How we evaluate AI agents across 1,000+ production scenarios

When you’re running AI agents in production - agents that call tools, follow multi-step operating procedures, and make real decisions - you need to know two things: are they making the right decisions, and could a different model make better ones?

Generic benchmarks can’t answer either question. Our agents process invoices, reconcile financial statements, and handle service requests across ~200,000 workflow runs per week - each run being one end-to-end agent execution on a real task. There’s no simple string to match against - the correct answer is a sequence of tool calls and a routing decision that a human reviewer would approve. To evaluate this, we built a replay-based framework with three phases:

1. Replay 1,000+ production scenarios with candidate models
2. Grade outcomes with an LLM judge
3. Compare models per workflow

Why Agent Evaluation Is a Different Problem

Our agents aren’t chatbots generating text. They’re making decisions that directly affect financial accuracy, operational efficiency, and homeowner experiences. A wrong GL code may mean money posted to the wrong account or a missed duplicate may mean paying the same vendor twice. A bad routing decision means requests may be stuck in the wrong approval queue, or an urgent request sitting unread.

At ~200,000 workflow runs per week, even a small accuracy drop compounds quickly. A 2% regression across a single high-volume workflow means thousands of wrong decisions per week - each one requiring human intervention to fix. And these agents don’t stay static. Model providers ship updates on their own timelines, prompts evolve as business rules change, and tool schemas shift as integrations mature. Every change is a potential regression, and manual QA can’t keep up.

Standard LLM benchmarks test models on isolated, generic tasks. Agent evaluation tests a fundamentally different thing: whether the full system - model + tool schemas + middleware + retry logic - produces outcomes a human would approve. The middleware matters because production infrastructure - retry logic, prompt caching, tool-call sequencing - directly shapes what the model sees and how it responds.

The failure modes are agent-specific:

These are real failures from our evaluation runs, but none would surface in a model benchmark.

Without robust evaluation, the risks are predictable: a faster or cheaper model that passes initial testing on straightforward cases can degrade on the edge cases that only emerge at production scale: threshold boundaries, ambiguous vendor matches, overlapping business rules. Limited test coverage won’t catch these until production complaints surface, and by then the damage compounds into customer escalations and internal stress.

Knowing what can go wrong is only half the problem. Before building the evaluation pipeline, the first question is: which scenarios do we test against?

Mining Scenarios From Production

Of course, we don’t write test scenarios by hand. We mine them from production data using a two-pronged approach: top-down from the operating procedure, and bottom-up from actual workflow runs.

Top-down: AOP analysis. Every workflow has an Agent Operating Procedure (“AOP”) - the decision tree the agent follows. We analyze each AOP to extract every decision branch, condition, and rule variation. For invoice processing, this means every routing path (approve, escalate, flag as duplicate, void), every payment type (check, ACH, credit card), every entity matching scenario (exact match, alias match, not found), every GL code determination method (from contract, historical, budget). If the AOP contains 20 distinct conditions, the framework automatically generates a test case for each one - not one generic category that summarizes them.

Bottom-up: Production run clustering. We query the distribution of real routing outcomes and cluster actual workflow runs by behavior pattern. This catches scenarios the AOP doesn’t explicitly describe - real-world edge cases that emerge from production traffic at scale. For example, production data revealed that 12% of service requests involved repeat issues at the same property - a pattern the operating procedure didn’t explicitly address, but one the agent needs to handle correctly. We cross-reference these clusters against the top-down analysis to fill coverage gaps.

Consolidation. The two analyses merge into a robust test suite of 1,000+ automatically generated scenarios - each characterized across the dimensions that matter for that workflow type. This granularity lets us root-cause failures at scale: not “fails on invoices” but “fails on multi-line-item invoices with unmatched vendors.”

For invoice processing alone, this produces 174 test cases across 4 routing categories - and that’s just one workflow type. Each references a real production run:

{
  "id": "wfrun_abc123",
  "test_case_id": "TC-1.1.1",
  "workflow_name": "Categorize Invoice",
  "use_case": "Approve - Check payment - 1 line item",
  "category": "Approve",
  "dimensions": {
    "routing": "approve_invoice",
    "payment_type": "check",
    "line_item_count": 1,
    "association": "found_exact",
    "provider": "found_exact"
  }
}

We started with invoice processing and expanded across our full platform - financial reconciliation, service request routing, owner communications, and email triage - covering the decision types that directly impact community management operations. A full evaluation run covers 1,000+ scenarios with edge cases deliberately overrepresented since that is where models diverge. When we expanded coverage beyond straightforward cases to include rejection edge cases, threshold boundaries, and entity matching failures, one model’s overall pass rate dropped from 91% to 80%. The 91% had been misleading - carried by straightforward cases that every model handles. Broader coverage exposed the real accuracy on the decisions that matter. The hard cases are where evaluation earns its keep.

The Replay-Grade-Compare Pipeline

With scenarios defined, the next step is running them. The pipeline has three stages, each designed to answer a specific question:

1. Can the model do the job?
2. Did it do it well?
3. Which model should we use where?

Replay

Take a production workflow run ID and a candidate model config. The system creates a fresh agent session with the same starting context, tool schemas, and operating procedure as the original run, then lets the candidate model execute.

Replays run in parallel with configurable concurrency and timeouts. The agent runs the full agentic loop: real tool schemas, multi-step execution, and the same middleware stack as production. We’re not benchmarking the model in isolation, rather we’re benchmarking the deployed agent.

This distinction matters. Models that perform well in isolation can behave differently in the full pipeline where production concurrency, middleware transformations, and real infrastructure constraints come into play.

Grade

Each completed replay is sent to an LLM judge - a reasoning model with high reasoning effort. The grader receives the full execution trace: system prompt, tool definitions, every message, every tool call and result.

Scoring dimensions:

When running multiple graders simultaneously, grades are combined conservatively: pass requires unanimous agreement, scores take the minimum, issues are the union.

Designing a Grader That Doesn’t Over-Fail

The grading prompt is where most evaluation frameworks break. Our first version was strict: check every step against the procedure in order. Result: a flood of false failures. An agent that checked for duplicates before verifying the vendor would fail, even though the final routing decision was correct. It was grading the sequence of steps, not the outcome.

We redesigned the grader around three principles:

1. Default to success. The grader starts by assuming the agent succeeded and only marks failure on clear evidence of a wrong outcome. This matters because operating procedures are written for humans, not scoring rubrics - they contain implicit judgment calls and alternate valid paths. A strict sequence-checker would penalize correct decisions that simply took a different route. Before marking failure, the grader must pass a three-question gate: Would a human reject this? Would approving it cause actual harm? Is this a genuinely wrong outcome, not process nitpicking?

2. Evaluate outcomes, not process. Operating procedures contain steps labeled “CRITICAL” and “MANDATORY.” These are guidance for the agent, not absolute requirements for the grader. Agent skipped a “MANDATORY” step but reached the correct decision? Pass, with a minor issue noted. This is a deliberate design choice in our grading prompt. Real procedures - written by humans, for humans - have overlapping and sometimes conflicting rules. Even experienced reviewers disagree on whether a specific step was “required.” Choosing one valid path over another is judgment, not error.

3. Issue taxonomy as diagnostic tool. When something goes wrong, we need to know what went wrong. Each issue is categorized (wrong_routing, wrong_field_value, skipped_step, misapplied_rule) and given a severity (defaulting to minor). This turns evaluation into a diagnostic tool, so that we can improve schema, prompting and test scenarios.

This redesign reduced false failures significantly and made results actionable. When a scenario fails now, it’s a real problem worth investigating.

Compare

The analyzer runs the same scenario set across candidate models and calculates per-workflow statistics:

Invoice processing (174 scenarios)

The key insight from this table: the two models fail on different scenarios. Only 8-15% of scenarios show a clear winner; the rest are ties. Across other workflow types, this pattern holds: neither model dominates everywhere. Understanding which scenarios each model wins is what drives per-workflow model selection.

Per-Workflow Model Selection

No single model wins everything. The model best at processing structured data with clear rules might not be best at classifying ambiguous text. The model with strongest reasoning might be 4x too expensive for simple routing.

The eval pipeline produces per-workflow recommendations, and each workflow runs its own model configuration independently. In practice, this means different models running on different workflow types, selected by data, not assumption.

Beyond Pass Rate

Pass rate is the headline metric, but production deployment decisions require a multi-dimensional view.

What We Learned

Across 1,000+ scenarios spanning use cases, the answer wasn’t “pick the best model;” rather, it was “pick the best model for each job.”

The real value wasn’t just model comparison. The structured grading output gave us actionable feedback on why scenarios failed. We used that feedback to iteratively improve prompts, tuning how models handled entity matching, GL code determination, and edge cases. Over successive evaluation rounds, a cheaper model’s accuracy improved to match the incumbent on workflows where it had initially lagged behind - turning evaluation into a continuous improvement loop, not just a pass/fail gate.

Five things we’d tell any team building agent evaluation:

1. Test the full agent loop, not the model in isolation. The middleware, tool schemas, and retry logic all affect outcomes. A model that aces benchmarks can fail in your pipeline.
2. Mine scenarios from production data, don’t invent them. Real traffic reveals edge cases no test designer would think to write.
3. The grading prompt is as important as the agent prompt. Our first grader produced a flood of false failures. Designing it to evaluate outcomes rather than process was the breakthrough.
4. Model selection is per-workflow, not global. Neither model dominates - they fail on different scenarios, and the evaluation framework is how we know which to trust with which decision.
5. The framework is what scales, not individual tests. New workflow? New model? Same pipeline. We run this before any model update, prompt change, or configuration adjustment reaches production. 200k workflow runs per week across 200 customers depend on it.

If building evaluation infrastructure for AI agents at scale sounds interesting, we’re hiring.