Agent Evaluation Readiness Checklist
Agent evaluation feels like it should be straightforward until you actually try to build it. The problem isn't tooling or infrastructure, it's that most teams skip the unglamorous work of understanding what they're actually measuring. You end up with eval pipelines that produce numbers without insight, or worse, numbers that actively mislead you about system quality.
The first mistake is building automation too early. Before writing any eval code, manually review 20 to 50 real agent traces. Not summaries, not logs, actual full traces showing tool calls, reasoning steps, and outputs. This sounds obvious but most teams skip it because it's tedious. Those 30 minutes of trace review will surface failure patterns that no automated system would catch initially. You'll see that your agent isn't hallucinating, it's getting malformed API responses. Or that "reasoning failures" are actually ambiguous prompts. The Witan Labs team discovered a single data extraction bug that explained a 23 point benchmark gap. Infrastructure issues constantly masquerade as agent intelligence problems.
Success criteria need to be unambiguous before you write a single eval. "Summarize this document well" is not a success criterion. "Extract the 3 main action items from this meeting transcript, each under 20 words, including owner if mentioned" is. If two domain experts can't agree on pass or fail, the task definition is broken. This matters more for agents than traditional ML because agent outputs are compositional, you're evaluating tool selection, parameter generation, and final response quality simultaneously.
Separate capability evals from regression evals explicitly. Capability evals measure progress on hard tasks you expect to fail initially, they should start with low pass rates and give you a hill to climb. Regression evals protect existing behavior and should maintain near 100% pass rates. Without this separation, teams either stop improving because they only guard existing behavior, or ship regressions because they only chase new capabilities. These require different datasets, different thresholds, and different response workflows.
Spend 60 to 80% of eval effort on root cause analysis, not automation. When something fails, categorize it: prompt problem, tool design problem, model limitation, tool execution failure, data pipeline issue. Keep an error taxonomy and update it as you review more traces. If you can't articulate why each failure occurs, you don't understand your system well enough to eval it meaningfully. Prompt problems need prompt fixes. Tool design problems need interface redesigns. Model limitations might need architecture changes or different models entirely. These require completely different interventions.
Start with trace-level evals, not run-level. A trace covers one complete agent turn: user input through final response. Grade it on three dimensions: final response correctness, trajectory reasonableness, and state changes. That last one is critical and often missed. If your agent schedules meetings, don't just check that it said "Meeting scheduled." Query the calendar API and verify the event exists with correct time, attendees, and description. The agent can claim success while the actual state is wrong.
Run-level evals, which test individual tool calls, are useful for debugging but brittle during development. If you're still changing tool definitions weekly, run-level evals break constantly. Thread-level evals covering multi-turn conversations are the hardest to implement. A practical pattern is N-1 testing: take real conversation prefixes from production and have the agent generate only the final turn. This avoids compounding errors from fully synthetic multi-turn simulations.
Dataset construction matters more than eval code quality. Every task needs a reference solution proving it's solvable. Test both positive cases where behavior should occur and negative cases where it shouldn't. If you only test "did it search when it should," you'll optimize for overuse. Source examples from dogfooding errors, production traces, and hand-written behavior tests. Set up a trace-to-dataset flywheel where production failures automatically seed new eval cases.
Assign eval ownership to a single domain expert, not a committee. Someone needs to maintain datasets, recalibrate LLM judges, triage new failure modes, and make judgment calls on ambiguous cases. Eval quality degrades fast without clear ownership because datasets become stale and failure taxonomies stop evolving.
The core insight is that agent evaluation is mostly error analysis, not infrastructure. Teams that succeed spend more time understanding failure modes than building eval pipelines. The automation comes after you know what you're measuring.