Evaluating agents: component scorers, trajectory checks, and end-to-end harnesses

Why evaluation is the hardest part of agentic engineering

An agent that looks correct in a demo can fail silently in production. It may call the right tool most of the time but pass subtly wrong arguments. It may reach the right final answer by an unreliable path that will break on a slightly different input next week. Evaluation is the discipline that catches these failures before users do — and it is harder for agents than for traditional software because the surface area is large: you must check not just the final output but also the intermediate steps the agent took to get there.

The goal of this sprint is to make agent evaluation concrete. You will learn three diagnostic depths, understand the difference between deterministic and model-based scorers, and build a small eval harness that demonstrates these ideas in runnable code.

Three diagnostic depths

Evaluation of an agent system naturally decomposes into three levels, each of which catches different failure modes.

Component evaluation tests a single step in isolation — most commonly a single tool call. Given a specific input, did the agent call the right tool? Did it pass the right arguments? Component checks are narrow and fast. They can be run as unit tests and make regression detection cheap. Their limitation is scope: a system where every component passes can still fail at the system level if the components are composed in the wrong order.

Trajectory evaluation tests the path the agent took through a task — the sequence of steps, the reasoning that connected them, and whether that path makes sense. A trajectory check might ask: did the agent try to retrieve context before calling a write tool? Did it attempt a retry after a tool failure? Did it avoid calling a destructive tool before checking preconditions? Trajectory checks are more expensive than component checks and harder to write, but they catch a class of failure that component checks miss entirely: an agent that does the right things in the wrong order.

End-to-end evaluation tests whether the agent satisfied the original goal — ignoring how it got there. This is the most important check for correctness from the user's perspective, but also the most expensive and the hardest to make deterministic. The evaluation criterion for an open-ended task (summarise this document, plan this trip) is hard to express as code, which is where model-as-a-judge enters.

Deterministic scorers vs. model-as-a-judge

A deterministic scorer computes a pass/fail result through code alone, with no model in the loop. Examples include checking whether the agent called a specific tool by name, verifying that an argument matched an expected value, counting whether all required fields are present in a JSON response, or asserting that a file was written to the right path. Deterministic scorers are fast, free to run, fully reproducible, and produce no variance between evaluation runs.

A model-as-a-judge scorer sends the agent's output to a second language model and asks it to rate quality, accuracy, or relevance according to a rubric. The approach was formalised by Zheng et al. (2023), who introduced MT-Bench and Chatbot Arena as structured frameworks for using models to evaluate open-ended responses. Model-as-a-judge is appropriate when the correctness criterion genuinely cannot be expressed as code — for example, judging whether a generated explanation is accurate and clear, or whether a plan is reasonable given underspecified constraints.

The tradeoffs between the two approaches are substantial. Deterministic scorers are reproducible, cheap, and require no model quota. Model-as-a-judge scorers are flexible but introduce variance (the judge model's outputs are stochastic), add latency and cost, and raise the question of which judge model to use and how to calibrate its rubric. Eval frameworks such as DeepEval and OpenAI Evals provide infrastructure for both approaches, including built-in judge templates and metric libraries.

Tracing and regression

An eval harness is most valuable when it is run repeatedly — not just once after a change but continuously as part of a development loop. To support this, the harness must be able to capture a run (record the inputs, the agent's decisions, and the outputs), and re-run a fixed suite against a new agent version to detect regressions.

Observability tools such as LangSmith attach to agent frameworks and record each step of a run as a structured trace, which can then be replayed or compared across versions. Even without a dedicated observability tool, a well-structured eval harness can serve as a lightweight regression guard: if all cases pass today and one fails tomorrow after a code change, you know exactly where the regression is. That tight feedback loop is what makes iterative agent improvement tractable.

Anthropic's guidance on building effective agents emphasises this iteration discipline: evaluating agents on real representative tasks, fixing the failures you discover, and re-running the suite after each fix to confirm forward progress without regressing. The eval harness is the tool that makes this loop mechanical rather than manual.