Evaluating Agents Means Testing the Whole Loop

An agent can produce a fluent final answer while taking unsafe intermediate actions. That is why evaluation must observe the loop: what it retrieved, which tools it selected, whether it respected authority boundaries, how it handled conflicting evidence, and whether it escalated when required.

I would build scenario suites around operational failures rather than only happy paths: a missing patient field, a stale market data snapshot, a malicious instruction embedded in a document, a tool timeout, and a task that exceeds the agent’s permission. The expected outcome may be a useful draft, a safe abstention, or a clear handoff—not always completion.

The NIST framework gives this work a vocabulary. Governance establishes ownership and risk tolerance; mapping identifies affected people and dependencies; measurement collects repeatable evidence; management acts on findings (Tabassi). The model’s benchmark score is one input to that larger system.

For high-stakes medicine and finance, I prefer releases that increase authority gradually. Begin with read-only research, then proposal generation, then carefully bounded actions. Each expansion should be earned with evidence from the prior stage.

I separate task success from safe behavior in the scorecard. An agent that completes a task by using an unauthorized tool or accepting a malicious instruction has not passed. Scenario tests should therefore assert intermediate tool choices, data boundaries, and escalation behaviour as first-class outcomes.

Human evaluation remains necessary for cases where the right answer depends on nuance, but it should be structured. Reviewers need written rubrics, blinded comparisons where practical, and a route to mark unsafe or unsupported behavior. Those annotations become a valuable regression set for the next release rather than a pile of anecdotes.