How to Test and Evaluate AI Agents Before They Touch Real Customers

To test an AI agent, build a set of 20 to 50 real work items with known good outcomes, run the agent on each one several times, score the results with a mix of code checks and human review, and rerun the whole set every time you change a prompt, tool or model. A demo cannot do this job. It shows the agent succeeding once, on a case someone chose because it worked. An eval set shows how often it succeeds on the cases you did not choose.

This guide is for whoever has to decide whether an agent is safe to switch on. It covers the eval set, scoring, regression runs, shadow mode and production monitoring. Terms follow Anthropic's engineering write-up on agent evals, which is worth reading alongside this.

Why do agent demos mislead?

Three things make a demo look better than the product will be.

• Selection. The demo case went well in rehearsal. Real traffic includes three questions in one message, the invoice scanned sideways, and the order that matches two records.
• Single trial. Agents are not deterministic. The same input can take a different path on the next run. One success tells you little about the second or the twentieth.
• Success is judged by feel. A fluent reply reads as correct. Whether the refund was actually issued, or the right record updated, only shows up if somebody checks the system, not the chat window.

Anthropic makes the second point in numbers. If an agent succeeds 75% of the time on each attempt, the chance it succeeds three times in a row is about 42% (0.75 cubed). For anything customer-facing, that gap matters.

Where does the eval set come from?

From your own work, not from imagination. The best source is the history of the job you want the agent to do. Anthropic's advice is to start small, with around 20 to 50 tasks drawn from real failures, because early changes tend to have large, visible effects. You do not need hundreds to begin.

Four places to collect items

1. Past tickets, emails or orders where you know what the correct handling was. Ask the person who did the job to pick a spread: easy, awkward, and "I had to check with my manager".
2. Corrections. Every time a human fixes the agent, that case becomes a test item.
3. Edge cases you can name. Missing data, a customer writing in another language, a request outside policy, two matching records, an attachment that will not open.
4. Cases where the agent should do nothing. A message that is spam, a question it has no right to answer, a refund above its limit. Test when a behaviour should happen and when it should not, otherwise the agent is tuned toward doing more and more.

What one eval item looks like

Keep each item boring and unambiguous. Here is an illustrative example for a support-triage agent. It is a worked example, not a real client case.

Each item should have a reference answer that a person agrees is right. If an item scores 0% across many trials, suspect the item first. Anthropic reports a case where fixing grading bugs and unclear task specs lifted a measured score from 42% to 95% without the agent changing at all, so read the transcripts before trusting a number.

What should you score?

Judge the outcome and the process separately. Outcome is the final state of the world: does the refund request exist, is the CRM row updated, was the right person alerted. Process is how it got there: which tools it called, with what arguments, and whether it respected the rules. There is usually more than one valid path to a correct outcome, so score the result, and use process checks only for things that must or must not happen.

Escalation is a score, not an afterthought

Two different mistakes hide under "escalation". A missed escalation is the agent pressing on when it should have stopped: the dangerous one. An unnecessary escalation is the agent handing over something it could have handled: expensive, but safe. Track them as two numbers. Most teams want missed escalations pushed close to zero first, then accept some unnecessary ones until the agent proves itself. The approval side of this is covered in our post on human-in-the-loop approvals for AI agents.

Three kinds of grader

Anthropic describes three. Code-based graders (string match, database check, tool-call check) are fast, cheap and repeatable, but brittle for nuanced output. Model-based graders (a second model scoring against a rubric) handle open-ended text but are not deterministic, cost more and need calibrating against humans. Human graders are the gold standard and slow. Use code wherever there is a right answer, a model grader for tone and completeness, and a weekly human sample to keep it honest.

How many times should each test run?

More than once. Run every item several times, for example five, and record how many runs pass. Two measures help here, both from the Anthropic write-up. pass@k is the chance of at least one success in k attempts; it suits cases where one good result is enough, like a coding assistant you can retry. pass^k is the chance that all k attempts succeed; it suits customer-facing agents where consistency matters. For a public-facing agent, watch the second one. An item that passes three runs out of five is not "mostly fine"; it is an item that fails 40% of the time and you have not fixed it.

How do you stop changes from breaking what worked?

Treat the eval set as a regression suite. Changing a prompt to fix one case routinely breaks another, and changing the model underneath shifts behaviour in ways nobody predicted. The rule is simple: after any of the following, rerun the full set and compare against the last run before it ships.

• A prompt or system-instruction edit
• A new, removed or reworded tool (tool descriptions steer behaviour as much as prompts do)
• A different model, or a provider version update
• A change to the knowledge the agent searches, such as new policy documents

Keep a small results table per run: pass rate overall, pass rate per tag, missed escalations, average cost per task. When a number drops, open the failing transcripts and read them. Anthropic stresses this step: reading transcripts tells you whether the agent made a real mistake or the grader rejected a valid answer. Failures should look fair to you. If they do not, fix the test.

Expect the set to age. Once an item always passes it tells you little, so keep adding fresh cases from production.

What is shadow mode and why run it first?

Shadow mode means the agent runs on live inputs but its actions are not executed. It proposes, a human does the real work as usual, and you compare the two. It is the closest thing to a real test that carries no customer risk, and it gives you the thing an eval set cannot: a sample of what real traffic actually looks like.

A simple rollout ladder

1. Offline evals. The set you built, until it passes at a threshold you wrote down in advance.
2. Shadow. Live traffic, no side effects. A person scores a sample of the proposed actions. Add any surprises to the eval set.
3. Assisted. The agent drafts, a person approves every action. Track how often people edit or reject.
4. Limited autonomy. The agent acts alone on the lowest-risk categories only; everything else still needs approval.
5. Widen slowly. Move a category up only when its numbers justify it, and keep the ability to move it back.

Write the pass thresholds before you start, such as "no missed escalations on the safety-tagged items and at least a stated pass rate on the rest". Setting the bar after seeing results lets weak agents through.

What should you monitor once it is live?

Evals tell you whether the agent is ready; monitoring tells you whether it is still ready. Anthropic recommends layering automated evals, production monitoring and periodic human review, because each catches things the others miss. Production monitoring reveals real behaviour at scale but has no ground truth by itself, so you pair it with sampled review.

Log every run in full: the input, each tool call with arguments, the outputs and the final result. Without a transcript you cannot tell a real failure from a bad test. It also matters for safety: our post on AI agent security and prompt injection explains why those logs are your evidence when something goes wrong.

How big is a first eval, in practice?

Small. A first useful version is a spreadsheet of 25 to 40 rows with an expected outcome for each, a script that runs the agent five times per row, and a one-page results summary. Writing down what "correct" means often exposes unclear policies that would have confused a human team too. On an automation platform, keep a saved list of test inputs and run it after each edit; in custom code, make the eval run part of deployment. If you want help setting this up around a real build, see our work on agentic AI development and AI automation with n8n.

Common mistakes when testing agents

• Grading the chat, not the system. A friendly "done!" is not an updated record.
• Testing only happy paths. Half the set should be awkward, missing data, or "should refuse".
• One run per item. You will mistake luck for reliability.
• A grader nobody checked. Sample its verdicts against a human every so often. Anthropic's multi-agent research team used a model judge but found that human testing caught issues automation missed.
• Tuning to the test. Keep a held-back batch you never tune against, or the set stops predicting anything.
• No owner. Someone has to be named for reading the weekly sample, or it quietly stops.

Frequently asked questions

How many test cases do I need to evaluate an AI agent?

Start with 20 to 50 real cases. That is enough to see large problems and judge the effect of changes. Grow the set from production failures over time. More cases help later, but a small set that you actually run beats a large one that you never finish.

Can another AI model grade my agent's work?

Yes, for open-ended output like tone or completeness, but it must be calibrated. Compare its scores to a human's on a sample, and use code checks wherever there is a clear right answer, such as a database field or a tool call.

What is the difference between an eval and monitoring?

An eval runs a fixed set of known cases to tell you whether the agent is ready or whether a change made it worse. Monitoring watches real traffic after launch. Evals give you ground truth but may not match real use; monitoring matches real use but has no answer key. You need both.

How long should shadow mode last?

Long enough to see a representative range of inputs, including the slow days and the odd ones. That depends on your volume, so set the exit condition as numbers (cases reviewed, rates reached) rather than a calendar date.

Do I need to re-test when the model provider updates the model?

Yes. A model update can change behaviour in small ways even when your prompts and tools are untouched. Rerun the eval set when a version changes and compare to the previous run before switching over.