How do you test an AI agent before launch?
Collect 20 to 50 real cases with a known correct result, run the agent on each several times, grade the outcome and the tool calls in the run log, and launch only at a pass rate agreed in advance.
The test loop before an agent goes live
This is the testing step inside our custom AI agent development service, written out in full. The method follows the evaluation guidance that model providers publish for their own agents: Anthropic's engineering team calls "20-50 simple tasks drawn from real failures" a great start (Anthropic, Demystifying evals for AI agents, January 9, 2026, checked October 30, 2026), and OpenAI's agent guide puts "set up evals to establish a performance baseline" as step one of choosing a model.
A few terms used on this page, as Anthropic defines them. A task (or test case) is "a single test with defined inputs and success criteria". Each attempt at it is a trial. A grader is "logic that scores some aspect of the agent's performance". The transcript, also called a trace, is the complete record of a trial: every model call, tool call and result.
What should an AI agent test set include?
Real past cases where the agent should act, matching cases where it should not, cases outside its job that must go to a person, edge cases such as empty or very long input, and every failure you have already seen.
A test set built only from easy cases tells you nothing. Anthropic's advice is to "test both the cases where a behavior should occur and where it shouldn't", because "one-sided evals create one-sided optimization". Its Claude documentation lists the edge cases to include: irrelevant or missing input, overly long input, harmful or off-topic user input, and ambiguous cases where even people disagree (Claude Docs, Define success criteria and build evaluations, checked October 30, 2026).
Here is how we split a 40-case starting set for an agent that reads inbound emails and updates a CRM. The proportions are our working practice, not a published standard; we change them to fit the job.
| Case type | Example for an email-to-CRM agent | Cases | What a pass means |
|---|---|---|---|
| Should act (common) | A new enquiry with name, service and budget | 12 | Contact created, fields correct, follow-up task set |
| Should not act | A supplier invoice, a newsletter, an auto-reply | 8 | No contact created, email labelled and left |
| Out of scope, hand to a person | A complaint, a refund request, a legal letter | 6 | Handed off with a short summary, nothing written |
| Edge cases | Empty body, a 30-page attachment, two people in one email | 6 | No crash, sensible handoff or a correct split |
| Known failures | Every case that went wrong in earlier testing | 8 and growing | Fixed, and stays fixed after every change |
Two rules make the set usable. Each case needs a written correct result, and Anthropic's test for a good case is one where "two domain experts would independently reach the same pass/fail verdict". If your own team cannot agree on the answer, the agent cannot be graded on it, so that case goes to the handoff group instead.
How do you grade an AI agent's answers and tool calls?
Use code checks for anything with one right answer, such as a field value or whether a tool ran. Use a model with a written rubric for tone and judgment. Use a person to check a sample, and to calibrate the model grader.
Anthropic's guide compares the three grader types. We use all three on most agents, each where it is strongest.
| Grader | Strengths (Anthropic) | Weaknesses (Anthropic) | What we grade with it |
|---|---|---|---|
| Code-based | Fast, cheap, objective, reproducible, easy to debug | Brittle to valid variations; lacking in nuance | CRM field values, which tools ran, output format, step and turn counts |
| Model-based | Flexible, scalable, captures nuance, handles open-ended tasks | Non-deterministic, more expensive than code, needs calibration with human graders | Reply tone, whether a summary is accurate, whether a handoff note makes sense |
| Human | Gold standard quality, matches expert user judgment | Expensive, slow, needs experts at scale | A sample of every run, and every case the model grader scored differently from last time |
Grade the outcome, check the path
Anthropic warns that "it's often better to grade what the agent produced, not the path it took", because agents find valid routes nobody planned for, and a test that demands one exact sequence of steps fails good runs. We follow that for the pass mark. We still read the path for two things: a required tool that was skipped (the agent claims it checked the calendar, but no calendar call is in the log), and an action that should never happen (a delete, a payment, an email to a customer without approval). Our guide to reading an n8n agent's execution log for skipped tools shows where that log lives in n8n.
Write the model grader's rubric like a checklist
A model grader is only as good as its rubric. Each criterion should be a yes or no question about the transcript ("Did the reply state the appointment time from the calendar result?"), never "rate the quality from 1 to 10". The Claude documentation adds that it is "generally best practice to evaluate with a different model from the one used to generate the evaluated output". So if the agent runs on one model, the grader runs on another.
How many times should you run each test case?
At least three times for any agent that customers rely on, because models are not deterministic. Report pass^k, the share of cases the agent gets right on every trial, which falls fast: 90% per trial becomes 59% over five trials.
Chance of passing all 5 trials (pass^5)
Anthropic defines two measures. pass@k "measures the likelihood that an agent gets at least one correct solution in k attempts". It suits tools where a person picks the best of several drafts. pass^k "measures the probability that all k trials succeed", which is what matters when the agent acts alone on every customer. Anthropic's own example: a 75% per-trial success rate over three trials gives (0.75)³, about 42%.
We ran the same arithmetic for the success rates we see during builds. pass^k is the per-trial rate to the power k; pass@k is 1 minus the failure rate to the power k.
| Per-trial success | pass^3 | pass^5 | pass^10 | pass@3 (for comparison) |
|---|---|---|---|---|
| 80% | 0.8³ = 51.2% | 32.8% | 10.7% | 99.2% |
| 90% | 0.9³ = 72.9% | 59.0% | 34.9% | 99.9% |
| 95% | 0.95³ = 85.7% | 77.4% | 59.9% | Over 99.9% |
| 99% | 0.99³ = 97.0% | 95.1% | 90.4% | Over 99.9% |
The same 90% agent looks finished on pass@3 (99.9%) and unfinished on pass^5 (59%). For an agent that books appointments or writes to your CRM without review, the pass^k column is the honest one. It is also why we keep a person approving outside actions at launch: approval covers the gap between a good average and every single run.
One practical rule from the same guide: "each trial should be 'isolated' by starting from a clean environment". If trial two can see the contact trial one created, the second result is meaningless. We run tests against a sandbox CRM or a test sub-account, reset between trials.
Which results show an AI agent is ready to launch?
Agree the launch gate before testing starts: a pass rate for normal cases, zero forbidden actions, correct handoffs on out-of-scope cases, valid structured output, and a measured cost and step count per case that fits the budget.
The thresholds depend on the job and the risk, so they are set with the owner of the process on the free strategy call. This is the gate we write into the plan; the example figures are illustrative for a CRM-writing agent with approval on at launch.
| Gate | How it is measured | Example threshold (illustrative) |
|---|---|---|
| Task success | Code checks plus model rubric, over 3 trials | pass^3 at or above the level the owner signs off |
| Forbidden actions | Code check on the transcript for delete, pay, send without approval | Zero, on every trial |
| Handoff | Out-of-scope cases handed over, not answered | All out-of-scope cases, every trial |
| Output shape | Structured fields present and valid | All runs that reach the write step |
| Cost and steps | Tokens and model calls per case from the provider logs | Inside the running-cost budget in the plan |
| Known failures | The failure group of the test set | All fixed before launch |
Anthropic separates capability evals, which "should start at a low pass rate" because they measure what the agent cannot yet do, from regression evals, which "should have a nearly 100% pass rate". Before launch, most of the set is capability work. After launch, the cases the agent passes reliably "graduate" into the regression suite that runs before every change.
What tools can you use to test an AI agent?
n8n has built-in Evaluations with a test dataset in a data table or Google Sheet. OpenAI offers trace grading on agent runs. Code-based agents can use any test runner plus a model grader.
- n8n Evaluations. n8n describes two levels: light evaluation during the build with a small hand-picked set, and metric-based evaluation with a larger dataset, often from production executions, run as regression tests. Datasets live in "data tables or Google Sheets". Built-in metrics are Correctness (AI-based), Helpfulness (AI-based), String Similarity, Categorization and Tools Used. Metric-based evaluation needs the Pro or Enterprise cloud plan or self-hosted Enterprise; Community and Starter users can use it on a single workflow (n8n Docs, metric-based evaluations, checked October 30, 2026). Most of our agents run on n8n automation builds, so this is where their test sets usually live.
- OpenAI trace grading. OpenAI defines a trace as "the end-to-end record of model calls, tool calls, guardrails, and handoffs for one run", and graders score those traces "to find regressions and failure modes at scale" (OpenAI, Agent evals, checked October 30, 2026).
- Code agents. For agents built on the OpenAI Agents SDK, the Claude Agent SDK or plain API calls, the test set is a file in your repository, the code checks are ordinary tests, and the model grader is one more API call per case. If the system uses several agents, test each agent alone and then the whole chain; our comparison of single agent vs multi-agent systems covers when a split is worth that extra testing.
Whatever the tool, Anthropic's blunt advice applies: "Read the transcripts!" A grader that passes a run you would have failed is a bug in the test, and you only find it by reading.
How much does it cost to test an AI agent?
Very little in model fees. On Anthropic's prices read October 30, 2026, a 50-case set run three times costs about $10.50 per full test run on Claude Sonnet 5.5 with an Opus 5.5 grader, and under $2 on Claude Haiku 5.5.
Our arithmetic, from the Claude pricing page (checked October 30, 2026). Assumptions, labelled: each case uses 20,000 input and 1,200 output tokens, the same "workload A" CRM agent used on our AI agent development page; each model grade reads 3,000 input tokens and writes 300; 50 cases x 3 trials = 150 runs and 150 grades.
| Line | Price per million tokens (in / out) | Arithmetic | Per full test run |
|---|---|---|---|
| Agent on Claude Sonnet 5.5 | $2 / $10 | 150 x (0.02 x $2 + 0.0012 x $10) = 150 x $0.052 | $7.80 |
| Grader on Claude Opus 5.5 | $4 / $20 | 150 x (0.003 x $4 + 0.0003 x $20) = 150 x $0.018 | $2.70 |
| Total, Sonnet agent | $7.80 + $2.70 | $10.50 | |
| Agent on Claude Haiku 5.5 | $0.10 / $0.50 | 150 x (0.02 x $0.10 + 0.0012 x $0.50) = 150 x $0.0026 | $0.39 |
| Grader on Claude Sonnet 5.5 | $2 / $10 | 150 x (0.003 x $2 + 0.0003 x $10) = 150 x $0.009 | $1.35 |
| Total, Haiku agent | $0.39 + $1.35 | $1.74 |
The grader calls do not depend on each other, so they can go through Anthropic's Batch API, which the same page lists at "a 50 percent discount on both input and output tokens". That takes the Opus grader line from $2.70 to $1.35. Even 20 full runs during a build cost about $210 on the Sonnet setup. The real cost of testing is the hours spent writing correct answers for each case, which is why we ask for real past cases with the outcome your team chose.
How do you keep testing an AI agent after launch?
Rerun the regression set before every prompt, tool or model change, add each real failure from production as a new case, and review a sample of live transcripts on a schedule. Production monitoring alone finds problems only after customers meet them.
Anthropic's comparison of methods is direct: production monitoring "reveals real user behavior at scale" but is "reactive; problems reach users before you know about them", and user feedback is "sparse and self-selected". Automated tests catch the problem first; live monitoring tells you which new cases to add. The trigger we see most often is a provider change: a model retirement or a removed API parameter, covered in the maintenance section of our AI agent build page. The test set is how the replacement model is proven before the switch. Our page on support after an automation build explains what happens in the first 6 months, when fixes to anything we built are free.
More questions about testing AI agents
What is an AI agent eval?
An eval is a repeatable test of an AI agent: a set of cases with known correct results, a way to run the agent on each one, and graders that score the outcome. The word comes from the model providers' own documentation. It differs from a single manual check because it runs the same way every time, so you can compare the agent before and after a change.
How do you test AI agent accuracy?
Write the correct result for each test case first, then compare the agent's output against it with a code check where there is one right answer and a model rubric where judgment is involved. Run each case at least three times and report the share passed on every trial. Accuracy on one run hides the cases the agent gets right only some of the time.
Can I test an AI agent without writing code?
Yes. In n8n, Evaluations take a test dataset from a data table or Google Sheet and score runs with built-in metrics such as Correctness and Tools Used. A spreadsheet of cases with the expected result, plus someone reading the run logs, is also a valid first test. Code helps once the set grows past what a person can check by hand.
How do you test a voice AI agent?
The same way, with the call transcript and the tool log as the record. Script test calls for each case type, including callers who interrupt, give a wrong date or ask for a person, and check what the agent booked or wrote. Our guide on building an AI receptionist covers test calls for voice agents step by step.
Should you stress test an AI agent?
Yes, for two reasons. Load: send the volume you expect at peak and check rate limits, timeouts and cost per case. Hostile input: include messages that try to change the agent's instructions or get it to act outside its job. The pass mark for both is the same as for normal cases: no forbidden actions, and a handoff when unsure.
Who should write the test cases?
The people who do the job today. They know the common cases, the strange ones and the correct outcome for each. We draft the structure and the grading, and the client's team supplies or approves the expected results. A test set written only by the builder tends to test what the builder already expected the agent to do.
How long does testing an AI agent take?
For a small agent, building the first test set and running it takes a few days inside the build, most of it spent gathering past cases and agreeing correct answers. Each rerun after a change takes minutes once grading is automated. Testing time grows with the number of case types and the risk of the actions the agent takes.
Sources and further reading
- Anthropic Engineering, Demystifying evals for AI agents (January 9, 2026: task, trial, grader definitions; grader types; pass@k and pass^k; 20-50 tasks; capability and regression evals), checked October 30, 2026
- Claude Docs, Define success criteria and build evaluations (eval design principles, edge cases, grading with a different model), checked October 30, 2026
- OpenAI, Agent evals (traces and trace grading), checked October 30, 2026
- OpenAI, A practical guide to building agents (evals as the first step in model choice), checked October 30, 2026
- n8n Docs, Evaluations overview and metric-based evaluations (datasets, built-in metrics, plan availability), checked October 30, 2026
- Claude pricing (Sonnet 5.5, Opus 5.5, Haiku 5.5 rates; Batch API discount), checked October 30, 2026
