A good demo proves that an agent can succeed once. An evaluation asks whether it succeeds for the right reasons, fails safely and keeps working after the system changes.
That difference is the gap between a prototype and a production release.
Start with the operating outcome
Do not begin with “Was the answer good?” Begin with the job.
For an onboarding agent, the outcome may be a complete account record, the required documents attached and an exception routed to the correct owner. For an account research agent, it may be a sourced brief added to the right CRM record without changing protected fields.
Each test should define:
- the starting state
- the available data
- the tools the agent may use
- the expected action or refusal
- the observable final state
- the reason a person should intervene
This makes evaluation about work completed, not how persuasive the output sounds.
Build five kinds of cases
Normal work
Use representative cases the system should complete without help. These establish the baseline path and confirm that integrations, permissions and output formats work together.
Missing information
Remove a required field, document or system response. The agent should request what it needs or route the task rather than inventing an answer.
Conflicting information
Give the agent two sources that disagree. Test whether it follows the agreed source hierarchy and shows the conflict to an operator.
Unsafe or disallowed work
Ask for an action outside the role, including a write to an unapproved system or a decision reserved for a person. A refusal is a successful outcome here.
Tool and system failure
Return a timeout, duplicate response or partial write. The agent should not silently repeat a consequential action. Recovery, cancellation and human handoff are part of the product.
Grade the trace, not only the final answer
The final result can look correct even when the process was unsafe. Review the steps that produced it:
| Dimension | What to inspect |
|---|---|
| Completion | Did the required business state change? |
| Grounding | Did the agent use approved and current sources? |
| Tool choice | Did it call the right tool with valid inputs? |
| Policy | Did it respect permissions and approval rules? |
| Efficiency | Did it finish without unnecessary loops or cost? |
| Handoff | Did it stop and route the case at the right boundary? |
IBM's 2026 study of production agents found that reliability remains the leading development challenge and that 74% of practitioners still depend primarily on human evaluation. That does not make automated evaluation unimportant. It means the evaluation system must combine repeatable checks with domain judgment.
Use automated checks where the answer is objective
Automated evaluation works well for questions such as:
- Was the correct record updated?
- Were all required fields present?
- Did the agent call a forbidden tool?
- Did it ask for approval before the write?
- Did the task exceed the allowed number of steps?
- Did the final output match the required schema?
Model based judges can help with softer criteria, but they should be calibrated against human review. A judge prompt that has never been compared with an operator's decision is another untested model in the system.
Turn every incident into a regression test
Production creates cases the design team did not imagine. When an operator finds a bad handoff, an incorrect tool call or an ambiguous policy decision, preserve the case and add it to the evaluation set.
The release cycle becomes:
- reproduce the failure
- add the case to the evaluation set
- change the prompt, tool or workflow
- run the full set
- review the affected traces
- release with a rollback path
This is how an agent becomes more reliable without fixing one incident and creating another elsewhere.
Define the release threshold before testing
A team should agree what must be true before the agent touches production. The threshold can include a minimum completion rate, zero forbidden actions, full success on high consequence cases and a maximum handoff error rate.
OpenAI's September 2026 Agents API announcement describes infrastructure for long running work, environments, tool use and subagent coordination. It also includes customer examples where improved harnesses and evaluation raised scores or reduced failures. The lesson is not that infrastructure guarantees quality. It is that production quality comes from the harness, environment and evaluation working together.
The evaluation set is the contract between what the team says the agent does and what the system can prove before release.