Traditional Tests Pass, but How to Prove the Agent Is Actually Better

When building ordinary software, the questions tests need to answer are often clear.

Is the balance sufficient? What does the API return? What result should a function produce given fixed inputs? Boundaries are written into code, expectations into assertions. As long as unit tests, integration tests, and end-to-end tests still pass after implementation changes, we at least know the original behavior hasn't been broken.

This approach remains important. The problem arises when Agents start performing continuous tasks on behalf of users.

When a Coding Agent is asked to fix a bug, it might read the relevant code and start working, or it might search several more files, modify the code, and then adjust based on test results. Sometimes it finds insufficient information and stops to ask the user for clarification. All three paths could complete the task, and all could also go astray midway.

If tests predefine which tools it must call, which lines it must modify, and which steps it must follow, the tests end up hard-coding a feasible path. What's actually measured is whether the Agent followed a preset script, not whether the task was done well.

Pass and Fail Are No Longer Enough

Traditional tests excel at judging deterministic outcomes. If the API returns 200, the test passes. If it returns 500, the test fails.

Agent results often require comparison.

For the same refactoring task, one version changes ten files, another changes five, and yet another changes only the core files. All three versions might run correctly, and existing tests might all pass. Next, you need to consider whether the code is maintainable, whether the scope of changes is reasonable, and whether new risks have been introduced.

Traditional tests can continue to check syntax, types, interfaces, and regressions, but they struggle to make these comparisons using assertions alone.

The execution process has also become longer. An Agent might first analyze the project, then design a solution, then modify code, run tests, and handle failures—the entire task spans many rounds. Running the same input multiple times may yield different steps and tool choices. Checking whether a single run followed a fixed path says little about its overall capability.

Code Isn't Broken, but Capability May Have Regressed

Suppose before a prompt modification, the task success rate is 70% and the average cost is $0.10. After the modification, the success rate rises to 75%, but the average cost also rises to $0.30.

Looking only at success rate, it improved by five percentage points. Factoring in cost, the conclusion isn't so simple. If the task is high-value, the cost might be acceptable. If the task is lightweight, tripling the cost might directly offset the success rate improvement.

Traditional tests might all pass before and after the modification. They can show that the code still behaves as expected, but they can't answer whether the Agent has improved, let alone whether the improvement is worth it.

Agent changes often affect prompts, models, tool selection, and execution rules. Even if the code doesn't error, the system's behavior can still change significantly. Without a stable evaluation method, releases easily revert to manually testing a few cases and then deciding by feel.

Eval Brings the Entire Task into the Evaluation Scope

Eval, as discussed here, can be understood as a method involving a fixed set of tasks, scoring rules, and repeated runs. Its evaluation target extends from a piece of code to the entire task.

Taking a Coding Agent as an example, you can give it a project and a problem to fix, and let it complete the modification. During evaluation, you check whether the task was solved, the quality of the code, whether regressions occurred, whether tool usage was reasonable, how many tokens were consumed, and how long it took.

These dimensions don't necessarily combine into a single score. Putting success rate, cost, and time together already clarifies the trade-offs of a change.

Eval also doesn't need to prescribe a single path for the Agent. As long as the task is ultimately completed, necessary constraints are met, and no new problems are introduced, different search and modification orders can be accepted.

This suits Agents well. Paths vary, but results can still be evaluated. A single run has randomness; after repeatedly executing the same set of tasks, versions have comparable records.

Two Systems Each Guard Their Own Domain

Unit tests, integration tests, and end-to-end tests should still be retained. Function logic, interface behavior, permission checks, data reads/writes, and tool execution all need explicit tests.

Eval handles another part. It cares whether the Agent completed the work, whether the quality is stable, whether the cost is acceptable, and whether capability rose or fell after changes.

Replacing traditional tests with Eval would miss basic code errors. Doing only traditional tests would fail to see the Agent's capability changes.

A more complete release process adds an extra check. Code first passes traditional tests, then the Agent runs a fixed Eval Suite. Only after results meet preset thresholds does the version enter release.

Without Eval, Optimization Easily Becomes Guesswork

Without Eval, the development process is often familiar. Modify a prompt, try a few cases, see decent results, then ship.

Six months later, it's hard to say which version was best, which change brought improvement, or which change caused a regression on certain tasks. A few successful demos can't replace long-term records.

With a fixed task set, the process becomes much simpler. Every time you change the Agent, run the same batch of tasks, score with the same rules, and compare success rate, regressions, cost, and time. Whether a change has value at least has a reviewable basis.

This method also has its own requirements. If the task set is too small, scoring rules change frequently, or test tasks are too far from real usage, Eval can also give misleading results. It needs continuous maintenance, just like unit tests—you can't write it once and forget it.

When an Agent team releases a version, it needs to answer two things simultaneously: whether the code passes tests, and whether the system performs better on fixed tasks.

Code passing only shows it still runs. Whether to release depends on whether it can consistently get things done on the same set of real tasks.