Twitter/X

Ankur Goyal argues that evals (evaluation suites) are not the same as tests and…

2026-08-04 · 15:49 UTC ·@ankrgyl ·0 min read

Brief

Ankur Goyal emphasizes that evals and tests serve distinct purposes: evals are crucial for measuring and iterating agent behavior, whereas traditional tests verify correctness in the software lifecycle. He argues teams need both, and that maturing eval practices for agents usually requires adding targeted tests that exercise agent-specific behaviors.

Why it matters

Ankur Goyal argues that evals (evaluation suites) are not the same as tests and serve different roles in the development lifecycle.

Key details

  • Evals are especially critical for agents, but Goyal insists you still need both evals and traditional tests.
  • A mature eval practice for agent systems typically implies you must also build tests specifically targeting agent behavior (he references this while building Braintrust).
Cleaned source text

Wrote up some thoughts on the differences between evals and tests.

They're not the same, and evals are critical for agents in particular, but you still need both. A mature eval practice built for agents often means testing the agents too.

Ankur Goyal (@ankrgyl)

Article

Evals ≠ tests

While building Braintrust I’ve been thinking about what evals are good at, what they should be used for, and how they differ from other parts of the software development lifecycle, like tests.

Though

— https://nitter.net/ankrgyl/status/2084667136618762550#m