Back to blog
• 5 min read • AI Engineering

The trap of a perfect eval score

A perfect score on an eval you wrote yourself proves less than it feels like it proves. There are two very different systems that both produce a 5/5: one where the agent is actually good at the task, and one where the eval never asked a question hard enough to expose where it isn’t. From the number alone, those two systems are indistinguishable. The number can’t tell you which one you’re looking at; only the design of the eval and the reasoning behind the score can.

That asymmetry is easy to lose sight of because a perfect score feels like a finish line. It’s actually closer to a prompt to go check your test data.

Treat the score as a summary, not the evidence

The concrete failure mode is trusting a bare number over the reasoning that produced it. A judge model that outputs score: 5 and nothing else is unfalsifiable: you can’t tell whether it checked anything or just defaulted to a high number for fluent-sounding text. A judge that’s required to explain itself is checkable in a way a bare number isn’t. Here’s a real example, the reasoning field from an LLM judge scoring a research agent’s report against its raw sources, pulled directly from an eval run:

The report accurately synthesizes the information from the provided sources. It correctly identifies that Erasure Coding in systems like AWS S3 typically relies on Reed-Solomon codes and Galois Field arithmetic, while also addressing the user’s specific request about the mathematical foundations and context of AWS S3.

That’s worth something a bare 5 isn’t: it names specific concepts (Reed-Solomon codes, Galois Field arithmetic) the report was checked against, which means a human reading it can independently confirm the judge actually engaged with the content rather than pattern-matching on tone. The structured output that produced it made this the design, not an accident:

class EvalScore(BaseModel):
    faithfulness: int = Field(
        description="Score 1-5: Does the report ONLY use facts present in the raw sources?"
    )
    relevance: int = Field(
        description="Score 1-5: Does the report actually answer the user's topic comprehensively?"
    )
    reasoning: str = Field(description="A brief explanation for the scores.")

If your eval harness only persists the numeric column and discards the reasoning, you’ve thrown away the part that lets you tell “genuinely good” apart from “unfalsifiable.”

Be explicit about what a small eval set can prove

Three test cases can tell you the pipeline runs end to end without crashing on ordinary inputs. Three test cases cannot tell you the system is robust, because “robust” is a claim about behavior across a distribution you haven’t sampled from, not about three points inside it. A concrete instance: a research agent’s eval harness runs exactly three fixed topics through the full pipeline, hardcoded, none of them adversarial, none chosen to try to make the system fail:

TEST_CASES = [
    "Explain the difference between LangGraph State and standard memory.",
    "What is the mathematical foundation of Erasure Coding in AWS S3?",
    "How does the PPO algorithm work in Reinforcement Learning from Human Feedback?",
]

All three came back 5/5 on faithfulness and relevance, with the critic inside the graph triggering zero revisions on any of them. Before that number can support a claim stronger than “the harness didn’t find a problem on three ordinary topics,” it needs company: a topic designed to be outside the system’s retrieval coverage, a topic that intentionally invites the model to state something it can’t source, and a repeat run of the same topic to check whether the judge’s own scoring is even stable. None of those were in the set. That’s not a flaw specific to this project, it’s the default state of most evals anyone writes for their own system: comfortable inputs, chosen because they’re representative of the happy path, not because they’re likely to break anything.

A zero on the “did anything need fixing” metric is a signal to check, not a win

The most counterintuitive part of reading eval output correctly is that a metric which should sometimes fire, and never does, is more suspicious than reassuring. In the research agent above, the critic loop exists specifically to catch and revise bad drafts, revision_count increments every time a draft is rejected, and it recorded zero across every single topic tested. There are two explanations for that, and they are not equally likely just because the number is the same:

  1. The drafts were good enough on the first pass that the critic genuinely had nothing to flag.
  2. The critic isn’t strict enough to catch what it should be catching, and a zero is what a critic that never rejects anything looks like too.

The independent judge’s reasoning, checking the same reports against the same sources through a completely separate code path, is what tips this toward explanation one rather than two, at least on the three topics actually tested. That’s a narrower claim than “the critic loop works,” and it’s worth stating the narrowing out loud: it’s a claim about three ordinary topics, one specific judge model, and zero adversarial pressure. “Zero revisions triggered, 5/5 across the board” is exactly the pattern that should make you go add a harder test case before it makes you write a changelog entry.

Back to top