Erik Caldwell
Connect on LinkedIn
Menu

The Topic Catalog · 12

AI Evaluation & Quality Engineering

Last updated · 10 min read

12.1 AI Evals & Golden Datasets

Priority: Must Understand

Executive Definition: An "eval" is a repeatable test that scores model or agent output against a defined standard (accuracy, relevance, tone, safety) usually run automatically against a curated "golden dataset" of representative inputs with known-good (or known-acceptable) outputs. Evals are the AI-era equivalent of a test suite, but because LLM outputs are non-deterministic and rarely have one single correct answer, evals typically score against a rubric or a reference answer within a tolerance, rather than an exact match.

Why It Matters: Conventional software testing assumes a deterministic function: same input, same output, pass/fail. LLM outputs vary between runs even at identical settings, and "correct" is frequently a matter of degree (a legally accurate but poorly worded answer vs. a fluent but subtly wrong one). Without golden datasets and evals, model or prompt changes ship on faith: the only signal that something regressed is a user complaint, by which point it is already in production. This is the foundational competency underneath every other evaluation topic below.

What I Need to Understand:

  • A golden dataset is a curated, versioned set of representative inputs (ideally including edge cases and known failure modes) with reference outputs or acceptance criteria: it needs the same change control and ownership as a codebase, not a one-time spreadsheet.
  • Evals fall into three broad families: exact/programmatic checks (does the output contain a required field, pass a regex, parse as valid JSON), similarity-based checks (embedding or n-gram overlap against a reference answer), and LLM-as-judge checks (see below): each has different cost, reliability, and failure characteristics.
  • Because there is often no single correct answer, most production evals score against a rubric with partial credit rather than binary pass/fail, and rubric design is itself a skill that requires domain expertise, not just engineering effort.
  • Golden datasets go stale as the product and real user traffic evolve: production evaluation (see continuous evaluation topic below) exists precisely because a static golden dataset cannot cover what real users will actually do.
  • Eval quality is only as good as the dataset's coverage of realistic failure modes; a dataset built only from happy-path examples will pass models that fail badly on messy real input.

Questions I Should Be Able to Ask My Team:

  1. Who owns and updates our golden datasets, and how often are they refreshed against real production traffic and known failure cases?
  2. What percentage of our golden dataset examples were sourced from actual production incidents versus hypothetical test cases?
  3. When we change a prompt, model, or RAG pipeline, is there a required eval run and pass threshold before it ships, or is this discretionary?

Technologies / Standards / Companies to Know: Retrieval Augmented Generation Assessment (RAGAS), DeepEval, Langfuse, Databricks MLflow evaluation, promptfoo.

Recommended Learning:

Time Investment: Half day

12.2 Trajectory Evaluation for Agents & Tool-Call Correctness

Priority: Should Understand

Executive Definition: Trajectory evaluation scores not just an agent's final answer but the full sequence of decisions it took to get there: which tools it called, in what order, with what arguments, and whether each step was necessary and correct. A single-turn chat eval only has to check the final output; an agent eval has to check the path, because two agents can reach the same correct final answer while one took an efficient, safe route and the other took an expensive, unauthorized, or unsafe one.

Why It Matters: An agent that books the wrong flight but then "corrects" it might show a correct final trajectory-blind eval score while having generated duplicate charges, hit an external API twice, or briefly held incorrect state that a downstream system already acted on. Final-answer-only evaluation is blind to exactly the failure modes (wasted tool calls, wrong tool selection, unauthorized actions taken and then walked back) that drive cost overruns and security incidents in agentic systems.

What I Need to Understand:

  • Trajectory correctness typically evaluates three separate things: tool selection (did it pick the right tool), argument correctness (did it call the tool with valid, correct parameters), and step efficiency/necessity (did it take an unnecessarily long or costly path, including retries and backtracking).
  • Academic benchmarks such as TRAJECT-Bench (arXiv:2510.04550) formalize trajectory-aware scoring specifically because prior agent benchmarks judged only the end state and missed these path-level failures.
  • Multi-step agent trajectories compound error: a small tool-argument mistake early in a chain can cascade into a completely wrong final answer that nonetheless "looks" plausible, which is why trajectory-level evals catch failures that final-answer evals miss.
  • Trajectory evaluation depends on having full step-by-step tracing (see AgentOps topic below): you cannot score a path you did not capture.
  • This is a newer and less standardized discipline than single-turn LLM evaluation; expect your team's tooling and rubrics here to be more custom-built than off-the-shelf.

Questions I Should Be Able to Ask My Team:

  1. Do our agent evals score the full tool-call trajectory, or only whether the final answer was correct?
  2. What is our defined "acceptable" number of tool calls or retries per task, and do we flag trajectories that exceed it even when the final answer is right?
  3. Can we replay a failed agent trajectory end-to-end from stored traces, and how far back does that trace history go?

Technologies / Standards / Companies to Know: TRAJECT-Bench, AgentOps, LangSmith, Arize.

Recommended Learning:

Time Investment: 2-3 hours

12.3 Hallucination, Groundedness & LLM-as-Judge

Priority: Must Understand

Executive Definition: Hallucination is a model generating confident, fluent output that is factually wrong or unsupported by its source material. Groundedness is the inverse property being measured for: whether an output can be traced back to and is fully supported by the retrieved context or source documents it was given (most relevant in retrieval-augmented generation). LLM-as-judge is the now-standard practice of using a separate (often stronger) model to score another model's output against a rubric, because human review does not scale to production volumes and exact-match testing does not work on open-ended text.

Why It Matters: Hallucination is not a bug that gets patched away; it is a structural property of how these models generate text, and it varies significantly by task. Public leaderboards such as Vectara's hallucination leaderboard track measured hallucination rates across models specifically because this remains an active, unsolved, model-dependent risk, not a solved problem you can assume away. LLM-as-judge is powerful but introduces its own failure mode: judge models have documented biases (verbosity bias, position bias, self-preference) that were characterized in the original MT-Bench paper (arXiv:2306.05685) and must be accounted for, not ignored.

What I Need to Understand:

  • Groundedness/faithfulness scoring (as formalized in frameworks like RAGAS) checks whether every claim in an output is supported by the retrieved context: this is the primary defense against hallucination in RAG systems specifically, and does not apply to open-domain generation without retrieval.
  • LLM-as-judge is a measurement tool, not ground truth: judge models need their own validation against human-labeled samples periodically, or scoring drift goes undetected.
  • Known LLM-as-judge biases include favoring longer answers, favoring the first answer shown in pairwise comparison, and favoring outputs stylistically similar to the judge's own outputs: mitigations include randomizing order, using multiple judges, and reference-based (not just pairwise) scoring.
  • Hallucination rate is not a single number for a model; it varies by task type and is actively tracked and updated.
  • Groundedness/hallucination checks should run both in pre-production evals and continuously in production (see continuous evaluation topic below), since retrieval quality and real user queries shift over time in ways a static eval set won't catch.

Questions I Should Be Able to Ask My Team:

  1. What groundedness or faithfulness metric are we using in production, and what threshold triggers a flagged response?
  2. If we use an LLM judge, how was it validated against human judgment, and how often is that validation refreshed?
  3. Do we distinguish between hallucination in retrieval-grounded tasks (unsupported claims) versus open-domain tasks (factually wrong claims), since the mitigation for each is different?

Technologies / Standards / Companies to Know: RAGAS, Vectara's Hughes Hallucination Evaluation Model (HHEM), TruthfulQA, DeepEval, G-Eval-style judge prompting.

Recommended Learning:

Time Investment: Half day

12.4 Red Teaming, Adversarial & Continuous Production Evaluation

Priority: Must Understand

Executive Definition: Red teaming is deliberately attacking your own AI system (prompt injection, jailbreaks, data exfiltration attempts) before an adversary does. Continuous production evaluation is the parallel discipline of testing changes safely once the system is live, using techniques like shadow testing (running a new model/prompt on real traffic without serving its output to users), canary deployments (serving the new version to a small percentage of real users first), and A/B testing (serving two versions to different user segments and comparing outcomes). Together these replace the false assumption that passing a pre-launch eval means a system is safe indefinitely.

Why It Matters: LLM systems fail in ways static testing does not anticipate: adversarial prompt injection tops the OWASP LLM Top 10, and model behavior drifts as providers update models behind the scenes, as retrieval corpora change, and as real user query patterns diverge from the golden dataset. Shadow and canary deployment patterns exist because rolling out a new model or prompt version directly to 100% of production traffic, with no fallback, is the same category of risk that canary deployment already solved for traditional software: except the failure modes here (a subtly worse groundedness rate, a new jailbreak surface) are much harder to catch with traditional health checks alone.

What I Need to Understand:

  • Red teaming for LLM/agent systems should be continuous, not a pre-launch audit: OWASP's GenAI Red Teaming initiative and NIST's AI RMF both frame adversarial testing as an ongoing control, not a gate you pass once.
  • Shadow testing captures real production output for comparison without any user-facing risk: it is the lowest-risk way to validate a new model/prompt version against real traffic patterns your golden dataset does not cover.
  • Canary deployment (serving to a small real-user percentage) tests actual user impact but carries real risk to that percentage of users, so it needs an automatic rollback trigger tied to the same eval metrics as pre-production testing.
  • A/B testing for LLM changes needs outcome metrics beyond user satisfaction: cost per outcome, hallucination/groundedness rate, and trajectory correctness should all be tracked per arm, not just top-line engagement.
  • Adversarial testing scope should explicitly include agentic risks (an agent being manipulated into taking unauthorized actions via injected instructions in retrieved content or tool outputs), which is a distinct and newer attack surface from classic prompt jailbreaks.

Questions I Should Be Able to Ask My Team:

  1. Do we red-team continuously in production, or only once before initial launch?
  2. What is our rollback trigger and threshold for a canary deployment that starts showing degraded groundedness or elevated hallucination scores?
  3. For agentic systems specifically, have we tested prompt injection delivered through retrieved documents or tool outputs, not just through direct user input?

Technologies / Standards / Companies to Know: OWASP GenAI Red Teaming Initiative, NIST AI RMF (AI 600-1), promptfoo, Microsoft PyRIT.

Recommended Learning:

Time Investment: Half day