nup_logo

Building AI Agents

Agent Evaluation:
From Metrics to Managed Quality


Alex Avdiushenko
November 3, 2026

Agenda

01 Why Agent Eval Is a Separate Discipline
02 Task Suite: The Most Valuable Artifact
03 Eval Infrastructure: Turning Checks into Engineering
04 Criteria & Metrics: Defining Success
05 Grading & LLM-as-Judge
06 Eval by Agent Type
07 Summary & Practice

What is Evaluation?

input task → system output → scoring logic → metric
  • Systematic, repeatable process — not a one-time manual check
  • Automated: runnable in CI, no real users required
  • Gives a number: progress is measurable, regressions are detectable
As systems grew more complex, eval had to evolve with them

How Evaluation Evolved

Generation What the system does What we evaluate Typical eval
Single-turn LLM 1 prompt → 1 answer Answer quality BLEU, ROUGE, exact match
Multi-turn LLM Dialog with memory, several turns Coherence, context, instruction following Dialog metrics, human eval
Agentic systems reasoning → action → observation loop, tools, env side-effects Full trajectory + outcome + environment state Graders: code + LLM-judge + human

What is an AI Agent?

Reasoning → Action → Observation → repeat
  • Calls tools: APIs, databases, search, code execution
  • Modifies environment state — actions have real-world consequences
  • Iterates until task is complete (or gives up)
Key insight: we must evaluate the whole chain, not just the final text

Why Evaluate Agents?

  • Reliability & Safety — catch bugs before production, not after
  • Understand Behavior — why did the agent make that decision?
  • Track Progress & Regressions — new model / prompt / tool → did it improve?
  • Iteration Speed — evals turn dev into a managed process; without them: guessing
  • Objective Communication — 'pass@1 dropped 8% on edge-cases' vs 'it seems worse'

Why Agent Eval is Hard

  • Compound errors — mistake at step 3/10 propagates, corrupts all subsequent steps
  • Multiple valid paths — different action sequences can reach the same correct outcome
  • Stochasticity — same prompt → different reasoning chains; classical assertEqual fails
  • Hidden failures — wrong tool, wrong param, but final text looks acceptable
  • Environment side-effects — agent says 'done', but did the DB actually change?

Agentic Eval — Full Chain, Not Just the Answer

Full Trace
What steps were taken · What actions executed · How decisions were made
Technical Effects
Which API called · What parameters passed · What external system returned
Logical Effects
Did the agent understand context · Respect constraints · Complete the task
Eval of an agent = test of the entire reasoning-to-action chain, not just the final text

Key Terms

Evaluation Suite Collection of tasks measuring specific agent abilities
Task / Test Case One test: user query + env state + success criteria
Trial One execution of a task. Multiple trials per task (stochasticity!)
Trajectory Full record: outputs, tool calls, reasoning, observations
Outcome Final environment state after trial — the real DB change, not the text
Grader Logic that evaluates one aspect of the agent's work
Eval Harness Infrastructure that runs evals: orchestrates tasks, captures results

The 4-Component Eval Pipeline

1
Task Suite
Representative set of tasks
→
2
Infrastructure
Runner, environment, logger
→
3
Criteria
Rules that define 'success'
→
4
Grading
Applying criteria to trajectories
Feedback loop: grading results feed back into all earlier stages
Weakness in any one component devalues all the others

Agenda

01 Why Agent Eval Is a Separate Discipline ✓
02 Task Suite: The Most Valuable Artifact
03 Eval Infrastructure: Turning Checks into Engineering
04 Criteria & Metrics: Defining Success
05 Grading & LLM-as-Judge
06 Eval by Agent Type
07 Summary & Practice

Building the Task Suite — Start Small

20–50 tasks from real problems is enough to start
  • v0 — 5–10 tasks from manual analysis of what the agent should do
  • v1 — run agent, find errors, add edge-cases and failure modes
  • v2 — add categories, balance, LLM-generated tasks for gaps
  • v3 — lock as regression suite (runs on every PR)
  • v∞ — add tasks from bug tracker and user feedback after launch
Key insight: A well-constructed benchmark dataset is the foundation — all other evaluation components (criteria, graders, monitoring) depend on it

Sources for Tasks

  • Manual analysis of user needs — always, from day one, what we want from agent
  • Production logs — best distribution representation (once deployed)
  • Bug tracker / support — capture real failure modes after launch
  • Public benchmarks — τ-bench, SWE-bench, ToolBench (for calibration vs SOTA)
  • LLM generation — features × scenarios × personas matrix, for scale
Rule: everything your team verifies manually → convert to a formal test

Scaling the Suite with LLM Generation

Generate tasks across 3 dimensions — then validate every generated task manually
Features
Agent functionalities
→ Search · Change · Cancel · Info
Example generation prompt:
Generate an airline user query with:
- Feature: change booking
- Scenario: incorrect data (booking not found)
- Persona: frustrated, uses short phrases
Format: query text only, 1–2 sentences.
Scenarios
Interaction situations
→ Edge-case · No results · Ambiguous · Error
⚠️ Always validate manually
  • Is the task unambiguous?
  • Is it actually solvable?
  • Does it make domain sense?
Personas
User types
→ Novice · Frustrated · Imprecise
Tag every task:
  • difficulty: easy / medium / hard
  • category: info / change / cancel
  • edge_case · needs_confirmation

Task = Query + Environment State

A task is NOT just a text prompt
  1. User query — what the user wants
  2. Initial environment state — DB, APIs, business policies in effect
  3. Expected goal state — how DB should look after correct execution
  • Agent says “Your flight is changed” — did the DB actually update?
  • Grade on state change, NOT on response text — core principle of serious agentic evals

Task Structure Example

Airline agent — change booking to cheaper flight
user_query: 'Reschedule ABC123 to tomorrow, only if cheaper' initial_environment: flights: SU2454_0220: { date: 2026-02-20, price: 25000 } SU2454_0221: { date: 2026-02-21, price: 24000 } bookings: ABC123: { flight_key: SU2454_0220, status: confirmed } policies: - 'Change is free if new flight is cheaper' - 'Require user confirmation before changing' expected_goal_state: bookings: ABC123: { flight_key: SU2454_0221, status: changed }
Grader compares final DB state to expected_goal_state
No text analysis needed — pure objective check
Policies are part of the environment, not just the prompt
Tools are declared — agent can only use listed tools

Properties of a Good Task Suite

  • Unambiguous — two experts must agree on pass/fail
  • Environment state — objective outcome check, not text analysis
  • Class balance — don't overfit on 'easy' categories
  • Reference solution — proves task is solvable, validates graders
  • Bidirectional — test when agent SHOULD and SHOULD NOT act
  • Reproducible — runs in CI; no dependency on live external APIs
  • Tagged — by difficulty, type, skill (enables regression analysis)

Agenda

01 Why Agent Eval Is a Separate Discipline ✓
02 Task Suite: The Most Valuable Artifact ✓
03 Eval Infrastructure: Turning Checks into Engineering
04 Criteria & Metrics: Defining Success
05 Grading & LLM-as-Judge
06 Eval by Agent Type
07 Summary & Practice

Principle Zero: Same Agent Code

The agent in eval MUST be the same agent as in production
  • Same code · Same prompts · Same tools — no 'simplified eval version'
  • Prompts and tool descriptions come from one source of truth
  • Only difference: test environment instead of production, test DB instead of real DB
  • If eval agent behaves differently from production — eval conclusions don't transfer

Infrastructure Components

Runner
Runs agent on full suite
Concurrency · Timeouts · Retry
Environment
Isolated env per trial
Clean DB state on every run
Logger / Tracer
Full trajectory captured
Reasoning · Tool calls · Observations
Iron User
LLM-based user simulator
Required for all multi-turn dialog tasks
Storage
All runs · Trajectories · Metrics
Searchable · Versioned
Comparator
Metric diff across versions
Regression detection & visualization

Environment: Real vs Mock

◀ more control · faster · cheaper more realistic · more honest ▶
Full Mock
in-memory DB fake APIs
When: Early dev · CI unit tests for tools
⚠️ Mock drift — diverges from real API over time
Hybrid
mock external APIs + real DB
When: Main eval mode best balance
⚠️ Keep test data fresh and realistic
Production API
all real calls
When: Final validation pre-release only
⚠️ Slow · expensive · external service dependencies

Iron User — Simulated User for Dialog Tasks

Without Iron User, dialog tasks cannot complete
  • Fixed goal — knows what it 'wants', stays in scenario
  • Naturalistic — short answers, doesn't reveal everything upfront
  • Does NOT hallucinate data not in the instruction

Iron User — Problems to Know

  • Non-deterministic — Iron User is also an LLM; its answers vary between runs
  • Can hallucinate — may invent data not in scenario despite instructions
  • Too cooperative — real users are vague, impatient; simulator inflates scores
  • Loop risk — agent error → simulator repeats → infinite loop
  • Fix: adaptive prompt, strict max_steps, correct API design (tools return enough info)
  • Iron User is necessary but not sufficient — periodic real-user validation remains important

Iron User — Simulation Strategies

τ-bench implements 4 strategies with increasing reliability — each builds on the previous
LLM
How:
LLM generates a response directly from the scenario instruction
Why:
Baseline variant. Fast. Prone to hallucinating data not in the instruction.
ReAct
How:
LLM first reasons (Thought:), then generates the response (Response:)
Why:
More consistent behavior. Reasoning step keeps simulator aligned with scenario.
Verify
How:
After generating, a 2nd LLM call checks quality. If bad → regenerate (up to 3 times)
Why:
Guards against simulator hallucinations. Quality gate before response is sent to agent.
Reflection
How:
On failed verification: LLM reflects on the error, then generates an improved response
Why:
Most reliable. Explicit self-correction loop. Used in highest-quality τ-bench evals.

Minimum Requirements for Your Own Harness

  • Runs on shared VM / CI — not on a dev laptop
  • Retry on API timeouts, graceful error handling
  • Auto-export results (traces + metrics) to storage after every run
  • Parallel task execution — serial execution of 100 tasks × 5 trials = hours
  • Version comparison — at minimum a metric diff in CSV
  • Isolation — clean environment state per trial (no leftover artifacts)

Eval Libraries & Platforms (2026)

Libraries
  • DeepEval — pytest-style, 40+ metrics, CI
  • RAGAS — reference-free RAG eval
  • Arize Phoenix
  • Braintrust AutoEvals — ready judge prompts
  • inspect-ai — UK AISI, 95+ benchmarks
Platforms
  • LangSmith — trajectory eval, framework-agnostic
  • W&B Weave — scorers, leaderboards, human review
  • OpenAI Platform — full harness, trace grading
  • Braintrust — harness + AutoEvals + dashboard
  • n8n Evaluations — no-code workflow builder

LangSmith — What It Looks Like

Experiment view
LangSmith Experiment View

LangSmith — Version Comparison

LangSmith Version Comparison

OpenAI Evals — What It Looks Like

OpenAI Evals Dashboard

Agenda

01 Why Agent Eval Is a Separate Discipline ✓
02 Task Suite: The Most Valuable Artifact ✓
03 Eval Infrastructure: Turning Checks into Engineering ✓
04 Criteria & Metrics: Defining Success
05 Grading & LLM-as-Judge
06 Eval by Agent Type
07 Summary & Practice

What Makes a Good Criterion?

Start with 2–3 criteria — teams that start with 10 end up measuring nothing
✓
Aligned
Corresponds to the agent's goal — not a proxy metric
✓
Operationalizable
Simple to verify — automatable, two experts must agree on pass/fail
✓
Human-readable
Colleagues understand what it means without explanation
✓
Discriminative
On failure: immediately clear what exactly to fix
✓
Stable
Independent of subjective interpretation — doesn't drift between runs

Two Groups of Criteria

Output-based (Formal)
  • Agent returned response in expected format
  • Correct tool called with correct parameters
  • Environment state changed as expected (booking in DB)
  • No runtime errors or exceptions
Implemented as: assert · unit tests · state checks
Trajectory-based (Behavioral)
  • Is the action chain logically consistent?
  • Was the task goal maintained throughout?
  • Were there unnecessary steps (inefficiency)?
  • Are decisions justified in the reasoning?
  • Did the agent lose context mid-trajectory?
Implemented as: LLM judge · count-based · reference compare
Both groups are needed — outcome-only grading misses process failures and policy violations

Universal Criterias

Start with 2–3 criteria — avoid metric drowning
Criterion Question to answer Type Example Grader
Usefulness Did the agent solve the user's task? Output-based State check · binary pass/fail
Correctness Errors, exceptions, invalid tool calls? Output-based Assert · static analysis
Groundedness All facts confirmed by tool data? Trajectory LLM judge · reference compare
Efficiency Minimum steps, no redundant tool calls? Trajectory Count-based · LLM judge
Stability Does success repeat on re-run? Statistical pass@k · pass^k
Safety No toxicity, data leaks, policy violations? Output / Traj Rule-check · classifier

Stability Metrics: pass@k and pass^k

pass@k — AT LEAST ONE of k trials succeeds
pass@k = 1 −
C(n − c, k) C(n, k)
↑ grows with k · pick best of k · code gen
pass^k — ALL k trials succeed
pass^k = p^k
p = per-trial success rate
↓ drops with k · customer-facing · production
pass@k vs pass^k chart

Agenda

01 Why Agent Eval Is a Separate Discipline ✓
02 Task Suite: The Most Valuable Artifact ✓
03 Eval Infrastructure: Turning Checks into Engineering ✓
04 Criteria & Metrics: Defining Success ✓
05 Grading & LLM-as-Judge
06 Eval by Agent Type
07 Summary & Practice

Grader Types — Pros & Cons

Type Methods ✅ Strengths ⚠️ Weaknesses
Code-based String/regex match · assert · DB state check · tool call check · transcript stats Fast · Cheap · Objective · Reproducible · CI-friendly Brittle to valid variations
Can't capture nuance
Human Expert review · inter-annotator agreement · A/B testing · spot-check Gold standard · Calibrates LLM judge · Catches subtle errors Slow · Expensive
Doesn't scale to CI
LLM-as-Judge Rubric scoring · pairwise comparison · reference-based · CoT judgment Flexible · Captures nuance · Scales to open-ended tasks Needs calibration (κ > 0.6)
Expensive · Non-deterministic
Rule: start with code-based, add LLM judge for what code can't check, calibrate with human labels

Measuring Agreement: Cohen's κappa

κ = (P_observed − P_expected) /
(1 − P_expected)
< 0.20
Poor
0.21–0.4
Weak
0.41–0.6
Moderate
0.61–0.8
Substantial
✅
0.81–1.0
Near-perfect
Corrects for chance agreement —
unlike raw % agreement
Target for LLM-as-Judge: κ > 0.6 (substantial agreement with human labels)
  • Without human labels → no ground truth to measure judge quality
  • Labeling forces you to externalize criteria (criteria drift is normal)
  • Expert critiques become the most valuable few-shot examples in the judge prompt
  • Periodically re-calibrate as the task suite grows

Combining Graders

One task — multiple graders — one score
  • Binary (AND) — all graders must pass — otherwise fail
  • Weighted — each grader gives a score, the result is a weighted sum
  • Hybrid — critical graders are must-pass, the rest are weighted (reasoning quality)

The Judge Building Cycle

Always start with human labels — no ground truth = can't validate
  • ① Manual labeling by experts: 30–100 trajectories, pass/fail + critique
  • ② Human labels → judge prompt: few-shot examples, criterias, scale definitions
  • ③ LLM judge runs on new tasks: automatic evaluation
  • ④ Compare with human labels: Cohen's κ + confusion matrix
  • κ < 0.6 → rework prompt → back to ③
  • κ ≥ 0.6 → judge ready for CI (target: substantial agreement)

Naive Judge and Its Problems

A good judge prompt needs 4 components
1
Task context
Goal · inputs · constraints
2
Agent data
Reasoning trace · tool calls · observations
3
Evaluation rubric
What to check · scale · concrete level definitions
4
Response format
JSON — so the output is parseable
Naive approach - 1 prompt, all criteria at once
Assess the agent's response quality: - usefulness (good / so-so / bad) - groundedness (good / so-so / bad) - efficiency (good / so-so / bad) Reply JSON: {"usefulness": "...", "groundedness": "...", "efficiency": "..."}
Problems this causes:
Leniency bias — LLM marks everything 'good', misses real failures
Halo effect — rating one criterion biases the others
Inconsistency — same trajectory twice → different scores

5 Improvement Techniques

  • (1) One criterion = one prompt — removes halo effect, allows independent tuning
  • (2) Chain-of-Thought: reason first, then score — forces justification
  • (3) Concrete rubric definitions with examples — biggest single improvement
  • (4) Escape hatch: allow 'Unknown / Insufficient data'
  • (5) Few shot examples (golden) — include tasks with known correct answers

Agenda

01 Why Agent Eval Is a Separate Discipline ✓
02 Task Suite: The Most Valuable Artifact ✓
03 Eval Infrastructure: Turning Checks into Engineering ✓
04 Criteria & Metrics: Defining Success ✓
05 Grading & LLM-as-Judge ✓
06 Eval by Agent Type
07 Summary & Practice

Conversational Agents

Agent type
Multi-turn dialog + tool calls + DB state management across several conversation turns
Grader
Combo grading:
State-based (DB diff) · LLM rubric judge (tone, policy compliance) · Transcript constraint (turn count)
Specifics
• Iron User required — without it multi-turn tasks cannot be completed
• Grade DB state, not text (expected_goal_state vs actual_state)
• Multi-dimensional: outcome + interaction quality + turn efficiency
Benchmark
τ-bench / τ2-bench (Sierra Research, 2024) — airline (50 tasks) + retail (115 tasks)
pass^k at k=5 drops to 15–20% for all SOTA — agentic tasks are far harder than single-turn evals
Leaders
Qwen3.5-397B 87.9% · Claude Opus 4.5 85.3% · GPT-5.2 84.8%
Human baseline: not published

Conversational Agents — τ2-bench Leaderboard

tau2-bench Leaderboard
pass^1 → pass^4 drops for ALL models. Even at pass^1 < 50% for the airline domain

Coding Agents

Agent type
Write, test, and debug code on real GitHub issues across large multi-file Python repositories
Grader
Code grader: fail-to-pass — tests that failed BEFORE the patch must pass AFTER
No LLM judge needed for core evaluation — test results are objective
Specifics
• Objective grading — tests pass or fail, minimal subjectivity
• A task may touch 1 to 15+ repository files
• Fail-to-pass + regression: previously passing tests must not break
Benchmark
SWE-bench Verified (bug fixing in 12 repos) · ProgramBench (rebuilding program from scratch)
Terminal-Bench — end-to-end tasks in Linux terminal (servers, scripts, networking)
Leaders
ProgramBench: Claude Opus 5 (xhigh) 4.5% · GPT-5.6 Sol 1.0% · SWE-bench SOTA ~79%
Human baseline: expert developers rebuild codebases reliably

Coding Agents — ProgramBench Leaderboard

ProgramBench Leaderboard
Rebuild entire codebases from binary + docs: frontier benchmark where SOTA resolves < 5% (Claude Opus 5: 4.5%).

Practice

Pick the champion by ROI — cost-efficiency on leaderboard (15–20 min)

  • Compute token cost in dollars from input_tokens, output_tokens and price per 1M tokens
  • Score / cost ratio — measure quality delivered per dollar spent on the benchmark
  • Filter candidates by min_score and positive cost; break ties stably

SOTA isn't just accuracy — in production, 10× cheaper at 95% quality often wins.

Research Agents

Agent type
Web search, information synthesis, report generation and multi-step factual Q&A
Grader
Exact match for factual tasks (BrowseComp) · LLM judge: groundedness + coverage + source quality
Human calibration required — subjectivity is higher than any other agent type
Specifics
• 'comprehensive', 'well-sourced' depend on context — most subjective agent type
• BrowseComp tasks can't be solved in one query — chained searches + cross-referencing required
• Human baseline is low — even experts with internet access fail many tasks
Benchmark
BrowseComp (OpenAI, 2025) — hard factual questions, exact match, deliberately non-templated
GAIA (Meta, 2023) — ~450 tasks, 3 difficulty levels, multi-step reasoning + tool use
Leaders
GAIA: multi-model agentic search 93.4% (surpassed human baseline ~92%)
BrowseComp: deep research models ~51% · Human baseline: only ~5–10%

Research Agents — GAIA Leaderboard

GAIA Leaderboard
Top research agents reach 93.4% on GAIA, officially surpassing human baseline (~92%). BrowseComp remains hard (~51%).

Computer Use Agents

Agent type
Computer Use agents — control browser and OS via screenshots, mouse clicks, and keyboard input
Grader
Execution-based: programmatic validators check URL, page state, or VM file system
No LLM judge needed — final system state is objective
Specifics
• Requires real or sandboxed GUI environment (VM with snapshot/restore before each run)
• Strong tool-calling agent ≠ strong GUI agent: different interfaces and different skills
• Agents take 1.4–2.7× more steps than humans and ~10× more time
Benchmark
WebArena (CMU, 2023) — 812 tasks on self-hosted websites, cross-site reasoning (5–15 steps)
OSWorld (HKU, 2024) — 369 tasks in real OSes (Ubuntu, Windows, macOS), 134 eval functions
Leaders
WebArena: DeepSeek v3.2 74.3% · OpAgent 71.6% · Human ~78%
OSWorld: Kimi K2.5 63.3% · Claude Sonnet 4.5 62.9% · Human 72.4%

Computer Use Agents — Leaderboards

Computer Use Agents Leaderboards (WebArena & OSWorld)
Specialized browser agents outperform general models on WebArena. OSWorld gap to human is closing.

Operational Metrics — Not Just Quality

Production readiness requires efficiency tracking alongside pass rates
  • Latency — task execution time p50/p95: compare across agent versions
  • Cost — API calls + compute: eval can become more expensive than production
  • Error rate — timeouts, exceptions, tool failures: must be below threshold

Agenda

01 Why Agent Eval Is a Separate Discipline ✓
02 Task Suite: The Most Valuable Artifact ✓
03 Eval Infrastructure: Turning Checks into Engineering ✓
04 Criteria & Metrics: Defining Success ✓
05 Grading & LLM-as-Judge ✓
06 Eval by Agent Type ✓
07 Summary & Practice

Moving to Practice

Now let's see how it all works in the notebook
1
Airline agent + task suite
Agent structure · 12-task suite · balance visualization
2
Code-based graders
state_grader · policy_grader · run on 3 tasks
3
Iron User v1 → v2
Infinite loop (77 steps) → stable run (8 steps)
4
LLM-as-Judge iterations
v1 (κ 0.25) → v4 (κ 0.72) — live calibration

Key Takeaways

1
Eval = 4 components
Task Suite · Infrastructure · Criteria · Grading
Weakness in any one devalues all others
5
LLM-as-Judge requires discipline
Per-criterion prompts + CoT + calibration
on human labels — the required minimum
2
Start early, not perfectly
20 tasks from real problems
> 0 tasks 'eventually'
6
Stochasticity is a property, not a bug
pass@k / pass^k and multiple trials
let you account for it
3
Eval is a loop, not a checkpoint
Results → new tasks,
improved criteria, better graders
7
Capability vs regression evals
Former drives progress,
latter prevents degradation
4
Grade the outcome, not the path
Agents find unexpected valid solutions.
Checking step sequences is brittle

Conclusion: From Eval to a Quality Culture

Flexibility by design
Put params (temperature, prompts, models) in configs from the start — experiments become routine
Quality goals before optimization
'90% accuracy for category X at latency <5s' protects against endless iteration
2–5 metrics, not twenty
80/20 rule: fewer but reliable. Add new ones when existing ones stop giving signal
Save every run
Agent version, params, date, per-task metrics — into a DB or file. Any run retrievable a month later
Tails matter more than averages
Analyze worst and best cases after each run. Every failure mode is a potential new eval task
Stability over average
Standard deviation is as important a signal as the mean. High variance = warning sign
Thresholds protect
'accuracy ≥ 85%', 'latency ≤ 3s' — no-go, even if the aggregate score formally improved
Quality first, then optimize
Start with the strongest model and full context. Lock in the quality ceiling — then optimize cost
The loop closes through production
User logs and feedback → new eval tasks. Online and offline metrics must correlate