| Generation | What the system does | What we evaluate | Typical eval |
|---|---|---|---|
| Single-turn LLM | 1 prompt → 1 answer | Answer quality | BLEU, ROUGE, exact match |
| Multi-turn LLM | Dialog with memory, several turns | Coherence, context, instruction following | Dialog metrics, human eval |
| Agentic systems | reasoning → action → observation loop, tools, env side-effects | Full trajectory + outcome + environment state | Graders: code + LLM-judge + human |
| Criterion | Question to answer | Type | Example Grader |
|---|---|---|---|
| Usefulness | Did the agent solve the user's task? | Output-based | State check · binary pass/fail |
| Correctness | Errors, exceptions, invalid tool calls? | Output-based | Assert · static analysis |
| Groundedness | All facts confirmed by tool data? | Trajectory | LLM judge · reference compare |
| Efficiency | Minimum steps, no redundant tool calls? | Trajectory | Count-based · LLM judge |
| Stability | Does success repeat on re-run? | Statistical | pass@k · pass^k |
| Safety | No toxicity, data leaks, policy violations? | Output / Traj | Rule-check · classifier |
| Type | Methods | ✅ Strengths | ⚠️ Weaknesses |
|---|---|---|---|
| Code-based | String/regex match · assert · DB state check · tool call check · transcript stats | Fast · Cheap · Objective · Reproducible · CI-friendly | Brittle to valid variations Can't capture nuance |
| Human | Expert review · inter-annotator agreement · A/B testing · spot-check | Gold standard · Calibrates LLM judge · Catches subtle errors | Slow · Expensive Doesn't scale to CI |
| LLM-as-Judge | Rubric scoring · pairwise comparison · reference-based · CoT judgment | Flexible · Captures nuance · Scales to open-ended tasks | Needs calibration (κ > 0.6) Expensive · Non-deterministic |
Pick the champion by ROI — cost-efficiency on leaderboard (15–20 min)
SOTA isn't just accuracy — in production, 10× cheaper at 95% quality often wins.