nup_logo

Building AI Agents

Production Engineering
for LLM Agents


Alex Avdiushenko
November 10, 2026

FlightAgent: Starting Point

FlightAgent
LLM
(brain)
GPT / Claude
Generates responses
and decisions
Tools
(hands)
search_flights()
book_ticket()
get_weather()
...
Loop
(spine)
Observe → Think →
Act → Observe
until task done
Works beautifully in Jupyter. Demo went great, everyone clapped
Ready for production? Not yet

Problem: Slow

5–10s
per request
(3 LLM calls × 1s + tools)
Google 200ms
Agent 5–10s

Problem: Unpredictably Expensive

Simple query: 3 steps  $0.01
Complex query: 15 steps  $0.30

Problem: Fails Silently

Tool Response
search_flights → TIMEOUT ❌
Status: 500 Internal Error
Data: none
Agent Response
“Found 3 flights to Istanbul”
Status: 200 OK ✅
Data: FABRICATED 🤔

Problem: Non-reproducible

Same Input:
"cheap flights
to Istanbul"
Run 1: "SU-123, ₽15K"
Run 2: "TK-456, ₽18K"
Run 3: "No flights found"

Problem: Black Box

Colleague changed prompt →
behavior changed. What exactly?
Don't know
Find out from support chat when
users complain
? ? ?
Steps? Cost? Where it broke?

Problem: Insecure

User
(untrusted)
FlightAgent
(no limits)
search_flights
get_bookings
admin_panel
DB access

Problem: Doesn't Scale

Each agent needs all of these:
Gateway
Auth
Observability
Eval
Queue
Deploy
FlightAgent
Gateway
Auth
Observability
Eval
Queue
Deploy
HotelAgent
Gateway
Auth
Observability
Eval
Queue
Deploy
InsuranceAgent
Gateway
Auth
Observability
Eval
Queue
Deploy
VisaAgent
Gateway
Auth
Observability
Eval
Queue
Deploy

7 Problems — All Solvable

1. Slow
2. Expensive
3. Fails
Silently
4. Non-
Reproducible
5. Black Box
6. Insecure
7. Doesn't
Scale

Agenda

01 Seven Problems ✓
02 Scaling the Prototype
03 Latency, Streaming & Cost Engineering
04 Resilience: Rate Limits, Fallbacks & Self-Healing
05 Tool Sandboxing & Execution Security
06 AgentOps: Tracing, Telemetry & OpenInference
07 Reference Architecture & Key Takeaways

Observability

Where to Start (Order Matters)

Classical service
Request → Response
1 step, deterministic
Fixed time & cost
Logs: 1 line per request
LLM Agent
Request → ??? → Response
3-15 steps, non-deterministic
3-15s, $0.01-$0.30
Logs: need per-step traces

What an Agent Actually Does

01
FlightAgent Request
user: 'cheap flights to
Istanbul'
3.1s total  |  $0.012
02
LLM Call #1
Parse intent
0.4s  |  320 tok
03
Tool: search_flights
SVO→IST, May 1-7
1.2s  |  ✅OK
04
LLM Call #2
Rank &
recommend
0.8s  |  1400 tok
05
Response
'Found 3 flights...'
0.2s

Structured Logging for Agents

print() / unstructured Structured logging
"Processing request..." {"event": "request_start",
"Called search_flights" "user_id": "u123",
"Got 5 results" "tool": "search_flights",
No context, no search "latency_ms": 1200}
Every LLM Call
Model, tokens,
latency, prompt
hash
Every Tool Call
Tool name, args,
result status,
latency
Every Decision
Why agent chose
this tool / this
response
Errors
Full context:
what was tried,
what failed

Trace Hierarchy — What to Look At

Session
User conversation
Multiple requests
Trace: Request #2
Step 4
Clarify dates
Step 5
Re-search
Trace: Request #1
Step 1
Parse Intent
Step 2
Search flights
Step 3
Rank results
Step 2 — Drill Down
LLM Call
0.4s | 320 tok
Tool Call
search_flights
1.2s | ✅

Key Metrics for LLM Agents

METRIC WHY IT MATTERS
Latency (p50, p95) User waits this long. p95 > 10s → users leave
Steps per request More steps = more $, more latency, more loops
Tokens per request Directly drives cost. Grows as context accumulates
Tool success rate Drop → external API broke or bad params
Task success rate The only metric users feel. Everything else is proxy
Error rate Spike → something broke. Silent errors are worse

Why Alerts — Not Dashboards

Without Alerts With Alerts
User complains in support chat Slack ping in 5 minutes
You check dashboard... manually Open trace → find root cause
3 hours later: 'LLM is slow' Fix before users notice
End-of-month: surprise $5K bill Budget alert → switch model
LATENCY
p95 > 10s for 5 min
→ Slack
Tool: LangFuse
alert rules
COST
daily > budget × 1.5
→ review traces
Tool: LangFuse
cost dashboard
ERROR RATE
> 5% for 10 min
→ PagerDuty
Tool: Grafana
or LangFuse

Observability Tools — How to Choose

LangFuse
Open-source, self-hosted
Python SDK, Docker
Traces, evals, cost
all in one dashboard
pip install langfuse
+ 3 lines of code
Grafana + Prometheus
Open-source, standard
Dashboards + alert rules
Thresholds, anomalies
Slack / Telegram notify
p95 latency, cost,
error rate alerts

Observability — Summary

Trace
Spans
Metrics
Alerts
Agent ≠ Microservice
Per-step observability
not per-request
Start with 2 metrics
Latency + Cost
on Day 1
Without Obs =
Blind
Can't optimize
what you can't see

Evaluation

Where to Dig?

62%
8 / 20 failing
Wrong tool
1 case
Called book
instead of search
Hallucination
2 cases
Price not
from API
Didn't clarify
3 cases
No date →
searched
randomly
Safety
2 cases
Booked without
confirmation
Main bucket: 'Didn't clarify' — 3/8 errors (38%). NOT a prompt problem → need few-shot
example.
Step 1: Error Analysis — categorize failing cases
Step 2: Open trace → find WHERE agent went wrong
Step 3: Category → approach (not all errors are prompt problems)

Quality Levers: Quick Wins

1. Tool Descriptions
+ Structured Output
WHY: LLM picks tool by its description. Vague desc → wrong tool called.
HOW: JSON schema forces valid output structure (no free text, no hallucination in field names)
2. Few-Shot
Examples
WHY: LLM learns from examples, not rules. 'Clarify dates' in prompt → ignored. Same as example → followed.
HOW: 3-5 input/output pairs in system prompt. +10-15% quality
3. Chain-of-Thought
WHY: complex queries need step-by-step. 'Find cheapest connecting flight' needs reasoning.
HOW: 'Think step by step' or structured reasoning. +10-20% quality

Quality Levers: Heavy Artillery

4. DSPy (sweet spot)
Algorithm finds best prompt
Auto-generates few-shot
Rewrites instructions
Needs: 20 test cases + metric
Result: +20-30% quality
5. Fine-tuning (last resort)
Train the model itself
Maximum quality + speed
Needs: 1000+ labeled examples
Cost: GPU time + maintenance
Only if 1-4 aren't enough

DSPy & CI

DSPy: Signature + Metric + Optimizer

Manual DSPy
Write prompt Define task + metric
Run eval Algorithm finds prompt
Edit prompt Automatic optimization
Repeat...
Signature
WHAT to do
(input → output)
Metric
HOW to measure
(your eval function)
Optimizer
HOW to find
(Bootstrap / MIPRO)

DSPy: FlightAgent — Real Code

Signature
(WHAT)
class FlightSearch(
    dspy.Signature
):
    """Find best flight"""
    query = dspy.InputField()
    flights = dspy.InputField()
    answer = dspy.OutputField()
Module
(HOW to call LLM)
agent = dspy.ChainOfThought(
    FlightSearch
)

# Adds "Reasoning: ..."
# before answer
# automatically
Metric
(HOW to score)
def metric(example, pred):
    return (
        pred.answer != "" and
        "cheapest" in pred.answer
    )
Optimizer
(find best prompt)
opt = BootstrapFewShot(
    metric=metric,
    max_demos=3
)
compiled = opt.compile(
    agent, trainset=data
)

DSPy: 62% → 91% in 2 Minutes

BEFORE: baseline prompt 62%
BootstrapFewShot: +3 examples 84% (+22pp)
MIPRO: rewrite instruction 91% (+29pp)

Multi-Agent Eval: Three Levels

Level 1: E2E Basket
20 user queries + expected output
Enough in 80% of cases
Level 2: Trace Debug
E2E fails → trace → find
sub-agent
1 failing e2e = 1 free
sub-test
Level 3: Assertions
No golden set needed
Valid JSON, prices > 0, no booking w/o confirm

Prompts as Code: CI Pipeline

Prompts NOT in git Prompts IN git
In Notion: who changed? Every change = PR + review
In UI: no history, no diff Diff: see what changed
In your head: bus factor 1 Rollback: 1 command
Regression: unnoticed CI: eval runs on every PR
Git Repo
prompts/
flight_agent_yaml
Pull Request
Changed prompt
Eval Suite
20 test cases
LLM-as-Judge
Score ≥
baseline?
✔Merge
Deploy
✖Block
Fix & retry

Quality ✔ — Now: Cost & Speed

✔ Observability
We can SEE
inside the agent
✔ Evaluation
We can MEASURE
and IMPROVE quality
Slow
Sequential
LLM calls
Expensive
Unpredictable
cost
Fails Silently
Fabricates
data

Agenda

01 Seven Problems ✓
02 Scaling the Prototype ✓
03 Latency, Streaming & Cost Engineering
04 Resilience: Rate Limits, Fallbacks & Self-Healing
05 Tool Sandboxing & Execution Security
06 AgentOps: Tracing, Telemetry & OpenInference
07 Reference Architecture & Key Takeaways

Cost Optimization

When You Need an Agent — and When You Don't

Your Agent
Can't solve with rules? Need reasoning? Need tools + loop?
Code
Sort by price
Validate dates
Format output
Classical ML
Intent classifier
Language detection
LLM (single call)
Parse user query
Generate response
LLM + loop
Multi-step
reasoning
Only when needed
Cost per 1K requests:
Sort by price: code
0 ms | $0
Detect language: fasttext
5 ms | $0
Parse intent: 1 LLM call
0.4s | $0.60
Full reasoning: LLM + tools
4s | $3.00

What the Company Pays For

LLM Inference
Every API call = money
70-90% of budget
Small vs Large = 60×
Server Hardware
Agent blocks while
waiting for LLM
Longer = more servers
1. Router
Fewer expensive
LLM calls
2. Speed
Parallel, streaming
fewer steps
3. Limits
Max steps
timeout, budget
4. Scale
Queue, cache
more requests

Calculate Before You Deploy

1K users × 3 requests × 4.2 LLM calls × 2,400 tokens = 30.24M tokens/day
Per call: 2,000 input + 400 output tokens. OpenAI Standard short-context rates / 1M tokens (input / output): GPT-6.1 Sol \$2 / \$10 · GPT-6.1 Terra \$1 / \$5 · GPT-6 Luna \$0.10 / \$0.50.
Configuration Price / 1M (input / output) Per day Per 30 days
100% Large — GPT-6.1 Sol \$2 / \$10 \$100.80 \$3,024
100% Medium — GPT-6.1 Terra \$1 / \$5 \$50.40 \$1,512
100% Small — GPT-6 Luna \$0.10 / \$0.50 \$5.04 \$151.20
Router 70/20/10 (S/M/L) \$0.47 / \$2.35 \$23.69 \$710.64
Router 80/15/5 (S/M/L) \$0.33 / \$1.65 \$16.63 \$498.96

Context Growth: Hidden Budget Eater

80% of tokens = resending previous
steps
Solutions: summarize, sliding window,
lean system prompt
Context Growth per Step (input tokens) Tokens 8000 7000 6000 5000 4000 3000 2000 1000 0 Step 1 Step 2 Step 3 Step 4 Step 5

Optimization: Four Levers

Start today: cost calc → max steps → defensive prompt → router
1.
Router — ~76-84% lower LLM cost on this example.
Small, 20% standard → Medium, 10% complex → Large
2.
Speed — -30-40% latency, -30% servers. Parallel tools,
streaming, fewer steps via DSPy
3.
Limits — predictable budget. Max steps (5-10), timeout
(30-60s), loop detection
4.
Scale — 100K+ users, -60% cost. Queue + worker pool, three
levels of caching

Not Every Request Needs a Large Model

Router
Classify complexity
Small
Simple, queries,
FAQ
Medium
Search, compare
Large
Multi-step, edge
cases
10% Complex ~ $0.034 / request 20% Standard ~ $0.017 / request 70% FAQ ~ $0.0017 / request failed? failed?

Model Router — How to Implement

Rules
if/else on query
length + keywords
Accuracy: ~70%
Classifier
Logreg on 200-300
labeled queries
Accuracy: ~85%
LLM-as-Router
Small model classifies
+1 LLM call
Accuracy: ~95%
CRITICAL: fallback if cheap model fails → retry at higher level, or bad answers on 15%

Speed & Resilience

Speed = Hardware Costs

Before: 4.2s/req After: 2.8s/req
1 worker ≈ 14 req/min 1 worker ≈ 21 req/min
100 req/min → 7 workers 100 req/min → 5 workers
Savings: 2 fewer servers × $300/mo = $600/mo
Parallel Tools
Run independent
tools at once
Streaming
0.3s to first word
vs 4s full wait
Fewer Steps
Better prompt
or DSPy

Without Limits, the Tail Eats Budget

Why agents loop Cost distribution
Tool returns empty → retry 95% normal: $0.003–$0.015
LLM misunderstands → repeat 4% expensive: $0.05–$0.15
Ambiguous query → search 1% catastrophic: $0.15–$0.50
→ refine → search → refine...
No stop condition in prompt 1% costs = 20% of normals
Max Steps
Hard stop: 5-10
Agent cannot exceed
regardless of prompt
Timeout
30-60s wall clock
Kills stuck requests
before cost explodes
Loop Detection
Same tool + same args
= loop → stop early
Alert to on-call

Queue + Worker Pool

Without queue: 100 users → 3 workers → 97 wait 30s+ → timeouts, retries, cascading failure
Accept instantly → return job_id → process async
Auto-scale: queue < 10 → 5 workers | > 50 → 20 | > 200 → backpressure
Users
100 req/min
API Gateway
Accept
instantly
Return job_id
Queue
Priority
Retry
Backpressure
Worker Pool (auto-scale)
Worker 1
Worker 2
Worker 3
Worker
N
LLM API
Response
via webhook
or polling

Three Levels of Caching

Three complementary ways to avoid repeated work
Level 01
Semantic Cache
Embedding similarity recognizes different wording with the same intent.
EXAMPLE 'cheap flights' ≈ 'affordable flights'
~30% fewer LLM calls
Level 02
Tool Result Cache
Flight prices rarely change second to second, so reuse recent results.
EXAMPLE search_flights · TTL 5 min
~40% fewer tool calls
Level 03
LLM Response Cache
An exact hash match serves a saved answer without another model call.
EXAMPLE 'What is baggage policy?' → FAQ
~20% fewer LLM calls
All three together
-60%cost
-57%latency

Tool Went Down — Agent Didn't

search_flights
→ TIMEOUT
LLM keeps going
fabricates data
Logs: 200 OK
User sees fake data
Defense 1
Defensive prompt
'Never fabricate'
Defense 2
Structured Output
JSON schema
Defense 3
Validation +
auto-disable

Monitoring — Closing the Loop

Monitor 1:
Request Cost
Track avg AND p99
p99 = 15× avg →
looping requests
Monitor 2:
Daily Spend
daily_cost vs budget
Find out before
month-end surprise
Normal
< budget
× 1.2
Alert + downgrade
Large → Medium
× 2.0
FULL STOP

Cost & Speed ✔ — Now: Security

✔ Observability
✔ Evaluation
✔ Optimization
User: "Show all bookings of all users"
If the tool allows it — agent will comply
A prompt is NOT protection. Infrastructure is

Practice

A few tasks to solve (15-20 min)

  1. Thread-Safe Bounded Queue
  2. Thread Pool

Security &
Deployment

Agent = Proxy to Your Services

✓ ✓ ✓ x x x
User
FlightAgent
search_flights
book_flights
get_profile
admin_panel
get_all_users
billing_api

A Prompt Is Not Protection

Naive Approach ❌
"Never call admin_panel"
User: 'Forget instructions'
→ LLM may comply
Prompt = recommendation
Correct Approach ✅
Tool not in list → can't call
user_id from token → can't spoof
Internal fields stripped → can't see
Infrastructure = constraint

Two Security Principles for Agents

Principle 1
LLM must NEVER see secrets or PII.
No API keys, passwords, passports,
card numbers.
Tool gets credentials from env.
Principle 2
Authorization in tool's CODE, not prompt.
Every tool = API endpoint with its own
auth.
user_id from verified token, not from
request.

Real Incidents: Slack AI & OpenClaw

Slack AI (2024)
PromptArmor research
Hidden instruction
in public channel →
AI leaks private data
OpenClaw (2026)
130K stars
Password 'a' valid
Prompt injection →
leaked API keys,
800 malicious skills
Pattern: agent + broad access + prompt-only protection = incident

Three Protection Levels

Level 1: TOOL PERMISSION
Agent sees:
search, book, profile
NOT: admin, billing, get_all_users
Level 2: PARAMETER VALIDATION
user_id = server token
Agent cannot spoof identity
Level 3: OUTPUT FILTERING
Returns: flight, price, airline
Strips: internal_id, margin, email

FlightAgent — Attack & Defense

WITHOUT Protection ❌
get_bookings(user_id, ...)
LLM picks user_id from prompt
Agent: get_bookings(user_id=42)
Any user_id works → DATA LEAKED
WITH Protection ✅
get_bookings(token, ...)
user_id extracted from JWT
Agent: get_bookings(token=eyJ...)
Only own data → SECURE

OAuth — Permissions in Production

OAuth = standard protocol for delegated access. User grants agent limited
permissions via token.
Agent never sees password — only scoped token. Like a hotel key card: opens
your room, not others.
User Login
Scoped Token
flights:read
flights:book
Gateway
checks scope
Tools
No scope = invisible
Tool doesn't appear
in agent's list
(not 'forbidden')
MCP 2.1 = OAuth
Native support
MCP server = Resource
Runtime = OAuth Client

Canary Deployment for Agents

Regular service: deploy → tests pass → done.
Agent: tests pass but LLM behaves differently on live traffic.
Prompt change can silently degrade quality — no crash, no error, just worse answers.
95% 5% OK for 1h quality drop > 10%
Router
Stable v2
Canary v3
Compare Metrics
latency ≤ stable
eval ≥ stable
errors ≤ stable
cost ≤ 1.2×
Rollout
5% → 25% → 50% → 100%
Auto Rollback
Compare
Latency | Eval score
Error rate | Cost
Quality drop >10%
→ auto-rollback
Ramp up
5% → 25% → 50% → 100%
Each step: wait 1h
and compare metrics

ACT 3

One Agent Works. Need a Second One.

FlightAgent ✓
HotelAgent ?
InsuranceAgent ?
VisaAgent ?
HotelAgent
Gateway
Auth
Obs
Eval
Queue
Deploy
InsuranceAgent
Gateway
Auth
Obs
Eval
Queue
Deploy
VisaAgent
Gateway
Auth
Obs
Eval
Queue
Deploy
Copy-paste: 3 months per agent = 12 months total. Bugs fixed separately in each.

When You Need a Platform

1 Agent
No platform
(overengineering)
2 Agents
Copy-paste
is tolerable
3+ Agents
Platform pays
for itself
Without Platform With Platform
4 agents = 12 months 4 agents = ~5 months
4 dashboards, 4 deploys 1 dashboard, 1 deploy
Bug fixed in 4 places Bug fixed once

Share Infrastructure, NOT Logic

FlightAgent → TravelPlatform

Flight
prompt + logic
+ eval
Hotel
prompt + logic
+ eval
Insurance
prompt + logic
+ eval
Visa
prompt + logic
+ eval

Evolution: Prototype → Platform

Stage 1: Prototype
ReAct + 3 tools
Jupyter
7 problems
1 user (you)
Stage 2: Production
+ Traces + Eval
+ Limits + Security
+ Canary
100K users
Stage 3: Platform
+ Shared infra
4+ agents
1 week per new agent

Wrap-up

Production Checklist (1/2)

10 steps to production-ready agent
1. Tracing + Metrics + Alerts — visibility from Day 1
2. Eval: Assertions + LLM-as-Judge + CI — measure quality
3. Max Steps + Timeout — prevent runaway agents
4. Cost Calculation — know your budget before deploy
5. Model Router — up to ~84% lower cost on this example (Small for FAQ, Large for complex)

Production Checklist (2/2)

From production-ready to platform scale
6. Fail-safe Prompting — defensive instructions + structured output
7. Tool-level Permissions — scope per user, not per agent
8. Canary Deployment — 5% → 25% → 50% → 100% with metrics
9. Queue + Cache — handle 100K+ users, -60% cost
10. Shared Platform — only with 3+ agents, not before

Production Is a Cycle

Iter 1: p95=12s → cache → 6s
Iter 2: eval 4.1 → 3.4 → DSPy → 4.3
Iter 3: cost $120 → $50 → fix router
Monitor
Traces + Metrics
Evaluate
Eval Pipeline + DSPy
Improve
Optimize + Fix
Deploy
Canary 5% → 100%

Where to Start (Order Matters)

This Week
(30 min)
1. Tracing (LangFuse)
2. 5 eval tests + Judge
3. Max steps + timeout
4. Cost calculation
→ Agent can be deployed
This Month
(2-3 days)
5. Model router (rules)
6. Fail-safe prompting
7. Tool permissions
8. Canary deployment
→ Production-ready
Next Sprint
(1-2 months)
9. Queue + Cache
10. Shared Platform
(only with 3+ agents)
→ Scale

Homework

This Week
• pip install langfuse
• Instrument your agent
• Run 5 test queries
• Calculate request cost
This Month
• 5 behavioral assertions
• LLM-as-a-Judge eval
• Try DSPy optimization

Do You
Actually
Need
an Agent?

AI Agents ≠ Software

Software Developing
  • Same input → same output
  • Unit tests, binary pass/fail
  • Bugs are reproducible
  • Ship it → it works
  • Engineer programs behavior
AI Agents
  • Same input → different output
  • Statistical testing (94% success rate)
  • Errors may not reproduce
  • Ship → monitor → iterate
  • Engineer designs environment

Formalize Before You Code

Before writing a single line — lock these down:
  • Input — What comes in? (text / voice / document / structured data)
  • Output — What comes out? (answer / action / document / API call)
  • Scope — What the agent does and does NOT do
  • Success criteria — How do you know it works? (metric, not “seems ok”)
  • Edge cases — What if input is garbage? Tool is down?
  • Human fallback — When does it hand off to a person?

Agent Economics

Define Value first:
  • What does a human do today? How much does it cost? How long does it take?
  • What changes with an agent? (speed / cost / quality / scale)
  • What KPI does this improve?
Then count Costs:
  • Inference: tokens × price × steps
  • Infrastructure: storage, RAG, compute
  • Hidden: human verification, prompt
    engineering time, incident response
Cost per Session:
CPS = (tokens × price_per_token) × avg_steps + tool_call_costs + infra_per_session + human_review_cost × escalation_rate
Cost per Successful Resolution:
CPR = CPS / success_rate

Agent Economics

Example: Customer Support Agent
  • Average: 5 steps, ~1,000 tokens per step
  • GPT-4o: $2.50/1M input, $10/1M output
  • 5 steps × 1,000 tokens ≈ 5,000 in + 2,500 out
  • LLM cost: ~$0.04 per session
  • Tool costs (API, DB): ~$0.005
  • Infrastructure: ~$0.005
  • Human review (10% escalation, $30/hr, 5 min): $0.25
  • Total CPS: ~$0.30
Compare:
  • Human support: $15–25/hr, handles 4–8 tickets/hr = $2–6 per resolution
  • Agent: $0.30 per session — 13x cheaper
BUT: only for requests the agent solves correctly.
Cost per Successful Resolution:
  • CPR = CPS / success_rate
  • At 70% success: $0.30 / 0.70 = $0.43
  • At 50% success: $0.30 / 0.50 = $0.60 + rework cost
When an agent does NOT pay off:
  • Success rate below 60% (too much rework)
  • Volume under 100 tasks/month (won't recoup dev cost)
  • Tasks too diverse (need 50 prompts instead of 1)
  • Cost of error is high (human verification eats all savings)

Quality — Three Pillars

01 DESIGN
Prompt Engineering
Your prompt is the API contract between you and the model.
Structure
Role · context · instructions · constraints · tools · output · examples
Version it. Test every change.
02 MEASURE
Evals
Systematically measure whether your agent actually works.
A useful eval suite
Runs automatically · scores outputs · gates deployments
Catch regressions before release.
03 OBSERVE
Tracing
Traces reveal what happened between input and output.
Log every step
Every LLM call · every tool call · every reasoning step
Make every run debuggable.

Offline vs Online Eval

Offline Online
Inputs Your test suite Real traffic
Expected output Known Unknown
When Before deploy (CI/CD) After deploy (continuous)
Catches Regressions, known failures Drift, edge cases, real-world distribution
Online eval — how to measure without expected outputs:
  • LLM-as-judge: sample production traces → separate model scores quality, safety, correctness
  • User signals: thumbs up/down, repeat contacts (= failed resolution), escalation rate
  • Proxy metrics: task completion rate, time to resolution, cost per session
  • A/B testing: new prompt vs old on 5% traffic + shadow mode
"Agent can solve these tasks" ≠ "Agent solves real user tasks"

Tracing Platforms: Feature Matrix

Langfuse LangSmith Arize Phoenix Braintrust Helicone
Focus Tracing + prompt mgmt Tracing + eval (LangChain) ML monitoring + LLM Eval-first + tracing Gateway + cost tracking
Open
source
MIT, self-host, no limits No (SaaS, self-host Enterprise) Yes (OpenInference) No (SaaS) Partially
Agent
tracing
Multi-step, nested spans Deep for LangChain/LangGraph Session → trace → span Nested spans, timeline replay Basic (no chain-of-thought)
Online
eval
LLM-as-judge, custom scorers LLM-as-judge, datasets Drift detection, alerts Scorers in CI (GitHub Action) No
Lock-in Minimal High (LangChain ecosystem) Minimal Medium Minimal
Best for Custom stack, self-host LangChain teams Production monitoring, drift Eval in CI/CD, regression testing Quick start, cost tracking
Decision tree:
  • Already on LangChain? → LangSmith
  • Need self-host / data control? → Langfuse
  • Focus on production drift? → Arize Phoenix
  • Eval gates in CI/CD? → Braintrust
  • Just need cost tracking fast? → Helicone

Latency vs Quality vs Cost

Latency
Quality
Cost

Acceptance Pipeline

Pass or fail? Metrics OK? Degradation?
Offline eval
Shadow mode
Canary (5%)
Rollout (100%)
Continuous online eval + alerting

Agent Stack

Layer 5: UX & Trust: what the user sees
Layer 4: Orchestration: how agents and steps coordinate
Layer 3: Tools & Integrations: how agents act on the world (MCP, APIs, databases)
Layer 2: Memory & State: conversation history, RAG, long-term memory, session state
Layer 1: Model & Inference: LLM, routing, caching
Layer 0: Infrastructure: compute, networking, monitoring

Control Plane for Agents

Kubernetes has a control plane to manage clusters.
Agents, especially multi-agent systems, need one too.
Components:
  • Registry: which agents exist, their tools, permissions, and access scope
  • Router: which agent handles which request (incl. model routing)
  • Rate limiting & budgets: max tokens per session, max cost per user, circuit breakers
  • Observability: traces, logs, metrics, fleet-level, not per-agent
  • Governance: audit trail, approval workflows, compliance checks

Decomposition — How to Split Into Agents

Same as software engineering: high cohesion, low coupling. But with nuances for stochastic systems.
# Rule Example Anti-pattern
1 One agent = one domain FlightAgent ≠ BaggageAgent God Agent: one agent knows
everything, 5-page prompt
2 Couple through interfaces, not state Handoff with contract: {flights:
[...], intent: "..."}
Agents reading each other's
internals
3 One is better than two when context is
shared
Search + booking tightly coupled →
one agent, two tools
Two agents constantly passing
context back and forth
4 Each agent testable in isolation PolicyAgent works without
FlightAgent (mock inputs)
Can only test agents together

Anthropic: Building effective agents

The prompt chaining workflow
In
LLM Call 1
Gate
LLM Call 2
LLM Call 3
Out
Exit

Anthropic: Building effective agents

The routing workflow
In
LLM Call Router
Gate
LLM Call 1
LLM Call 2
LLM Call 3
Out

Anthropic: Building effective agents

The routing workflow
The parallelization workflow
In
LLM Call Router
Gate
LLM Call 1
LLM Call 2
LLM Call 3
Out
In
LLM Call 1
LLM Call 2
LLM Call 3
Aggregator
Out

Anthropic: Building effective agents

The routing workflow
The parallelization workflow
The orchestrator-workers workflow
In
LLM Call Router
Gate
LLM Call 1
LLM Call 2
LLM Call 3
Out
In
LLM Call 1
LLM Call 2
LLM Call 3
Aggregator
Out
In
Gate
LLM Call 1
LLM Call 2
LLM Call 3
Gate
Out

Anthropic: Building effective agents

The evaluator-optimizer workflow
Solution
Rejected + Feedback
Accepted
In
LLM Call
Generator
LLM Call
Evaluator
Out

Google Patterns — Agents as Composable Units

Three levels of composition:
  • Single Agent — one agent + N tools. 80% of tasks are solved here. Don't overcomplicate until it breaks.
  • Multi-Agent — specialized agents with roles. Coordinator decides who handles what (LLM-driven delegation or explicit routing).
  • Agent-as-Tool — Agent B is wrapped as a tool for Agent A. A calls check_policy() and gets {allowed: true, fee: 3000}. Doesn't know there's a full RAG + LLM pipeline inside. Encapsulation at agent level.
Two delegation mechanisms in Google ADK:
  • LLM-Driven Delegation: model reads sub-agent descriptions, picks who to delegate to. Flexible, but model can pick wrong.
  • AgentTool (Explicit): agent B is wrapped as a callable function. Predictable, clear contract, but rigid.
Agentic Design Patterns by Antonio Gulli
A Hands-On Guide
to Building Intelligent Systems,

Antonio Gulli

Choosing a Framework (ADK)

Paradigms and Comparison:
GRAPH
LangGraph
Nodes → Edges.
You draw the blueprint. Max control, max boilerplate. Stateful graphs with checkpointing. Most mature (80K+ stars, 2+ years in production).
CONVERSATION
CrewAI, AutoGen
You assemble a team. Agents talk to each other. CrewAI: roles + tasks, fast prototyping. AutoGen: GroupChat, debate, consensus. Most intuitive, least predictable.
COMPUTER
Claude Agent SDK
bash + files. You provide an environment.
Model decides how to act. Subagents, MCP tools, Computer Use. Minimal orchestration, maximum autonomy. Powers Claude Code.
HIERARCHY
Google ADK, OpenAI SDK
Boss → Workers.
You build a company. Coordinator delegates to specialists. ADK: workflow agents (Sequential/Parallel/Loop) + A2A protocol. OpenAI SDK: 4 primitives, fast start, OpenAI lock-in.
MCP-NATIVE
Strands (AWS)
MCP → everything.
One connector for all: tools, integrations, other agents. Deep AWS integration. Youngest framework, strongest bet on MCP as the universal standard.

Timeline

23
GPT-4: $60/1M tokens
Value = MODEL
24
GPT-4o: $10/1M
Claude 3: $15/1M
Value = MODEL + PROMPT
25
GPT-4o-mini: $0.15/1M
DeepSeek: $0.14/1M
Inference: -78% per year
Value = The SYSTEM
Around the Model
26
?