| print() / unstructured | Structured logging |
|---|---|
| "Processing request..." | {"event": "request_start", |
| "Called search_flights" | "user_id": "u123", |
| "Got 5 results" | "tool": "search_flights", |
| No context, no search | "latency_ms": 1200} |
| METRIC | WHY IT MATTERS |
|---|---|
| Latency (p50, p95) | User waits this long. p95 > 10s → users leave |
| Steps per request | More steps = more $, more latency, more loops |
| Tokens per request | Directly drives cost. Grows as context accumulates |
| Tool success rate | Drop → external API broke or bad params |
| Task success rate | The only metric users feel. Everything else is proxy |
| Error rate | Spike → something broke. Silent errors are worse |
| Without Alerts | With Alerts |
|---|---|
| User complains in support chat | Slack ping in 5 minutes |
| You check dashboard... manually | Open trace → find root cause |
| 3 hours later: 'LLM is slow' | Fix before users notice |
| End-of-month: surprise $5K bill | Budget alert → switch model |
| Manual | DSPy |
|---|---|
| Write prompt | Define task + metric |
| Run eval | Algorithm finds prompt |
| Edit prompt | Automatic optimization |
| Repeat... |
class FlightSearch(
dspy.Signature
):
"""Find best flight"""
query = dspy.InputField()
flights = dspy.InputField()
answer = dspy.OutputField()
agent = dspy.ChainOfThought(
FlightSearch
)
# Adds "Reasoning: ..."
# before answer
# automatically
def metric(example, pred):
return (
pred.answer != "" and
"cheapest" in pred.answer
)
opt = BootstrapFewShot(
metric=metric,
max_demos=3
)
compiled = opt.compile(
agent, trainset=data
)
| Prompts NOT in git | Prompts IN git |
|---|---|
| In Notion: who changed? | Every change = PR + review |
| In UI: no history, no diff | Diff: see what changed |
| In your head: bus factor 1 | Rollback: 1 command |
| Regression: unnoticed | CI: eval runs on every PR |
| Configuration | Price / 1M (input / output) | Per day | Per 30 days |
|---|---|---|---|
| 100% Large — GPT-6.1 Sol | \$2 / \$10 | \$100.80 | \$3,024 |
| 100% Medium — GPT-6.1 Terra | \$1 / \$5 | \$50.40 | \$1,512 |
| 100% Small — GPT-6 Luna | \$0.10 / \$0.50 | \$5.04 | \$151.20 |
| Router 70/20/10 (S/M/L) | \$0.47 / \$2.35 | \$23.69 | \$710.64 |
| Router 80/15/5 (S/M/L) | \$0.33 / \$1.65 | \$16.63 | \$498.96 |
| Before: 4.2s/req | After: 2.8s/req |
|---|---|
| 1 worker ≈ 14 req/min | 1 worker ≈ 21 req/min |
| 100 req/min → 7 workers | 100 req/min → 5 workers |
| Why agents loop | Cost distribution |
|---|---|
| Tool returns empty → retry | 95% normal: $0.003–$0.015 |
| LLM misunderstands → repeat | 4% expensive: $0.05–$0.15 |
| Ambiguous query → search | 1% catastrophic: $0.15–$0.50 |
| → refine → search → refine... | |
| No stop condition in prompt | 1% costs = 20% of normals |
A few tasks to solve (15-20 min)
| Without Platform | With Platform |
|---|---|
| 4 agents = 12 months | 4 agents = ~5 months |
| 4 dashboards, 4 deploys | 1 dashboard, 1 deploy |
| Bug fixed in 4 places | Bug fixed once |
| Offline | Online | |
|---|---|---|
| Inputs | Your test suite | Real traffic |
| Expected output | Known | Unknown |
| When | Before deploy (CI/CD) | After deploy (continuous) |
| Catches | Regressions, known failures | Drift, edge cases, real-world distribution |
| Langfuse | LangSmith | Arize Phoenix | Braintrust | Helicone | |
|---|---|---|---|---|---|
| Focus | Tracing + prompt mgmt | Tracing + eval (LangChain) | ML monitoring + LLM | Eval-first + tracing | Gateway + cost tracking |
| Open source |
MIT, self-host, no limits | No (SaaS, self-host Enterprise) | Yes (OpenInference) | No (SaaS) | Partially |
| Agent tracing |
Multi-step, nested spans | Deep for LangChain/LangGraph | Session → trace → span | Nested spans, timeline replay | Basic (no chain-of-thought) |
| Online eval |
LLM-as-judge, custom scorers | LLM-as-judge, datasets | Drift detection, alerts | Scorers in CI (GitHub Action) | No |
| Lock-in | Minimal | High (LangChain ecosystem) | Minimal | Medium | Minimal |
| Best for | Custom stack, self-host | LangChain teams | Production monitoring, drift | Eval in CI/CD, regression testing | Quick start, cost tracking |
| # | Rule | Example | Anti-pattern |
|---|---|---|---|
| 1 | One agent = one domain | FlightAgent ≠ BaggageAgent | God Agent: one agent knows everything, 5-page prompt |
| 2 | Couple through interfaces, not state | Handoff with contract: {flights: |
Agents reading each other's internals |
| 3 | One is better than two when context is shared |
Search + booking tightly coupled → one agent, two tools |
Two agents constantly passing context back and forth |
| 4 | Each agent testable in isolation | PolicyAgent works without FlightAgent (mock inputs) |
Can only test agents together |
check_policy() and gets {allowed: true, fee: 3000}. Doesn't know there's a full RAG + LLM pipeline inside. Encapsulation at agent level.