Real-World Cases
Industry systems, architecture trade-offs
In the foundational sessions we built core primitives from scratch: loops, tools, MCP, memory, and evals.
Now it's time to focus on understanding real systems in depth
and rigorously defending architectural choices.
Task A
In-Depth Case Studies
Deep-dive into a publicly shared agent architecture,
SDK, or production system.
Reverse-engineer or look at GitHub design choices,
problems encountered,
run reproduction benchmarks, and deliver a 45-min long technical presentation.
Be ready to defend your findings.
Individual format
Task B
Technical Debates
Structured, evidence-based debates on high-stakes
industry questions.
Two opposing teams prepare empirical data,
benchmark citations, and sharp arguments to stress-test
architectural decisions.
Adversarial format: 2 teams (2-3 students each)
Objective
Maximizing Cohort Utility
Every talk and debate should provide actionable insights,
architectural diagrams, and evaluated trade-offs
that expand the collective engineering knowledge of the entire cohort.
Deliverables committed to the course materials
Task A: Case Study & 1-Hour Talk
Investigate a public architecture or production framework, then teach it to the room
Talk Structure (60 Minutes)
- 40–45 min: Technical Deep-Dive & Architecture Walkthrough
- 10 min: Live Demonstration, Code Reproduction, or Benchmark Results
- 10 min: Interactive Q&A and Discussion with the Cohort
Aim for engineering depth rather than high-level slides
Required Deliverables
- System Architecture Diagram: Detailed components, state flow
- Reproduction or Prototype: Working code sample, trace logs, or benchmark
- Failure Modes & Trade-Offs: Where does the design fail? What does it cost?
- Slide Deck & Notes: Better to publish for peer access
Code & slides submitted to the course materials
Key Metric
Can another engineer and AI agent implement or debug this system based solely on your presentation and artifacts?
Case 1: PI.dev
Architecture In-Depth
An open-sourced autonomous developer agent harness & sandbox runtime
Core Challenge: Autonomous coding agents require fast execution loops,
untrusted command sandboxing,
rich filesystem state management, and resilient error recovery across
hours of uninterrupted execution.
Loop & Context Management
- How does PI.dev structure the primary agent loop?
- Context window budgeting & prompt cache prefix stability.
- Summarization vs truncation for long terminal outputs.
- Handling partial tool streaming and token budgeting.
Sandbox & Execution Isolation
- MicroVM vs container isolation for bash & file ops.
- Bidirectional synchronization of workspace files.
- Security boundaries: protecting against rogue tool invocations.
- Interception and human-in-the-loop permission gates.
Subagents & Recovery
- Parent-child agent delegation protocols.
- How does the system detect and break infinite command loops?
- State persistence across disconnects and browser sessions.
- Cost & latency tracking per resolved engineering issue.
Deliverable
End-to-end architecture diagram + simulated reproduction of the execution harness on sample coding tasks.
Case 2: Local Models in Google's Antigravity SDK
Edge execution, on-device agent loops, and hybrid cloud-local dispatch
Core Challenge: Running agent loops with local open models (e.g. Gemma, Llama) eliminates API costs and preserves privacy,
but introduces strict memory limits, higher schema error rates, and constrained reasoning depth.
Local Runtime Architecture
- Interface design in Google's Antigravity SDK.
- Runtime backends: ONNX Runtime, Metal/CUDA, Ollama, vLLM.
- Constrained decoding & grammar-guided JSON generation.
- Memory footprint: KV-cache sizing on consumer hardware.
Tool Calling Fidelity
- Comparing tool call accuracy: 7B/9B models vs frontier cloud models.
- Sensitivity to schema complexity and parameter count.
- Self-correction loops when a local model emits invalid arguments.
- Benchmarking end-to-end task completion rates.
Hybrid Orchestration
- Dynamic routing: when to run locally vs dispatching to cloud.
- Privacy tiers: classifying sensitive queries for local-only processing.
- Fallback cascades upon local reasoning exhaustion.
- Comparative cost/energy analysis per resolved task.
Deliverable
Working prototype running on a local open model, benchmarked against cloud API on a test suite of tools.
Case 3: What Happens in Context No Longer Stays in Context
Consolidating in-context agent trajectories into persistent memory & model weights
Core Challenge: Trajectory experience shifts token distributions, but keeping history in-context is expensive, hits window limits, and vanishes on reset.
Continual learning approaches consolidate in-context behaviors into persistent memory and model weights.
ByteDance Seed & Monash
Experience Distillation
- Trajectory Distillation: Teacher (task + trial experience τexp) distills policy into student without extra environment steps.
- Retains ≥64.8% of in-context learning gains, vs only 3.8% for direct SFT on identical experience.
- Matches RL baselines with ≥9.6× fewer environment samples across 749 SWE tasks & text games.
- Internalizes trial-and-error reasoning directly into model parameters.
Microsoft Research
Online Experiential Learning
- Two-Stage Continual Loop: Extract reusable lessons from trajectories → consolidate into weights via On-Policy Context Distillation.
- Reasoning Efficiency: Qwen3-1.7B cuts response length to ~70% on Frozen Lake by round 3 while boosting accuracy.
- On-Policy Consistency: Agent learning from own experience outperforms distillation from larger models (Qwen3-4B).
- Preserves out-of-distribution instruction-following without task collapse.
Research & Reproduction
Engineering Takeaways
- Context vs Weights: When to maintain episodic memory vs triggering weight consolidation.
- Sample Efficiency: Replacing expensive online RL exploration with in-context trial-and-error distillation.
- Failure Modes: Catastrophic forgetting, over-consolidation, and off-policy drift.
- Deliverable: Comparative architecture diagram + reproduction on a gym/code benchmark.
Case 4: Omarchy — Can You Vibe Code an OS?
Agentic desktop environments, and the boundaries of LLM systems programming
Core Challenge & Doctrine: DHH's Omarchy asserts: “When you can vibe code whatever app comes to your mind, you should be able to vibe code your operating system.”
Can autonomous agents reliably build, configure, and self-heal an entire operating system, or are they orchestrating mutable userland glue over a mature Linux kernel?
Userland Reality
What is Actually “Vibe Coded”?
- Kernel vs. Userland: The Linux kernel, drivers, and C toolchains remain fixed; agents generate Hyprland/Wayland configs, systemd units, and glue scripts.
- Malleable Desktop: Natural language prompts generate custom status widgets, keybindings, workflow automations, and live system diagnostics.
- Tight Feedback Loops: Shell exit codes, logs (journalctl), and instant UI visual rendering provide immediate ground-truth feedback.
- Agentic Sysadmin: Replacing manual StackOverflow / ArchWiki research with autonomous problem-solving loops.
Engineering Limits
Bottlenecks & Failure Modes
- Security & Privilege Escalation: Giving agents raw sudo / root shell access opens critical prompt-injection and destructive execution hazards.
- State Drift & Config Rot: Mutating live system files without declarative reproducibility (like NixOS) leads to non-deterministic, irreversible system breakage.
- Deep Systems Programming: LLMs struggle with kernel concurrency, low-level memory allocators, and hardware drivers lacking unit-test harnesses.
- Maintenance Burden: Unvetted, vibe-coded bash/glue scripts accumulate silent regressions and edge-case fragility.
Architectural Lessons
Key Takeaways & Deliverables
- Agents as Desktop Operators: Shifting from isolated IDE sandboxes to first-class desktop orchestrators with telemetry access.
- Declarative Foundations: Why atomic, declarative architectures (Nix/Guix) are essential for reliable, rollback-safe agentic OS mutation.
- Evaluation: Benchmarking agent OS reliability against benchmarks like OS-World and Linux system administration suites.
- Deliverable: Architecture breakdown + live demo of an agent diagnosing and self-healing a broken desktop configuration.
Task B: Structured Technical Debates
Sharpening architectural decisions through rigorous, evidence-based contest
Debates force both teams to examine empirical evidence, quantify cost and latency trade-offs,
and uncover the boundary conditions where opposing approaches succeed or fail.
1. Opening Theses
5 min per team. Clear thesis statement, foundational principles, and prime evidence supporting the assigned position.
2. Cross-Exam
5 min per team. Direct, pointed questions exposing weak assumptions, edge cases, and hidden failure modes in the opponent's model.
3. Rebuttals
4 min per team. Countering challenges with concrete production data, benchmarks, or real-world incident postmortems.
4. Q&A & Verdict
10 min. Audience & instructor probing questions, followed by cohort vote based on technical rigor, not rhetorical flair.
Rule
Every claim must be backed by benchmark citations, mathematical models, or verified production architecture studies.
Debate 1: Regulate & Slow Down vs. Open Velocity
Motion: "Frontier AI development should be actively slowed down and subjected to strict government licensing."
- Catastrophic Misuse Risks: Autonomous cyberwarfare, CBRN weapon synthesis, and uncontainable propagation cannot be fixed post-release.
- Alignment & Loss of Control: Capabilities consistently outpace interpretability; safety standards must be legally binding.
- Externalities & Liability: Unregulated model releases impose massive societal costs without developer accountability.
- Precedents: Aviation, nuclear, and pharma prove that rigorous pre-deployment certification enhances sustainable progress.
- Regulatory Capture: Heavy licensing regimes protect incumbent tech monopolies and suffocate open-source ecosystems.
- Game-Theoretic Futility: Unilateral national slowdowns forfeit technical leadership to geopolitical rivals with zero safety treaties.
- Defense Needs Frontier Models: The only viable defense against malicious autonomous agents is a more capable defensive agent.
- Tremendous Opportunity Cost: Slowing AI directly delays breakthroughs in disease cure, clean energy, and scientific automation.
Core Tension
Can regulatory gatekeeping prevent catastrophic tail risks without halting decentralized technical and economic progress?
Debate 2: Multi-Agent Systems vs. Single Strong Agent
Motion: "Multi-agent systems are the dominant future architecture for complex tasks, not unnecessary complexity."
- Separation of Concerns: Narrow prompts, dedicated tool subsets, and tailored schemas outperform monolithic system prompts.
- Independent Verification: Generator-critic and planner-worker topologies eliminate self-confirmation bias.
- Context Window Hygiene: Subagents maintain compact, focused contexts, avoiding catastrophic distraction in 50+ step runs.
- Parallelism: Independent sub-tasks run concurrently across multiple agents, shrinking wall-clock latency.
- Compounding Failure Rates: If each agent has a 90% success rate, a 5-agent sequential pipeline succeeds only 59% of the time.
- Coordination Tax: 3x–6x higher token consumption, enormous latency penalties, and fragile inter-agent communication.
- SWE-bench Empirical Truth: The highest-scoring submissions are almost universally single agents with great tools and tight loops.
- Frontier Model Capabilities: As models scale, the need for artificial role-playing collapses into standard tool execution.
Core Tension
Does multi-agent modularity provide real robustness, or is it an accidental anti-pattern born of weak model capabilities?