nup_logo

Building AI Agents

Case Studies & Tech Debates
Topics, Formats & Guidelines


Alex Avdiushenko
Fall 2026

Real-World Cases

Industry systems, architecture trade-offs

In the foundational sessions we built core primitives from scratch: loops, tools, MCP, memory, and evals. Now it's time to focus on understanding real systems in depth and rigorously defending architectural choices.
Task A
In-Depth Case Studies
Deep-dive into a publicly shared agent architecture, SDK, or production system. Reverse-engineer or look at GitHub design choices, problems encountered, run reproduction benchmarks, and deliver a 45-min long technical presentation. Be ready to defend your findings.
Individual format
Task B
Technical Debates
Structured, evidence-based debates on high-stakes industry questions. Two opposing teams prepare empirical data, benchmark citations, and sharp arguments to stress-test architectural decisions.
Adversarial format: 2 teams (2-3 students each)
Objective
Maximizing Cohort Utility
Every talk and debate should provide actionable insights, architectural diagrams, and evaluated trade-offs that expand the collective engineering knowledge of the entire cohort.
Deliverables committed to the course materials

Task A: Case Study & 1-Hour Talk

Investigate a public architecture or production framework, then teach it to the room

Talk Structure (60 Minutes)
  • 40–45 min: Technical Deep-Dive & Architecture Walkthrough
  • 10 min: Live Demonstration, Code Reproduction, or Benchmark Results
  • 10 min: Interactive Q&A and Discussion with the Cohort
Aim for engineering depth rather than high-level slides
Required Deliverables
  • System Architecture Diagram: Detailed components, state flow
  • Reproduction or Prototype: Working code sample, trace logs, or benchmark
  • Failure Modes & Trade-Offs: Where does the design fail? What does it cost?
  • Slide Deck & Notes: Better to publish for peer access
Code & slides submitted to the course materials
Key Metric Can another engineer and AI agent implement or debug this system based solely on your presentation and artifacts?

Case 1: PI.dev Architecture In-Depth

An open-sourced autonomous developer agent harness & sandbox runtime

Core Challenge: Autonomous coding agents require fast execution loops, untrusted command sandboxing, rich filesystem state management, and resilient error recovery across hours of uninterrupted execution.
Loop & Context Management
  • How does PI.dev structure the primary agent loop?
  • Context window budgeting & prompt cache prefix stability.
  • Summarization vs truncation for long terminal outputs.
  • Handling partial tool streaming and token budgeting.
Sandbox & Execution Isolation
  • MicroVM vs container isolation for bash & file ops.
  • Bidirectional synchronization of workspace files.
  • Security boundaries: protecting against rogue tool invocations.
  • Interception and human-in-the-loop permission gates.
Subagents & Recovery
  • Parent-child agent delegation protocols.
  • How does the system detect and break infinite command loops?
  • State persistence across disconnects and browser sessions.
  • Cost & latency tracking per resolved engineering issue.
Deliverable End-to-end architecture diagram + simulated reproduction of the execution harness on sample coding tasks.

Case 2: Local Models in Google's Antigravity SDK

Edge execution, on-device agent loops, and hybrid cloud-local dispatch

Core Challenge: Running agent loops with local open models (e.g. Gemma, Llama) eliminates API costs and preserves privacy, but introduces strict memory limits, higher schema error rates, and constrained reasoning depth.
Local Runtime Architecture
  • Interface design in Google's Antigravity SDK.
  • Runtime backends: ONNX Runtime, Metal/CUDA, Ollama, vLLM.
  • Constrained decoding & grammar-guided JSON generation.
  • Memory footprint: KV-cache sizing on consumer hardware.
Tool Calling Fidelity
  • Comparing tool call accuracy: 7B/9B models vs frontier cloud models.
  • Sensitivity to schema complexity and parameter count.
  • Self-correction loops when a local model emits invalid arguments.
  • Benchmarking end-to-end task completion rates.
Hybrid Orchestration
  • Dynamic routing: when to run locally vs dispatching to cloud.
  • Privacy tiers: classifying sensitive queries for local-only processing.
  • Fallback cascades upon local reasoning exhaustion.
  • Comparative cost/energy analysis per resolved task.
Deliverable Working prototype running on a local open model, benchmarked against cloud API on a test suite of tools.

Case 3: What Happens in Context No Longer Stays in Context

Consolidating in-context agent trajectories into persistent memory & model weights

Core Challenge: Trajectory experience shifts token distributions, but keeping history in-context is expensive, hits window limits, and vanishes on reset. Continual learning approaches consolidate in-context behaviors into persistent memory and model weights.
ByteDance Seed & Monash
Experience Distillation
  • Trajectory Distillation: Teacher (task + trial experience τexp) distills policy into student without extra environment steps.
  • Retains ≥64.8% of in-context learning gains, vs only 3.8% for direct SFT on identical experience.
  • Matches RL baselines with ≥9.6× fewer environment samples across 749 SWE tasks & text games.
  • Internalizes trial-and-error reasoning directly into model parameters.
Microsoft Research
Online Experiential Learning
  • Two-Stage Continual Loop: Extract reusable lessons from trajectories → consolidate into weights via On-Policy Context Distillation.
  • Reasoning Efficiency: Qwen3-1.7B cuts response length to ~70% on Frozen Lake by round 3 while boosting accuracy.
  • On-Policy Consistency: Agent learning from own experience outperforms distillation from larger models (Qwen3-4B).
  • Preserves out-of-distribution instruction-following without task collapse.
Research & Reproduction
Engineering Takeaways
  • Context vs Weights: When to maintain episodic memory vs triggering weight consolidation.
  • Sample Efficiency: Replacing expensive online RL exploration with in-context trial-and-error distillation.
  • Failure Modes: Catastrophic forgetting, over-consolidation, and off-policy drift.
  • Deliverable: Comparative architecture diagram + reproduction on a gym/code benchmark.

Case 4: Omarchy — Can You Vibe Code an OS?

Agentic desktop environments, and the boundaries of LLM systems programming

Core Challenge & Doctrine: DHH's Omarchy asserts: “When you can vibe code whatever app comes to your mind, you should be able to vibe code your operating system.” Can autonomous agents reliably build, configure, and self-heal an entire operating system, or are they orchestrating mutable userland glue over a mature Linux kernel?
Userland Reality
What is Actually “Vibe Coded”?
  • Kernel vs. Userland: The Linux kernel, drivers, and C toolchains remain fixed; agents generate Hyprland/Wayland configs, systemd units, and glue scripts.
  • Malleable Desktop: Natural language prompts generate custom status widgets, keybindings, workflow automations, and live system diagnostics.
  • Tight Feedback Loops: Shell exit codes, logs (journalctl), and instant UI visual rendering provide immediate ground-truth feedback.
  • Agentic Sysadmin: Replacing manual StackOverflow / ArchWiki research with autonomous problem-solving loops.
Engineering Limits
Bottlenecks & Failure Modes
  • Security & Privilege Escalation: Giving agents raw sudo / root shell access opens critical prompt-injection and destructive execution hazards.
  • State Drift & Config Rot: Mutating live system files without declarative reproducibility (like NixOS) leads to non-deterministic, irreversible system breakage.
  • Deep Systems Programming: LLMs struggle with kernel concurrency, low-level memory allocators, and hardware drivers lacking unit-test harnesses.
  • Maintenance Burden: Unvetted, vibe-coded bash/glue scripts accumulate silent regressions and edge-case fragility.
Architectural Lessons
Key Takeaways & Deliverables
  • Agents as Desktop Operators: Shifting from isolated IDE sandboxes to first-class desktop orchestrators with telemetry access.
  • Declarative Foundations: Why atomic, declarative architectures (Nix/Guix) are essential for reliable, rollback-safe agentic OS mutation.
  • Evaluation: Benchmarking agent OS reliability against benchmarks like OS-World and Linux system administration suites.
  • Deliverable: Architecture breakdown + live demo of an agent diagnosing and self-healing a broken desktop configuration.
Reference Omarchy by DHH (omarchy.org/#watch) • Agentic Linux distribution, desktop vibe coding & system-level autonomy

Task B: Structured Technical Debates

Sharpening architectural decisions through rigorous, evidence-based contest

Debates force both teams to examine empirical evidence, quantify cost and latency trade-offs, and uncover the boundary conditions where opposing approaches succeed or fail.
1. Opening Theses
5 min per team. Clear thesis statement, foundational principles, and prime evidence supporting the assigned position.
2. Cross-Exam
5 min per team. Direct, pointed questions exposing weak assumptions, edge cases, and hidden failure modes in the opponent's model.
3. Rebuttals
4 min per team. Countering challenges with concrete production data, benchmarks, or real-world incident postmortems.
4. Q&A & Verdict
10 min. Audience & instructor probing questions, followed by cohort vote based on technical rigor, not rhetorical flair.
Rule Every claim must be backed by benchmark citations, mathematical models, or verified production architecture studies.

Debate 1: Regulate & Slow Down vs. Open Velocity

Motion: "Frontier AI development should be actively slowed down and subjected to strict government licensing."

Team Affirmative (Pro-Regulation) Safety & Caution
  • Catastrophic Misuse Risks: Autonomous cyberwarfare, CBRN weapon synthesis, and uncontainable propagation cannot be fixed post-release.
  • Alignment & Loss of Control: Capabilities consistently outpace interpretability; safety standards must be legally binding.
  • Externalities & Liability: Unregulated model releases impose massive societal costs without developer accountability.
  • Precedents: Aviation, nuclear, and pharma prove that rigorous pre-deployment certification enhances sustainable progress.
Team Negative (Pro-Velocity) Open Innovation
  • Regulatory Capture: Heavy licensing regimes protect incumbent tech monopolies and suffocate open-source ecosystems.
  • Game-Theoretic Futility: Unilateral national slowdowns forfeit technical leadership to geopolitical rivals with zero safety treaties.
  • Defense Needs Frontier Models: The only viable defense against malicious autonomous agents is a more capable defensive agent.
  • Tremendous Opportunity Cost: Slowing AI directly delays breakthroughs in disease cure, clean energy, and scientific automation.
Core Tension Can regulatory gatekeeping prevent catastrophic tail risks without halting decentralized technical and economic progress?

Debate 2: Multi-Agent Systems vs. Single Strong Agent

Motion: "Multi-agent systems are the dominant future architecture for complex tasks, not unnecessary complexity."

Team Affirmative (Multi-Agent Specialization) Modular Teams
  • Separation of Concerns: Narrow prompts, dedicated tool subsets, and tailored schemas outperform monolithic system prompts.
  • Independent Verification: Generator-critic and planner-worker topologies eliminate self-confirmation bias.
  • Context Window Hygiene: Subagents maintain compact, focused contexts, avoiding catastrophic distraction in 50+ step runs.
  • Parallelism: Independent sub-tasks run concurrently across multiple agents, shrinking wall-clock latency.
Team Negative (Single Strong Agent + Tools) Linear Simplicity
  • Compounding Failure Rates: If each agent has a 90% success rate, a 5-agent sequential pipeline succeeds only 59% of the time.
  • Coordination Tax: 3x–6x higher token consumption, enormous latency penalties, and fragile inter-agent communication.
  • SWE-bench Empirical Truth: The highest-scoring submissions are almost universally single agents with great tools and tight loops.
  • Frontier Model Capabilities: As models scale, the need for artificial role-playing collapses into standard tool execution.
Core Tension Does multi-agent modularity provide real robustness, or is it an accidental anti-pattern born of weak model capabilities?

Evaluation Rubric & Standards

What we value in technical presentations and debate performances

33% of Score
Technical Rigor & Depth
Demonstrated understanding of inner mechanics, protocols, schemas, and wire formats. No hand-waving or marketing slides; precise engineering terminology and accurate explanations.
33% of Score
Empirical Evidence & Demos
Grounding arguments in verified numbers, live prototypes, reproduction code, or trace logs. Testing boundaries and demonstrating concrete failure modes under pressure.
33% of Score
Clarity & Peer Value
Clean visual structure, effective audience engagement, ability to defend positions during Q&A, and high-utility artifacts committed to the repository for the whole cohort.
Artifacts All presentation slides, benchmark scripts, and architecture specs should be uploaded into the course materials.

Next Steps: Team Formation & Topics

How to get started with your topic

Step 1
Form Teams
Group into teams of 2–3 students for Task B (Debates) or individually for Task A (Cases). Coordinate with peers in the course chat.
Step 2
Select or Propose Topic
Choose one of the showcased topics or propose an alternative system/motion using the same template. Confirm topic selection with the instructor.
Questions or new case ideas? Reach out in the course channel or discuss with me in chat.