System Design: AI Software Engineer (From Autocomplete to Autonomous App Builder)
Goal: Build a system that predicts the next line of code in 300ms, refactors 12 files in 45 seconds, and builds an entire app from a one-sentence spec over 4 hours.
1 million developers. 100 million completions per day. This is the blueprint.
How to read this: Three stories, three time scales.
Story one: a keystroke becomes ghost text in 300ms. Story two: "refactor auth to JWT" becomes a 12-file diff in 45 seconds.
Story three: "build me a SaaS app" becomes a deployed product in 4 hours. Each story goes deeper into the system.
1. Problem Statement and Scale
"Help developers write code." Sounds simple. It isn't. Five problems make this hard:
- The blank page. A developer types
func processPayment(and stares at an empty body. The system has 300 milliseconds to predict what comes next before the developer types another character. Miss that window and the suggestion is useless. - The 10,000-file maze. Someone says "refactor auth from sessions to JWT." The relevant code lives in 12 files out of 10,000. The developer doesn't know which 12. The system needs to find them, understand them, plan the changes, execute across all 12, run tests, and fix anything that breaks. Under a minute.
- The "build me an app" problem. Developer has an idea and zero code. The system takes a one-sentence spec, asks the right questions, designs the architecture, scaffolds the project, builds every module, handles errors, and treats "make the sidebar darker" and "actually switch to GraphQL" as equally valid mid-flight corrections. Autonomously. Over hours.
- The quality wall. Every suggestion must parse, type-check, only reference imports that actually exist, match the project's coding style, and not introduce security holes. One hallucinated import and the developer spends 20 minutes debugging "module not found."
- The money math. 100M completions per day. With API pricing, each completion costs $0.001 at the cheapest tier and $0.05 at the most expensive. With 1M developers, blended compute runs $4.5-6.5M/month. Revenue needs to exceed that. Self-hosting GPUs cuts compute to ~$500K/month, which completely changes the economics.
Scale targets:
| Metric | Target |
|---|---|
| Developers | 1,000,000 |
| Completions per day | 100,000,000 |
| Agent sessions per day | 1,500,000 |
| Autonomous build sessions per day | 50,000 |
| QPS (average / peak) | 1,200 / 3,000 |
| P50 completion latency | < 400ms |
| P99 completion latency | < 800ms |
The Three Levels
AI code assistants are three products stacked on top of each other:
| Level | What It Does | Time Budget | Example |
|---|---|---|---|
| L1: Autocomplete | Predicts next lines as the developer types | 300ms | GitHub Copilot, Cursor |
| L2: Codebase Agent | Searches, reads, edits, tests across files | 10-60s | Cursor Agent, Copilot Agent, Claude Code |
| L3: AI Software Engineer | Builds apps from spec, runs for hours | Minutes-hours | Claude Code, OpenAI Codex, Cursor Cloud Agents |
These levels stack. L3 runs L2's agent loop for every subtask.
L2 uses L1's context engine to read code. Skipping levels does not work.
The deeper into the stack, the less the model matters and the more the system around it matters.
| Level | Model's Contribution | System's Contribution | What Determines Quality |
|---|---|---|---|
| L1 | ~50% | ~50% | Context assembly + inference equally |
| L2 | ~25% | ~75% | Retrieval, tools, and verification dominate |
| L3 | ~10% | ~90% | Scheduling, memory, and recovery dominate |
How Real Systems Map to These Levels
| System | L1 (Autocomplete) | L2 (Agent) | L3 (Autonomous) | Primary Strength |
|---|---|---|---|---|
| GitHub Copilot | Best-in-class | Strong (agent mode + coding agent + sub-agents, GA March 2026) | Emerging (coding agent: issue → PR) | Inline completion + deep IDE integration |
| Cursor | Good | Strong (codebase-aware agent) | Strong (Cloud Agents on VMs, multi-agent, Automations platform) | IDE-integrated agent + strongest autonomous UX |
| Claude Code | N/A (CLI, no ghost text) | Strong (tool-based, subagents) | Strong (auto mode, /loop background tasks, hours-long sessions) | Deep reasoning + autonomous workflows |
| OpenAI Codex | N/A (Codex CLI for terminal) | Strong (cloud sandbox per task) | Strong (GPT-5.3-Codex, parallel worktrees, 7+ hour tasks) | Cloud-native autonomy + ChatGPT integration |
As of March 2026, everyone has decent L2 agents. The real fight is at L1 (Copilot still wins on raw completion speed) and L3 (Cursor, Claude Code, and Codex are racing for autonomous territory).
Nobody covers all three levels equally well yet. The architecture here is the union of all of them.
2. Requirements
2.1 Functional Requirements
| ID | Requirement | Priority |
|---|---|---|
| FR-01 | Inline code completion: predict next lines from cursor position | P0 |
| FR-02 | Multi-line completion: generate entire function bodies, code blocks | P0 |
| FR-03 | Fill-in-the-middle: complete code where cursor is between existing code | P0 |
| FR-04 | Codebase-aware suggestions: use project files, imports, types as context | P0 |
| FR-05 | Multi-file agent: search, read, edit, create, delete files across a project | P0 |
| FR-06 | Tool execution: run shell commands (tests, build, lint) and use results | P0 |
| FR-07 | Streaming responses: token-by-token delivery with sub-200ms TTFT | P0 |
| FR-08 | Code review: analyze PR diffs for bugs, security issues, missing tests | P1 |
| FR-09 | Project scaffolding: create new projects from natural language spec | P1 |
| FR-10 | Iterative build: implement features through multi-turn feedback loops | P1 |
| FR-11 | Long-running sessions: maintain context and progress across hours of work | P1 |
| FR-12 | Memory: remember project architecture, decisions, and conventions across sessions | P1 |
| FR-13 | Acceptance tracking: log shown/accepted/rejected/partial for model improvement | P1 |
| FR-14 | Multi-model routing: select optimal model per task for cost/quality tradeoff | P2 |
| FR-15 | Deployment pipeline: generate CI/CD and deploy to cloud platforms | P2 |
2.2 Non-Functional Requirements
| Requirement | Target |
|---|---|
| Inline completion latency (TTFT) | P50 < 200ms, P99 < 500ms |
| Agent task completion | P50 < 30s, P99 < 120s |
| Availability | 99.9% (8.7 hours downtime/year) |
| Completion acceptance rate | > 25% of shown suggestions accepted |
| Completion persistence rate | > 80% of accepted kept after 30 seconds |
| Post-processing rejection rate | < 5% of model outputs rejected for invalid syntax |
| Cost per inline completion | < $0.002 |
| Cost per agent task | < $0.10 |
| Zero-retention mode | Enterprise: code never stored or used for training |
| Multi-language support | 30+ programming languages via tree-sitter grammars |
3. System Architecture
🔒 Premium section
4. Design Principles
🔒 Premium section
5. Technology Selection
🔒 Premium section
6. Capacity Planning
🔒 Premium section
7. Platform Data Model
🔒 Premium section
8. End-to-End Request Flow
🔒 Premium section
9. IDE Plugin Architecture
🔒 Premium section
10. Local Context Engine
🔒 Premium section
11. Context Assembly and Prompt Engineering
🔒 Premium section
12. Model Gateway and Routing
🔒 Premium section
13. Inference System
🔒 Premium section
14. Post-Processing Pipeline
🔒 Premium section
15. Streaming and UX
🔒 Premium section
16. The Agent Loop
🔒 Premium section
17. Execution Sandbox
🔒 Premium section
18. Codebase RAG
🔒 Premium section
19. AI Code Review
🔒 Premium section
20. Phase 1: Understanding the Spec
🔒 Premium section
21. Phase 2: Architecture Generation
🔒 Premium section
22. Phase 3: Scaffolding
🔒 Premium section
23. Phase 4: The Build Loop
🔒 Premium section
24. Live Preview and Error Recovery
🔒 Premium section
25. Long-Running Execution: Checkpointing and Recovery
🔒 Premium section
26. Long-Running Memory
🔒 Premium section
Tech Stack
🔒 Premium section
Architecture Decisions
🔒 Premium section
Conventions
🔒 Premium section
27. Deployment
🔒 Premium section
28. Failure Strategy and Recovery
🔒 Premium section
29. Multi-Agent Orchestration
🔒 Premium section
30. Task Queue
🔒 Premium section
31. Control Plane
🔒 Premium section
32. Caching Architecture
🔒 Premium section
33. Feedback Loop and Model Improvement
🔒 Premium section
34. Safety and Privacy
🔒 Premium section
35. Observability
🔒 Premium section
36. Cost Engineering
🔒 Premium section
37. Multi-Tenant Architecture
🔒 Premium section
38. API vs Self-Hosted
🔒 Premium section
39. Common Pitfalls
🔒 Premium section
40. The Maturity Model: What to Build First
🔒 Premium section
41. Where This Breaks in Real Life
🔒 Premium section
42. End-to-End Walkthrough: "Add Stripe Billing to My SaaS App"
🔒 Premium section
Related Resources
🔒 Premium section
Conclusion
🔒 Premium section