CrackingWalnuts
The system design round changed. RAG pipelines, model serving, evaluation, agents, and cost are now fair game. These are the AI system design questions, each worked end to end the way the interviewer scores them.
Design a multi-tenant AI agent platform that monitors restaurant operations across thousands of restaurants. The system ingests real-time data from POS systems, delivery platforms (DoorDash, Uber Eats, Grubhub), payment processors, and inventory systems. It detects anomalies (revenue leaks, unusual refund patterns, commission errors), investigates them using LLM-powered agents with tool access, and triggers automated actions like dispute filing and reorder requests. Handle 50M+ events/day with sub-60-second detection latency.
Design a complete AI Software Engineer system that operates at three levels: - Inline autocomplete in 300ms (serving 100M completions/day). - A codebase-aware agent that refactors across 12 files in 45 seconds. - A fully autonomous builder that takes a natural language spec, asks clarifying questions, designs the architecture, scaffolds the project, builds each module with error recovery, responds to feedback over hours, and deploys to production. Cover the local context engine (tree-sitter AST, dependency graphs, LSP, git), multi-model routing, agent tool system with sandbox execution, long-running checkpointing and crash recovery, persistent project memory, multi-agent orchestration, and the economics of serving 1M developers at $20/month.
Design ChatGPT for 1 billion weekly users, about 3.6 billion messages a day and about 103,000 a second at peak. A large language model (LLM) writes answers in tokens, chunks of text about three quarters of a word long. Its learned numbers are called weights. Models run in pools, groups of running copies (replicas) of one model, on NVIDIA B200 GPUs (the chips that do the maths). The product also remembers users, reads files, searches the web and runs code. Some messages go to a model that thinks for about a minute first. Every turn sends the whole chat again, and a prefix cache keeps the work on the prompt's start. Answers stream and must survive a dropped phone.
Design one platform that serves a company's ML models. Its main user is a home feed that asks for 20,000 fresh feeds a second at peak, each within 120 ms. A fresh feed is a new first page, not page 2. Each feed runs four models in a row: the user tower, an ANN search, a pre-ranker and a ranker. Together they cut 100 million items to 25. A fraud model and an image model share the platform. Model teams ship new versions every week. And when a part fails, users shouldn't see an empty feed. A slightly worse one is fine.
Design an enterprise RAG (Retrieval-Augmented Generation) platform that serves 10M queries per day across 500+ engineering teams. The platform ingests 2M+ internal documents from Confluence, GitHub, Slack, Google Docs, and S3, lets engineers ask natural language questions, and returns accurate, cited answers grounded in internal knowledge. Handle hybrid retrieval, multi-strategy chunking, model routing for cost optimization, hallucination mitigation with citation enforcement, and continuous evaluation loops.
Design a vector database like Pinecone or Milvus for storing and searching high-dimensional embeddings. Support similarity search (cosine, Euclidean, dot product) over 1 billion vectors at 768 dimensions, metadata filtering, hybrid search (vector + keyword), and real-time inserts. Handle 10K queries/sec with sub-10ms latency using HNSW indexing and product quantization for memory efficiency. Consider index building, quantization trade-offs, and sharding strategies.
Design the serving platform behind a hosted chat model API. It serves a 70-billion-parameter model to many tenants, at 10,000 requests a second on average and 25,000 at peak. Prompts are about 1,500 tokens and answers about 247. The first token must appear within a second. GPUs are almost all of the cost, so every GPU-hour must do as much useful work as possible.
Design the system that pre-trains a 70-billion-parameter language model from random weights. It reads 15 trillion tokens on 1,024 NVIDIA B200 GPUs, with 8-bit numbers for most of the maths. The run lasts about two months and costs over ten million dollars. So every GPU must stay busy and the loss curve healthy. When hardware fails, the run should lose minutes of work, not days.
Design the platform that decides whether a new release of a chat model replaces the current one. A new release arrives every week, and smaller prompt changes every day. The decision must come within 2 hours. It must be honest about noise and hard to fool with a biased judge or a leaked test. And people must be able to stop a release that every number passes.
Design the safety pipeline around a chat and agent platform that serves 25,000 requests a second at peak. It checks every request, every streamed answer and every tool action against a written policy. It must add little delay and refuse as few harmless requests as possible. And it must keep tools and data safe even when the model is tricked.
Design the system that ships a new model version to production safely. Covers shadow deploys, percentage rollouts, automatic rollback, quality monitoring, and how to detect a bad model before millions of users notice.
One payment for 12 months. No auto-renewal.
One yearly plan. No auto-renew.