CrackingWalnuts
The system design round changed. RAG pipelines, model serving, evaluation, agents, and cost are now fair game. These are the AI system design questions, each worked end to end the way the interviewer scores them.
Design a multi-tenant AI agent platform that monitors restaurant operations across thousands of restaurants. The system ingests real-time data from POS systems, delivery platforms (DoorDash, Uber Eats, Grubhub), payment processors, and inventory systems. It detects anomalies (revenue leaks, unusual refund patterns, commission errors), investigates them using LLM-powered agents with tool access, and triggers automated actions like dispute filing and reorder requests. Handle 50M+ events/day with sub-60-second detection latency.
Design a complete AI Software Engineer system that operates at three levels: - Inline autocomplete in 300ms (serving 100M completions/day). - A codebase-aware agent that refactors across 12 files in 45 seconds. - A fully autonomous builder that takes a natural language spec, asks clarifying questions, designs the architecture, scaffolds the project, builds each module with error recovery, responds to feedback over hours, and deploys to production. Cover the local context engine (tree-sitter AST, dependency graphs, LSP, git), multi-model routing, agent tool system with sandbox execution, long-running checkpointing and crash recovery, persistent project memory, multi-agent orchestration, and the economics of serving 1M developers at $20/month.
Design ChatGPT, a conversational AI for 100M daily users and 1B messages/day, with answers streamed token by token. Cover the inference pipeline (prefill, decode, KV-cache, continuous batching, speculative decoding), three-tier context memory, RAG, a bounded tool-calling agent loop, multi-model routing, and RLHF/DPO alignment. Consider three-layer safety, resumable streaming, GPU fleet sizing, and cost per conversation.
Design an enterprise RAG (Retrieval-Augmented Generation) platform that serves 10M queries per day across 500+ engineering teams. The platform ingests 2M+ internal documents from Confluence, GitHub, Slack, Google Docs, and S3, lets engineers ask natural language questions, and returns accurate, cited answers grounded in internal knowledge. Handle hybrid retrieval, multi-strategy chunking, model routing for cost optimization, hallucination mitigation with citation enforcement, and continuous evaluation loops.
Design a vector database like Pinecone or Milvus for storing and searching high-dimensional embeddings. Support similarity search (cosine, Euclidean, dot product) over 1 billion vectors at 768 dimensions, metadata filtering, hybrid search (vector + keyword), and real-time inserts. Handle 10K queries/sec with sub-10ms latency using HNSW indexing and product quantization for memory efficiency. Consider index building, quantization trade-offs, and sharding strategies.
Design the serving stack behind a hosted language model API: 10,000 requests per second, a 70-billion-parameter model, streaming tokens to users in under 2 seconds. Covers prefill vs decode, the KV cache, continuous batching, GPU economics, and streaming recovery.
Design the training stack to train a 70-billion-parameter language model on a thousand GPUs over a month. Covers data parallel, tensor parallel, pipeline parallel, FSDP, checkpointing, failure recovery, and how to not burn millions of dollars on a wasted run.
Design the platform that decides whether a new model is safe to ship. Covers golden sets, LLM-as-judge, online vs offline evals, regression gates, and how to catch model quality drops before customers do.
Design the system that decides which inputs the model refuses and which outputs the user is allowed to see. Covers input classifiers, output classifiers, jailbreak defense, prompt injection, age-gating, region rules, and the constant tension between safety and usefulness.
Design the system that ships a new model version to production safely. Covers shadow deploys, percentage rollouts, automatic rollback, quality monitoring, and how to detect a bad model before millions of users notice.
A limited number of memberships. Save $34.
One yearly plan. No auto-renew.