System Design: ML Model Serving Platform
From one prediction to a fleet of 100 models
🔒 Premium section
1. Problem Statement
Training is an offline throughput job. Serving is an online latency and availability problem.
A training run may take hours and retry a failed step. Our ranking request has 50ms. It cannot download a model, wait for a new GPU, or keep retrying until something works.
Five tensions shape the design:
| Concern | Design consequence |
|---|---|
| Small requests underuse GPUs | Benchmark dynamic batching inside the p99 budget |
| PyTorch, TensorFlow, ONNX, TensorRT, and XGBoost differ | Standardize platform contracts while allowing multiple runtimes |
| Models change frequently | Validate, warm, canary, publish, and roll back immutable versions automatically |
| Models compete for heterogeneous hardware | Place by profiled memory, compute, bandwidth, topology, interference, and isolation rather than weight size alone |
| Offline and online features can diverge | Pin one feature contract and test training-serving parity |
One tempting design is a separate web service for every model. That leaves each team to solve batching, rollout, GPU placement, deadlines, metrics, and security again. It also produces many slightly different inference APIs.
We will instead standardize the serving contract and lifecycle. Models may use different runtimes underneath.
2. Functional Requirements
State the required behaviour before choosing Triton, KServe, MLflow, or Feast:
| ID | Requirement | Priority |
|---|---|---|
| FR-01 | Expose a client-facing prediction API for declared entity, media, text, or prepared-tensor inputs, plus an internal tensor runtime API | P0 |
| FR-02 | Support PyTorch, TensorFlow SavedModel, ONNX, TensorRT, and XGBoost through approved runtimes | P0 |
| FR-03 | Batch compatible requests with a configurable maximum batch size and latency window | P0 |
| FR-04 | Deploy several immutable model versions at once and route by exact version | P0 |
| FR-05 | Canary new versions on a bounded traffic share and roll back automatically on trusted guardrails | P0 |
| FR-06 | Register, approve, deploy, and audit models through a managed lifecycle | P0 |
| FR-07 | Retrieve online features when the model contract requires them and apply the versioned training-time transforms | P0 |
| FR-08 | Scale from queue depth, deadline pressure, latency, and hardware utilization | P0 |
| FR-09 | Run stable A/B assignments and collect exposure and outcome data | P1 |
| FR-10 | Send shadow traffic to a candidate without changing the live response | P1 |
| FR-11 | Place models on exclusive GPUs, time-sliced GPUs, MPS, MIG, or CPUs from workload policy | P1 |
| FR-12 | Execute approved multi-model DAGs with bounded fan-out and one deadline | P1 |
| FR-13 | Load models and run warmup probes before they receive traffic | P1 |
| FR-14 | Enforce per-team model, GPU-memory, concurrency, queue, and request-rate quotas | P0 |
| FR-15 | Detect schema, freshness, feature, output, and slice-level drift | P1 |
| FR-16 | Route only to healthy replicas that have the requested version loaded and warm | P0 |
| FR-17 | Reject or cancel work that cannot finish inside the caller's remaining deadline | P0 |
| FR-18 | Join sampled predictions with delayed outcomes and compute model quality by slice | P1 |
| FR-19 | Keep serving the last known-good route when the registry or deployment controller is unavailable | P0 |
Ensemble inference is useful, but it is not special at the platform boundary. An approved DAG still needs bounded fan-out, versioned dependencies, one deadline, and observable failure policy.
3. Non-Functional Requirements
These are the interview constraints. They determine the queue policy, replica count, warm capacity, and failure reserve.
| ID | Requirement | Target |
|---|---|---|
| NFR-01 | Model execution latency | Under 10ms p50 and 25ms p99 for the ranking workload |
| NFR-02 | End-to-end latency | Under 20ms p50 and 50ms p99, including feature lookup and post-processing |
| NFR-03 | Throughput | 50,000 sustained predictions/second, with reserved capacity for one-zone loss |
| NFR-04 | Accelerator utilization | 60 to 75% planned utilization for Tier-1 pools while p99 and failover reserve hold |
| NFR-05 | Availability | 99.95% for each Tier-1 model endpoint, not only the fleet average |
| NFR-06 | Model load time | Under 30 seconds up to 2GB and under 120 seconds up to 8GB |
| NFR-07 | Zero-downtime deployment | A pre-staged warm version begins serving within 60 seconds of approval |
| NFR-08 | Canary rollback | Under 30 seconds from trusted anomaly detection to stable-only routing |
| NFR-09 | Horizontal scalability | Near-linear throughput growth until a shared router, feature store, network, or control-plane limit appears |
| NFR-10 | Model count | More than 100 models concurrently loaded and serving |
| NFR-11 | Feature freshness | Each feature contract declares and monitors maximum age; request-time features use current context |
| NFR-12 | Scaling reaction | Make a scale-up decision within 30 seconds; node provisioning and model warmup may take longer |
| NFR-13 | Queue latency | Under 5ms p99 for Tier-1 models during planned load |
| NFR-14 | Failure-domain resilience | Lose one GPU node or one availability zone without dropping below reserved Tier-1 capacity |
| NFR-15 | Regional recovery | Control-plane RPO under 5 minutes and serving RTO under 30 minutes |
| NFR-16 | Overload behaviour | Keep queues bounded, preserve Tier-1 reserve, and reject before useless execution |
The numbers are calibrated estimates, not promises for every model or GPU. Actual throughput depends on the graph, shapes, batch size, precision, hardware, runtime version, and traffic mix. Benchmark every important model release.
Under overload, queues stay bounded. Tier-1 traffic keeps its reserve. Work that cannot finish before its deadline is rejected before execution instead of wasting a GPU and timing out later.
4. Design Principles and System Boundaries
🔒 Premium section
5. High-Level Approach & Technology Selection
🔒 Premium section
6. High-Level Architecture
🔒 Premium section
7. Back-of-the-Envelope Estimation
🔒 Premium section
8. Data and Release Contracts
🔒 Premium section
9. API Contracts
🔒 Premium section
10. Deep Dives
🔒 Premium section
11. Identify Bottlenecks
🔒 Premium section
12. Failure Scenarios
🔒 Premium section
13. Deployment Strategy
🔒 Premium section
14. Observability
🔒 Premium section
15. Security
🔒 Premium section
16. Reference Design and Feasibility
🔒 Premium section
Related AI Engineering Chapters
🔒 Premium section
Further Reading
🔒 Premium section
Related System Designs
🔒 Premium section
