How an LLM Actually Runs in Kubernetes

The complete stack, from Kubernetes to the next token

Deploying an LLM in Kubernetes can sound like three steps: put the model in a container, run that container in Kubernetes, and send prompts to it. That mental model hides almost everything that matters in production.

A real deployment has several different systems doing different jobs:

Kubernetes does not know how an LLM generates text. It does not manage KV cache. It does not split transformer layers across GPUs. It does not know that one model server already has 8,000 tokens of a prompt cached while another does not.

Kubernetes gives us machines, networking, containers, health checks, secrets, and GPUs. The inference runtime does the model-specific work.

This article builds the full picture from the bottom up using one concrete example:

We want to serve a 70B-class instruction model to many SaaS tenants. The model does not fit on one GPU, so one model replica uses four GPUs. We use Kubernetes, vLLM, and later llm-d when we have a fleet of replicas.

The exact model or GPU is not important. The architecture is.

Before opening any layer, keep the complete request path in view:

Client
  -> Gateway: authenticate, validate, limit and meter
  -> HTTPRoute / InferencePool: identify the model fleet
  -> Endpoint Picker: select a suitable replica
  -> vLLM: tokenize, schedule, prefill and decode
  -> CUDA / NCCL: execute kernels and coordinate GPU ranks
  -> Gateway: stream tokens and propagate cancellation

The first half of this article explains what happens inside one replica. The second half explains how Kubernetes operates a fleet of those replicas. Advanced features such as multi-node execution and separate prefill/decode workers come last because most deployments should not begin there.

Design target for the running example

We need a workload and latency target before choosing replicas. The following numbers are illustrative, not promises for a particular model or GPU:

Capacity is measured in tokens, not just requests. At 20 requests per second with the typical lengths above, the fleet receives roughly:

input demand  = 20 x 1,000 = 20,000 input tokens/second
output demand = 20 x   250 =  5,000 output tokens/second

Benchmark one complete four-GPU replica with the real prompt distribution and concurrency. Suppose separate phase measurements show up to 30,000 input tokens/second for prefill and 2,000 output tokens/second for decode:

replicas required by prefill = ceil(20,000 / 30,000) = 1
replicas required by decode  = ceil( 5,000 /  2,000) = 3

The larger result, three replicas, is only a lower bound. A combined worker shares GPU time between prefill and decode, so independent phase ceilings cannot prove that three replicas can sustain the mixed workload. Run a second benchmark with the real prompt/output distribution, concurrency and latency targets. Then add headroom for traffic variation, rollout and failure recovery. If that mixed test confirms three replicas, a four-replica plan provides one spare failure unit in this example. Re-run the benchmark whenever the model, quantization, context distribution, runtime version or GPU type changes.

1. From a GPU to a model server

Start with a Kubernetes worker node containing four GPUs. A GPU by itself cannot accept an HTTP request such as:

POST /v1/chat/completions

We need software that loads the model into GPU memory and turns it into a server.

That is the role of an inference runtime such as vLLM.

For a model that needs four GPUs, the pod can request all four:

resources:
  limits:
    nvidia.com/gpu: 4

Kubernetes schedules that pod onto a node with four available GPUs.

Inside the pod, vLLM sees the GPUs and starts the model runtime.

If we configure:

--tensor-parallel-size 4

vLLM treats those four GPUs as one tensor-parallel model replica.

The boundary is simple: Kubernetes allocates the GPUs to the pod; vLLM turns those GPUs into an inference server. Kubernetes is not implementing inference.

2. How a model generates text

Before discussing caches or multiple GPUs, it helps to understand one request on one model.

Suppose the user asks:

Explain Kafka partitioning in simple terms.

For a chat API, the input is not tokenized directly from the JSON messages array. The model's chat template first renders the roles and message text into one prompt string. The tokenizer then converts that rendered prompt into token IDs. A template change can therefore change the model input even when the API request is identical, which is why the tokenizer and chat template must be versioned with the model.

The model does not work directly with words; it works with those token IDs and the vectors they map to.

Prefill

The complete input prompt is first processed through the transformer.

If the input contains 8,000 tokens, prefill processes those 8,000 input tokens.

At the end of prefill, the model produces probabilities for the next token.

The probabilities might favour Kafka over A or Partition. A configured decoding strategy then chooses one token. Sampling is common, while greedy decoding simply takes the highest-probability token.

Suppose the first token is:

Kafka

Now the model needs the second token.

It runs another forward step and perhaps produces:

partitions

Then:

allow

Then:

messages

So generation is fundamentally a loop:

Two latency measurements are useful here:

• TTFT — Time To First Token: strongly affected by prompt length, queueing, and prefill.
• Inter-token latency: how quickly tokens arrive after generation has started.

This distinction becomes important later because prefill and decode use the GPU differently.

3. The hidden cache: KV cache

There is an obvious problem with the decode loop.

Imagine the model has already processed:

Kafka partitions allow applications to

and now wants to generate the next word.

A naive implementation could recompute the entire sequence on every step. That would waste enormous amounts of compute.

Transformers therefore keep the important attention state from earlier tokens. That state is called the KV cache.

KV means Key and Value.

At a simplified level, attention calculates values called Query, Key, and Value:

Q = XWq
K = XWk
V = XWv

For old tokens, their K and V values do not need to be calculated again during every decode step.

So after prefill we keep them:

When a new token is generated, we calculate the K/V state for that token and append it.

This is why the KV cache is central to inference performance.

It is also why context length consumes GPU memory. More active tokens mean more KV-cache memory.

The KV cache is not Redis

The KV cache normally lives in GPU memory managed by the inference runtime.

It is not:

Redis key → prompt

and it is not your conversation database.

If the vLLM pod disappears, that cache can disappear too.

Your durable conversation history should still live in your application/database.

Think of KV cache as:

Temporary accelerator state that lets the model continue generation efficiently.

4. Prompt caching is reuse of KV cache across requests

The phrase prompt caching can be confusing because it sounds like we are caching the text of the prompt.

For vLLM, the useful concept is Automatic Prefix Caching.

Consider these two requests.

Request 1:

SYSTEM
You are our architecture tutor.
Here are 6,000 tokens of rules and examples...

USER
Explain Kafka partitioning.

Request 2:

SYSTEM
You are our architecture tutor.
Here are the same 6,000 tokens of rules and examples...

USER
Explain Cassandra consistency.

Without prefix caching, both requests repeat the work for the 6,000 shared tokens. With prefix caching, the first request leaves reusable KV blocks behind. When Request 2 arrives:

The model output is not cached. The expensive attention state for the common prefix is reused.

That distinction matters:

vLLM implements prefix caching by hashing token blocks and tracking the KV blocks associated with those hashes.

Prompt layout affects cache hits

Suppose every request begins with a changing timestamp:

Current time: 10:31:02
Tenant: acme

[6,000 static tokens]

The prefix changes immediately, which reduces reuse.

A better layout is:

A useful rule is:

Put stable content early and highly dynamic content later when the semantics allow it.

Do not distort prompt semantics just to obtain cache hits, but avoid unnecessary variation at the beginning.

5. Many users, one GPU model

Now suppose three users make requests at roughly the same time. Processing them strictly one after another often leaves throughput unused, especially during decode.

vLLM loads the model weights into GPU memory when the replica starts. The CPU does not reload the complete model for every request. During batch-one decode, one sequence advances by one token while the GPU reads the weights needed by each layer. That is often too little work to use the GPU's wide parallel hardware efficiently.

Batching lets one model step advance several sequences. The GPU performs larger matrix operations and amortizes weight-memory access and kernel-launch overhead across more token calculations. Prefill is different because one long prompt already provides parallel work across many input tokens. Chunked prefill lets the scheduler share GPU time between those long prompts and latency-sensitive decode work.

LLM serving engines use continuous batching.

Conceptually:

Requests can enter and leave the batch as generation progresses.

This is very different from a classic fixed batch used in offline machine learning.

This scheduler lives inside the vLLM engine. On every engine step it looks at waiting requests, running requests, the token budget and available KV-cache blocks. It then decides things such as:

• Which new prompts can start prefill?
• Which active requests need another decode token?
• How much KV-cache space is available?
• How many tokens can run in this iteration?
• Should a long prefill be broken into chunks?

You normally do not write this scheduling engine yourself when using vLLM.

The decision applies to the whole model replica. If the replica uses four tensor-parallel GPUs, the scheduler does not assign request A to GPU 0 and request B to GPU 1. It chooses a batch of token work, and all four GPUs cooperate to execute that batch.

6. Multi-tenancy is more than putting tenantId in a header

Suppose the model is shared by three companies. They need several different kinds of isolation.

Identity and authorization isolation

The public gateway should authenticate the user and derive the tenant from trusted identity claims.

Do not trust a public request such as:

{
  "tenantId": "acme"
}

unless the caller is authorized to act for that tenant.

The gateway should forward an internal, trusted tenant identity.

Quota isolation

If Tenant C sends 50,000 requests, it should not be able to consume all capacity.

Typical controls include:

requests per minute
input tokens per minute
output tokens per minute
max concurrent requests
max context length
max generated tokens

These controls usually live in your gateway/admission layer, often backed by Redis or another fast shared store.

Fairness isolation

Rate limits alone are not enough.

Imagine all tenants are below their formal quota, but Tenant C has 1,000 requests waiting. A production platform may need weighted fairness or priority so one workload does not dominate queueing latency.

This is one area where inference-aware routing and scheduling become important at fleet scale.

Prefix-cache isolation

This one is easy to miss.

If two unrelated tenants share the same vLLM server, should one tenant's request be allowed to reuse prefix-cache blocks created by another tenant?

For a shared trust domain that might be acceptable. For stricter isolation, it may not be.

vLLM supports a per-request cache_salt for prefix-cache isolation. Requests with different salts do not share the same prefix-cache namespace.

Conceptually:

The salt should be generated and controlled by the server, not supplied arbitrarily by an untrusted client.

Important: cache_salt is a prefix-cache security control. It is not complete tenant isolation by itself.

Network isolation

Clients should normally not call vLLM directly.

Expose the public gateway, keep vLLM behind an internal Service, and use a Kubernetes NetworkPolicy so only approved gateway or router namespaces can reach the model-server pods. This works only when the cluster's network plugin enforces NetworkPolicy.

Hard isolation

Some customers may require a dedicated deployment and GPU pool. This costs more, but it gives a separate process, cache and resource failure domain.

A practical SaaS platform often has both:

7. Serving a giant model: one model across four GPUs

Now we reach the reason our example needs four GPUs.

A 70B-class model in BF16 can require roughly 140 GB just for model parameters before considering KV cache and runtime overhead. It therefore cannot fit on a single 80 GB GPU.

We can shard the model across four GPUs using tensor parallelism.

This does not mean four independent copies of the model.

It is one logical model replica whose work is distributed across four GPUs.

What is actually sharded?

Transformer layers contain large matrix multiplications.

A large weight matrix can be split across the four ranks so each GPU calculates its local part.

But a later operation may need a combined result.

That means the GPUs must communicate.

Common collective operations include:

AllReduce
AllGather
ReduceScatter

NVIDIA's NCCL library performs these GPU collective operations efficiently over technologies such as NVLink, NVSwitch, PCIe, and high-speed networking.

Where does AllReduce or AllGather actually happen?

If we use vLLM, we normally do not write this communication code ourselves.

Assume Kubernetes gives one vLLM pod four NVIDIA GPUs and we start the model with tensor parallel size 4:

vLLM understands how the model is sharded. When a transformer operation produces partial results that must be combined, its distributed runtime invokes a collective such as:

AllReduce
AllGather
ReduceScatter

A tensor-parallel layer hides this coordination behind a clear ownership boundary:

So vLLM decides where communication is required. On NVIDIA systems, NCCL is the standard library that performs the collective communication. vLLM can also use optimized communication paths internally, but the application developer still does not write GPU socket or NVLink code.

This communication is not something that happens once after the model finishes. It happens inside transformer execution. A common tensor-parallel pattern needs synchronization after parts of attention and the MLP. During decode, those collectives are repeated as every new token passes through the model's layers.

That is why four GPUs do not automatically mean four times the speed: extra compute and memory come with communication cost. Fast links such as NVLink or NVSwitch can matter greatly for tensor parallelism.

AllGather and AllReduce solve different problems:

Kubernetes is not part of this inner loop. Its job was mainly:

"Place this pod on a node and give it four GPUs."

If tensor parallelism crosses machines, NCCL can also communicate over high-speed network fabrics such as InfiniBand or RoCE. That makes network topology a first-class production concern.

If we built the runtime ourselves

Then Python/PyTorch code would explicitly create a distributed process group and invoke collectives.

A simplified example looks like:

import torch
import torch.distributed as dist


dist.init_process_group(backend="nccl")
rank = dist.get_rank()
torch.cuda.set_device(rank)

# each rank owns a local model shard
local_output = run_local_shard(input_tensor)

# combine partial results when the sharding strategy requires it
dist.all_reduce(local_output)

Usually the same Python program runs once per GPU rank.

You do not write socket code for NVLink. NCCL handles the transport.

For production serving, use vLLM unless building an inference runtime is itself your product or research goal.

8. What happens to KV cache across four GPUs?

Tensor parallelism also affects the KV cache.

Suppose attention heads are divided across GPUs.

Then each GPU can own the KV state associated with its part of the attention computation.

From the client perspective this is invisible.

The client still sends one request to one model-server endpoint:

POST /v1/chat/completions

It does not send one request to each GPU.

The model runtime coordinates the GPU ranks internally.

9. One request, end to end

Now put everything together.

Assume Tenant acme sends a request with a large common system prompt.

Step 1: Client calls the gateway

Client
  ↓
POST /api/chat
Authorization: Bearer ...

The gateway validates the token and obtains:

tenantId = acme
userId   = 8172

Step 2: Gateway applies tenant policy

The gateway checks things such as:

Acme concurrency limit: 20
Input token budget:      allowed
Model permission:        qwen-70b allowed
Requested max output:    1,000 tokens

If the tenant has exceeded its policy, reject before consuming GPU capacity.

Step 3: Gateway builds the model request

The application creates a stable prompt structure:

[system instructions]
[shared examples]
[relevant retrieved context]
[conversation]
[user question]

It also attaches a server-managed cache namespace, for example:

cache_salt = HMAC(platformSecret, tenantId)

Do not use the tenant name itself as a security secret.

Step 4: Request reaches vLLM

The gateway calls the internal vLLM service.

Gateway
   ↓
ClusterIP Service
   ↓
vLLM pod

vLLM tokenizes the rendered prompt.

Suppose there are 8,000 input tokens.

Step 5: Prefix-cache lookup

This is the first request with this prefix, so there is a miss:

Step 6: Prefill across four GPUs

The model is tensor-parallel across four GPUs.

The GPUs compute their local shards and communicate whenever the tensor-parallel layer requires it.

At the same time, KV-cache state is created for the input tokens.

Step 7: First output token

After prefill, the model produces logits for the first new token.

Sampling selects a token.

The server streams it to the client.

Step 8: Decode

For each subsequent token:

The model does not re-prefill the original 8,000 tokens each time.

Step 9: Request completes

Usage accounting can record:

input tokens
output tokens
model
latency
cache hit/miss statistics
tenant

Be careful with metric cardinality. Per-tenant usage often belongs in an accounting/event system rather than putting thousands of tenant IDs into every Prometheus metric.

10. The second request is where prefix caching becomes visible

A few seconds later Acme sends another question using the same large system prompt.

[7,000+ identical prefix tokens]
[new conversation suffix]
[new question]

The request reaches the same vLLM replica.

This time the server can reuse cached KV blocks for the matching prefix.

This can reduce prefill work and improve time to first token.

Now suppose Tenant Beta sends exactly the same textual prefix but has a different cache_salt.

Acme salt  ≠ Beta salt

The cached blocks are not shared across those cache namespaces.

That is one layer of tenant protection.

11. A practical code ownership map

This is the question teams often struggle with: What do we actually have to write?

For a production system based on vLLM, the answer is much less than people expect at the GPU layer and much more than they expect at the platform layer.

If your backend is Java, keeping the product/platform layer in Java is completely reasonable.

The inference engine itself does not need to be Java.

12. Example Java gateway responsibilities

The gateway should not be a giant ML framework. Its job is product policy and admission.

A simplified request flow could be:

public Flux<String> chat(AuthPrincipal principal, ChatRequest request) {
    String tenantId = principal.tenantId();

    tenantPolicy.assertModelAllowed(tenantId, request.model());
    tenantQuota.acquireConcurrencySlot(tenantId);

    String cacheSalt = cacheSaltService.forTenant(tenantId);

    VllmRequest internalRequest = VllmRequest.builder()
        .model(request.model())
        .messages(promptBuilder.build(principal, request))
        .maxTokens(Math.min(request.maxTokens(), 1000))
        .cacheSalt(cacheSalt)
        .stream(true)
        .build();

    return vllmClient.stream(internalRequest)
        .doFinally(signal -> tenantQuota.releaseConcurrencySlot(tenantId));
}

Production code needs stronger error handling than this example, but notice what is not present:

no CUDA
no KV-cache allocation
no attention implementation
no AllReduce code

Those are runtime responsibilities.

13. Kubernetes: the minimum production shape

For our four-GPU model replica:

The GPU nodes need the normal NVIDIA Kubernetes stack: working host drivers, NVIDIA container support, and the device plugin so Kubernetes can advertise nvidia.com/gpu resources.

There are two useful deployment stages:

One replica
Gateway -> ClusterIP Service -> one vLLM pod

Replica fleet
Gateway -> HTTPRoute -> InferencePool -> EPP decision -> selected vLLM pod

The Service example below is enough to learn how one internal replica is exposed. Once several replicas need inference-aware routing, a blind Service is no longer the decision-maker. A Gateway implementation that supports the Inference Extension sends request information to the EPP, and an HTTPRoute names the InferencePool as its backend.

A simplified vLLM Deployment looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: qwen-70b
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: qwen-70b
  template:
    metadata:
      labels:
        app: qwen-70b
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:<tested-version>
          args:
            - --model
            - <your-70b-model>
            - --tensor-parallel-size
            - "4"
            - --enable-prefix-caching
            - --gpu-memory-utilization
            - "0.90"
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 4
          volumeMounts:
            - name: dshm
              mountPath: /dev/shm
      volumes:
        - name: dshm
          emptyDir:
            medium: Memory

Do not copy this blindly into production. Real manifests should also include:

• a long enough startup probe for model loading;
• readiness/liveness behavior appropriate for the runtime;
• node selectors/affinity for the correct GPU type;
• taints/tolerations for GPU nodes;
• model-cache/storage strategy;
• resource requests for CPU and RAM;
• secrets for gated model access if required;
• security context;
• observability;
• controlled rollout behavior.

Expose vLLM internally:

apiVersion: v1
kind: Service
metadata:
  name: qwen-70b
spec:
  type: ClusterIP
  selector:
    app: qwen-70b
  ports:
    - port: 8000
      targetPort: 8000

The public internet should normally see your gateway, not this service.

How the Gateway API resources connect

An InferencePool does not receive public traffic by itself. The complete logical relationship is:

Gateway
  <- attached HTTPRoute
       -> backendRef: InferencePool/qwen-pool
            -> endpointPickerRef: Service/qwen-epp
            -> selector: app=qwen-server
                 -> matching vLLM pods

The exact proxy deployment depends on the chosen Gateway implementation, but each object has one job:

This distinction prevents a common mistake: creating an InferencePool and expecting it to behave like an ingress endpoint without a Gateway and route.

Streaming, cancellation and graceful shutdown

An LLM request may keep its HTTP connection open for minutes. Every proxy on the path therefore needs streaming-friendly timeouts and must avoid buffering the entire response.

When the client disconnects:

disconnect
  -> gateway cancels the upstream request
  -> router/sidecar propagates cancellation
  -> vLLM stops scheduling more decode work
  -> quota slot and KV blocks are released

Without propagation, the GPU keeps generating tokens nobody will receive. Put an absolute request deadline and an output-token limit on every request as a second line of defence.

Pod termination needs the reverse sequence:

1. Mark the replica unready so it receives no new requests.
2. Let active streams drain for a bounded period.
3. Cancel requests that exceed that period.
4. Terminate the model process.

Set terminationGracePeriodSeconds to match the drain budget and make the readiness probe reflect whether the server can accept new work. A liveness probe should detect a stuck process, not restart a healthy server merely because model loading or one request is slow.

Current vLLM supports --shutdown-timeout N to drain active requests for a bounded number of seconds after SIGTERM; without it, shutdown aborts active requests immediately. Keep the Kubernetes grace period longer than the vLLM shutdown timeout so the process can finish its own cleanup before SIGKILL.

14. Why ordinary load balancing starts to break

Everything is fairly straightforward with one model replica.

Now traffic increases, so we create a second replica.

Each replica still needs four GPUs. A normal Kubernetes Service might spread connections across both endpoints.

For a normal stateless HTTP service, that is often fine.

For LLM inference, the replicas are not equivalent at every moment.

Suppose Replica A already has Acme's 7,000-token prefix cached while Replica B does not. Sending Acme's next request to A can avoid substantial prefill work.

A blind round-robin decision may send it to B and throw away that locality.

There is another case:

Always choosing A because of cache locality can also be wrong.

So the routing decision is no longer simply:

Which pod is next?

It becomes something closer to:

Which pod has useful prefix cache?
Which pod has available KV-cache capacity?
How many active requests does it have?
How much token work is already queued?
What priority does this workload have?
Would cache affinity save more than queueing would cost?

This is why ordinary load balancing is not enough for a serious LLM fleet.

15. llm-d: running the whole fleet

This is the problem llm-d is designed to address. vLLM runs a model server; llm-d adds fleet-level routing and scheduling around many model servers.

The word scheduler now appears at two different levels. The vLLM scheduler from section 5 chooses work inside one model server. The llm-d Endpoint Picker chooses which model-server endpoint receives a request. Kubernetes has another scheduler below both of them, but that scheduler places pods on nodes rather than routing inference requests.

Kubernetes scheduler  -> Which node should run this pod?
llm-d EPP scheduler   -> Which vLLM endpoint should receive this request?
vLLM scheduler        -> Which requests and tokens run in the next model step?

For one request, the decisions happen in that order. Kubernetes has already placed the model pods. The EPP might choose vLLM B because its queue is short, even if vLLM A has a better prefix-cache hit. Once the request reaches B, B's internal scheduler can run part of its prefill alongside decode work from requests already in progress. llm-d does not build that GPU batch, and vLLM does not choose between the fleet's independent replicas.

A simplified architecture is:

Where is llm-d installed?

llm-d is not installed inside vLLM as a Python library. It is deployed as Kubernetes components.

In the simplest standalone setup, the llm-d Router includes a proxy such as Envoy plus the Endpoint Picker (EPP). The model servers are separate vLLM pods. The current llm-d quickstart installs the router with Helm and deploys model-server manifests separately.

The cluster also needs the Kubernetes Gateway API Inference Extension resources, including InferencePool.

What is an InferencePool?

An InferencePool is roughly an LLM-aware grouping of model-server pods. It uses Kubernetes labels to discover the pods that belong to the pool and points to the EPP that should choose among them.

A simplified resource looks like:

apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
  name: qwen-pool
spec:
  selector:
    matchLabels:
      app: qwen-server
  targetPorts:
    - number: 8000
  endpointPickerRef:
    name: qwen-epp
    port:
      number: 9002

This does not itself decide that pod A is better than pod B. It defines which endpoints belong to the pool. The EPP performs the intelligent selection.

Where are the routing rules stored?

The central EPP policy is an EndpointPickerConfig written as YAML. It looks like a Kubernetes object, but in current llm-d it is configuration, not a CRD. It is normally stored in a ConfigMap or file and mounted into the EPP container.

The EPP reads this configuration on startup, so changing it normally means updating the ConfigMap and restarting/rolling the EPP.

The scheduler's mental model is:

For example, filters can restrict requests to pods with a particular role, while scorers can consider prefix-cache affinity, KV-cache utilization or load. A picker then chooses an endpoint from the scored candidates.

So these are closer to routing policies than hard-coded application rules.

What does llm-d know about each vLLM server?

The EPP can make decisions using inference-specific information such as:

• prefix-cache locality;
• KV-cache utilization;
• request queue depth;
• active requests and token load;
• workload priority and fairness policy.

Routing and endpoint scoring are core EPP behavior. Holding requests at the gateway and enforcing priority or fairness between traffic flows require EPP Flow Control to be enabled and configured. They are not automatic in every llm-d installation.

These signals are also best-effort observations, not locks. Metrics can be delayed, queues can change, and a cached prefix can be evicted after an endpoint is scored. Cache-aware routing improves the odds of reuse; it cannot guarantee a cache hit.

So the current routing idea is better described as:

Prefer useful cache locality, but do not stay sticky to a server when its load makes another server a better choice.

llm-d therefore becomes the place to manage fleet-level concerns such as routing, cache locality, fairness, priorities, multiple replicas, and advanced prefill/decode layouts.

vLLM still executes the transformer. llm-d decides where a request should execute.

Well-lit paths

llm-d publishes well-lit paths: measured starting points packaged as guides, recipes and deployment configuration. Platform teams often call the same idea a golden path or paved road. The settings still need validation against your model, hardware and traffic.

The current llm-d release has several foundation and workload paths. Three are especially relevant to the architecture in this article:

Intelligent inference scheduling is prefix-cache-aware routing across vLLM replicas. That is what this section has been describing. Published llm-d benchmarks show that routing and scheduling can improve throughput and time to first token for specific models, hardware and traffic distributions. Treat those results as evidence for testing the pattern, not as a portable multiplier.

Prefill/decode disaggregation splits the two phases onto separate pods, which is section 18. Published results show that this can improve throughput and time to first token on long-prompt workloads, but the result depends strongly on the model, P/D ratio, hardware and network.

Wide expert parallelism spreads the experts of a large mixture-of-experts model such as DeepSeek-R1 across many GPUs, using an all-to-all communication backend and data parallelism on top.

The third one does not apply to our example. Wide expert parallelism only pays on a mixture-of-experts model, and the 70B in this article is dense, so its experts do not exist to spread. That is worth knowing before you go looking for the setting.

These are three relevant paths, not the complete llm-d catalogue. Current releases also document optimized baselines, predicted-latency routing, tiered prefix caching, flow control, autoscaling and workload-specific paths. Treat published numbers as evidence that a pattern can work, not as a capacity promise for a different deployment.

16. The architecture after scaling out

A realistic shared production setup can look like this:

Notice the two different dimensions of scaling:

If each model replica needs four GPUs, two replicas need eight GPUs. This scale-out axis is easy to confuse with tensor parallelism, which makes several GPUs cooperate inside one replica.

17. What if the model is larger than one entire machine?

Suppose a node has eight GPUs but the model still cannot fit on that node.

Now we need a multi-node strategy.

A common approach is:

Tensor parallelism inside a node
        +
Pipeline parallelism across nodes

Very simplified:

Here, TP = 8 and PP = 2. Each pipeline stage is a logical model partition. A stage does not have to equal one pod or one machine, but mapping one tensor-parallel group to each node is common because the GPUs inside a node usually have the fastest links.

Suppose the vLLM scheduler selects one decode token for request A:

1. The eight GPUs on node A jointly execute layers 0 to 39. Tensor-parallel collectives combine their partial results inside that stage.
2. Stage 0 produces an intermediate activation tensor. The distributed execution layer sends that tensor across the network to stage 1.
3. The eight GPUs on node B jointly execute layers 40 to 79. The final stage produces the logits used to choose the next token.
4. On the next decode step, the new token starts at stage 0 and follows the same path again.

The model weights do not move between nodes for every token. Each stage keeps the weights for its own layers. Its KV-cache entries also stay with those layers. What crosses the pipeline boundary during normal execution is the intermediate activation needed by the next stage.

The responsibilities are separate:

vLLM scheduler      -> decides what token work should run
vLLM executor       -> coordinates the distributed workers
TP/PP runtime       -> maps each model partition to its workers
NCCL / PyTorch      -> moves tensors between participating ranks
GPU workers         -> execute their local model partition

Ray can launch and manage the workers in a multi-node vLLM deployment, but it is not required for every deployment. Current vLLM also supports a multi-node multiprocessing backend. In either case, the launcher manages worker processes; the model runtime performs TP and PP communication.

At startup, each worker receives a rank, the total world size and a rendezvous address. Kubernetes networking, often through a headless Service, makes the pod addresses reachable. The workers join one distributed process group; then PyTorch and NCCL can send activations and run collectives between the ranks. Kubernetes connects and places the pods, but it does not decide which model layers a rank owns.

Networking matters much more because model execution now crosses machines.

You need fast interconnects and careful topology.

Why a plain Deployment is not enough for this

Section 13 quietly assumed one pod is one replica. Multi-node breaks that assumption, and it breaks it in every direction at once.

A Deployment treats each pod as an independent replica it can create, delete and reschedule on its own. For a model sharded across four nodes that is wrong four times over:

Three of four pods running is not 75 percent of a model. A parallel group with a missing rank cannot serve a single token, so a partially started group is worth exactly nothing.

Scaling to two replicas should mean two more complete groups, not two more loose pods.

Losing one worker makes the entire group useless, so restarting only the dead pod leaves the other three waiting for a rank that never rejoins.

And the pods have to land near each other on the fast interconnect. A scheduler that places them wherever there is room can put two ranks on opposite sides of the datacenter.

LeaderWorkerSet

LeaderWorkerSet is the Kubernetes API built for exactly this. It makes a group the unit of replication rather than a pod: one leader plus a set of workers, sometimes described as a super pod.

The properties that matter here are:

• Group lifecycle. LWS creates and manages the leader and workers as one replica. True all-or-nothing admission requires integration with Kueue or another gang scheduler; the default Kubernetes scheduler may still leave part of a group pending.
• Two templates. The leader and the workers can have different pod specs, because they usually do.
• The group is the unit of scaling and of rolling update. Groups upgrade one at a time, and the pods inside a group upgrade together.
• Optional topology placement. With exclusive-topology placement configured, a group can map to one rack or other topology domain so its ranks use the intended network fabric.
• restartPolicy: RecreateGroupOnPodRestart. One pod dies and the whole group restarts, which sounds heavy-handed until you remember that the surviving pods were useless anyway.

That last point is the one to carry forward. Earlier we said four GPUs are one failure unit. Across nodes, the whole group is the failure unit, and it is also the scaling unit and the rollout unit.

Do not confuse lifecycle grouping with scheduling admission. LWS creates the related pods and tracks them as one replica. If the cluster must reserve every GPU in the group before any rank starts, combine LWS with Kueue topology-aware scheduling or another gang scheduler. Without that integration, Kubernetes may create all pods while some remain pending for resources.

vLLM documents LeaderWorkerSet as its multi-node deployment path, NVIDIA Dynamo uses it, and llm-d's wide expert parallelism path from section 15 ships as a wide-EP-on-LWS guide. If you end up on the third well-lit path, this is the object underneath it.

Current LWS releases also provide DisaggregatedSet for coordinating deployments with separate roles such as prefill and decode. It is useful when those roles need coordinated placement, scaling and rollout. It is not required for the simpler two-Deployment llm-d example used below.

Do not start with multi-node inference unless the model requires it. A model that fits inside one multi-GPU node is much simpler operationally.

Pipeline parallelism should not be confused with the next design. In PP, different layer ranges form one distributed model replica and intermediate activations move between its stages. In prefill/decode disaggregation, separate model servers perform different phases of a request and transfer KV state.

18. Prefill/decode disaggregation: how llm-d actually routes one request to two workers

This is the part that can look magical until the pieces are separated.

Normally one vLLM server does both phases:

request -> one vLLM replica -> prefill -> keep KV locally -> decode -> stream tokens

The same replica processes the prompt, keeps the resulting KV cache in its own GPU memory and uses it for every decode step. This combined path avoids network KV transfer and is the simpler default. We consider separating the phases only after understanding why their resource profiles differ.

Why the two phases want different hardware

Both phases read the same weights. They differ in how much arithmetic they do per byte read, and that one ratio decides which part of the GPU runs out first.

Take the running example. A 70B model in BF16 is about 140 GB of weights, sharded four ways, so about 35 GB sits on each GPU. For an H100 SXM, using the advertised 989 TFLOP/s dense BF16 tensor-core peak and 3.35 TB/s memory-bandwidth peak gives an idealized balance point near 295 FLOPs per byte. This is a roofline model, not a latency prediction: real kernels do not sustain every advertised peak.

In this simplified model, one batch-one decode step does roughly two FLOPs per parameter for one token. That is about 1 FLOP per byte, far below the balance point. Reading a 35 GB shard at the peak bandwidth would take about 10.4 ms, while the ideal arithmetic time would be much smaller. Neither is an expected production latency: a real step also reads KV state, launches kernels and communicates across GPUs. The comparison only explains why batch-one decode tends to be memory-bandwidth-bound.

The same simplified calculation gives much higher arithmetic intensity for prefill because thousands of prompt tokens can be processed together. That tends to make a long prefill compute-bound.

These are workload-dependent tendencies, not permanent labels. Batch size, kernels, context length, quantization, KV traffic and inter-GPU communication move the balance. The asymmetry is still the reason separating the phases can help.

With prefill/decode disaggregation, we intentionally create different vLLM deployments for the two phases:

Both sets can belong to the same InferencePool. The role labels tell the EPP which pods are eligible for which phase.

Is llm-d simply using a fixed rule: "prefill left, decode right"?

No. The behavior is configured in the EPP. For P/D serving, llm-d uses a disaggregation profile handler with separate scheduling profiles for decode and prefill.

An illustrative configuration looks like this:

apiVersion: llm-d.ai/v1alpha1
kind: EndpointPickerConfig

plugins:
  - type: label-selector-filter
    name: prefill-pods
    parameters:
      matchExpressions:
        - key: llm-d.ai/role
          operator: In
          values: [prefill]

  - type: label-selector-filter
    name: decode-pods
    parameters:
      matchExpressions:
        - key: llm-d.ai/role
          operator: In
          values: [decode]

  - type: prefix-cache-scorer
  - type: max-score-picker

  - type: prefix-based-pd-decider
    parameters:
      # Illustrative only. Tune these from real workloads.
      nonCachedTokens: 512
      promptTokens: 1024

  - type: disagg-profile-handler
    parameters:
      profiles:
        prefill: prefill
        decode: decode
      deciders:
        prefill: prefix-based-pd-decider

schedulingProfiles:
  - name: prefill
    plugins:
      - pluginRef: prefill-pods
      - pluginRef: prefix-cache-scorer
      - pluginRef: max-score-picker

  - name: decode
    plugins:
      - pluginRef: decode-pods
      - pluginRef: prefix-cache-scorer
      - pluginRef: max-score-picker

The exact thresholds and scorer mix are workload-specific. The important idea is the structure:

One request, step by step

Suppose the user sends a 12,000-token prompt.

1. Request reaches the llm-d proxy

The proxy receives the normal OpenAI-style request. It consults the EPP before choosing a model-server endpoint.

The proxy does not itself calculate prefix-cache affinity or GPU load. That is the EPP's job.

2. EPP chooses the decode worker first

In the default decode-first P/D flow, the EPP runs the decode scheduling profile, filters to decode-capable workers, scores them for cache locality and load, and selects D2. llm-d now knows where final token generation would happen.

3. The decider asks whether remote prefill is worth it

Suppose D2 already has most of the prompt cached. Shipping the request through a separate prefill worker may be unnecessary.

The P/D decider considers the prompt length and the uncached suffix on the selected decode server. If only 500 tokens are missing, local prefill may be cheaper. For another request, most of the prompt may be missing:

This is important: P/D disaggregation can be a per-request decision, not simply a permanent rule that every request must use two servers.

4. If needed, EPP chooses a prefill worker

The EPP runs the prefill profile, filters to prefill-capable workers, scores them and chooses P2. It returns the decode worker as the main request destination and communicates the selected prefill endpoint through routing metadata. In the current sidecar architecture this includes a header such as:

x-prefiller-host-port: <P2-address>

5. The request lands on the decode pod's routing sidecar

The decode pod contains a routing proxy sidecar. This sidecar orchestrates the two-step request. Prefill pods do not need that sidecar for this flow.

6. Prefill worker creates the KV cache

P2 processes the prompt and creates the KV state. The sidecar and vLLM use the model server's KV-transfer protocol to coordinate how D2 can retrieve those KV blocks.

The coordination happens before any tensor moves. An out-of-band handshake exchanges NIXL agent metadata used to establish the connection. The prefiller also returns remote memory descriptors and the block IDs that identify this request's KV data. No KV tensor has crossed the wire yet.

7. KV is transferred to the decode worker

The cache is not one shared, coherent allocation. It is copied. Each vLLM pod owns its own paged KV pool. RDMA lets D2 read registered memory on P2, but D2 still copies those blocks into its own pool. For a short time, the same KV data exists on both workers.

The actual KV tensors can be transferred using NIXL. For production, llm-d recommends high-bandwidth networking such as InfiniBand, RoCE or EFA; TCP fallback is useful mainly for development because KV transfer is large and latency-sensitive.

D2 is the side that initiates. It issues a one-sided RDMA read against the descriptors from the handshake, pulling the blocks out of P2's memory. P2 does not push them, and once the handshake is done P2's CPU is not involved at all.

With GPUDirect RDMA-capable infrastructure, the data path can avoid copying the KV through normal application buffers and CPU memory.

"Large" deserves a number. A particular 70B architecture using grouped-query attention with 80 layers, 8 KV heads and head dimension 128 in BF16 needs 2 x 80 x 8 x 128 x 2 bytes per token, which is 320 KiB. A 12,000-token prompt is therefore about 3.66 GiB of KV for one request, split across the four GPUs of the tensor-parallel group. The number is architecture-specific: a model with 64 KV heads would use eight times as much for the same layer count, head dimension and data type.

On a 400 Gb/s class link that is tens of milliseconds. On ordinary 25 GbE it is over a second, which is longer than the prefill it was supposed to save. That is the whole reason llm-d treats TCP as a development fallback.

NIXL is therefore solving a different problem from NCCL:

8. Decode starts from the transferred KV

D2 does not need to recompute the whole prompt. It pulls the remote KV state into blocks it has already allocated, then starts producing output tokens. In llm-d's default vLLM nixlv2 request flow, the routing sidecar treats this as a two-phase sequence: the prefill response returns transfer parameters before the decode request is forwarded. Some connector and runtime implementations can overlap parts of the data movement with computation, but that is version- and backend-specific rather than a property to assume.

The final response streams back through the sidecar and gateway to the client.

A different answer: pool the cache instead of copying it. Systems such as Mooncake can provide a distributed KV-cache store backed by memory and storage across the cluster. Prefill instances write KV into the pool and later workers read it back. LMCache provides related tiered and remote-cache integrations for vLLM.

The trade is different from a point-to-point copy. A pooled tier costs a hop through slower memory, but what lands in it can be reused by later requests and by other instances, not just by the one decode pod waiting for this prompt.

What if the decider says not to disaggregate?

Then no prefill endpoint is attached to the request. The selected decode vLLM server simply performs both phases locally:

So the infrastructure supports both paths at the same time.

Where each piece of P/D configuration lives

This is the practical map:

This separation is useful because none of your application code needs to contain logic such as:

if (promptTokens > 10000) {
    callPrefillPod();
    copyKv();
    callDecodePod();
}

That fleet-level orchestration belongs in the serving infrastructure. Your application can continue to send a normal model request.

Should every production system start with P/D disaggregation?

No. Start with combined vLLM servers unless measurements show a reason to separate the phases. P/D adds a sidecar, more scheduling policy, KV transfer, high-speed networking requirements and more failure modes. It becomes valuable when long prefills interfere with decode latency or when independent scaling/specialization materially improves the workload.

19. Model files are part of production architecture too

Large models can be tens or hundreds of gigabytes.

You do not want every pod restart to spend a long time pulling the same weights across the internet.

Common approaches include:

node-local NVMe cache
shared high-performance filesystem
persistent volume cache
pre-baked model images in specific environments

Whichever approach you choose, account for:

• cold-start time;
• disk capacity;
• model versioning;
• checksum/integrity verification;
• rollout of new weights;
• access to gated/private models.

A model rollout should be treated like a major application rollout, not like replacing a tiny container image.

Pin one immutable serving release containing or referencing all compatibility-sensitive parts:

model weights
tokenizer
chat template
generation defaults
quantization format
vLLM image and arguments
KV-transfer layout and connector version

Two pods with the same model name but different tokenizers, chat templates or KV layouts are not interchangeable. Do not mix incompatible prefill and decode workers in one pool. Bring up a new pool, validate it, shift a small amount of traffic, and keep the old pool available for rollback until active requests have drained.

Protect the model supply chain

Treat model weights and serving images as executable production inputs:

• allow only approved registries and model repositories;
• pin images by digest and verify model checksums or signatures;
• use a dedicated ServiceAccount with minimal RBAC;
• mount credentials only into pods that need them;
• restrict unnecessary outbound network access;
• run the container without host privileges or writable host mounts;
• use workload identity and encrypted service-to-service traffic when the threat model requires it.

Never log raw authorization headers, prompts, generated text or cache_salt. Send content to a separately governed audit system only when product and privacy requirements require retention.

20. Observability: measure tokens, not only requests

Traditional HTTP metrics are useful but incomplete.

Two requests can have completely different costs:

Request A
100 input tokens
20 output tokens

Request B
30,000 input tokens
4,000 output tokens

Both are "one request" but they are not remotely equal workloads.

Useful production measurements include:

request rate
input tokens / second
output tokens / second
time to first token
inter-token latency
end-to-end latency
queue time
active sequences
KV-cache utilization
prefix-cache hit rate
GPU utilization
GPU memory utilization
request rejection / quota rate
model load/startup time

For a multi-tenant product also track usage for chargeback and capacity planning, but avoid turning every tenant into a high-cardinality Prometheus label.

Autoscale on work waiting, not only GPU utilization

GPU utilization alone is a poor scaling signal. A decode-heavy replica can be memory-bandwidth-bound while its arithmetic units look underused, and a new replica can appear idle for minutes while weights load.

Useful scaling inputs include:

queued input tokens
queued requests by priority
running sequences
KV-cache pressure
observed TTFT and TPOT
predicted latency for admitted work
ready replicas, excluding replicas still loading

Current llm-d guidance separates two common paths. KEDA with EPP metrics scales a homogeneous pool from inference demand observed by the EPP. HPA with Workload Variant Autoscaler metrics is useful when several model or hardware variants share capacity. Treat either as a starting point and validate its signals against model-loading time and the workload SLO.

Scale whole model replicas. In the running example, one scale step means four GPUs, not one. For a multi-node model managed by LeaderWorkerSet, it means one whole leader-worker group.

Keep a minimum number of warm replicas and reserve enough capacity for the chosen failure target. Scaling from zero is usually unsuitable for an interactive model whose weights take minutes to load. Scale-down must use the same drain sequence as a rollout so active streams are not cut off and useful prefix cache is not discarded unnecessarily.

21. Failure cases worth designing before launch

GPU failure

If one GPU in a four-way tensor-parallel replica fails, the logical model replica is usually unhealthy. Treat the four GPUs as one failure unit for that replica. If the replica spans nodes under a LeaderWorkerSet, the failure unit grows to the whole group, and the group restarts together.

Pod restart

Model weights reload and local KV/prefix cache may be lost.

Your conversation state must not depend on that cache surviving.

One tenant floods the service

Admission control and fairness must stop one tenant from monopolizing the shared pool.

Router sends traffic to a cold replica

Prefix-cache reuse falls and TTFT can increase. An inference-aware router helps once there are multiple replicas.

Model rollout doubles GPU demand temporarily

A four-GPU replica can make rolling updates surprisingly expensive. Plan capacity and rollout strategy carefully.

Large prompt unexpectedly consumes KV memory

Enforce maximum context and output limits at admission time, not after the request has already occupied GPU capacity.

Gateway or EPP failure

Decide explicitly whether routing fails open or fails closed. InferencePool.endpointPickerRef.failureMode supports both behaviours. Fail-open can preserve availability through simpler routing, but it may lose cache locality, fairness or other policy. Fail-closed preserves the routing contract but rejects traffic when the EPP is unavailable. Run the gateway and EPP with more than one replica and test both the chosen mode and recovery.

If EPP Flow Control is enabled, its waiting queues are held in memory. Multiple EPP replicas improve availability, but they do not make requests already waiting inside one failed EPP durable. Clients and gateways still need bounded timeouts and a retry policy that avoids duplicating requests after output may have started.

Client disconnect during generation

If cancellation stops at a proxy, vLLM may continue decoding and holding KV memory. Trace a disconnect across every hop and alert when upstream cancellation does not promptly reduce active sequences.

With vLLM P/D disaggregation, a small cancellation window can strand blocks on the prefiller until VLLM_NIXL_ABORT_REQUEST_TIMEOUT expires. The documented default is 480 seconds. Tune it only after measuring the longest legitimate delay before a decoder pulls KV; making it too short can free blocks that a slow decoder still needs.

Prefill or KV transfer failure

In a disaggregated path, the prefiller can fail before returning transfer metadata, or the decoder can fail while pulling KV. Bound both phases with timeouts. vLLM's default KV-load policy fails the request; recompute can fall back to local prefill but may cause latency spikes on the decode pool. Do not retry client errors or blindly repeat a request after output may already have streamed.

Zone loss

Replicated control-plane pods are not enough if every ready model replica uses GPU nodes in one zone. Define whether the service must remain within its latency SLO after a zone loss, then keep warm GPU capacity in the surviving zones. A PodDisruptionBudget helps with voluntary disruptions, but it does not create spare GPUs or protect against an entire zone failure.

Retry after partial output

Before the first token, a gateway can often retry a safe request on another replica. After tokens have streamed, retrying can duplicate or change the answer. Record whether output started and return a clear terminal error rather than silently beginning a second generation.

22. A sensible production rollout path

Do not start by deploying every advanced component.

Phase 1 — Make one replica correct

Implement:

authentication
tenant identity
quotas
streaming
usage accounting
prefix caching
tenant cache salt
metrics

Understand the behavior of prefill, decode, KV cache, and memory usage.

Phase 2 — Add more replicas

Now cache locality and load-aware routing matter.

This is where llm-d becomes valuable.

Deploy a new model or runtime as a separate versioned pool. Send shadow traffic first, then a small canary slice. Compare correctness, TTFT, TPOT, error rate, KV usage and tokens per GPU-second before increasing traffic. Route assignment must be stable enough that one user is not switched between model versions in the middle of a conversation unless the product permits it.

Phase 3 — Improve fleet efficiency

Based on measurements, consider:

cache-aware routing tuning
priority/fairness classes
KV offloading / external cache systems
prefill/decode disaggregation
quantization
speculative decoding
multi-node models

Do these because a measured bottleneck requires them, not because the features exist.

At every phase, rehearse rollback. Stop new traffic to the bad pool, drain or cancel its active streams according to policy, and retain the previous immutable release until recovery is complete. A rolling update that temporarily needs twice the GPUs is not safe unless that surge capacity really exists.

23. The entire request in one picture

Here is the full architecture we have built up.

24. The five ideas to remember

If the rest of the article fades, keep these five ideas.

1. Kubernetes does not run transformer inference

Kubernetes allocates resources and runs the model-server container. vLLM is the component that turns the model into an inference service.

2. KV cache and prompt caching are related but different

KV cache avoids recomputing old tokens while a request generates. Prefix caching allows later requests with the same prefix to reuse already-computed KV blocks.

3. A giant model can be one logical server across many GPUs

Tensor parallelism shards one model replica across GPUs. vLLM coordinates the execution and NCCL performs the low-level collective communication.

4. Multi-tenancy needs several isolation layers

Authentication, quotas, fairness, network isolation, prefix-cache isolation, logging boundaries, and sometimes dedicated GPU pools are separate concerns.

5. Once you have many replicas, ordinary load balancing is too naive

A good router should understand cache locality and current inference load, and once prefill and decode are split it picks a prefill/decode pair rather than a single endpoint. That is the role llm-d can play around a fleet of vLLM servers.

Final mental model

Ask one question at each layer:

Your gateway
"Is this tenant allowed to make this request, and how much may it consume?"

Kubernetes
"Where should the model-server process run?"

llm-d
"Which model-server replica or prefill/decode pair should receive the request?"

vLLM
"Which requests and tokens should the GPU execute now?"

KV cache and prefix cache
"What previous attention work can be reused?"

Tensor parallelism
"How do several GPUs execute one model together?"

NCCL
"How do GPUs inside one model replica exchange tensors?"

NIXL
"How does KV state move from a prefill server to a decode server?"

Once these responsibilities are separated, the system is much easier to understand and operate.

Further reading

The concepts in this article map directly to the current official documentation:

• vLLM Kubernetes deployment: https://docs.vllm.ai/en/stable/deployment/k8s/
• vLLM Automatic Prefix Caching: https://docs.vllm.ai/en/latest/design/prefix_caching/
• vLLM optimization and batching guidance: https://docs.vllm.ai/en/latest/configuration/optimization/
• vLLM parallelism and scaling: https://docs.vllm.ai/en/latest/serving/parallelism_scaling/
• vLLM production stack: https://docs.vllm.ai/en/latest/deployment/integrations/production-stack/
• llm-d architecture and Endpoint Picker: https://llm-d.ai/docs/dev/architecture/core/router/epp
• llm-d EPP configuration (EndpointPickerConfig): https://llm-d.ai/docs/dev/architecture/core/router/epp/configuration
• llm-d API reference, including InferencePool failure modes: https://llm-d.ai/docs/api-reference
• llm-d current well-lit paths: https://llm-d.ai/docs/well-lit-paths
• llm-d autoscaling architecture: https://llm-d.ai/docs/dev/architecture/advanced/autoscaling
• llm-d disaggregated serving architecture: https://llm-d.ai/docs/architecture/advanced/disaggregation
• llm-d vLLM disaggregation operations: https://llm-d.ai/docs/architecture/advanced/disaggregation/operations-vllm
• llm-d P/D disaggregation well-lit path: https://llm-d.ai/docs/well-lit-paths/foundations/pd-disaggregation
• llm-d quickstart / installation shape: https://llm-d.ai/docs/dev/getting-started/quickstart
• Gateway API InferencePool reference: https://llm-d.ai/docs/dev/api-reference/inferencepool
• llm-d API / InferencePool concepts: https://llm-d.ai/docs/api-reference
• llm-d token-aware routing: https://llm-d.ai/blog/sticky-until-saturated-token-aware-routing
• Kubernetes LeaderWorkerSet: https://lws.sigs.k8s.io/docs/overview/
• LeaderWorkerSet with Kueue topology-aware scheduling: https://lws.sigs.k8s.io/docs/examples/tas/
• Kubernetes DisaggregatedSet: https://lws.sigs.k8s.io/docs/reference/disaggregatedset.v1/
• NVIDIA NCCL documentation: https://docs.nvidia.com/deeplearning/nccl/
• NVIDIA H100 specifications: https://www.nvidia.com/en-us/data-center/h100/

Related System Designs

• ML Model Serving Platform: Predictive inference for ranking, vision, XGBoost, and bounded model responses
• LLM Inference Serving Platform: Token streaming, prefill and decode, KV cache, and multi-GPU model groups
• Distributed Training a 70B Model: Training-time sharding, checkpointing, and exact recovery across 1024 GPUs
• LLM Evaluation Platform: Golden sets, LLM-as-judge, regression gates, and online evaluation
• LLM Safety Pipeline: Input and output policy enforcement, tool authorization, and appeals