How a 284B Model Can Run on a Consumer GPU

From Tokens to Expert Offloading

The central problem has four parts.

A dense model uses the same path for every token

A large language model (LLM) is built from many layers. Each layer contains learned numerical values called parameters. Two important parts of a Transformer layer are attention and a feed-forward network, or FFN.

In a dense model, each layer has one fixed FFN that every token uses. Increasing the model size can improve its capacity, but it also increases the weights that must be stored, moved, and used.

Inference means using an already-trained model to answer a request.

What an MoE model changes

A Mixture-of-Experts, or MoE, layer replaces one FFN with many alternative FFNs. Each alternative FFN is called an expert.

An expert is not a person, a complete model, or necessarily a human-readable subject specialist. It is one learned computation block inside one layer.

A small learned component called the router scores the experts for the current token and selects only a few. The selected experts process that token, and their outputs are combined.

This gives the model many stored parameters without using every parameter for every token. In other words, MoE increases stored parameter capacity while keeping the active computation path smaller.

The new problem created by MoE

A dense model can also be too large for GPU memory. In that case, its weights must be stored outside VRAM or divided across devices. Because every token uses almost the entire dense model, there is no small expert subset to load selectively.

MoE creates a different version of the memory problem. All experts still have to exist somewhere, but each token uses only a few of them. The router can select a different subset for every token and at every layer.

A consumer GPU may not have enough VRAM, the memory attached to the GPU, for the complete expert pool. The serving runtime must therefore keep track of the selected experts and decide where each one should execute.

Now inference becomes a placement and data-movement problem:

Which experts were selected?
Where are their weights?
Can they reach a processor quickly enough?

Where FreeToken fits

FreeToken is an inference runtime, not a new model. The CPU and GPU are the processors that perform the calculations. System RAM and GPU VRAM are the memories that hold the weights they use.

FreeToken keeps the complete expert-weight pool in system RAM, the machine's main memory. Recently used expert weights may also remain in a smaller cache in GPU VRAM.

When the router selects an expert whose weights are not already in the VRAM cache, FreeToken has two choices. It can copy those weights from system RAM across PCIe, the connection between system RAM and the GPU, into VRAM. The GPU can then execute that expert.

Or it can leave the weights in system RAM and let the CPU execute that expert directly from there. The partial CPU and GPU outputs are then merged exactly.

Part 1 — How a normal dense LLM works

1. What is a parameter?

An LLM is ultimately a huge collection of learned numerical values.

A single learned value is often called a weight.

The broader word parameter includes learned weights and, depending on the architecture, other learned values such as biases or normalization parameters.

Imagine a tiny neural network:

weight_1 =  0.423
weight_2 = -1.172
weight_3 =  0.081
weight_4 =  2.314
...

Training repeatedly changes these numbers so the network becomes better at predicting the correct output.

Therefore:

14B model

roughly means:

14,000,000,000 learned parameters

Those parameters are distributed across the model:

Model parameters
│
├── token embeddings
│
├── attention projection weights
│
├── feed-forward network weights
│
├── normalization parameters
│
└── output projection weights

Three concepts are easy to mix up:

PARAMETER
A learned number stored in the model.

TOKEN
A piece of text being processed or generated.

ACTIVATION
A temporary number/vector produced while
processing a particular input.

A parameter persists across requests.

An activation depends on the request.

For example:

stored model parameter
0.4821

may remain unchanged for months.

But:

"The capital of France is"
              ↓
temporary hidden vectors

exist only while the model processes that request.

This distinction explains statements such as:

284B total parameters
but
13B active parameters

"Active" means participating in the current computation path. It does not mean the inactive parameters disappeared.

2. How many bytes does a parameter need?

A parameter is a number, and a number has to be represented using some number of bits.

A common training or inference representation is 16 bits.

Very roughly:

16 bits = 2 bytes
8 bits  = 1 byte
4 bits  = 0.5 byte

So a simple storage estimate is:

number of parameters
×
bytes per parameter

For 35B parameters:

35B × 2 bytes
≈ 70 GB

At 8 bits:

35B × 1 byte
≈ 35 GB

At 4 bits:

35B × 0.5 byte
≈ 17.5 GB

What is quantization?

Quantization means representing model weights using lower precision.

For example, instead of storing a weight using 16 bits, a quantized model file may represent groups of weights using 8-bit or 4-bit values plus scaling information.

Conceptually:

Lower precision can reduce:

• storage;
• RAM requirements;
• VRAM requirements;
• memory bandwidth.

But real model file sizes are not exactly:

parameters × bits

because quantized formats may also need:

• scaling values;
• metadata;
• padding;
• some numerical arrays, called tensors, stored at higher precision;
• separate shared weights;
• runtime buffers.

So:

35B at 4 bits ≈ 17.5 GB

is a useful mental estimate, not a guarantee about a specific file.

Quantization reduces the number of bytes that must be stored and moved.

It does not change this fundamental fact:

All parameters required by the model still need to exist somewhere.

3. What is a token?

An LLM does not directly process sentences as human-readable strings.

Before inference, a tokenizer converts text into token IDs.

For example:

"The capital of France is"

may become pieces resembling:

"The"
" capital"
" of"
" France"
" is"

The exact split depends on the tokenizer.

The process is:

An autoregressive language model generates one new token at a time.

For example, the prompt "The capital of France is" may produce " Paris". The model appends that token to the context, runs again, and may then produce ".". The inference loop is therefore: compute next-token scores, choose one token, append it, and repeat.

Inference has two phases, which sections 20 and 21 explain in detail:

Both the input and output are processed as tokens.

4. What does a token look like inside the model?

A model cannot calculate directly with the literal word:

France

The token is converted into a vector.

A vector is simply a list of numbers:

"France"
    ↓

[0.13, -0.72, 0.44, 0.09, ...]

A real LLM hidden vector can contain thousands of numbers.

A matrix is a rectangular grid of numbers. Matrix multiplication is the operation neural networks repeatedly use to transform vectors with learned weights. A tensor is the broader name for a numerical array: a vector is a one-dimensional tensor, a matrix is a two-dimensional tensor, and model software also uses tensors with more dimensions.

The initial token vector comes from an embedding table.

You can think of an embedding as the model's starting numerical representation for a token.

But that vector does not stay fixed. Each layer reads the previous representation and produces a new hidden state:

embedding
→ layer 1 hidden state
→ layer 2 hidden state
→ ...
→ final hidden state

Why context matters

Consider:

"Java is a programming language."

"I traveled across Java."

The token "Java" begins from the same vocabulary identity, but after contextual processing its hidden representation can differ.

Conceptually:

An MoE router does not simply see:

word == "Java"

It receives this contextual hidden vector. That is why the same token can be routed differently in different sentences.

5. What is a Transformer layer?

Most modern LLMs contain many repeated computational blocks.

The examples use a tiny imaginary model with five Transformer layers. Real models commonly have dozens.

A simplified Transformer layer contains two major pieces:

Real layers also contain normalization, residual connections, positional information, and architecture-specific details. Those details do not change the dense-versus-MoE distinction, so this article uses the simpler path:

Attention
   ↓
FFN

Attention gathers information from other token positions, while the FFN transforms the information held at the current position.

Neither of these directly says:

"Choose Paris as the next token."

The final next-token decision happens only after the representation has passed through all layers.

6. What is attention?

Take this sentence:

"The capital of France is"

The model eventually needs to predict what comes next.

For the final position, some earlier information is particularly relevant:

"capital"
"France"

and some may be less relevant:

"The"
"of"

Self-attention lets the representation at one token position gather information from other token positions.

A conceptual example:

Those numbers are only illustrative.

Real attention operates over vectors and many dimensions.

Optional attention detail: Query, Key, and Value

These mechanics explain attention more precisely. You can skip them for the expert-offloading mechanism, although section 22 briefly returns to Key and Value when it explains the KV cache.

Attention usually derives three vectors:

A useful intuition:

Query
"What am I looking for?"

Key
"What kind of information does this token contain?"

Value
"What information should be taken from this token
if it is relevant?"

The current Query is compared with Keys:

After normalization, the scores determine how Values are combined.

Very simplified:

The result is another vector.

Optional attention detail: parameters versus scores

The phrase attention weights can mean two different things, so it is better to be precise.

Learned parameters:

Wq
Wk
Wv
Wo

These are stored in the model.

Runtime scores:

France  → 0.55
capital → 0.35

These are temporary and change with the request.

Optional attention detail: causal masking

A text-generating model should not look at future tokens that have not been generated.

If the sequence is:

A B C D

while processing C, the model may use:

A B C

but not future D.

A causal mask enforces this.

Optional attention detail: multiple heads

Transformers normally run several attention heads in parallel:

Each head has its own learned projections, so different relationships can be captured.

Attention lets information move between token positions.

7. What is an FFN?

FFN stands for Feed-Forward Network.

The FFN does not directly choose the next token.

For the sentence:

"The capital of France is"

Attention may produce a representation that now contains useful contextual information related to:

France
+
capital
+
the current sentence structure

Then the FFN transforms that representation.

Attention gathers information; the FFN processes and transforms what was gathered.

Optional math detail: what does an FFN look like?

A simplified FFN:

Suppose a hidden vector contains 4,096 numbers.

The FFN may expand it into a much larger intermediate vector:

Those large matrices contain many parameters.

Modern LLM FFNs often use gated structures such as SwiGLU and therefore may contain more than the two simple matrices shown above, but the role remains similar:

input hidden state
       ↓
large learned non-linear transformation
       ↓
output hidden state

Attention and FFN do different jobs

Attention:

mixes information across token positions

FFN:

applies the same learned transformation
independently to each token position
within that layer

For example:

The same FFN parameters are reused for all positions in that layer.

This is the exact component that MoE will later replace with many experts.

8. Where does the next token actually get chosen?

After the final Transformer layer, the model has a final hidden vector for the current generation position.

That vector is projected into one score for every token in the vocabulary.

A logit is an unnormalized score.

For example:

To make the arithmetic checkable, suppose softmax is applied only to these five displayed logits. They become approximately:

A real model normalizes scores for its entire vocabulary, so its probabilities also depend on every logit not shown here.

Then the decoding strategy selects a token.

So:

Attention
does NOT directly choose "Paris"

FFN
does NOT directly choose "Paris"

Router
will NOT directly choose "Paris"

Experts
will NOT directly choose "Paris"

They all help transform hidden representations.

The final vocabulary projection produces next-token scores.

9. Walk one token through our 5-layer model

Use the same prompt:

"The capital of France is"

The following explanation is conceptual. Individual real layers are not assigned human-readable jobs such as "detect geography."

Each layer repeats the same structural operation: attention mixes information across token positions, and the FFN transforms each position's updated hidden state. Because one layer receives the output of the previous layer, the representation at the final position can gradually encode relationships associated with:

capital-of-France

without containing that phrase literally.

After the fifth layer, the output projection turns the final hidden state into vocabulary logits and the decoding strategy chooses the next token. The complete simplified path is:

10. What is a dense model?

A normal dense LLM uses essentially the full parameterized network for each token's forward pass—the computation that moves an input through the model to produce an output.

Our toy dense model:

There is no decision like:

For this token, skip FFN-3
and use some other FFN instead.

Each layer has its normal path.

A toy parameter count

Suppose every layer contains:

attention/shared parameters = 20M
FFN parameters              = 80M

Then one layer has:

20M + 80M = 100M

Five layers:

5 × 100M = 500M total parameters

A token passes through all five layers, so in this simplified model:

Total parameters  ≈ 500M
Active parameters ≈ 500M

Scale the same idea to a dense 14B model:

Dense 14B

Total parameters
≈ 14B

Active per token
≈ almost the full model

That does not mean parameters execute one by one:

parameter 1
parameter 2
...
parameter 14,000,000,000

The GPU performs large parallel matrix operations.

"Active" simply means the token's forward pass uses that parameterized path.

Part 2 — Mixture of Experts

11. What problem is MoE trying to solve?

Suppose we want to increase the amount of learned parameter capacity available to the model.

With a dense architecture:

14B
 ↓
70B
 ↓
200B

normally means both:

more weights stored

and:

more weights used for each token

More active weights mean more:

• memory reads;
• compute;
• GPU time;
• cost.

What if we could build a much larger collection of parameters but choose only a small part for a particular token?

That is the core idea of a Mixture-of-Experts (MoE) model.

This is commonly called sparse activation.

Sparse activation does not mean:

most stored weights are zero

It means:

many parameter blocks exist,
but only selected blocks participate
for a given token

This is why total and active parameter counts describe different things. A dense 284B model uses nearly its entire network for each token. A 284B-total MoE model may use only a much smaller selected part of its expert pool.

Experts not selected for one token can still be selected by other tokens. They let the model store more learned capacity without making every token use all of it.

The two models therefore do not perform the same computation, and the parameter label alone does not predict equal answer quality. Quality also depends on the data, training process, architecture, and how well the router and experts learned.

12. What is an expert?

In many Transformer MoE architectures, an expert is an FFN.

In this article, you can read "expert" as "one alternative FFN inside an MoE layer."

Dense Transformer layer (simplified):

Both diagrams omit normalization and residual connections so the FFN difference is easier to see.

MoE Transformer layer (simplified):

Each expert has its own parameters.

So instead of:

one large FFN

the model has:

many FFNs

but only a few are selected for one token.

The diagram above is a simplified top-2 routing example. Expert-pool size and the number selected vary across models. The 284B DeepSeek-V4-Flash discussed later has 256 routed experts in each of 43 MoE layers and selects six for each token.

13. What is the router?

The router is a small learned neural network inside an MoE layer.

Its job is:

Given this token's current hidden state, which experts should process it?

Suppose the layer has eight experts.

The router computes scores:

If the model uses top-2 routing:

select E3
select E5

The router also usually produces gate values used when combining selected expert outputs.

A simplified picture might be:

E3 contribution = 0.55
E5 contribution = 0.45

Then:

output
≈
0.55 × E3(hidden)
+
0.45 × E5(hidden)

The details differ among MoE architectures, but the central idea is stable.

The router is not hand-written logic

It is not:

if topic == "SQL":
    choose_database_expert()

if language == "Java":
    choose_programming_expert()

The router sees the token's current hidden vector, something more like:

[0.17, -0.92, 0.33, ...]

This is why:

"Java is a programming language"

and:

"I traveled across Java"

can route differently.

The router receives contextual representations, not just token strings.

14. How the router learns

Experts are not manually assigned topics such as geography or programming. During model training, prediction errors update both the experts and the router. Over many examples, an expert can become useful for certain hidden-state patterns, and the router learns when selecting it improves the prediction.

Those patterns may combine syntax, language, token shape, arithmetic, or features that have no simple human label. At inference time, the trained router receives the current contextual vector and produces expert scores. FreeToken preserves that decision.

15. Why MoE training needs load balancing

This kind of load balancing happens while the model is being trained.

Early in training, one expert may become slightly better than the others. The router then sends it more tokens, so that expert receives more training and becomes even stronger. Eventually, a few experts could receive nearly all the work while the others learn very little. This feedback loop is called expert imbalance or routing collapse.

The training process prevents this with additional rules that encourage the router to use the expert pool more broadly. The goal is not perfectly equal traffic. It is to stop a small number of experts from receiving almost every token.

After training, the learned router is saved as part of the model. During normal inference, it scores each token's hidden state and selects experts using what it has already learned. The training-time balancing rules do not retrain the router for each request.

Do not confuse this training-time balancing with FreeToken's serving-time scheduling. During inference, the trained router's selection is fixed for that step. FreeToken later decides where the already-selected expert runs; it does not rebalance training traffic or change the router's choice.

16. Walk the same 5-layer example through MoE

Use the same prompt:

"The capital of France is"

Suppose our imaginary MoE model has eight experts in each layer and selects two per token.

To avoid implying that the whole prompt shares one route, the diagram follows only the final prompt position, "is". Every other prompt position is routed independently.

At every layer, attention first changes this position's hidden representation. That layer's router then makes a fresh decision from the new state, so the chosen experts can differ from the previous layer. The expert numbers below are illustrative, but the changing route is the important part:

For a different prompt:

"Write a Java binary search"

The routes can be different:

Layer 1 → E1,E4
Layer 2 → E6,E8
Layer 3 → E2,E6
Layer 4 → E3,E8
Layer 5 → E1,E5

The expert numbers do not represent human-readable specialties. They illustrate only that different inputs can activate different paths.

17. Numeric example: dense versus MoE

The following toy model makes the parameter arithmetic concrete.

Dense version

Five layers.

Each layer has:

Attention/shared parameters = 20M
FFN parameters              = 80M

Per layer:

20M + 80M = 100M

Five layers:

5 × 100M = 500M total parameters

The full path runs:

Active/token ≈ 500M

MoE version

Keep:

20M attention/shared parameters

But replace one 80M FFN with 8 separate 80M experts.

Per layer:

Expert pool:

8 × 80M = 640M

Total per layer:

20M + 640M
= 660M

Five layers:

5 × 660M
= 3.3B total parameters

Suppose the router activates only one expert per layer.

The active path per layer is:

20M shared
+
80M selected expert
=
100M

Across five layers:

5 × 100M
= 500M active

So:

In this toy example, MoE stores 6.6 times as many parameters while activating the same 500M parameters for one token. That is a statement about stored and active parameter counts, not a claim of 6.6 times better answers.

18. Real examples: Qwen3.6 and DeepSeek-V4-Flash

The same distinction appears in real model descriptions.

Qwen3.6-35B-A3B

The official Qwen model card describes:

Total parameters: ~35B
Activated parameters: ~3B

40 layers
256 experts
8 routed experts active per token
+ 1 shared expert

The A3B in the model name is a clue:

35B total
A3B ≈ 3B activated

So:

35B stored parameters
      ↓
~3B active parameter path
for a token

DeepSeek-V4-Flash

The official DeepSeek model card confirms the 284B-total, 13B-active architecture. The FreeToken paper reports the routing details used in its evaluation:

Total parameters: 284B
Active parameters: ~13B

43 MoE layers

each MoE layer:
256 routed experts
6 selected

So it is not:

choose six experts once
for the entire query

It is:

at each MoE layer
for each token
router chooses the active experts

The ~13B number represents the model's overall active parameter path, including shared/non-expert components and selected experts across the network.

19. Routing happens per token and per layer

Routing does not happen like this:

Instead, if the prompt has tokens:

"Explain"
" how"
" PostgreSQL"
" indexing"
" works"

at MoE Layer 10 we might conceptually see:

Then at Layer 11 the hidden representations have changed.

Routing happens again:

Routing therefore changes both from one layer to the next and from one generated token to the next:

Generated token t
→ one routing pattern

Generated token t+1
→ another routing pattern

Generated token t+2
→ another routing pattern

This dynamic routing behavior is exactly why a fixed expert-placement policy can perform poorly.

20. What is prefill?

Suppose you send a prompt with 10,000 tokens.

Before generating the first answer token, the model must process that existing sequence.

This first phase is called prefill.

In a normal chat:

system prompt
+
conversation history
+
new user message

all may be part of prefill.

In a coding agent:

system prompt
+
repository instructions
+
conversation
+
source code
+
tool calls
+
tool outputs
+
reasoning context

can make the prompt much larger.

TTFT, or time to first token, is heavily affected by prefill.

TTFT means:

21. What is decode?

Prefill's final hidden state produces the scores used to select the first output token. The engine then appends that token and performs another one-token model pass to produce the next token. This repeated one-token phase is called decode.

prefill produces token 1
       ↓
first decode pass produces token 2
       ↓
later decode pass produces token 3
       ↓
...

Decode usually advances one new token per sequence at a time.

For each generated token:

FreeToken uses very different strategies for prefill and decode because their MoE working sets behave differently.

22. What is a Key–Value (KV) cache?

In attention, a Query represents what the current token position is looking for. Keys describe what previous positions offer, and Values carry the information retrieved from those positions.

Attention has already computed Key and Value vectors for the previous tokens.

Without caching, every generated token would repeatedly recompute large portions of old attention state.

The KV cache stores those past Key and Value tensors.

Suppose tokens A, B, C, and D have already been processed:

A → K_A, V_A
B → K_B, V_B
C → K_C, V_C
D → K_D, V_D

Store them:

KV cache

K_A V_A
K_B V_B
K_C V_C
K_D V_D

When new token E arrives, the model computes its state and lets its Query attend to cached Keys:

Q_E
 ↓
compare against
K_A K_B K_C K_D K_E

Then add:

K_E V_E

to the cache.

Important:

As a conversation grows, KV/cache state may consume more VRAM.

The growing request state leaves less VRAM available for FreeToken's expert cache.

23. Why a long prefill can destroy MoE sparsity

For one decode token:

token
 ↓
router
 ↓
only a few experts

That is sparse.

But consider a 20,000-token prompt.

Each token may choose different experts:

Token 1     → E3,E7,...
Token 2     → E9,E12,...
Token 3     → E31,E92,...
...
Token 20000 → E5,E104,...

Each individual token is sparse.

But the union of all routes can cover nearly every expert.

This means prefill can behave like:

effectively dense expert working set

even though decode is sparse.

The FreeToken paper reports this as one of the central edge-MoE problems.

For its FP4 DeepSeek-V4-Flash example, prefill may require streaming roughly 140 GB of expert weights across the CPU–GPU link.

The paper estimates this transfer alone at roughly:

if the movement is exposed rather than overlapped with computation.

So MoE solves compute sparsity, but prefill can create a massive data-movement problem.

24. The serving problem created by MoE

Sparse activation reduces the expert work for one token, but it does not remove the complete expert pool. The next token may select a different subset, so all experts must remain available.

For every token and MoE layer, the serving engine must answer:

Which experts were selected?
Where are their weights?
Are they already in VRAM?
Should weights missing from the VRAM cache move to the GPU
or run from system RAM on the CPU?
What should the GPU cache evict?

For small-batch decode, this work can be memory-bound. The system may spend more time moving expert weights into compute units than performing arithmetic on them.

FreeToken is designed around that data-movement cost.

25. When is MoE a good choice?

MoE is attractive when:

• you want much larger total parameter capacity;
• only a smaller active path is needed for each token;
• compute per token matters;
• you can afford storing the full expert pool;
• your training/serving stack can handle routing and communication.

A dense model may be better when:

• the model comfortably fits target hardware;
• deployment simplicity matters;
• latency predictability matters;
• system RAM is limited;
• CPU/GPU communication is slow;
• training complexity is not worth the sparse-compute benefit.

MoE trades deployment simplicity and predictable data movement for a larger stored capacity at a smaller active-compute cost. That sparsity is the opportunity FreeToken exploits.

Part 3 — How FreeToken works

26. The main decode bottleneck: moving weights

LLM inference is not always limited by raw arithmetic. Many small-batch decode workloads are limited by moving weights.

Suppose an expert's weights are already inside GPU VRAM.

The GPU can read local VRAM extremely quickly.

Modern high-end consumer GPUs can have VRAM bandwidth around the terabyte-per-second range.

If the expert is instead in system RAM and FreeToken chooses GPU execution, its weights must travel:

That transfer path may be far slower than reading weights already in VRAM. FreeToken does not use it for every cache miss; for some misses, the weights stay in system RAM and the CPU executes the expert there.

The FreeToken paper gives a useful comparison for consumer hardware:

So an expert already in VRAM can be dramatically easier to consume.

This leads to the central insight:

In sparse MoE decode, moving expert weights can cost more time than performing the arithmetic on those weights.

That is why expert placement, caching, and bandwidth scheduling matter so much.

27. CPU, RAM, GPU, VRAM, and PCIe

FreeToken treats the full personal computer as one inference platform.

FreeToken uses two processors, two memory pools, and one connection between them.

CPU

The general-purpose processor.

It can execute expert computation directly from system RAM.

System RAM / DRAM

The large memory attached to the CPU.

RAM is typically much larger and cheaper per GB than GPU VRAM.

GPU and GPU VRAM

The GPU is the parallel processor. GPU VRAM is the memory attached directly to it. VRAM is much faster for GPU computation than system RAM, but often too small for a huge MoE model.

RAM and VRAM store weights; they do not execute the model. CPU cores execute weights read from system RAM. GPU cores execute weights read from VRAM.

PCIe

PCI Express is the connection over which a discrete GPU communicates with host memory.

System RAM
    ↕
   PCIe
    ↕
GPU VRAM

A simple analogy:

If the expert is already on the desk:

fast

If a new expert must be fetched over the road for every step:

data movement becomes expensive

28. FreeToken is a runtime, not a model

A supported MoE model already contains:

• its trained parameters;
• MoE experts;
• routers;
• attention layers;
• architecture.

FreeToken does not train those routers.

FreeToken does not decide what the expert "knows."

FreeToken receives the router's expert selection and decides how to serve the computation efficiently.

FreeToken is especially designed for large MoE models whose full routed-expert pool is larger than GPU VRAM.

29. Where do all the parameters live?

This answers the original "8 GB GPU" puzzle.

FreeToken does not put the entire model into GPU VRAM.

Its offload design uses a two-level expert-memory hierarchy.

The model does not fit in VRAM. Its working set is distributed across the machine.

30. Why "35B model on an 8 GB GPU" is easy to misread

Take Qwen3.6-35B-A3B.

Section 2 calculated roughly 70 GB at 16 bits or 17.5 GB at a theoretical 4 bits. Neither fits into 8 GB of VRAM.

But the paper's laptop is not:

8 GB total memory

It has additional system RAM:

8 GB GPU VRAM

+

32 GiB system RAM

A GiB is a binary memory unit and is slightly larger than a GB. The distinction does not change the placement idea.

The routed-expert pool therefore lives in system RAM, while recently used expert weights and non-expert model components use VRAM. For some GPU-cache misses, the weights stay in system RAM and the CPU executes the selected expert. The model is distributed across the machine; it is not somehow compressed into 8 GB.

31. Decode: what happens when the router selects experts?

Consider one token at one MoE layer.

Use a toy layer that selects 12 experts so the cache split is easy to see. Real models can select different numbers.

FreeToken checks the GPU expert cache.

Suppose eight of the twelve selected experts already have their weights in VRAM. The other four do not:

The interesting problem is the four misses.

Only cache misses require a CPU-or-GPU decision. On a cache hit, the selected expert's weights are already in VRAM, so the GPU executes that expert without another weight transfer.

32. A cache miss has two possible execution paths

If an expert is in system RAM but not GPU VRAM, there are two basic choices.

Option A — Copy it to the GPU

Benefits:

• GPU computation is fast;
• the expert remains cached;
• a future token may hit it.

Cost:

• expert weights must cross PCIe now.

Option B — Compute it on the CPU

Benefits:

• no GPU transfer needed;
• weights are already in host memory.

Cost:

• CPU/host memory bandwidth is much lower than GPU VRAM bandwidth;
• the expert does not automatically become a future GPU-cache hit.

A fixed runtime might always favor one path.

FreeToken chooses between the two paths for the current set of cache misses.

Why use both? Assigning every cache miss to direct CPU execution would forgo available GPU capacity for those misses and could overload the host-memory path. Copying every cache-miss expert to VRAM could overload PCIe.

FreeToken assigns each cache-miss expert to one of the two paths. In one path, its weights stay in RAM and the CPU executes it. In the other, its weights are copied into VRAM and the GPU executes it. The two branches can run at the same time.

33. CPU execution and PCIe transfer share host-memory bandwidth

It may look as though these are independent resources:

Path 1
RAM → PCIe → GPU

Path 2
RAM → CPU

But both paths begin by reading expert weights from the same host memory subsystem.

Therefore they compete for the machine's host-memory bandwidth.

You cannot simply assume:

CPU bandwidth
+
PCIe bandwidth
=
fully independent throughput

FreeToken models this contention.

The paper's decode design uses bandwidth measured on the actual machine to choose how each cache miss is handled.

34. FreeToken's q* policy

The decision applies only to experts selected by the router whose weights are not already in GPU VRAM. Their weights are not missing from the machine; they still exist in system RAM.

Call the number of these GPU-cache misses m. In the earlier example, the router selected 12 experts, eight were already in VRAM, and four were not. Therefore:

m = 4

Each of those four experts begins in system RAM. FreeToken has two choices for each one:

CPU path
leave the weights in system RAM
→ CPU reads them and executes the expert

transfer-then-GPU path
copy the weights from system RAM across PCIe into VRAM
→ GPU executes the expert

The two paths run at the same time.

If FreeToken copies too many cache-miss expert weights, PCIe transfer takes longer and the GPU branch finishes last. If it assigns too many cache misses to direct CPU execution, the CPU has more expert weights to read and process, so the CPU branch finishes last. The layer cannot combine the expert results until both branches finish.

FreeToken therefore divides the m misses so the two branches should finish at roughly the same time. It measures both paths on the actual machine at startup:

Fast PCIe compared with host memory
→ copy more cache-miss expert weights to VRAM
→ execute them on the GPU

Slow PCIe compared with host memory
→ leave more cache-miss expert weights in system RAM
→ execute them on the CPU

The value q* estimates how many GPU-cache misses should use the transfer-then-GPU path. The remaining misses use direct CPU execution.

Let:

m
=
number of unique selected experts absent from VRAM
for this step/layer

B_PCIe
=
measured host-to-GPU expert-transfer bandwidth

B_Host
=
measured effective host-side expert bandwidth

FreeToken uses the approximate policy:

q* = m × B_PCIe / B_Host

q* tells the runtime roughly how many cache-miss expert weights should cross PCIe and enter the GPU cache before GPU execution. For the remaining misses, the weights stay in system RAM and the CPU executes the experts there.

Why does this ratio balance the work? Let one expert contain S bytes. Copying q experts to the GPU takes approximately:

GPU-fill time ≈ qS / B_PCIe

The paper approximates the exposed time of this transfer-dominated GPU branch with the weight-copy time. GPU expert evaluation is therefore not a separate term in this bandwidth model.

The PCIe transfer is already reading from host memory, so the CPU branch receives only the remaining host bandwidth. That residual cannot be negative:

remaining CPU bandwidth
≈ max(B_Host - B_PCIe, 0)

When B_Host is greater than B_PCIe, the CPU time is approximated as:

CPU time ≈ (m - q)S / (B_Host - B_PCIe)

Setting these two times approximately equal and solving for q produces:

q* ≈ m × B_PCIe / B_Host

If PCIe can consume all available host bandwidth, no residual bandwidth remains for concurrent CPU expert execution. In that case, q* is clamped to m, so every miss takes the transfer-then-GPU path.

If m is zero, there is nothing to schedule. Otherwise, the runtime rounds and clamps the result between one and m. Keeping at least one fill lets the GPU cache continue warming.

Simple example from the paper

Suppose:

m = 4 experts whose weights are absent from VRAM

B_PCIe : B_Host
≈
1 : 4

Then:

q*
=
4 × 1/4
=
1

So:

At the same time:

8 cache-hit experts
→ GPU

Then FreeToken combines the CPU and GPU partial outputs.

This preserves the model's exact selected-expert computation.

FreeToken does not skip low-scoring experts or approximate the MoE output in this path.

It changes where computation happens, not which experts the router chose.

35. Why different machines need different CPU/GPU splits

FreeToken does not assume that theoretical hardware specifications are enough. It measures actual bandwidth using the model's expert-tensor shapes.

Two measured consumer machines in the paper have very different bandwidth ratios:

RTX 5090 desktop: balancing branch completion times

This is the paper's gaming desktop: an RTX 5090 with 32 GB of VRAM, 192 GiB of DDR5 system RAM, and a Ryzen 9 9950X3D. It is the personal machine used to demonstrate that the 284B-total DeepSeek-V4-Flash model can be served without fitting in VRAM.

Use a hypothetical decode step with 10 unique GPU-cache misses. Such a count could occur in a toy model or across a small batch of tokens. For this desktop:

B_PCIe / B_Host
≈
49.0 / 53.8
≈
0.91

If 10 experts miss:

q*
≈
10 × 0.91
≈
9

So the machine should often copy most cache-miss expert weights to VRAM for GPU execution.

The split is not intended to give the CPU and GPU equal numbers of experts. It balances their estimated completion times: the decode step waits for both branches, so q* tries to keep either branch from becoming the slower one.

Conceptually:

RTX 4060 laptop: slower PCIe path

For the laptop:

B_PCIe / B_Host
≈
11.8 / 47.5
≈
0.25

For 10 misses:

q*
≈
10 × 0.25
≈
2–3

So:

The laptop should use the CPU much more heavily.

This is counterintuitive if you look only at GPU names.

The correct policy depends on the sustained PCIe and host-memory bandwidth available to that workload. Those values are affected by the link, memory channels, CPU kernel, and machine layout.

These two measurements sit at opposite ends of the host-to-PCIe balance.

Why FreeToken profiles the machine

At startup, FreeToken calibrates how the machine moves and processes the model's expert tensors. It uses those measurements for runtime scheduling instead of guessing from product specifications.

36. Why fixed expert placement performs poorly

Dynamic MoE routing changes every token.

Suppose at one moment:

Token t
→ E3 E7 E9 E12

next:

Token t+1
→ E3 E7 E22 E41

next:

Token t+2
→ E3 E17 E41 E62

A placement chosen when the model loads cannot perfectly follow this.

The FreeToken paper contrasts its policy with two broad baseline styles:

llama.cpp-style static placement

The paper describes llama.cpp as assigning MoE tensors to devices at model load.

That placement is relatively static.

KTransformers-style hot subset

The paper describes KTransformers as keeping a "hot" subset of experts in GPU memory and executing the rest on CPU.

That is more MoE-aware, but it still does not dynamically split every step's residual misses according to current bandwidth in the same way.

FreeToken

FreeToken uses a shared GPU expert cache. At each layer and decode step, it checks which experts the router selected.

For some cache misses, the weights remain in system RAM and the CPU executes the expert. For others, the weights cross PCIe into the GPU cache before GPU execution. The split uses measured host-memory and PCIe bandwidth instead of a placement fixed when the model loads.

37. Why the GPU uses an LRU expert cache

Routing is dynamic, but adjacent tokens often reuse experts.

Continue with the earlier 12-expert toy example. This is only an illustration of reuse; DeepSeek-V4-Flash selects six routed experts for each token.

Imagine:

Eight experts overlap.

That is temporal locality:

Recently used experts often have an elevated chance of being used again soon.

So FreeToken uses a shared GPU expert cache with an LRU-like policy.

LRU means Least Recently Used.

recently used
→ keep

not used for a while
→ good eviction candidate

Conceptually:

GPU expert cache

E3
E7
E9
E12
E17
E24
E5
E22
E48
...

Measured miss-rate difference

At the RTX 5090 cache capacities evaluated in the paper:

Those numbers are on identical routing traces at equal cache capacity in the paper's comparison.

This matters because every avoided miss saves a costly host-memory path.

38. Prefill needs a different strategy: full-layer double buffering

The q* split described above is for decode. Long prefill behaves differently because thousands of prompt tokens can collectively select nearly every expert in a layer.

During decode:

one token
→ sparse expert set
→ cache is useful

During long prefill:

thousands of tokens
→ union of routes touches nearly all experts

FreeToken therefore does not rely only on on-demand expert fetches during prefill.

It uses full-layer double buffering.

Suppose the GPU is computing MoE layer L.

At the same time, PCIe loads all experts for layer L+1.

Two buffers alternate roles: while one supplies the current layer to the GPU, PCIe fills the other with the next layer. The key is that FreeToken loads the complete next-layer expert set before that layer's individual token routes are known.

Because long prefill is expected to touch most experts anyway, this allows transfer to start early and run continuously in the background.

Why double buffering helps

Without overlap, transfer and computation happen one after another. With double buffering, transfer of the next layer runs while the GPU computes the current layer.

The paper reports that on its RTX 5090 Qwen3.6 experiment, disabling the second buffer reduced throughput by:

With overlap, processing an 8,192-token chunk took about as long as streaming the 64.4 GB expert pool once over the measured PCIe link. This indicates that transfer overlapped much of the expert computation.

If VRAM cannot spare two complete layer buffers, FreeToken falls back to on-demand loading instead of oversubscribing memory.

39. Why coding agents invalidate cached state

Expert movement is only one source of latency. Coding agents also send repeated requests whose long contexts change between turns.

A normal one-shot prompt might look like:

User
  ↓
Model
  ↓
Answer

A coding agent has a much more dynamic context:

Agent frameworks often modify older context to save space.

Examples described in the FreeToken paper include patterns such as:

• removing old thinking blocks;
• replacing older tool outputs with placeholders;
• retaining only a recent observation window.

The next diagram shows what happens when a later turn removes an old tool output and appends a new suffix:

A request-state checkpoint saves reusable computation for a particular request prefix. Any checkpoint taken after the edited region is no longer valid.

If checkpoints are poorly placed, the engine may have to reprocess thousands of old tokens.

On consumer GPUs, that can cause very large TTFT spikes.

40. How semantic anchors preserve reusable work

FreeToken handles full-attention KV state and recurrent state differently. A prefix tree finds the longest exact prompt prefix whose KV state can be reused.

Some hybrid models also have recurrent layers that compress the earlier sequence into one evolving state. A saved copy of all recurrent-layer state can be large, so the runtime can keep only a limited number of recurrent-state checkpoints.

FreeToken places those limited recurrent-state checkpoints at meaningful boundaries instead of only at arbitrary token positions such as:

token 2000
token 4000
token 6000

Useful candidate boundaries include:

Meaningful boundaries help because agent frameworks tend to edit entire semantic blocks. The diagram shows candidate checkpoint locations; the limited cache does not necessarily retain every old boundary.

Suppose the tool output disappears:

FreeToken can restore from the deepest checkpoint whose prefix is still valid.

Then:

reuse preserved prefix
       ↓
re-prefill only changed/new suffix

The mechanisms are different:

Expert cache
=
model weights

KV cache
=
request attention state

Recurrent-state checkpoint
=
compressed state for recurrent layers

Semantic anchor
=
a smart location at which to save
that reusable request state

These are different mechanisms.

41. Why time to first token matters

For an interactive coding agent, decode speed is not the only metric.

A coding-agent turn can look like this:

agent calls tool
      ↓
context changes
      ↓
next model request
      ↓
2 minutes before first token

Even if subsequent decode is fast, the agent feels broken.

The paper reports worst-turn TTFT across its evaluated cells:

Some turns can take so long to produce the first token that the agent application stops waiting and reports a timeout. Faster token generation after that point does not help because the request has already failed.

Semantic prefix/state reuse is therefore not only a speed optimization. It can prevent a slow turn from becoming a failed agent request.

42. GPU memory is not static during a long conversation

VRAM is needed for:

non-expert model weights

runtime buffers

KV / recurrent state

expert cache

At the beginning of a conversation:

short context
      ↓
small KV/state requirement
      ↓
more VRAM available
for expert cache

Conceptually:

After many turns:

long context
      ↓
larger KV/state demand

Now:

On a personal computer, other applications may also consume VRAM:

• browser;
• desktop compositor;
• game;
• video workload.

So a memory split chosen at startup may become wrong later.

FreeToken can rebuild the expert cache at scheduler-safe points using a revised memory budget, without reloading the complete host expert pool or restarting the whole engine.

This is safe because the host expert pool remains the source of truth:

Host expert pool
=
source of truth

GPU expert cache
=
temporary performance optimization

Changing the cache changes speed, not model correctness.

43. Startup also matters on personal hardware

A datacenter server may stay alive for weeks.

A local inference engine may be:

start model
use it
close it
start another model

Loading a huge expert pool from solid-state storage can itself be expensive.

The paper gives an illustrative DeepSeek-V4-Flash FP4 example:

~140 GB expert pool

solid-state storage speed
~7 GB/s

Even ideal sequential reading takes roughly:

140 / 7
≈ 20 seconds

before GPU warmup and other work.

FreeToken loads expert tensors directly into the host-memory arrangement used during inference, avoiding another full copy. It then keeps those memory pages resident so serving does not have to fetch them from storage again.

For local inference, startup time matters alongside steady-state throughput.

44. What performance does the paper actually report?

The evaluation compares FreeToken with llama.cpp, Ollama, KTransformers, and MoE-Infinity where each engine supports the tested model and configuration.

In the paper's cross-hardware coding-agent study, FreeToken leads the strongest supported baseline by about 1.3 to 2.1 times across five consumer systems. The exact result changes with the model, prompt, machine, software version, and baseline support.

This supports a limited conclusion: FreeToken's complete runtime outperformed the supported baselines in those tested configurations. A separate replay of identical routing traces shows that its LRU cache also produced fewer expert misses than the compared placement policies at equal cache capacity. Neither result is a universal FreeToken-versus-Ollama promise.

45. How applications use FreeToken

FreeToken runs as a local server rather than only as a benchmark program. Applications can send it requests through familiar OpenAI- or Anthropic-compatible APIs, so they do not need to understand expert caching or CPU/GPU scheduling. The runtime hides those mechanisms behind the same kind of request-and-response interface used with hosted models.

46. What FreeToken adds

FreeToken builds on established inference techniques such as prefix reuse and optimized GPU kernels.

Its main contribution is coordinating all the hardware in one machine for large MoE models. The complete expert-weight pool stays in system RAM, while a recently used subset is cached in VRAM.

During decode, selected expert weights absent from the VRAM cache take one of two paths. They remain in system RAM for direct CPU execution, or they cross PCIe into VRAM before GPU execution. The split follows the machine's measured host-memory and PCIe bandwidth.

That distinction matters: many individual operations already existed, but FreeToken assembles them into a runtime designed for a model whose total weights do not fit in GPU memory.

47. Optional implementation detail: how dynamic cache control avoids CPU stalls

An expert cache changes at every step. The selected experts determine the misses and evictions. The runtime also decides how many misses become GPU fills.

If the CPU had to synchronize with the GPU at every MoE layer to control these decisions, that synchronization itself could hurt latency.

The FreeToken paper describes keeping routing-dependent cache control on the GPU using device-resident data structures compatible with CUDA Graph execution.

A CUDA Graph lets a GPU runtime capture a repeated execution structure and replay it with lower CPU-launch overhead.

Dynamic MoE routing normally fights against static graph capture:

token 1
different experts

token 2
different experts

token 3
different number of misses

FreeToken represents changing cache-control information as data inside a fixed-shape captured execution structure.

This is an implementation detail, but it helps explain why a seemingly simple "just cache experts" idea requires careful systems engineering.

48. What FreeToken does not solve

48.1 It does not create memory from nowhere

An 8 GB GPU paired with only 8 GB of system RAM cannot run an arbitrary 284B model through this design.

FreeToken still needs enough system RAM for the resident expert pool and enough storage space for the model files. Solid-state-drive capacity cannot replace the required RAM. Only a changing expert subset is cached in VRAM, so "runs on an 8 GB GPU" does not mean "requires only 8 GB of total memory."

48.2 Quantization is still important

Lower-precision weights help make the host-resident model practical.

48.3 PCIe and RAM bandwidth still matter

Slow host memory, a slow PCIe link, or slow CPU kernels will limit performance.

FreeToken schedules those resources; it does not remove their physical limits.

48.4 MoE is required for the main expert-offload advantage

A normal dense model does not have:

256 experts
choose 6

There is no sparse expert pool for this policy to exploit in the same way.

48.5 It does not make inference literally free

The name does not mean zero electricity, hardware cost, latency, or computation.

The practical benefit is that users can run supported open-weight models on hardware they own instead of paying a hosted API for every token. The name should not be interpreted as a claim about zero cost.

49. End-to-end example on a memory-constrained machine

Assume the model's complete expert pool fits in system RAM but not in GPU VRAM. Also assume the non-expert model components and runtime state fit in VRAM while leaving some space for an expert cache. The MoE router selects only a few experts for each token.

Step 1 — Load the model.

The complete routed-expert pool is loaded into host memory.

solid-state storage
 ↓
System RAM
 ↓
complete expert pool

Non-expert GPU-resident components and runtime state occupy VRAM.

The remaining space becomes an expert cache.

Step 2 — User sends a prompt.

"Explain PostgreSQL indexes."

Tokenizer creates token IDs.

Step 3 — Prefill.

For a fresh request with no reusable prefix state, all existing prompt tokens move through the model. If an earlier prefix is still valid and cached, FreeToken can instead reuse that state and prefill only the changed or new suffix, as sections 39 and 40 explain.

Across many prompt tokens, routing may collectively touch most experts.

FreeToken pipelines:

load next layer's experts over PCIe
              ∥
compute current layer on GPU

This full-layer double buffering is used when VRAM can hold both layer buffers. If it cannot, FreeToken falls back to on-demand expert loading.

Step 4 — Request state is built.

KV and/or recurrent state is retained for future generation.

Step 5 — Decode starts.

The final logits produced during prefill select the first output token. The first decode step then feeds that generated token through the model to produce the second output token. Each later decode step feeds the previously generated token through the model to produce the next one.

At an MoE layer:

Attention
   ↓
Router
   ↓
select this token's routed experts

Step 6 — FreeToken checks expert residency.

Some selected experts may already be in the GPU cache.

hits
→ GPU immediately

Others miss.

Step 7 — q scheduling.*

The runtime compares measured PCIe bandwidth with host-side expert bandwidth. For a cache miss assigned to the GPU path, the expert weights cross PCIe, enter the GPU cache, and are then executed by the GPU.

For a miss assigned to the CPU path, the expert weights stay in system RAM and the CPU executes the expert there. Slower PCIe therefore results in fewer cache misses using the GPU path.

Step 8 — CPU and GPU run concurrently.

GPU:
cached experts
+
newly filled experts

CPU:
remaining misses

Step 9 — Exact merge.

Their partial outputs are combined.

GPU result
   +
CPU result
   ↓
MoE output

Step 10 — Move to next layer.

The next MoE layer receives the hidden state, and its router makes a new expert selection. The same cache check and CPU/GPU split happen again.

Step 11 — Final vocabulary scores.

After the final model layers:

hidden state
   ↓
vocabulary logits
   ↓
next output token

Step 12 — Next token.

Routing happens again.

Recent expert reuse creates cache hits.

This continues token after token.

That is how a model whose total weights do not fit in VRAM can still generate tokens without changing the router's selected experts.

50. End-to-end system diagram

This view brings the execution path and memory placement together. Follow the green path for cache hits. On a miss, the runtime reads from the complete expert pool in system RAM.

The value (q^*) determines how many GPU-cache misses copy expert weights across PCIe for GPU execution. For the remaining misses, the weights stay in system RAM for CPU execution. The two partial results meet at an exact merge.

The complete expert pool remains the source of truth in system RAM. VRAM holds a changing expert cache. FreeToken changes where selected experts execute; it does not change which experts the model router selected.

51. Common questions and misconceptions

Does every LLM have an MoE router?

No.

A dense model has no expert selector because there is only the normal FFN path.

Dense:
Attention → FFN

MoE:
Attention → Router → selected experts

Does the router decide the next word?

No.

It only chooses which FFN experts process the current hidden state.

The final model output layer produces next-token scores. The decoding strategy chooses a token from those scores.

Is an expert a separate full LLM?

No.

An expert is usually an FFN block inside one MoE layer.

Does one expert handle one topic?

Not necessarily.

Expert specialization emerges during training and can be much more abstract than human topic categories.

Does routing happen once per question?

No.

Routing happens per token at each MoE layer.

If only 13B parameters are active, can we delete the other 271B?

No.

The next token may choose different experts.

Do active parameters all need to fit in VRAM simultaneously?

"Active parameters" describes the computational path, not necessarily one single simultaneously resident buffer.

FreeToken schedules experts layer by layer and uses a cache. It does not simply load one fixed 13B subset at the start of the query.

52. Glossary

Activation.

A temporary numerical value produced while processing a request.

Active parameters.

Parameters that participate in the current token's computational path.

Attention.

A Transformer mechanism allowing token positions to gather information from other positions.

Attention head.

One set of attention projections operating in parallel with other heads.

Attention score.

A request-specific relevance score between positions.

Bandwidth.

How much data can move through a memory/link per second, commonly measured in GB/s.

CUDA Graph.

A CUDA mechanism for capturing and replaying GPU execution structures with reduced launch overhead.

Checkpoint.

A saved point from which computation can be reused. In this article, a request-state checkpoint stores computation for a particular request prefix.

Decode.

The autoregressive phase where the model generates new tokens.

Dense model.

A model in which essentially the same full parameterized network is used for every token rather than conditionally selecting expert blocks.

DRAM.

Dynamic Random Access Memory. In this article, usually ordinary system RAM attached to the CPU.

Embedding.

A numerical vector used as a token's initial learned representation.

Expert.

In many MoE Transformers, an independently parameterized FFN selected by a router.

Expert cache.

GPU memory used to keep recently useful expert weights resident.

Feed-Forward Network (FFN).

A learned non-linear transformation applied to each token position in a Transformer block.

FP8 / FP4.

Low-precision floating-point representations used to reduce model memory/bandwidth.

Forward pass.

The computation that moves an input through the model to produce an output.

GB / GiB.

GB is a decimal unit of one billion bytes. GiB is a binary unit of 1,073,741,824 bytes.

Hidden state.

The current internal vector representation of a token at a particular layer.

Host memory.

System RAM attached to the CPU.

Inference.

Using an already-trained model to produce an answer or prediction.

KV cache.

Stored Key/Value attention tensors for previous tokens, allowing generation to reuse past attention state.

Logit.

An unnormalized score for a possible output token.

LRU.

Least Recently Used. A cache eviction policy that preferentially removes entries not used recently.

Matrix.

A rectangular grid of numbers. Neural networks use matrix operations to transform vectors with learned weights.

Mixture of Experts (MoE).

A model architecture with many expert parameter blocks in which only a subset is selected for a token.

Parameter.

A learned value stored in a model.

PCIe.

PCI Express, the host-to-device interconnect used by discrete GPUs in PCs.

Prefill.

The phase where an inference engine processes the existing input prompt before producing the first new token.

Quantization.

Representing weights at lower precision to reduce storage and bandwidth requirements.

Query / Key / Value.

Vectors used in attention. Queries are matched with Keys, and Values carry information that is aggregated.

Recurrent state.

A compact state maintained by certain recurrent/hybrid sequence layers to summarize prior context.

Router.

A learned MoE component that scores experts for a hidden state and selects which ones execute.

Sparse activation.

Using only selected parameter blocks for a particular input/token.

System RAM.

Main memory used by the CPU.

Temporal locality.

The tendency for recently used items—in this case experts—to be used again soon.

Tensor.

A numerical array. Vectors and matrices are one- and two-dimensional tensors; model software also uses tensors with more dimensions.

Token.

A tokenizer-produced unit of text processed or generated by the model.

Transformer.

The neural architecture family used by most modern LLMs.

TTFT.

Time To First Token: elapsed time between a request and the first generated token.

Vector.

An ordered list of numbers representing model state.

VRAM.

Memory attached directly to the GPU.

53. Sources and verification notes

The neural-network sections intentionally simplify some mathematics to build intuition first. The FreeToken implementation details and benchmark numbers in this article are based on the current paper and repository as of August 2026.

1. FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution Shuo Yang, Xiaoze Fan, Melissa Pan, Haocheng Xi, Zhe Wang, Shanlin Sun, Kurt Keutzer, Song Han, Matei Zaharia, Chenfeng Xu, Ion Stoica. https://arxiv.org/abs/2608.16157 PDF: https://arxiv.org/pdf/2608.16157

2. FreeToken source repository https://github.com/FlashML-org/FreeToken

3. FreeToken installation requirements https://github.com/FlashML-org/FreeToken/blob/main/docs/install.md

4. FreeToken Quick Start / API compatibility https://github.com/FlashML-org/FreeToken/blob/main/docs/quickstart.md

5. FreeToken CLI reference / hardware bandwidth calibration https://github.com/FlashML-org/FreeToken/blob/main/docs/cli.md

6. FreeToken supported-model and backend documentation https://github.com/FlashML-org/FreeToken/blob/main/docs/models.md

7. Qwen3.6-35B-A3B official model card https://huggingface.co/Qwen/Qwen3.6-35B-A3B

8. DeepSeek-V4-Flash official model card https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash

9. Attention Is All You Need Vaswani et al. https://arxiv.org/abs/1706.03762

10. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity Fedus, Zoph, Shazeer. https://arxiv.org/abs/2101.03961

11. Mixtral of Experts Mistral AI. https://arxiv.org/abs/2401.04088

12. Transformer Feed-Forward Layers Are Key-Value Memories Geva et al. https://arxiv.org/abs/2012.14913

Final takeaway

A claim such as "a 284B model runs on a 32 GB GPU" can sound as though the complete model has been squeezed into VRAM. It has not.

MoE makes each token's active computation sparse. FreeToken keeps the full expert-weight pool in system RAM and caches a changing subset in VRAM.

During decode, a selected expert whose weights are absent from VRAM takes one of two paths. Its weights remain in system RAM for CPU execution, or they cross PCIe into VRAM for GPU execution. The whole machine—not the GPU alone—serves a model whose complete parameter pool would never fit in VRAM.