- FreeToken 290b model: Understand the edge-serving approach behind frontier-scale sparse models.
- Core design: Combine GPU expert caching, CPU execution, and PCIe transfers.
- Best setup: Measure host and PCIe bandwidth before tuning the runtime.
- Main advantage: Reduce decode misses and hide prefill transfers behind computation.
- Hardware scope: Support ranges from an 8 GB laptop GPU to workstation-class hardware.
FreeToken 290b model: What the System Does
FreeToken is an edge-native serving system for large Mixture-of-Experts models. The FreeToken 290b model search term generally points toward the paper’s frontier-scale serving context, although the documented demonstration uses DeepSeek-V4-Flash at 284B total parameters, with about 13B active parameters per token. The important distinction is that sparse activation reduces computation, but the complete expert pool still requires substantial host memory and storage.
Instead of requiring the entire model to fit in GPU memory, FreeToken treats the GPU, CPU, host memory, and PCIe interconnect as one inference platform. Non-expert weights remain on the GPU, while the full routed-expert pool stays in host memory. A dynamic GPU cache stores the expert-layer pairs that are most useful for the current workload.
The system targets agentic sessions, where long contexts and repeated tool calls create both prefill and decode pressure. Its design addresses three recurring problems:
- Prefill transfer cost: Large expert pools must move through the CPU-to-GPU link.
- Decode cache misses: Each token can request experts that are not currently resident.
- Changing resources: Browsers, games, desktop applications, and growing KV caches can reduce available VRAM.
The research paper, “FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution”, describes the architecture, implementation, and evaluation published on August 24, 2026.
Sparse Compute
MoE routing activates only a small expert subset for each token, making frontier-scale computation more practical on local hardware.
Elastic Cache
A shared LRU cache tracks recently routed experts and can be resized as the available GPU memory budget changes.
Hybrid Execution
Cache misses can be transferred to the GPU or executed directly on the CPU according to measured bandwidth.
The total parameter count does not equal the active compute per token. FreeToken makes sparse computation usable while managing the larger memory footprint of the complete expert pool.
How FreeToken Handles Prefill and Decode
FreeToken separates inference into two phases because prefill and decode have different bottlenecks. Prefill processes the prompt and determines time to first token, while decode generates tokens one at a time and is more sensitive to expert locality and bandwidth balance.
During prefill, the system uses full-layer double buffering. While the GPU computes one layer, the next layer’s expert weights stream over PCIe into a second buffer. This overlaps transfer and computation rather than waiting for every expert movement to finish before continuing.
Agentic workloads also edit conversation history frequently. Thinking blocks, tool calls, tool outputs, and conversation turns can be removed or replaced between requests. FreeToken places recurrent-state checkpoints at these semantic boundaries so that a surviving prefix can be reused. Only the changed suffix needs to be processed again.
During decode, routed experts are checked against the shared GPU cache. Cache hits run directly on the GPU. Misses are divided between GPU cache fills and CPU execution using a bandwidth-derived ratio.
| Inference phase | Main pressure | FreeToken response | Practical result |
|---|---|---|---|
| Prefill | Expert transfer and prompt recomputation | Full-layer double buffering and semantic checkpoints | Lower transfer exposure and less repeated work |
| Decode | Expert misses and limited host bandwidth | LRU caching plus CPU-GPU miss splitting | More balanced per-token execution |
| Multi-turn agent use | Context edits after tools or reasoning | Prefix reuse at semantic anchors | Shorter re-prefill for preserved history |
| Runtime changes | Fluctuating VRAM and KV cache demand | Elastic expert-cache resizing | No engine restart for every memory adjustment |
The miss policy uses two measured values:
- Bₚ: Pinned host-to-device expert-transfer bandwidth.
- Bₕ: Effective host-side bandwidth available to the CPU expert kernel.
For a step with m missing experts, the approximate cache-fill count is:
q* ≈ m × Bₚ / Bₕ
A higher PCIe share favors more GPU cache fills. A stronger CPU path leaves more misses for direct CPU execution. This is a runtime policy rather than a fixed hardware tier list.
The optimal CPU-GPU split depends on measured bandwidth, memory layout, CPU behavior, and the deployed PCIe link. Hardware specification sheets are useful for planning, but FreeToken’s policy should be based on runtime measurements.
FreeToken 290b model Hardware and Performance Guide
The documented evaluation covers several consumer and workstation configurations. Results vary by model, quantization, host memory, PCIe generation, and workload, so the figures below should be treated as reported reference points rather than universal guarantees.
The system serves Qwen3.6-35B-A3B, DeepSeek-V4-Flash, and a workstation-class GLM-5.2 demonstration. The largest example uses a 753B-parameter MoE model on a single RTX PRO 6000 Blackwell with 96 GB of memory.
| Model or tier | Total parameters | Active parameters | Reported deployment |
|---|---|---|---|
| Qwen3.6-35B-A3B | 35B | 3B class | Consumer GPUs, including laptop hardware |
| DeepSeek-V4-Flash | 284B | 13B | RTX 3090, 4090, 5090-class systems |
| GLM-5.2 | 753B | 40B | RTX PRO 6000 Blackwell, 96 GB |
| FreeToken paper scope | 20+ MoE models | Model-dependent | 8 GB laptop GPU to workstation GPU |
On an RTX 5090, the paper reports approximately 77–83 tokens per second for Qwen3.6 and 22–25 tokens per second for DeepSeek-V4-Flash across the tested workloads. The reported result is between 1.5× and 2.3× the strongest baseline depending on model and scenario.
For the cross-hardware Qwen3.6 coding workload, FreeToken led the strongest baseline by approximately:
| Hardware configuration | Reported advantage |
|---|---|
| RTX 3090 | 1.3× |
| RTX 4090 | 1.3× |
| RTX 5090 server | 1.9× |
| RTX 5090 desktop | 2.1× |
| RTX 4060 laptop | 1.8× |
| RTX PRO 6000 with GLM-5.2 | 2.0× versus llama.cpp |
The RTX 4060 laptop example is notable because the 8 GB system used an NVFP4 build and reached 39.3 tokens per second, reported as 92% of the RTX 4090 rate in that comparison. The result demonstrates why host bandwidth and quantization can matter as much as GPU model names.
FreeToken’s strongest gains come from coordination. A smaller GPU can remain competitive when the runtime uses its PCIe link, CPU bandwidth, cache capacity, and model format efficiently.
Step-by-Step FreeToken Setup Workflow
A practical deployment should begin with the memory hierarchy, not with cache-size guesses. The host-resident expert pool remains the source of truth, so GPU cache capacity affects speed and latency rather than model correctness.
Identify the Model and Expert Format
Confirm the model’s total parameters, active parameters, precision, expert count, and checkpoint layout. FreeToken’s FTW format normalizes expert banks into a runtime-friendly layer–expert structure and can avoid tensor discovery and repacking at startup.
Measure Host and PCIe Bandwidth
Profile pinned expert-transfer bandwidth and effective CPU expert-processing bandwidth on the target machine. Use these values to estimate the cache-fill portion of each decode miss rather than relying only on advertised bandwidth.
Reserve the Memory Budget
Allocate GPU memory for non-expert weights, activations, and the KV cache before assigning the remaining space to expert slots. Keep the cache elastic because long agent sessions increase KV-cache demand.
Prepare the Host Expert Pool
Load experts directly into their final host layout, then pin populated memory for DMA when the platform supports it. This avoids unnecessary page faulting and reduces startup overhead.
Warm Through Real Workloads
Start with representative prompts, tool calls, and multi-turn sessions. Let the shared LRU cache learn the active routing pattern, then evaluate decode speed, time to first token, miss rate, and tail latency.
| Setup check | Recommended action | Why it matters |
|---|---|---|
| GPU memory | Split space between KV cache and expert slots | Context growth changes the correct balance |
| Host memory | Keep the complete expert pool available | Host storage remains the correctness source |
| PCIe path | Use pinned memory where supported | DMA transfer determines cache-fill speed |
| CPU execution | Pin workers near the GPU’s NUMA node | Avoid unnecessary memory-access penalties |
| Startup format | Prefer prepacked FTW-style layouts | Reduce discovery and repacking work |
Begin with a cold-cache test and a multi-turn agent test. A single short prompt may hide the transfer, cache, and context-reuse behavior that determines real-world performance.
Limitations, Checklist, and Best Practices
FreeToken improves the serving system, but it does not eliminate the physical cost of large models. The complete expert pool may still require hundreds of gigabytes of host memory or storage. Platform support also depends on operating-system and driver behavior, especially for pinned or registered memory.
When the fast DMA path cannot be established, the runtime can fall back to a pure-CPU MoE backend. Non-expert layers remain on the GPU, while activations, routing metadata, and aggregated outputs cross the device boundary. This improves deployability but can reduce peak transfer performance.
Use the following checklist before comparing results:
Deployment Readiness Checklist:
- Confirm the model precision and complete expert-pool size
- Measure pinned PCIe transfer and CPU expert bandwidth
- Reserve VRAM for both KV cache and expert slots
- Test cold-start, single-turn, and multi-turn workloads
- Track cache miss rate, TTFT, decode speed, and tail latency
The most useful metrics are not limited to average tokens per second. For agentic workloads, long-tail TTFT can determine whether a client waits successfully or reaches its timeout threshold. The paper reports FreeToken staying below 44 seconds for worst-turn TTFT in its tested cells, while each baseline exceeded 150 seconds somewhere in the evaluation.
| Metric | What to monitor | Interpretation |
|---|---|---|
| Decode throughput | Mean tokens per second | Measures generation efficiency |
| TTFT | Mean and worst-turn latency | Captures prompt transfer and recomputation |
| Expert miss rate | Misses as a share of routed reads | Shows cache locality quality |
| Cache capacity | Percentage of expert pool resident | Connects VRAM allocation to reuse |
| Startup time | Disk load plus first response | Measures practical on-demand usability |
Evaluate the same harness, prompts, model precision, and success criteria across engines. Agent trajectories can diverge, making raw wall-clock comparisons misleading when requests do different amounts of work.
FreeToken FAQ
Q: Is the FreeToken 290b model a separate officially named model?
The supplied research identifies FreeToken as a serving system, not a standalone 290B model checkpoint. Its main documented example is DeepSeek-V4-Flash at 284B total parameters, so the 290b wording should be treated as a search label for the frontier-scale serving topic.
Q: What makes FreeToken different from static CPU-GPU placement?
Static systems decide expert placement at load time or prefill time. FreeToken uses a shared LRU cache that follows decode routing and divides unavoidable misses between PCIe cache fills and direct CPU execution.
Q: Can FreeToken run large MoE models on an 8 GB laptop GPU?
The evaluation reports an 8 GB RTX 4060 laptop configuration for Qwen3.6 using NVFP4. The full model still relies on host-resident experts, so GPU memory alone does not contain the complete checkpoint.
Q: Why are multi-turn agent workloads important?
Tool calls and context edits repeatedly trigger prefill. FreeToken uses semantic checkpoints to preserve reusable prefixes, while its expert cache tracks routing locality during decode.