- FreeToken setup guide: Treat the GPU, CPU, RAM, storage, and PCIe link as one serving system.
- Hardware fit: Sparse activation reduces compute demand, but the complete expert pool still needs host memory.
- Bandwidth policy: Measure host and PCIe bandwidth instead of relying only on specification sheets.
- Cache strategy: Use elastic GPU expert caching alongside KV-cache management for agentic workloads.
What FreeToken Requires
FreeToken is an edge-native serving system for large Mixture-of-Experts models. A practical setup begins by separating active computation from total model storage. Only a subset of experts handles each token, but the full routed-expert pool still has to remain available in host memory or another storage tier.
The reference design keeps the complete expert pool in CPU memory while non-expert weights remain on the GPU. Remaining VRAM becomes an elastic expert cache shared across MoE layers. This arrangement allows a consumer machine to serve models whose total weights exceed available VRAM, provided the host-memory and interconnect budgets are suitable.
| Resource | Setup role | What to check |
|---|---|---|
| GPU VRAM | Non-expert weights, KV cache, expert cache | Leave room for changing context length |
| Host memory | Source of truth for routed experts | Must hold the deployed expert pool |
| PCIe link | Moves missing experts to the GPU | Measure effective transfer bandwidth |
| CPU and DRAM | Executes selected misses in place | Measure bandwidth with target tensors |
| NVMe storage | Loads the host-resident pool | Use sufficient capacity and sustained read speed |
GPU Tier
Larger VRAM supports a bigger expert cache and improves decode hit rates. The RTX 5090-class configuration in the study served substantially larger working sets than an 8 GB laptop GPU.
Host Tier
Host memory stores the full routed-expert pool. Dual-channel DDR4 or DDR5 bandwidth can become the limiting factor when the CPU handles cache misses.
Link Tier
PCIe bandwidth determines how quickly missing experts reach the GPU. Laptop links, especially PCIe x8 connections, can expose more transfer latency.
Do not size the machine from active parameters alone. Sparse activation lowers per-token computation, while the complete expert pool can still exceed GPU and system-memory budgets.
FreeToken Setup Guide: Step-by-Step
Use this FreeToken setup guide as a deployment sequence rather than a single command recipe. The available reference describes the system design and evaluation, while release-specific installation details should be checked through the FreeToken project page, published in 2026.
Choose a Supported Model
Start with an MoE checkpoint whose expert representation and precision are supported by the runtime. The reference evaluation includes DeepSeek-V4-Flash, Qwen3.6-35B-A3B, and GLM-5.2 across different hardware tiers.
Reserve Host Memory
Confirm that the full routed-expert pool fits in available host memory with room for the operating system and concurrent applications. Do not treat disk capacity as a substitute for runtime memory.
Measure Bandwidth
Profile effective pinned expert-transfer bandwidth over PCIe and CPU-side expert-processing bandwidth using the deployed tensor shapes. These values determine the miss split.
Set the Runtime Budget
Reserve VRAM for non-expert weights and the KV cache first. Allocate the remaining budget to the shared expert cache, leaving enough flexibility for longer agent sessions.
Run a Cold-Cache Test
Start with a cold cache, verify the first request completes, then compare decode speed and time to first token after the cache warms through normal serving.
| Setup phase | Main decision | Validation signal |
|---|---|---|
| Model selection | Check precision and expert layout | Runtime can load the checkpoint |
| Memory reservation | Fit experts plus system overhead | No allocation or paging failure |
| Bandwidth profiling | Record host and PCIe rates | Stable measurements on target hardware |
| VRAM allocation | Balance experts against KV cache | Context growth does not exhaust VRAM |
| First request | Test cold-start behavior | Request completes without a warmup-only path |
Record bandwidth measurements for each machine. Two systems with the same GPU can produce different results when their DRAM channels, PCIe links, or concurrent workloads differ.
Prefill and Decode Configuration
FreeToken uses different tactics for the two major inference phases. Prefill processes the prompt and strongly affects time to first token. Decode generates tokens one at a time and is more sensitive to expert-cache misses and host bandwidth.
During prefill, the system uses full-layer double buffering. While the GPU computes one layer, the next layer’s experts stream over PCIe. This overlaps transfer and computation instead of exposing the entire expert movement interval as GPU idle time. If the cache cannot spare two full-layer buffers, the design falls back to on-demand loading to avoid oversubscribing VRAM.
Agentic sessions also benefit from semantic-aware state caching. Checkpoints are placed around boundaries such as thinking segments, tool calls, tool outputs, and conversation turns. When a harness edits a whole block of history, the runtime can reuse the preserved prefix and recompute only the new suffix.
| Phase | Primary bottleneck | FreeToken response |
|---|---|---|
| Prefill | Full expert movement and repeated context recomputation | Double-buffered layer loading and semantic checkpoints |
| Decode | Missing experts and limited CPU bandwidth | Shared LRU cache plus bandwidth-adaptive execution |
| Long sessions | Growing KV-cache demand | Elastic split between KV pages and expert slots |
| Restart | Loading the complete expert pool | Direct loading into final host layout |
During decode, routed experts already in the shared LRU cache execute on the GPU. Missing experts are divided between two paths:
- Cache-fill path: Transfer selected experts over PCIe, execute them on the GPU, and retain them for future reuse.
- CPU path: Execute other missing experts directly from the host-resident pool without changing GPU residency.
- Merge path: Combine GPU and CPU partial outputs while preserving exact MoE computation.
The approximate cache-fill count is:
q* ≈ m × BP / BH
Here, m is the number of missing experts, BP is measured pinned-transfer bandwidth, and BH is measured host-side expert-processing bandwidth. The runtime rounds the result and continues warming the cache even when CPU execution handles most misses.
Compare throughput on both short single-turn prompts and multi-turn agent traces. A configuration that looks strong in isolated decode may degrade when repeated prefills and context edits dominate.
Cache, Storage, and Runtime Tuning
The CPU-resident expert pool remains the source of truth, so GPU cache capacity changes performance rather than model correctness. This is important on personal computers where browsers, desktop applications, and other GPU workloads can change the available VRAM budget during a session.
FreeToken can rebuild the GPU expert cache at scheduler safe points without restarting the engine or reloading the host pool. The cache is shared across MoE layers and uses logical layer–expert identifiers, allowing residency and execution to follow the current routed working set.
| Tuning area | Recommended approach | Avoid |
|---|---|---|
| Expert cache | Let LRU follow recent routing locality | Permanently pinning a prefill-only hot set |
| KV cache | Increase with session length, but protect expert capacity | Assigning all free VRAM to context |
| Host pool | Load directly into final runtime layout | Repacking large banks at every launch |
| Pinned memory | Pin populated buffers after loading | Pinning empty buffers and faulting pages unnecessarily |
| Startup | Serve cold-cache requests through the normal path | Requiring a separate full warmup phase |
Before Serving:
- Confirm the complete routed-expert pool fits in host memory
- Measure effective PCIe transfer and CPU expert bandwidth
- Reserve VRAM for non-expert weights and growing KV cache
- Verify the selected model precision and expert layout
- Test both cold-cache startup and warmed decode
For storage, the reference design introduces the FreeToken Weight format, which normalizes expert banks into a runtime-friendly layout. Preformatted banks let the engine read aligned chunks directly into exact-size host buffers, reducing tensor discovery and repacking work during launch. If a platform cannot establish the required pinned or registered memory path, the runtime can use a pure-CPU MoE backend, trading peak transfer performance for broader deployability.
Consumer hardware is not dedicated infrastructure. Close unnecessary applications, monitor sustained temperatures, and repeat measurements after changing desktop, browser, or GPU workloads.
Validation, Troubleshooting, and FAQ
Validate a deployment with metrics that expose the real bottleneck. Decode throughput shows how efficiently the current expert working set is served, while time to first token reveals prefill transfer and recomputation costs. Tail latency matters for agent clients because a single long turn can make an otherwise acceptable setup feel unavailable.
| Symptom | Likely cause | First response |
|---|---|---|
| Slow first token | Serialized expert transfer or repeated prefill | Check double-buffer availability and prefix reuse |
| Low decode rate | Excessive cache misses | Increase expert-cache budget if KV demand allows |
| CPU saturation | Too many misses assigned to host execution | Reprofile host bandwidth and inspect the miss split |
| GPU underuse | PCIe or host-memory bottleneck | Compare measured transfer bandwidth with CPU bandwidth |
| Startup delay | Disk loading or runtime repacking | Use the prepared weight layout and faster storage |
Q: What is the main purpose of FreeToken?
FreeToken serves large MoE models on edge hardware by coordinating GPU execution, CPU execution, host memory, PCIe transfers, and elastic expert caching.
Q: Does FreeToken require the entire model to fit in VRAM?
No. The design keeps the complete routed-expert pool in host memory while using GPU VRAM for non-expert weights, KV cache, and a shared expert cache.
Q: Why are bandwidth measurements important?
The best split between PCIe cache fills and direct CPU execution depends on the deployed machine. FreeToken derives that split from measured host and transfer bandwidth.
Q: How should I validate a FreeToken setup?
Test cold and warm requests, monitor decode throughput and time to first token, then repeat the test with realistic multi-turn or agentic prompts.
The design and reported evaluations are documented in the FreeToken research paper, published on August 24, 2026. Use the project release materials for current installation instructions.