- FreeToken benchmark: Measures local serving performance for large mixture-of-experts models.
- Core idea: Combines GPU, CPU, system memory, and PCIe bandwidth as one inference platform.
- Best use case: Running open-weight MoE models that exceed available GPU memory.
- Setup path: Install the runtime, confirm model files, launch a local endpoint, and test requests.
- Main advantage: Adaptive caching and CPU-GPU execution improve consistency across consumer hardware.
FreeToken benchmark Overview
FreeToken benchmark refers to the evaluation of an edge-native serving system designed for large mixture-of-experts, or MoE, models. Instead of requiring the full model to fit in VRAM, FreeToken keeps the complete expert pool in host memory while using the GPU as an elastic cache and execution resource.
The system targets local inference on machines with different GPU capacities, PCIe links, CPU bandwidth, and memory budgets. Its benchmark focuses on decode throughput, time to first token, multi-turn agent workloads, and performance across several consumer GPU classes.
Video Highlights:
- FreeToken combines GPU VRAM, CPU processing, and system RAM for local model serving.
- The desktop application supports Windows, Linux, and macOS workflows.
- The command-line interface can launch a model and expose an OpenAI-compatible local endpoint.
- Runtime statistics include token speed, processed tokens, requests, and cache activity.
| Benchmark area | What it measures | Why it matters |
|---|---|---|
| Decode throughput | Tokens generated per second | Shows sustained generation speed |
| TTFT | Time to first token | Indicates prompt and startup responsiveness |
| Expert cache | Routed experts retained in VRAM | Reveals how effectively limited GPU memory is used |
| Cross-hardware scaling | Results across GPU and host configurations | Shows portability beyond one test machine |
Treat throughput and TTFT as separate metrics. A system can generate tokens quickly after startup while still producing an inconvenient first-token delay on long prompts.
How FreeToken Serves Large MoE Models
MoE models contain many experts but activate only a small subset for each token. This reduces active computation, but the complete expert collection can remain much larger than a consumer GPU’s VRAM. FreeToken addresses that mismatch with a two-level memory hierarchy.
The CPU-resident expert pool stores the full routed-expert weights and acts as the source of truth. Non-expert model weights remain on the GPU, while remaining VRAM is divided between the KV cache and an elastic expert cache.
| Runtime layer | Primary role | Adaptive behavior |
|---|---|---|
| GPU memory | Stores non-expert weights, KV cache, and selected experts | Cache capacity can change during runtime |
| System memory | Holds the complete expert pool | Remains available even when VRAM is limited |
| PCIe link | Moves selected experts into GPU cache | Transfer work is balanced against CPU execution |
| CPU | Executes some cache misses directly | Uses measured host bandwidth rather than fixed placement |
During prefill, FreeToken uses full-layer double buffering. While the GPU computes one layer, the next layer’s experts can move across PCIe. This overlaps transfer and computation instead of waiting for each expert movement to finish before execution continues.
During decode, the system uses a shared least-recently-used expert cache. Recently routed experts are more likely to remain in VRAM, while misses are divided between GPU cache fills and direct CPU execution.
The paper describes this balance with the q-star policy. The runtime profiles host-side expert bandwidth and PCIe transfer bandwidth, then uses those measurements to decide how many missing experts should move to the GPU and how many should execute on the CPU.
Elastic Cache
GPU expert capacity can be rebuilt at safe runtime points without reloading the host-resident model pool.
Semantic State
Checkpoints at thinking, tool-call, and conversation boundaries reduce unnecessary recomputation during agent sessions.
Adaptive Misses
Cache misses can become GPU work or direct CPU work, depending on measured bandwidth.
Graph Compatibility
Device-resident control data keeps routing-dependent decisions compatible with captured CUDA execution.
More system memory does not automatically create higher performance. PCIe bandwidth, host-memory bandwidth, CPU execution speed, and model layout all influence the final result.
FreeToken Setup Guide
FreeToken can be used through a desktop application or a command-line workflow. The CLI is useful for repeatable testing because it exposes the model path, launch arguments, endpoint behavior, and runtime output more directly.
The setup flow below follows the documented quick-start pattern. Confirm that the model files are already present at the path supplied to the launch command; the runtime cannot serve a model that is missing from that location.
Choose the Runtime
Select the desktop application for a guided workflow, or use the CLI for scripting and benchmark automation. The available desktop builds cover Windows, Linux, and macOS according to the published project materials.
Install the Package
Install FreeToken with the project’s recommended UV-based command: uv pip install freetoken. If UV is not installed, add it first and ensure its executable is available in your system path.
Confirm the Model Path
Download or prepare a supported local model, then verify that the supplied directory contains the required files. A valid installation alone does not guarantee that a model can launch.
Launch the Server
Start the runtime with the documented model argument, such as freetoken start --model <model-name-or-path>. Use the exact command syntax supported by the installed release.
Check the Endpoint
Query the local OpenAI-compatible endpoint, inspect the served model, and send a test chat-completion request before beginning a longer benchmark.
| Setup checkpoint | Expected result | Troubleshooting focus |
|---|---|---|
| UV command works | Package installation begins | Check installation and PATH configuration |
| Dependencies finish | Runtime becomes available | Review Python and package errors |
| Model path resolves | Model files are detected | Confirm directory and file permissions |
| Server starts | Local service listens | Check model compatibility and memory |
| Test request succeeds | Completion response returns | Verify endpoint, model name, and request format |
The runtime console can expose useful operational information, including token-per-second speed, request counts, processed-token totals, and cache-hit information. GPU VRAM and system RAM availability are also important when deciding whether a model configuration is suitable.
Run a short request before a long evaluation. This confirms the model path, endpoint name, memory allocation, and basic CPU-GPU execution path at once.
FreeToken benchmark Results
The published evaluation tests FreeToken on agentic workloads rather than relying only on short, isolated prompts. The scenarios include math reasoning, coding tasks with tool use, native coding-agent requests, and an email/calendar workflow.
The benchmark compares FreeToken with edge-serving systems including llama.cpp, Ollama, and KTransformers. The evaluation uses multiple GPU and host configurations, including RTX 3090, RTX 4090, RTX 5090, an RTX 4060 laptop, and an RTX PRO 6000 Blackwell workstation.
| Model | Model scale | Active parameters | Evaluation note |
|---|---|---|---|
| DeepSeek-V4-Flash | 284B | 13B | Routed experts use an MXFP4 deployment |
| Qwen3.6-35B-A3B | 35B | 3B | Tested in BF16, with an NVFP4 laptop build |
| GLM-5.2 | 753B | 40B | Frontier-scale demonstration on RTX PRO 6000 |
On the RTX 5090 evaluation, FreeToken reports 77–83 tokens per second on Qwen3.6 and 22–25 tokens per second on DeepSeek-V4-Flash. These figures represent the tested workloads and configurations, not a universal speed guarantee for every system.
| Workload result | Reported FreeToken outcome | Comparison described in evaluation |
|---|---|---|
| Qwen3.6 decode | 77–83 tok/s | 1.8–2.3 times the strongest baseline by workload |
| DeepSeek-V4-Flash decode | 22–25 tok/s | 1.5–1.9 times the strongest baseline by workload |
| RTX 4060 laptop | 39.3 tok/s on Qwen3.6 NVFP4 | 92% of the tested RTX 4090 rate |
| RTX PRO 6000 with GLM-5.2 | 14.9 tok/s | Compared with 7.3 tok/s for llama.cpp |
| Multi-turn TTFT | Worst turn below 44 seconds in tested cells | Baselines crossed 150 seconds in at least one cell |
The benchmark also reports that FreeToken’s cache policy reduced decode-time expert misses compared with static or prefill-based placement. At the tested RTX 5090 capacities, the reported miss rates were 16% for Qwen3.6 and 39% for DeepSeek-V4-Flash, compared with higher rates for the evaluated baseline policies.
For additional technical detail, consult the FreeToken research paper, which explains the bandwidth-adaptive execution policy, semantic-aware caching, implementation, and evaluation methodology.
The strongest results depend on model format, cache capacity, host bandwidth, and workload shape. Use the published numbers as reference points rather than guaranteed local performance.
Practical Testing Checklist
A useful local benchmark should record more than peak token speed. Test the same model, prompt format, quantization, context length, and workload across each configuration. For agentic serving, preserve multi-turn context and tool-call behavior because repeated prefill can expose differences that single-turn tests hide.
Benchmark Preparation:
- Verify the model path and confirm the intended model format
- Record GPU VRAM, system memory, CPU type, and PCIe link configuration
- Run a short warm-up request before collecting measurements
- Measure both decode throughput and time to first token
- Repeat multi-turn or tool-use workloads when evaluating agent performance
| Test variable | Keep consistent | Record separately |
|---|---|---|
| Model | Same checkpoint and precision | File format and quantization |
| Prompt | Same text and token budget | Context length and turn count |
| Runtime | Same launch options | Cache size and CPU thread count |
| Hardware | Same machine for direct comparisons | Background applications and memory pressure |
| Metrics | Same measurement window | Mean, tail, and failed requests |
Use the runtime console to monitor token speed, request totals, processed tokens, and cache behavior. If performance changes between runs, check whether another application has consumed VRAM or whether the available host-memory bandwidth has shifted.
A practical test sequence is:
- Start with a short prompt to validate correctness.
- Measure a single-turn generation for baseline decode speed.
- Add a long prompt to observe prefill and TTFT.
- Run several turns with context reuse.
- Repeat with background GPU activity if you want to study elastic resource behavior.
Prioritize tail TTFT for interactive agents. A stable response time across turns can be more valuable than a higher peak rate on a short prompt.
FreeToken FAQ
Q: What is the FreeToken benchmark?
It is an evaluation of FreeToken, an edge-native serving system for large MoE models. The benchmark measures decode throughput, time to first token, expert-cache behavior, and performance across consumer hardware.
Q: Does FreeToken require the entire model to fit in VRAM?
No. FreeToken keeps the complete expert pool in host memory and uses available GPU memory as an elastic cache. The model still requires sufficient system memory, storage, and compatible execution support.
Q: How does FreeToken handle cache misses?
The runtime can transfer selected missing experts to the GPU or execute other misses directly on the CPU. A measured-bandwidth policy determines the balance for the deployed machine.
Q: Can I use FreeToken through a local API?
Yes. The documented workflow exposes a local OpenAI-compatible endpoint after the model server starts, allowing a compatible client or coding agent to send chat-completion requests.
FreeToken is best understood as a local MoE serving system, not a game or content code platform. Its value comes from coordinating memory, bandwidth, caching, and execution.