- FreeToken quantization mainly concerns serving supported low-precision checkpoints efficiently on local hardware.
- MoE design lets only a small subset of model parameters activate for each token.
- VRAM limits do not remove the need for sufficient system RAM to hold the complete model.
- Best hardware fit currently means Linux, NVIDIA GPUs, CUDA 13, and a recent driver.
- Main advantage appears when a large MoE model cannot fit entirely inside GPU memory.
FreeToken quantization and the core serving model
FreeToken is an edge-native inference engine, not a new language model and not a standalone quantization algorithm. In the context of FreeToken quantization, the important question is how the runtime serves supported low-precision checkpoints when the full mixture-of-experts model exceeds available VRAM.
A mixture-of-experts model stores a large pool of experts but routes each token through only a limited selection. DeepSeek-V4-Flash is described as having 284 billion total parameters, with approximately 13 billion active for each token. That sparse activation makes local execution more practical, while the complete expert pool still creates a major memory and transfer challenge.
| Concept | What it means | Why it matters |
|---|---|---|
| Quantized checkpoint | A model stored using reduced-precision weights | Lowers storage and memory pressure |
| MoE model | A model with many experts and sparse token routing | Reduces active computation per token |
| Active parameters | Parameters used for the current token | Determines much of the immediate compute load |
| Full checkpoint | All model and expert weights | Still must reside somewhere on the system |
| Edge serving | Inference across GPU, CPU, RAM, and PCIe | Makes consumer hardware a unified runtime |
The paper describes FreeToken supporting multiple model families and precision formats. Its evaluation includes a natively MXFP4-quantized DeepSeek-V4-Flash checkpoint, an NVFP4 release for an 8 GB laptop configuration, and Qwen3.6-35B-A3B in BF16. This means “quantization support” depends on the specific model checkpoint and runtime path rather than on a universal conversion rule.
Video Highlights:
- FreeToken is designed for large MoE models that exceed GPU memory.
- Adaptive expert caching keeps frequently routed experts in VRAM.
- CPU execution and PCIe transfers can work together on cache misses.
- Long coding-agent sessions are an important target workload.
Choose a supported quantized checkpoint first, then calculate total system memory needs. An 8 GB GPU can accelerate a larger model, but it cannot store the entire checkpoint by itself.
How quantized MoE weights move through memory
FreeToken organizes inference around a two-level expert-memory hierarchy. The host system keeps the complete routed-expert pool, while non-expert weights remain on the GPU. Available VRAM is then used as an elastic expert cache shared across MoE layers.
This arrangement changes the role of quantization. Reduced-precision weights lower the size of the host-resident model and reduce transfer volume, but the runtime still has to decide which experts should be in VRAM, which missing experts should be transferred, and which can be executed directly by the CPU.
| Memory layer | Primary contents | Runtime role |
|---|---|---|
| GPU memory | Non-expert weights, KV cache, selected experts | Fast execution and active-state storage |
| Host RAM | Complete expert pool | Source of truth for model weights |
| PCIe link | Expert transfers and activation traffic | Connects CPU-side storage to GPU execution |
| NVMe storage | Checkpoint files and FTW data | Supplies model data during startup |
| CPU cache path | Recently used host-side data | Supports direct execution of misses |
During prefill, FreeToken uses full-layer double buffering when enough cache capacity is available. While the GPU computes one layer, the next layer’s experts can stream over PCIe. This overlaps data movement with computation instead of exposing every transfer as separate GPU idle time.
During decode, routing becomes more fine-grained. The runtime maintains a shared LRU cache based on recently selected experts. A cache hit executes on the GPU. A miss can be added to the cache through PCIe or executed directly on the CPU, depending on measured host and transfer bandwidth.
The bandwidth-adaptive policy is central to this design. A fast PCIe connection may favor transferring more missing experts, while stronger host-memory bandwidth can make direct CPU execution more attractive. The decision is made for the deployed machine rather than copied from a fixed hardware profile.
Quantization reduces checkpoint size but does not eliminate memory requirements. Large models still need enough RAM for the complete host-resident expert pool, plus operating-system and application overhead.
Supported formats, hardware, and performance profile
FreeToken’s documented accelerated setup is currently specialized. The command-line path described in the available material requires Linux on an x86-64 computer, an NVIDIA GPU, CUDA 13, and a recent driver. The project highlights RTX 30, RTX 40, and RTX 50 series hardware.
| Hardware or platform | Status in the available documentation | Key consideration |
|---|---|---|
| RTX 30 series | Highlighted support | Host RAM remains important for oversized MoE models |
| RTX 40 series | Highlighted support | PCIe and CPU bandwidth affect cache misses |
| RTX 50 series | Highlighted support | Strong fit for large local MoE experiments |
| RTX 4060 laptop | Evaluated configuration | 8 GB VRAM, 32 GB system memory, NVFP4 checkpoint |
| RTX PRO 6000 Blackwell | Frontier-scale evaluation | Used for GLM-5.2 demonstration |
| Apple Silicon | No comparable path described | Do not assume a native equivalent |
| CPU-only execution | Not the primary target | Specialized GPU serving is the focus |
The published evaluation reports 77–83 tokens per second for Qwen3.6-35B-A3B and 22–25 tokens per second for DeepSeek-V4-Flash on an RTX 5090 configuration. On an RTX 4060 laptop with 8 GB of VRAM and 32 GB of system memory, the official NVFP4 Qwen configuration reached 39.3 tokens per second.
A separate workstation result served GLM-5.2, described as a 753 billion parameter model, at 14.9 tokens per second on a single RTX PRO 6000. These figures are tied to specific checkpoints, hardware, workloads, and precision formats. They should be treated as reference points rather than universal performance guarantees.
| Model or configuration | Precision or format | Reported hardware | Reported result |
|---|---|---|---|
| Qwen3.6-35B-A3B | BF16 | RTX 5090 | 77–83 tokens per second |
| DeepSeek-V4-Flash | MXFP4 routed experts | RTX 5090 | 22–25 tokens per second |
| Qwen3.6-35B-A3B | Official NVFP4 release | RTX 4060 laptop, 8 GB VRAM | 39.3 tokens per second |
| GLM-5.2 | NVFP4 routed experts | RTX PRO 6000 Blackwell | 14.9 tokens per second |
| Qwen3.6-35B-A3B | Community-reported quantized test | RTX 5080 system | Around 100 tokens per second reported |
The strongest use case is a model that does not fit in VRAM but remains manageable in system memory. If a quantized model fits completely inside the GPU, a general-purpose runtime may already deliver excellent speed, and FreeToken’s transfer-oriented advantages may be less important.
FreeToken is most compelling when an NVIDIA GPU, sufficient system RAM, and a large MoE checkpoint must work together for interactive local inference.
FreeToken quantization setup workflow
Use this workflow to evaluate a supported quantized model without treating published benchmark numbers as guaranteed results.
Confirm the platform
Verify that the machine uses Linux, an x86-64 processor, a supported NVIDIA GPU, CUDA 13, and a recent driver. Also record available VRAM, system RAM, PCIe link width, and host-memory bandwidth.
Select a supported checkpoint
Choose an official or documented checkpoint with a compatible precision format, such as MXFP4, NVFP4, or the listed BF16 evaluation configuration. Confirm that the model family is supported before downloading its weights.
Calculate total memory needs
Treat GPU memory as acceleration space, not as the complete storage location. Reserve enough system RAM for the full expert pool, runtime state, the operating system, and the growing KV cache.
Prepare the runtime format
Use the project’s documented loading path. Where applicable, the FreeToken Weight format stores expert banks in the runtime layout, reducing checkpoint discovery and repacking work during startup.
Test with your real workload
Measure prompt latency, time to first token, decode speed, and stability across multi-turn sessions. Coding agents and tool calls can reveal behavior that short single-prompt tests miss.
The runtime can dynamically resize and rebuild the GPU expert cache at scheduler safe points. This is useful when browser windows, desktop applications, or other GPU workloads change the available VRAM during a session.
For agentic workloads, context handling matters as much as raw decode speed. FreeToken anchors recurrent-state checkpoints at semantic boundaries such as thinking segments, tool calls, tool outputs, and conversation turns. When an agent edits a prior block, the runtime can reuse the unchanged prefix and re-prefill only the new suffix.
| Test metric | What to record | Why it matters |
|---|---|---|
| VRAM usage | Cache, KV cache, and non-expert allocation | Shows whether memory is balanced |
| System RAM usage | Host-resident model and runtime overhead | Detects memory pressure |
| Time to first token | Average and slowest turns | Reveals prefill and context costs |
| Decode speed | Tokens per second by workload | Enables fair engine comparisons |
| Cache behavior | Hit and miss rate | Shows whether expert locality helps |
| Session stability | Long context and repeated tool calls | Tests practical agent serving |
Run the same model, precision, prompt sequence, and agent harness across engines. Compare long-session behavior separately from short single-turn throughput.
Readiness checklist and engine comparison
Before adopting FreeToken for regular local serving, check the constraints that most often determine whether the setup is practical.
Readiness Checklist:
- Confirm Linux, x86-64, NVIDIA GPU, CUDA 13, and a recent driver
- Select a documented model and compatible quantization format
- Reserve system RAM for the complete host-resident checkpoint
- Measure PCIe transfer and CPU memory bandwidth on the target machine
- Test long-context and tool-calling workloads before daily use
FreeToken is not positioned as a universal replacement for llama.cpp. llama.cpp supports a broader range of operating systems, processors, GPU vendors, Apple Silicon devices, and model formats. FreeToken instead focuses on oversized MoE models and the coordination of GPU memory, CPU execution, host RAM, and PCIe transfers.
| Runtime | Main strength | Main limitation for this use case |
|---|---|---|
| FreeToken | Adaptive MoE serving across GPU and CPU resources | Narrower platform and model ecosystem |
| llama.cpp | Broad hardware and model compatibility | Static hybrid placement can miss changing expert locality |
| KTransformers | CPU expert execution and hybrid serving | Reported policies may be less adaptive to hardware balance |
| Ollama | Accessible local model workflow | Not the primary specialized target for oversized MoE serving |
The project also provides OpenAI-compatible and Anthropic-compatible APIs, allowing local models to connect with supported coding and agent tools. Compatibility at the API layer does not guarantee identical behavior across every client, so test authentication, context handling, tool calls, and timeout settings individually.
For the technical design and evaluation details, read the FreeToken research paper. The paper identifies FreeToken as an Apache 2.0 open-source system and lists the release destination as flashml.ai.
Start with one supported quantized MoE checkpoint and a reproducible benchmark. Expand only after confirming memory headroom, acceptable first-token latency, and stable agent sessions.
FreeToken quantization FAQ
Q: Is FreeToken itself a quantization algorithm?
No. FreeToken is an inference and serving engine. The term FreeToken quantization usually refers to running compatible low-precision checkpoints through its MoE serving system.
Q: Does an 8 GB GPU run a 35B model using only 8 GB of memory?
No. The evaluated laptop configuration used 8 GB of VRAM and 32 GB of system memory. The GPU handled acceleration while host memory held the remaining model weights.
Q: Which quantization formats are discussed for FreeToken?
The available evaluation describes MXFP4 routed experts for DeepSeek-V4-Flash, an official NVFP4 Qwen3.6 release for the laptop test, and BF16 for the main Qwen3.6 comparison.
Q: When is FreeToken more useful than a general local runtime?
It is most useful when a large MoE model exceeds VRAM, the computer has sufficient system RAM, and adaptive expert caching or CPU-GPU coordination can reduce transfer stalls.