- FreeToken api refers to a local inference stack for serving large mixture-of-experts models.
- Core advantage: Routing-aware expert caching reduces unnecessary CPU and PCIe bottlenecks.
- Best hardware fit: Recent NVIDIA GPUs with sufficient system memory and CUDA support.
- Main limitation: The public release is beta-oriented and primarily targets Linux NVIDIA setups.
- Key expectation: Results depend heavily on VRAM, host bandwidth, PCIe bandwidth, and model format.
What Is FreeToken api?
FreeToken is an edge-native serving system designed to run large open-weight mixture-of-experts models on personal hardware. Rather than treating the GPU as the only usable resource, it coordinates GPU memory, CPU memory, CPU execution, and the PCIe interconnect as one inference platform.
The system targets models whose full expert pool exceeds available VRAM but whose sparse per-token computation remains practical. For example, DeepSeek-V4-Flash is described as having 284B total parameters and 13B active parameters, while GLM-5.2 is listed at 753B total parameters and 40B active parameters. Only a subset of routed experts participates in each token, but the complete expert pool still has to remain accessible.
Video Highlights:
- FreeToken is positioned as a new local engine for large MoE models.
- The main comparison focuses on routing-aware caching versus static placement.
- Reported results include consumer GPUs, laptops, and a workstation-class GPU.
- Tail latency is treated as an availability concern for agentic workloads.
| Term | Meaning |
|---|---|
| MoE | A model architecture containing many experts while routing each token through only a few |
| Expert pool | The complete collection of routed expert weights |
| Prefill | Processing the existing prompt or context before generation begins |
| Decode | Generating new tokens one step at a time |
| TTFT | Time to first token, including the work required before output begins |
| Edge serving | Running inference on personal or consumer hardware rather than a datacenter cluster |
Think of FreeToken as a serving runtime, not a model. You still need a compatible model checkpoint, supported hardware, adequate host memory, and a correctly prepared runtime format.
How FreeToken Handles MoE Memory
FreeToken separates the model into a GPU-resident portion and a host-resident expert pool. Non-expert weights remain on the GPU, while the CPU-resident pool acts as the source of truth for routed experts. Available VRAM becomes an elastic expert cache shared across MoE layers.
This design matters because sparse computation does not eliminate memory pressure. A token may activate only six experts from a much larger pool, but the runtime must still be ready to access whichever experts the router selects next.
Shared Expert Cache
- Uses a global LRU residency space
- Tracks layer-expert pairs
- Follows changing token-level routing
Adaptive Miss Handling
- Sends some misses through PCIe
- Executes other misses directly on the CPU
- Balances work using measured bandwidth
Elastic Memory
- Adjusts the GPU cache at safe points
- Shares capacity with the KV cache
- Can respond to changing VRAM availability
During decode, FreeToken identifies cache hits and misses on the GPU. Hits execute directly from VRAM. Misses are divided between GPU cache fills and CPU execution according to measured host-side and PCIe bandwidth. This prevents the runtime from relying on one fixed strategy for every machine.
| Runtime Component | Primary Location | Main Responsibility |
|---|---|---|
| Non-expert weights | GPU memory | Remain available for normal model execution |
| Complete expert pool | Host memory | Provides the source weights for routed experts |
| Expert cache | Remaining GPU memory | Retains recently used layer-expert pairs |
| KV cache | GPU memory budget | Stores attention state for active contexts |
| Routing metadata | GPU and runtime buffers | Identifies selected experts and cache status |
The important change is not simply moving more weights between CPU and GPU. FreeToken treats a missing expert as either transferable data or executable work, then chooses between those paths according to the deployed machine.
FreeToken api Setup Workflow
A reliable setup begins with compatibility checks rather than performance expectations. The published material identifies CUDA and NVIDIA Linux-style environments as the primary supported target, while requests for broader Windows, macOS, and older-GPU support were still visible around the 2026 public launch period.
Use the following workflow to prepare a local deployment without assuming that every model or operating system is supported.
Confirm the Hardware Profile
Record GPU memory, host memory, PCIe link width, CPU memory bandwidth, and operating system. These values influence how much of the expert cache can remain in VRAM and how efficiently misses can be served.
Choose a Compatible Model
Start with a model listed by the project’s published evaluation, such as Qwen3.6-35B-A3B, DeepSeek-V4-Flash, or the GLM-5.2 demonstration. Verify the required precision and checkpoint layout before preparing storage.
Prepare the Runtime Format
FreeToken uses a FreeToken Weight format that normalizes expert banks into a layout suitable for direct loading. A prepared format can reduce startup work by avoiding repeated tensor discovery and repacking.
Measure Before Tuning
Let the runtime profile host-side expert processing and pinned transfer bandwidth. These measurements determine the balance between cache fills and direct CPU execution.
Test an Agentic Workload
Evaluate more than isolated token speed. Use a multi-turn coding, reasoning, or tool-calling workflow and monitor TTFT, long stalls, cache behavior, and completion reliability.
| Setup Check | Why It Matters | Recommended Action |
|---|---|---|
| NVIDIA CUDA environment | The fast path is centered on CUDA-compatible hardware | Confirm the driver and CUDA stack before model preparation |
| Host memory capacity | The complete expert pool may exceed VRAM by a wide margin | Reserve enough memory for the selected checkpoint |
| PCIe bandwidth | GPU cache fills depend on host-to-device transfer speed | Prefer a wide, high-bandwidth link where available |
| Model precision | Weight size and kernel compatibility vary by format | Match the checkpoint precision to the supported runtime path |
| Concurrent applications | Browsers, games, and desktop workloads can change VRAM availability | Leave headroom and test under realistic usage |
The project documentation and release material point to the official FreeToken project page for access to the system. The technical design is documented in the FreeToken research paper.
Preflight Checklist:
- Confirm a supported NVIDIA CUDA environment
- Verify host memory and available GPU memory
- Select a model and matching precision
- Prepare or obtain the required FreeToken Weight format
- Benchmark a realistic multi-turn workload
Do not treat a successful model download as proof of runtime compatibility. The public support profile emphasizes NVIDIA CUDA and POSIX Linux, while Apple Silicon, older NVIDIA cards, and broader platform support may require future project changes.
Performance and Hardware Comparison
FreeToken’s reported gains are strongest when the workload combines a large MoE model, limited VRAM, repeated expert routing, and long-running agentic turns. The evaluation compares it with actively maintained edge engines across multiple consumer systems and a workstation-class RTX PRO 6000.
On an RTX 5090, the published results report 77–83 tokens per second for Qwen3.6-35B and 22–25 tokens per second for DeepSeek-V4-Flash. The same evaluation reports a worst-turn TTFT below 44 seconds for FreeToken, while baseline stalls reached 232 seconds for llama.cpp, 179 seconds for Ollama, and 946 seconds for KTransformers in at least one tested cell.
| Model or Tier | Hardware Example | FreeToken Result | Reported Comparison |
|---|---|---|---|
| Qwen3.6-35B-A3B | RTX 5090 | 77–83 tok/s | 1.8–2.3x the strongest baseline |
| DeepSeek-V4-Flash | RTX 5090 | 22–25 tok/s | 1.5–1.9x the strongest baseline |
| Qwen3.6-35B-A3B | RTX 4060 laptop, 8 GB | 39.3 tok/s | About 92% of the reported RTX 4090 rate |
| GLM-5.2 | RTX PRO 6000 Blackwell, 96 GB | 14.9 tok/s | About 2.0x llama.cpp’s 7.3 tok/s |
| Qwen3.6-35B-A3B | RTX 5090 desktop | Noted as leading baseline comparison | Host bandwidth reduced the result by about 4% versus the server setup |
The cache policy also produced a substantial difference in the reported routing-trace replay. At an RTX 5090 serving capacity, FreeToken’s global LRU missed 16% of Qwen3.6 expert reads and 39% of DeepSeek-V4-Flash reads. The comparison figures listed 62% and 89% misses for llama.cpp’s static split in the same model order.
| Placement Strategy | Routing Awareness | Strength | Main Tradeoff |
|---|---|---|---|
| FreeToken global LRU | Token-level and continuously updated | Tracks the current expert working set | Requires dynamic cache control |
| llama.cpp static split | Fixed by layer placement | Predictable and straightforward | Can miss changing routed experts |
| KTransformers hot placement | Updated around prefill behavior | Can keep selected experts on GPU or CPU | May not follow every decode-time shift |
| CPU-only expert path | No GPU cache dependency | Broad fallback behavior | Limited by host memory bandwidth |
The evaluation also reports that full-layer double buffering improved prefill throughput by hiding expert movement behind computation. Disabling the second buffer reduced throughput by 19% at 4K tokens, 25% at 8K, and 26% at 16K in the cited Qwen3.6 test.
The headline throughput numbers come from the project’s own evaluation. They use identical weights and multiple workloads, but independent third-party benchmarks were not established in the supplied material. Treat tail latency, compatibility, and repeatability as equally important.
Limitations, Best Use Cases, and FAQ
FreeToken is most compelling for users who already own recent NVIDIA hardware, have substantial system memory, and run MoE models through coding or reasoning agents. The benefit is less obvious when the chosen model already fits comfortably in VRAM, when the platform lacks the required CUDA path, or when the workload is a short single-turn request.
The system also does not make local inference costless in a practical sense. Hardware, electricity, storage, cooling, and setup time remain part of the decision. Its value is instead connected to privacy, local control, reduced rate-limit exposure, and the ability to keep a model available without depending on a hosted service.
| User Profile | Fit | Reason |
|---|---|---|
| Recent NVIDIA desktop owner | Strong | Has the best chance of using the cache and PCIe path effectively |
| 8 GB NVIDIA laptop owner | Conditional | The reported laptop result is promising, but thermal and memory limits still matter |
| Apple Silicon user | Limited in the supplied 2026 support profile | No published Mac fast path was identified |
| Older NVIDIA GPU owner | Uncertain | Older hardware support was listed as an open request |
| Short single-turn user | Moderate | Long-context and agentic workloads better expose FreeToken’s advantages |
| Coding-agent user with large MoE models | Strong | Tail latency and repeated routing are central target problems |
Q: What is FreeToken api used for?
It is used to serve large open-weight mixture-of-experts models on personal hardware by coordinating GPU memory, host memory, CPU execution, and PCIe transfers.
Q: Does FreeToken require a specific model?
The runtime is model-dependent. The supplied evaluation names Qwen3.6-35B-A3B, DeepSeek-V4-Flash, and GLM-5.2, but each model still requires a compatible precision and prepared runtime layout.
Q: Is FreeToken better than llama.cpp for every user?
No. FreeToken targets recent NVIDIA systems running large MoE workloads. llama.cpp remains the more broadly established option when platform coverage and existing hardware compatibility are the priority.
Q: Why does FreeToken use both CPU execution and GPU caching?
A missed expert can be transferred to the GPU or executed where it already resides. FreeToken uses measured bandwidth to divide misses between those paths instead of relying on one fixed policy.
Use FreeToken when your workload is limited by large MoE expert pools and long agentic turns. For general local inference, first compare platform support, model availability, setup effort, and tail latency rather than relying on peak tokens per second alone.