FreeToken github: Local MoE Setup Guide & Benchmarks

FreeToken github is the public home of an edge-native inference engine designed for running frontier-scale open-weight Mixture-of-Experts models on personal and consumer hardware. The project combines GPU memory, system RAM, CPU compute, and interconnect bandwidth as one flexible serving platform rather than treating the GPU as an isolated device.

The official FreeToken GitHub repository provides the project description, desktop application direction, CLI installation notes, research citation, acknowledgments, and license information. It also links to the paper, developer Slack, community Discord, and community WeChat channels.

Video Highlights:

• FreeToken is designed for large MoE models that do not fit entirely in GPU memory.
• Adaptive expert caching can keep frequently used model components closer to the GPU.
• Agent workloads benefit from reduced context recomputation during repeated tool calls.
• The project is specialized rather than a universal replacement for every local inference runtime.

A Mixture-of-Experts model can contain a very large total parameter count while activating only a smaller subset of experts for each token. That sparse execution pattern makes frontier-scale local inference more practical, but the complete checkpoint still needs to be stored somewhere.

FreeToken addresses this challenge with several coordinated runtime strategies. Its bandwidth-adaptive CPU–GPU co-execution policy evaluates whether an operation should remain on the CPU or move to the GPU. The decision depends on the real system rather than a fixed hardware split.

The project also uses global LRU expert caching. In practical terms, frequently needed experts can remain in a faster memory tier while less useful entries are replaced. This is particularly relevant when a model’s full weights reside partly in system memory.

Double-buffered prefill streaming attempts to hide transfer delays behind work that the GPU is already performing. Transfers still consume bandwidth, but overlapping them with computation can reduce visible waiting. This approach becomes more important as prompts grow longer and agents repeatedly update their working context.

The official project description also emphasizes graph-compatible execution and semantic-aware caching. These features are aimed at workloads where the prompt changes incrementally, such as coding agents that read files, call tools, receive results, and submit follow-up requests.

The repository presents two main access paths: a desktop application for Windows or Linux and a command-line installation path using uv or pip. The accelerated CLI route is currently most closely associated with Linux, x86-64 hardware, Nvidia graphics, CUDA 13, and a recent driver.

Use the following process to keep the first setup focused and verifiable.

FreeToken’s strongest use case is a large MoE model that does not fit entirely inside the GPU. In that situation, the runtime can use host memory while adapting the division of labor between the CPU and GPU.

Reported testing includes an RTX 5090 running Qwen 3.5 35B A3B at approximately 77–83 tokens per second across selected workloads. DeepSeek V4 Flash was reported at approximately 22–25 tokens per second in the same broad testing context. These figures come from project-related testing and should be treated as workload-specific rather than universal guarantees.

A separate reported community result used an RTX 5080, 64 GB of system memory, and a Ryzen 9 9950X3D. The user reported approximately 100 tokens per second for Qwen 3.5 35B A3B, with one example near 110 tokens per second. This is an early community result, not a controlled benchmark.

Token generation speed is only one part of the experience. Agent workflows can expose long delays during context processing, especially when a client repeatedly edits the conversation. FreeToken’s semantic-aware caching is intended to preserve unchanged context and reduce redundant recomputation.

For a fair comparison, test identical checkpoints, quantization, prompt lengths, context settings, and client behavior. A small model that fits entirely in VRAM may already perform very well under another runtime. FreeToken becomes more compelling when memory movement and long context updates are the dominant constraints.

FreeToken is specialized. The project highlights Windows and Linux desktop access, while the accelerated command-line documentation is focused on Linux, x86-64 systems, Nvidia GPUs, CUDA 13, and recent drivers. It is not presented as a universal CPU runner or a broad replacement for every hardware ecosystem.

The project also does not remove the need for sufficient storage and system RAM. A large checkpoint remains large even when sparse execution reduces the number of active parameters per token. Model format, quantization, context length, and agent behavior all affect the final memory requirement.

The Apache License 2.0 makes the repository suitable for inspection and reuse within the license terms. The project acknowledges inspiration and reused code from systems including SGLang, vLLM, FlashInfer, LightLLM, flash-linear-attention, and llama.cpp. Review the repository’s current license and notices before distributing modified components.