FreeToken github moe: Setup Guide & MoE Runtime Comparison

FreeToken is an edge inference engine designed to serve large mixture-of-experts (MoE) models on personal hardware. If you are searching for “FreeToken github moe,” the important distinction is that FreeToken is serving software, not a new language model. It loads supported checkpoints and coordinates the GPU, CPU, system memory, and PCIe connection during inference.

The system targets models whose total parameter count is much larger than the available VRAM. An MoE model activates only a subset of experts for each token, making its per-token computation manageable. However, the complete expert pool still has to remain somewhere in memory. FreeToken keeps the full pool in host memory while using VRAM as an adaptive expert cache.

Video Highlights:

• FreeToken is positioned as a specialized alternative to llama.cpp for oversized MoE models.
• Reported tests include Qwen3.6-35B-A3B, DeepSeek-V4-Flash, and GLM-5.2.
• The runtime is designed for long coding-agent and tool-calling sessions.
• Hardware requirements depend on VRAM, system RAM, CPU bandwidth, and PCIe bandwidth.

The FreeToken research paper describes three major mechanisms:

The paper reports that FreeToken can serve models from a 35-billion-parameter class on an 8 GB laptop GPU, provided the remaining weights fit in system memory. That does not mean the model requires only 8 GB of total memory. The GPU holds active computation and cached experts, while system RAM stores the larger host-resident pool.

FreeToken is most interesting when a model does not fit entirely in VRAM. If the complete quantized model already fits on your GPU, a conventional runtime may remain highly competitive because it avoids repeated transfers between system memory and VRAM.

The reported RTX 5090 results show different behavior across models and workloads:

These figures come from the project’s published evaluation, so they should be treated as reported results rather than universal guarantees. Your model precision, prompt length, system memory, CPU architecture, PCIe link, and background applications can materially change performance.

The system also emphasizes time to first token (TTFT) during agentic workloads. A coding agent repeatedly reads files, calls tools, receives output, and sends new requests with an increasingly large context. FreeToken uses prefix and recurrent-state checkpoints at semantic boundaries, allowing unchanged context sections to be reused after tool calls or context edits.

The evaluation compares FreeToken with llama.cpp, KTransformers, Ollama, and MoE-Infinity. The reported advantage is largest in cases where expert weights must move frequently and where the workload contains long, changing contexts.

The accelerated documentation described in the available material focuses on Linux x86-64, an NVIDIA GPU, CUDA 13, and a recent driver. The project also references desktop support for Windows and Linux, but the strongest documented path remains Linux with NVIDIA hardware.

Use the following setup sequence as a planning guide. Confirm the current commands and supported checkpoints through the project’s official release channel before installing, because runtime support can change.

The project’s design uses an FTW storage format to normalize expert banks into a layout that can be loaded efficiently. If a supported distribution provides a preprocessed format, it may reduce startup work by avoiding repeated tensor discovery and repacking.

FreeToken should not be treated as a universal replacement for llama.cpp. The two projects optimize for different priorities. llama.cpp supports a much broader mix of operating systems, CPUs, GPU vendors, Apple Silicon devices, and GGUF models. FreeToken instead focuses on the difficult case of serving very large MoE models whose experts overflow GPU memory.

FreeToken’s reported advantages come from combining several policies rather than relying on one optimization:

• Shared LRU expert caching follows token-level routing locality.
• Double-buffered prefill overlaps the next layer’s expert movement with current computation.
• Bandwidth-adaptive scheduling measures the actual machine instead of assuming every system has the same PCIe and DRAM balance.
• Elastic cache resizing adjusts the VRAM split between experts and growing KV cache demand.
• Prefix reuse helps agent sessions avoid recomputing unchanged context after tool interactions.

Choose FreeToken when most of the following statements are true:

• Your target model is an MoE checkpoint that exceeds available VRAM.
• You have enough system RAM for the full model.
• Your GPU is NVIDIA and the software stack is supported.
• You care about long-running coding or tool-calling agents.
• You are comfortable with a more specialized setup.

Choose llama.cpp or another general runtime when portability, model breadth, Apple Silicon, AMD support, CPU operation, or GGUF compatibility matters more than peak performance on oversized MoE models.

FreeToken’s architecture makes large local models more accessible, but it does not remove their storage or memory requirements. An 8 GB GPU does not turn a 35B or 284B checkpoint into an 8 GB installation. The complete model still needs host memory, storage, and a compatible runtime.

For reliable testing, use the same model, precision, prompt, client, and workload across runtimes. A short one-shot prompt is useful for checking that the server works, but it does not represent a multi-turn coding agent. Run several turns, include tool calls if relevant, and note the slowest TTFT rather than only the average.

The project’s published evaluation reports that FreeToken’s worst tested turn stayed below 44 seconds, while baseline runtimes crossed much higher delays in some cells. Those values are useful for understanding the design target, but local results will vary with hardware and software versions.