FreeToken paper: MoE Serving Comparison and Setup Guide

The FreeToken paper presents an edge-oriented approach to serving large mixture-of-experts models when model weights exceed available GPU memory. Its full title is FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution. The paper was submitted to arXiv on August 17, 2026, and lists researchers including Song Han, Matei Zaharia, and Ion Stoica.

Rather than treating inference as a simple GPU-only workload, FreeToken focuses on the movement and placement of expert weights across GPU memory, system memory, and the processor. This distinction matters because sparse computation does not automatically mean sparse memory requirements. A model may activate only a small group of experts for each token while still needing access to the complete expert pool.

Paper profile:

Video Highlights:

• Explains why very large MoE models can run on a single workstation under the right memory strategy.
• Compares FreeToken’s routing-aware placement with a fixed layer-based expert split.
• Discusses reported throughput, tail latency, hardware support, and independent benchmark limitations.

The central research question is straightforward: what should an inference engine do when the next token requests an expert that is not currently resident on the GPU? A static placement policy can be predictable, but it may ignore the runtime routing behavior that determines which experts are actually needed. FreeToken’s design attempts to adapt execution to those changing access patterns.

For the formal reference, read the FreeToken paper on arXiv.

A mixture-of-experts model contains many specialized feed-forward modules, commonly called experts. A router selects a limited number of experts for each token. This creates sparse computation: only part of the network performs arithmetic for a given token. However, the complete parameter set still needs to remain accessible because the router can choose a different expert on the next token.

FreeToken treats this changing demand as a systems problem. The engine can decide whether an expert should remain on the GPU, stay in system memory, or be processed closer to where it is stored. The goal is to reduce expensive transfers across the hardware bus while preserving enough flexibility to follow runtime routing.

The reference comparison described in the available technical discussion uses a fixed layer-based policy as a contrast. In that model, the operator selects how many early layers keep their MoE weights on the processor. The split is established before inference begins and does not change when token routing changes.

FreeToken instead emphasizes routing-aware behavior. In the reported experiment, the systems replayed identical routing traces under the same cache size while changing only the placement policy. The discussion reports a 16% expert-read cache miss rate for FreeToken against 62% for the compared static split under a memory configuration associated with an RTX 5090. These numbers describe the cited test conditions rather than a universal result for every model or device.

The practical takeaway is that cache misses are not minor bookkeeping events. Each miss can require a weight transfer or a change in execution location. When this happens repeatedly during autoregressive generation, the cost appears as lower throughput and, more importantly, long pauses during individual turns.

The available benchmark discussion presents FreeToken as especially relevant to recent Nvidia systems running MoE models that are too large for ordinary GPU-only loading. Reported results include Qwen 35B on an RTX 5090, DeepSeek V4 Flash on similar hardware, and GLM on a workstation-class card.

The figures below are transcribed as reported measurements from the available discussion. They should be treated as research results pending broader third-party reproduction.

Throughput is useful, but it does not tell the entire story for interactive agents. A system that produces a high average rate but occasionally stalls for several minutes may be less useful than a slower engine with predictable response times.

The reported worst-turn comparison is therefore important. FreeToken’s slowest cited single turn remained below 44 seconds, while the discussion places llama.cpp at 232 seconds, another implementation at 179 seconds, and KTransformers at 946 seconds in the cited scenario. These are workload-specific results, not general guarantees.

A separate comparison issue concerns denominators. The discussion contrasts FreeToken’s decode figures with a cloud-agent trace value of 33 tokens per second, but that trace measurement reportedly includes time to first token and intermediate reasoning. A like-for-like comparison with a pure decode median produces a smaller advantage than the headline chart suggests. This does not invalidate the benchmark; it means readers should compare measurements with matching definitions.

FreeToken’s strongest use case is narrow but meaningful: a user with a recent Nvidia GPU, substantial system memory, and a workload built around large MoE models may benefit from routing-aware serving. The advantage becomes more visible when static placement causes frequent expert transfers or when long stalls cause an agent watchdog to terminate a task.

That profile is different from the average local-AI user. The project information summarized in the available material identifies Nvidia CUDA and POSIX Linux as the supported environment, with beta-level packaging. Requests for older Nvidia cards, dual-GPU Docker support, GGUF, Windows fixes, and Apple Silicon support were still visible around the August 2026 launch period.

FreeToken should not be evaluated only by peak tokens per second. Installation friction, model compatibility, memory capacity, operating system support, and stability can outweigh a benchmark lead. A tool that runs on fewer systems may still be valuable as a specialized research engine, but it is not automatically the best default for every local deployment.

The comparison with llama.cpp illustrates this tradeoff. The established project is described as having broader backend and platform coverage, while FreeToken concentrates on a newer execution policy for a narrower hardware profile. These tools should therefore be viewed as serving different priorities: portability and maturity on one side, specialized MoE efficiency on the other.

A reliable comparison requires more than launching two engines and reading the fastest number. Use the same model files, quantization, prompt set, context length, generation settings, and hardware conditions. Record both average throughput and the slowest meaningful turns.

For reproducibility, publish the exact command-line options, commit or release identifier, model checksum, and test date. The public project was described as having a young history in August 2026, so results may change quickly as kernels, supported formats, and placement policies evolve.

FreeToken’s research direction is significant because it treats expert movement as a first-class inference concern. Still, the available evidence supports a measured conclusion rather than a universal replacement claim. The project was young, platform coverage was limited, and the benchmark set required broader independent validation.

The paper also raises a larger ecosystem question. If routing-aware expert caching proves useful, mature inference engines may eventually adopt similar mechanisms. In that case, FreeToken’s lasting contribution could be its execution policy rather than long-term dominance as a standalone runtime.