FreeToken rtx 5090: Setup Guide, Results & Tips

FreeToken is an edge-native inference runtime rather than a new AI model. On an RTX 5090, its main purpose is serving large mixture-of-experts models whose complete expert pool does not fit in GPU memory. The runtime keeps the full model in host memory while using the GPU as an adaptive working cache.

This approach matters because MoE models activate only a fraction of their total parameters for each token. DeepSeek-V4-Flash, for example, is listed with 284 billion total parameters and approximately 13 billion active parameters. The active computation can fit within the practical memory capacity of a high-end GPU, but the complete checkpoint still requires much more storage elsewhere.

Video Highlights:

• FreeToken is designed for oversized MoE models rather than every local model workload.
• A single RTX 5090 reportedly reached more than 20 tokens per second with DeepSeek-V4-Flash.
• The runtime combines expert caching, overlapped transfers, and CPU execution.
• Long coding-agent sessions reveal larger differences than short synthetic prompts.

The accelerated FreeToken path is currently centered on Linux, x86-64 systems, NVIDIA GPUs, CUDA 13, and a recent driver. The project highlights RTX 30-, 40-, and 50-series hardware, placing the RTX 5090 in its primary target range.

The GPU is only one part of the configuration. The complete expert pool must reside in system memory when it exceeds VRAM. An RTX 5090 does not reduce the model’s total memory requirement; it improves how the available GPU, CPU, RAM, and PCIe resources are coordinated.

The research setup also distinguishes between PCIe bandwidth and host-side expert-processing bandwidth. Those values should be measured on the actual computer instead of inferred only from product specifications.

The 2026 evaluation measures decode throughput and time to first token across math, coding-agent, native protocol, and email/calendar workloads. This is important because an agent repeatedly changes its context, calls tools, and sends new requests. A short one-shot prompt does not expose the same prefill and cache behavior.

On an RTX 5090, FreeToken sustained approximately 77–83 tokens per second with Qwen3.6-35B-A3B and 22–25 tokens per second with DeepSeek-V4-Flash across the tested workloads. The reported advantage over the strongest supported alternatives ranged from roughly 1.5x to 2.3x, depending on the model and scenario.

The RTX 5090 results are strongest when the model is larger than the GPU’s available VRAM. FreeToken’s shared LRU cache follows routing locality between tokens, while its bandwidth policy decides how to serve experts that miss the cache.

The evaluation reports that FreeToken’s cache missed approximately 16% of Qwen3.6 expert reads and 39% of DeepSeek-V4-Flash reads at the tested RTX 5090 serving capacity. Comparable placement approaches showed higher miss rates in the same replayed traces.

FreeToken organizes memory into a hierarchy. The host-resident expert pool stores the complete routed-expert weights, while non-expert weights remain on the GPU. Remaining VRAM becomes a shared expert cache that can be divided differently as the session changes.

During prefill, the runtime uses full-layer double buffering when memory allows. While the GPU processes one layer, the next layer’s experts can move over PCIe. This hides part of the transfer cost behind active computation. During decode, the runtime uses a shared LRU cache to track experts selected by recent routing decisions.

A cache miss is not treated as a single fixed operation. FreeToken estimates a split between:

• GPU cache fills, which transfer missing experts over PCIe and retain them for reuse.
• Direct CPU execution, which processes experts where their weights already reside.
• Concurrent execution, which allows both paths to contribute to the current token.

This division depends on measured host bandwidth and PCIe bandwidth. A desktop with strong PCIe connectivity may favor more GPU cache fills, while a system with higher host-memory bandwidth may perform more direct CPU work.

The runtime also supports dynamic reconfiguration at scheduler safe points. Because host memory remains the source of truth, changing GPU cache capacity affects performance rather than model correctness. This is useful on personal computers where browsers, desktop applications, and other GPU workloads may change the available VRAM budget.

FreeToken is not a universal replacement for llama.cpp, Ollama, or KTransformers. Its strength comes from specialization. The project focuses on large MoE checkpoints, NVIDIA acceleration, host-resident experts, and agentic serving.

Llama.cpp offers broader hardware and operating-system coverage, including CPU, NVIDIA, AMD, and Apple Silicon paths. It also has a large GGUF ecosystem and substantial maturity. Ollama emphasizes accessible local model management, while KTransformers targets hybrid CPU-GPU execution for supported model families.

The best choice depends on whether your model fits inside VRAM and whether your workload is interactive or agentic.

For technical details, consult the FreeToken research paper on arXiv and the project release referenced by the authors at flashml.ai. These links provide the strongest basis for checking supported models, implementation changes, and deployment guidance as the project develops.