FreeToken vram: Setup Guide for Local MoE Inference

FreeToken vram is best understood as part of a unified local inference system. FreeToken serves mixture-of-experts models by combining GPU memory, system RAM, CPU processing, and the PCIe link between them. The complete routed-expert pool can remain in host memory while available VRAM stores non-expert weights, runtime state, KV cache pages, and recently used experts.

This design matters because an MoE model may have a very large total parameter count but activate only a smaller subset for each token. DeepSeek-V4-Flash is described as a 284B-parameter model with approximately 13B active parameters per token. That makes local serving more practical, but it does not eliminate the need to hold the wider expert pool somewhere.

Video Highlights:

• Single-GPU testing demonstrates local FreeToken serving with an RTX 3090.
• DeepSeek-V4-Flash reaches roughly 10 to 11 tokens per second in one server configuration.
• The desktop client is easier to launch but may produce lower throughput than the server path.
• System RAM capacity and memory bandwidth have a direct effect on practical performance.

The practical lesson is simple: more VRAM helps, but VRAM alone does not define the supported model range. A system with a large GPU and insufficient host memory may still fail to load a frontier-scale model. Conversely, a system with enough RAM can run a model with more flexible offload behavior, although response speed depends heavily on bandwidth.

FreeToken’s memory requirements vary by model, quantization format, context length, and the amount of VRAM reserved for expert caching. The available references describe 32 GB of system RAM as a practical starting point for a single-GPU setup, with 64 GB providing a more comfortable operating range. Larger models can require substantially more.

The complete model footprint should not be confused with active parameters. Sparse activation reduces computation for each token, while inactive experts still need to remain in host memory or another storage tier. This is why a model can be computationally feasible but still fail because the combined usable VRAM and RAM budget is too small.

A longer context also changes the memory balance. As an agent session grows, KV cache demand can increase while the expert working set remains comparatively stable. FreeToken therefore needs to adjust the division between KV cache pages and expert-cache slots instead of treating the initial launch allocation as permanent.

The paper FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution, published in 2026, describes this elastic memory approach. The GPU expert cache can be rebuilt at scheduler safe points without restarting the engine or reloading the complete CPU-resident expert pool.

Follow these steps to prepare a local FreeToken deployment. The exact interface and supported model list can change because the desktop application is described as beta software in the available testing material.

A first launch should be treated as a baseline test rather than a final performance result. Record memory usage, prompt processing time, decode speed, and whether the model remains stable across several requests.

FreeToken’s performance depends on how effectively it keeps frequently used experts in VRAM and how it handles cache misses. During decode, routed experts may already be resident or may need to move from host memory. FreeToken uses a shared LRU-style expert cache so recent routing behavior can influence future residency.

The runtime also divides cache misses between GPU transfers and direct CPU execution. This split is based on measured host-side processing bandwidth and pinned transfer bandwidth rather than a single fixed rule. That approach is useful because a desktop with strong PCIe performance and a laptop with a narrow link can benefit from different allocations.

Prefill and decode behave differently. Prefill processes a large prompt and can require movement across much of the expert pool. FreeToken uses full-layer double buffering to overlap expert transfers with GPU computation where the available memory budget permits. Decode is more incremental, but repeated cache misses can still reduce responsiveness.

Testing from the supplied material found that a single RTX 3090 configuration produced approximately 10.5 tokens per second in one server-side scenario. A desktop client test reached about 8.8 tokens per second under a different setup. These figures are configuration-specific rather than universal benchmarks, so use them as reference points instead of guarantees.

Not every model will work equally well in FreeToken. The referenced testing found that a dense Qwen 3.8 27B BF16 configuration failed to start in one environment, producing an unexpected engine exit. This illustrates why model compatibility, quantization format, and runtime support matter alongside memory capacity.

Large models can also fail for a straightforward reason: insufficient usable RAM. One test indicated that a GLM-5.2 NVFP4 configuration required hundreds of additional gigabytes of system memory beyond the available budget. A larger workstation configuration may be necessary for such a model.

Use a controlled troubleshooting sequence:

• Test the same model with a short prompt.
• Confirm whether the failure occurs during loading or generation.
• Check whether the model uses a supported quantization format.
• Compare usable memory before and after closing other applications.
• Save server logs when reporting a reproducible engine error.

The most reliable strategy is to change one variable at a time. Switching the model, client, operating system package, memory allocation, and reasoning mode simultaneously makes it difficult to identify the actual cause.