FreeToken api server: Setup Guide & Runtime Tips

FreeToken is an edge-native serving system for large open-weight mixture-of-experts models. It is not a game or entertainment platform; its purpose is to expose local model inference through a serving runtime that can be used by compatible applications and agent frameworks.

The system is designed around a two-level memory hierarchy. The complete expert pool remains in host memory, while the GPU stores non-expert weights and an elastic cache of routed experts. This allows models larger than available VRAM to run without placing every expert permanently on the graphics card.

Video Highlights:

• FreeToken focuses on local inference for frontier-scale MoE models.
• The central challenge is moving or executing missed experts efficiently.
• Reported results cover consumer GPUs, laptops, and workstation-class hardware.
• Hardware compatibility matters more than headline throughput alone.

A useful way to understand FreeToken is to separate model capacity from active computation. An MoE model may contain hundreds of billions of total parameters, yet each token activates only a small subset of experts. The reference paper describes DeepSeek-V4-Flash as activating six of 256 routed experts per layer, with approximately 13 billion active parameters out of 284 billion.

That sparsity makes local serving possible, but it does not remove the memory problem. Every expert must remain accessible because the router can select different experts for each token. FreeToken therefore treats cache policy and bandwidth scheduling as first-class parts of the API server design.

Before configuring the FreeToken api server, confirm that the machine matches the runtime’s intended deployment profile. The public material describes a CUDA-centered beta environment, with the strongest results on recent NVIDIA GPUs and sufficient host memory for the full expert pool.

The most important checks are:

• NVIDIA GPU availability and suitable CUDA support.
• Enough system memory for the selected model’s complete expert pool.
• A PCIe connection capable of moving expert weights efficiently.
• Host-memory bandwidth that can sustain CPU-side expert execution.
• Adequate storage capacity and read speed for model files.
• Sufficient free VRAM for both non-expert weights and the runtime cache.

The paper’s evaluation spans an RTX 4060 laptop, RTX 3090 and RTX 4090 systems, RTX 5090 hardware, and an RTX PRO 6000 workstation. These results should be treated as reference points rather than guarantees for every configuration.

For example, the reported 39.3 tokens per second on an 8 GB RTX 4060 laptop uses an NVFP4 build and a specific model configuration. A different model format, driver, memory layout, or workload may produce a different outcome.

The API server also needs to account for dynamic memory pressure. Browsers, desktop compositors, games, and other applications can claim VRAM while FreeToken is running. Its elastic memory approach is intended to resize the expert cache at scheduler-safe points rather than requiring a complete engine restart every time the available budget changes.

A reliable setup begins with the model and memory plan, not with throughput tuning. Follow the sequence below to reduce failed launches and make later comparisons meaningful.

The setup sequence matters because FreeToken’s main benefits appear during sustained serving. A short prompt may not show the value of prefix reuse, semantic checkpoints, or an adaptive expert cache. Multi-turn agent workloads repeatedly edit context and route tokens through changing expert combinations, making them more representative of the design goals.

FreeToken’s most distinctive mechanism is its shared LRU expert cache. During decode, the router identifies the experts required for the next token and checks which ones are already resident. Cache hits execute on the GPU. Misses are divided between GPU cache fills and direct CPU execution according to measured bandwidth.

This differs from a fixed layer split. A static placement policy decides where experts live before token-by-token routing changes are known. FreeToken instead lets cache residency follow recent routing behavior. The cited replay results show lower miss rates under equal cache capacity, especially when the workload exhibits short-range expert locality.

The q-star policy estimates how many misses should be fetched over PCIe and how many should run on the CPU. The split is based on the measured relationship between pinned transfer bandwidth and host-side expert-processing bandwidth. This is important because a desktop with dual-channel DDR5 and a laptop with LPDDR5 may have very different optimal behavior, even when their GPUs are similar.

For tuning, prioritize tail latency rather than average decode speed alone. The evaluation reports FreeToken worst-turn TTFT below 44 seconds across tested cells, while some baselines crossed 150 seconds. In a real agent integration, a long stall can trigger a client timeout or idle watchdog even if the average tokens-per-second number looks competitive.

Use these practical rules:

• Keep the host system free of unnecessary memory pressure.
• Compare identical model weights and precision across engines.
• Measure multi-turn tasks, not only isolated prompts.
• Track first-token latency and worst-turn latency.
• Record cache size, context length, and CPU thread limits.
• Repeat tests after other applications claim VRAM.

The official FreeToken research paper provides the technical description of semantic-aware caching, pipelined prefill, elastic memory management, and bandwidth-adaptive execution. Use it as the reference when release behavior differs from early summaries.

FreeToken is promising for a specific deployment profile, but it is not automatically the best choice for every local inference user. The strongest fit is a recent NVIDIA system with enough RAM, an MoE workload, and a reason to keep inference local for long periods.

Its early public status also matters. The project has a small public history in the supplied 2026 materials, and several practical requests remain important for broader adoption, including wider hardware coverage, additional operating systems, dual-GPU workflows, and common model formats.

Local serving can reduce dependence on hosted rate limits and keep prompts on the user’s machine, but it transfers responsibility to the operator. Protect API credentials, bind the service deliberately, and avoid exposing an unauthenticated endpoint to a shared network.