- FreeToken app provides a desktop interface for running local Mixture-of-Experts models.
- Best use case: Models whose active experts exceed available GPU VRAM capacity.
- Core advantage: Adaptive CPU–GPU scheduling, expert caching, and background streaming.
- Installation options: Use the Windows or Linux desktop app, or install the CLI with
uv. - Important limitation: Models that fit comfortably in VRAM may run faster with a simpler runtime.
FreeToken app Overview
The FreeToken app is a local inference interface and engine designed for running large open-weight Mixture-of-Experts models on consumer hardware. Rather than requiring the entire model to fit inside GPU memory, it coordinates the GPU, CPU, system RAM, and PCIe connection as one flexible execution platform.
FreeToken is especially relevant when a model is too large for available VRAM but still practical for the rest of the computer. Its desktop application provides a graphical way to inspect compatible models, launch conversations, and adjust engine settings. The command-line version offers more control for development and benchmarking workflows.
Video Highlights:
- FreeToken is positioned as an alternative to tools such as Ollama and llama.cpp for selected MoE workloads.
- The engine uses GPU expert caching instead of permanently assigning model layers to one device.
- Q-star scheduling decides whether missing experts should move to the GPU or run on the CPU.
- Double-buffered prefill overlaps computation and expert transfers.
- Performance depends strongly on VRAM capacity, PCIe bandwidth, RAM speed, and model format.
The project is maintained as an open-source Apache License 2.0 project. Its official documentation and installation resources are available through the FreeToken GitHub repository.
| Area | FreeToken approach | Why it matters |
|---|---|---|
| Model type | Mixture-of-Experts models | Targets sparse models with large total parameter counts |
| Memory strategy | GPU cache plus system RAM | Avoids requiring every expert to remain in VRAM |
| Scheduling | Bandwidth-adaptive CPU–GPU execution | Adjusts to the local machine |
| Interface | Desktop app and CLI | Supports both visual and development workflows |
| License | Apache License 2.0 | Permits broad open-source use under the license terms |
Begin with the desktop app if you want a visual model overview. Use the CLI when you need reproducible commands, project integration, or detailed testing.
How FreeToken Handles Large MoE Models
A Mixture-of-Experts model may contain hundreds of billions of total parameters while activating only a smaller subset for each token. That sparsity can reduce computation, but it does not remove the memory challenge: the full model still needs to be available somewhere.
FreeToken treats GPU memory as an active expert cache. System RAM remains the source of truth for the complete model, while frequently requested experts are kept closer to the GPU. Because routers often reuse experts across nearby tokens, a relatively small hot set can serve much of the workload.
Expert Cache
Frequently used experts remain near the GPU, reducing repeated transfers for locally reused routing patterns.
Q-Star Policy
Cache misses can be divided between GPU transfer and direct CPU execution according to measured hardware bandwidth.
Double-Buffered Prefill
While one layer is being processed, the next layer's required experts can stream in the background.
During prompt processing, also called prefill, long inputs may touch a broad range of experts. FreeToken addresses this by overlapping data movement and computation instead of waiting for each transfer to finish before processing continues.
During generation, the engine must handle cache misses. Pulling an expert over PCIe may be beneficial on one system, while calculating directly on the CPU may be better on another. The Q-star policy measures the local balance between RAM and PCIe bandwidth and selects a suitable split for the current workload.
| Mechanism | Function | Main benefit | Main dependency |
|---|---|---|---|
| Global LRU cache | Keeps recently used experts available | Improves reuse across tokens | Expert locality and cache size |
| Q-star execution | Chooses GPU transfer or CPU execution for misses | Adapts to different hardware layouts | RAM and PCIe bandwidth |
| Double buffering | Streams future experts during current computation | Reduces visible transfer stalls | Concurrent transfer and compute support |
| FTW format | Loads weights into the engine's required layout | Reduces startup repackaging work | Model availability in FTW-compatible form |
| Semantic anchors | Preserves useful state and KV-cache checkpoints | Can reduce repeated context recomputation | Agentic context and cache behavior |
The engine also includes semantic anchor checkpoints for recurrent state and key-value caches. These are intended to reduce redundant context recomputation when agentic workflows edit context through tool calls or thinking blocks.
FreeToken is not a universal replacement for every local inference engine. Its strongest case is an MoE model that crosses the available VRAM threshold without making CPU execution the bottleneck.
FreeToken app Setup Steps
The official project supports a desktop app for Windows and Linux, along with a Python-based CLI installation. Choose one path based on how you plan to use the runtime.
Check the Hardware Profile
Review available GPU VRAM, system RAM, PCIe connectivity, and operating system support. FreeToken is designed for consumer systems, but very large frontier models may still exceed the capacity of a powerful desktop.
Choose the Interface
Select the Windows or Linux desktop app for guided setup, model visibility, and chat-oriented use. Select the CLI when you need scripting, project integration, or repeatable benchmark commands.
Install the CLI When Needed
The official repository recommends installing the accelerated package with uv pip install "freetoken[accel]". Building from source is also documented through the repository's virtual-environment workflow.
Load a Suitable MoE Model
Start with a model that is too large for a comfortable full-GPU load but remains realistic for your RAM and storage. Confirm that the model format and runtime requirements are supported.
Tune and Measure
Run a consistent prompt or coding task, then compare speed, stability, and memory behavior. Resize the GPU expert cache during a live server test when you need to study the performance curve.
| Setup path | Recommended for | Strength | Limitation |
|---|---|---|---|
| Desktop app | New users and visual monitoring | Guided interface and model overview | Current app workflow is more limited for project-file operations |
CLI with uv | Developers and testers | Scriptable and configurable | Requires command-line familiarity |
| Source build | Contributors and advanced users | Direct access to project code | More setup and maintenance responsibility |
The desktop app is currently listed for Windows and Linux. It can show which models appear feasible for the local system, but its current workflow may not provide the same project-folder or file-modification capabilities as a coding-focused integration.
Use a small, repeatable test before committing to a large model. A short benchmark can reveal whether transfers, CPU work, or cache misses are limiting your system.
Performance Comparison and Tuning
The most important decision is whether the model fits comfortably in GPU memory. If it does, a conventional fully resident GPU setup may be faster because it avoids FreeToken's streaming and scheduling overhead. If the model exceeds VRAM, FreeToken can keep mathematical work on the GPU while moving only the experts that are needed.
A reported workstation comparison illustrates this distinction. With an 8-bit Qwen model estimated at about 38 GB on a 32 GB GPU, a layer-splitting setup reached approximately 58 tokens per second, while FreeToken reached approximately 132 tokens per second on the same coding task. The task duration was about 14 minutes and 20 seconds versus 4 minutes and 40 seconds.
The same comparison showed a different result with a 4-bit version that fit inside 32 GB of VRAM. The fully GPU-resident configuration reached approximately 240 tokens per second, while FreeToken reached approximately 225 tokens per second. These figures are hardware- and workload-specific rather than universal targets.
| Scenario | Observed result | Preferred approach |
|---|---|---|
| Oversized 8-bit MoE model | FreeToken reached about 132 tokens/sec in the cited test | Test FreeToken first |
| Layer-split comparison | Alternative runtime reached about 58 tokens/sec | Useful baseline |
| Smaller 4-bit model fitting VRAM | Full-GPU setup reached about 240 tokens/sec | Prefer the simpler full-GPU path |
| FreeToken cache reduced from 58% to 40% | Speed loss was about 7% in the cited test | Consider a smaller cache |
| Cache reduced below roughly 20% | Performance declined sharply in the cited test | Avoid aggressive undersizing |
FreeToken also supports resizing its GPU expert cache while a server is running. In the cited test, reducing the cache from 58% to 40% caused only a modest speed reduction, while going below approximately 20% led to a much sharper decline as PCIe traffic became saturated.
Full VRAM Fit
Prefer a runtime that can keep the model fully on the GPU and avoid unnecessary streaming overhead.
Slight VRAM Overflow
FreeToken may recover performance by caching hot experts and adapting CPU–GPU execution.
Limited PCIe Link
Expect more sensitivity to cache misses and transfer volume, especially with a small expert cache.
Large System RAM
Extra RAM helps hold the complete model, but it does not eliminate bandwidth limits.
Token-per-second results depend on model quantization, prompt length, hardware, cache size, and task shape. Use published figures as comparison points, not guaranteed results.
Practical Checklist and FAQ
Use this checklist before evaluating a large local model. It focuses on compatibility and measurement rather than assuming that the largest available model will be the best choice.
Preflight Checklist:
- Confirm Windows or Linux support for the selected interface
- Measure available GPU VRAM and system RAM
- Verify that the target model is a compatible Mixture-of-Experts workload
- Run the same prompt or coding task for every runtime comparison
- Record cache size, generation speed, stability, and transfer behavior
| Checkpoint | What to verify | Recommended action |
|---|---|---|
| Operating system | Desktop app support | Use Windows or Linux for the graphical interface |
| Model architecture | MoE routing and supported weights | Avoid assuming dense models receive the same benefit |
| Memory balance | VRAM versus total model size | Test streaming when the model exceeds VRAM |
| Connection speed | PCIe transfer behavior | Watch for saturation during cache misses |
| Evaluation method | Same task and prompt | Compare runtimes under matched conditions |
Q: What is the FreeToken app designed to run?
It is designed for local inference with large open-weight Mixture-of-Experts models across consumer GPUs, CPUs, system memory, and interconnects.
Q: Should I use FreeToken when my model fits fully in VRAM?
Not necessarily. A fully GPU-resident model may be faster because it avoids the streaming and scheduling overhead used by FreeToken.
Q: Does FreeToken work only through a desktop interface?
No. FreeToken offers a desktop app for Windows and Linux plus a CLI that can be installed with the documented accelerated package.
Q: Can I change the GPU expert cache without restarting?
The engine supports live GPU expert-cache resizing on a running server, allowing you to test memory and speed tradeoffs without a restart.
Start with the official repository instructions, keep a record of your hardware and model configuration, and judge FreeToken by matched local tests.