- FreeToken gguf support is native for Gemma-4, while most listed models use Hugging Face safetensors.
- Best use case: Large MoE checkpoints that exceed available GPU VRAM but fit in system memory.
- Core advantage: Adaptive expert caching, CPU work, and PCIe transfers can reduce waiting between tokens.
- Key limitation: The accelerated setup is focused on Linux, NVIDIA GPUs, CUDA 13, and x86-64 systems.
- Important check: A GGUF file is not automatically compatible unless FreeToken supports its architecture and format.
FreeToken gguf Compatibility Explained
FreeToken is a local inference engine rather than an AI model. Its purpose is to run supported checkpoints on available hardware, especially large mixture-of-experts models whose complete weights cannot fit in VRAM. The current model documentation states that FreeToken loads Hugging Face safetensors checkpoints directly and supports native GGUF for Gemma-4.
That means “FreeToken gguf” should not be interpreted as universal GGUF compatibility. GGUF is a model container format, but the runtime still needs matching architecture support, kernels, and loading logic. Check the official FreeToken model documentation before converting or downloading a checkpoint.
Video Highlights:
- FreeToken targets oversized MoE models rather than every local inference workload.
- Adaptive expert caching keeps frequently used experts available in VRAM.
- Hybrid execution can combine GPU transfers and CPU computation.
- Reported tests include large models running beyond the limits of their graphics cards.
| Format or source | Current FreeToken handling | Practical meaning |
|---|---|---|
| Hugging Face safetensors | Direct loading for known-good checkpoints | Primary route for listed model families |
| GGUF | Native support documented for Gemma-4 | Do not assume every GGUF model will load |
| Converted FreeToken checkpoint | Optional fast-load format | ft serve --model can auto-detect the result |
| Unsupported architecture | No documented guarantee | Verify model support before conversion |
A file extension alone does not confirm compatibility. Verify the model family, checkpoint structure, quantization, and documented backend support before preparing a FreeToken deployment.
Supported Models and Runtime Backends
The documented model list identifies DeepSeek-V4 and GLM-5.2 as known-good Hugging Face checkpoints. The documentation also explains several MoE backends that determine where experts are stored and how cache misses are handled.
The auto setting selects a default based on the model type. Dense models resolve to fused, while MoE models generally use offload and can move to hybrid when a machine benchmark recommends it.
fused
Experts remain resident on the GPU. This can be efficient when the available VRAM is sufficient, but it requires the most GPU memory.
offload
Experts live in host RAM, while an LRU cache keeps selected expert slots on the GPU. Cache misses stream across PCIe.
hybrid
Each step can fetch some experts over PCIe and calculate others on the CPU, with overlapping work when the hardware favors that split.
| Backend | Expert location | Best-fit situation |
|---|---|---|
fused | GPU VRAM | The model and active workload fit comfortably on the GPU |
offload | Host RAM with GPU cache | Large MoE models exceed VRAM capacity |
cpu | CPU handles cache misses | CPU memory bandwidth is preferable to repeated transfers |
hybrid | CPU and PCIe transfers | Benchmarking shows a mixed path is faster |
auto | Selected automatically | A reasonable starting point for supported models |
FreeToken’s design is especially relevant when the model is larger than the graphics card but still fits within total system memory. It treats GPU compute, CPU compute, RAM capacity, and PCIe bandwidth as one system instead of relying on a fixed split.
Run ft bench bw once per machine before relying on hybrid execution. The resulting bandwidth profile helps FreeToken estimate whether fetching or CPU computation is more suitable for cache misses.
FreeToken Setup Steps for a Supported Checkpoint
Before setup, confirm that the host meets the documented direction: Linux, an x86-64 computer, an NVIDIA GPU, CUDA 13, and a recent driver. The project highlights RTX 30, RTX 40, and RTX 50 series cards, but actual performance depends on model size, quantization, RAM, VRAM, CPU speed, and PCIe bandwidth.
Verify the checkpoint
Start with a model listed in the official documentation, such as a supported DeepSeek-V4 or GLM-5.2 Hugging Face checkpoint. For GGUF, confirm that the architecture is explicitly supported; the documented native example is Gemma-4.
Check memory capacity
Make sure the complete checkpoint can be stored across VRAM and system RAM. An 8 GB GPU does not reduce the model’s total storage requirement to 8 GB.
Benchmark the machine
Use ft bench bw to measure the local memory and transfer behavior. This helps determine whether offload, cpu, or hybrid is the better MoE path.
Prepare the optional fast-load format
Run the documented checkpoint conversion only if faster loading is useful. Conversion is optional, and the serving command can detect the resulting format automatically.
Launch and monitor
Start the model with ft serve --model and observe memory use, first-token delay, generation speed, and stability during long contexts or repeated agent requests.
| Setup check | What to confirm | Why it matters |
|---|---|---|
| Operating system | Linux-focused accelerated path | Current instructions are not a universal desktop setup |
| GPU | NVIDIA card with suitable driver | Documented acceleration centers on NVIDIA and CUDA |
| CUDA | CUDA 13 | Required by the documented command-line direction |
| System RAM | Enough for offloaded weights | Remaining model weights must live somewhere |
| Checkpoint | Supported architecture and format | Loading depends on runtime support, not only file extension |
Test model loading and a short prompt before beginning long-context or agent workloads. This catches checkpoint, driver, and memory problems without committing to a lengthy session.
Performance, Memory, and llama.cpp Comparison
FreeToken’s strongest case is not that it replaces every local runtime. Its specialization is running large MoE models when their total weights exceed VRAM. The engine can cache active experts, overlap transfers with computation, and choose between GPU movement and CPU work based on the host machine.
Reported tests described in the available material include Qwen3.6-35B-A3B at roughly 77–83 tokens per second on an RTX 5090, and DeepSeek-V4 Flash at roughly 22–25 tokens per second in tested workloads. An RTX 4060 laptop with 8 GB of VRAM and 32 GB of system memory reportedly reached 39.3 tokens per second with an official 4-bit Qwen checkpoint. These are reported results, not a guarantee for every configuration.
| Workload condition | FreeToken’s relevance | Comparison note |
|---|---|---|
| Model fits entirely in VRAM | Lower | A mature runtime may already be very fast |
| MoE model exceeds VRAM | High | Expert caching and offload become central |
| Long agent context | High | Repeated tool calls can expose transfer delays |
| CPU-only operation | Limited | FreeToken is not positioned as a universal CPU runner |
| Apple Silicon setup | Unclear | No comparable path is documented in the supplied material |
The comparison with llama.cpp depends on the workload. llama.cpp supports a much broader range of operating systems, processors, GPUs, and model formats, including a large GGUF ecosystem. FreeToken is younger and narrower, but that narrower focus can be useful for oversized NVIDIA-based MoE deployments.
For an agent, generation speed is only one metric. First-token delay, context reuse, tool-call turnaround, and long-session stability can matter more than a short synthetic prompt. A runtime that avoids multi-minute stalls may feel more responsive even when headline tokens-per-second numbers look similar.
Treat published and community numbers as configuration-specific. Compare the same model, quantization, prompt length, context size, hardware, backend, and memory layout before drawing conclusions.
Validation Checklist and FAQ
Use this checklist when evaluating a FreeToken gguf workflow or any supported checkpoint. The goal is to confirm compatibility and system fit before tuning performance.
Preflight Checks:
- Confirm the model architecture appears in FreeToken documentation
- Verify whether the checkpoint uses Hugging Face safetensors or supported native GGUF
- Measure system RAM, GPU VRAM, and available storage
- Run ft bench bw before selecting hybrid execution
- Test a short prompt before long-context or agent workloads
| Signal | Healthy result | Action if it fails |
|---|---|---|
| Model loading | Checkpoint initializes without format errors | Recheck architecture and checkpoint layout |
| VRAM use | Cache and context leave operating headroom | Reduce cache pressure or choose another backend |
| First-token delay | Stable across repeated requests | Inspect transfers, context length, and RAM pressure |
| Generation rate | Consistent for the selected workload | Compare backend choices using the same prompt |
| Long context | No severe stalls or memory exhaustion | Reduce context size or review system capacity |
Q: Does FreeToken support every GGUF model?
No universal support is established. The documentation specifically identifies native GGUF support for Gemma-4, while most listed checkpoints use Hugging Face safetensors. Check architecture support before using another GGUF file.
Q: Is FreeToken better than llama.cpp for every computer?
No. llama.cpp has broader hardware, operating system, and GGUF coverage. FreeToken is more specialized for large MoE models that exceed GPU VRAM on supported NVIDIA systems.
Q: Can an 8 GB GPU run a model larger than 8 GB?
It can be possible when enough system RAM holds the remaining weights and the checkpoint is supported. The full model still requires its total storage across the computer.
Q: What should I do if a DeepSeek-V4 checkpoint fails?
Confirm that the inference/config.json subdirectory remains present, because the documented model arguments are read from that location.
Start with documented checkpoints, validate memory requirements, and benchmark the actual machine. Format support and hardware behavior matter more than the GGUF label alone.
Official FreeToken model documentation is the reference point for supported checkpoints, backend flags, conversion notes, and model-specific requirements.