- FreeToken edge native moe serving coordinates CPU and GPU resources for large mixture-of-experts models.
- Bandwidth-aware execution adapts computation to each machine’s memory and interconnect limits.
- Double buffering overlaps data movement with computation during the prefill stage.
- Adaptive decoding responds to expert-cache misses instead of relying on fixed placement.
- Reported results include up to 40 tokens per second on a laptop and 15 on a workstation for a 753B model.
FreeToken edge native moe serving explained
FreeToken edge native moe serving is a research system for running very large mixture-of-experts, or MoE, language models across ordinary local hardware. Instead of treating limited GPU memory as an automatic blocker, the design divides work between the graphics processor, central processor, host memory, and available interconnects.
The central idea is resource orchestration. A consumer computer may have a capable GPU but insufficient graphics memory for a frontier-scale model. Moving every operation through the same slow path creates stalls. FreeToken instead adapts execution to bandwidth, cache state, and the current phase of inference.
The project was published on arXiv on August 17, 2026, under the title FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution. The listed authors include Shuo Yang, Xiaoze Fan, Melissa Pan, Haocheng Xi, Zhe Wang, Shanlin Sun, Kurt Keutzer, Song Han, Matei Zaharia, Chenfeng Xu, and Ion Stoica.
Video Highlights:
- Large MoE models can be distributed across CPU and GPU resources.
- Prefill uses overlapping transfers, computation, and checkpointing.
- Decode adapts expert loading when cache misses occur.
- Reported tests span an 8 GB laptop through a 96 GB workstation.
- Agent workloads benefit from faster decoding and shorter response-start delays.
| Concept | FreeToken approach | Practical effect |
|---|---|---|
| GPU memory limit | Distribute model work across GPU, CPU, and host memory | Makes larger models more practical on local systems |
| Slow transfers | Adapt execution to measured bandwidth | Reduces avoidable idle time |
| Expert loading | Use cache-aware placement and dynamic loading | Limits repeated movement of active experts |
| Long prompts | Overlap transfer and computation during prefill | Improves prompt-processing throughput |
| Agent edits | Keep checkpoints at token anchors | Avoids unnecessary full re-prefill work |
Treat FreeToken as a systems and inference research project, not a conventional end-user application. Its main contribution is how resources are coordinated during model serving.
Bandwidth Adaptive
Execution changes according to CPU memory bandwidth, GPU capacity, and the connection between them.
Cache Aware
Recently used experts remain available when possible, reducing the cost of repeated expert transfers.
Pipeline Oriented
Data movement and computation are arranged to overlap instead of waiting in a single sequential queue.
Architecture and inference phases
FreeToken separates inference into two important phases: prefill and decode. Prefill processes the input context, while decode generates new tokens one at a time. These phases have different performance pressures, so using one fixed scheduling policy for both can leave hardware underused.
During prefill, the system uses full-layer double buffering. While one portion of the model is being computed, another portion can be transferred or prepared. This arrangement is especially important when expert weights cannot remain permanently resident in GPU memory.
The system also maintains state checkpoints at special token anchors. In agent workflows, a user or tool may edit part of the conversation. Rather than rebuilding the entire prompt state after every edit, checkpoints can help preserve reusable computation and reduce repeated prefill work.
| Inference phase | Main challenge | FreeToken technique | Expected benefit |
|---|---|---|---|
| Prefill | Processing long prompts while moving model data | Full-layer double buffering | Better overlap between transfer and computation |
| Prefill | Recomputing after agent edits | Token-anchor state checkpoints | Lower repeated prompt-processing cost |
| Decode | Expert weights may miss the cache | Adaptive miss-handling policy | More balanced CPU and GPU utilization |
| Decode | CPU and GPU can wait on different resources | Dynamic expert loading and in-place CPU work | Fewer avoidable idle periods |
The decode stage uses a different strategy. When an expert is missing from the active cache, FreeToken can balance loading that expert over the interconnect with computation performed in place on the CPU. The policy is designed to keep neither side idle when the other can continue useful work.
This distinction matters because MoE models do not activate every expert for every token. The system must identify which experts are needed, determine where they are available, and choose whether moving or locally processing the work is more efficient.
Large parameter count does not directly equal the amount of computation used for every token in an MoE model. However, the full model still creates substantial storage and movement demands, which FreeToken addresses through scheduling.
Prepare the active context
Prefill the prompt while arranging model data transfers and computation through the double-buffered pipeline.
Record reusable state
Preserve checkpoints at selected token anchors so suitable agent edits can reuse previous computation.
Track expert availability
Monitor which experts are already cached and identify misses during token generation.
Select the execution path
Balance expert loading across the bus with in-place CPU execution according to current resource conditions.
Continue adaptive decoding
Reevaluate placement as cache state and workload conditions change instead of keeping one static policy.
Hardware coverage and performance profile
FreeToken was evaluated across varied local configurations rather than one idealized machine. The reported test range extends from an 8 GB laptop to a 96 GB workstation. These systems differ substantially in host-memory bandwidth and interconnect performance, making adaptive scheduling important.
One compact laptop configuration had interconnect bandwidth below 12 GB per second. At the other end, the strongest workstation configuration reached 178 GB per second of CPU memory bandwidth. Those differences affect whether an expert should be moved to the GPU, processed on the CPU, or retained in a cache for later use.
| Test environment | Reported hardware characteristic | Why it matters |
|---|---|---|
| Compact laptop | 8 GB memory class; interconnect below 12 GB/s | Transfer delays become a major scheduling constraint |
| RTX 4060 laptop | Portable GPU configuration | Tests whether coding-agent workloads remain practical |
| RTX 5090 desktop | High-end desktop GPU | Demonstrates faster agent decoding and prompt handling |
| Large workstation | Up to 96 GB system memory; CPU bandwidth up to 178 GB/s | Provides more room for host-side execution |
| Workstation MoE test | 753B-parameter model | Shows the system’s behavior on an unusually large model |
The reported results vary by workload and hardware. On one laptop setup, FreeToken approached 40 tokens per second while serving a 753 billion parameter model. On a workstation, the same broad test category reached about 15 tokens per second for that model and was reported as approximately twice the throughput of comparison serving engines.
On an RTX 5090 desktop, coding-agent workloads exceeded 76 tokens per second in the reported tests. Another workload involving a large model exceeded 22 tokens per second, while a separate model reached more than 80 tokens per second. These figures should be read as benchmark observations tied to specific configurations, model variants, and workloads rather than universal device guarantees.
| Workload or configuration | Reported FreeToken result | Interpretation |
|---|---|---|
| Laptop with large MoE model | Nearly 40 tokens/s | Shows the value of adaptive local execution |
| Workstation with 753B model | About 15 tokens/s | Reported at roughly twice rival-engine throughput |
| RTX 4060 coding-agent test | More than 39 tokens/s | Demonstrates strong performance on a mobile GPU setup |
| RTX 5090 desktop coding-agent test | More than 76 tokens/s | Indicates higher throughput on a powerful desktop |
| Long-prompt prefill | More than 6,600 tokens/s at 16,000 tokens | Highlights pipeline scaling during prefill |
The results demonstrate that hardware coordination can change the practical serving limit. They do not mean every laptop or workstation will reproduce the same throughput.
Caching, misses, and tuning priorities
Caching is one of FreeToken’s most important performance mechanisms. Because MoE inference activates selected experts, a useful cache can prevent repeated transfers. A poor placement policy increases misses, forcing the system to move or recompute data at inconvenient times.
The reported evaluation compares a least-recently-used cache policy with static and prefill-based placement strategies. FreeToken’s least-recently-used policy substantially reduced expert-cache misses across the tested models. This approach is intuitive for changing workloads: experts used recently are more likely to remain useful during nearby decoding steps, although workload behavior can vary.
| Policy | Placement behavior | Strength | Risk |
|---|---|---|---|
| Least recently used | Retains recently accessed experts | Adapts to changing token demand | May not predict a sudden workload shift |
| Static placement | Keeps a predetermined expert layout | Simple and predictable | Can waste space when demand changes |
| Prefill-based placement | Uses prompt-stage activity to guide later placement | Connects initial context to cache setup | May become stale during long decoding |
| Adaptive execution | Chooses CPU, GPU, or transfer paths dynamically | Responds to current bandwidth and misses | Requires more runtime coordination |
For long prompts, prefill throughput reportedly scaled beyond 6,600 tokens per second at a 16,000-token context in the tested setup. The double-buffered pipeline was compared with non-pipeline execution and baseline systems, with the overlapping design showing stronger throughput in the reported measurements.
A practical tuning order follows the system’s architecture:
- Measure host-memory bandwidth and interconnect behavior before choosing a fixed placement.
- Separate prefill analysis from decode analysis because their bottlenecks differ.
- Watch expert-cache misses instead of judging performance only by total model size.
- Preserve reusable state in agent workflows where prompts are edited repeatedly.
- Compare CPU-side execution and transfer costs under the actual workload.
Serving Review Checklist:
- Identify available GPU memory and host-memory capacity
- Measure the effective CPU-GPU interconnect bandwidth
- Check prefill throughput separately from decode throughput
- Monitor expert-cache misses during representative workloads
- Record time to first token and steady-state token throughput
Use representative prompts, agent actions, and model variants. A short chatbot prompt may hide the transfer costs that appear during long-context or tool-using workloads.
Use cases, limits, and research context
FreeToken is most relevant to researchers, infrastructure engineers, and advanced local-inference users interested in edge-native model serving. Its reported agent workloads include coding-oriented tasks, where time to first token and sustained decode speed both affect usability.
The system also emphasizes that serving massive MoE models is not only a memory-capacity problem. Memory remains important, but the pathway between storage locations, the speed of host computation, cache behavior, and scheduling decisions can determine whether available hardware is used efficiently.
Reported time-to-first-token results stayed within a few seconds in the highlighted RTX 5090 agent tests, while comparison systems sometimes required much longer or did not complete successfully. This measure is distinct from decode throughput: a system may generate tokens quickly after a slow start, or begin quickly but sustain lower output speed.
| Metric | What it measures | Why it matters |
|---|---|---|
| Time to first token | Delay before generation begins | Important for interactive assistants and agents |
| Decode throughput | Generated tokens per second | Indicates sustained response speed |
| Prefill throughput | Context tokens processed per second | Matters for long prompts and tool histories |
| Cache-miss rate | Frequency of unavailable expert data | Reveals transfer and placement pressure |
| Resource utilization | CPU and GPU activity during serving | Shows whether one processor is sitting idle |
FreeToken does not remove the need for suitable hardware, model support, or careful evaluation. Performance depends on memory capacity, bandwidth, interconnect speed, model structure, prompt length, and the behavior of the serving workload. The published material also describes a research evaluation rather than a universal compatibility list.
For technical readers, the primary reference is the FreeToken arXiv paper, listed as arXiv:2608.16157 in distributed, parallel, and cluster computing. The paper’s August 2026 publication makes it the appropriate starting point for implementation details, experimental methodology, and the authors’ formal terminology.
Start with the architecture, then reproduce a small prefill or cache experiment before attempting a full large-model serving deployment.
Q: What is FreeToken edge native moe serving?
It is a research approach for serving large mixture-of-experts language models on local or edge hardware by coordinating CPU computation, GPU computation, host memory, transfers, and expert caching.
Q: Why does FreeToken use different policies for prefill and decode?
Prefill processes the input context in bulk, while decode generates tokens sequentially and may encounter expert-cache misses. Their bottlenecks differ, so FreeToken uses overlapping pipeline execution for prefill and adaptive balancing during decode.
Q: What hardware does FreeToken target?
The reported evaluation covers systems ranging from an 8 GB laptop to a 96 GB workstation, including RTX 4060 laptop and RTX 5090 desktop configurations. Results depend on the exact hardware and workload.
Q: Does FreeToken guarantee a specific token-per-second result?
No. The published figures are measurements from selected models, devices, and workloads in 2026. Actual throughput can change with memory, interconnect bandwidth, prompt length, cache behavior, and scheduling conditions.