- FreeToken tokens per second depends on GPU cache hits, system memory, and PCIe bandwidth.
- RTX 3090 testing reached approximately 10–11 decode tokens per second in a local setup.
- Desktop client results measured about 8.8 tokens per second in comparable testing.
- Model choice matters because MoE and dense models create very different memory demands.
- Best improvement usually comes from faster memory, more usable RAM, and reduced background load.
FreeToken tokens per second: What the Metric Means
FreeToken tokens per second describes decode throughput: the speed at which a locally served language model generates output after prompt processing. It is different from time to first token, prompt-processing speed, and total response time. For interactive chat, decode throughput is the number users usually notice while an answer is appearing.
FreeToken is an edge-native serving system for large mixture-of-experts models. Its design keeps the complete expert pool in host memory while using available GPU memory as an elastic expert cache. The result is that performance depends on the entire machine, not only the graphics card.
Video Highlights:
- A single RTX 3090 setup produced roughly 10–11 decode tokens per second.
- The desktop client reported approximately 8.8 tokens per second during a later test.
- System RAM capacity and bandwidth were identified as major performance factors.
- Background applications may reduce available VRAM and affect throughput.
| Metric | Meaning | Why It Matters |
|---|---|---|
| Decode tokens per second | Generated output rate | Indicates conversational responsiveness |
| Prompt processing speed | Input-token processing rate | Affects startup delay for long prompts |
| Time to first token | Delay before output begins | Important for agents and large contexts |
| Total response time | Full request duration | Combines prompt and decode costs |
The published FreeToken research paper explains that the system adapts execution to measured host-memory and PCIe bandwidth. This is why two computers with the same GPU can produce noticeably different results.
Treat tokens per second as a workload measurement rather than a fixed product specification. Change the model, context length, thinking mode, or memory conditions, and the result may change substantially.
Observed FreeToken Performance by Setup
The available testing shows a practical gap between server-side and desktop-client measurements. On a single RTX 3090, DeepSeek V4 Flash stayed near 10 to 11 tokens per second during ordinary chat prompts. A separate desktop-client run reported 8.8 tokens per second. These figures are useful reference points, but they are not universal ratings.
The difference can come from client overhead, operating-system behavior, model settings, background processes, or measurement variation. Thinking mode also changes the workload. A response generated with maximum reasoning can have a lower visible output rate and a longer completion path than a short direct answer.
| Setup | Model or Mode | Reported Decode Rate | Practical Reading |
|---|---|---|---|
| Single RTX 3090 | DeepSeek V4 Flash, local server | About 10–11 tok/s | Suitable for interactive chat |
| Desktop client | DeepSeek V4 Flash, thinking enabled | About 8.8 tok/s | Easy setup, somewhat slower result |
| Single RTX 4090 comparison | DeepSeek V4 Flash | No fixed result provided | Use as a hardware comparison target |
| RTX PRO 6000 research tier | GLM-5.2 NVFP4 | 14.9 tok/s | Frontier-scale demonstration |
| RTX 5090 research tier | DeepSeek V4 Flash | 22–25 tok/s | Research benchmark, not a 3090 expectation |
The research evaluation reports higher results on newer hardware because it uses an optimized serving runtime, measured bandwidth scheduling, and specific benchmark conditions. Those figures should not be copied directly into a consumer desktop estimate.
Interactive Chat
Around 8–11 tok/s can feel responsive for ordinary local conversations, depending on prompt and model behavior.
Agent Workloads
Tool calls and repeated long prompts make time to first token more important than decode rate alone.
Reasoning Mode
Extended thinking may lower visible speed and increase total completion time, even when hardware remains unchanged.
A useful comparison method is to test the same model, prompt style, context size, and generation settings on each configuration. Record several responses instead of relying on a single completion. MoE routing changes from token to token, so individual results can fluctuate.
A desktop-client result, a server-side measurement, and a research benchmark may use different runtimes and settings. Compare them directionally unless every test condition matches.
Hardware Factors That Change Throughput
FreeToken shifts much of the performance discussion from raw GPU compute to memory movement. Mixture-of-experts models activate only part of their parameters for each token, but the full expert pool may still need to remain available in system memory. When an expert is not cached on the GPU, the runtime can transfer it over PCIe or execute it directly on the CPU.
That makes system memory capacity essential. The testing indicates that 32 GB may be a starting point for some local configurations, while 64 GB provides a more comfortable margin. Larger models can require far more. The research material describes deployments requiring hundreds of gigabytes of host memory, especially for frontier-scale models.
| Hardware Factor | Effect on FreeToken | Recommended Interpretation |
|---|---|---|
| GPU VRAM | Determines expert-cache capacity | More VRAM can reduce cache misses |
| System RAM capacity | Holds host-resident experts | Insufficient usable RAM prevents loading |
| RAM bandwidth | Feeds CPU execution and transfers | Faster memory can improve decode performance |
| PCIe link width | Controls expert transfer rate | x16 generally offers more headroom than x8 |
| Background GPU use | Reduces available VRAM | Browsers, recording tools, and games may interfere |
| Storage speed | Influences startup and model loading | NVMe storage can reduce initial loading delay |
The research paper reports approximate platform-level differences for expert transfer: RTX 4090- and RTX 3090-class systems using PCIe 4.0 x16 can take several seconds to move large expert sets during prefill, while narrower laptop links may take longer. Decode behavior is different because caching and CPU co-execution can hide some transfer cost.
For a single-GPU FreeToken system, prioritize usable RAM capacity first, then memory bandwidth and PCIe conditions. Additional GPU compute alone may not remove a host-memory bottleneck.
A clean test environment also matters. Close unnecessary GPU-heavy software, avoid running multiple model services simultaneously, and leave enough RAM for the operating system. If the runtime reports insufficient RAM or VRAM, lowering expectations about tokens per second will not solve the loading problem; the model configuration must fit first.
Step-by-Step FreeToken Benchmark Setup
Use this process to measure a reliable local baseline. The goal is not to chase one unusually high number, but to create a repeatable test that helps explain changes.
Record the Machine
Write down the GPU model, VRAM, system RAM, RAM generation and speed, operating system, PCIe link, and storage type. These details explain many differences between results.
Choose One Model
Start with the same supported model for every comparison. DeepSeek V4 Flash is the clearest reference in the available single-GPU testing, while dense models may behave differently.
Prepare a Clean Session
Close competing GPU workloads and background inference services. Keep the model, context size, temperature, thinking mode, and sampling settings consistent.
Warm Up the Runtime
Send one short prompt before recording results. The first response may include loading and cache-warming costs that do not represent steady-state decode speed.
Average Several Runs
Run at least three comparable prompts and record the displayed decode rate, prompt-processing rate, and time to first token. Use the median or average rather than the highest result.
| Test Variable | Keep Constant | Example Record |
|---|---|---|
| Model | Same checkpoint and quantization | DeepSeek V4 Flash |
| Prompt | Similar length and task | Short factual question |
| Generation | Same thinking and sampling mode | Thinking enabled or disabled |
| Runtime | Same client or server path | Desktop client or API server |
| Result | Record multiple runs | 10.5, 10.1, 10.8 tok/s |
When comparing the desktop client with a server-connected interface, test them separately. The available results suggest the desktop path can be convenient while producing a lower measured rate. That does not automatically mean the model is slower in every workload; it means the interface and runtime path should be documented.
Record both the displayed rate and the response conditions. A number without model, context, runtime, and thinking-mode details is difficult to reproduce or compare.
Optimization Checklist and Common Limits
FreeToken’s strongest advantage comes from coordinating GPU memory, host memory, CPU execution, and PCIe transfers. The practical tuning approach is therefore to remove avoidable contention before changing models or expecting a higher output rate.
Performance Checklist:
- Confirm that the model fits within usable system RAM and VRAM
- Use the fastest stable RAM configuration available on the system
- Close unnecessary GPU-heavy applications before benchmarking
- Keep model, prompt length, and thinking settings consistent
- Run multiple warm tests and record median tokens per second
| Symptom | Likely Cause | First Action |
|---|---|---|
| Model fails to start | Not enough usable RAM or VRAM | Choose a smaller model or add memory |
| Rate varies widely | Expert routing and cache misses | Average several runs |
| Desktop result is lower | Client or runtime overhead | Compare with the server path |
| Long initial delay | Prefill or expert loading cost | Measure time to first token separately |
| Other applications slow inference | Shared GPU or memory resources | Close background workloads |
The available testing also identifies software maturity as a consideration. FreeToken is described as beta software, and one dense Qwen configuration exited unexpectedly during testing. A failed model launch should be treated as a compatibility or resource issue rather than as a tokens-per-second result.
For troubleshooting, check logs, verify the model’s supported format, confirm available memory after the operating system and other applications are accounted for, and repeat the test after restarting the runtime. Avoid interpreting one failed launch as proof that every model configuration is incompatible.
A model that fits conceptually may still fail because of precision format, runtime support, memory registration, or available host resources. Confirm compatibility before comparing speed.
Q: What is a good FreeToken tokens per second result on an RTX 3090?
The available single-GPU testing reached approximately 10 to 11 decode tokens per second with DeepSeek V4 Flash. Treat that as a practical reference, not a guaranteed rating.
Q: Why did the desktop client report about 8.8 tokens per second?
The desktop client may include different runtime behavior, interface overhead, memory conditions, or settings. The result is directionally consistent with interactive local inference but should be compared under matching conditions.
Q: Does more system RAM always increase tokens per second?
More RAM mainly determines whether a model can load and how much cache space remains available. Once capacity is sufficient, RAM bandwidth and PCIe behavior may matter more for throughput.
Q: Why can two computers with the same GPU perform differently?
FreeToken uses host memory and PCIe transfers when experts are not resident in VRAM. RAM bandwidth, PCIe link width, background applications, operating system behavior, and runtime choice can all change the result.