FreeToken rtx 3090: Setup Guide & Performance Tips

FreeToken is a local AI serving system rather than a game or entertainment title. Its purpose is to make large open-weight mixture-of-experts models more practical on consumer hardware by combining GPU memory, system RAM, CPU execution, and PCIe transfers.

An RTX 3090 provides 24 GB of VRAM. That is enough for many smaller local models, but it is not enough to hold the complete expert pool of models such as DeepSeek V4 Flash. FreeToken addresses this limitation by keeping the full expert weights in host memory while using the GPU as an elastic cache for frequently routed experts.

The result is a hybrid workflow:

• Non-expert model weights remain on the GPU.
• The complete expert pool resides in system memory.
• Frequently used experts are cached in VRAM.
• Missing experts can be transferred to the GPU or processed on the CPU.
• The runtime adjusts its behavior according to measured PCIe and host-memory bandwidth.

This design makes the FreeToken rtx 3090 combination useful for experimentation with models that would otherwise require a much larger GPU or a hosted API.

Reference: The FreeToken research paper describes the bandwidth-adaptive execution model, elastic expert cache, and edge-oriented serving design.

Video Highlights:

• A single RTX 3090 runs DeepSeek V4 Flash with system-memory offload.
• Server-side testing reached approximately 10–11 tokens per second.
• The desktop client measured approximately 8.8 tokens per second in a comparable test.
• The setup used a browser-based chat interface through Open WebUI.
• Memory capacity and bandwidth had a major effect on the experience.

The RTX 3090 is only one part of the system. FreeToken’s model capacity depends heavily on usable system RAM, host-memory bandwidth, storage speed, and the amount of VRAM available after other applications claim resources.

A practical starting point is 32 GB of system memory, although 64 GB is a more comfortable target for larger MoE experiments. The reference testing also examined systems with substantially more RAM for models requiring a much larger host-resident expert pool. A model may report insufficient memory even when the GPU itself appears underused, because FreeToken evaluates usable RAM and VRAM together.

Memory bandwidth is also important. The reference material compares dual-channel DDR4 and DDR5 systems, with DDR5 generally offering higher host bandwidth. However, specifications alone do not determine performance because FreeToken measures the deployed machine’s actual transfer and CPU-processing behavior.

Before launching a model, check the following:

• Available system RAM after the operating system and background applications are accounted for.
• Free VRAM on the RTX 3090.
• PCIe link width and generation.
• RAM channel configuration and effective memory speed.
• NVMe capacity for model files and converted weights.
• CPU thermal and power limits during sustained inference.

Use this workflow to prepare a clean first test. The available package and interface may vary as the project develops, so verify the current release instructions before installation.

A clean first run is more useful than an overloaded one. Do not start with a model that already approaches the limits of your memory budget, because an early failure may be caused by allocation pressure rather than a software defect.

Performance depends on model architecture, quantization, prompt length, active experts, RAM speed, PCIe behavior, and whether the runtime is running as a server or desktop application. The available RTX 3090 test results should therefore be treated as practical reference points rather than guaranteed specifications.

In one server-side test, DeepSeek V4 Flash generated approximately 10–11 tokens per second on a single RTX 3090. The desktop client measured approximately 8.8 tokens per second in a comparable configuration. The difference suggests that the interface and runtime path can affect throughput, although the exact result will vary by operating system and system configuration.

The paper’s design explains why decode speed can remain usable despite the model exceeding VRAM. FreeToken maintains a shared LRU expert cache, allowing recently routed experts to remain on the GPU. For misses, the runtime calculates a split between PCIe cache fills and direct CPU execution based on measured bandwidth.

Several factors can reduce results:

• OBS or other GPU encoding workloads may affect available memory.
• Browser tabs and desktop applications can consume VRAM.
• Slow or single-channel RAM can constrain CPU-side expert processing.
• Longer prompts increase prefill work and context-management costs.
• Dense models may behave differently from sparse MoE models.
• Beta software issues can cause model-specific startup or engine failures.

FreeToken’s hybrid design introduces more variables than a GPU-only local model. Troubleshooting should begin with memory and environment checks before changing advanced runtime settings.

If a model fails to start, reduce competing memory use first. A model-specific engine error does not necessarily mean the RTX 3090 is defective. The reference testing found that a dense BF16 model could fail while another MoE model loaded successfully under the same general setup.

If generation is slower than expected, inspect host-memory bandwidth and CPU activity. FreeToken may need to process more misses on the CPU when the GPU cache is small or the routing pattern changes. Increasing RAM capacity helps compatibility, while faster memory and a stronger platform can improve the rate at which host-resident experts are processed.

The FreeToken project description and paper identify the system as research-oriented software released through the project’s official channel. Check the current project page at flashml.ai for release information, supported models, and platform updates.