FreeToken: Making the Local Machine an Elastic Inference Platform

Most local serving systems treat a personal computer as a small GPU that must fit the model. FreeToken starts from a different premise. The machine is a unified, elastic inference platform whose CPU, GPU, and interconnect can be orchestrated continuously as workloads and available resources change.

The system is designed for Mixture-of-Experts models and for agentic workloads that repeatedly edit context and call tools. Its core techniques include bandwidth-adaptive execution (deciding in real time whether missing experts should be moved to the GPU or run on the CPU), semantic-aware state caching at natural boundaries, full-layer double buffering during prefill, and elastic reallocation of GPU memory between KV cache and experts.

The practical result is that machines people already own can serve models far larger than their VRAM. A 35B model runs on a laptop. A 284B model runs on a gaming desktop. A 753B model runs on a single workstation GPU. Throughput remains usable across real coding and tool-using agent workloads.

This Ledger Entry expands how the reader thinks by shifting the problem of local AI from “make the model smaller” to “orchestrate the heterogeneous machine as a single system.”