The Minimum Viable Baseline for Local LLMs

Running large language models locally requires compression. The practical question is how far you can go before quality becomes unusable.

Across benchmarks and deployment experience, a clear baseline has emerged: 4-bit weight quantization combined with 8-bit KV cache. This combination preserves roughly 92–99% of full-precision (BF16) quality and is the current minimum viable floor for most practical applications.

The quality spectrum is consistent:

•⁠ ⁠FP8 → ~99%

•⁠ ⁠INT8 → ~97–98%

•⁠ ⁠Q5_K_M → ~95–97%

•⁠ ⁠Q4_K_M → ~92–94%

•⁠ ⁠Q2_K → ~80–85% (noticeable degradation)

Going below 4-bit weights or 8-bit KV cache moves into research-grade territory. Quality loss becomes especially visible on reasoning, mathematics, and code tasks.

Weights and KV cache must both be addressed. At longer context lengths, the KV cache often consumes more memory than the model weights themselves. Compressing only one of them is incomplete.

This Ledger Entry expands how the reader thinks by replacing vague advice about quantization with a concrete, evidence-based decision boundary for local model deployment.