Which quantisation should I choose to run an LLM locally?

What Quantisation Does to Your Model

Quantisation compresses a model's weights from higher-precision formats like FP16 to lower-bit representations such as INT8 or INT4. Each weight occupies fewer bits, so the model fits in less RAM and inference gets faster because memory bandwidth, not raw compute, is usually the bottleneck. The cost is rounding error: every quantised weight is an approximation of the original, and those small errors accumulate through layers, subtly shifting token probabilities and occasionally producing less coherent or less accurate output.

At 8-bit, degradation is barely measurable for most tasks. At 4-bit, you save the most memory but will notice softer reasoning, more repetition, and weaker performance on long-context or multilingual work. The sweet spot for many local setups is 5 or 6 bits, where quality loss stays marginal and the footprint drops enough to run on consumer GPUs. Molly takes this further by keeping one quantized base and routing each request to a small LoRA specialist, so you trade almost nothing for the convenience of a single local system.

Memory Savings vs. Quality Loss

Choosing a lower bit-width directly reduces VRAM consumption, allowing larger models to fit on consumer hardware. An 8-bit quantization roughly halves memory requirements compared to 16-bit floats with negligible quality degradation. Dropping to 4-bit cuts memory by 75 percent, enabling 7-billion parameter models to run on 8GB GPUs. However, this aggressive compression increases perplexity and degrades nuanced reasoning, particularly in complex mathematics or coding tasks where precision is critical.

To mitigate the quality loss inherent in aggressive 4-bit quantization, architectural efficiency matters. Molly is an orchestrator that runs on your own hardware, keeping a library of small LoRA domain specialists over one quantized base and routing each request to the right one. This approach preserves the memory savings of a single heavily compressed model while recovering specialized accuracy, effectively sidestepping the traditional trade-off between broad capability and local hardware limits.

Comparing GGUF, AWQ, GPTQ, and EXL2

GGUF offers the broadest hardware compatibility, running on CPU, GPU, or a mix of both through llama.cpp. Its k-quant variants let you trade precision for size in fine increments, making it the safest default when you are unsure of your setup. AWQ and GPTQ are GPU-centric weight-only formats; AWQ generally preserves slightly more accuracy at equivalent bit widths by protecting salient weights, while GPTQ is older and more widely supported across inference engines.

EXL2, used with ExLlamaV2, supports per-layer mixed-precision bit rates, letting you target a specific model size while maximising quality where layers are sensitive. It is fast on modern GPUs but offers no CPU fallback. If your stack routes requests across specialists on local hardware, GGUF's flexibility may matter more than EXL2's peak throughput, since not every workload can assume a dedicated GPU.

Matching Formats to Your Hardware

Your GPU's VRAM is the hard ceiling. A 4-bit quantisation like GGUF Q4_K_M fits roughly 1.5 GB per billion parameters, so an 8B model needs around 12 GB to load with context. If you have 24 GB, you can run that same model at Q8 or step up to a 13–14B at Q4. Below 4-bit, perplexity rises noticeably and output quality degrades; above 8-bit, you gain almost nothing perceptible but halve your headroom for context and batch size.

For mixed workloads, pick the format your runtime actually supports natively. GGUF works well across CPU and GPU with llama.cpp, while AWQ and GPTQ are optimised for specific CUDA kernels and offer faster inference on supported cards but break entirely on unsupported ones. If your hardware varies or you need multiple domains served from one footprint, an orchestrator like Molly keeps one quantized base on disk and swaps small LoRA adapters per request, avoiding the cost of loading several full models.

Common questions

How does quantisation affect model quality beyond what benchmarks show?

Quantisation mainly degrades performance on nuanced reasoning, long-context coherence, and rare-token generation. Perplexity scores may look fine, but edge-case behaviour—multilingual output, code precision, or subtle instruction-following—often degrades more than aggregate metrics suggest. Always test on your actual workload, not just standard benchmarks.

Which quantisation formats work best for CPU-only versus GPU setups?

GGUF (via llama.cpp) is the dominant choice for CPU and mixed CPU/GPU inference, offering flexible k-quants that offload layers efficiently. For GPU-only setups, AWQ and GPTQ provide faster inference with minimal quality loss, while EXL2 offers fine-grained bit-rate control. Match the format to your hardware path.

Is there a practical threshold below which quantisation becomes unusable?

Below 3 bits per weight, quality drops sharply for most models. 4-bit quantisation (Q4_K_M or AWQ 4-bit) is widely considered the sweet spot, preserving nearly all capabilities while halving memory versus FP16. At 2-bit or 3-bit, expect noticeable degradation unless the base model is exceptionally large.