How much VRAM do I actually need to run a 70B model?
What a 70B Model Actually Requires
At 16-bit precision, a 70B parameter model needs roughly 140 GB of VRAM just to load the weights. Quantize to 8-bit and you drop to about 70 GB. At 4-bit, you're around 35 to 40 GB. These are floor numbers — they do not account for the KV cache, which scales with context length and batch size. A 4K context window in 4-bit can easily add another 4 to 8 GB depending on architecture.
This means a single 24 GB consumer card cannot run a 70B model without offloading layers to system RAM, which tanks throughput. Two such cards in 4-bit can work but leave little headroom. The honest trade-off: you either spend on an 80 GB professional card and run 4-bit comfortably, or you accept CPU offloading and slow inference. Molly sidesteps this by keeping one quantized base on the GPU and routing each request to the right small LoRA specialist, so a single card serves many domains without reloading full models.
How Quantization Changes the Math
A 70B model at FP16 needs roughly 140 GB of VRAM just to hold the weights, which puts it out of reach for most consumer setups. Quantization to 4-bit brings that down to around 38–40 GB, fitting on a single 48 GB card or two 24 GB cards with room for the KV cache. INT8 lands in the middle at about 70 GB, splitting the difference between fidelity and footprint.
The trade-off is real but smaller than people fear. Going from FP16 to 4-bit typically costs 1–3 percentage points on benchmarks like MMLU, and for most practical tasks the difference is barely noticeable. This is why approaches like Molly's make sense: it runs on your own hardware, keeps one quantized base model and a library of small LoRA specialists, routing each request to the right adapter. You get near-full-quality responses without holding multiple full models in memory.
Splitting the Load Across Cards
A 70B model in 4-bit quantization occupies roughly 35 to 40 GB of VRAM for weights alone, which means even a single 48 GB card leaves little headroom for KV cache and activation buffers. Tensor parallelism distributes each layer's matrices across GPUs, keeping latency low but demanding NVLink-grade interconnect bandwidth. Pipeline parallelism is more forgiving on slower links like PCIe, since layer groups stay local to each card, but it introduces sequential bottlenecks that increase latency for single-request inference.
The real cost of splitting is communication overhead, which can eat 15 to 30 percent of throughput depending on topology and batch size. Smaller batches suffer disproportionately because fixed synchronization costs don't amortize well. One practical approach
Picking the Right Configuration for You
A 70B parameter model at FP16 needs roughly 140 GB of VRAM just for weights, which means dual 80 GB GPUs at minimum. Quantization changes the math dramatically: 4-bit quantization brings that down to around 40 GB, fitting on a single 48 GB card with enough room left for a modest context window. The trade-off is measurable quality degradation, typically 1-3 points on standard benchmarks, though it is barely noticeable for most conversational and coding tasks.
If your workflow spans multiple domains, running several full fine-tunes is impractical on consumer hardware. Molly runs on your own machine, keeping a library of small LoRA adapters over a single quantized 70B base and routing each prompt to the right specialist. This means you pay the VRAM cost of the base model once, then swap adapters that are typically 50-200 MB each, instead of loading separate multi-gigabyte models for every task.
Common questions
How does quantization affect VRAM requirements for a 70B model?
Quantization significantly reduces VRAM needs. At FP16, a 70B model requires roughly 140GB. With 8-bit quantization, that drops to around 70GB, and 4-bit quantization can bring it down to approximately 35–40GB, making it feasible on fewer or more accessible GPUs.
How does the KV cache scale with context length and batch size?
The KV cache grows linearly with both context length and batch size. For a 70B model, each token's cache can consume several megabytes. Long contexts (e.g., 32K tokens) with larger batch sizes can add tens of gigabytes on top of model weights, so plan VRAM headroom accordingly.
Does fine-tuning a 70B model require significantly more VRAM than inference?
Yes. Fine-tuning stores gradients and optimizer states, often tripling or quadrupling VRAM compared to inference. Full fine-tuning of a 70B model in FP16 may need 400GB+. Techniques like LoRA reduce this dramatically by training only low-rank adapter weights, making it feasible on far less VRAM.