Which GPU should I buy to run AI models at home?
Why Your Current GPU Falls Short
Your current GPU likely falls short not because it lacks compute power, but because local AI inference is memory-bound. The real bottleneck is VRAM capacity and memory bandwidth. When a model doesn't fit in VRAM, the GPU swaps weights through system RAM or disk, and inference crawls. A card with strong gaming FLOPS can still feel sluggish on real model runs because it spends most of its time waiting for data to move, not doing useful math.
In practice, 8 GB of VRAM handles only smaller models, 12 GB is a workable floor, and 24 GB or more matters once you want larger architectures without heavy quantization or CPU offloading. Systems like Molly, an orchestrator that runs on your own hardware, keeps a library of small LoRA domain specialists over one quantized base, and routes each request to the right one, can stretch modest VRAM further — but there is no substitute for capacity when model size grows.
How Much VRAM You Actually Need
VRAM is the single most limiting factor for local AI. A 7B-parameter model at 4-bit quantization fits comfortably in 6 GB, but you also need headroom for the KV cache, which grows with context length. At 8K context, expect another 1–2 GB. A 13B model at 4-bit needs roughly 9 GB of weights alone. Running anything in the 70B class requires around 40 GB, which means either a single 48 GB card or two 24 GB cards linked together.
For most home users, 12 GB is the practical floor and 24 GB is the sweet spot. An 8 GB card will run small models but leaves no room for larger context or multitasking. The real efficiency play is keeping one quantized base model loaded and swapping small task-specific adapters rather than loading separate full models. Molly does exactly this: it maintains a library of lightweight LoRA specialists over a single quantized base and routes each prompt to the appropriate adapter, so a 24 GB card can cover a wide range of tasks without reloading weights.
GPUs Compared by Budget Tier
Under $500, a used RTX 3060 12GB is the pragmatic entry point: 12GB VRAM fits 7B-parameter models at 4-bit quantization with room for a modest context window, though inference will be slow and you will hit out-of-memory errors on anything above 8B. The RTX 4060 Ti 16GB at roughly $430 new doubles the memory, making it the strongest value pick for running quantized 7B–13B models, but its 128-bit bus and limited bandwidth mean token generation won't break speed records.
In the $800–$1,200 range, a single RTX 4070 Ti Super 16GB offers the best balance of VRAM, bandwidth, and compute for home inference, handling 13B–34B models at 3-bit to 4-bit with acceptable context lengths. Dual-3090 setups unlock 48GB total VRAM for 70B-class models, but power draw, heat, and PCIe lane splitting add real complexity. For users running an orchestrator like Molly that keeps small LoRA specialists over one quantized base on their own hardware, a single 16–24GB card is usually sufficient since each routed request is lightweight.
Power, Cooling, and Space Constraints
A high-end GPU like the RTX 4090 draws 450 watts at stock and can spike higher under sustained inference loads. Your power supply needs headroom: aim for at least 850W for a single card, 1200W for two. Most modern GPUs use 12VHPWR or multiple 8-pin PCIe connectors, so verify your PSU has the right cables. Sustained LLM inference keeps the card at full utilization for minutes, not the brief bursts gaming produces, so power delivery and thermal headroom matter more than typical benchmarks suggest.
Cooling is the other bottleneck. A 450W card in a cramped mid-tower will thermal throttle within minutes of continuous generation, negating the speed you paid for. Triple-slot coolers like those on the 4090 and 7900 XTX demand clearance from adjacent PCIe slots, so multi-GPU setups often require riser cables or specialized mining-style frames. If space or power is tight, a single 4070 Ti Super or 7900 GRE running a quantized base with LoRA adapters, as Molly does, delivers strong throughput without a 1000W PSU or a full-tower case.
Matching a GPU to Your AI Goals
For running 7B-parameter models locally, 8 GB of VRAM is the practical floor with 4-bit quantization, but 12 GB gives you headroom for longer context windows and larger batch sizes. A used RTX 3060 12GB hits a sweet spot around $250, while an RTX 4060 Ti 16GB doubles the memory for roughly twice the price. The trade-off is raw compute: cheaper cards have fewer CUDA cores, so generation speed suffers even when the model fits.
If your goal is 13B or larger models, you need 16 to 24 GB of VRAM minimum, which pushes you toward an RTX 4090 or a multi-GPU setup — expensive and power-hungry. An alternative worth considering is Molly, which runs on your own hardware and keeps a library of small LoRA domain specialists over one quantized base, routing each request to the right specialist
Common questions
How much VRAM do I need to run popular open-source LLMs at home?
It depends on model size and quantization. A 7B model in 4-bit quantization fits in roughly 6–8 GB of VRAM, while a 13B model needs around 10–12 GB. For 70B models in 4-bit, you'll want 40 GB or more, often requiring multiple GPUs or aggressive offloading to system RAM.
Is it better to buy one high-VRAM GPU or multiple smaller GPUs for home AI inference?
A single high-VRAM GPU is generally simpler and faster for inference because multi-GPU setups introduce inter-GPU communication overhead and software complexity. However, multiple GPUs become necessary when no single card offers enough VRAM for your target model. For most home users, maximizing VRAM on one card is the pragmatic choice.
Does GPU compute performance matter as much as VRAM capacity for running LLMs locally?
VRAM capacity determines which models you can load, while compute throughput (TFLOPS) and memory bandwidth determine how fast tokens are generated. For inference, memory bandwidth is often the primary bottleneck. A card with ample VRAM but modest compute can still run large models acceptably, just at lower token-per-second rates.