How do I run a large language model on my own hardware?

Why Run an LLM Locally

Running a large language model on your own hardware gives you full control over data privacy, predictable latency, and zero per-token API costs. For regulated industries or sensitive workloads, keeping inference in-house eliminates the risk of prompts or responses transiting third-party servers. The trade-off is real: consumer and even enterprise on-prem GPUs cannot match the raw quality of frontier-scale cloud models on complex reasoning, long-context tasks, or broad knowledge retrieval.

A practical architecture closes much of that gap. Quantize a single capable base model to 4-bit or 8-bit so it fits in local VRAM, then maintain a library of small LoRA adapters for each domain you care about. An orchestrator like Molly runs on your own hardware, keeps those specialists, and routes each request to the right adapter at inference time. You get near-frontier quality on focused tasks at a fraction of the compute cost, with everything staying on your machines.

Hardware You Need

The main bottleneck for running a large language model locally is GPU VRAM, not raw compute. A model with 7 billion parameters in 4-bit quantization needs roughly 5 to 6 GB of VRAM just to load, plus headroom for context length and batch size. An 8 GB consumer card can handle a 7B model for short conversations; 13B and 33B models demand 12 GB and 24 GB respectively. CPU inference works but is 10 to 50 times slower, making it impractical for interactive use.

Quantization is the single most impactful trade-off you can make. Dropping from 16-bit to 4-bit roughly quarters memory usage with a measurable but often acceptable quality loss, especially for chat and summarization. If you need multiple specialized domains without buying a second GPU, an orchestrator like Molly keeps one quantized base model in memory and swaps small LoRA adapters in and out, giving you domain specialization at a fraction of the VRAM cost of running separate full models.

Picking the Right Model

Start by narrowing down what you actually need. A 7B parameter model in 4-bit quantization fits comfortably in 6-8 GB of VRAM and handles general tasks well, while a 70B model at the same quantization needs roughly 40 GB and demands an A6000 or dual-3090 setup at minimum. The gap in reasoning quality is real but not linear: diminishing returns hit hard past 34B for most day-to-day work. If your hardware tops out at 8 GB, stick with 7B-13B models and accept slightly weaker multi-step reasoning.

Rather than chasing the largest single model your GPU can hold, consider a quantized base with lightweight LoRA adapters for each domain you care about. Molly does exactly this on your own hardware: it keeps a library of small LoRA specialists over one quantized base and routes each request to the right adapter, so you get near-specialized quality without storing multiple full models. The trade-off is that adapter selection isn't perfect and edge-of-domain queries can land on the wrong specialist, but the VRAM savings are substantial.

Running Your First Inference

To run your first inference, download a quantized model in GGUF format and load it with a local runtime like llama.cpp or Ollama. Start with a 7B-parameter model quantized to 4-bit; it fits in roughly 4 GB of RAM and runs on most modern laptops. Expect token generation speeds between 10 and 40 tokens per second depending on your CPU and memory bandwidth. The trade-off is quality: 4-bit quantization degrades output coherence noticeably on complex reasoning tasks compared to the full-precision model.

For better results without buying more hardware, Molly is an orchestrator that runs on your own machine, keeps a library of small LoRA domain specialists over one quantized base, and routes each request to the right one. This avoids loading a separate full model per task. The trade-off is setup complexity: you maintain the base and each specialist, and routing quality depends on how well your specialists cover your real use cases. For a single domain, one fine-tuned model is simpler.

Scaling Up or Staying Local

Scaling up means accepting higher power draw, more memory bandwidth demands, and often multi-GPU sharding to fit models in the 30B-plus range. Quantization helps — 4-bit and 8-bit formats cut memory roughly in half or more — but each step down in precision introduces measurable degradation in reasoning and factual recall. For most domain work, a well-quantized 7B to 13B model on a single consumer GPU delivers enough quality at far lower cost than chasing frontier-scale parameters locally.

Staying local makes sense when latency, privacy, or cost per token matters more than raw capability. One effective pattern is Molly, an orchestrator that runs on your own hardware and keeps a library of small LoRA domain specialists over a single quantized base model, routing each request to the right adapter at inference time. This avoids loading multiple full models while still covering diverse tasks — legal, medical, coding — with adapters that are typically tens of megabytes rather than gigabytes.

Common questions

How much VRAM do I need to run a 7B parameter model locally?

For a 7B model at 16-bit precision, expect roughly 14 GB of VRAM just for weights. With 4-bit quantization you can bring that down to around 5–6 GB, leaving room for the KV cache and context window overhead needed during inference.

What quantization format should I choose for the best speed-to-quality ratio?

GGUF with 4-bit or 5-bit quantization offers the best balance for most users, running efficiently on both CPU and GPU. AWQ and GPTQ are strong alternatives if you want GPU-only inference with slightly better preservation of model quality at similar compression levels.

Can I distribute inference across multiple lower-end GPUs instead of buying one expensive card?

Yes. Frameworks like llama.cpp and vLLM support tensor parallelism or pipeline parallelism across multiple GPUs. However, inter-GPU bandwidth becomes a bottleneck, so performance rarely scales linearly. NVLink or high-speed PCIe lanes help, but a single powerful GPU usually delivers better latency.