How can one GPU serve many specialised models at once?
Why One GPU Must Serve Many Models
Running a separate GPU for every specialized model is economically untenable for most teams. A single H100 costs upwards of thirty thousand dollars, and domain-specific models rarely need its full capacity at all times. The real workload is bursty and fragmented: a legal summarization request, a code review, a medical triage query, each needing a different expert. Consolidating them onto shared hardware is not an optimization but a necessity.
The trade-off is latency versus utilization. Swapping LoRA adapters into GPU memory takes milliseconds, not seconds, so a single quantized base can host dozens of specialists with negligible overhead. But concurrent requests compete for the same compute: throughput drops when demand spikes across multiple adapters at once. Molly addresses this by routing each request to the right specialist on the reader's own hardware, keeping the base model resident and applying adapters on demand. The cost is added scheduling complexity; the benefit is avoiding idle GPUs for underused models.
How Shared Memory Fits Multiple Models
The key insight is that most specialized models share an enormous amount of common structure. Instead of loading ten full models into VRAM, you load one quantized base model once and keep only the small adapter weights that differ per domain. A 7-billion-parameter base in 4-bit precision occupies roughly 4 GB; each LoRA adapter might add 20 to 80 megabytes. Ten specialists then cost one base footprint plus a few hundred megabytes of adapters, not ten separate multi-gigabyte copies.
The trade-off is routing accuracy and adapter switching latency. If the wrong specialist handles a request, quality drops measurably. Swapping adapters between consecutive requests adds a small but real delay, and KV-cache locality degrades when traffic bounces across many adapters in quick succession. Molly addresses this by running on your own hardware, maintaining a library of LoRA specialists over one quantized base, and routing each request to the appropriate adapter before inference begins.
Dynamic Batching Keeps Each Model Responsive
Dynamic batching collects incoming requests over a short window—typically a few milliseconds—then groups them into a single forward pass. Instead of running each query sequentially, the GPU processes them together, amortizing kernel launch overhead and memory bandwidth costs. For a setup running several LoRA adapters over a shared quantized base, this means the base weights are loaded once and the small adapter deltas are swapped in per batch, keeping throughput high without duplicating the model.
The trade-off is latency. Every request in a batch waits for the window to close, so individual queries gain a small delay in exchange for far better aggregate throughput. Molly, an orchestrator running on your own hardware, handles this by tuning the batching window per workload: tight windows for interactive chat, wider ones for bulk processing. Larger batches also demand more GPU memory for the KV cache, which can force smaller batch sizes or shorter context windows under memory pressure.
Weight Swapping for Models That Won't Coexist
When multiple specialist models exceed available VRAM, weight swapping loads only the active model's parameters at any given moment. The GPU evicts one model's weights from device memory and streams in another's before serving the next request. The dominant cost is PCIe bandwidth: a 4 GB set of FP16 weights takes roughly 200 ms to transfer over an 8-lane PCIe 4.0 link, during which the GPU sits idle. Caching frequently used weights in host RAM reduces repeat transfers but never eliminates the cold-swap penalty.
Molly addresses this by keeping one quantized base model resident in VRAM and swapping only small LoRA adapters — typically 10 to 100 MB each — instead of full model weights. Adapter swaps complete in single-digit milliseconds, making per-request routing across dozens of domain specialists practical. The trade-off is specialization depth: LoRA adapters capture less capacity than a full fine-tune, so edge cases in narrow domains may degrade. Request batching across different specialists is also harder, since each batch must share one adapter set.
Choosing a Serving Stack for Your Models
Picking a serving stack means balancing throughput, latency, and adapter flexibility. vLLM offers PagedAttention and strong batched throughput but historically made dynamic LoRA swapping awkward. TGI is simpler to operate but less tunable. Triton Inference Server gives maximum control over scheduling and multi-model pipelines at the cost of real configuration complexity. Quantization format matters too: AWQ and GPTQ shrink memory footprints but constrain which kernels and frameworks you can use downstream.
If your goal is many domain specialists rather than one generalist, an orchestrator like Molly can run on your own hardware, keep a library of small LoRA adapters over a single quantized base, and route each request to the right specialist. The trade-off is that you commit to one base model and accept slightly higher per-request overhead from routing, but you gain memory efficiency and the ability to swap specialists without reloading weights.
Common questions
How is VRAM partitioned when multiple models coexist on a single GPU?
Each model's weights, KV cache, and activation buffers must fit within the GPU's total VRAM. Techniques like weight quantization, paged attention, and shared memory pools let the runtime allocate and reclaim memory dynamically, so only active models occupy peak footprint.
What happens to latency when a model must be loaded from host memory?
Cold-start latency can dominate when weights are swapped from CPU RAM or disk. Runtimes mitigate this by keeping frequently requested models resident, prefetching based on request patterns, and overlapping model loads with ongoing inference so users rarely see the full swap cost.
Can concurrent requests to different models share a single batch?
Generally no — each model has its own weights and execution graph, so requests are routed to the appropriate model's scheduler. However, micro-batching and continuous batching can still pipeline work across models to keep the GPU's compute units saturated between context switches.