LLM Hardware Requirements Calculator

Understanding Local LLM Hardware Requirements

Running large language models locally has become increasingly viable in 2026, with consumer hardware like the NVIDIA RTX 5090 (32GB GDDR7) and Apple Mac Studio M5 Ultra (192GB unified memory) capable of running models that previously required enterprise-grade infrastructure. But choosing the right hardware requires understanding two critical constraints: VRAM capacity (does the model fit in memory?) and memory bandwidth (how fast can it generate tokens?).

VRAM capacity determines whether a model can run at all. A 70B-parameter model at Q4 quantization requires approximately 35GB just for weights — exceeding the RTX 5090's 32GB VRAM and requiring either Q3 quantization or CPU offloading. Memory bandwidth determines the user experience: the RTX 5090's 1,792 GB/s delivers roughly 140-300 tokens per second for 7B-20B models, while the Mac Studio M5 Ultra's 819 GB/s delivers strong performance for larger models that fit in its generous memory.

This calculator computes exact VRAM requirements for any combination of model size, quantization format, and context length, then shows which hardware options can run it natively versus requiring CPU offloading. For cost analysis of local versus cloud deployment, see the Local LLM Break-Even Calculator. To understand how context window affects your choice, use the Context Window Calculator. To size your workload before choosing hardware, count your actual usage with the Token Counter Calculator, and compare the API bill you are replacing with the LLM API Cost Calculator. For the broader decision on whether to self-host, see Local vs Hosted LLMs: The Decision Framework, and for how to serve the models once you have the hardware, vLLM vs Ollama: Production Serving 2026.

How to Use This Calculator

Select your target model size from the dropdown. The options cover the most common open-weight models available in 2026, from efficient 7B models like Llama 3 and Mistral to large 120B+ mixture-of-experts architectures like gpt-oss.

Choose a quantization format. Q4 (4-bit) is the recommended default for most use cases — it offers minimal quality loss while halving VRAM requirements compared to FP16. Q3 saves more VRAM but with noticeable quality degradation. Q5 preserves more quality at the cost of additional VRAM. FP16 represents full precision and should only be used when quality is paramount and hardware allows.

Select your expected context length. This is the maximum number of tokens the model will process in a single request, including both input prompt and generated output. Longer contexts require more VRAM for the KV cache (key-value attention cache), which stores attention state from previous tokens. A 128K context adds roughly 6.4 GB of VRAM overhead on top of model weights.

The calculator displays the total VRAM required and shows compatibility with current hardware options. "Fits natively" means the entire model loads into GPU VRAM for maximum performance. "Offloading required" means some layers must be stored in system RAM and transferred over PCIe, reducing throughput by 60-80%. "Does not fit" means the model exceeds even offloading capacity.

The VRAM Calculation Formula

The approximate VRAM required for model weights follows from first principles. Each parameter stored at $B_q$ bits requires $B_q / 8$ bytes:

Where $P$ is parameter count in billions and $B_q$ is bits per weight for the quantization format. For example, a 70B model at Q4 (4 bits) requires $(70 \times 4) / 8 = 35$ GB for weights alone.

The KV cache grows with context length. Each token in the context requires storing key and value vectors for all attention heads across all layers. The approximate overhead is:

Where $C$ is context length in tokens. This is an approximation — actual KV cache size depends on the model's architecture (number of layers, attention heads, hidden dimensions). Models with grouped query attention (GQA) like Llama 3 have smaller KV caches than models with full multi-head attention.

Total VRAM required includes weights, KV cache, and overhead for activations and buffers:

Where $W_{overhead}$ is typically 1-3 GB for activation memory, CUDA buffers, and framework overhead[llama-cpp-docs].

Quantization Formats Explained

Quantization reduces the precision of model weights from FP16 (16-bit floating point) to lower-bit representations. This trades a small amount of model quality for significant VRAM savings:

Q4 is the recommended default for most use cases. The quality loss is minimal for most tasks — benchmarks show Q4 models retain 95-98% of FP16 performance on common benchmarks — while halving VRAM requirements. This is why most local LLM practitioners default to Q4_K_M (a specific Q4 variant with optimized kernel implementations).

Hardware Options (2026)

The RTX 5090[nvidia-rtx5090] offers 78% more memory bandwidth than the RTX 4090[nvidia-rtx4090] (1,792 vs 1,008 GB/s) and 33% more VRAM (32 vs 24 GB). For LLMs, bandwidth is the primary performance determinant, so the 5090 delivers roughly 60-80% higher tokens/second on bandwidth-bound workloads. However, current street prices ($3,000-5,000) significantly exceed MSRP due to supply constraints.

Apple's Mac Studio M5 Max and M5 Ultra[apple-mac-studio] offer a different value proposition: massive unified memory (128-192GB) at lower bandwidth (819 GB/s). This makes them ideal for running very large models (70B-120B+) at the cost of lower tokens/second compared to NVIDIA GPUs. They also offer silent operation and lower power consumption.

Practical Tips for Local LLM Hardware

Prioritize bandwidth over VRAM for interactive use. A model that runs at 30 tok/s feels responsive; one at 5 tok/s feels sluggish. If you must choose between more VRAM with lower bandwidth versus less VRAM with higher bandwidth, choose bandwidth for interactive applications like chat interfaces and coding assistants.

Consider CPU offloading for non-interactive workloads. If you are running batch processing (summarization, classification, overnight report generation), CPU offloading is acceptable. A 70B Q4 model on an RTX 5090 drops from ~40 tok/s native to ~14-22 tok/s with offloading, but for batch work this may be perfectly adequate — and it saves thousands of dollars in hardware costs.

Factor in the full system cost. A GPU is not just its purchase price. The RTX 5090 requires a 1000W power supply, adequate cooling, and a motherboard with sufficient PCIe lanes. Multi-GPU setups require even more infrastructure including proper case airflow and potentially dedicated power circuits. Budget for the complete system, not just the card.

Watch for VRAM fragmentation. Some frameworks cannot use 100% of available VRAM — they reserve portions for CUDA contexts, display buffers (if the GPU drives a monitor), and framework overhead. Expect usable VRAM to be 1-3 GB less than the card's rated capacity. If your model requires exactly 32 GB, it may not fit on a 32 GB card in practice.

Test with your actual workload. The calculator provides estimates based on common configurations. Your actual VRAM usage depends on the specific inference framework (llama.cpp, vLLM, Ollama), batch size, and model architecture. Always test with your target model before purchasing hardware. Most frameworks report actual VRAM allocation at startup, making this verification straightforward.

Consider future-proofing. Model sizes continue to grow, but quantization improvements also continue. Buying hardware with more VRAM than you currently need extends its useful life as models evolve. The difference between 24 GB and 32 GB VRAM often determines whether a card remains useful for 1-2 additional years.

Apple Silicon for large models on a budget. If your primary goal is running the largest possible model regardless of speed, Apple Mac Studio with unified memory offers exceptional VRAM per dollar. A Mac Studio M5 Ultra with 192 GB can run models that would otherwise require $10,000+ in enterprise GPU infrastructure, at the cost of lower tokens/second.

Limitations

This calculator provides estimates, not guarantees. Actual VRAM usage varies by inference framework, model architecture, and configuration:

• Architecture variations: Models with grouped query attention (GQA) like Llama 3 have smaller KV caches than models with full multi-head attention. Mixture-of-experts models have lower active parameter counts than their total parameter count suggests — a 30B total MoE model with 3B active parameters uses VRAM proportional to the total parameter count for weights, but the KV cache depends on the actual activated layers. Sliding window attention models like Mistral have KV caches that cap at the window size regardless of total context length.

• Framework overhead: Different frameworks have different memory management strategies. vLLM uses PagedAttention for efficient KV cache management, while llama.cpp uses contiguous allocation. Actual overhead ranges from 1-5 GB depending on framework and configuration. vLLM typically has higher baseline overhead but better memory efficiency at scale due to PagedAttention eliminating KV cache fragmentation.

• Batch size: This calculator assumes batch size 1 (single user). Serving multiple concurrent users multiplies KV cache requirements — 4 concurrent users roughly quadruple the KV cache footprint. Production deployments serving multiple users should multiply the KV cache estimate by expected concurrency.

• Quantization variants: Q4 includes multiple variants (Q4_K_M, Q4_K_S, Q4_0) with slightly different VRAM usage and quality characteristics. Q4_K_M is the recommended variant for quality but may use marginally more VRAM than Q4_0. The calculator uses average values that may differ from specific variants by 5-10%.

• Memory bandwidth is not the only bottleneck. While bandwidth is the primary constraint for token generation (decode phase), the prefill phase (processing the input prompt) is compute-bound on tensor cores. GPUs with higher tensor core throughput (like the RTX 5090 with 5th-gen tensor cores) show proportionally better prefill performance despite similar bandwidth improvements.

• System RAM requirements for offloaded models: When using CPU offloading, system RAM speed and PCIe bandwidth become limiting factors. PCIe 4.0 x16 provides ~32 GB/s bandwidth versus the GPU's internal 1,792 GB/s — a 56x bottleneck. Fast DDR5 RAM (6000+ MT/s) partially mitigates this by improving CPU-side inference for offloaded layers.

Always verify with your specific hardware and software configuration before making purchasing decisions.

Frequently Asked Questions

❓ Can I run a 70B model on an RTX 5090?

✅ At Q4 quantization, a 70B model requires ~35 GB for weights alone, exceeding the RTX 5090's 32 GB VRAM. You would need Q3 quantization (~26 GB) to fit natively, or accept CPU offloading at Q4 which drops throughput to 14-22 tok/s. For comfortable Q4 70B, use 2× RTX 5090 (64 GB) or Mac Studio M5 Max (128 GB).

❓ How much VRAM do I need for a 7B model?

✅ A 7B model at Q4 requires ~3.5 GB for weights. Add 1-2 GB for KV cache and overhead, totaling ~5-6 GB. This fits comfortably on any modern GPU including the RTX 4060 Ti (8 GB). At FP16, the same model needs ~14 GB.

❓ Is the RTX 5090 worth it over the RTX 4090 for LLMs?

✅ The RTX 5090 offers 78% more memory bandwidth (1,792 vs 1,008 GB/s) and 33% more VRAM (32 vs 24 GB). For LLMs, bandwidth is the primary performance determinant, so the 5090 delivers roughly 60-80% higher tokens/second. However, at $3,000-5,000 current street price (well above $1,999 MSRP), the value proposition depends on your throughput needs.

❓ Can I use multiple GPUs to run larger models?

✅ Yes. Two RTX 5090s provide 64 GB total VRAM, enough for 70B Q4 or 120B Q3. However, multi-GPU inference requires software support (vLLM, llama.cpp with tensor parallelism) and the GPUs should ideally be on the same PCIe root complex for best performance.

❓ How does context length affect VRAM usage?

✅ The KV cache grows linearly with context length. At Q4, each 10K tokens of context adds roughly 0.5 GB of VRAM. A 128K context window adds ~6.4 GB on top of model weights. For very long contexts (200K+), the KV cache can exceed the model weights in size.

❓ What is the best budget GPU for local LLMs?

✅ The used RTX 4090 (24 GB) at $1,600-2,000 offers the best VRAM per dollar for local LLMs. It fits 14B Q4 models natively and can run 70B with Q3 quantization or partial offloading. Newer RTX 5090 offers better performance but at a significant price premium.

❓ Do I need a GPU with ray tracing cores for LLMs?

✅ No. Ray tracing cores are irrelevant for LLM inference. What matters is VRAM capacity and memory bandwidth. Tensor cores help with quantized inference in some frameworks, but the primary bottleneck is always memory bandwidth.

❓ How do I check my current VRAM usage during inference?

✅ Use nvidia-smi for NVIDIA GPUs (shows VRAM utilization in real-time) or Activity Monitor on macOS (Memory tab). Most inference frameworks (llama.cpp, vLLM, Ollama) also report VRAM allocation at startup.

❓ What is the difference between Q4_K_M and Q4_0?

✅ Both are 4-bit quantizations but with different kernel implementations. Q4_K_M uses a more sophisticated quantization scheme that preserves quality better, especially for larger models. Q4_0 is simpler and faster but with slightly lower quality. Most practitioners default to Q4_K_M for the best balance.

❓ Can I run LLMs on CPU only?

✅ Yes, but slowly. CPU inference uses system RAM instead of VRAM, so you can run much larger models if you have sufficient RAM. However, expect 5-20x slower tokens/second compared to GPU inference. A 70B model on a high-end CPU might achieve 2-5 tok/s versus 20-40 tok/s on a GPU.