How Much VRAM Do You Really Need for Local LLMs?

Can You Run It? The Question Every Gamer Already Knows

Every PC gamer knows the feeling: you open a store page, scroll to the system requirements, and hold your breath. "Can I run it?" The GPU shader count, the RAM floor, the dreaded minimum versus recommended specs. Local AI has inherited that exact anxiety, repackaged as a single metric: VRAM. When you read about a 70B model running "locally," the unspoken question is whether your graphics card actually has the memory to hold it.

The good news is that the math is refreshingly simple — more predictable than game performance, which depends on drivers, resolution, and scene complexity. An LLM either fits in your VRAM or it does not, and the calculation is a straightforward equation you can do on a napkin. This guide teaches you that equation, walks through the real numbers for 2026 hardware, and helps you answer the "can I run it?" question for any model.

The VRAM Equation in Plain Terms

Every model consumes memory proportional to its parameters, and that number depends on how you store the weights. The core rule of thumb:

This gives you gigabytes from a parameter count. At 16-bit precision, a 7B model needs about 14GB of weights alone. That is why almost everyone who runs models locally uses quantization — storing weights in fewer bits (8, 4, or even 3 bits) at a modest quality cost, which is how a 7B model fits into a mainstream 8GB card.

Quantization is not a lossy afterthought; it is the entire reason local AI is possible on consumer hardware. The 4-bit GGUF format is the de facto standard for local inference, and the numbers below use it as the baseline.

The precision ladder shows why the bit choice dominates everything else. The same 7B model goes from unfittable on most consumer cards at FP16 to comfortable on an 8GB card at Q4:

Each step down roughly halves memory while adding a little quantization error. That is why the community standard for local use is Q4: it fits mainstream cards while staying close enough to the original quality that most users cannot tell the difference on everyday tasks. Going below Q4 is only worth it when the alternative is not running the model at all.

Quantized Sizes: The Numbers That Matter

Here is the practical reality for popular model sizes, in 4-bit quantization (the default you will encounter in GGUF files):

The "realistic total" column matters more than the raw weights. Your context window, KV cache, and the running application all live in the same memory. A 70B model at Q4 needs roughly 40GB of weights, plus several more gigabytes for a reasonable context window — which is why you will rarely see a 70B running comfortably on a single 24GB card without aggressive quantization or a short context[mlc-ai-vram].

The 2026 Hardware Landscape

The current consumer flagship is the NVIDIA RTX 5090 with 32GB of GDDR7 memory and 1,792 GB/s of bandwidth[nvidia-rtx5090]. That 32GB is the dividing line: it comfortably handles 7B, 14B, and even 32B models at Q4, but a 70B model only fits at Q3 or smaller — for Q4 70B, you are looking at either two cards or a workstation part.

The other popular route is unified memory. Apple's Mac Studio M5 Ultra packs up to 192GB of unified memory shared between CPU and GPU, which can run 120B+ parameter models natively that a discrete GPU could not touch[quantized-benchmarks]. The trade-off is bandwidth: 819 GB/s on the M5 Ultra versus 1,792 GB/s on the 5090, which directly caps tokens-per-second generation speed.

Beyond these consumer options there are two practical routes for the 70B-plus tier. The first is multi-GPU: two RTX 4090s give 48GB of combined VRAM, which fits a 70B at Q4 — but you need motherboard support, a high-wattage power supply, and a runtime like vLLM that shards the model across cards. The second is the new class of desktop AI appliances such as the NVIDIA DGX Spark with 128GB of unified memory, which puts a serious fraction of a datacenter node on your desk at $4,699. Each route trades money, complexity, and speed differently; the right choice depends on whether you need one model at scale or many models at convenience.

Why Context Eats VRAM (the Part Nobody Explains)

Almost every guide tells you the weight size and stops there. But the reason your "40GB for a 70B" plan falls apart in practice is the KV cache — memory the model uses to remember what has been said so far while generating. Every token in your prompt and every token the model has produced gets stored in this cache, scaled by the number of attention layers and heads.

The KV cache is why your usable VRAM shrinks as your conversation grows. A short prompt on a 7B model might use 1-2GB of cache; a long RAG document or a multi-turn chat can eat 4-8GB or more. The formula for cache size is roughly proportional to context length × model size, which is why the same model feels "fine" at a 2K context and suddenly OOMs at 32K.

This creates the practical rule that most people learn the hard way: your model fits, but your conversation does not. When you plan VRAM, decide your context budget first, then add it to the weight size. The Context Window Calculator helps you size realistic context needs for your workload before you buy hardware around the wrong assumption.

How to Check Your Real VRAM Usage

Theory gets you in the ballpark; measurement gets you the answer. Once you have a model running, you can confirm exactly how much memory it really uses instead of trusting the marketing numbers, and the tools to do it are free and built into your operating system.

On Windows, open the Task Manager and go to the Performance tab, then GPU. You will see "Dedicated GPU memory" — that is your VRAM. Run your model, start a conversation, and watch the number rise as the prompt grows. The difference between the idle reading and the in-use reading is the model's true footprint, and it will climb as your context window fills, which is the KV-cache effect in action.

On Linux, the command-line tools do the same job. nvidia-smi shows a live per-process memory breakdown for NVIDIA cards, while rocm-smi and radeontop cover AMD hardware. A simple loop that samples nvidia-smi --query-gpu=memory.used every few seconds while you chat with the model gives you a clear picture of both the peak and the steady-state usage.

The practical reason to measure is that the advertised figures and the real numbers rarely match. A "4-bit 7B" model ships weights of roughly 4GB, but the running process regularly holds 5-6GB once the tokenizer, runtime buffers, and a normal conversation are included. Knowing your real ceiling — not the theoretical one — is what turns "should fit" into "does fit." If you find yourself right at the edge, the levers are the same ones from earlier: drop to a smaller quantization, shorten the context budget, or offload layers to system RAM.

The VRAM Calculation Walked Through

Let us do the napkin math for a 70B model at Q4, because that is the case that confuses most people. First, the weights:

That is 35GB just for the weights, before anything else. Add a few gigabytes for the KV cache at a moderate context window, plus the model runtime overhead, and you are at roughly 40-44GB. Against an RTX 5090's 32GB, it does not fit — so the realistic options are Q3 quantization (smaller, some quality loss) or CPU offloading, where part of the model lives in system RAM and only active layers stream through the GPU.

For a 7B model at Q4, the same math gives ~4GB of weights. Add context and overhead, and an 8GB card (RTX 4060 Ti) handles it comfortably, which is why 7B models are the sweet spot for budget local setups[mlc-ai-vram].

Step-by-Step: Can You Run It?

Let us apply the process to a real decision, the kind you will actually face. Imagine you want to run a 32B parameter model for local coding assistance, with a 16K context window, on a budget.

Step 1 — compute the weights. At Q4, 32B parameters means 32 × 4 / 8 = 16GB of weights. That alone rules out any 8GB card and most 12GB ones.

Step 2 — budget the context. A 16K context on a 32B model needs roughly 3-5GB of KV cache plus runtime overhead. Realistic total: 20-24GB.

Step 3 — compare against hardware. An RTX 4090 (24GB) fits this at Q4 with a moderate context. An RTX 4060 Ti (8GB) does not even come close. The Mac Studio M5 Ultra (192GB) fits it trivially — but at 819 GB/s its generation speed is capped below what the 4090 achieves with its 1,008 GB/s, so your real choice is between the 24GB card for speed or the Mac for sheer capacity.

Step 4 — sanity-check the economics. Before spending, run your real workload through the LLM Hardware Requirements Calculator to confirm the model+context combo, and the Local LLM Break-Even Calculator to confirm you actually use enough tokens to justify the hardware over cloud API calls. A 32B model you use twice a month is cheaper as cloud tokens than as a $1,500 GPU.

That four-step flow — weights, context, hardware, economics — is the entire skill of answering "can I run it?" for any model you will ever see.

Practical Tips for Buying or Repurposing a GPU

• Check your real constraint before buying. If you want 7B models, an 8GB card is enough; if you want 32B+, you are on the 24-32GB tier. Know the target before the budget.
• Watch out for "weights only" marketing. Advertised VRAM figures are often weights-only. Add 2-6GB for context and overhead when planning.
• Quantization is your friend, not a cheat. Q4 (4-bit) is the quality/size sweet spot for most local use. Q3 fits bigger models but the quality loss shows on complex reasoning.
• Bandwidth determines speed, VRAM determines fit. A 70B model on a huge unified-memory Mac runs slower than a 7B on a fast gaming card. The model you can fit is not the same as the model you can enjoy.
• Test before you commit. Download a GGUF file and measure your actual tokens-per-second with your real context before spending thousands on a bigger card.
• When NOT to buy a GPU: if your workload is bursty and small, renting cloud tokens is cheaper — use the Local LLM Break-Even Calculator to check before you spend.

Limitations and Edge Cases

The VRAM equation is a strong heuristic, not a precise oracle. Real usage varies with context length, batch size, KV-cache behavior, and the exact runtime (llama.cpp, MLX, vLLM all manage memory slightly differently). The "realistic total" column assumes a moderate context window; heavy RAG workloads with large contexts can push 5-10GB beyond the table. CPU offloading blurs the line further — a model can "run" while partially streaming through system RAM, at a steep speed penalty. And unified-memory systems (Apple Silicon, DGX Spark) have a different memory model where the entire system RAM is the pool, so the math changes entirely.

Frequently Asked Questions

❓ How much VRAM do I need for a 7B model?

✅ About 4GB of VRAM for the weights at Q4 quantization, plus 1-2GB for context and overhead. An 8GB card like the RTX 4060 Ti is the comfortable entry point for 7B models.

❓ How much VRAM does a 70B model need?

✅ Roughly 35GB for Q4 weights, and 40-44GB with context and overhead. That exceeds a single RTX 5090's 32GB, so 70B at Q4 realistically needs two cards, a workstation GPU, or a high-VRAM unified-memory Mac.

❓ Can I run a 70B model on 24GB VRAM?

✅ Only at Q3 or smaller quantization, or with CPU offloading. Q4 70B needs more than 24GB; you would sacrifice quality or speed to squeeze it in.

❓ What is quantization and why does it matter?

✅ Quantization stores model weights in fewer bits (8, 4, or 3 instead of 16), shrinking memory use roughly proportionally at a small quality cost. It is the reason local AI is feasible on consumer GPUs.

❓ Does more VRAM always mean faster output?

✅ No. VRAM determines whether a model fits; bandwidth determines how fast it generates. A huge-model Mac can be slower than a small-model gaming card because of the bandwidth gap.

❓ Is CPU offloading a good idea for a 70B model?

✅ It lets a model run on insufficient VRAM by streaming layers through system RAM, but at a significant speed penalty. It is a stopgap, not a comfortable long-term setup.

❓ Can I run an LLM on a laptop with integrated graphics?

✅ You can, but only small models at low speed. A 1-3B model in Q4 fits in shared system memory and will crawl along at a few tokens per second. For anything interactive, a dedicated GPU or an Apple Silicon Mac with unified memory is the practical entry point.

❓ What happens when a model does not fit in VRAM?

✅ Two things: either the runtime crashes with an out-of-memory error, or it silently falls back to CPU offloading and runs drastically slower. The fix is smaller quantization, a shorter context window, or a bigger card.

❓ Is a 32GB card enough for everything in 2026?

✅ It handles the realistic consumer sweet spot: up to 32B models at Q4 and 70B at Q3, with room for context. It does not run 70B at Q4 or 120B-class models comfortably — for those you need dual-GPU, a workstation, or unified memory in the 96-192GB range.

❓ Does batch size change VRAM needs?

✅ Yes, dramatically. Processing multiple requests at once multiplies the KV-cache footprint per active sequence. Serving frameworks like vLLM handle batching efficiently, but a single-user setup with llama.cpp typically processes one request at a time, keeping cache usage lower.