Tokens per Second Benchmarks Explained

A Few Moments Later… How Fast Is "Fast"?

Every interface in the world of local AI eventually shows you that dreaded spinner, and on the wrong setup it sits there long enough that your brain supplies the meme: "A few moments later…" That pause is a number wearing a disguise. Somewhere inside your machine, the model is grinding out tokens — fragments of words — and the only question that matters is how many of them it produces per second.

Tokens per second (tok/s) is the universal speedometer of local LLMs, quoted in every benchmark and every GPU review. But it is also one of the most misleading numbers in the field, because the same model can measure 45 tok/s or 793 tok/s depending on how you test it. This guide explains what the number actually means, why it moves so dramatically, and how to read a benchmark without fooling yourself.

What a Token Actually Is

Before speed makes sense, the unit has to. Models do not read words; they read tokens, which are chunks of text roughly three-quarters of a character on average in English. The word "calculator" might be one token or three, depending on the tokenizer, and this is not idle trivia — it is the reason the same prompt can cost a different amount across providers, as the Token Counter Calculator shows in practice.

Because tokens are the unit of both billing and speed, "tokens per second" is the single number that connects all three corners of the local AI decision: how fast the model answers (tok/s), how big the model is (parameters), and what it costs to run (hardware amortized over time). A model doing 50 tok/s reads roughly 100-150 words per second — comfortably faster than you can read. A model stuck at 5 tok/s feels like a slow internet connection in 1998.

The Single-User Number Is Not the Whole Story

Here is the trap: most consumer benchmarks report tok/s at one user, one request. That number describes a private chat with your own model, and for that scenario it is the right metric. But the moment you put a model behind an API or a team, the picture changes entirely, because serving engines handle many requests at once and the interesting number becomes throughput — total tokens produced across all users per second.

The mechanism that changes everything is continuous batching. A naive server waits for one request to finish before starting the next. A batched server fills idle GPU cycles with other requests, keeping the silicon busy. This is why vLLM's key innovation — continuous batching plus PagedAttention — lets one GPU serve dramatically more total traffic than a tool that processes requests one at a time[anyscale-continuous-batching].

The Concurrency Curve: Back Street vs Highway

Think of it like driving. At one car, a quiet back street and a six-lane highway are both quick — there is no traffic, so lane count does not matter. That is the single-user case, where Ollama and vLLM are genuinely close, with Ollama even slightly ahead on some hardware[redhat-ollama-vllm].

Now imagine rush hour. The back street gridlocks at a handful of cars; the highway keeps absorbing lane after lane of traffic. That is the concurrency curve for local inference. At a single request both tools idle along, but as concurrency climbs, the batching engine pulls ahead dramatically:

Under sustained load, the Red Hat benchmark on an A100 40GB with Llama 3.1 8B measured vLLM peaking around 793 tok/s of combined throughput versus about 41 tok/s for Ollama — roughly a 19x gap that emerged only as concurrency rose from 1 to 256 users[redhat-ollama-vllm]. Ollama even stayed behind when told to run 32 parallel workers. The exact multiplier varies by model, GPU, and workload — it is not a universal constant.

Throughput vs Latency: Two Numbers, Two Questions

The tok/s number hides a second, equally important split: throughput versus latency. Throughput is how much work the machine does per second (useful for a server). Latency is how long one specific user waits for their answer (useful for a chat). They pull in opposite directions — serving more users at once raises throughput but can stretch how long any single request takes to start producing.

That is why real serving guides pay attention to the 99th-percentile latency (P99), the wait experienced by the slowest 1% of requests. In the same Red Hat test, vLLM delivered an 80-millisecond P99 under concurrency while Ollama's tail latency ballooned to 673 milliseconds — an 8x difference that matters enormously for interactive applications, even though average latencies looked acceptable[redhat-ollama-vllm]. Average numbers flatter the tool that fails occasionally; percentiles expose it.

For a single user running a private model, latency and throughput are almost the same number, and the headline tok/s figure is honest enough. For a server, ignore the single-user benchmark entirely — read the concurrency curve and the P99 column.

There is a third metric hiding between them: time to first token (TTFT), the delay between submitting a prompt and the first word appearing. TTFT is dominated by prompt processing — the model reading and attending to your entire input before it can start generating. A long RAG context or a multi-turn history makes TTFT grow, independently of how fast the model then generates subsequent tokens. Under concurrency, TTFT is where a poorly batched server shows its worst face, because queued requests wait behind others before their prompt is even processed.

The three numbers tell different stories. Tok/s describes sustained generation, P99 describes worst-case wait under load, and TTFT describes how the experience starts. A model can look great on all three for a chat and collapse on TTFT for a long-document workload, or vice versa. When you read a benchmark, ask which of the three it is actually reporting — many reviews only quote tok/s and quietly ignore the other two, which is exactly where the misleading numbers hide.

What Realistic Speeds Look Like in 2026

Consumer hardware in 2026 lands in predictable bands, and knowing them keeps expectations sane. A 7B model in Q4 on a fast gaming card typically generates in the 120-300 tok/s range, which is more than fast enough for interactive chat. A 70B model on the same hardware crawls to single digits or low tens, because the memory bandwidth and compute demand scale with size[quantized-benchmarks]. Unified-memory Macs sit in the middle: large models fit easily, but generation speed is capped by their bandwidth ceiling.

The practical translation: for a personal assistant or coding autocomplete, anything above roughly 30-40 tok/s feels instant, and the difference between 80 and 200 tok/s is barely noticeable to a human. The tok/s arms race only matters when you are serving many users, where the concurrency curve decides whether your hardware keeps up at all.

What Speed Feels Like: Tokens to Real Words

Numbers on a screen are abstract until you translate them into the experience of actually waiting. Because a token is roughly three-quarters of a word, you can estimate reading speed by multiplying tok/s by 0.75 to get words per second, then by 60 to get words per minute. The result is a feeling you already have a reference for.

At 50 tok/s, a model produces about 2,250 words per minute — several times faster than any human reads. Long answers materialize almost instantly, and the only real pause is the initial delay before the first token, which is dominated by prompt processing rather than generation speed. This is the experience most people mean when they say local AI "feels fast enough."

Drop to 10 tok/s and the math becomes 450 words per minute. Still readable, but now you watch the text assemble across a couple of seconds, and a 500-word answer takes nearly a minute. The "A few moments later…" feeling starts here, not because the model is slow in absolute terms, but because it is slower than your patience.

The distinction between these two experiences is almost never the headline benchmark. A review that says "50 tok/s" tells you nothing about whether the first token arrived in 200 milliseconds or 4 seconds — and that first-token latency is what you actually notice when you hit Enter. When you test a model yourself, watch the gap between pressing Enter and seeing the first word, then judge the generation rate separately. Both matter, but they are different problems with different fixes: prompt processing is about compute and prompt length, while generation speed is about memory bandwidth.

A useful mental anchor: compare the model's reading speed against your own. If a model produces text faster than you can read it, generation speed is effectively a non-issue — your attention is the bottleneck, not the hardware. That threshold sits near 15-20 tok/s for a comfortable reader, which is why budget local setups on 7B models feel so satisfying and why a slow 70B model on the wrong card feels broken even when it is "technically working." Speed, in the end, is a relationship between the machine and the person waiting.

How to Measure Your Own Tokens per Second

You do not need a lab to get your number — just ten minutes and a model file. The two most common tools, Ollama and vLLM, both make measurement straightforward, and both are open source with their own documentation[ollama-github][vllm-github].

With Ollama, the fastest path is interactive: download a GGUF model, run ollama run <model>, and ask it to produce a fixed amount of text. The terminal prints timing statistics when the response finishes, including a tokens-per-second figure for that exact prompt on your exact hardware. Keep the prompt the same across tests and you get a repeatable baseline.

With vLLM, the honest number requires a tiny bit of setup because the tool is built for serving, not single chats. Point it at your model file, start the OpenAI-compatible server, and send a request through its /v1/chat/completions endpoint with a known prompt length. The response metadata includes timing information you can turn into tok/s, and you can raise concurrency by firing several requests in parallel to build your own concurrency curve.

Whichever tool you use, standardize three things so the number is comparable: the model (same quantization), the prompt length (same token count), and the hardware (same GPU and driver). Change any one and the benchmark changes with it. Also measure a couple of times — GPU thermals and background load shift first-run numbers by 10-20%.

Once you have your real tok/s, plug your model choice and hardware into the LLM Hardware Requirements Calculator to sanity-check that the setup makes sense before committing to it.

Practical Tips for Reading Benchmarks

• Ask "at what concurrency?" A tok/s number without a concurrency context is meaningless. 45 tok/s at one user and 45 tok/s at 50 users are opposite results.
• Match the model to your hardware. Benchmarks on A100s do not predict RTX 5090 performance. Find numbers for your actual card and quantization.
• Check the P99, not the average, for serving. Average latency hides the requests that actually time out.
• Read your own tok/s before buying. Download a GGUF, run the LLM Hardware Requirements Calculator for the model+context combo, then measure the real speed on your machine.
• Do not pay for more speed than you can perceive. Single-user chat above ~40 tok/s is already instant. The budget is better spent on a bigger model than a faster small one.
• When the numbers disagree, trust the method. A benchmark that does not state its concurrency, GPU, quantization, and context is marketing, not measurement.

Limitations and Edge Cases

Tokens per second is a snapshot, not a law. Real numbers shift with context length (longer prompts slow generation), temperature sampling (which changes how much the model reuses cache), quantization, and the exact runtime. Batch efficiency also depends on request mix — varied prompt lengths batch better than uniform ones. And tok/s is silent about quality: a blazing-fast small model is still a small model. Use it to compare how well a given model runs on a given box, not to compare models against each other.

Frequently Asked Questions

❓ How many tokens per second is 'good' for a local LLM?

✅ For interactive chat, roughly 30-40 tok/s already feels instant to most people. Above that, gains are barely perceptible. For coding autocomplete, similar thresholds apply. Only servers need the 500+ tok/s numbers.

❓ Why is my model slower than the benchmark said?

✅ Benchmarks use specific hardware, quantization, and context lengths. Longer context, a bigger model, a lower-bandwidth card, or background processes all cut real speed. Measure your own setup rather than trusting a headline number.

❓ Why does vLLM beat Ollama under concurrency but not at one user?

✅ Continuous batching. At a single request there is nothing to batch, so Ollama's simpler pipeline can even edge ahead. As requests pile up, vLLM fills idle GPU cycles with other work, multiplying total throughput.

❓ Is a 70B model always slower than a 7B?

✅ Roughly yes on the same hardware, because every token pass touches more parameters. The gap is often 10x or more. This is why fitting a big model matters less than fitting one that generates at a usable speed.

❓ What is the difference between throughput and latency?

✅ Throughput is total tokens produced per second across all users (a server metric). Latency is how long one user waits for their response (an experience metric). Serving more users raises throughput but can worsen tail latency.

❓ Can I use tok/s to compare two different models?

✅ Only loosely. Tokens are tokenizer-dependent, so two models can split the same text into different token counts. Tok/s compares how efficiently a model runs on a machine, not how good the model is.