Understanding Tokens: A Complete Guide to How AI Models Process Text

What Are Tokens and Why Do They Matter?

A token is the atomic unit of computation for large language models. When you type a prompt into ChatGPT, Claude Code, OpenCode, or Gemini, the model converts your text into a sequence of numerical vectors called tokens, processes them through its neural network, and generates output tokens one at a time. This tokenization step is invisible to the user but determines everything about cost, latency, and context capacity[openai-tokenizer].

The approximate token count for a given text follows:

Where $C$ is character count and $R$ is the model-specific characters-per-token ratio. For GPT-5.6 models, $R \approx 3.8$. For Gemini, $R \approx 4.0$. This means a 10,000-character document consumes about 2,500 tokens on GPT-5.6 and 2,500 on Gemini — a small difference per document that compounds into significant cost variance at production scale.

Three factors make token awareness essential for anyone using LLMs. First, every API call is billed by the token — understanding tokenization is understanding your costs[openai-tokens]. Second, the context window is measured in tokens — exceeding it means data loss[anthropic-tokens]. Third, agentic workflows consume tokens multiplicatively, not additively — a single agent task can consume 5-10x the tokens of a simple chat completion. These factors make token management a core engineering and business concern, not an academic curiosity.

How Tokenization Works: Byte-Pair Encoding

The dominant tokenization algorithm across all major providers is Byte-Pair Encoding (BPE), adapted from data compression by Sennrich et al. in 2016[sennrich-2016]. BPE starts at the byte level and iteratively merges the most frequent adjacent pairs to build a vocabulary. Each merge creates a new token that represents a common subword unit.

The algorithm can be expressed as:

Where $a_k b_k$ is the most frequent adjacent pair in the current vocabulary at step $k$. After 50,000-100,000 merges, the resulting vocabulary reflects the statistical patterns of the training corpus. Common words like "the" become single tokens. Rare words split into their frequent subword components: "unhappiness" becomes "un", "happi", "ness" because those subwords appear frequently enough across the corpus to be learned.

The vocabulary is a direct function of the training data. OpenAI's tokenizer — trained on English text and code — produces a vocabulary optimized for those domains. Google trains on broader multilingual data, producing a vocabulary that handles non-English text more efficiently but slightly over-tokenizes English[huggingface-tokenizers]. These training differences mean token counts are not portable across providers — a prompt that uses 1,000 tokens on GPT-5.6 might use 1,050 tokens on Claude Opus 5 and 980 on Gemini 3.6 Flash.

DeepSeek V4's architectural choices further separate it from the rest: its Mixture-of-Experts design activates only 49B of 1T+ total parameters per token, achieving 1M context windows at a fraction of the computational cost per token compared to dense models[deepseek-v4]. This architectural difference means that even when two models produce identical token counts, the cost to the provider — and by extension the price to the user — can differ dramatically.

Token Efficiency Across Languages

Token efficiency is heavily language-dependent. English tokenizes at roughly 1.3 tokens per word, or 3.7-4.0 characters per token. Chinese, Japanese, and Korean tokenize much less efficiently because each character typically maps to 1-2 tokens — meaning the same semantic content in Chinese uses 2-3x more tokens than in English[anthropic-tokens].

This has direct cost implications for multilingual applications. A chatbot serving English and Japanese customers pays 2-3x more per Japanese interaction for the same semantic content. The difference is invisible to users — the model still processes their text — but visible on the API bill. When budgeting for multilingual deployments, token efficiency by language must be factored into per-market cost projections.

Code also tokenizes differently. Programming keywords like "function" and "return" are single tokens, but camelCase variable names split into subword units. "calculateMonthlyRevenue" becomes "calculate", "Monthly", "Revenue" — three tokens instead of one. Code consumes roughly 20-30% more tokens per character than English prose, making code completion and code review tasks proportionally more expensive than text processing tasks of equivalent length.

Token Consumption in Agentic AI

Agentic AI — where the model acts autonomously, calls tools, reflects on results, and iterates — fundamentally changes how tokens are consumed. A chat completion follows a simple pattern: input → output. An agentic task follows a pattern closer to: input → reasoning → tool call → tool response → reasoning → tool call → ... → final output. Each step in this loop adds tokens on both sides of the equation[mcp-spec].

The token consumption for an agentic task over $k$ turns is:

Where $N_{i,\text{prompt}}$ is the running conversation history, $N_{i,\text{tools}}$ is MCP tool responses, $N_{i,\text{reasoning}}$ is chain-of-thought, and $N_{i,\text{output}}$ is the visible output. Over 10 turns, a single agentic task may consume 20,000-50,000 tokens — 10x a simple chat completion for the same final deliverable.

Consider an agentic code review running in Claude Code:

1. Turn 1: System prompt (1,200 tokens) + file content (2,000 tokens) + review output (1,000 tokens) = 4,200 tokens
2. Turn 2: History (4,200) + tool call (200) + git diff response (3,000) + analysis output (800) = 8,200 tokens
3. Turn 3: History (8,200) + tool call (200) + lint results (1,500) + fix output (1,200) = 11,100 tokens
4. Turn 4-10: Continues accumulating. By turn 10, history alone exceeds 30,000 tokens.

The compound effect is dramatic: 10 turns consume roughly 10x the tokens of a single chat completion, and each turn is more expensive than the last because the history grows.

How Different Tools Manage Tokens

The same model behaves differently across tools because each environment injects system prompts, manages conversation state, and handles memory with unique strategies[anthropic-tokens].

Claude Code's diff-based retention is unique: it preserves structural changes (file edits, git commits) while discarding intermediary reasoning tokens. This makes it efficient for coding workflows where the change history matters more than the reasoning that produced it[anthropic-tokens]. OpenClaw's relevance-weighted compaction preserves messages by recency and relevance, which is more appropriate for general agentic tasks where context from earlier turns may remain important.

The system prompt overhead is invisible to the user but consumes real tokens at the provider's per-token rate. OpenCode's ~1,800-token system prompt costs $0.009 per session on GPT-5.6 Sol input pricing, or roughly $270/month for 30,000 sessions — before any actual work tokens are consumed.

MCP and Tool Response Token Costs

Model Context Protocol (MCP) servers amplify token consumption by injecting tool responses into the context window as input tokens[mcp-spec]. An agent connected to a filesystem MCP server, a PostgreSQL MCP server, and a GitHub MCP server can generate tool responses totaling 10,000-50,000 tokens per task.

A typical MCP interaction consumes:

Where $N_{\text{call}}$ is the tool call itself (typically 100-300 tokens including tool definition and arguments) and $N_{\text{response}}$ is the tool's response. A database query returning 50 KB of results adds approximately 12,500 input tokens. A web search returning 10 results adds 3,000-5,000 tokens. A file read adds just 50-200 tokens but may be repeated across many files.

The challenge with MCP is that tool responses are unpredictable. You cannot know in advance whether a search will return 3 results or 30, or whether a database query will return 10 rows or 10,000. This uncertainty makes budget planning for MCP-enabled agents harder than for simple chat completions.

Session Compaction and Context Management

When the cumulative context — including all history, tool responses, reasoning, and output — approaches the model's limit, the system must decide what to discard. This process is called context compaction[lost-in-middle].

Each compaction strategy has different implications for performance:

• Summarization (ChatGPT): Replaces old conversation history with a condensed summary. This preserves key facts but loses nuance, tone, and exact phrasing. Useful for general chat where only the current topic matters.
• Sliding window (Cursor): Drops the oldest messages when a hard limit is reached. Simple and predictable but loses context that may become relevant again later. Problematic for tasks where information from early turns is revisited later.
• Relevance-weighted (OpenClaw): Preserves messages by recency and relevance score. More sophisticated but introduces unpredictability: the user cannot know which messages survived compaction.
• Diff-based (Claude Code): Preserves structural changes to files while discarding intermediary reasoning. Optimized for coding: you keep the "what changed" and lose the "why we changed it."

Research on the lost-in-the-middle phenomenon shows that even without compaction, not all context positions are equal[lost-in-middle]. Models perform best when relevant information appears at the beginning or end of the context, and performance degrades for information in the middle. This means compaction strategy matters more than raw context window size. A well-compacted 200K context can outperform a poorly managed 1M context for tasks requiring consistent access to information across a session.

Budgeting Tokens for Production

Token budgeting for production systems must account for three categories: fixed overhead, variable content, and agentic multiplier.

Fixed overhead: System prompt + tool definitions + conversation starter templates. These are predictable and should be measured once per environment. A typical agentic setup runs 1,500-2,500 tokens of fixed overhead per session.

Variable content: User inputs, MCP tool responses, and model outputs. These vary by task and cannot be predicted exactly, but historical averages should guide budgeting. Log actual token consumption in production and use the 90th percentile for capacity planning.

Agentic multiplier: The ratio of total tokens consumed to visible output tokens. A multiplier of 3-5x is typical for non-coding agentic tasks. A multiplier of 5-10x is typical for coding agentic tasks with tool calls and iteration. Measure your specific multiplier by comparing API dashboard token counts to your content estimation.

A production budget formula for a single agent session with 10 turns on GPT-5.6 Terra:

Where $P_{\text{blended}}$ is the blended price accounting for input/output ratio (approximately $5.50/M for GPT-5.6 Terra at a 3:1 input:output ratio). At 2,000 tokens fixed overhead + 3,000 tokens per turn variable, a 10-turn session costs about $0.18. The same session without the agentic multiplier — just a single chat completion — would cost about $0.02. The agentic multiplier is 9x for this scenario.

Putting It All Together: A Practical Example

Consider a developer building a documentation generator using OpenCode with GPT-5.6 Terra. The agent reads a codebase, generates documentation, and writes it to markdown files over multiple turns.

Session profile: 15 turns, 3 MCP tools (filesystem, git, web search), 5 file reads per turn, 2 documentation files generated.

Token breakdown per turn: 1,800 system prompt + 3,000 conversation history (cumulative) + 500 file read responses + 1,000 reasoning + 800 output = approximately 7,100 tokens per turn average. Over 15 turns, total consumption is approximately 106,500 tokens.

Cost at GPT-5.6 Terra: Approximately $0.59 for the entire session. The same work as a manual documentation task would cost roughly $0.07 in chat completions — but the agentic approach requires human review time that the chat approach does not. The 8.4x token multiplier includes the value of autonomous operation.

Optimization levers:

1. Increase compaction threshold to reduce history overhead (saves ~20%)
2. Cache the system prompt (90% discount on 1,800 tokens per session)
3. Reduce unnecessary file reads (profile which turns actually need which files)
4. Use batch API for the documentation generation step (50% discount)

Frequently Asked Questions

❓ How many tokens does a typical agentic task consume versus a chat completion?

✅ An agentic task with 10 turns consumes roughly 10x the tokens of a single chat completion for the same final deliverable. Each turn adds conversation history, tool responses, and reasoning to the context, compounding token consumption.

❓ Why does the same model cost different amounts in different tools?

✅ Each tool injects its own system prompt (500-2,000 tokens), manages conversation history differently, and applies different compaction strategies. These invisible overheads add to the base token count of your actual content, changing effective costs by 20-70%.

❓ How much does MCP add to token consumption?

✅ MCP tool responses vary from 50 tokens (file read) to 50,000 tokens (database query). A typical MCP-enabled agent with 10 tool calls per task consumes 5,000-50,000 tokens in tool response input alone, beyond the conversation content.

❓ What is the best compaction strategy for long-running agents?

✅ Relevance-weighted compaction (used by OpenClaw) balances recency and importance, preserving context that remains relevant. Diff-based retention (used by Claude Code) works best for coding where structural changes matter more than reasoning.

❓ How do I budget tokens for a production agent?

✅ Measure fixed overhead once (system prompt, tool defs), track variable content by task type, and establish your agentic multiplier (ratio of total tokens to visible output). Use the 90th percentile for capacity planning and add 20% headroom for retries.

❓ Does token counting work the same for reasoning models?

✅ No. Deep reasoning models generate extensive hidden chain-of-thought that is billed as output tokens but not returned to the user. This hidden output can multiply visible output by 2-4x on complex tasks. Budget accordingly.

❓ How does session compaction affect response quality?

✅ Aggressive compaction improves response speed and reduces costs but may lose nuance from earlier turns. The lost-in-the-middle effect means models already struggle with mid-context information; compaction exacerbates this by removing context that could help.

❓ Should I use the same tokenization strategy across all models?

✅ No. Each provider's tokenizer has different ratios. Optimize text for the specific model you use most. For multi-model systems, budget for the most expensive tokenizer and treat the savings from cheaper models as upside.