Token Counter Calculator

What Is a Token?

A token is the fundamental unit language models use to process text. When you send a prompt to ChatGPT, Claude, Gemini, or any LLM, the model does not read characters or words the way a human does. It breaks your input into chunks called tokens, each corresponding to a subword unit that the model processes as a numerical vector[openai-tokenizer]. Every token processed is billable at the provider's per-token rate, making token awareness the foundation of LLM cost management[openai-pricing]. This tokenization step is invisible to the user but determines everything from cost to response quality.

The approximate relationship between tokens and text follows a simple formula:

Where $N_t$ is the estimated token count, $C$ is the character count of the text, and $R$ is the model-specific characters-per-token ratio. For GPT-5.6 models, $R \approx 3.8$, while Gemini models average closer to $R \approx 4.0$. This means a 10,000-character document runs approximately 2,500-2,700 tokens.

To estimate the dollar cost of processing those tokens:

Where $P_{\text{input}}$ and $P_{\text{output}}$ are the per-token prices (price per million tokens divided by 1,000,000). Output tokens are typically 3-6x more expensive than input tokens because generating text requires sequential autoregressive inference, while input processing can be parallelized[anthropic-pricing].

For agentic applications — where the model makes multiple tool calls, revises its reasoning, or interacts with MCP servers — the effective token consumption is the sum of all turns:

Agentic loops multiply token consumption. A single agentic task that calls 3 tools, generates 2 intermediate reasoning steps, and produces a final response may consume 5-10x more tokens than a simple chat completion.

Tokenizer Differences Across Providers

Not all tokenizers are created equal. OpenAI's GPT-5.6 family uses a BPE tokenizer trained on English text and code, reaching roughly 3.8 characters per token. Anthropic's Claude models use a SentencePiece-based tokenizer that averages slightly higher at 3.9 characters per token. Google's Gemini tokenizer targets multilingual efficiency at 4.0 characters per token, a deliberate trade-off that penalizes English slightly but improves Japanese and Arabic tokenization[google-pricing].

The BPE algorithm works by iteratively merging the most frequent byte pairs in a training corpus, building a vocabulary of 50,000-100,000 tokens[sennrich-bpe]. Common words like "the" and "and" become single tokens, while rare words split into subword units. "Tokenization" might split into "token" and "ization" as two tokens. A camelCase variable like "calculateMonthlyRevenue" might split into "calculate", "Monthly", and "Revenue" as three tokens.

This variability has direct consequences for cost. The same prompt sent through ChatGPT (OpenAI tokenizer), Claude Code (Anthropic tokenizer), and OpenClaw (DeepSeek tokenizer) can differ by 5-15% in token count, which translates to real dollar differences at scale. DeepSeek's aggressive pricing — $0.14 per million input tokens for V4 Flash — makes tokenizer efficiency particularly important when every token saved reduces cost in the most budget-friendly tier available[deepseek-pricing]. The calculator above provides model-specific estimates rather than a single universal count.

Token Consumption in Agentic Workflows

Traditional chat completions follow a simple input-output pattern: you send a prompt, the model responds, the conversation ends. Agentic AI changes this fundamentally. An agentic system — whether running in Claude Code, OpenCode, Cursor, or a custom MCP client — makes decisions, calls tools, processes results, and iterates. Each cycle consumes tokens.

Consider an agentic code review task. The agent reads a file (2,000 tokens input), generates a review (1,000 tokens output), calls a git diff tool (500 tokens input from tool output), generates a pull request summary (800 tokens output), and writes test suggestions (1,200 tokens output). Total: 2,500 input tokens and 3,000 output tokens over one multi-step task — roughly 3x the token consumption of a simple chat completion for the same final result.

MCP (Model Context Protocol) servers compound this. An agent connected to a Postgres MCP server, a GitHub MCP server, and a file system MCP server consumes input tokens for each tool's response. A database query returning 500 rows of results might consume 3,000-5,000 input tokens from the tool response alone, before the model does anything with that data.

The calculator above models base token consumption. For agentic workloads, multiply the result by the expected number of agent iterations to estimate true consumption.

How Different Tools Tokenize the Same Model

The same model behaves differently across tools because each environment injects system prompts, manages conversation context, and handles memory differently. These factors add 500-3,000 tokens of overhead per conversation.

[anthropic-cache] Each tool adds overhead tokens for system prompts and memory management:

The system prompt overhead is invisible to you but consumes real tokens. Claude Code injects approximately 1,200 tokens of system instructions about coding conventions, file operations, and safety guidelines. OpenCode injects around 1,800 tokens covering tool definitions, MCP server configurations, and session management. These overheads are billed like any other token and must be factored into cost estimates.

Session compaction — the mechanism that decides which conversation history to keep or discard — also varies dramatically. OpenClaw compacts when the context budget reaches 70%, preserving the most relevant messages while discarding low-value reasoning. Claude Code uses byte-level diff-based retention that keeps structural changes while discarding intermediary analysis. ChatGPT summarises the conversation after roughly 8K tokens, losing specificity in favor of conciseness. These compaction strategies affect token consumption patterns and whether the model "remembers" details from earlier in the conversation.

Practical Examples

A typical customer support conversation on ChatGPT: 600 token system message, 500 token initial user query, 300 token assistant response, three follow-up exchanges averaging 200+ tokens each. Total: approximately 2,700 tokens. The same conversation handled by an OpenCode agent with MCP access to a knowledge base would add tool call overhead, roughly doubling consumption.

A code review across 3 files (500 tokens each): 1,500 input tokens, 800 output tokens with suggestions, plus tool call tokens for reading each file. Running through Claude Code adds 1,200 tokens of system prompt overhead. The actual consumption is roughly 3,500-4,000 tokens, not 2,300.

A document summarization of a 10-page report (approximately 6,000 chars, 1,500 tokens): The summary output is approximately 300 tokens. Total direct consumption is 1,800 tokens. Running through an MCP-enabled agent that also searches the web for supplementary data adds another 2,000-3,000 tokens from search results alone.

Practical Tips for Token Management

The most effective way to reduce token consumption is prompt optimization. Remove filler phrases, use direct instructions, and structure prompts to minimize overhead. A well-crafted 200-token prompt can accomplish the same task as a poorly written 500-token prompt, saving 60% of input costs.

For agentic workloads, minimize the number of tool calls per task. Each tool call adds input tokens from the tool's response. Batch operations when possible: rather than reading 10 files individually with 10 separate tool calls, concatenate them into a single read operation if the tool supports it.

Monitor session compaction frequency. If your agentic workflow consistently triggers compaction, it is a sign that context is accumulating faster than necessary. Consider reducing conversation history retention or increasing compaction thresholds.

Use the Token Counter Calculator above to estimate consumption for your specific text, and cross-reference with the LLM API Cost Calculator for cost projections. The Context Window Calculator helps ensure your combined input and output fit within the model's limits before you send the request.

Limitations

• Approximate estimates: Token counts are based on statistical ratios and may differ from actual tokenizer output by 10-15%. For production billing, always use the official tokenizer library (tiktoken for OpenAI, sentencepiece for Anthropic and Google).
• No multimodal support: This calculator estimates text tokens only. Images, audio, and video consume tokens through separate encoders with different ratios that are not modeled here.
• Tool overhead excluded: The token overhead from MCP servers, tool calls, and system prompts is not included in the base estimate. Add 500-3,000 tokens per environment for system prompt overhead.
• No agent loop modeling: Agentic workflows that iterate multiple times consume tokens multiplicatively, not additively. The calculator models a single input-output cycle.
• Tokenizers evolve: Providers periodically update their tokenizer implementations. Ratios can shift by 5-10% with new releases.
• Non-English variance: Estimates are most accurate for English prose. Chinese, Japanese, Korean, and mixed-language content may deviate significantly.

Frequently Asked Questions

❓ How many tokens does an average English word consume?

✅ One English word typically equals 1.25 to 1.35 tokens due to subword tokenization. A 1,000-word document uses approximately 1,250-1,350 tokens depending on the model's tokenizer.

❓ Why does the same text use different token counts in different tools?

✅ Each tool injects its own system prompt (500-2,000 tokens), manages conversation history differently, and applies session compaction at different thresholds. These invisible overheads add to the base token count of your actual content.

❓ Do agentic AI workflows consume more tokens than chat?

✅ Yes, significantly. Agentic tasks that make tool calls, process tool outputs, and iterate on reasoning can consume 3-10x more tokens than a simple chat completion for the same objective.

❓ How does MCP affect token consumption?

✅ MCP server responses are consumed as input tokens. A database query returning 500 rows can add 3,000-5,000 input tokens. Every MCP tool call adds its response to the context.

❓ What is session compaction and how does it save tokens?

✅ Session compaction discards low-value conversation history when the context approaches its limit. Different tools use different strategies: summarization (ChatGPT), selective retention (OpenClaw), or byte-level diff (Claude Code). Compaction reduces token consumption but may lose nuance.

❓ How accurate is this token counter for code?

✅ Code tokenizes differently than prose. Keywords like 'function' are single tokens, but variable names and string literals split into multiple subwords. The estimate may be 15-25% off for code-heavy inputs.

❓ Should I use different token counters for different providers?

✅ Yes, for production systems. The calculator estimates based on model-specific ratios, but exact counts require the actual tokenizer library (tiktoken for OpenAI, sentencepiece for Anthropic and Google).

❓ What is the single biggest factor in token consumption?

✅ Conversation history accumulation in long-running sessions. A 50-turn conversation with 200 tokens per turn consumes 10,000+ tokens just for history. Use sliding windows or summarization to keep history bounded.