How text is cut into tokens
A tokenizer is a vocabulary of common pieces of text. A common word is one token, a rare one several; the space is
usually glued to the start of a word. The output was taken with js-tiktoken on the o200k_base vocabulary (GPT-6 and
GPT-5); Claude has its own vocabulary, built the same way. The Russian rows show why other languages cost more.
| Text | Tokens | How it’s cut |
|---|---|---|
Hello, how are you? |
6 | Hello · , · how · are · you · ? |
Привет, как дела? (Russian, the same) |
6 | Пр · ивет · , · как · дела · ? |
Check from your phone over mobile data. |
8 | a token per word, plus the period |
Проверяем с телефона по мобильному интернету. (Russian, the same) |
11 | П · ровер · яем · с · телефона · по · мобиль · ному · интернет · у · . |
strawberry |
3 | st · raw · berry |
2026-10-18 |
6 | 202 · 6 · - · 10 · - · 18 |
docker compose up -d |
5 | docker · compose · up · - · d |
Words in inflected languages get cut finer: endings and prefixes go as separate tokens. Over a large volume the gap is smaller than on one phrase:
| Measured: 34 site pages, text only, no code | RU | EN |
|---|---|---|
| characters | 31,911 | 33,719 |
tokens, o200k_base (newer OpenAI models) |
9,332 | 8,072 |
| characters per token | 3.42 | 4.18 |
| tokens per word | 1.80 | 1.32 |
tokens, cl100k_base (GPT-4, 2023) |
13,252 | 8,062 |
The same meaning in Russian is 1.16 times as many tokens on the new vocabulary and 1.64 times on the 2023 one. Newer tokenizers have nearly closed the gap; what that means for choosing a prompt language — Which language to prompt in.
A rough estimate without a counter: a page of English text (about 3,000 characters, 500 words) is roughly 700 tokens on
o200k_base; on Claude, 15–20% more. A 300-page book is 150–250 thousand tokens.
What the price is made of
price = input × input_price + output × output_price
↑ everything sent: system, history, files, tools
↑ the answer + reasoning (also output)
| Term | What’s in it | How to make it cheaper |
|---|---|---|
| input | system prompt, the whole chat history, attached files, tool descriptions, tool results | caching: repeated input is 10–40 times cheaper, depending on the model (Prompt cache) |
| output | the answer text | ask for brevity; JSON instead of prose (JSON by schema) |
| reasoning | the draft reasoning before the answer, even when hidden | a lower effort on simple tasks (Match reasoning to the task) |
| images | on Claude, one token per 28×28-pixel square: a 1000×1000 screenshot is 1,296 tokens, 4K up to 4,784 | crop to the part of the screen you need; 5.5 models downscale anything over 2576 pixels on the long side anyway |
Counting
Estimate the tokens in your text
For OpenAI models the count is exact; for Claude it’s 15–20% low, but you see the order of magnitude:
mkdir -p ~/tok && cd ~/tok && npm init -y >/dev/null && npm i [email protected] cat > count.mjs <<'EOF' import { getEncoding } from 'js-tiktoken'; import fs from 'node:fs'; const text = fs.readFileSync(process.argv[2], 'utf8'); const tokens = getEncoding('o200k_base').encode(text).length; console.log(`${text.length} chars, ${tokens} tokens, ${(text.length / tokens).toFixed(2)} chars per token`); EOF node count.mjs ~/site/CLAUDE.md # 19325 chars, 6423 tokens, 3.01 chars per tokenThat’s an agent instruction file in Russian with paths and commands: 3.01 characters per token versus 3.42 for plain Russian text. Code, paths and tables get cut finer than prose.
Count exactly for Claude — count_tokens
A separate API endpoint counts input without generating: the same request that will do the work, but no answer. Use the same model that will run it — different generations have different tokenizers.
jq -n --rawfile t ~/site/CLAUDE.md \ '{model: "claude-sonnet-5-5", messages: [{role: "user", content: $t}]}' | curl -s https://api.anthropic.com/v1/messages/count_tokens \ -H "x-api-key: $ANTHROPIC_API_KEY" \ -H "anthropic-version: 2023-06-01" \ -H "content-type: application/json" \ -d @- # {"input_tokens":…}The key is in an environment variable, not in the command — how to put it there is in First API request.
Turn it into money
A request: 20,000 input tokens (code and a question), 2,000 output, with reasoning on, taking another 3,000.
Model Input Output (2,000 + 3,000) Total per request Haiku 5.5 ($0.10 / $0.50) $0.002 $0.0025 $0.0045 Sonnet 5.5 ($2 / $10) $0.04 $0.05 $0.09 Opus 5.5 ($4 / $20) $0.08 $0.10 $0.18 Fable 5.1 ($10 / $50) $0.20 $0.25 $0.45 A single request is cheap on any tier. It gets expensive when there are thousands of requests or the history is long.
Price an agent session — the history rides along every turn
On every turn an agent sends the whole history: its previous answers, command output, the files it read. A 40-turn session, with the history growing from 15,000 to 150,000 tokens:
input per session ≈ turns × average history length = 40 × ~82,000 ≈ 3.3M tokens Sonnet 5.5 without cache: 3.3M × $2 ≈ $6.60 for input alone with cache (reads at $0.20): ≈ $0.66 + writing new chunks — several times cheaperThat’s why agents keep caching on, and why a long session is cheaper split into tasks with a short summary between them (Long conversations).
Compare with the actual bill — usage in the response
Every API response carries the bill:
"usage": { "input_tokens": 1840, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 18200, "output_tokens": 4920 }input_tokensis input outside the cache,cache_read_input_tokenswas read from the cache at the low price,cache_creation_input_tokenswas written to the cache (1.25 times the regular input price), andoutput_tokensis the answer including reasoning. Daily and monthly spend is in the provider’s console, where you also set a spending limit.
How to check
Send the same long request twice in a row with caching (Prompt cache): in the second response
cache_read_input_tokens should nearly equal the first one’s input, and input_tokens should be small. If both times
show cache_read_input_tokens: 0, the cache didn’t work and you’re paying full price for input.
If it doesn’t work
The monthly bill is several times what you “roughly estimated”
- Cause
- You priced one request, but an agent or a chat resends the whole history each time: turn one hundred carries the previous 99.
- Fix
- Count from the
usagein responses, turn on the prompt cache (Prompt cache), and cut long sessions into tasks (New task, new chat).
The tiktoken count doesn’t match Claude’s usage
- Cause
tiktokenis OpenAI’s tokenizer. Claude has its own: on ordinary text it gives 15–20% more tokens, more still on code and non-English text.- Fix
- For Claude, count with
count_tokensusing the same model that will do the work.
Switched to a new model — more tokens on the same prompts
- Cause
- Claude Opus 4.7 and newer have a different tokenizer: the same text is 1–1.35 times as many tokens as on Opus 4.6 and older.
- Fix
- Recount the budget with
count_tokenson the new model instead of carrying over the old figures.
The model can’t count the letters in a word or mixes up digits in a long number
- Cause
- It sees tokens, not letters:
strawberryisst,raw,berryto it, and the date2026-10-18is six pieces. - Fix
- Leave character-level work to a script; the model writes the script (Where models fail).