basforge

Search titles, text and the “If it doesn’t work” cases.

    RU

    Tokens: measuring text size and the price of a request

    A model reads and writes in tokens — pieces of words. Text in other languages is often longer in tokens than the same text in English (Russian by 16%), and you pay for every input and output token, reasoning included. Count the tokens in your text, the price of one request and the price of an agent’s work — before the bill arrives.

    How text is cut into tokens

    A tokenizer is a vocabulary of common pieces of text. A common word is one token, a rare one several; the space is usually glued to the start of a word. The output was taken with js-tiktoken on the o200k_base vocabulary (GPT-6 and GPT-5); Claude has its own vocabulary, built the same way. The Russian rows show why other languages cost more.

    Text Tokens How it’s cut
    Hello, how are you? 6 Hello · , · how · are · you · ?
    Привет, как дела? (Russian, the same) 6 Пр · ивет · , · как · дела · ?
    Check from your phone over mobile data. 8 a token per word, plus the period
    Проверяем с телефона по мобильному интернету. (Russian, the same) 11 П · ровер · яем · с · телефона · по · мобиль · ному · интернет · у · .
    strawberry 3 st · raw · berry
    2026-10-18 6 202 · 6 · - · 10 · - · 18
    docker compose up -d 5 docker · compose · up · - · d

    Words in inflected languages get cut finer: endings and prefixes go as separate tokens. Over a large volume the gap is smaller than on one phrase:

    Measured: 34 site pages, text only, no code RU EN
    characters 31,911 33,719
    tokens, o200k_base (newer OpenAI models) 9,332 8,072
    characters per token 3.42 4.18
    tokens per word 1.80 1.32
    tokens, cl100k_base (GPT-4, 2023) 13,252 8,062

    The same meaning in Russian is 1.16 times as many tokens on the new vocabulary and 1.64 times on the 2023 one. Newer tokenizers have nearly closed the gap; what that means for choosing a prompt language — Which language to prompt in.

    A rough estimate without a counter: a page of English text (about 3,000 characters, 500 words) is roughly 700 tokens on o200k_base; on Claude, 15–20% more. A 300-page book is 150–250 thousand tokens.

    What the price is made of

    price = input × input_price + output × output_price
              ↑ everything sent: system, history, files, tools
                                       ↑ the answer + reasoning (also output)
    Term What’s in it How to make it cheaper
    input system prompt, the whole chat history, attached files, tool descriptions, tool results caching: repeated input is 10–40 times cheaper, depending on the model (Prompt cache)
    output the answer text ask for brevity; JSON instead of prose (JSON by schema)
    reasoning the draft reasoning before the answer, even when hidden a lower effort on simple tasks (Match reasoning to the task)
    images on Claude, one token per 28×28-pixel square: a 1000×1000 screenshot is 1,296 tokens, 4K up to 4,784 crop to the part of the screen you need; 5.5 models downscale anything over 2576 pixels on the long side anyway

    Counting

    1. Estimate the tokens in your text

      For OpenAI models the count is exact; for Claude it’s 15–20% low, but you see the order of magnitude:

      mkdir -p ~/tok && cd ~/tok && npm init -y >/dev/null && npm i [email protected]
      cat > count.mjs <<'EOF'
      import { getEncoding } from 'js-tiktoken';
      import fs from 'node:fs';
      const text = fs.readFileSync(process.argv[2], 'utf8');
      const tokens = getEncoding('o200k_base').encode(text).length;
      console.log(`${text.length} chars, ${tokens} tokens, ${(text.length / tokens).toFixed(2)} chars per token`);
      EOF
      node count.mjs ~/site/CLAUDE.md
      # 19325 chars, 6423 tokens, 3.01 chars per token

      That’s an agent instruction file in Russian with paths and commands: 3.01 characters per token versus 3.42 for plain Russian text. Code, paths and tables get cut finer than prose.

    2. Count exactly for Claude — count_tokens

      A separate API endpoint counts input without generating: the same request that will do the work, but no answer. Use the same model that will run it — different generations have different tokenizers.

      jq -n --rawfile t ~/site/CLAUDE.md \
        '{model: "claude-sonnet-5-5", messages: [{role: "user", content: $t}]}' |
      curl -s https://api.anthropic.com/v1/messages/count_tokens \
        -H "x-api-key: $ANTHROPIC_API_KEY" \
        -H "anthropic-version: 2023-06-01" \
        -H "content-type: application/json" \
        -d @-
      # {"input_tokens":…}

      The key is in an environment variable, not in the command — how to put it there is in First API request.

    3. Turn it into money

      A request: 20,000 input tokens (code and a question), 2,000 output, with reasoning on, taking another 3,000.

      Model Input Output (2,000 + 3,000) Total per request
      Haiku 5.5 ($0.10 / $0.50) $0.002 $0.0025 $0.0045
      Sonnet 5.5 ($2 / $10) $0.04 $0.05 $0.09
      Opus 5.5 ($4 / $20) $0.08 $0.10 $0.18
      Fable 5.1 ($10 / $50) $0.20 $0.25 $0.45

      A single request is cheap on any tier. It gets expensive when there are thousands of requests or the history is long.

    4. Price an agent session — the history rides along every turn

      On every turn an agent sends the whole history: its previous answers, command output, the files it read. A 40-turn session, with the history growing from 15,000 to 150,000 tokens:

      input per session ≈ turns × average history length = 40 × ~82,000 ≈ 3.3M tokens
      Sonnet 5.5 without cache:   3.3M × $2   ≈ $6.60 for input alone
      with cache (reads at $0.20): ≈ $0.66 + writing new chunks — several times cheaper

      That’s why agents keep caching on, and why a long session is cheaper split into tasks with a short summary between them (Long conversations).

    5. Compare with the actual bill — usage in the response

      Every API response carries the bill:

      "usage": {
        "input_tokens": 1840,
        "cache_creation_input_tokens": 0,
        "cache_read_input_tokens": 18200,
        "output_tokens": 4920
      }

      input_tokens is input outside the cache, cache_read_input_tokens was read from the cache at the low price, cache_creation_input_tokens was written to the cache (1.25 times the regular input price), and output_tokens is the answer including reasoning. Daily and monthly spend is in the provider’s console, where you also set a spending limit.

    How to check

    Send the same long request twice in a row with caching (Prompt cache): in the second response cache_read_input_tokens should nearly equal the first one’s input, and input_tokens should be small. If both times show cache_read_input_tokens: 0, the cache didn’t work and you’re paying full price for input.

    If it doesn’t work

    The monthly bill is several times what you “roughly estimated”

    Cause
    You priced one request, but an agent or a chat resends the whole history each time: turn one hundred carries the previous 99.
    Fix
    Count from the usage in responses, turn on the prompt cache (Prompt cache), and cut long sessions into tasks (New task, new chat).

    The tiktoken count doesn’t match Claude’s usage

    Cause
    tiktoken is OpenAI’s tokenizer. Claude has its own: on ordinary text it gives 15–20% more tokens, more still on code and non-English text.
    Fix
    For Claude, count with count_tokens using the same model that will do the work.

    Switched to a new model — more tokens on the same prompts

    Cause
    Claude Opus 4.7 and newer have a different tokenizer: the same text is 1–1.35 times as many tokens as on Opus 4.6 and older.
    Fix
    Recount the budget with count_tokens on the new model instead of carrying over the old figures.

    The model can’t count the letters in a word or mixes up digits in a long number

    Cause
    It sees tokens, not letters: strawberry is st, raw, berry to it, and the date 2026-10-18 is six pieces.
    Fix
    Leave character-level work to a script; the model writes the script (Where models fail).