basforge

Search titles, text and the “If it doesn’t work” cases.

    RU

    Which language to prompt in: non-English costs more tokens, not more misunderstanding

    The same text in Russian takes 1.16 times as many tokens as in English on the o200k_base vocabulary; on the 2023 vocabulary it was 1.64 times. Strong models understand a prompt in another language as well as in English, so write in the language you phrase things most precisely in, and set the answer language explicitly.

    What another language costs

    This site exists in Russian and English, which gives a direct comparison: the same pages, two languages.

    Vocabulary RU, tokens EN, tokens RU / EN
    o200k_base (GPT-6, GPT-5) 9,332 8,072 1.16
    cl100k_base (GPT-4, 2023) 13,252 8,062 1.64

    Measured on 34 of this site’s pages, text only, no code. Claude has its own vocabulary, and for non-English text it gives more tokens than o200k_base; for the exact number use count_tokens (Tokens). Code, commands and paths are the same in both languages, so in a prompt with a large config the gap is smaller. Other languages differ from English by their own ratio — measure on your texts.

    What to choose

    Case Prompt language
    a chat, an explanation, a task in your own words the one you think in: precise phrasing matters more than 16%
    a system prompt sent a million times English, if money matters; put the answer language in the rules
    a small local 4–8B model (A model on your own hardware) English: such models are trained mostly on it and make more mistakes in other languages; how many more, test on your own prompts in both
    terms, commands, config names as they are, untranslated: entryPoints, not a translation of it

    Set the answer language explicitly

    A model answers in the question’s language, but in a long context full of English documents it may slide into English — or, with documents in another language, out of it. One line in the system prompt settles it: “Answer in German; keep code, commands and terms as in the original.”