How it works
| Term | What it means |
|---|---|
| LLM | a large language model: a neural network that continues text token by token (Language models) |
| token | a piece of text, the unit of counting: about 4.2 characters on average in English (Tokens) |
| tokenizer | the vocabulary of pieces and the cutting rule; each family has its own |
| parameters, weights | the numbers inside the network; 8B is 8 billion |
| transformer | the architecture of nearly every LLM: attention layers and feed-forward layers |
| attention | the mechanism that links every token to every previous one |
| context, context window | everything the model takes into account in one request, and its limit in tokens (A model request, block by block) |
| inference | running a finished model to generate an answer, as opposed to training |
| MoE | mixture of experts: part of the subnetworks works on each token; 30B-A3B means 3B active out of 30B (How much memory a model needs) |
| multimodal model | accepts more than text: images, PDFs, audio |
| embedding | a vector of hundreds of numbers, close for texts close in meaning (Your own data in a model) |
Training
| Term | What it means |
|---|---|
| pretraining | training to predict the next token on trillions of tokens of text |
| knowledge cutoff | the last thing that made it into the data: the model knows nothing later (Put the date, versions and environment in the prompt) |
| base model | after pretraining: continues text but doesn’t answer questions |
| instruct, chat model | fine-tuned to answer in a dialogue |
| SFT | fine-tuning on “question — good answer” examples |
| RLHF | reinforcement learning from human ratings; the source of politeness and flattery |
| RLAIF, constitutional AI | the same, but a model grades by written principles |
| reasoning model | trained to write its thinking before the answer (Match reasoning to the task) |
| distillation | a small model learns from a big one’s answers |
| fine-tuning | further training of a finished model on your own examples (Your own data in a model) |
| LoRA | cheap fine-tuning: you train a small add-on to the weights, not all the weights |
Request and response
| Term | What it means |
|---|---|
| system prompt | instructions for the whole conversation, with priority over turns (A model request, block by block) |
| prompt | the whole request text; narrowly, your instruction (A prompt, block by block) |
| few-shot | a few example answers right in the prompt (An example over a description) |
| sampling | picking the next token from the probabilities with some randomness |
temperature, top_p, top_k |
knobs for that randomness; removed on newer Claude models (Repeatable answers) |
max_tokens |
the ceiling on answer length, required in the API |
stop_reason |
why the model stopped: end_turn, max_tokens, tool_use (A model request, block by block) |
effort |
Claude’s reasoning depth: low … max (Match reasoning to the task) |
| streaming | the text arrives in chunks as it’s generated |
| structured output | an answer strictly by a JSON schema (JSON by schema) |
| prefill | writing the start of the answer for the model; disabled on newer Claude models |
| prompt cache | a repeated request prefix is read 10–40 times cheaper (Prompt cache) |
| batch processing | requests in a batch, answered within 24 hours, at half price (Non-urgent API requests go as a batch at half price) |
| compaction | replacing the old part of the history with a summary as the window fills up (Long conversations) |
| TTFT | time to first token of the answer |
| tokens per second | generation speed (How much memory a model needs) |
Size and running
| Term | What it means |
|---|---|
| open weights | the weights can be downloaded and run locally; training data is usually closed |
| quantization, quant | storing weights with fewer bits: Q4_K_M, Q8_0 (How much memory a model needs) |
| GGUF | the model file format for llama.cpp and Ollama |
| llama.cpp | an engine that runs models on CPU and GPU; Ollama is built on it |
| Ollama | a program that downloads models, runs them and serves an API on 11434 (A model on your own hardware) |
| vLLM | a server for multi-user GPU inference that handles many requests at once |
| VRAM | video memory: the model and its context must fit in it (How much memory a model needs) |
| KV cache | memory for the context, growing with every token (How much memory a model needs) |
| speculative decoding | a small model guesses tokens and a big one verifies them in a batch — faster |
Data and search
| Term | What it means |
|---|---|
| RAG | retrieving document chunks before answering and putting them into the prompt (Your own data in a model) |
| vector database | stores embeddings and finds the nearest ones: pgvector, Qdrant |
| chunk | a piece of a document for the index, usually 300–800 tokens |
| reranker | a model that re-sorts search results by closeness to the question |
| BM25 | classic keyword search; finds exact names that vector search misses |
| grounding, citation | an answer backed by a source with a pointer to the place in it (Long document first) |
Agents
| Term | What it means |
|---|---|
| tool, function calling | a function the model can ask to have called (Agents and tools) |
| agent | a model with tools in a “call — result” loop until the task is done |
| MCP | an open standard for connecting tools to agents |
| server tool | runs at the provider: web search, code execution |
| subagent | an agent the main one hands part of the task to, with its own clean context |
| sandbox | an isolated environment where the agent is allowed anything |
CLAUDE.md, AGENTS.md |
a file in the repository with standing instructions for the agent (Long conversations) |
Quality and risks
| Term | What it means |
|---|---|
| hallucination | a plausible but wrong answer (Where models fail) |
| sycophancy | agreeing with the user against the facts (Review in a new chat) |
| prompt injection | instructions hidden in the input: a web page, a file, an email (Someone else’s text is data, not commands) |
| jailbreak | an attempt to get around the model’s rules by wording |
| refusal | the model won’t answer under its safety rules; in the API, stop_reason: refusal |
| benchmark | a set of tasks for comparing models; some of it may have leaked into training |
| eval | your own task set for testing a model or prompt on your own work |
| data contamination | test tasks ended up in the training data, inflating the score |