Tokens and the context window

What it isCopy Anchor Link

A token is a piece of text, usually a word or part of a word. "unbelievable" might be three tokens. Code, numbers and non-English text usually cost more tokens per character than plain English prose. A rule of thumb for English is about 4 characters or 0.75 words per token.

The context window is the maximum number of tokens the model can see at once. It holds your system prompt, the whole conversation so far, any files or tool results, and the answer being written. When it is full, something has to be dropped or summarized.

Why it mattersCopy Anchor Link

Cost, speed and quality all follow token count. Developers who use a model through an API pay per input and output token. Long inputs are slower. Past a point, quality drops too: models get worse at using facts buried in the middle of a very long context, a failure often called context rot.

Try itCopy Anchor Link

Paste a paragraph of prose, a JSON blob and some code into a tokenizer viewer and compare tokens per character. Then count the tokens in the longest message you have ever sent a chatbot.

Worked example: counting tokens and costCopy Anchor Link

The sentence "I love learning about AI!" is about 7 tokens. A 1,000-word email is around 1,300 tokens. If a model charges $3 per million input tokens, that email costs about $0.004 to read. Cheap once, but an agent that re-reads a 50,000-token conversation 30 times pays for 1.5 million tokens: about $4.50. Tokens are the unit of cost, speed and memory.

How text becomes tokensCopy Anchor Link

A tokenizer holds a fixed list of pieces, built by repeatedly merging character pairs that appear often in training text (this method is called BPE). To encode text it starts from single characters and joins pairs using those learned merges, in the order it learned them. For example, "t" and "h" join into "th", then "th" and "e" into "the". Then it replaces each resulting piece with an ID number. The model only ever sees those numbers.

Common words are usually one piece. Rare words split into several. An invented split: "unbelievable" becomes "un", "believ", "able". This also explains why models struggle to count letters. The letters were never given to the model, only piece IDs.

Why long chats get expensiveCopy Anchor Link

Every turn resends the whole conversation as input. Invented numbers: each turn adds 500 tokens. Turn 1 sends about 500. Turn 2 sends turns 1 and 2 together, about 1,000. Turn 3 sends all three, about 1,500, and so on. Over 10 turns the total sent is 500 × (1 + 2 + ... + 10) = 500 × 55 = 27,500 tokens, although the conversation holds only 5,000 tokens of text. Cost grows faster than the number of turns. Providers can discount text that repeats at the start of every call, but the window still fills.

What goes wrong when the window fillsCopy Anchor Link

Symptoms are easy to spot:

1. The assistant forgets an instruction from the start, because the app trimmed old messages.
2. A reply ends mid-sentence, because the limit was reached.
3. Costs per message keep rising during one session.

To confirm, count tokens before sending. Most APIs also report why generation stopped, for example hitting the length limit.

Common mistakesCopy Anchor Link

Assuming a bigger window means a better answer. More text can bury what matters. Another mistake is forgetting that the answer also uses window space, so a nearly full window leaves little room to reply.