Module 2 · Know the tool
Tokens and the context window
In short
Models read and write in tokens. A full context window makes the start or middle get lost, and a tight output limit can cut an answer short or leave it empty.
After this lesson you can
- Explain what a token is and why costs and limits are counted in it
- Recognise the symptoms of a context window that is too full
- Understand why an output limit that is too small can break an answer
Terms in this lesson
Understand
A language model does not read letter by letter or word by word as we do. It reads and writes in pieces of text called tokens. A token can be a short word, part of a word, a punctuation mark, or a space with a few letters. Common words usually take one token; long or rare words are split into several.
As a rough picture: one page of ordinary text, about 500 words, comes to somewhere between a few hundred and about a thousand tokens. The exact number depends on the language and the model. Indonesian text usually takes more tokens than English text with the same content, because many tokenisers were built with more English examples, and our words are often long with affixes such as "memper-" and "-kan". So the same document can cost more and fill limits sooner when written in Indonesian.
Why does this matter? Because almost every limit and cost is counted in tokens. Paid services count the tokens going in (prompt, material, conversation history) and the tokens coming out (the answer). Chat apps that do not charge per token still have usage limits counted in a similar way.
The limit you feel most is the context window: the number of tokens the model can see at once in one go. It holds more than your latest question. Standing instructions, the whole conversation history, files you paste and the answer being written all go in. As a conversation keeps growing, the window slowly fills.
When it is full, the app has to choose. Some drop the earliest part. Some summarise the older conversation into a short note, so the detail is lost. And even before it is full, models tend to be less careful with the middle of very long text: the start and the end get more attention, the middle is easier to skip. So what gets lost may be an instruction at the start, a decision in the middle, or the middle chapter of a report.
Besides the context window there is an output limit: the most tokens the model may write in one answer. This limit is mostly set by the app or by the service settings developers use, not by your request. Writing "answer as long as possible" does not get past it. If the limit is too small, the answer stops mid-sentence or a table is cut off in the middle, often with no error message.
Reasoning models make this trickier. This kind of model writes out thinking steps before the answer, and those thinking tokens usually count against the same output limit. If the limit is tight, the tokens run out on thinking and the answer comes back empty. If you use a tool that lets you set this limit, do not set it too frugally. If you only use a chat app, just split big tasks into parts and ask clearly for the continuation.
Before
The prompt
[pastes an 80-page annual report] I need a summary of this report.
What came back
This report covers the company's performance over the year. The opening explains the vision and market challenges. At the end, management concludes the year was challenging but the company still grew, and sets out plans for next year.
After
The prompt
I will send this annual report chapter by chapter. This is chapter 4, on performance by region. I need five main points and every target and actual figure it mentions, for the managers' meeting on Friday. If a figure has nothing to compare against, flag it. [chapter 4 text]
What came back
Chapter 4, five main points: 1. The eastern region beat its sales target: 112 percent of target. 2. The western region fell short: 87 percent, because two warehouses closed for renovation. 3. Shipping costs rose in every region. 4. New customers grew fastest in the central region. 5. Next year's targets are raised for all regions. Flagged: the shipping cost rise (point 3) has no comparison figure.
What changed
- The first summary only took the start and end of the report. The middle chapters, where the key figures are, vanished.
- Sending chapter by chapter keeps each part small and clearly visible to the model.
- Naming the purpose and reader tells the model what to look for: target and actual figures.
Common mistake
- Mistake
- Using the same single conversation for weeks for all your work.
- Why it hurts
- The context window fills with old, irrelevant material. The app starts dropping or summarising older parts, the model mixes details from different tasks, and important instructions from last week quietly stop being used.
- Do this instead
- One task, one conversation. When a conversation gets long or answers start drifting, ask for a summary of decisions and rules in force, then start a new conversation with that summary.
Try
Follow what goes into the context window in one go.
Step 1 of 4
Standing instructions
Rules and role that apply to the whole conversation.
After two hours in one conversation, the model starts using a table format you banned at the start and forgets a decision made in the middle. What is the best step?
Sorting card
Which adds to the tokens going in, and which uses the output limit?
The last two hours of conversation
A reasoning model's thinking steps
Standing instructions in project settings
A 20-page minutes attachment
A long table the model is writing
5 cards left
A prompt to carry context into a new conversation
Works with any AI assistant.
Your prompt
I need a summary of this conversation to paste into a new one. Write: the goal of the work, the format rules to follow, decisions already made, and open questions. At most ‹batas› words. Do not add anything that is not in the conversation.
Take home
Look after your context window
- One task, one conversation.
- I send long documents in parts, with a clear purpose for each.
- When answers start forgetting rules, I move to a new conversation with a summary.
- I treat a cut-off or empty answer as a sign of the output limit, not of the task being done.
What went wrong for us
Reasoning models get an output floor
- Context
- For short tasks it is natural to set a small output limit to save cost.
- What tends to go wrong
- A reasoning model spends tokens thinking before it answers, and those count against the same output limit. With a limit that is too small, answers can come back empty without any error.
- What it costs if ignored
- The feature gives no results, and no error explains why.
- Root cause
- The token budget runs out on thinking before the answer is written.
- How we do it
- Our code sets a floor on output tokens for reasoning models.
- The rule we keep
- An empty answer is treated as a failure to investigate.
The lessonUnderstand where tokens go before you cut them.
What it means for your workAn empty or cut-off answer is not a sign the task is done. The limit may have run out before the answer was written.