Kamus AI
Token
A piece of text a model reads and writes, usually part of a word.
Models do not read letters or whole words, they read tokens. One Indonesian word is often two or three tokens. Input limits, output limits and usage costs are all counted in tokens. Indonesian text usually takes more tokens than English text with the same meaning.
Also called: Token
Related terms
Appears in 13 lessons
- AI as a highly trained text predictor
- Tokens and the context window
- Why the answers keep changing
- Why models make things up
- A first agent that only reads
- Reading an agent's trace
- Context is not memory
- Skills as procedures
- Stop conditions
- The cost of one agent task
- Per-user credit metering
- Step traces and silent failures
- Project: a metered release