Tokenization defines the model’s alphabet
How to study this lesson
Read one section, trace the worked example, and try the knowledge check. Bookmark saves a shortcut; notes appear in My notebook. Mark lesson complete is your own assessment and does not mark its lab passed.
By the end, you can
- Trace a ranked pair-merge tokenizer.
- Explain why token counts depend on the tokenizer and text.
- Connect next-token cross-entropy to generation.
Text must become discrete model inputs
A language model operates on token IDs, not directly on the visual words a reader sees. A tokenizer maps text into a sequence from a finite vocabulary. Depending on its design, units can include characters, byte sequences, subwords, punctuation, spaces, and special control tokens. The embedding lookup then maps each ID to a learned vector. Tokenization therefore sits between raw text and all later model computation.
The same sentence can have different token counts under different tokenizers. Code, rare names, non-English text, and unusual whitespace can fragment differently. A fixed characters-per-token estimate is only a rough planning approximation. For an actual request budget, use the tokenizer or counting interface associated with the selected model and account for protocol formatting as well as the visible user text.
Pair merging builds reusable pieces
Byte-pair-encoding-style tokenizers learn or apply ordered merge rules. Start from smaller symbols, find an eligible adjacent pair, merge according to the learned ranking, and repeat while rules apply. A production implementation includes many details beyond this outline, but the key idea is that frequent pieces can become a single vocabulary item without needing a separate whole-word entry for every possible word.
Our toy word is banana. Starting with b, a, n, a, n, a, the pair a+n can merge twice, leaving b, an, an, a. A later b+an rule creates ban, and an+a creates ana. The result is ban, ana. This example uses hand-authored character rules; it does not reproduce a provider tokenizer or claim that banana has two tokens in a particular model.
Normalization and boundaries have consequences
Before or during segmentation, a tokenizer may use normalization, byte conversion, or special handling of whitespace and control markers. These choices influence reversibility and what strings the model can distinguish. Two visually similar Unicode strings can contain different code points; a normalization step might merge that distinction or preserve it. Never assume a displayed character is one byte or one token.
Special tokens are also part of a protocol. A user typing text that resembles a special marker should not automatically gain the authority of a system message. The application and tokenizer interface must distinguish ordinary text from structured control information. This is another reason to use the provider's supported message interface instead of concatenating a homemade transcript with guessed delimiters.
Training predicts the next ID
For a sequence of token IDs t1 through tT, an autoregressive model learns conditional distributions over the next token given earlier tokens. A training objective sums or averages negative log probabilities assigned to the observed next IDs. If the correct next token receives probability 0.25, its contribution to negative log likelihood is about 1.386 nats. Assigning it probability 0.5 lowers that contribution to about 0.693.
The objective provides supervision at many positions in a text, but it is still a statistical prediction objective. It does not directly prove factual correctness, task success, or permission to act. Later training and system design can change behavior, while evaluation must still measure the outcomes your application needs. A fluent continuation can remain unsupported by the available evidence.
Generation exposes representation tradeoffs
At generation time, a model produces logits over the vocabulary, converts them into a distribution under the chosen decoding procedure, selects a token, and appends it to the context. The process repeats until a stopping condition is reached. Different tokenizations change sequence length, vocabulary size, and the granularity of these choices, influencing computation and how errors appear in text.
The code traces a small ranked merge process and prints the UTF-8 byte length of a short non-ASCII string. It is deliberately inspectable and cannot estimate a production model's billable tokens. Use it to understand why string length, byte length, and token count are separate concepts. In the exercise, you will change rule ordering to see that a tokenizer is an algorithm with a versioned vocabulary, not a universal property of a word.
Attention, under your control
One query [q, 1], four fixed keys, and a two-dimensional head. Change the query or temperature and inspect where attention goes.
scoreᵢ = (query · keyᵢ) / (√2 × temperature). Then apply softmax. Lower temperature concentrates weight; it does not add evidence. These four labels are synthetic teaching tokens, not a trained model.
Work through the code
The deterministic toy applies the lowest-ranked available pair and breaks equal-rank ties by position. It uses character symbols and three hand-written rules. The UTF-8 example separates code-point count from byte length. Neither count is a substitute for a selected model’s tokenizer.
def tokenize_trace(word, ranks):
symbols = list(word)
trace = [symbols.copy()]
while len(symbols) > 1:
candidates = [(ranks[pair], i) for i in range(len(symbols) - 1)
if (pair := (symbols[i], symbols[i + 1])) in ranks]
if not candidates:
break
_, index = min(candidates)
symbols[index:index + 2] = [symbols[index] + symbols[index + 1]]
trace.append(symbols.copy())
return trace
ranks = {("a", "n"): 0, ("b", "an"): 1, ("an", "a"): 2}
for stage in tokenize_trace("banana", ranks):
print(" | ".join(stage))
text = "café"
print("characters:", len(text))
print("utf8 bytes:", len(text.encode("utf-8")))
b | a | n | a | n | a b | an | a | n | a b | an | an | a ban | an | a ban | ana characters: 4 utf8 bytes: 5
Run Python snippets locally with the prerequisites named above. The module coding lab runs directly in your browser.
Pause and reason
For the word abc, compare a tokenizer whose first merge is a+b with one whose first merge is b+c. No further merge rules exist. What are the token sequences and why must tokenizer identity be stored with token IDs?
Check your understanding
Which quantity can reliably determine a request’s model token count?
Your notes
Explain the mechanism in your own words. Add a failure you want to test.