Tokenization

why not just use words?

A word-level vocabulary would need an entry for every word in every language the model supports, still couldn't handle typos or made-up words, and would waste capacity on rare words that appear once in the training data. A character-level vocabulary handles anything but makes sequences much longer, which is expensive since attention cost grows with sequence length. Byte-pair encoding (BPE) is the practical middle ground almost every current LLM uses.

how BPE builds its vocabulary

# simplified BPE training loop
vocab = set(all_bytes)                 # start from raw bytes/characters
corpus = tokenize_to_vocab_units(text)

for _ in range(num_merges):
    pair = most_frequent_adjacent_pair(corpus)
    vocab.add(merge(pair))             # e.g. ("t", "h") -> "th"
    corpus = apply_merge(corpus, pair)

# repeat until vocab reaches the target size (e.g. 100k-200k tokens)

The result: common whole words ("the", "and") usually end up as single tokens because they were frequent enough to get fully merged, while rare or made-up words get split into multiple sub-word pieces. This is a byproduct of the training corpus's frequency statistics, not a hand-designed rule.

checking token counts yourself

pip install tiktoken

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
tokens = enc.encode("Tokenization isn't the same as splitting on spaces.")
print(len(tokens))   # 10 -- more tokens than words, because of punctuation
                      # and the split on "isn't"
print(enc.decode(tokens[:3]))

Rule of thumb for English prose: roughly 4 characters or ~0.75 words per token. Code, non-English text, and text with unusual punctuation typically tokenize less efficiently than that.

practical consequences

Symptom	Root cause
API cost is higher than expected for "short" prompts	Non-English text, code, and rare vocabulary tokenize less efficiently than plain English
Model struggles to count letters in a word, or reverse a string	It's operating on tokens, not characters — "strawberry" may be 2-3 tokens, none of which expose individual letters to the model directly
Context window fills up faster than expected	Both prompt and completion tokens count against the same budget — see Context Windows

where to go from here

The Transformer Architecture — what happens to tokens once they're embedded.
Context Windows — why token counts are the budget every prompt has to fit inside.
Full-Text Search — a different, older kind of tokenization — worth contrasting.