How an LLM reads your sentence

A language model has never seen a letter. Before any attention, any transformer block, any prediction, your sentence is chopped into pieces called tokens and each piece is replaced by a number. Almost every strange behaviour people notice in LLMs — bad spelling, bad arithmetic, higher costs for Hindi than English — traces back to this one step.

The problem tokenization solves

A neural network takes numbers. Text is not numbers. So we need a rule for turning "the cat sat" into something like [1820, 9059, 3332]. There are three obvious approaches and two of them fail.

Option 1: one number per word

Build a dictionary of every word in English, assign each an ID. It seems natural, and it breaks immediately:

• English has hundreds of thousands of words, and the dictionary needs an entry for every one — a vast embedding table where most rows are barely ever trained.
• New words appear constantly. "Rizz", "COVID-19", your surname, a typo. Every one becomes <UNK> and the meaning is simply lost.
• The model cannot see that "walk", "walked", "walking" and "walks" are related. To it they are four unrelated IDs, as different from each other as "walk" and "tractor".

Option 2: one number per character

Only about a hundred symbols needed, and nothing is ever unknown. But now "internationalisation" is 21 separate steps for the model to process, sequences become enormously long, and since attention cost grows with the square of sequence length, this is ruinously expensive. Worse, a single character carries almost no meaning on its own.

Option 3: subwords — the one that works

Keep common words whole, and split rare words into meaningful fragments. "The" stays one token. "Unhappiness" becomes un + happi + ness. A misspelling or a name the model has never seen still breaks into pieces it recognises, so nothing is ever truly unknown.

This gets you a vocabulary of roughly 30,000–200,000 tokens, sequences that stay short, and graceful handling of anything unfamiliar. Every major model — GPT, Claude, Gemini, Llama — uses a variant of this idea.

Lab: tokenize anything

The tokenizer below is a genuine byte-pair-encoding implementation running in your browser. It uses a small merge table trained on English text, so it is far cruder than a production tokenizer — but the mechanism is exactly the same. Type anything and watch it split.

How byte-pair encoding is trained

BPE is startlingly simple. You start with individual characters and repeatedly merge the most frequent adjacent pair. Work through this tiny example by hand.

Corpus: low low low lower lowest. Begin with characters:

Round 1. The most frequent adjacent pair is l o, appearing 5 times. Merge it into a single unit lo:

Round 2. Now lo w appears 5 times. Merge:

Round 3. Count again. low e now appears twice — in "lower" and "lowest" — while every other pair appears only once. So it wins:

Round 4. Everything remaining appears once, so the tie is broken by order of first appearance. lowe r merges:

Stop after however many merges you budgeted for — production tokenizers run 50,000 or more. The learned merge list is the tokenizer. To tokenize new text you replay those merges in the order they were learned. Notice that the algorithm discovered "low" and the suffix "er" without anyone teaching it English morphology. It is pure frequency counting.

Consequences you have probably noticed

Why models miscount letters

Ask a model how many r's are in "strawberry" and it often gets it wrong. The reason is structural: to the model, that word may be two or three tokens such as str + aw + berry. It never sees the individual letters. Asking it to count characters is like asking someone to count the brush strokes in a printed word — the information is not in the representation.

Why arithmetic is fragile

Numbers tokenize inconsistently. "1234" might be one token while "1235" splits into "123" + "5". Digits that should line up for column addition end up in different positions, so the model has to learn arithmetic in a representation actively hostile to it. Newer tokenizers force digits to split individually, which measurably improves arithmetic.

Why Hindi costs more than English

This one matters for anyone building for India. Tokenizers are trained on corpora that are overwhelmingly English, so English gets efficient merges and other scripts do not. The same sentence in Devanagari, Tamil or Bengali can take two to five times as many tokens as its English translation. Three real consequences:

• You pay more per sentence, because APIs bill per token.
• You fit less into a fixed context window.
• Quality is lower, because meaning is spread across more, smaller fragments.

This is one of the strongest arguments for tokenizers trained specifically on Indian-language corpora, and it is a genuinely open area of work.

The code

A working BPE trainer in about thirty lines. Run it in the playground and watch it discover subwords.

from collections import Counter

corpus = "low low low lower lowest newer newer wider"

# start with characters, end-of-word marked with 
words = [tuple(w) + ("</w>",) for w in corpus.split()]

def pair_counts(words):
    c = Counter()
    for w in words:
        for i in range(len(w) - 1):
            c[(w[i], w[i+1])] += 1
    return c

def merge(words, pair):
    out = []
    for w in words:
        new, i = [], 0
        while i < len(w):
            if i < len(w)-1 and (w[i], w[i+1]) == pair:
                new.append(w[i] + w[i+1]); i += 2
            else:
                new.append(w[i]); i += 1
        out.append(tuple(new))
    return out

for step in range(10):
    counts = pair_counts(words)
    if not counts: break
    best = counts.most_common(1)[0][0]
    words = merge(words, best)
    print(f"merge {step+1}: {best[0]!r} + {best[1]!r}")

print("final:", sorted(set(words)))

Check yourself

1. Why does a character-level model need far more compute per sentence than a subword model, even though its vocabulary is tiny? (Sequences are much longer, and attention cost scales with the square of sequence length.)
2. You are building a chatbot for Marathi speakers with a 4,000-token context limit. Roughly how much Marathi text fits compared with English? (Perhaps a quarter to a half as much — measure it, do not guess.)
3. Why would forcing every digit to be its own token improve a model's arithmetic? (Digit positions become consistent, so place value is learnable.)
4. Use the lab: find an English word under twelve letters that splits into four or more tokens. What do those words have in common?

Where this goes next

Your sentence is now a list of integers. The next lesson turns each of those integers into a vector — an embedding — and then attention lets every one of those vectors look at every other one. That is where the model stops processing text and starts processing meaning.