What Is Tokenization in LLMs? Tokens Explained

Tokenization is the step that turns raw text into the numbered units a language model actually processes, called tokens. A model never sees letters or words the way you do. Before any prediction happens, your input is chopped into tokens, each token is mapped to an integer ID, and the model works entirely with those IDs. Tokenization is the boundary between human-readable text and the numeric world a model lives in — and understanding it explains a surprising number of otherwise baffling model behaviors.

It also touches your wallet and your limits directly: APIs bill per token, and a model’s context window is measured in tokens, not words or characters. Knowing roughly how text becomes tokens is the difference between guessing at costs and estimating them.

A token is not a word

The most common misconception is that a token equals a word. It doesn’t. Modern tokenizers use subword units — chunks that sit between a single character and a whole word. A common short word like “the” is usually one token. A longer or rarer word gets split into pieces: “tokenization” might become token + ization, and an unusual name might break into several fragments.

A useful rule of thumb for English is that one token is roughly four characters, or about three-quarters of a word. So 1,000 tokens is somewhere near 750 words. This is only an approximation — it shifts with language, punctuation, and formatting — but it is close enough to estimate whether a document will fit in a context window. For a precise count, our LLM token cost calculator tokenizes real text and shows the totals.

Spaces and punctuation matter too. In most tokenizers a leading space is part of the token, so ” cat” (with a space) and “cat” (without) are different tokens. This is why concatenating strings carelessly can change token counts in ways that look arbitrary until you know the rule.

Why subword tokenization won

The design is a deliberate compromise between two bad extremes.

• Character-level tokenization — one token per character — has a tiny vocabulary and can represent any string, but sequences become enormously long, which is slow and wastes the context window.
• Word-level tokenization — one token per word — keeps sequences short but needs a gigantic vocabulary and still fails on any word it has never seen, including typos, new slang, and made-up product names.

Subword tokenization gets the best of both. Common words stay whole and cheap, while rare or novel words are decomposed into familiar pieces the model can still work with. Nothing is truly “out of vocabulary,” because in the worst case a string falls back to individual characters. The most widely used algorithm for building these vocabularies is byte-pair encoding (BPE), which starts from bytes and repeatedly merges the most frequent adjacent pair into a new token until it reaches a target vocabulary size — often on the order of tens of thousands of tokens.

What happens after the split

Tokenization produces a list of integer IDs, but the model can’t reason about bare integers either. Each token ID is looked up in an embedding table and turned into a vector — a list of numbers that encodes meaning in a high-dimensional space. Those vectors are what actually flow through the transformer layers. If you want the deeper story on that representation, our explainer on vector embeddings picks up exactly where tokenization ends: tokens become IDs, IDs become embeddings, embeddings get processed.

On the way out, the process runs in reverse. The model predicts a probability distribution over the next token, one token is chosen, its ID is appended, and the whole sequence is fed back in to predict the token after that. This token-at-a-time loop is why large language models generate text incrementally, and why streaming responses appear to type themselves.

The quirks tokenization explains

Once you know how tokenization works, several classic model failures stop being mysterious.

• Counting letters. Ask a model how many times the letter “r” appears in a word and it often gets it wrong. The word arrived as one or two tokens, not as individual letters, so the model never saw the characters it is being asked to count.
• Arithmetic oddities. Numbers tokenize inconsistently — “1234” might be one token while “12345” splits differently — which makes digit-by-digit math harder than it should be.
• Non-English inefficiency. Tokenizers trained mostly on English spend more tokens per word on other languages and scripts, so the same sentence can cost noticeably more tokens — and more money — in one language than another.
• Reversing strings. Flipping the characters of a word is hard for the same reason as counting: the model operates on tokens, not letters.

None of these are reasoning failures exactly; they are artifacts of the input format. They are worth remembering when you design prompts around tasks that depend on individual characters.

Why the count matters in practice

Because tokens are the unit of both billing and context, the token count is the number to watch when you build with models.

• Cost. Input and output tokens are priced separately, and long system prompts, retrieved documents, and conversation history all add up. Techniques like prompt caching exist specifically to avoid re-paying for the same tokens on every request.
• Context limits. Everything the model considers — instructions, retrieved context, prior turns, and its own output — must fit within the token budget of the context window. Overflow means truncation or dropped history.
• Latency. More tokens in and out means more compute, which means slower responses. Trimming a verbose prompt often speeds things up as much as it saves money.

These are the practical reasons that “just count the words” isn’t good enough. Tokens, not words, are what the model and the invoice actually measure.

The takeaway

Tokenization splits text into subword tokens, maps each to an integer ID, and hands those IDs to the model, which turns them into embeddings and processes them one token at a time. A token is roughly four characters or three-quarters of a word in English, and it is the true unit of both context windows and API pricing. Knowing this explains why models miscount letters, why non-English text costs more, and why trimming a prompt saves both money and time — and it makes estimating the size and cost of any AI workload far less of a guess.