Tokenization Explained: How AI Breaks Text Into Tokens

When you type:

"How do I build an AI application?"

you might imagine that an AI model receives the sentence exactly as you see it.

It doesn't.

Before a modern language model can process your text, the text usually passes through a process called:

Tokenization

Tokenization converts text into smaller units called tokens.

Those tokens are then mapped to numerical IDs, which can be converted into vectors and processed by a neural network.

The simplified pipeline is:

Human Text
    ↓
Tokenizer
    ↓
Tokens
    ↓
Token IDs
    ↓
Embeddings
    ↓
Transformer
    ↓
Prediction

This seemingly simple step is fundamental to how modern Large Language Models process language.

And understanding tokenization will make many AI concepts much easier to understand.

Table of Contents

1. What Is Tokenization?
2. Why Do AI Models Need Tokens?
3. Characters vs Words vs Tokens
4. What Exactly Is a Token?
5. Simple Tokenization Example
6. Token IDs
7. Vocabulary
8. Subword Tokenization
9. Why Words Are Split
10. Byte Pair Encoding
11. WordPiece
12. SentencePiece
13. Special Tokens
14. Tokens and Embeddings
15. Tokens Inside Transformers
16. Context Windows
17. Why Token Count Matters
18. Tokenization and AI Costs
19. English vs Other Languages
20. Code Tokenization
21. Numbers and Tokenization
22. Emojis and Special Characters
23. Tokenization Example
24. What Happens When You Send a Prompt?
25. Tokenization in LLM Training
26. Tokenization During Inference
27. Common Tokenization Problems
28. Tokenizer vs Embedding Model
29. Frequently Asked Questions
30. Key Takeaways
31. Conclusion

What Is Tokenization?

Tokenization is the process of converting text into smaller units called tokens so that a language model can process the text numerically.

For example:

"Hello world"

might conceptually become:

["Hello", "world"]

But modern LLM tokenizers often use subword or byte-level units, so the actual result may look different.

For example:

"unbelievable"

could potentially be represented as multiple pieces rather than one whole word.

The exact tokenization depends on the tokenizer.

Why Do AI Models Need Tokens?

Computers work with numbers.

A neural network cannot directly perform matrix operations on:

"Hello"

Instead, the text needs to become numerical data.

The process looks approximately like:

"Hello"
   ↓
Token
   ↓
Token ID
   ↓
Embedding Vector
   ↓
Neural Network

This connects natural language to mathematical computation.

Characters vs Words vs Tokens

There are several possible ways to split text.

Consider:

"I love programming."

Character-based

I
(space)
l
o
v
e
...

This can create extremely long sequences.

Word-based

I
love
programming
.

This is simpler but has limitations.

Subword-based

A tokenizer can divide words into reusable pieces.

For example, conceptually:

program + ming

or:

program + ing

The exact split depends on the tokenizer.

Modern language models commonly use tokenization strategies that operate at the subword or byte level.

What Exactly Is a Token?

A token is a unit produced by a tokenizer.

A token can represent:

• A complete word
• Part of a word
• Punctuation
• Whitespace-related patterns
• Numbers or parts of numbers
• Symbols
• Special control markers
• Byte-level pieces

For example:

Hello, world!

could be represented by a tokenizer using pieces corresponding to:

Hello
,
 world
!

The exact tokens vary by tokenizer.

This is important:

There is no universal tokenization of text.

Different models can tokenize the same sentence differently.

A Simple Tokenization Example

Consider:

I love Python.

A conceptual tokenizer might produce:

["I", " love", " Python", "."]

Notice something interesting.

The space can sometimes be incorporated into a token representation.

That's because modern tokenizers aren't necessarily simply:

split(" ")

They can use learned vocabularies and byte-level rules.

Token IDs

Tokens themselves are still not what the neural network directly manipulates.

Each token corresponds to an integer ID in the tokenizer's vocabulary.

For example, conceptually:

"I"       → 42
" love"   → 381
" Python" → 9271
"."       → 13

These numbers are illustrative only.

A real tokenizer has its own vocabulary and IDs.

So:

Text
 ↓
Tokens
 ↓
Token IDs

Vocabulary

A tokenizer has a vocabulary containing the token units it knows how to represent.

Imagine a tiny vocabulary:

0 → <PAD>
1 → <UNK>
2 → I
3 → love
4 → Python
5 → .

Then:

"I love Python."

could become:

[2, 3, 4, 5]

Real LLM vocabularies are far larger and often contain many subword or byte-level tokens.

Why Not Just Use Every Word?

At first glance, a word-based vocabulary seems attractive.

Suppose we build a vocabulary containing:

cat
dog
computer
programming
developer
...

But language contains enormous numbers of possible words, names, spellings, and variations.

Consider:

program
programming
programmer
programmers
programmed
programmable

If every variation were treated as an entirely separate vocabulary item, the vocabulary could become unnecessarily large.

Subword tokenization provides a useful compromise.

Subword Tokenization

Subword tokenization breaks uncommon words into smaller reusable pieces.

Imagine:

"programming"

becoming:

"program" + "ming"

Then the same pieces can potentially help represent:

program
programming
programmer
programmed

The actual segmentation depends on the tokenizer.

This approach gives models a balance between:

Word-level representation

and:

Character-level representation

Why Words Are Split

Consider an unusual word:

"microservice"

A tokenizer may have a token for:

micro

and another for:

service

Instead of requiring:

"microservice"

to exist as a single vocabulary item.

This allows the tokenizer to handle many previously unseen or uncommon combinations.

The Unknown Word Problem

Older word-level tokenization approaches often faced the out-of-vocabulary problem.

Suppose the vocabulary contains:

cat
dog
computer

Then the model encounters:

QuantumCryptography

If the whole word isn't in the vocabulary, a simple word tokenizer may need an unknown token:

<UNK>

Subword approaches reduce this problem because unfamiliar words can often be decomposed into smaller known pieces.

Byte Pair Encoding

One important tokenization approach is:

Byte Pair Encoding — BPE

BPE originated as a compression technique and was later adapted for tokenization.

The basic intuition is:

Frequently occurring sequences can be merged into reusable units.

Imagine starting with small units:

a
b
c
d
...

and discovering that:

t + h

frequently occurs.

You can merge them:

th

Then perhaps:

th + e

becomes:

the

Through repeated merging, a vocabulary of useful pieces emerges.

Modern implementations can operate at different levels, including byte-level variants.

BPE Intuition

Imagine training data contains:

play
player
playing
played

The tokenizer may discover reusable pieces.

Conceptually:

play
play + er
play + ing
play + ed

Now the same pieces can represent many words.

Again, the exact behavior depends on the tokenizer and training process.

WordPiece

WordPiece is another subword tokenization approach.

It became particularly associated with Transformer-based NLP systems such as BERT.

Instead of simply applying the exact same merging procedure as BPE, WordPiece uses a vocabulary-learning strategy designed around useful subword units and likelihood objectives.

The practical result is similar in spirit:

Words
 ↓
Subword pieces
 ↓
Token IDs

SentencePiece

SentencePiece is a tokenizer framework that treats text as a sequence of symbols and can train tokenization models without requiring traditional whitespace-based word boundaries.

This is particularly useful for multilingual systems and languages where whitespace does not naturally separate words in the same way as English.

SentencePiece has been used with approaches such as:

• BPE
• Unigram language model tokenization

Special Tokens

Language models often use special tokens that aren't ordinary words.

Examples can include conceptual tokens such as:

<BOS>

Beginning of sequence.

<EOS>

End of sequence.

<PAD>

Padding.

<UNK>

Unknown token.

Modern chat models can also use special control structures to distinguish different message roles or formatting.

The exact special-token vocabulary depends on the model.

Tokens and Embeddings

Now connect this article with the previous one.

We learned:

Text
 ↓
Embedding

But there's an important intermediate step.

A simplified LLM pipeline looks like:

Text
 ↓
Tokenizer
 ↓
Token IDs
 ↓
Embedding Lookup
 ↓
Vectors
 ↓
Transformer

For example:

"Hello AI"
      ↓
["Hello", " AI"]
      ↓
[15496, 9552]
      ↓
Embedding vectors
      ↓
Transformer

The numbers here are only illustrative.

What Is an Embedding Lookup?

Suppose the model has a token vocabulary of 50,000 tokens.

Each token has an associated vector.

Conceptually:

Token ID 0  → Vector 0
Token ID 1  → Vector 1
Token ID 2  → Vector 2
...
Token ID 49,999 → Vector 49,999

When a token ID arrives, the model retrieves its corresponding vector.

This is essentially an embedding-table lookup.

Tokens Inside Transformers

Once tokens have become vectors, the Transformer can process them.

A simplified view:

Text
 ↓
Tokenizer
 ↓
Token IDs
 ↓
Embeddings
 ↓
Positional / position-related information
 ↓
Transformer Layers
 ↓
Output Logits
 ↓
Next Token

This is the bridge from raw text to neural computation.

Context Windows

Every language model has a limit on how many tokens it can process within a particular context.

This is commonly called the:

Context Window

For example, a hypothetical model might support:

128,000 tokens

Another model might support a different context length.

The important point is:

Context limits are generally measured in tokens, not words.

Why Token Count Matters

Consider:

"I am learning machine learning."

That's only a few words.

But tokenization may produce a different number of tokens depending on the tokenizer.

Therefore:

Words ≠ Tokens

This matters when working with:

• LLM APIs
• Long documents
• Prompt engineering
• RAG
• Context windows
• AI pricing
• Model performance

Tokens and AI Costs

Many AI services price usage based partly on tokens.

A simplified pricing model might look like:

Input Tokens × Input Price
+
Output Tokens × Output Price

Therefore, token efficiency can matter when building production AI applications.

For example:

Huge Prompt
 ↓
100,000 tokens
 ↓
Higher cost

versus:

Optimized Prompt
 ↓
20,000 tokens
 ↓
Lower cost

Exact pricing varies by provider and model, so token economics should always be checked against the specific service being used.

Tokenization Is Not the Same as Word Count

Suppose your prompt contains:

"Build a scalable microservices architecture."

You might count:

6 words

But the tokenizer could produce a different number of tokens.

This is why developers working with LLMs should think in terms of tokens, not just words or characters.

English vs Other Languages

Tokenization efficiency can vary dramatically between languages.

For example, a tokenizer trained heavily on English data may represent English text relatively efficiently.

Other languages may require more tokens for the same amount of semantic content.

This can affect:

• Context usage
• Cost
• Latency
• Maximum usable text
• Model behavior

Multilingual tokenizer design therefore matters enormously for global AI systems.

Code Tokenization

Tokenizers can process code too.

Consider:

def hello():
    print("Hello World")

The tokenizer doesn't necessarily see:

def
hello
(
)
:

as simple words.

It may divide the code into a mixture of programming-language pieces, symbols, whitespace patterns, and other tokens.

This allows LLMs to process:

• Python
• JavaScript
• Java
• Rust
• C++
• SQL
• HTML
• CSS
• Shell commands

and many other forms of code.

Why Code Tokenization Matters

Imagine:

getUserById()

A tokenizer might represent parts of this identifier using reusable pieces.

Likewise:

HTTP
GET
/api/users

contains punctuation and symbols that need to be represented.

Good tokenization helps language models handle programming syntax efficiently.

Numbers and Tokenization

Numbers can be interesting.

Consider:

123456789

A tokenizer might represent the number as one token or several tokens depending on its vocabulary and tokenization scheme.

Similarly:

3.1415926535

may be divided into multiple pieces.

This is one reason LLMs can sometimes struggle with precise arithmetic: tokenization and neural representation are not equivalent to having a dedicated mathematical calculator.

Emojis and Special Characters

Consider:

🔥 🚀 ❤️

These aren't necessarily one token each.

Depending on the tokenizer, Unicode characters and emoji sequences may be represented by multiple token pieces.

This can affect token counts.

What Happens When You Send a Prompt?

Let's follow a real conceptual pipeline.

You type:

Explain recursion in Python.

Step 1 — Text

"Explain recursion in Python."

Step 2 — Tokenization

The tokenizer converts it into token pieces.

Conceptually:

["Explain", " recursion", " in", " Python", "."]

The exact output depends on the tokenizer.

Step 3 — Token IDs

[ID₁, ID₂, ID₃, ID₄, ID₅]

Step 4 — Embedding

Each token ID maps to a vector.

ID
 ↓
Embedding Vector

Step 5 — Transformer

The vectors pass through Transformer layers.

Vectors
 ↓
Attention
 ↓
Feed-Forward Networks
 ↓
More Transformer Layers

Step 6 — Output Logits

The model produces scores for possible next tokens.

"Recursion"
0.42

"is"
0.18

"can"
0.11

...

The model then selects or samples a next token according to its decoding strategy.

Step 7 — Repeat

The generated token becomes part of the growing sequence.

Prompt
 ↓
Next Token
 ↓
Next Token
 ↓
Next Token
 ↓
...

Eventually, the generated tokens are decoded back into text.

Tokenization During Training

During training, huge collections of text are processed.

The pipeline looks like:

Training Corpus
      ↓
Tokenizer
      ↓
Token IDs
      ↓
Model
      ↓
Prediction
      ↓
Loss
      ↓
Backpropagation
      ↓
Parameter Updates

The tokenizer itself is usually established as part of the model's design and training pipeline rather than being dynamically relearned from scratch during every inference request.

Tokenization During Inference

When you use a trained LLM:

Your Prompt
    ↓
Same tokenizer family / configured tokenizer
    ↓
Token IDs
    ↓
Model
    ↓
Generated Tokens
    ↓
Decoded Text

The tokenizer is an essential part of the model ecosystem.

You generally shouldn't assume that a tokenizer from one model will produce identical IDs or token boundaries for another model.

Why You Should Never Assume Token IDs Are Universal

This is important for developers.

You might see:

"hello" → 15339

and assume:

hello = 15339

forever.

That's incorrect.

Token IDs depend on the tokenizer's vocabulary.

Another tokenizer could assign a completely different ID.

So:

Tokenizer A:
hello → 15339

Tokenizer B:
hello → 9821

could both be perfectly valid.

Tokenizer vs Embedding Model

These are two different components.

Tokenizer

Converts:

Text
 ↓
Tokens
 ↓
Token IDs

Embedding Model

Converts information into:

Vector Representation

In a language model:

Text
 ↓
Tokenizer
 ↓
Token IDs
 ↓
Embedding Layer
 ↓
Transformer

In a dedicated semantic-search system:

Text
 ↓
Embedding Model
 ↓
Embedding Vector

The second setup doesn't necessarily expose the same tokenization/embedding process as an LLM's internal architecture.

Why Tokenization Matters for RAG

Remember our previous article?

RAG systems often perform:

Document
 ↓
Chunking
 ↓
Embedding
 ↓
Vector Database

Tokenization matters because the LLM ultimately has a token-based context limit.

Suppose you retrieve:

100 huge chunks

You may exceed the model's context capacity.

A better pipeline considers:

Document
 ↓
Chunk
 ↓
Token Count
 ↓
Embedding
 ↓
Retrieval
 ↓
Context Selection
 ↓
LLM

Token awareness is therefore an important part of production RAG design.

Common Tokenization Problems

1. Assuming One Word = One Token

False.

A word can become:

1 token

or:

2 tokens

or:

many tokens

depending on the tokenizer.

2. Assuming One Character = One Token

Also false.

Tokenizers can represent multiple characters as one token or split characters into multiple pieces.

3. Assuming Tokens Have Meaning Like Words

A token can be:

part of a word
punctuation
whitespace-related text
symbol
byte-level unit

Tokens aren't necessarily semantic concepts.

4. Ignoring Token Limits

If you're building an LLM application, you need to understand the model's context capacity.

5. Using the Wrong Tokenizer

Token IDs and token boundaries are tokenizer-specific.

Use the tokenizer associated with the model or service you're working with.

Tokenization and Prompt Engineering

Tokenization also matters when writing prompts.

Suppose you're building an AI application that sends:

50,000 tokens

of instructions and documents for every request.

That's potentially inefficient.

You may improve the architecture by:

Large Context
     ↓
Retrieval
     ↓
Relevant Information
     ↓
Smaller Prompt
     ↓
LLM

This is one reason RAG can be more practical than simply putting an entire knowledge base into every prompt.

Tokenization and Long Context

Modern models support increasingly large context windows, but larger context doesn't eliminate the need for good information management.

If your application has:

1 million tokens of data

you may still not want to send all of it for every question.

Instead:

1M tokens
   ↓
Search / Retrieval
   ↓
Relevant 10K tokens
   ↓
LLM

This reduces unnecessary computation and can improve relevance.

A Bigger Picture

At this point, we can connect the previous three articles.

Article 17

We learned:

Neural Networks

Article 18

We learned:

Embeddings

Article 19

We learned:

Tokenization

Together:

Human Language
      ↓
Tokenization
      ↓
Token IDs
      ↓
Embeddings
      ↓
Neural Network
      ↓
Transformer
      ↓
Prediction
      ↓
Generated Token
      ↓
Text

This is the foundation for understanding how modern LLMs process language.

Frequently Asked Questions

What is a token in AI?

A token is a unit produced by a tokenizer from text. It can represent a word, part of a word, punctuation, whitespace-related content, or other symbols.

Is one word always one token?

No. A word can be represented by one or multiple tokens depending on the tokenizer.

What is tokenization?

Tokenization converts text into token units that can subsequently be mapped to numerical IDs.

What is a token ID?

A token ID is an integer identifying a particular token in a tokenizer's vocabulary.

What is BPE?

Byte Pair Encoding is a tokenization approach based on creating reusable units through frequent pair merging; modern implementations may use byte-level variants.

What is WordPiece?

WordPiece is a subword tokenization method strongly associated with models such as BERT.

What is SentencePiece?

SentencePiece is a tokenization framework that can learn subword segmentation directly from text without requiring conventional whitespace tokenization.

Why do LLMs use tokens instead of words?

Tokens provide a practical balance between vocabulary size, sequence length, and the ability to represent unfamiliar words and symbols.

Do tokens have meaning?

Not necessarily. A token can be a semantic word, a subword, punctuation, whitespace-related pattern, or other unit.

Why are tokens important for LLM APIs?

Token counts can affect context usage, latency, and—in many services—cost.

Are token IDs universal?

No. Token IDs are specific to a tokenizer vocabulary.

What comes after tokenization?

Typically:

Token IDs
 ↓
Embeddings
 ↓
Transformer

for a language model.

Key Takeaways

• Tokenization converts text into tokens.
• Tokens are not necessarily complete words.
• A token can be a word, subword, punctuation, symbol, or other unit.
• Token IDs represent tokens numerically.
• Tokenizers have vocabularies.
• Modern LLMs commonly use subword or byte-level tokenization approaches.
• BPE, WordPiece, and SentencePiece are important tokenization technologies.
• Tokenization varies between models.
• One word can correspond to multiple tokens.
• Token counts affect context usage.
• Token counts can affect AI costs.
• Token IDs are not universal across tokenizers.
• Tokens are transformed into vectors through embedding mechanisms before deeper neural processing.
• Tokenization is an essential bridge between human language and neural-network computation.

Conclusion

When you type:

"Build an AI application."

the model doesn't simply receive those words as humans perceive them.

Instead, the journey looks roughly like:

"Build an AI application."
             ↓
         Tokenizer
             ↓
     Token Pieces
             ↓
         Token IDs
             ↓
        Embeddings
             ↓
       Transformer
             ↓
      Neural Network
             ↓
     Next-Token Scores
             ↓
       Generated Token
             ↓
            Text

And that explains something fundamental about Large Language Models:

LLMs operate on numerical representations of token sequences, not directly on human-readable words.

Once you understand this pipeline, the architecture of modern AI starts becoming much less mysterious.

But we've reached an even more interesting question.

We know that:

Text
 ↓
Tokens
 ↓
Vectors

But how does the model understand which tokens are related to which other tokens?

How does it know that in:

"The developer opened the laptop because it was slow."

the word "it" is related to something earlier in the sentence?

How can a model connect information separated by dozens, hundreds, or thousands of tokens?

The answer is one of the most revolutionary ideas in modern AI:

Attention.