How LLMs Are Trained: Pretraining, Fine-Tuning, RLHF & Alignment Explained

Introduction

How does an AI system go from this:

Raw Text
Books
Web Pages
Code
Documents
Articles

to this:

"Explain recursion in simple terms."

and produce a useful answer?

The answer isn't a single training step.

Modern Large Language Models go through a pipeline of multiple stages.

A simplified version looks like:

Raw Data
   ↓
Data Collection
   ↓
Data Cleaning
   ↓
Tokenization
   ↓
Training Dataset
   ↓
Pretraining
   ↓
Base Model
   ↓
Instruction Tuning
   ↓
Preference Optimization
   ↓
Safety & Evaluation
   ↓
Deployment
   ↓
AI Assistant

The remarkable part is that the original training objective can be surprisingly simple:

Predict the next token.

Yet when this objective is combined with enormous datasets, powerful Transformer architectures, large-scale computation, and extensive post-training, the resulting models can perform tasks such as:

• writing
• summarization
• translation
• programming
• question answering
• reasoning
• classification
• data analysis
• creative generation

So how does this actually happen?

Let's go through the entire lifecycle.

Table of Contents

1. What Is an LLM?
2. The LLM Training Pipeline
3. Step 1: Collecting Training Data
4. Step 2: Cleaning the Data
5. Step 3: Deduplication
6. Step 4: Filtering and Quality Control
7. Step 5: Tokenization
8. Step 6: Building Training Sequences
9. Step 7: The Transformer
10. Step 8: Next-Token Prediction
11. Step 9: Calculating Loss
12. Step 10: Backpropagation
13. Step 11: Updating Parameters
14. What Happens During Pretraining?
15. Why Scale Matters
16. What Does the Model Actually Learn?
17. The Emergence of Capabilities
18. The Base Model
19. Why a Base Model Isn't a Chatbot
20. Supervised Fine-Tuning
21. Instruction Tuning
22. Preference Data
23. RLHF
24. Reward Models
25. Preference Optimization
26. DPO and Related Methods
27. AI Alignment
28. Safety Training
29. Evaluation
30. Model Checkpoints
31. Deployment
32. Inference
33. Why LLMs Sometimes Hallucinate
34. Common Misconceptions
35. Frequently Asked Questions
36. Key Takeaways
37. Conclusion

1. What Is an LLM?

LLM stands for:

Large Language Model

At its core, an LLM is a neural network trained to model sequences of tokens.

Modern LLMs are typically based on the Transformer architecture.

The simplified structure is:

Text
 ↓
Tokenizer
 ↓
Tokens
 ↓
Embeddings
 ↓
Transformer
 ↓
Logits
 ↓
Probabilities
 ↓
Next Token

The model repeatedly predicts what token should come next.

That sounds simple.

But the scale changes everything.

2. The LLM Training Pipeline

A modern LLM development process can be visualized as:

                 ┌──────────────────┐
                 │   Raw Data       │
                 └────────┬─────────┘
                          ↓
                 ┌──────────────────┐
                 │ Cleaning &       │
                 │ Filtering        │
                 └────────┬─────────┘
                          ↓
                 ┌──────────────────┐
                 │ Tokenization     │
                 └────────┬─────────┘
                          ↓
                 ┌──────────────────┐
                 │ Training Data    │
                 └────────┬─────────┘
                          ↓
                 ┌──────────────────┐
                 │ Pretraining      │
                 └────────┬─────────┘
                          ↓
                 ┌──────────────────┐
                 │ Base Model       │
                 └────────┬─────────┘
                          ↓
                 ┌──────────────────┐
                 │ Fine-Tuning      │
                 └────────┬─────────┘
                          ↓
                 ┌──────────────────┐
                 │ Preference /     │
                 │ Alignment        │
                 └────────┬─────────┘
                          ↓
                 ┌──────────────────┐
                 │ Evaluation       │
                 └────────┬─────────┘
                          ↓
                 ┌──────────────────┐
                 │ Deployment       │
                 └──────────────────┘

Each stage solves a different problem.

3. Step 1: Collecting Training Data

Before a model can learn, we need data.

Potential training sources can include:

• books
• websites
• documentation
• public-domain material
• code
• educational content
• articles
• conversations
• mathematical material
• scientific papers
• other licensed or appropriately sourced datasets

The exact composition of a commercial model's training corpus is usually proprietary.

The important idea is that data quality matters enormously.

A model cannot learn useful patterns from information it never receives.

4. Step 2: Cleaning the Data

Raw internet-scale data is messy.

Imagine collecting billions of documents.

You might encounter:

Duplicate pages
Spam
Broken HTML
Navigation menus
Advertisements
Malformed text
Low-quality content
Repeated boilerplate
Machine-generated content
Irrelevant material

Feeding everything directly into training would be a terrible idea.

So data pipelines perform extensive preprocessing.

Conceptually:

Raw Data
   ↓
Remove corruption
   ↓
Remove unwanted content
   ↓
Normalize formats
   ↓
Quality filtering
   ↓
Training Corpus

The quality of this pipeline can significantly affect the resulting model.

5. Step 3: Deduplication

Suppose the same article appears on 500 websites.

If the model sees essentially the same content repeatedly, that can distort the training distribution.

Therefore, large-scale training pipelines often perform deduplication.

There can be several levels:

Exact deduplication

Remove identical documents.

Near-duplicate detection

Detect documents that are mostly the same.

Template removal

Remove repeated boilerplate such as:

Header
Navigation
Footer
Cookie notices

The goal is not simply:

More data.

It is:

Better useful data.

6. Step 4: Filtering and Quality Control

Not every piece of text is equally useful.

A training pipeline can apply different filters to improve the corpus.

For example:

Language Detection
      ↓
Quality Filtering
      ↓
Safety Filtering
      ↓
Spam Detection
      ↓
Code Filtering
      ↓
Document Classification

Different datasets may also be weighted differently.

For example, a training mixture might contain multiple categories:

General Web
   +
Books
   +
Code
   +
Mathematics
   +
Science
   +
Reference Material

The exact mixture is an important design decision.

7. Step 5: Tokenization

Computers don't directly process sentences as words.

The text is converted into tokens.

Consider:

"Artificial intelligence is powerful."

A tokenizer might represent it conceptually as:

["Artificial", " intelligence", " is", " powerful", "."]

But tokens aren't necessarily whole words.

A word can be divided into multiple pieces.

For example:

unbelievable

could conceptually become:

un
believ
able

The exact result depends on the tokenizer.

These tokens are then mapped to integer IDs:

Token
  ↓
Token ID

For example:

Artificial → 18291
intelligence → 7312
is → 318
powerful → 9214

The numbers above are only illustrative.

8. Step 6: Building Training Sequences

After tokenization, the data becomes sequences of token IDs.

For example:

[125, 731, 982, 44, 891, 321]

The model processes sequences of tokens.

A language-model training example can look like:

Input:
The cat is sitting on the

Target:
mat

More generally:

Input:
T₁ T₂ T₃ T₄ T₅

Target:
T₂ T₃ T₄ T₅ T₆

This is the foundation of next-token prediction.

9. Step 7: The Transformer

The token sequence enters a Transformer.

From our previous article:

Tokens
 ↓
Embeddings
 ↓
Positional Information
 ↓
Self-Attention
 ↓
Feed-Forward Network
 ↓
Residual Connections
 ↓
Normalization
 ↓
Repeated Transformer Blocks
 ↓
Output Representation

A large model may contain many Transformer layers.

Each layer transforms the representations.

The final representation is converted into logits over the vocabulary.

10. Step 8: Next-Token Prediction

This is the heart of autoregressive language modeling.

Suppose the training text is:

The capital of France is Paris.

The model can be trained on several prediction tasks:

"The" 
→ predict "capital"

"The capital"
→ predict "of"

"The capital of"
→ predict "France"

"The capital of France"
→ predict "is"

"The capital of France is"
→ predict "Paris"

For each position, the model tries to predict the next token.

This means a single sequence can provide many training signals.

11. Step 9: Calculating Loss

The model produces logits for possible next tokens.

Suppose:

Correct token: Paris

The model predicts:

Paris       → 0.15
London      → 0.35
Berlin      → 0.20
Madrid      → 0.10
Other       → 0.20

The correct token only received 15%.

The loss will therefore be relatively high.

The model receives a signal:

Increase the probability of the correct token.

After billions of examples, these small signals accumulate into parameter changes.

12. Step 10: Backpropagation

The loss is propagated backward through the network.

Conceptually:

Prediction
    ↓
Loss
    ↓
Gradient
    ↓
Transformer Layer N
    ↓
Transformer Layer N-1
    ↓
...
    ↓
Embedding / Earlier Layers

Backpropagation calculates gradients indicating how the parameters contributed to the error.

13. Step 11: Updating Parameters

The optimizer then updates the model parameters.

Conceptually:

[
W_{new}=W_{old}-\eta\nabla_WL
]

Where:

• (W) = model parameters
• (L) = loss
• (\nabla_WL) = gradient
• (\eta) = learning rate

This happens repeatedly.

Batch 1
 ↓
Update

Batch 2
 ↓
Update

Batch 3
 ↓
Update

...

Millions of updates

Eventually, the model's parameters encode increasingly useful patterns.

14. What Happens During Pretraining?

Pretraining is the enormous first stage of learning.

A simplified loop:

Get batch
   ↓
Tokenize / load tokens
   ↓
Forward pass
   ↓
Predict next tokens
   ↓
Calculate loss
   ↓
Backpropagation
   ↓
Gradient calculation
   ↓
Optimizer update
   ↓
Repeat

This happens over a massive training corpus.

The model doesn't receive a lesson saying:

"Here is how grammar works."

Instead, grammar and other patterns can emerge because learning to predict language requires capturing useful regularities in the data.

15. Why Scale Matters

One of the most important ideas in modern AI is scale.

Three major factors are often discussed together:

Model Size
     +
Training Data
     +
Compute

Increasing these can improve capabilities, although the relationship isn't simply:

Bigger = always better.

Data quality, architecture, optimization, training strategy, and inference techniques also matter.

Modern AI engineering is therefore an exercise in balancing multiple resources.

16. What Does the Model Actually Learn?

This is where things become fascinating.

The model isn't simply memorizing a giant dictionary.

Its parameters can encode complex statistical representations.

During training, it can learn patterns involving:

Syntax

Subject
Verb
Object

Semantics

word meanings
relationships
context

Facts and associations

Paris ↔ France
Tokyo ↔ Japan

Code structure

function
variable
loop
class
API

Mathematical patterns

equations
operations
relationships

Long-range dependencies

Earlier context
      ↓
Later prediction

These capabilities emerge from optimization over huge datasets.

17. The Emergence of Capabilities

A model trained to predict tokens may eventually demonstrate abilities that weren't explicitly programmed as individual features.

For example:

Next-token prediction
        ↓
Language patterns
        ↓
Syntax
        ↓
Semantic representations
        ↓
Reasoning-like patterns
        ↓
Programming ability
        ↓
Translation
        ↓
Question answering

This is one of the most interesting aspects of large-scale machine learning.

The training objective can be relatively simple while the resulting learned representations become highly complex.

However, "emergence" should not be interpreted as magic.

The capabilities arise from the interaction of:

• architecture
• data
• optimization
• scale
• training objectives
• learned representations
• inference and post-training techniques

18. The Base Model

After pretraining, we have what is commonly called a base model.

It has learned to model its training distribution.

But imagine giving it:

Explain photosynthesis to a 10-year-old.

A base model may continue the text in many possible ways rather than behaving exactly like a polished assistant.

That's because:

Base Model

and:

Instruction-Following Assistant

are not necessarily the same thing.

This is why post-training matters.

19. Why a Base Model Isn't a Chatbot

A useful mental model is:

Pretraining
      ↓
"Learn language and patterns"

while post-training can help with:

"Follow instructions"
"Be helpful"
"Follow desired behavior"
"Respect safety constraints"
"Format answers appropriately"

A chatbot experience is therefore usually the result of more than pretraining alone.

20. Supervised Fine-Tuning

One important post-training technique is supervised fine-tuning, often called SFT.

The model is given examples of desired behavior.

For example:

User:
What is recursion?

Assistant:
Recursion is a programming technique...

Another:

User:
Write a Python function to reverse a string.

Assistant:
def reverse_string(s):
    return s[::-1]

The model is trained on examples like these.

The objective remains fundamentally based on predicting tokens, but now the training data is specifically designed to teach desired response patterns.

21. Instruction Tuning

Instruction tuning is closely related to supervised fine-tuning.

The training data emphasizes:

Instruction
      ↓
Desired Response

For example:

Instruction:
Summarize this article.

Response:
The article explains...

Or:

Instruction:
Convert this SQL query to PostgreSQL.

Response:
SELECT ...

Over many examples, the model learns to respond to instructions more effectively.

22. Preference Data

There is another problem.

Suppose we have two responses:

Response A

Very long,
irrelevant,
poorly structured answer.

Response B

Concise,
accurate,
helpful answer.

Both may be grammatically valid.

Which one should the model prefer?

This introduces the idea of preference data.

Humans or other evaluators can compare candidate responses.

For example:

Prompt
  ↓
Model generates A and B
  ↓
Evaluator prefers B

This preference information can be used during post-training.

23. RLHF

RLHF stands for:

Reinforcement Learning from Human Feedback

A simplified historical pipeline looks like:

Pretrained Model
       ↓
Supervised Fine-Tuning
       ↓
Generate Responses
       ↓
Human Preferences
       ↓
Reward Model
       ↓
Reinforcement Learning
       ↓
Improved Model

The important idea is that human preferences provide an additional training signal.

Instead of asking only:

"Did the model predict the next token?"

we can also ask:

"Which response is more useful or preferable?"

24. Reward Models

In a traditional RLHF pipeline, a reward model can learn to predict human preferences.

Imagine:

Prompt
  ↓
Response A ─┐
            ├──→ Reward Model → Score
Response B ─┘

If humans consistently prefer B, the reward model can learn patterns associated with preferred responses.

The resulting reward signal can then be used to optimize the language model.

25. Preference Optimization

Reinforcement learning isn't the only way to use preference data.

Modern systems can use direct preference optimization techniques.

The general idea is:

Preferred Response
        ↑
        │
Model learns preference
        │
        ↓
Rejected Response

The objective is to make preferred responses more likely than rejected alternatives.

This can simplify parts of the post-training pipeline.

26. DPO and Related Methods

One well-known method is:

DPO — Direct Preference Optimization

Instead of explicitly training a separate reward model and then running a traditional reinforcement-learning procedure, DPO directly optimizes the model using preference pairs.

Conceptually:

Prompt
 ↓
Preferred Answer
Rejected Answer
 ↓
Preference Objective
 ↓
Model Update

DPO is one example of a broader family of preference-optimization approaches.

The exact post-training recipe varies considerably across models and organizations.

27. AI Alignment

Alignment is a broad term.

In the context of language models, it can involve making model behavior better match desired goals, instructions, values, and constraints.

Examples include:

Helpfulness
Honesty
Safety
Instruction following
Robustness
Appropriate refusal behavior

Alignment isn't one single algorithm.

It is better understood as a broad area encompassing:

• training
• evaluation
• preference optimization
• safety methods
• policy constraints
• system design
• monitoring

28. Safety Training

Large AI systems need to be evaluated for potentially harmful behavior.

Safety work can include:

• red-team testing
• adversarial prompts
• policy evaluations
• harmful-content testing
• jailbreak testing
• privacy testing
• robustness evaluation

The goal is to identify failure modes before deployment.

A simplified process:

Model
 ↓
Adversarial Testing
 ↓
Find Failure
 ↓
Improve Training / System
 ↓
Retest
 ↓
Deploy

This process can continue after deployment as well.

29. Evaluation

Training loss alone isn't enough.

A model can have excellent training metrics and still perform poorly on important real-world tasks.

Evaluation may include:

Knowledge
Reasoning
Coding
Math
Instruction Following
Safety
Truthfulness
Robustness
Long Context
Multilingual Performance

Human evaluation can also be important.

A complete evaluation system might look like:

Automated Benchmarks
        +
Human Evaluation
        +
Adversarial Testing
        +
Real-world Testing

30. Model Checkpoints

Training extremely large models is expensive.

The model's state is therefore periodically saved.

For example:

Checkpoint 10,000
Checkpoint 20,000
Checkpoint 30,000
Checkpoint 40,000

Checkpoints allow engineers to:

• resume training
• compare model states
• investigate regressions
• select useful versions
• recover from failures

Training infrastructure must therefore be designed for reliability as well as raw computational speed.

31. Deployment

Eventually, the model needs to serve users.

But a training model and a production serving system have very different requirements.

Training focuses on:

Learning

Deployment focuses on:

Latency
Throughput
Reliability
Cost
Memory
Scalability
Safety

A production architecture might look like:

User
 ↓
API
 ↓
Load Balancer
 ↓
Inference Servers
 ↓
GPU / AI Accelerators
 ↓
LLM
 ↓
Generated Response

32. Inference

When you send a prompt to an LLM, the model performs inference.

Suppose you ask:

What is recursion?

The model processes your prompt and predicts tokens.

Perhaps:

Recursion

Then:

is

Then:

a

Then:

programming

and so on.

Conceptually:

Prompt
 ↓
Predict token
 ↓
Append token
 ↓
Predict next token
 ↓
Append token
 ↓
Repeat

This continues until a stopping condition is reached.

33. Why LLMs Sometimes Hallucinate

One of the most important things to understand is that next-token prediction does not automatically guarantee factual accuracy.

The model is fundamentally generating likely continuations according to its learned representations and current context.

Therefore, it can produce:

Fluent answer
+
Incorrect information

This is commonly called a hallucination.

For example:

Question
 ↓
Model generates confident response
 ↓
Response sounds plausible
 ↓
But factual claim is incorrect

This is why retrieval, tools, verification, better training, and careful system design can be important for factual tasks.

34. Common Misconceptions

Misconception 1: LLMs are trained by manually teaching every fact

No.

Most of the learning comes through large-scale automated training objectives over datasets.

Misconception 2: Next-token prediction is too simple to create intelligence

The objective is simple, but the model, data, and optimization scale can be enormous.

Learning to predict tokens well requires discovering many useful structures in the data.

Misconception 3: Fine-tuning creates the model from scratch

Usually, fine-tuning starts from an already pretrained model.

Pretraining
    ↓
Base Model
    ↓
Fine-Tuning
    ↓
Specialized / Instruction-Following Model

Misconception 4: RLHF is how the entire LLM is trained

No.

RLHF is a post-training technique, not the entirety of LLM training.

The majority of foundational learning generally occurs during pretraining.

Misconception 5: A chatbot searches the internet for every answer

Not necessarily.

A language model can generate answers from its learned parameters and current context.

Systems may additionally use search, retrieval, databases, APIs, or other tools when designed to do so.

35. Frequently Asked Questions

How long does it take to train an LLM?

It depends enormously on model size, dataset size, hardware, training efficiency, and training objectives. Large models can require substantial compute over extended periods.

How much data is required?

There is no universal number. Modern models can be trained on extremely large token datasets, but quality and composition matter as much as raw quantity.

Does an LLM read every document like a human?

No. Training converts data into numerical representations and processes it through batches of tokens.

Does the model store every training document?

Not in the simple sense of a database containing every document. Information is learned through changes to model parameters, although memorization of particular content can occur.

Why does the model need fine-tuning?

Pretraining teaches broad patterns. Fine-tuning and other post-training methods can improve instruction following and desired behaviors.

Is RLHF still the only way to align models?

No. RLHF is one approach among several preference-learning and alignment techniques.

Does training change the model's architecture?

Usually, training primarily changes the model's parameter values. Architecture is generally defined before training, although model development can involve architectural experimentation.

Why is LLM training so expensive?

Because large models require enormous amounts of computation, memory, storage, networking, and engineering infrastructure.

36. Key Takeaways

The complete lifecycle can be summarized as:

                 RAW DATA
                    ↓
              DATA CLEANING
                    ↓
              TOKENIZATION
                    ↓
             TRAINING DATA
                    ↓
              PRETRAINING
                    ↓
              BASE MODEL
                    ↓
          SUPERVISED FINE-TUNING
                    ↓
          PREFERENCE OPTIMIZATION
                    ↓
            SAFETY TRAINING
                    ↓
               EVALUATION
                    ↓
                DEPLOYMENT
                    ↓
              AI ASSISTANT

The most important concepts are:

Pretraining

Teaches broad patterns from massive datasets.

Next-Token Prediction

Provides the fundamental learning objective for autoregressive language models.

Fine-Tuning

Adapts an existing pretrained model to more specific behavior or domains.

Instruction Tuning

Improves the ability to follow user instructions.

Preference Optimization

Uses preferred versus less-preferred responses to shape model behavior.

RLHF

Uses human feedback in a reinforcement-learning-based post-training pipeline.

Alignment

A broad effort to make model behavior better match desired objectives and constraints.

Evaluation

Measures whether the model actually performs well and behaves appropriately.

Inference

The process of using the trained model to generate outputs.

37. Conclusion

A modern LLM isn't created by simply writing:

model = AI()

and pressing a button.

It is the result of a massive engineering pipeline.

It begins with data:

Web
Books
Code
Documents
Other Sources

Then:

Cleaning
   ↓
Filtering
   ↓
Tokenization
   ↓
Training

The model repeatedly predicts the next token.

When it makes mistakes:

Loss
 ↓
Backpropagation
 ↓
Gradients
 ↓
Optimizer
 ↓
Parameter Updates

After enormous numbers of updates, the model becomes a powerful base model.

Then additional stages can teach it to:

Follow instructions
Produce useful responses
Respect preferences
Handle safety constraints
Perform specialized tasks

Finally:

Evaluation
   ↓
Deployment
   ↓
Inference
   ↓
User

And that brings us to one of the biggest questions in modern AI:

If an LLM is essentially a huge mathematical function trained on enormous amounts of data, where exactly does its “knowledge” live?

Does the model store facts?

How are concepts represented inside billions of parameters?

Can we look inside a neural network and understand what individual neurons are doing?