Caching: How to Make Software Systems Faster and More Scalable

A system can have powerful servers, a well-designed database, and excellent APIs — and still become slow when millions of users start accessing it.

One of the most common solutions is caching.

Caching is one of the fundamental building blocks of scalable software architecture. Almost every large-scale application uses caching somewhere in its architecture.

From social media feeds and e-commerce product pages to banking dashboards, video platforms, and search engines, caching helps systems reduce database load and serve frequently requested data much faster.

In this article, we'll understand caching from the ground up and gradually move toward the problems engineers face when designing distributed caching systems.

What Is Caching?

A cache is a temporary storage layer that keeps frequently accessed data closer to the application.

Instead of repeatedly retrieving the same data from a slower source such as a database, the application can retrieve it from the cache.

A simplified architecture looks like this:

User
  |
  v
Application Server
  |
  v
Cache
  |
  v
Database

Without caching:

User
  |
  v
Application
  |
  v
Database

If thousands of users request the same information, the database may have to process thousands of identical queries.

With caching, the first request can fetch the data from the database and store it in the cache.

Future requests can then retrieve the data directly from the cache.

Why Do We Need Caching?

There are three major reasons.

1. Reduce Latency

Memory is significantly faster to access than disk-based database storage.

If an application can retrieve frequently requested information from memory instead of executing a database query every time, response times can improve dramatically.

For example:

Database query
    ↓
50 ms

Cache lookup
    ↓
1 ms

The exact numbers depend on the architecture, network, database, and workload, but the principle remains:

Avoid expensive work when the same result is needed repeatedly.

2. Reduce Database Load

Imagine an application receives:

100,000 requests

and every request executes a database query.

The database suddenly becomes responsible for processing:

100,000 database requests

Now suppose 80% of those requests are for data that rarely changes.

A cache could potentially handle a large portion of those reads.

The database may only need to process the requests that aren't already cached.

3. Improve Scalability

Caching allows the application to serve more requests without proportionally increasing database capacity.

This becomes extremely important as traffic grows.

Without Cache

1,000 requests
      |
      v
Database


With Cache

1,000 requests
      |
      v
    Cache
      |
      |-- 850 requests served
      |
      v
Database
   150 requests

The cache absorbs much of the repetitive workload.

Cache Hit and Cache Miss

Two terms appear constantly in system design interviews:

Cache Hit

Cache Miss

Suppose the application wants:

user:123

The application checks the cache.

Cache Hit

If the data exists:

Application
    |
    v
Cache
    |
    v
Data found

This is a cache hit.

The application can return the data immediately.

Cache Miss

If the data isn't available:

Application
    |
    v
Cache
    |
    X
Not found
    |
    v
Database
    |
    v
Store in Cache
    |
    v
Return response

This is a cache miss.

The application retrieves the data from the database and commonly stores it in the cache for future requests.

Cache Hit Ratio

A useful metric is the cache hit ratio.

Cache Hit Ratio =
Cache Hits / Total Cache Requests

For example:

Cache requests = 1,000,000

Cache hits = 900,000

Cache hit ratio = 90%

A high hit ratio generally means the cache is doing a good job for that workload.

However, blindly maximizing hit ratio isn't always the goal.

A cache should store useful and frequently accessed data, not simply as much data as possible.

The Cache-Aside Pattern

One of the most commonly used caching strategies is Cache-Aside.

It is also called Lazy Loading.

The application is responsible for interacting with both the cache and database.

The flow is:

Application
    |
    v
Check Cache
    |
    +---- Hit ----> Return Data
    |
    +---- Miss
           |
           v
       Database
           |
           v
      Store in Cache
           |
           v
       Return Data

Example:

GET /users/123

The application first checks:

user:123

in Redis.

If it exists:

Redis → User

If it doesn't:

Redis → MISS

Database → User

Database User
      ↓
Redis SET user:123
      ↓
Response

The next request can be served from Redis.

Why Cache-Aside Is Popular

Cache-aside is popular because it is relatively simple and flexible.

The application controls:

• What gets cached
• When data is cached
• How long it remains cached
• What gets invalidated

It also avoids automatically caching every piece of database data.

However, the application becomes responsible for managing cache behavior correctly.

Read-Through Cache

In a read-through cache, the application primarily interacts with the cache.

The cache itself knows how to retrieve missing data from the underlying data source.

Application
     |
     v
   Cache
     |
     +---- Hit ----> Return
     |
     +---- Miss
            |
            v
         Database
            |
            v
          Cache
            |
            v
        Application

This can simplify application logic, but it requires infrastructure or libraries that support this model.

Write-Through Cache

Now consider writes.

Suppose a user changes their name.

With a write-through cache:

Application
     |
     v
Cache
     |
     v
Database

The cache writes the updated value to the database as part of the write operation.

Conceptually:

Write
  |
  v
Cache
  |
  v
Database

The advantage is that the cache remains relatively fresh.

The downside is that writes can become more expensive because the system must update multiple layers.

Write-Back / Write-Behind Cache

A write-back cache takes a different approach.

The application writes to the cache first.

The cache updates the database later.

Application
     |
     v
Cache
     |
     |  later
     v
Database

This can make writes extremely fast.

But it introduces risk.

What happens if the cache fails before the data reaches the database?

Potentially, data can be lost.

Therefore, write-back caching should be used carefully, particularly when data durability is critical.

Write-Around Cache

Another strategy is write-around caching.

Writes go directly to the database instead of the cache.

Write:

Application
    |
    v
Database

Later, when the application reads the data:

Application
    |
    v
Cache
    |
    X
    |
Database
    |
    v
Cache

This can be useful when newly written data isn't immediately expected to be read.

Comparing Caching Strategies

There is no universally best strategy.

The correct choice depends on the workload.

TTL — Time To Live

Caches have limited memory.

If data remained in the cache forever, eventually the cache could become full of stale or unused information.

That's where TTL comes in.

TTL means:

Time To Live

Example:

user:123

TTL = 10 minutes

After 10 minutes, the cache entry expires.

10:00 → Stored
10:05 → Available
10:09 → Available
10:10 → Expired

The next request may need to retrieve fresh data from the database.

Why TTL Is Important

TTL provides a simple mechanism for limiting stale data.

For example:

Weather information

TTL = 1 minute

Product catalog

TTL = 10 minutes

User profile

TTL = 1 hour

Static configuration

TTL = several hours

The correct TTL depends on how frequently the underlying data changes and how much staleness the application can tolerate.

Cache Eviction

What happens when the cache becomes full?

The cache needs to decide which entries should be removed.

This is called eviction.

Common policies include:

LRU — Least Recently Used

Remove the data that hasn't been accessed for the longest time.

Example:

A → recently used
B → recently used
C → old
D → very old

If space is required:

D gets removed

LRU is one of the most common strategies.

LFU — Least Frequently Used

Remove data that has been accessed the fewest times.

Example:

Product A → 10,000 requests
Product B → 5,000 requests
Product C → 2 requests

Product C is a candidate for eviction.

LFU can be useful when frequency is more meaningful than recency.

FIFO — First In, First Out

The oldest cached entry is removed first.

A → first
B
C
D → newest

If space is required:

A gets removed

It is simple but doesn't account for popularity.

TTL-Based Expiration

Entries can also be removed simply because their TTL expires.

For example:

Product:123
TTL = 300 seconds

After 300 seconds, it becomes invalid.

Many real systems combine TTL with other eviction mechanisms.

Redis

When engineers talk about caching, one technology appears frequently:

Redis.

Redis is an in-memory data store commonly used for:

• Caching
• Sessions
• Rate limiting
• Distributed locks
• Counters
• Leaderboards
• Queues
• Temporary data

A simplified architecture:

                    +----------------+
                    |      Redis     |
                    |     Cache      |
                    +----------------+
                         ↑       ↓
                         |       |
User → Load Balancer → Application
                         |
                         v
                    +-----------+
                    | Database  |
                    +-----------+

Redis stores data primarily in memory, making it well suited for fast lookups.

Example: Caching a Product

Suppose an e-commerce application has:

GET /products/500

Without caching:

Client
  |
  v
API Server
  |
  v
Database
  |
  v
Product

With Redis:

Client
  |
  v
API Server
  |
  v
Redis
  |
  +---- HIT ----> Product
  |
  +---- MISS
          |
          v
      Database
          |
          v
        Redis
          |
          v
       Product

If the product is requested thousands of times, most requests may never reach the database.

Cache Invalidation

One of the hardest parts of caching is keeping cached data synchronized with the source of truth.

Imagine:

Database:

Product price = ₹999

Cache:

Product price = ₹999

Now the product price changes:

Database:

Product price = ₹799

But the cache still contains:

₹999

Users may receive incorrect information.

This is the classic cache invalidation problem.

A common approach is:

Update Database
      |
      v
Delete/Update Cache

For example:

UPDATE products
SET price = 799

DELETE product:500 FROM CACHE

The next request will miss the cache and retrieve the latest value.

Cache Invalidation Strategies

There are several approaches.

1. Delete on Update

When data changes:

Database updated
      ↓
Cache deleted

The next read repopulates the cache.

This is often simple and reliable.

2. Update Cache

Instead of deleting:

Database updated
      ↓
Cache updated

Now both layers contain the new value.

3. Short TTL

Allow cached data to become stale for a limited amount of time.

For example:

TTL = 60 seconds

This reduces the amount of invalidation logic but introduces a window where stale data can be served.

The Famous Cache Invalidation Problem

There is a famous engineering observation:

There are only a few truly difficult things in computer science, and cache invalidation is one of them.

The reason is simple.

Once data exists in multiple places, keeping those copies synchronized becomes a distributed systems problem.

Cache Stampede

Imagine a very popular page.

Millions of users request:

homepage

The cached value expires.

Suddenly:

Cache MISS
Cache MISS
Cache MISS
Cache MISS
Cache MISS
...

Thousands of application servers simultaneously query the database.

The database gets overwhelmed.

This is called a:

Cache Stampede

or

Thundering Herd

How Do We Prevent Cache Stampedes?

Several techniques can help.

1. Locking

Only one request rebuilds the cache.

Request A → Cache MISS → gets lock → Database

Request B → Cache MISS → waits

Request C → Cache MISS → waits

After A populates the cache:

Cache populated

B → Cache
C → Cache

2. Staggered Expiration

Instead of allowing thousands of keys to expire at exactly the same time, expiration times can be slightly randomized.

For example:

Base TTL = 300 seconds

Actual TTL:
302
315
307
324
309

This spreads database load over time.

3. Background Refresh

The system refreshes popular cache entries before they expire.

Cache
  |
  | nearing expiration
  v
Background Worker
  |
  v
Database
  |
  v
Updated Cache

This can be particularly useful for highly popular data.

Cache Penetration

Another problem occurs when users repeatedly request data that doesn't exist.

For example:

GET /users/999999999

The database has no such user.

The application doesn't cache the result.

An attacker or poorly designed client repeatedly requests the same nonexistent ID.

Every request becomes:

Cache MISS
     ↓
Database
     ↓
NOT FOUND

The database gets hammered.

This is called cache penetration.

Preventing Cache Penetration

One strategy is to temporarily cache negative results.

For example:

user:999999999 = NOT_FOUND

TTL = 60 seconds

Another strategy is to validate IDs before querying the database.

Bloom filters can also be used in large systems to quickly determine whether a requested key is likely to exist.

Cache Avalanche

Imagine thousands of cache entries all expire around the same time.

10:00

Cache
├── Product A → expired
├── Product B → expired
├── Product C → expired
├── Product D → expired
└── Product E → expired

Suddenly, a huge number of requests reach the database.

This is called a cache avalanche.

Solutions include:

• Randomized TTL
• Staggered expiration
• Background refresh
• Multiple cache layers
• Rate limiting
• Database protection mechanisms

Hot Keys

Sometimes one particular key receives an enormous number of requests.

For example:

homepage

or:

product:iphone-xyz

Suppose:

10 million requests

all hit the same cache key.

Even though the database is protected, the cache itself may become a bottleneck.

This is called a hot key problem.

Possible solutions include:

• Local caching
• Key replication
• Request coalescing
• CDN caching
• Sharding
• Precomputation

Local Cache vs Distributed Cache

Caching can happen at different layers.

Local Cache

Each application server has its own memory cache.

             Application 1
             +----------+
             |  Cache   |
             +----------+

Users → Load Balancer

             +----------+
             |  Cache   |
             +----------+
             Application 2

Advantages:

• Extremely fast
• No network request
• Simple

Disadvantages:

• Data duplicated
• Cache differs between servers
• Limited by server memory

Distributed Cache

A shared cache is used by multiple application servers.

                 +---------+
                 |  Redis  |
                 +---------+
                  ↑   ↑   ↑
                  |   |   |
               App1 App2 App3

Advantages:

• Shared cache
• Consistent cache view
• Larger overall cache capacity
• Useful for distributed applications

Disadvantages:

• Network latency
• Cache infrastructure must be highly available
• More operational complexity

Multi-Level Caching

Large systems may use more than one cache.

For example:

User
 |
 v
CDN
 |
 v
Application
 |
 v
Local Cache
 |
 v
Redis
 |
 v
Database

Each layer has a different purpose.

CDN

Useful for globally distributed content.

Local Cache

Extremely fast application-level data.

Redis

Shared distributed cache.

Database

Source of truth.

This layered approach can significantly reduce database traffic.

Caching APIs

Consider an application:

GET /users/123

A possible architecture:

                   ┌─────────────┐
                   │    Client   │
                   └──────┬──────┘
                          │
                          v
                   ┌─────────────┐
                   │Load Balancer│
                   └──────┬──────┘
                          │
                          v
                   ┌─────────────┐
                   │ API Server  │
                   └──────┬──────┘
                          │
                   ┌──────v──────┐
                   │    Redis   │
                   └──────┬──────┘
                          │
                       Cache Miss
                          │
                          v
                   ┌─────────────┐
                   │  Database   │
                   └─────────────┘

The API server doesn't necessarily need to query the database for every request.

Caching and Consistency

Caching introduces a fundamental trade-off:

Freshness vs Performance

If you cache data for a long time:

Performance ↑
Freshness   ↓

If you aggressively invalidate data:

Freshness   ↑
Complexity  ↑
Database Load ↑

System designers need to understand how stale the application can tolerate being.

For a social media follower count, a few seconds of staleness may be acceptable.

For a bank balance, stale data can be dangerous.

Therefore:

Cache according to business requirements, not simply because caching is available.

What Should You Cache?

Good candidates include:

• Frequently accessed data
• Expensive database queries
• Product information
• User profiles
• Configuration
• Computed results
• API responses
• Session information
• Frequently accessed reference data

Poor candidates include:

• Data that changes constantly
• Highly unique one-time queries
• Extremely large objects
• Sensitive data without proper security controls
• Data where stale values are unacceptable

A Practical Caching Strategy

When designing a system, ask:

Step 1: What is expensive?

Identify:

Database queries
External APIs
Complex calculations
Expensive aggregations

Step 2: What is frequently requested?

Look for:

Popular products
Popular posts
User profiles
Trending content
Configuration

Step 3: How stale can the data be?

Define:

1 second?
1 minute?
1 hour?
Never?

Step 4: Choose a strategy

For many applications:

Cache-Aside + TTL

is a strong starting point.

Step 5: Design invalidation

Ask:

When does this data change?
Who invalidates the cache?
What happens if invalidation fails?

Step 6: Handle failure

What happens if Redis goes down?

A robust application should usually degrade gracefully:

Redis unavailable
       ↓
Application
       ↓
Database

The system may become slower, but it shouldn't necessarily become completely unavailable.

Example: Designing a News Feed

Suppose we are designing a social media news feed.

Users frequently request:

GET /feed

Generating the feed may require:

• Fetching followed users
• Fetching posts
• Sorting posts
• Ranking content
• Applying personalization

Doing all of this for every request could be expensive.

We could cache the result:

User 123 Feed
      |
      v
Redis
      |
      +---- HIT → Return feed
      |
      +---- MISS
              |
              v
          Feed Service
              |
              v
          Database
              |
              v
            Redis

But now we have another question:

What happens when a user creates a new post?

We need an invalidation or update strategy.

This demonstrates why caching isn't simply:

Put Redis in front of database.

The real challenge is deciding what to cache, when to cache it, and when to invalidate it.

Caching in a Large-Scale Architecture

A mature system might look like:

                         Users
                           |
                           v
                         CDN
                           |
                           v
                    Load Balancers
                           |
              +------------+------------+
              |            |            |
              v            v            v
            App 1        App 2        App 3
              |            |            |
              +------------+------------+
                           |
                           v
                    Distributed Cache
                         Redis
                           |
                    Cache Miss Only
                           |
                           v
                        Database
                           |
                           v
                    Read Replicas

Notice something important:

The database still exists.

Caching does not replace the database.

The database generally remains the source of truth.

The cache is an optimization layer.

Common Caching Mistakes

Mistake 1: Caching Everything

More cache doesn't automatically mean a better system.

Mistake 2: No TTL

Permanent cached data can become stale.

Mistake 3: Ignoring Invalidation

If the underlying data changes, the cache needs a strategy.

Mistake 4: One Giant Cache

A single cache can become a critical point of failure.

Mistake 5: Ignoring Cache Failures

Always consider:

What happens if Redis is unavailable?

Mistake 6: Using the Same TTL Everywhere

Different data has different freshness requirements.

Mistake 7: Forgetting Hot Keys

One extremely popular key can create unexpected bottlenecks.

System Design Interview Questions About Caching

1. What is caching?

A mechanism for temporarily storing frequently accessed data closer to the application to reduce latency and backend load.

2. What is a cache hit?

When requested data is found in the cache.

3. What is a cache miss?

When requested data isn't found in the cache and must be retrieved elsewhere.

4. What is cache-aside?

The application checks the cache first and retrieves data from the database on a miss.

5. What is TTL?

Time To Live — the duration for which cached data remains valid.

6. What is LRU?

Least Recently Used — an eviction policy that removes data that hasn't been accessed recently.

7. What is a cache stampede?

A situation where many requests simultaneously attempt to regenerate expired cache data, potentially overwhelming the backend.

8. What happens if Redis goes down?

The application should have a fallback strategy, often retrieving data from the database while protecting the database from a sudden traffic spike.

9. How do you handle stale data?

Possible approaches include:

• Short TTL
• Explicit invalidation
• Cache updates
• Versioning
• Event-driven invalidation

10. When should you not use caching?

When data changes extremely frequently, has strict freshness requirements, or has a low reuse rate that doesn't justify the caching complexity.

Interview Cheat Sheet

Remember this flow:

Request
   |
   v
Cache?
   |
   +---- HIT ----> Return
   |
   +---- MISS
          |
          v
       Database
          |
          v
       Cache
          |
          v
       Return

Remember these concepts:

Cache Hit
Cache Miss
Cache-Aside
Read-Through
Write-Through
Write-Back
Write-Around
TTL
LRU
LFU
Cache Invalidation
Cache Stampede
Cache Penetration
Cache Avalanche
Hot Keys
Local Cache
Distributed Cache

And most importantly:

Caching = Performance + Scalability
           +
        Complexity

Final Takeaway

Caching is one of the most powerful tools available to a system designer.

It can:

• Reduce latency
• Reduce database load
• Increase throughput
• Improve scalability
• Reduce expensive computation

But caching also introduces new problems:

• Stale data
• Invalidation
• Cache failures
• Stampedes
• Hot keys
• Consistency
• Memory limitations
• Operational complexity

A strong system designer doesn't simply say:

"Let's add Redis."

A strong system designer asks:

"What data should we cache, why should we cache it, how long should it live, how will it be invalidated, and what happens when the cache fails?"

That mindset is what turns caching from a technology choice into a system design decision.