Back to blog
• 6 min read • Backend Engineering

Solving Race Conditions at 10K RPS

Ten thousand people. One hundred tickets. That combination broke my first implementation in ways I didn’t anticipate. I built a Flash Sale engine thinking I understood concurrency. The system crashed on day one, oversold tickets, and taught me more about distributed systems than any textbook.

The Naive Approach (And Why It Fails)

When I first sketched out the architecture, the logic seemed simple enough:

  1. Read the current inventory from the database
  2. Check if tickets are available
  3. If yes, decrement the count and create the order

Here’s what that looks like in Python with SQL:

@app.post("/reserve")
async def reserve_ticket(user_id: str):
    # Step 1: Read
    result = await db.fetch_one("SELECT inventory FROM events WHERE id = 1")
    current_inventory = result["inventory"]

    # Step 2: Check
    if current_inventory > 0:
        # Step 3: Write
        await db.execute(
            "UPDATE events SET inventory = inventory - 1 WHERE id = 1"
        )
        await db.execute(
            "INSERT INTO orders (user_id, event_id) VALUES (?, ?)",
            user_id, 1
        )
        return {"status": "success"}
    return {"status": "sold_out"}

This works perfectly in development with one user. It fails catastrophically in production.

The Race Condition Window

The bug hides in the microseconds between read and write. At 10,000 requests per second, thousands of operations squeeze into that gap. Picture this sequence:

  1. User A reads inventory: 1 ticket remaining
  2. User B reads inventory: 1 ticket remaining (before User A writes)
  3. User A buys: inventory → 0
  4. User B buys: inventory → -1

We’ve oversold. This is the classic “Lost Update” problem in database theory.

Attempt 1: Database Row Locking

My first fix used SELECT FOR UPDATE, the standard approach for database consistency:

async with db.transaction():
    result = await db.fetch_one(
        "SELECT inventory FROM events WHERE id = 1 FOR UPDATE"
    )
    # ... rest of the logic

This guarantees correctness—no overselling—but the performance is terrible. At high concurrency, requests queue up waiting for locks. Latency spikes from 15ms to over 2 seconds as contention increases. For a Flash Sale where users expect instant feedback, this is unacceptable.

Moving to Redis

The system needed to handle 10K requests per second without overselling. Database locking hit that, but response times spiked to 2 seconds. I moved the hot path into Redis after reading about atomic Lua scripts.

Redis executes Lua scripts atomically—meaning the entire script runs as a single, indivisible operation. While the script runs, no other commands execute. This delivers the same correctness guarantees as database locking, but keeps latency under 15ms instead of 2000ms.

Here’s the Lua script that powers the inventory check:

local inventory_key = KEYS[1]
local user_reservation_key = KEYS[2]
local user_id = ARGV[1]
local ttl = tonumber(ARGV[2])

-- Check if user already has a reservation
local existing = redis.call('ZSCORE', user_reservation_key, user_id)
if existing then
    return {err = "ALREADY_RESERVED"}
end

-- Check inventory
local current = tonumber(redis.call('GET', inventory_key) or 0)
if current <= 0 then
    return {err = "SOLD_OUT"}
end

-- Atomic decrement and reservation
redis.call('DECR', inventory_key)
redis.call('ZADD', user_reservation_key, ttl, user_id)

return {ok = "SUCCESS"}

Redis executes Lua scripts on a single thread. No other commands run during that script. The check and decrement happen as one indivisible operation, eliminating the race condition entirely.

The Distributed State Machine

Reservations are more complex than simple purchases. A user reserves a ticket, has 10 minutes to pay, and if they don’t, the ticket goes back into the pool. This is a classic distributed state machine problem.

I implemented this using Redis Sorted Sets (ZSET) where the score is the expiration timestamp:

# When reserving: ZADD reservations <expiry_timestamp> <user_id>
# The reaper worker runs every second:
expired = redis.zrangebyscore('reservations', 0, current_time)
for user_id in expired:
    redis.incr('inventory')  # Return ticket to pool
    redis.zrem('reservations', user_id)

The ZRANGEBYSCORE operation is O(log N), meaning even with thousands of active reservations, finding the expired ones takes microseconds.

Event-Driven Architecture

Users hate refreshing the page to see if tickets are available. I implemented WebSockets with Redis Pub/Sub for real-time inventory updates:

  1. When a reservation succeeds, the backend publishes to a channel
  2. All connected clients receive the update within 15ms
  3. The UI updates without polling

This eliminated the “F5 spam” behavior typical of high-demand sales and reduced server CPU load by 40% compared to polling-based approaches.

The Write-Behind Pattern

Redis is fast but volatile. For durability, orders eventually need to reach PostgreSQL. But writing to the database synchronously would reintroduce the latency we’re trying to avoid.

The solution is an async Write-Behind queue:

  1. Successful reservations go into a Redis List
  2. A background worker consumes the list and batches writes to PostgreSQL
  3. If the worker crashes, the list persists in Redis (with AOF enabled)

This decouples the user-facing latency from database write performance. The user gets instant confirmation; durability happens asynchronously.

Load Testing and Chaos Engineering

I built a “Bot Swarm” simulator that runs directly in the browser. It spawns N concurrent agents that authenticate, reserve, and pay—creating realistic lock contention patterns for testing.

For automated testing, I used k6 to simulate 10,000 concurrent users:

export const options = {
  stages: [
    { duration: '30s', target: 1000 },
    { duration: '1m', target: 10000 },
    { duration: '30s', target: 0 },
  ],
};

The results: 8,500+ RPS sustained, 12ms average latency, zero overselling incidents.

Lessons for System Designers

Memory-first architectures are powerful: Keeping hot state in Redis and cold state in PostgreSQL gives you the best of both worlds—speed and durability.

Atomicity is non-negotiable: You cannot fix race conditions with application-layer logic. You need database (or Redis) level atomic operations.

Observability matters: The Chaos Simulator page visualizes the difference between the naive SQL approach and the Redis Lua approach in real-time. I built this visualization to prove the system worked, not just to show it off.

Fairness is a feature: Rate limiting (token bucket), purchase limits per account, and queue fairness algorithms need planning from day one. Users trust systems that treat everyone equally.

Try It Yourself

The system is live here. Open the Chaos Simulator to see the race condition in action, then watch how the Redis-based atomic approach maintains perfect inventory counts even under extreme load.

The code demonstrates patterns applicable beyond ticketing: inventory management, seat reservations, limited-quantity sales—any domain where correctness under concurrency matters.

Back to top