Flash Sale Engine

The problem

Imagine 10,000 people trying to buy the last concert ticket at the same instant. A typical web app handles it like this:

1. Read available tickets from database      → 1 remaining
2. (some processing happens here...)
3. Write new ticket count back to database    → 0 remaining

The gap between step 1 and step 3 is the problem. While User A is between those steps, Users B through Z also read “1 remaining” and proceed to buy. The result: you sold 26 tickets but only had 1. This is a classic lost update problem, and it gets worse linearly with traffic.

The textbook fix is row-level locking (SELECT ... FOR UPDATE), which forces every request to wait in line. Safe, but slow — latency jumps from ~15ms to 2000ms+ because every transaction blocks on the previous one.

This project takes a different approach: move the inventory counter out of the database entirely. Redis executes Lua scripts atomically on a single thread, so the “check inventory” and “decrement counter” steps can’t be interleaved by any other request. No locks, no waiting, no overselling. Typical reservation latency: ~12ms.

Architecture

The system uses two-tier state management:

Ticket lifecycle:

1. Reserve — Redis Lua script atomically checks inventory >= quantity, decrements the counter, and creates a hold with a 30s TTL in a sorted set
2. Pay — Removes the hold, pushes the order to a Redis List (queue), increments the user’s ticket count
3. Expire — A background reaper runs ZRANGEBYSCORE every second, finds expired holds, and returns them to inventory
4. Persist — A separate async worker drains the Redis List and inserts confirmed orders into PostgreSQL

The reserve endpoint never touches PostgreSQL. The pay endpoint never blocks on a database write. The database only sees confirmed, paid orders.

Key implementation details

Atomic inventory control

The entire check-decrement-hold operation runs as a single Lua script inside Redis. Since Redis is single-threaded, no other command can interleave:

local count = tonumber(redis.call('GET', KEYS[1]))
if count >= tonumber(ARGV[3]) then
    redis.call('DECRBY', KEYS[1], ARGV[3])
    redis.call('ZADD', KEYS[2], ARGV[1], ARGV[2])
    redis.call('SET', KEYS[3], ARGV[3])
    return 1
end
return 0

Executes in ~0.05ms. No locks, no retries, no optimistic concurrency failures.

Write-behind persistence

Orders are not written to PostgreSQL during the /pay request:

/pay  →  RPUSH order_queue {user, qty, status}
           ↓
worker loop  →  LPOP  →  INSERT INTO orders

The endpoint returns immediately after pushing to the queue. If the database is slow, the queue absorbs the burst. The worker drains it asynchronously.

Real-time inventory updates

Inventory change → PUBLISH channel "tickets" count
                        ↓
               FastAPI subscriber → WebSocket broadcast → all connected clients

No polling. Sub-15ms propagation to 1,000+ concurrent clients.

Fairness and abuse prevention

• Per-user cap — max 5 tickets per JWT identity, tracked in Redis (user_tickets:{id}). Works across tabs, browsers, and sessions; the limit is enforced server-side.
• Rate limiting — sliding window counter in Redis. Exceed the limit, get a 429 Too Many Requests.

Cross-cloud event streaming

Order events can be published to GCP Pub/Sub from the /pay endpoint. The integration is optional; if GCP_PROJECT_ID isn’t configured, the import is skipped and the system behaves identically without it.

/pay → asyncio.create_task(publish_to_pubsub(user, qty))
         ↓ (fire and forget)
       GCP Pub/Sub topic → subscription → downstream consumers

Production observability

Every request is timed and emitted as Prometheus histograms. The /metrics endpoint exposes request counts, latency percentiles, reservation outcomes, payment outcomes, queue depth, and active reservations. Structured JSON logging via structlog makes the service easy to monitor in Loki, CloudWatch, or Datadog.

AI-powered bot detection

A real-time fraud-detection layer scores every reservation using a scikit-learn Isolation Forest trained on bot-vs-human behavioral patterns. Features include request velocity, quantity patterns, unique users per IP, user-agent presence, and pay latency. The model is trained offline via python -m backend.scripts.train_fraud_model and loaded at startup. Set FRAUD_DETECTION_ENABLED=true to block high-risk requests.

Microservices + Saga orchestration

A gRPC-based microservices architecture demonstrates distributed-systems patterns:

• reservation-service — inventory and reservation lifecycle (Redis Lua / optimistic fallback)
• payment-service — payment confirmation with PostgreSQL outbox pattern for exactly-once event publishing
• notification-service — consumes order events from Redis/RabbitMQ
• api-gateway — REST facade that orchestrates the Reserve → Pay → Notify Saga, compensating by releasing the reservation if payment fails

Client → API Gateway → reservation-service (reserve)
             ↓
         payment-service (pay + outbox)
             ↓
      notification-service (event)

docker compose -f docker-compose.microservices.yml up --build

Getting started

Prerequisites: Docker and Docker Compose.

docker-compose up --build
open http://localhost:8000

Test with bot swarm: open /live, set bot count to 20–50, click DEPLOY SWARM, watch inventory drop to 0 without going negative.

Run the chaos simulator: open /simulator, set concurrency to 150 threads. System A (naive read-check-write) shows counter drift and data-integrity loss; System B (Redis Lua) locks at exactly 0 with 100% integrity.

Headless load testing:

brew install k6
k6 run k6/load_test.js

Run tests:

pytest backend/tests -v --tb=short --cov=backend --cov-fail-under=60

Tech stack

Infrastructure

Multi-cloud deployment configurations in infra/:

infra/
├── terraform/
│   ├── modules/s3/          # AWS S3 static website hosting
│   ├── modules/gcp_pubsub/  # GCP Pub/Sub topic + dead letter queue
│   └── modules/rds/         # AWS RDS PostgreSQL (Multi-AZ, encrypted)
├── k8s/                     # Raw manifests — Deployment, HPA, PDB, Ingress
├── helm/flash-sale/         # Helm chart with templated values
└── database/                # Hardened local PostgreSQL + PgBouncer

Provisioned with Terraform. The Kubernetes configs include an HPA (2–10 replicas based on CPU/memory), a PodDisruptionBudget, and non-root security contexts.

Project structure

backend/
├── main.py              # FastAPI app — endpoints, Lua scripts, background workers
├── config.py            # Environment-based configuration
├── observability.py     # Prometheus metrics + structured logging
├── messaging.py         # Redis List / RabbitMQ order publisher abstraction
├── fraud_detection.py   # AI bot / fraud detection with feature store
├── pubsub_client.py     # GCP Pub/Sub integration (optional)
├── rabbitmq_worker.py   # RabbitMQ consumer with DLQ
├── scripts/             # Offline training / data pipelines
│   └── train_fraud_model.py
├── services/            # Microservices (reservation, payment, notification, gateway)
│   ├── reservation_service.py
│   ├── payment_service.py
│   ├── notification_service.py
│   ├── gateway.py
│   └── common.py
├── proto/               # gRPC protobuf definitions
└── tests/               # pytest suite (unit + integration)
frontend/
├── landing.html         # / — landing page
├── simulator.html       # /simulator — naive vs atomic comparison
├── index.html           # /live — booking interface
└── script.js            # Bot swarm, WebSocket client, UI logic
k6/
└── load_test.js         # Auth-aware load test script