💥 Production Postmortems

Real-world outage stories and the lessons learned

Every Monday/Wednesday
  1. #4

    Our AWS Bill Went from $200 to $14,000 in One Week

    How a misconfigured Lambda function caused a $14,000 AWS bill in one week. The debugging process and how to prevent it.
  2. #3

    The Day We Accidentally Deleted the Production Database

    A postmortem about accidentally running a destructive command on the production database. How it happened and how we recovered.
  3. #2

    How a Missing Database Index Took Down Our API for 3 Hours

    A production postmortem about a missing PostgreSQL index that caused cascading API failures. The debugging process and fix.
  4. #1

    We Deployed to Production on a Friday — Here's What Happened

    A realistic production postmortem about a Friday deploy gone wrong. What broke, why, and the lessons learned.