Reliability
Browse all articles, tutorials, and guides about Reliability
Posts
Stop Building Webhook Retries Yourself
We pointed a webhook sender at a receiver built to fail the common ways production fails: outages, 429s, timeouts, dead endpoints, bad signatures. Then we watched the retries, the schedule, the signature checks, and the replay happen without writing any of it. Here is the run, with the code and the attempt logs.
The Postmortem Nobody Reads, and the One They Do
Most incident write-ups are compliance artifacts: written once, filed, and never opened again. The difference between those and the postmortems engineers actually forward to each other comes down to a handful of choices about audience, structure, and follow-through.
GitHub's 2.9B Monthly Commits: Anatomy of an Outage
GitHub's August 17 outage began with a missed sidecar limit and escalated through retry storms. Learn which reliability controls your platform needs next.
Running a Background Job That Must Not Be Lost
A queue gets your job to a worker, not to the finish line. What happens when the worker dies halfway, and a durable executor in 90 lines of TypeScript.
What It Actually Takes to Deliver a Webhook in Production
Sending a webhook is one HTTP POST. Delivering one is a retry schedule, a signature scheme, an idempotency story, and a way to answer "did you get it?" six hours later. Here is the whole problem, and a working Node implementation of both sides.
PostgreSQL 18.2 Broke Standbys: The 18.x Upgrade Footguns
PostgreSQL 18.2 shipped regressions bad enough to force an out-of-cycle 18.3: halted standbys, substring crashes, and a pg_trgm segfault. Here is what went wrong and the quieter Postgres 18 upgrade traps to check before you patch.
When One Data Center Room Got Hot: AWS US-EAST-1, Coinbase, and the DR Drill That Was Not
On May 7, 2026, cooling failed in a single hall of one US-EAST-1 data center. Coinbase, FanDuel, and CME Group went down for hours, and Coinbase publicly confirmed their backup systems did not work as expected. Here is what happened, the multi-AZ checklist that would have caught it, and the AWS Fault Injection Simulator commands to run the drill before the next thermal event.