The Night the Database Died

Blast Radius
Episode 1 15 min

31 January 2017: a tired engineer wipes a database directory on the wrong server, and GitLab.com's production database starts deleting itself. Then the team reaches for five layers of backups.

Julian Ashcroft

0:01 The thirty-first of January, twenty seventeen. Late evening. An engineer types a command to wipe a database directory... and presses enter.

0:15 Then he sees which machine he's on. D B one. Not D B two. This is the production database of GitLab dot com.

0:28 And it is deleting itself, while he watches. That was the disaster. It was not the catastrophe. The catastrophe came next...

0:41 when the team reached for the backups.

Nora Whitfield

0:45 I'm Nora Whitfield. Before we begin... one honest disclosure. Everything in this story is real.

0:52 The company, the timestamps, the commands. All of it happened, and all of it comes from the public record. The voice about to tell it to you... is not real.

1:03 Neither is mine. We are synthetic... generated with AI. The facts are human.

1:10 The reconstruction is ours.

Julian Ashcroft

1:13 I'm Julian Ashcroft, and I'm your narrator. This is Blast Radius... stories from the nights our systems failed... and the people who brought them back.

1:26 This is a true story.

System

1:28 Incident declared. Start the clock.

Julian Ashcroft

1:33 January, twenty seventeen. GitLab is a young company, founded in twenty fourteen... about a hundred people, many of them working remotely. Their product is where developers keep their code...

1:48 their bug reports, their code reviews, their conversations. Trust is the entire business. You give them your work, and they keep it safe. Behind GitLab dot com there are two big database servers.

2:04 D B one... the primary. Every issue, every comment, every merge request lands here. And D B two...

2:13 the secondary. Its only job is to follow along, copying everything the primary does, a heartbeat behind. If the primary ever dies, the secondary steps in.

2:25 That's the design. That's the safety net. Keep those two names in mind. D B one.

2:34 D B two. One letter of difference... on machines that look identical when you're logged in. That day, GitLab is under attack.

2:46 Not a clever one — spammers, hammering the site, creating garbage at industrial speed. The database is drowning in load. And under that load, something snaps: the secondary falls behind. Replication...

3:02 stops. Now there is no follower. No copy, a heartbeat behind. Whatever happens to D B one...

3:16 happens to GitLab. An engineer on the infrastructure team steps up to fix it. It's late in Europe. He'd meant to sign off around eleven.

3:27 But replication is broken... so he stays. One more fix... and then sleep.

3:35 Oh — and one small detail. Earlier that afternoon, before testing some new software, an engineer took a snapshot of the production database by hand... and loaded it into the staging environment. A copy for testing.

3:53 Nobody thought about it twice. Remember it anyway. The fix for broken replication is fragile. GitLab's own report would later call it badly documented...

4:07 a handful of shell scripts. The secondary's data is stale and inconsistent... so you wipe it clean, and let it copy everything fresh from the primary. Erase the follower...

4:21 so it can follow again. But tonight, the copy tool won't cooperate. The team changes settings. Restarts the database.

4:32 Tries again... and it just hangs. Silent. No error.

4:38 No progress. Anyone who has run a production system knows this exact feeling. Eleven at night... a tool sitting there doing nothing...

4:50 and not one message to tell you why. He decides the secondary's data directory isn't empty enough. Fine. Wipe it properly, then.

5:03 One command. He's typed it a thousand times. Here is what the record shows. He believed he was on the secondary.

5:13 He was on the primary. Later, GitLab would change the prompt on every one of its servers... so the hosts would be easier to tell apart. One letter of difference.

5:28 The command went to D B one. Seconds later, he sees it. Cancels it. Four and a half gigabytes left...

5:38 out of three hundred. What happens next tells you everything about this company. No cover-up. No quiet scramble.

5:49 They take the site down, and say so in public. And then, just after midnight, the company's status account publishes a sentence... that no communications department in the world would have approved. Listen.

Record

6:06 We accidentally deleted production data... and might have to restore from backup.

Julian Ashcroft

6:13 That's real. They said it. In public. That night.

6:19 And they went further. The internal document tracking the disaster was opened to the whole world... and later, they put the recovery itself on a live stream. Thousands of strangers, watching engineers try to resurrect a database in real time...

6:40 like a surgery theater with a chat window. But before any of that can work... they need something to restore from. They need the backups.

6:54 Time to line up the suspects. Suspect number one: the spammers. They lit the fire — the load that broke replication in the first place. Guilty of arson, maybe.

7:08 But they didn't delete anything. Suspect number two: the tooling. The copy process that hung, silently, whatever they tried... and pushed a tired human toward a manual fix at eleven at night.

7:25 Closer. Suspect number three... the engineer. The man at the keyboard.

7:32 The obvious answer. The one everyone reaches for first. Hold that thought. Because this show has a rule, and the rule exists for a reason: if one tired person typing one command in the wrong window can erase your company...

7:52 the person was never the problem. The system was. The question is not who. The question is...

8:00 what else had already failed, silently, before he ever touched the keyboard. And no — in case you're wondering. Not this time. This time, it wasn't DNS.

8:14 It was about to be something much worse. Just after half past eleven. The team calls for help, and starts down the list of backups. GitLab had five separate ways to recover a lost database.

8:30 Five layers of safety net. Count them with me. Layer one. The regular database dumps.

8:38 A scheduled job, exporting everything. The team goes looking for the files... and finds nothing. The dump tool had been failing...

8:51 every single time it ran. A version mismatch: the backup binary was too old for the database it was backing up. It failed silently... and nobody knew.

9:04 Why did nobody know? Because the failure notifications... the emails that scream when a backup breaks... were never arriving.

9:15 Rejected by the mail system for a technical formality. The alarm was ringing in a room with nobody in it. Layer two. The uploads to cloud storage.

9:30 The bucket is empty. The job feeding it had depended on layer one. Layer three. Azure disk snapshots.

9:41 Never enabled for the database servers. Layer four. The disk-level snapshots on the machine itself. Normally taken just once every twenty-four hours.

9:54 The last regular one could be up to a day old. Restoring it could mean erasing a whole day of the world's work. And layer five? Replication itself.

10:07 The very thing that broke tonight. Five safety nets. Not one of them holding. And the whole building is on fire.

10:20 And then someone remembers one more copy. The one nobody thought of as a backup at all. The snapshot for staging. The copy an engineer took by hand that afternoon, for the testing environment...

10:38 about six hours before the deletion. It exists. It's real. It is, by pure luck...

10:51 the only survivor. So here is the reveal. The true root cause of the GitLab data loss... is not a command typed in the wrong window at eleven at night.

11:06 The database didn't die when he pressed enter. Parts of GitLab's safety had been dead long before that night. The dumps that produced nothing. The alerts that reached no one.

11:19 The snapshots that were never enabled. The deletion didn't cause the catastrophe. The deletion revealed it. A backup you have never restored...

11:35 is not a backup. It's a hope. GitLab had five hopes... and one accident.

11:46 The recovery is brutal and slow. The staging environment runs on cheaper infrastructure... so copying the data back takes hour... after hour...

11:58 around eighteen of them, streamed live, to an audience of thousands. The next evening, GitLab dot com comes back. The final damage: about six hours of database data, gone. Issues, merge requests, users, comments, snippets.

12:21 The code itself — the repositories — survived untouched. They were never in that database. Painful. Survivable.

12:32 Lucky. Then comes the part that turned a disaster into a legend. The postmortem. Public.

12:42 Complete. Unflinching. Every failed backup layer, named and explained. And the engineer?

12:52 In GitLab's reports, he is never named. The postmortem doesn't stop at the man at the keyboard. It goes after the systems that let one command do this much damage. The sentence, then.

13:07 Not for a person — for a system. Backups, actually monitored... alerts that actually arrive... restores, actually tested, on a schedule, forever.

13:22 And on every database prompt... the hostname, impossible to miss... so no one ever again has to guess which window is loaded. They fell in public...

13:34 so they rebuilt in public. Rigor. Respect. And the discipline to say: the system failed the human...

13:44 not the other way around. The dashboards are green again. The runbook is longer than it was yesterday. Somewhere in the Netherlands...

13:55 an engineer finally gets some sleep. So. The verdict. It wasn't the spammers.

14:04 It wasn't the replication lag. It wasn't the man at the keyboard — it was never really him. It was the backups that never ran... the alerts that never arrived...

14:18 the restores that were never tested. And honestly? This time... it wasn't even DNS.

14:28 One question before you go, for everyone with a pager on their nightstand: have you tested your restore this quarter? Not the backup. The restore.

14:43 Blast Radius is a synthetic-voice production, made with AI and based entirely on the public record — GitLab's own postmortem and their live-streamed recovery. Links, timestamps, and sources are in the show notes. Thanks for keeping me company tonight.

System

15:01 Incident resolved. Stop the clock.

Julian Ashcroft

15:07 ... Sleep with your pager on...