Forty-Five Minutes

Blast Radius
Episode 2 24 min

On the morning of August 1, 2012, one of the largest market makers on the New York Stock Exchange rolled out new trading code to eight servers. One of them kept the old code, and a reused configuration flag woke up logic that had been dead for nearly a decade. In the forty-five minutes before anyone could stop it, Knight Capital's system sent millions of orders into the market and lost about $460 million, more than the firm had. This is the story of a deployment with no kill switch, alerts nobody read, and how a company that moved a tenth of US equity volume was gone within a week.

Julian Ashcroft

0:01 New York. The first of August, twenty twelve. The stock market opened minutes ago. And inside one of the biggest trading firms in America, a machine is sending orders nobody meant it to send.

Engineer

0:17 It's not counting the fills. It just keeps sending. Child orders, over and over, straight into the market.

Commander

0:23 What changed?

Engineer

0:25 The new code. It went out this week. It has to be the new code.

Commander

0:31 Then pull it. Roll it back. Now.

Julian Ashcroft

0:36 It's the move any of us would make. Take out what changed. Go back to what worked. On seven servers, the new code comes out.

Engineer

0:48 It's worse. It's spreading to the other servers.

Julian Ashcroft

0:54 There is no kill switch. No written plan for this. And in an inbox, ninety-seven emails from before the opening bell sit unanswered, each carrying the same words...

1:07 Power Peg disabled. The clock keeps running. Forty-five minutes. More than four hundred and sixty million dollars.

1:21 By the end of that week, a firm that handled a tenth of America's stock trading will be fighting for its life.

Nora Whitfield

1:30 I'm Nora Whitfield. Before we begin... one honest disclosure. Everything in this story is real.

1:38 The company, the timeline, the numbers. All of it happened, and all of it comes from the public record. The voice about to tell it to you... is not real.

1:49 Neither is mine. We are synthetic... generated with AI. The facts are human.

1:55 The reconstruction is ours.

Julian Ashcroft

1:59 I'm Julian Ashcroft, and I'm your narrator. This is Blast Radius... stories from the nights our systems failed... and the people who brought them back.

2:12 This is a true story. Tonight's case is Knight Capital. In the summer of twenty twelve, it was one of the largest market makers in American stocks. Between what it traded for itself and for its customers, about ten percent of all trading in listed US equities went through Knight.

2:32 At the heart of it sat an order router called SMARS. It took large orders, the parent orders, and broke them into smaller child orders for the market. It ran on eight servers. And somewhere inside it sat a piece of code nobody had used since two thousand and three.

2:53 Unused... but still callable. That summer, the New York Stock Exchange was about to launch something new, a Retail Liquidity Program, starting on the first of August.

3:06 Knight wrote new code for it. And to switch that code on, it reused an old flag. What was at stake? A year later, the S E C put it in its order against the firm:

Record

3:22 The speed with which automated trading systems enter orders into the marketplace can turn an otherwise manageable error into an extreme event with potentially wide-spread impact.

Julian Ashcroft

3:34 An ordinary deployment. An old switch, flipped for a new reason. And forty-five minutes.

System

3:43 Incident declared. Start the clock.

Julian Ashcroft

3:49 Let's go back. Before the alarms... before the inbox. On an ordinary trading day, Knight Capital was a machine that simply worked.

4:01 Parent orders came into SMARS. SMARS cut them into child orders and sent them out to the market. Quietly. All day long.

4:10 To understand that morning, you need to meet one old part of that machine. Its name was Power Peg. Knight stopped using it in two thousand and three. But the code was never taken out.

4:24 It stayed on the production servers... and it could still be called. Power Peg had one brake. A counter that watched the fills, and stopped the child orders once the parent order was complete.

4:39 In two thousand and five, Knight moved that counting function to an earlier point in the code. Power Peg was never tested again. And so, silently, its brake stopped working. Dead code.

4:54 No brake. Sitting on live servers for years, with no reason for anyone to look at it. A systems-safety writer later put a question to every engineer listening tonight:

Record

5:08 Raise your hand if you can definitively state all of the active and inactive code execution paths in your application right now.

Julian Ashcroft

5:16 There were warning signs, too. In October twenty eleven, after a weekend disaster-recovery test, one of Knight's trading desks kept quoting from test data on Monday morning, and lost nearly seven and a half million dollars.

5:34 Knight fixed that one problem. Not the wider controls behind it. Knight did have a guard on prices. It capped how far an order could stray from the market.

5:47 But it didn't cover orders that arrived before the bell, for the opening auction. Remember that. July, twenty twelve. The exchange's Retail Liquidity Program launches on the first of August, and Knight's plan is a tidy one.

6:07 Delete Power Peg. Reuse its old flag. Any order that arrives with that flag set to yes will now run the new R L P code.

Commander

6:17 The launch date is fixed. Will the code be on every server in time?

Engineer

6:24 Eight servers. We go by hand, a few each day. All eight before the first.

Julian Ashcroft

6:32 From July twenty-seventh, it goes out, server by server, over successive days. Copy the code. Move to the next. The kind of work engineers do every week.

Engineer

6:47 That's the rollout. We're ready for the first.

Julian Ashcroft

6:52 But no second person checked the deployment, and no written step asked anyone to. On one of the eight servers, the new code never arrived. The old Power Peg code was still there. And on that server, the flag that now meant R L P...

7:11 still meant Power Peg. The thirty-first of July. The launch is ready. The servers are humming.

7:20 Tomorrow morning, at nine thirty, the market opens. The first of August, twenty twelve. Eight oh one in the morning, Eastern time. The market won't open for another hour and a half.

7:35 But inside Knight, something has already started talking. An internal system begins sending automated emails to a group of staff. B NET reject messages, they're called. They mention SMARS.

7:51 And each one says the same thing.

Record

7:55 Power Peg disabled.

Julian Ashcroft

7:58 Before the opening bell, ninety-seven of them arrive. Here is how the S E C's order describes those messages:

Record

8:08 Knight did not design these types of messages to be system alerts, and Knight personnel generally did not review them when they were received.

Julian Ashcroft

8:18 Not alarms. Just mail. The kind that lands every morning and gets skimmed past. A year later, the S E C's press release looked back at those warnings:

Record

8:32 they provided an opportunity to identify and fix the problem before the markets opened.

Julian Ashcroft

8:39 Nobody takes it. Nine thirty. The opening bell. Orders flow into SMARS carrying the reused flag.

8:50 On seven servers, that flag means the new code. On the eighth, it still means Power Peg. And Power Peg wakes up. It starts sending child orders.

9:04 But its brake stopped working seven years earlier. It never counts the fills. So it never stops.

Engineer

9:14 Something's wrong in PMON. The thirty-three account... the positions keep piling up.

Commander

9:23 Then stop the orders.

Engineer

9:26 PMON can't stop anything. It only watches. No automated alerts, no link to order entry.

Julian Ashcroft

9:38 Out in the market, traders see it before anyone at Knight can explain it. Stocks are suddenly flooded with orders. In the first hour of trading, a managing director at Wedbush Morgan told Reuters what it looked like from the outside:

Record

9:57 they are trading like millions of shares, 100 shares at a time, so something went haywire somewhere

Engineer

10:06 We're looking. We're looking. It has to be the new code.

Commander

10:12 Where's the switch? How do we shut it off?

Julian Ashcroft

10:17 There isn't one. Knight has no kill switch, and no written procedure for a morning like this. Writing about it later, one engineer who studied the S E C's order put it in one line:

Record

10:33 As it turns out there was no kill switch.

Julian Ashcroft

10:36 Prices start to lurch. The exchange's own safeguards pause trading in some of the hardest-hit stocks. Then Knight speaks. Its spokesperson, in a statement quoted by CNNMoney:

Record

10:51 Knight notified its market-making clients this morning to route listed orders away

Julian Ashcroft

10:57 A market maker, telling its own clients to take their orders somewhere else. The director of research at IndexUniverse told Reuters:

Record

11:07 The fact that Knight told people to go trade somewhere else is I am sure unprecedented in the industry

Julian Ashcroft

11:15 Inside, the clock keeps running. Child orders by the million, from one forgotten server. And the team reaches for the most natural fix there is. Every case has its suspects.

11:30 So, while the clock runs, let's line them up. Suspect one. The exchange. Months earlier, on Facebook's opening day, Knight said it lost around thirty million dollars to a Nasdaq glitch.

11:45 But this time the trades ran on the New York Stock Exchange... and a source told Reuters the exchange's own systems and circuit breakers had no problems. Suspect two. The new code.

11:59 It's the team's first instinct, and a fair one. But on seven of the eight servers, the new code was doing exactly what it was written to do. Suspect three. Whoever copied the code.

12:15 No. Not on this show. A systems-safety writer said it plainly:

Record

12:21 The engineers and managers at Knight Capital did not set out that morning of August 1, 2012 to lose $460 million.

Julian Ashcroft

12:31 Which leaves the oldest suspect in our trade. Was it DNS? It was not DNS. Not this time.

12:44 The real culprit was already in the building. It had been sitting there, quietly, since two thousand and three. The market is moving about eight million shares a minute. And inside Knight, people are hunting for one fault in a live system, still connected, with no written plan to follow.

Commander

13:08 The new code went out this week. Take it off the servers.

Engineer

13:15 Uninstalling it from the seven now.

Julian Ashcroft

13:19 The seven servers that were working. Now the orders arriving there carry the reused flag into the old code, with no brake. The S E C's order:

Record

13:32 This action worsened the problem, causing additional incoming parent orders to activate the Power Peg code that was present on those servers

Julian Ashcroft

13:42 The same engineer, years later, in one line:

Record

13:47 In other words, they removed the working code and left the broken code.

Julian Ashcroft

13:52 Don't judge them too fast. A systems-safety writer asked the question every on-call engineer should hear:

Record

14:01 Would it be surprising if one of the ways you attempt to rectify the scenario is to roll back to the last known working version?

Engineer

14:09 It's not one server anymore. It's all of them.

Julian Ashcroft

14:12 In thirty-seven stocks, Knight is now more than half of all the trading, and prices swing more than ten percent. A co-manager of trading at Themis Trading told Reuters:

Record

14:27 Grandma's mutual fund is now being whipped around by algos that don't even understand what they're doing

Julian Ashcroft

14:34 On the exchange floor, at Knight's booth, traders field calls and questions from exchange officials. Every minute is a lifetime. Then, about forty-five minutes after the open... the orders stop.

14:50 Two hundred and twelve parent orders. Millions of child orders. More than four million executions, in a hundred and fifty-four stocks. And when someone finally looks at the book...

15:05 Knight holds about three and a half BILLION dollars of stock it never meant to buy. And it's short another three point one five billion. The culprit has a name. Power Peg.

15:19 Retired in two thousand and three, but never removed. Its brake broken in silence since two thousand and five. And on the first of August, one reused flag woke it up. The eighth server, the one the new code never reached.

15:39 The S E C's order:

Record

15:42 Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added.

Julian Ashcroft

15:56 The ninety-seven emails that said Power Peg disabled, built as mail, not as alarms. The price guard that skipped orders sent in before the bell, and the two hundred and twelve orders that woke Power Peg came in for the opening auction.

16:15 The rollback that carried the flag to seven more servers. And no kill switch at the end of any of it. Another part of Knight's systems even knew those parent orders were filled. That never reached SMARS.

16:31 Not one villain. A forgotten path, a reused switch, a rollout by hand, and warnings nobody was meant to read. A chain... and on that morning, every link held.

16:49 The orders have stopped. The damage hasn't. Knight is holding billions of dollars of stock it never meant to own. A co-head of trading at Themis Trading told CNNMoney:

Record

17:03 you can't unwind positions like this quickly

Julian Ashcroft

17:07 By the S E C's count, the morning cost Knight more than four hundred and sixty million dollars, and left it with problems in its net capital. Its clients had been told to send their orders elsewhere. The firm had days, not weeks, to find new money.

17:26 It found it. A group of outside investors stepped in, and Knight survived... for a while. Within months it agreed to combine with another trading firm, GETCO.

17:42 On the first of July, twenty thirteen, the two became KCG Holdings. Then came the sentencing. October sixteenth, twenty thirteen. The S E C censured Knight, ordered it to cease and desist, and fined it twelve million dollars.

18:01 Knight settled without admitting or denying the findings. It was the agency's first enforcement action under the market access rule, the rule that says checks against erroneous orders must run automatically, before an order leaves the firm.

18:20 The order's sharpest line is about the simplest fix of all:

Record

18:25 A written procedure requiring a simple double-check of the deployment of the RLP code could have identified that a server had been missed and averted the events of August 1.

Julian Ashcroft

18:36 And the S E C's press release looked further back:

Record

18:41 The firm also reacted to prior events too narrowly and did not adequately consider the root causes of previous incidents.

Julian Ashcroft

18:50 That's the October twenty eleven desk again. One problem fixed. The system behind it left alone. But an enforcement order isn't a postmortem.

19:02 It records which rules were broken. A blameless review asks a different question: how did every step make sense to the people taking it? A systems-safety researcher, quoted in an essay on the S E C's order:

Record

19:21 In a lot of these accidents each individual component worked exactly the way it was expected to work.

Julian Ashcroft

19:29 So, the real action items. Not one of them is a person. The order required an independent consultant to review four things: how Knight deployed code, how it routed orders, how it enforced its capital limits, and how it responded to incidents.

19:49 Read them as lessons for the rest of us. Deployments that a machine verifies, not a checklist someone has to remember. That engineer again:

Record

20:02 Any time your deployment process relies on humans reading and following instructions you are exposing yourself to risk.

Julian Ashcroft

20:11 Dead code taken out, not just switched off. Warnings built as real alarms, sent to someone whose job is to act on them. Limits that act on their own. And a way to stop the machine, before the morning you need one.

20:31 The chief of the S E C's Market Abuse Unit put it as a question every team can ask:

Record

20:39 Brokers and dealers must look at each component in each of their systems and ask themselves what would happen if the component malfunctions and what safety nets are in place to limit the harm it could cause.

Julian Ashcroft

20:52 It wasn't the exchange. A source told Reuters its systems and circuit breakers had no problems. It wasn't the new code. On seven servers, it did exactly what it was written to do.

21:06 And it wasn't whoever copied it. Nobody at Knight set out that morning to lose four hundred and sixty million dollars. It was a piece of code retired in two thousand and three and never removed. A brake that broke in silence in two thousand and five.

21:27 An old flag given a new meaning. One server the rollout missed, with no second check to catch it. Ninety-seven warnings sent as mail. And no way to stop the machine once it started.

21:43 It was a system. Every part of it made sense on its own. Together, in forty-five minutes, they nearly cost a firm everything. So, for whoever carries the pager tonight, one lesson.

22:01 Know how to stop your system BEFORE the morning you need to. Where the switch is. Who is allowed to pull it. And what will actually be running when they do.

22:13 Write that down. Then let's hand down the verdict. So. The verdict.

System

22:22 Incident resolved. Stop the clock.

Julian Ashcroft

22:26 ... Sleep with your pager on... Tonight's case was built from the public record. Our main source is the S E C's order against Knight Capital Americas, from October twenty thirteen, along with the agency's press release on the settlement.

22:46 We also drew on same-day reporting from Reuters and CNNMoney, on the essay Knightmare: A DevOps Cautionary Tale, and on an incident analyst's reading of the S E C's order. Links, timestamps, and sources are in the show notes. No one who worked at Knight that morning is named in this story, and no one will be.

23:09 They did what made sense with what they could see. The voices you heard are synthetic. The script was drafted with AI and checked against the record. Somewhere tonight, on a server nobody thinks about, there is code nobody remembers.

23:29 Maybe it's yours. Go and look. Blast Radius comes from AllTuner Originals.