Episode summary
The command was supposed to check capacity on Facebook's backbone network. Routine. The kind of thing that happens a hundred times a day inside a company that serves three and a half billion people. Then the backbone disconnected. Not partially. Not in one region. Every link, gone.
Within three minutes, Facebook's DNS servers noticed they could no longer talk to the data centers. They had a safety mechanism built in: if they lose contact with the backbone, the network must be unhealthy, so the responsible thing is to withdraw their BGP routes. They did exactly that. They erased themselves from the internet.
This episode is the story of the next six hours. The cascade that followed. The badge readers that locked engineers out of their own buildings. The out-of-band access that wasn't actually out of band. And the lesson that matters most for anyone who runs infrastructure: the safety mechanisms you build for partial failures are the things most likely to turn a partial failure into total erasure.
The internet is 75,000 networks held together by a protocol that runs on trust. When one of those networks tells the others it no longer exists, the others believe it. Instantly.
Show notes
- Cold open: 11:39 AM ET, the maintenance command, the global backbone disconnect.
- What the internet actually is: 75,000 autonomous systems held together by BGP. AS32934 is Facebook. The protocol runs on trust.
- The first fifteen minutes: verified timeline from Cloudflare, Kentik, and Catchpoint data. The TCP cliff. The DNS cache TTL that delayed the visible outage by eleven minutes.
- The cascade: internal tooling on the same backbone. Workplace, ticketing, monitoring, dashboards, badge readers, all dependent on the network they were trying to fix.
- Why the safety net failed: the DNS servers worked exactly as designed. The BGP protocol worked exactly as designed. The interaction between correctly-behaving components under untested conditions is the whole story.
- Getting back in: physical access to data centers. Out-of-band paths that depended on in-band infrastructure. The careful staged restart that took six hours.
- The lesson for any operator: every system has hidden dependencies. The scariest are the safety mechanisms, because nobody tests what happens when the safety mechanism is the thing that fails.
- What's next: Episode 2, "The Phone Book That Broke the Internet." A college student in New Jersey, a botnet of hijacked webcams, and the day DNS itself nearly came down.
Sources & fact-check
- engineering.fb.com: Facebook's own post-mortem, by Santosh Janardhan, Oct 5 2021
- blog.cloudflare.com: Cloudflare's BGP and DNS analysis
- kentik.com: Kentik's traffic analysis (TCP/UDP graphs, the "tera-lapse" calculation)
- wikipedia.org: consolidated public timeline