The importance of backup power

We’re located in the northern Virginia area – where Friday night brought a derecho which is basically a hurricane on land. Unfortunately, our county lost 911 service, and 3 days later, it’s still not quite back up. The 911 service is run by Verizon, which said that both primary and backup power was lost.

Amazon Web Services lost Netflix, Foursquare, Pinterest, and other sites.

So – assuming that these services were in a traditional data center, what happened? These buildings are supposed to have backup generators – why didn’t they kick in? Did they not test the generators, or the ATS (automatic transfer switch)?

People pay data centers for continuous power – and most offer 5 9s of power (~5 minutes/yr down time) or more, so practically any power outage is unacceptable. The folks that purchase these services expect that downtime to be minimal – otherwise, they’d just put their servers in their offices and skip paying for the data center (really, who’s paying for the security?)

I’m looking forward to reading about the post-mortems (if available) to see what exactly happened.