The Northeast Blackout Struck 15 Years Ago Today

Just past 4 PM on August 14, 2003, a widespread power outage rippled across the Northeastern and Midwestern United States as well as throughout Ontario. By the time the events had run their course, more than 55 million people across two countries had lost power. In some cases, service took weeks to restore. Initially, it wasn’t clear what had happened or gone wrong, forensic analysts spent weeks pouring over logs from multiple power plants across the region to see how the failure had developed and metastasized.

What makes the entire affair remarkable is that it happened due to a simple software bug. Power distribution grids must maintain a very close alignment between the load on the network, its supply, and its ability to transmit power from Point A to B. At 12 PM, incorrect telemetry data took a power flow monitoring tool offline at the Midwest Independent Transmission System Operator. The telemetry problem was corrected, but the monitor wasn’t restarted. Meanwhile, at FirstEnergy, a race condition in the company’s software stalled alarms from sounding and prevented the power monitoring team from observing any data that would’ve showed them that the power distribution grid had begun to fail.

The main problem kicked off when high-voltage transmission lines began to sag under heavy load and made contact with trees. As power lines heat up, the metal inside of them expands, causing them to sag. In this case, the lines hit tree branches that hadn’t been properly trimmed. This causes a transient flashover event as current carried by the line is transferred to other objects. The grid is designed to deal with this kind of problem, but at the time, it dealt with it by disconnecting the lines in question — which increased the load on other high-voltage lines in the same area. And as the current feeding through those lines started to increase, the same problem happened again. And again. And again. Other plant operators began to notice the problem, but they focused on fixing issues in their local network rather than informing other plant operators.

The first line in Northeast Ohio failed at 2:02. The problem built from there, until 15 138kV lines failed nearly at once at 3:41 PM in northern Ohio. A post-mortem report on the blackout describes it thusly:

At approximately 16:10, something remarkable happened: the power began to flow backwards through the system. Power began to flow in a counterclockwise fashion, from Pennsylvania, through New York, into Ontario, and then into Michigan and Ohio. Typically, the 3700 MW of load served by this rever[s]ed power flow was not met by Ontario, but Michigan and Ohio had lost all of their other major tie lines.

Wikipedia’s timeline of the event offers additional color. As failures cascaded across the system, power networks began taking automatic actions designed to preserve local stability. In this case, however, those actions — like cutting lines and diverting power to still-functioning ties — only served to heighten the failure as surges hit some areas and under-voltage conditions struck others.

As lines failed, some areas became isolated. This sudden disruption between power demand and supply caused power plants to automatically drop offline to prevent damage. Once 4:10 hit, the failures accelerated. I know that the below is a giant wall of text, but give it a skim to see how the collapse built.

The Atlantic has put together a photo essay of the event and how it impacted New Yorkers, including those stuck in the city’s transit systems. The blackout was a telling example of how vulnerable modern infrastructure can be to disruption and a reminder that our sophisticated failure prevention systems are only as effective as the scenarios they contemplate. Long term, the blackout led to the declaration of the North American Electric Reliability Council (NERC) as an Energy Reliability Organization (ERO), whose job was to develop and enforce mandatory reliability standards with regard to the national power grid. Previously, compliance with NERC policies aimed at encouraging grid reliability and stability was strongly encouraged, but voluntary.

Feature image by Wikimedia

Continue reading

Google Struggled to Patch New Stadia Game After Closing Development Studio

Google's in-house game developers were supposed to lead the charge, but now most of them are out of work, and there's no one to issue prompt patches for a brand new game. It's just one more embarrassing misstep for Stadia.

Boeing Is Still Struggling to Fix Latest Starliner Glitch In Time for Launch

Boeing was supposed to be riding high on the success of its long-delayed Starliner spacecraft right now, but the launch was called off again last week. At the time, Boeing only said there was a problem with "unexpected valve position indications," but the extent of the issue is more severe than initially reported.

Smallest Flying Structure Ever Made Inspired By Storybooks and Seeds

Researchers have devised the smallest human-made fliers ever constructed: a whole new cadre of biomimetic winged microchips, the size of a grain of sand. What is this, a helicopter for ants?

New Research Says Dinosaur-Killing Asteroid Struck Earth in Springtime

Scientists have studied the so-called Chicxulub impact for decades but only on millennia timescales. For the first time, researchers have zeroed in on when the asteroid or comet walloped Earth, and they say it happened on an otherwise pleasant spring day.