During the Super Bowl, no less!

Naturally, there’s a lot of speculation as to the cause. So far, all I’ve found is a cryptic statement:

A joint statement from Entergy New Orleans, which provides power to the stadium, and Superdome operator SMG shed some light on the chain of events, which apparently started at the spot where Entergy feeds power into the stadium’s lines. The problem occurred shortly after Beyonce put on a halftime show that featured extravagant lighting and video effects.

“A piece of equipment that is designed to monitor electrical load sensed an abnormality in the system,” the statement said. “Once the issue was detected, the sensing equipment operated as designed and opened a breaker, causing power to be partially cut to the Superdome in order to isolate the issue. … Entergy and SMG will continue to investigate the root cause of the abnormality.”

Far be it from me not to jump in on this stuff. You see, I used to work for Entergy, a whopping big electrical utility, one of whose branches supplies power in New Orleans. Further adding to my credentials, I was a “relay foreman”, that is one of the guys who’s supposed to work on “A piece of equipment that is designed to monitor electrical load”.

We call ’em “protective relays”. They’re a very interesting subset of the electrical power field. My job has been to make sure that a) they sense and abnormality and b) they operate as designed, usually opening breakers to get all those electrons away from the abnormal condition.

I don’t know what voltage level they feed into the SuperDome, but it’s probably in the 13,000 volt range. You don’t want to let 13 kV flow unimpeded very long. Stuff melts. Burns. Explodes. I used to have a client that had one of those events and the magic boxes failed to isolate the fault. It vaporized concrete.

So what happened?

Could’ve been something as simple as an overloaded circuit. Like an old kitchen when you try to run the toaster, the coffeepot and the microwave all at the same time on the same circuit. Too much current flows, the breaker thinks in its tiny little mechanical brain “If this keeps up, we’re gonna burn the place down” and it clicks. Somebody has to go reset it.

Naturally the stuff used on higher voltages is more complex, some just barely so, other stuff being little boxes with dozens of wires and little microprocessor hearts. Those devices have to be carefully programmed with settings tuned to the individual circuit situation. They depend on other devices in the circuit to feed information as to the state of things. If the information coming is suggests a problem, then –click!–, except this time, it’s not your Eggo and a cup of coffee, it’s a major production unit at a petrochemical plant or a city block or a hospital or, Heaven forbid!, the biggest public event of the calendar year.

So it could’ve actually been a problem, and the system worked to take the problem down (along with half the lights in the SuperDome) and left the electrical folks jumping out their collective asses to determine the problem and find out a way to safely return power to the affected circuits. Most modern critical power systems are built with redundancy so that loss of one power source can be overcome by routing power from a second source by the simple expedient of closing a few breakers, something I set up in MY stations to be accomplished by remote control.

However, first thing I have to do, dictated by both prudence AND national codes and regulations, is investigate the cause of the trip. The last thing I want to do is toss a bucket of electrons at a bit of equipment that had only begun to fail. The second go can be spectacular. So while the nation watched, I’ve got a picture of a bunch of power guys alternately shitting themselves and scrambling madly to look at fault records on computers and actually walking through the affected electrical equipment spaces sniffing for burnt smells and looking for visible damage.

Then it’s time to turn things on. And to the nation’s sigh of relief, the game was finished.

In my career, there have been many times that I was the poor schmuck standing in the substation when the lights went out. If you’re a contractor, as I was, hired to work on “A piece of equipment that is designed to monitor electrical load”, which was my specialty, then when the lights go out, you get to meet a lot of people VERY fast, because they will ALL come running to your little workspace to see what you did. Happily, only once in my career was I ever directly responsible for one of those outages, and fortunately that one outage was in a facility that used to trip off and go in the dark several times a week. I was there to fix that problem, and ultimately I did just that.

I’ve seen the magic boxes fail. Not often, but it happens. Worse, though, is I’ve seen the magic boxes incorrectly programmed. It goes both ways: not sensitive enough or TOO sensitive. The second one is easier found. The first one is often found in the aftermath of “It should’ve tripped here, not THERE!”, where “THERE” is the main substation feeding an entire petrochemical plant. I’ve seen the boxes and their associated circuits wired wrong, to where they would NEVER trip. And I’ve seen power supplies just sadly die when the system needed power to trip circuit breakers. That last one cost the power to a third of a city of 80,000. You can hardly imagine the joy that it brought. Or my glee when **I** found the problem that caused it, a poor connection on a battery bank.

Will we wever know the whole story of the Super Bowl outage? Somebody will. Entergy has some brilliant engineers (and some supernumerary dunces, too, but that’s a different story) who can dig down and find the cause. I know. I’ve been in on some of those kinds of meetings, including those where a fault in a 230 kV transmission yard rocked every industry in southwest Louisiana. They’ll KNOW what caused it. I know WE did. But will the story ever get out? Only in bits and pieces.