Okay. So you not only have a power outage, but you have a power outage in the middle of the biggest sporting event of the year, in front of a hundred million+ viewers.

You have to know that Entergy (the utility company) and the SuperDome people have been investigating their butts off over this, all the while muttering under their collective breath “Please-please-PLEASE don’t let it be MY screwup!”

Reports are coming out.

Relay equipment manufacturer blames Entergy for Super Bowl blackout
Super Bowl blackout

The Times-Picayune By Richard Thompson, NOLA.com | The Times-Picayune
on February 08, 2013 at 12:58 PM, updated February 08, 2013 at 7:49 PM

Without criticizing it as faulty, Entergy New Orleans on Friday blamed an electrical relay device in its electrical switchgear for the 34-minute power outage that brought Sunday’s Super Bowl to a halt. Entergy officials had zeroed in on the switchgear –– which controls the flow of electricity from the power company to the stadium and act somewhat like circuit-breakers –– in the wake of the partial outage, and were working with experts from the company that built the equipment to determine what happened.

Folks, this is what I did for a living for years. Let me explain: “zeroed in on the switchgear” is fog because that’s where the electricity is turned on and off. It does indeed “act somewhat like circuit-breakers” because, you know, Hey! we call them ‘circuit breakers’. (or in some cases ‘circuit switchers’ and a few other terms). The difference between MY circuit breaker at 15,000 volts and YOUR circuit breaker at 240 volts is that My breaker requires external circuitry to trip, things like a power supply and a ‘brain’ we call a protective relay.

When you (or your knowledgeable electrician) buys a circuit breaker, the circuit load is determined and you buy a breaker based on the load and the wire size. You have a twenty-amp circuit, you buy a twenty-amp breaker. Says so on the handle. Easy-peasy.

It’s not that simple when the electrons get bigger. I have a fifteen-thousand volt (15 kV – that’s a ‘class’. Actual voltage is usually something like 13.8 kV or 13.2 kV or something) system, I can only buy circuit breakers with numbers like 1200, 2000, 3000 and 4000 amps. (Not exactly, but hey, we’re just discussing here) Suppose I need a whole 150 amps at 13.8 kV for a transformer. There are no 150-amp breakers. Nope, I use that 1200-amp breaker and then set up a ‘relay’ to trip that breaker based on that 150 amp load. If the current is 200 amps, the relay might take ten minutes to trip. If the current is 900 amps, it may take two seconds. This is carefully determined, has a name (inverse time-overcurrent characteristic) and the determination of the setting is the subject of an engineering study.

At the end of the study, the engineer hands a technician a document and said technician goes to the device, plugs in a laptop, gets into the relay and applies the settings. Now, if everything is done correctly: the engineer, the constructor, the technician, everybody, then my 1200 amp breaker acts like your 150 amp breaker.

But as they say in the business, “Shit happens.”

The switchgear’s manufacturer, S&C Electric Co., said that its experts had concluded the outage was “a result of the electric load current exceeding the trip setting for the switchgear relay as set by the system operators.”

“Based on the onsite testing, we have determined that if higher settings had been applied, the equipment would not have disconnected the power,” said Michael J.S. Edmonds, vice president of strategic solutions for the company. “S&C continues to work with all those involved to get the system back online, and our customers can continue to rely on the quality and performance of our products.”

Seen it happen before, with my own eyes. Here’s what happens sometimes: The technician is there in the switchgear building. He doesn’t have ONE relay to put settings into. He’s got a dozen. Or more. And he has a folder on his laptop with a corresponding dozen ‘setpoint files’. It’s easy to get confused, especially when breakers and files have cryptic alphanumeric names. And so the file that was supposed to set a breaker for eight hundred amps gets loaded into the relay for a breaker that was supposed to be seeing a thousand amps.

Or the engineer can make the mistake. Running those studies to determine the settings of a large power system is an exercise in the application of meticulously detailed data into a specialty bit of expensive (I know, we bought one) programming. Put in an error, like a ‘6’ instead of an ‘8’, and you know the saying: “Garbage in, garbage out,” so the technician takes a faulty file to upload.

Or the equipment can be assembled incorrectly. A cable carrying eight hundred amps doesn’t connect to the relay because that cable would be as thick as your arm and would be carrying 13,800 volts. Instead, the current goes through a clever little device called a current transformer (CT), where eight hundred amps passing through transforms down to five amps (and 400 amps equals 2.5, and 200=1.25. It’s a transformer) and the five amps goes to the relay. The relay does its calculations based on that five amps, assuming that the current transformer is correct. If you and the relay think that the CT is 800 amps and the manufacturer installed a 600, then when 600 amps is flowing, the relay THINKS it’s actually 800, and acts accordingly.

Now, lest I get pushed into a box, what I just discussed in a bit above elementary detail is what system protection folks call a ‘basic overcurrent’ scheme. It’s but one of many different applications of protective relays. There are many others, the study and application of which makes for an interesting career. I’ve been on the upper end of ‘technician’ and lower end of ‘engineer’ in the field since 1989. I know much of the business. Nobody knows it all.

But in S&C’s defense, I can say that most often, the problem does not lie in the magic box, the relay. Nope, it’s in the application of the relay: Setpoints and inputs.

“We will continue to do more testing but we believe that we have zeroed in on the device that caused the outage, and we have removed it from service,” said Michael Burns, a utility spokesman.

The council’s Utility Committee, which regulates Entergy New Orleans, called the meeting in the wake of the Super Bowl blackout, which occurred early in the second half of the championship game between San Francisco and Baltimore.

Entergy New Orleans CEO Charles Rice told committee members that the device, “for some unknown reason at this particular time, did not react the way that it should.”

When pressed, Rice said the utility was not prepared to label the equipment as faulty, but said Entergy “installed the device as we received it.”

So “we have zeroed in on the device that caused the outage”. Big whoo! The little box was probably sitting there shining a red LED saying “I did it”, like a cat depositing a dead bird on your front step, all proud that it was doing what it was supposed to do. And I’m not a user of S&C’s product, but MOST modern relays keep records of exactly what was going on at the time of the operation, so there’s that, waiting to be downloaded and studied.

MR. Rice’s statement is couched in foggy verbiage.

And we may never know the truth.