Laziness in the data center

Laziness in the data center

Some data center operations people are not too smart. They do dumb things that screw up systems for the customers and sometimes even get hurt. I heard some crazy stories from my friends who had worked in the data centers of various hosting places, and this is one of them.

As the story goes, it started as an ordinary Saturday afternoon. My friend rolled in to the parking lot at the data center where he worked and noticed the boss's car in one of the spaces. This was never a good sign, since he would never be around on a weekend under normal conditions.

Once inside, he found out why: one of the techs -- we'll call him "Z" -- had gotten a good shock when "his cutters slipped". The resulting short circuit had gone down Z's arm and out through his elbow into another rack. This blew the PDU for a big area and took out a bunch of machines.

Naturally, they wanted to know what Z had been doing which could have caused such a thing to happen. Z said that he had been working to clean up a rack which was going to be rebuilt, and one of the tasks was to remove all of the old power cables. According to this guy, he was going to cut one of the wire ties holding the whole mess together, and he "missed" or "slipped" and somehow bit into a live wire. The rest was inevitable.

Now, you have to realize, he wasn't using some kind of bushwhacking knife or cutter here. The data center guys had these tiny little snips to take care of wire ties. They were maybe 3 inches long, and they didn't open all that far. You could fit one around a zip tie easily, but a power cable? And by mistake? Highly unlikely.

My friend who worked there with Z asked around and started putting it together. It seems that he was supposed to remove a bunch of power cords, and they had all been zip-tied into a huge bundle by design. Only some of them were supposed to come out. Some of them would stay around.

To do something like this, you have two options. You either cut the bundle open, pull the cables out, then re-tie the rest and repeat every couple of inches, or you do what my friend thinks this guy Z did.

Realize that you can't just pull a power cable out from either end. It's bigger on the ends than it is in the middle and won't fit through the ties. Ah, but, if you could cut it in the middle, you could pull it out the top and the bottom and skip redoing the bundles!

As near as anyone can figure, he decided to do this. Using those tiny little snips, he must have FORCED it onto the power cord, and then really squeezed it like crazy to make it go. My friend tried to duplicate this on a dead cable and found it took two or three good bites before it would work.

This was no accident. He wanted to cut that cable.

Trouble is, he cut the WRONG cable. He cut one which was still live.

Oops.