The "fixers" you might already have without realizing it

The "fixers" you might already have without realizing it

One of the things I used to do back when I was in the web hosting business was to troll the support ticket queues to see what sort of gunk had built up at the bottom. I'd frequently find tickets which had been mishandled in all sorts of interesting ways. There were ways to set tickets to statuses where nobody would (usually) ever see them again, and you better believe it happened a lot.

Part of the problem was that there might be several hundred tickets open at any given time, and hardly anyone bothered to look at the queue that way. The glacial speed of the ticketing system at times didn't do much to encourage this kind of behavior. Indeed, I could only get so much done through it before giving up.

More often than not, I found the insanity by looking through the database directly for things which didn't seem right. I could see things which made no sense which, after some investigation, ultimately boiled down to massive human error.

I stumbled across a particularly messy anomaly one morning. There was a fairly Big Cheese type customer who had called in one evening with a report of trouble on their account. They had reached one of the "Big Cheese" second shift techs who then proceeded to mishandle the call by creating a ticket which would never be seen again.

Basically, this customer's account number was a five digit number. Let's say it was 98765. Instead of opening a ticket against that account and one or more of the computers or net devices on it, this brilliant tech opened one against computer number 98765. That particular server was associated with a completely different account which wasn't even a Big Cheese type customer. This other account was a relatively little affair.

Given that the ticket had been created for the wrong account, all sorts of badness happened. First of all, you're supposed to list the requester of a ticket when it's a phone call. That requester would then have to be one of the contacts on the account. Since this was the wrong account, there's no way the person on the phone could have been in the contact chooser dropdown list. Still, this tech managed to create the ticket by using himself as the requester, and just put some weird comment about "how it wouldn't let him choose (actual contact name)". Apparently this failed to alert him to a big error in the works.

What really surprises me about this whole thing is that he didn't notice things were amiss. When you get someone on the phone and they start talking about an account, you're supposed to pull up that account and then go into the contact name they claim to be. At that point you hit them with their secret question and do nothing else until they provide their secret answer. Easy, right?

The only thing I can figure is that he never actually authenticated that caller and so just barreled ahead into ticket creation. After all, if he had done the proper auth thing, he would have already been in the right account, and the ticket would have been too.

The final insult was that this happened to a customer who was probably #2 or #3 in terms of monthly billing. It was some fairly big problem with their multi-site config stuff, and it just sat there and rotted.

Interestingly enough, this broken ticket sat there and rotted through second shift and then someone on third pounced on it. They added a comment to the ticket which of course went to the wrong customer. That customer had no idea what any of this was about, naturally.

When I got in the next morning, I reached one of the Big Cheese first shift types via the internal company IM and told him about the situation. He then had the fun task of creating a new ticket to track things since the old one could not be salvaged. He also had to call up the customer and explain why their request had been rotting for the past 18 or so hours, unnoticed and unloved by multiple shifts.

If you really care about making things work, you need someone who can float around and look for this kind of insanity. They need the ability to dig around in more things than you might realize. It also helps if they are well-connected and know the people on other teams who actually get things done.

Big enough companies might already have someone like this lurking in the shadows doing this very job. It might be an entirely unofficial role they have taken on, perhaps spurred by an internal value like "having the ability means you have the responsibility".

My challenge for boss types is simple: see if you have someone like this who's putting out fires for you. Then make sure they realize you are aware of it and appreciate it. Then do something to make their life easier and give them appropriate powers and/or mandates to make things happen. They've probably seen a bunch of badness and might even have some useful data about patterns. Listen to them and act on it.

It's like free consulting services. You're already paying this person a salary. You just have to take them seriously.

Alternatively, do nothing. Mediocrity rarely kills companies, and being exceptional is difficult. It's far easier to just leave things alone.