Author: Jason

Currently, I deal with two companies on a regular basis to whom which we pay a lot of money.  One of them, a co-location facility, is fantastic at communication.  Every time they have scheduled maintenance or even unplanned issues, I get an email.  In fact, I get several.  For a scheduled issue I get an announcement, a reminder, a notification of start, a progress update if the issue is long enough, more if the issue keeps going, a notification of resolution and an after action report.  For unscheduled issues it’s the same only without the lead time.  I even get these emails when the issue isn’t going to affect us, just so I know what is going on and on the off chance that is does end up affecting us.

The other, to whom we pay much more money and they handle out internet and phones, we get nothing.  Even when we have outages, getting an explanation from them is like pulling teeth.  Our internet access, which is needed for our customers to use our products, goes down for twenty minutes, or bounces up and down for several hours (down for 30 seconds, up for 3 minutes, down for 20 second, up for a minute, down for a minute, up for 5, etc..) and the best I get from them is “we’ll look into it”.

Having worked in the telecom industry, and from dealing with various companies over the years, I know that all this equipment is monitored and logged.  Someone, somewhere had to have gotten notification of the problem, so why didn’t they toss out an email that said, “Hey, we are seeing issues in XXX area and some of our customers may be affected. We are working on it!” and later a nice “We found the problem and it was resolved!”  I mean, shit, the co-location facility provides an after action report with more detail than I know what to do with…

The failure was in the 3rd card of the router in room 3E1 that caused packet loss to the network that exceeded the threshold and switched the room to its backup route at the same time room 3E2 was being fault tested which overloaded the network and crashed it.  Our customers in 3E2 were immediately switched to their primary route experiencing only a few seconds of outage but room 3E1 was offline for 63 seconds while the backup route was restarted.  The 3rd card of the router in 3E1 was replaced and the room was placed back on its primary route within one hour.  We have scheduled a replacement of parts of the backup route for tomorrow to ensure it doesn’t crash again when it is needed.  Next week we will begin a full equipment test of all routes that should take two weeks to complete to ensure this same issue doesn’t happen to the other rooms not affected at all by this issue.

All of this comes without me ever once having to call anyone.  Meanwhile, our telecom tells us:

We think there was an outage in XXX, which shouldn’t affect you but might have, and since your service is up now I’m going to close the ticket.

And this is after calling to open a ticket, waiting three hours, escalating through four levels of management and threatening to cancel our service and go with another provider.  I wish I could say that my threat was more than empty, but as I said, I used to work in telecom, and this is par for the course.  No matter who we switch to we’d get the same level of information about failures out of them.

Seriously guys… if there was a line out, or an equipment failure, or someone crossed a line or rebooted something they shouldn’t have, just admit it.  Tell me exactly what it is so that I understand that you actually know what happened, because your current level of non-communication only leads me to believe that you have no idea what happened.

This applies to just about every business in the world.  Admit fault, explain failure and detail resolution.  Even if it all goes over your customer’s head, you have at least demonstrated that you know what you are doing.  Hand waving and secrecy just makes your customers have less faith in your abilities.  If your team is competent, you should want to show it off.