Tag: failure

Tell Your Customers

Random Thoughts
We (don't) Care!
Don't let this be your motto!

Currently, I deal with two companies on a regular basis to whom which we pay a lot of money.  One of them, a co-location facility, is fantastic at communication.  Every time they have scheduled maintenance or even unplanned issues, I get an email.  In fact, I get several.  For a scheduled issue I get an announcement, a reminder, a notification of start, a progress update if the issue is long enough, more if the issue keeps going, a notification of resolution and an after action report.  For unscheduled issues it’s the same only without the lead time.  I even get these emails when the issue isn’t going to affect us, just so I know what is going on and on the off chance that is does end up affecting us.

The other, to whom we pay much more money and they handle out internet and phones, we get nothing.  Even when we have outages, getting an explanation from them is like pulling teeth.  Our internet access, which is needed for our customers to use our products, goes down for twenty minutes, or bounces up and down for several hours (down for 30 seconds, up for 3 minutes, down for 20 second, up for a minute, down for a minute, up for 5, etc..) and the best I get from them is “we’ll look into it”.

Having worked in the telecom industry, and from dealing with various companies over the years, I know that all this equipment is monitored and logged.  Someone, somewhere had to have gotten notification of the problem, so why didn’t they toss out an email that said, “Hey, we are seeing issues in XXX area and some of our customers may be affected. We are working on it!” and later a nice “We found the problem and it was resolved!”  I mean, shit, the co-location facility provides an after action report with more detail than I know what to do with…

The failure was in the 3rd card of the router in room 3E1 that caused packet loss to the network that exceeded the threshold and switched the room to its backup route at the same time room 3E2 was being fault tested which overloaded the network and crashed it.  Our customers in 3E2 were immediately switched to their primary route experiencing only a few seconds of outage but room 3E1 was offline for 63 seconds while the backup route was restarted.  The 3rd card of the router in 3E1 was replaced and the room was placed back on its primary route within one hour.  We have scheduled a replacement of parts of the backup route for tomorrow to ensure it doesn’t crash again when it is needed.  Next week we will begin a full equipment test of all routes that should take two weeks to complete to ensure this same issue doesn’t happen to the other rooms not affected at all by this issue.

All of this comes without me ever once having to call anyone.  Meanwhile, our telecom tells us:

We think there was an outage in XXX, which shouldn’t affect you but might have, and since your service is up now I’m going to close the ticket.

And this is after calling to open a ticket, waiting three hours, escalating through four levels of management and threatening to cancel our service and go with another provider.  I wish I could say that my threat was more than empty, but as I said, I used to work in telecom, and this is par for the course.  No matter who we switch to we’d get the same level of information about failures out of them.

Seriously guys… if there was a line out, or an equipment failure, or someone crossed a line or rebooted something they shouldn’t have, just admit it.  Tell me exactly what it is so that I understand that you actually know what happened, because your current level of non-communication only leads me to believe that you have no idea what happened.

This applies to just about every business in the world.  Admit fault, explain failure and detail resolution.  Even if it all goes over your customer’s head, you have at least demonstrated that you know what you are doing.  Hand waving and secrecy just makes your customers have less faith in your abilities.  If your team is competent, you should want to show it off.

Defining Failure

Gaming

If you travel in the gaming blog circles, you might have heard about or even read a little anonymous diatribe about Warhammer Online.  And there are responses.  One.  Two.  Three.  Four.  Five.  Six.  And I’m sure there are more…  But really, I don’t want to talk about that.  Instead, let’s talk about what constitutes a failure in the MMO world.

I’ve seen a number of places, in comments on the above linked posts and all around the Internet that Warhammer “failed”.  However, they sold 1.2 million boxes, which I have to assume covered a good bit, if not all of the development costs.  We know they bled subscribers, and the last official numbers were that they had 300,000 subscribers as of March 2009.  They have cut back on servers, down to 9 (4 US, 2 UK, 2 German, 1 French) and are most certainly down below the reported 300k.  Still… if we can assume that the box sales and the first couple of months recouped the development investment, and if the current operating costs are below their subscription revenue, while the returns for the investors aren’t good, is an MMO operating in the black a failure?  I mean, they haven’t shut the game off yet, so I kinda have to assume they operate in the black, or damn close to it.  I could be wrong.

Clearly, the game did not perform as well as people would have hoped.  They didn’t make WoW-style money hats to wear while driving dump trucks of money to the bank, but did they lose money?  Is the game bleeding cash?  Each perspective on a game defines failure in different ways.  An investor, for example, will define failure as earning less money than other, less risky, options.  If he earned less on his cash than he would have just putting it in a savings account, then it’s an epic failure.  A publisher or game company probably defines success or failure on the affect the game has on both the bottom line AND the company reputation.  If a game is making money but the press keeps bringing the game up as being crap or failing, then overall the game is probably a failure since it might affect getting future investors to give you their money.

For me, as a player and a wanna-be developer, success means the game runs and I, as a player, can play it and we, as the developers, are still able to release more content.  Failure exists only when the game is in the red and we have to shut it off to keep from bankrupting everyone involved.

How many MMOs have truly failed?  Asheron’s Call 2, The Matrix Online, Tabula Rasa, APB, Motor City Online, The Sims Online… Are there more bodies in the MMO graveyard?  How do you define failure?