The Disaster Disaster Recovery Plan

Mushroom CloudOnce upon a time, I worked for a company that put out a mandate of updating our disaster recovery plans.  Seeing as how most of the company didn’t have them, it really meant creating disaster recovery plans.  Due to a confluence of events, I happened to be the senior guy in our department who wasn’t a manager, and so it fell to me to craft our plans.

As with any good company, we did have some preparations in place even if we didn’t have a formal plan.  We had an off site backup data center which we could switch over to should a service at the primary site go offline.  And in that site we had an approximate copy of our primary site.  Approximate in that the intention of the backup site was to limp the company along until the primary site could be restored, not in that the backup could become the new primary.  So where the primary site had 3 servers doing a task, the backup site had 1, enough to do the job but not enough to do it without frustration.

So there began a series of meetings between departments as we began updating and creating our recovery plans, budgets were outlined and all the ducks were put in a row.  In one of these meetings, I noticed a flaw in one of the other department’s plans.  I brought it up, but since I’d only been there a year and was dealing with people who’d been with the company for 5, 10, even 25 years, I was ignored.  I was told, “You don’t know what you’re talking about.”  To that end, I decided to be sure I was right.  I began researching the other plans and looking for flaws.

My own department’s plan was simple: money.  We needed to spend the money to duplicate our server functions.  I called the companies we licensed software from and got the okay and licenses to maintain a fail-over site without spending any more money on licenses.  I wrote up hardware orders so that where we had 3 servers doing a task at the primary location we would have 3 servers waiting to take over if the primary failed.  My plan done, I had plenty of time to look at other people’s work.  So I did, and I made a nuisance of myself, sending emails and showing up for meetings I wasn’t invited to in an effort to actually make our company’s disaster recovery capable of recovery.  Eventually, I got told to stop.  The message was clear.  I was to focus on my own plan and leave everyone else alone.

I dove back into my disaster recovery plan.  You see, because of the flaws in the other plans, my original plan wouldn’t work.  It had to be redone.  I went to my boss and made one request.  On presentation day, I wanted to go last.

The day came to show off our new plans.  I sat in the back and waited through each department.  One by one they went to the podium and showed charts and laid out plans that illustrated they were well on their way to being ready, each department patting themselves on the back.  Finally, it was my turn.  I got up and started handing out my plan.  It was very short.  A cover sheet and then just two pieces of paper beneath that.  As I made my way through the room people began muttering to each other.  I got to the podium and said, “If you will turn to page one of the packet I’ve handed out you will clearly see the full extent of my disaster recovery plan.”

It was a copy of my resume.

“If the primary data center were to go offline, I would, in reaction to this disaster, begin sending out copies of my resume in an effort to find another job, because I certainly wouldn’t want to work here anymore.”  I could see my boss turning red with rage.  I could also see the managers for other departments shooting dirty looks at me.  Then I opened up my PowerPoint presentation.  I quickly showed the single page of my real disaster recovery plan: buy servers, install software, use extra license keys I’d already obtained.  Then I showed how my plan would still fail due to a flaw in Department X’s plan.  Then I showed that without fixing another flaw in Department Y’s plan, Departments A, B and C would fail.  And then I showed how Department M had overlooked a critical piece of hardware for which there was no backup and rendered everyone moot because the only working mainframe terminal in the backup site would be the one hooked directly to the mainframe.  Their plan actually had them unhooking a piece of equipment, loading it on a truck, and driving it nearly two thousand miles to the backup site, rather than actually purchasing a duplicate – probably because it was extremely expensive.  “So, as you can clearly see, my only reasonable course of action – since I was instructed not to involve myself in the affairs of other departments – is to find another job.”

The fallout from that meeting was huge.  First, I got yelled at.  Then, I got apologized to as they discovered I was right.  Eventually, new plans were drawn up and big money was spent, but our recovery plan was actually capable of recovering from disaster.  To date, that company has not had a disaster from which recovery was needed, but that’s not the point.  The point is that each and every department concerned themselves only with their own particular areas and no one had been assigned the task of looking at the places where they relied on another department.  Each one was happy to be able to say “We have a back up for our functions” and didn’t bother to examine if slot where their tab was supposed to insert was being covered, they just assumed it was someone else’s responsibility and it would be handled.

Since then, I’ve always tried to make sure I keep an eye on the big picture when I do things.  And I try to be open to suggestions and/or criticism from others on the off-chance that I’ve missed something big because I’m too close to it.  Outside that, there is no point to this story other than I just like to share it.

The Tools You Need

There is an old saying that goes something like this:

Give a man a fish and he’ll eat for a day. Teach a man to fish and he’ll eat for a lifetime.

Throughout my career in computers, I have often referred to various tasks as “handing out fish” or “fishing lessons” depending on how I decided to approach it.  In recent years, I have often leaned toward fishing lessons over handing out fish because I hate having to repeat work, though sometimes if a task will never need to be repeated obviously teaching is a waste.

More recently, I’ve come to realize that I enjoy working with people who can figure things out, where I don’t need to actually teach them, but I can just give them the tools they need and they’ll teach themselves.  This I have come to refer to as “giving them a pole and showing them the ocean.”  Of course, this technique does require that the person is familiar in the arena.  I couldn’t do this for a programming job with the guy who cuts my neighbor’s lawn, but if I’m talking to another programmer then he should already be familiar with enough basics that if I give them a language and a goal they should be able to fill in all the missing steps themselves.

Since I put this in the gaming category, how does this pertain?

I’ve always admired EVE Online, even if I didn’t overly enjoy the game, because the game itself is little more than a set of mechanics and some tutorials in how to use them.  If that is all the effort you even put into the game, EVE is shallow, bland, repetitive and boring.  However, using the tools of the game and the Internet (in the form of message boards and other bits) and stepping outside the safeguards of high security space, the players have crafted themselves a very deep game of social interactions and political intrigue that rivals the plots of many popular novels.  And it is a game you can’t teach.  You can’t be an EVE player and teach someone how to get involved in the machinations of the social entities.  You can only give them a pole and show them the ocean.

Seeing Redd

Back in March I read a book called The Looking Glass Wars.  I enjoyed it enough to pick up the sequel, Seeing Redd.

We are back in Wonderland, and its been… well, some short period of time since the end of the first book.  The book uses “lunar cycles” to measure time, but its unclear if a lunar cycle in Wonderland is the same as one on Earth, but I suspect it isn’t.  Here on Earth, a lunar cycle means from full moon to full moon (or new moon to new moon, if you wish), which runs about a month.  In Wonderland, I am guessing that its much shorter, perhaps that the author might in fact mean just one day, as is the moon comes up and then the moon goes down.  In some places the book says its been a full lunar cycle, and you can easily imagine a month having gone by, yet in other places when an urgent task is required to be done in two or three lunar cycles… well, would waiting two or three months really make the task urgent?

In any event, the book keeps with that same similar style from the first book where things are said but not explained, which is good that the book isn’t bogged down with explaining exactly what an orb generator or a scorpspitter is, and yet, given the way that I read books, constructing the world in my imagination, without details I’m left to fill in the gaps myself, or just to have gaps.  Filling in the gaps myself often leads me to having to revise my image of something later when a new detail does emerge, which then causes me a little confusion concerning earlier events…

Despite those issues (which might be my issues alone), the book still reads well.  It moves along at a good clip, hits the high parts and delivers a decent punch.  I enjoyed it.

Directed Instancing

One of the things I find discussed quite often is the use of instancing in games.  Some people like it, some people hate it, and there is every shade of gray in between.  The problem with no instancing is that places can get crowded… you go out to kill orcs and find that all the orcs are being killed.  The problem with too much instancing is that it can seem like you are the only one there… messing around in Guild Wars with their version of instancing, in the city there are people, then I go outside and am usually alone.

Just spit balling here… but what about a “directed instancing”, where you have an area that is an instance designed for a few dozen people/groups each unit performing different tasks.  The idea would be that before leaving “town”, you or your group has to select a task.  Once you leave town, the task is locked and you zone out into an instance that all but guarantees that no one is performing the same task you are, but may be populated with people performing any one of the couple dozen tasks the area supports.  You might head into the forest to kill orcs and run across a player chopping lumber, another hunting bears for meat and skins, a group searching for a missing child, and a raid about to siege the wizard tower at the far end.

Would that work?

Perfect Inefficiency

I dropped by the local branch of Bank of America last week to do some banking. While I was waiting for them to open at 9 AM, I sat in my car and listened to music and watched the road crew continue to work along side Satellite Blvd.

They’ve been digging up the side of the road for a while now, either laying some new cable or preparing to widen the road or something, I’m not sure. So there is a crew of six guys there. Five of the men are literally leaning on their shovels, while the sixth man operating a mini-backhoe is thrashing around trying to align himself correctly in order to use this machine to put dirt into a two foot hole. It took him over thirty minutes to complete the task… a task which the five guys with shovels could have completed in under two minutes.

I just don’t understand why people would want to deliberately work so inefficiently. The gasoline alone that it must have taken to operate that thing for over a half hour… It hurts to see my tax dollars wasted.

Prey

There is one thing Michael Crichton has always excelled at, and that is taking a real field of study or research, finding a disturbing way it could be abused, manipulated or get out of control, and then construct a story around it that manages to inform, thrill and scare you all at the same time.

Prey is no different. The subject here is nanotechnology, small molecule sized machines programmed to work together to perform a task. The angle, what happens when the machines are coded with animal behavior logic that is intended to be open-ended and self-optimizing. In other words, the program makes decisions and learns from its mistakes.

The book was a pretty fast read, like most of Crichton’s books, and there is enough technojargon to give the book that science filled feel to it, but it never goes so far as to obscure the story from the read in mumbo jumbo that only learned people in the know would understand.

In short, the book didn’t suck.

The Definition of Insanity

The clinical definition of insanity is the repetion of the same task expecting a different result. Like, if you have a button that turns on a red light, pushing it over and over again expecting one of those times for the light to be green instead.

So, at work, there is a woman who asks me to create an entry in our database. She does this about once a week. Every week she provides me with the name of the record. Every week, I ask her to send me more information and list about twenty fields that need to be filled out. Every week she replies with the information I need and says she forgot.

The question is, is she insane because she keeps sending me one piece of data expecting me to be able to create the record or am I insane for expecting her to learn and give me all the information in the first email?

Goals

Sometimes one of the best things to do in online games in to set goals for yourself. And I don’t mean strictly the ones the designers lay out for you. Levels, items, zones, quests… they’re all decent goals, but often times they wind up being diminished by the fact that so many other people have done them.

So, with that in mind, in every game I play I always find little things to task myself with. The most recent one is in City of Heroes. I’m not playing CoH as much as I used to… heck, I’m not playing any game as much as I used to, but my time is currently divided between World of Warcraft and City of Heroes… WoW when Jodi is around, and CoH when its just me. Anyway, in CoH I’m playing Ishiro Takagi, a scrapper, martial arts and regeneration. Given enemies his own level, he’s fairly unbeatable. It takes guys 2 or 3 levels higher or groups of 6 or more to put him in the dirt. Ishiro is level 26, and currently prowling the streets and rooftops of Striga Isle, Talos Island, Dark Astoria and Independance Port… I go into Terra Volta every now and then, and when I’m sent to the hospital I remember why I don’t go in there alone, heh.

In Independance Port there is a bridge. Its long and covered in bad guys. I start at the north end and begin wading through. I haven’t made it yet… either I run into a boss who cleans my clock, or an afore mentioned group of 6 or more who I just can’t thin out before they get the upper hands. As I level it will get easier, and that’s sorta the point. My goal isn’t levelling, its something else, which if I keep doing it will lead to levelling, but in and of itself is far more satisfying than watching the experience bar fill up… instead I watch my progress across the bridge.

I’m halfway there.

Soon.

Trade Skills: The Good and the Bad

So today I decided to try out the trade skills of Dark Age of Camelot. I’ve been familiar with the old UO style, and the EQ style, and never bothered AC (are there trade skills in AC? I never cared to stay long enough to find out). What follows is, as always, my opinion.

We’ll begin with how to do them, and I’ll stick with tailoring since its what I chose to do. I talk to the Tailoring Master and say I want to be a tailor. He says, “Great!”, and I get a skill point. I buy a tailoring kit and some supplies: wool thread, leather, cloth. Deciding ahead of time that skilling up on your own just costs money you never get back, I take the option of doing consignments.

For those who don’t know, you can make stuff in DAoC and sell it to a vender at low levels of skill for anywhere from 50% to 95% of the cost of making the item (barring failures), OR you can have the master of your skill task you to make something and deliver it for him in which case you average about 120% of the cost of making the item. Yes, that’s right EQ fans, you make a PROFIT!

So I start asking what he wants me to make, I make it, and I deliver it. I started with 13 silver pieces and today I ended with 12 silver pieces. I know what you are thinking, “Didn’t he say profit?” Yeah, I did. The last task I got asked to make was leather boned gauntlets which require bronze studs. I had to pay 3 silver for 20 bronze bars to make 20 bronze studs so that I could use 1 stud for the gauntlets (yep, you buy things in amounts of 20 from what I have seen so far). So in reality I had 15 silver before I bought the bronze bars, and I haven’t even turned in the gauntlets for my reward yet. Pretty neat, eh?

On top of that, the trade skills are a tree. Tailoring isn’t just tailoring. Tailoring is just the act of making the final product. There are sub-skills like clothworking, leatherworking, and metalworking, and I earn skill in those as well depending on the content of the item I am crafting.

There are two annoying nit picky bits with trade skills though. First, when doing consignments, you have to find the people. You start to cheer and shout out loud when you get a repeat customer because you already found them. Finding people is made easier though because you can ask any guard where to find them and they will literally point the way. Of course, nothing is fool proof and they will point directly to the person, so all I need now is the ability to walk through walls. Second, at least in EQ, you hit combine and you know immediately if you failed or succeeded. In DAoC you get a progress bar. Now you and I both know full well that the game knows before you even start to display the bar that the item was a success or not, making me wait 10 second just to tell me I have to try again is a pain in the ass, especially when you end up having to try 8 or 10 times before you succeed. But then again, not having to physically put items into my sewing kit is nice, so is not having everything automatically eaten on failures.

In the end, DAoC has a more well thought out Trade Skill system. That’s plain as day. However, all EQ would need to do in order to catch up again would be to put in Trade Skill Quests where you are given a no drop note with an order on it, make the item, then return the item and note for money. Oh, that and venders where you can store buy more materials for skilling up, hunting rare supplies is a bitch.