Disaster Recovery ... what's your plan?

With the tragedy in South it got me thinking recently about DR plans (disaster recovery). 

There are hundreds of churches that just got demolished and my heart goes out to them.  It didn't hit me until today that I'd wager almost all of them had no IT DR plan.  How can I say that?  I've yet to talk to a smaller sized church that has a good backup system in place let alone any DR plans.  Their membership database is on the secretaries PC, their financial records are on the Finance Directors PC, etc.  There might be a floppy disk at someone's home with a database backup ... and we all know how reliable floppies are.  My hope is that with the advanced warning, churches got their critical data off to a safe place.

Does your church have a DR plan?  Block out a day this week to focus on creating one ... seriously, do it!  Sit down and think absolute worst case senarios ... if you came back to work tomorrow and your building was burnt to the ground what would you do?  Where's your critical data?  How would you get systems up and running? Don't forget about phones.  Ministry still has to occur even if your computers/servers are black and toasty.  In this age, staff rely on technology so if you're responsible for your churches IT you must have a DR plan regardless of church size.

At Granger we're immune to hurricanes and flooding, but tornados are a real threat as well as fire.  Our DR plan isn't by any means perfect, but we have a plan and periodically we tweak on it and test it.  I'll go over our DR plan in several installments over the next few weeks lest this be a flippin' hugely long post.  ;-)

PART 1)  Power Outages - suddenly the power goes out in your building!  Yup, though it seems very simple, this is part of your DR plan.  Servers should never, never go through a hard shutdown.  Make sure your critical equipment is plugged into a good UPS (Uniterruptable Power Supply).  We use APC UPS products.  Great, so now your servers are on battery backup.  What happens when the batts tank?  A good UPS unit will have an interface to connect to a server/PC.  This allows you to schedule what happens when the battery reaches a certain level.  So when the batt hits say 10% the server will shut itself down properly.

Last week Ed and I actually tested part of our power outage strategy.  We yanked the plugs out of wall for the server rack and waited.  The main UPS picked up without any hestitation.  It's a 2200VA unit powering 5 servers, 1 switch, 1 monitor, and 1 DLT tape drive.  Within moments of going on batt power Ed and I both received Nextel text messages and emails telling us the servers were running on batt power.  Our main server is connected to the UPS unit and we have 2 events set in the power management utility.  Upon going on batt power a simple script runs which sends the messages (assuming our ISP's are not down). 

Once the batt reaches 20% remaining capacity (took about 12 mins) a second script runs sending us messages that the servers are shutting down, then the script tells all the servers to power down.  One by one the servers do a gracefull shutdown until the rack is ghostly quiet and all blinking lights are dark.  So should we have a power outage for longer than 10 mins it's very comforting to know that the servers will shut themselves down nicely and send a little message that they are doing so.

We didn't get around to testing the IDF which has our ISA box, but it's setup pretty much the same.  We also didn't test all the switches from the server rack to the CSU/DSU which must be up if we are to get the text messages...not critical, but it's certainly nice to have to get the heads up if we're offsite.

We'll be budgeting for more UPS power next year...I'd like to have 30 mins of runtime for the servers.  I'd like to also have UPS units for our checkin PC's so in the event of power outage you could still run child checkin ... again assuming our ISP's are still available.

So there's Part 1 of our DR plan.  Stay tuned for more...

Comments

[Stuart Cowen]

Good post! I've just inherited a church infrastructure that was a Johnson & Johnson Medical complex until 3 years ago and they desperately need a DR Plan. One thing I would say is that as soon as the UPS goes to battery power, have everything shutdown immediately. I used to wait until the 10% or 20% mark, but then a year goes by without an outage, or I've added more devices to the battery pack and suddenly my shutdown window at 20% is much shorter.

Blog on, Brother!

[Jason Powell]

That's why we test it periodically ... twice a year at min. We used to wait until 10%, but found it just barely left enough time to get everything powered down.
Power goes out here more than any other place I've ever been, so we leave some time before everything starts shutting down.
I just opened my latest issue of Windows IT Pro magazine and wouldn't you know it ... they have a article on DR and steps to create a DR plan. Talk about timing :-)