I'd Rather Not Relive the Last 8 Hours

Sorry I haven't posted in while folks ... I'm soooo burried lately.  If you've sent me an email, please be patient.

Since I'm already sitting at my PC here at home I figured I'd writeup the events of the past 8 hrs ... get out your hankies! :-)

Here are a few back story items to keep in mind ...
My wife is out of town with several of her friends this weekend.
Ed is out of town with his wife on a couples retreat.

Sunday 2am-ish:  I awake to hear my cellphone indicating I have a text message.  The only text messages I get are either an alert that the power has been lost at church and our UPS systems are running on batt power OR that Postini, our offsite anti-spam/anti-virus solution, can't contact our email server.  I stumbled downstairs and read the message:  "unable to contact mail host".  Hmm, that's odd, but it's not a critical alert ... prob just a temp burp.  Back to bed I go.

Sunday ... 15 minutes later:  I get another text message.  This time Postini indicates that they've started spooling our email ... crud, this is not good.  I stumble downstairs to my PC ... crud, I can't VPN ... crud, I can't get to our OWA (outlook web access) either.  Nooooooooooooooooo!

Now my sleepy brain goes into panic mode ... what if our SBC connection at church is down?  What do I have to do to enable our backup ISP connection?  What if our backup is down also?  I start to perspire.  Ok brain, let's calm down and think this through.  It's possible our ISA firewall has had a glitch and just needs rebooted.  Any other time and I'd just hop in the car and drive to church ... but, having to dress-n-drag 2 sleepy kids with me isn't something I'd like to do.

So let's see if SBC is having problems.  GRRRR, their website sucks if you're trying to find a phone number to call to get support.  Since I cache all my GCC email at home and found my SBC docs and called up the 24hr support line.  Troy answers and I ask him to check and see if he can find any issues with their router at our location.  Troy has a good demeaner for talking my call at 2am.
He does several tests on the router and it all looks good.  He goes a step further and checks the ARP tables and can see traffic trying to get to internal addresses with no luck.  I thank him for his help.

Sunday 2:30am-ish:  So the good news is that it's not an ISP problem.  The bad news is that whatever the problem is will require a trip to the church.  Hmm, go now or wait until 7am??  I'm the worst morning person so I decide to bite the bullet and go now since I'm already up and probably won't be able to sleep knowing there's a problem

I go out and start up the minivan so it warms up ... luckily the weather isn't bad at all.  Aaron has some warm jammies on so I slap some shoes on him and carry him to the van.  Jacob is next ... man, how much does this kid weigh?!  I cover them with blankets and tell them to go to sleep.

Sunday 2:45am-ish:  We arrive at the church ... I pull the van up on the sidewalks and right to the front door.  The van is toasty warm so I kill the motor and let the kids sleep ... don't worry, I can see the van from just outside the room I'll be working in.  I open the tech room door and see no power lights from any of the computers on top of the audio rack.  After a few moments I conclude that the UPS powering our ISA firewall has failed...grrr.  Here's the kicker, I just bought a new UPS last week to replace it.  It's sitting right in front of me still in the box ... serenity now!!!!  For a moment I contemplate unboxing and setting up the new unit ... then laugh for thinking that at 3am.

So I route all the stuff on that UPS over to non-UPS powerstrips.  Power stuff on and waalaa things are working again ... a few moments later I get text messages from Postini that connection has been established and unspooling has begun.  Back to the van ... back home ... haul kids back up to bed ... dang, I'm out of shape! ... test VPN and OWA from home PC ... life is good and child check-in is ready to go for the 8:45am service.

Sunday 3:10am:  I lay down with a warm feeling that all is well in the world and look forward to sleeping in late.

Sunday 8:20am-ish:  I think I hear my cellphone ringing?  Once I actually awake I realize it's not a dream.  My cellphone, which is now right beside the bed, is saying I've got a vmail from Shelley, our First Impressions leader ... A HUGE deflating feeling comes over me ... a call from Shelley can only mean that something is wrong with the check-in PC's.  Noooooooooooooooooooo!

I give her a buzz and she explains the situation ... everything is fine except that no Sunday activities are available to select in the check-in app.  They've rebooted with no change in results.  Andy, our F1 guru, is out of town on his anniversary and thus should have his phone off...which he does.  Unfortunately, I don't know enough about F1 to know how to get into the activity setup area ... but all signs seem to point to the activities not being setup or somesuch.  I feel bad that I can't help, but tell Shelley I'm at a loss and starting thinking about plan B for check-in. 

I head down to my PC, VPN into the church and start testing ... everything is normal from a network standpoint.  By this time the 8:45am service has already begun ... luckily it's the lightest attended service we have.  However, the upcoming 10:15am service is almost always are highest attended ... yikes!  I finally remote control one of the checkin PC's (I should note that both boys are still asleep at this point) and when I launch the check-in app it won't take my activity code at all.  I call Shelley and let her know I'm going to check with Fellowship One and see if they can help.  I hit the F1 support page and see a red status indicater noting that they're having problems with activity codes...doh!  I update Shelley.  I send in a support ticket to F1 just to let them know our church is having issues as well as others.  Within a couple minutes I get a reply back that they are working quickly to resolve the issue.

I keep checking the check-in app every couple minutes and around 10:07am I'm able to get in and see the Sunday activities.  I quickly call Shelley and they've noticed the same and have started routing people from the backup handwritten system to the check-in PC's.  Better late than never :-)
A minute later I get an email from F1 support that things are back up and running.

By now my kids are up and asking for breakfast ... they say they kinda remember riding in the van :-)

So what did I learn from this?  Several things...but I'm tired and it'll have to wait for a later post :-)

Comments

[Sean Sperte]

"It's sitting right in front of me still in the box ... serenity now!!!!"

Now, that made me laugh.

If you're still looking for the lesson in all of this, one of 'em has got to be DON'T ALLOW SO MANY PIVOTAL PERSONNEL TO SKIP TOWN AT THE SAME TIME! Eesh.

[Matt Singley]

The lesson is actually this: don't let your wife leave town. ;) JK of course.

Man, that's a rough 8 hours. Agreed with Sean that it's hard to let so many key players go at the same time, but when anniversaries come, when vacations have a small window of opportunity, when the wife is ready to get a much needed rest...what do you do? Pray. That's about it! Hope for the best, prepare for the worst. Sorry you had such a rough weekend. At least we have a new episode of 24 tomorrow. :P

[Marcus Monroe]

Backup people perhaps.. always good to have second hand people to fill in spots.

[Jason Powell]

Backup people? ... man, don't tell my wife I need a backup for her!!!!!!!!!

Alas, the pivotal person in this senario is my wife. Perhaps the real lesson in this is that I need to appreciate her even more than I already do.

In the end there's really nothing any backup person could have done ... even if Ed was in town I would not have called him at 2am. He's a morning person so I would have waited until 7am IF I could not have gotten things up and running at 2.
And there's nothing a backup F1 person could have done either ... they would have called me anyway to ensure there wasn't some networking issue onsite. Of course they would have probably contacted F1 support sooner than I did and dicovered the root of the issue.

I've got a good list of things started for part II of this post...stay tuned.

[Ed Buford]

Everytime I go away...

[andy hosier]

Everytime I go away... (too)

[Marcus Monroe]

hmph, what I meant by back up people would be that for every person you have doing something, have a person who can feel that spot.

I would think that if you got a text, wouldn't ed get a text as well?

As for the F1 problem, that would be out of your hands. What if there were ways that the guy who does your F1 stuff was a little more inclined to defining if something was a network problem?

[Dan]

Another ironic thing...I was talking with Ed at the retreat about how he was afraid something would happen while he was gone. Ahh, irony... :-)

[Bob Brown]

You need to back up yourself. ;)
http://www.imdb.com/title/tt0117108/

[Jason Powell]

I'd probably end up with a clone like their 3rd clone ;-)

[Matt Wilson]

Same exact thing happened to me with the down Check-In system. It was a nightmare because I kept thinking it was us. Ruined our 8:30am service for check-in. But we were okay for 10:30am. Seems like it happened for every church running F1. I wonder what kind of chaos it caused at Lakewood Church with Joel Osteen. They process like 2000 kids on a Sunday!