Sorry I haven't posted in while folks ... I'm soooo burried lately. If you've sent me an email, please be patient.
Since I'm already sitting at my PC here at home I figured I'd writeup the events of the past 8 hrs ... get out your hankies! :-)
Here are a few back story items to keep in mind ...
My wife is out of town with several of her friends this weekend.
Ed is out of town with his wife on a couples retreat.
Sunday 2am-ish: I awake to hear my cellphone indicating I have a text message. The only text messages I get are either an alert that the power has been lost at church and our UPS systems are running on batt power OR that Postini, our offsite anti-spam/anti-virus solution, can't contact our email server. I stumbled downstairs and read the message: "unable to contact mail host". Hmm, that's odd, but it's not a critical alert ... prob just a temp burp. Back to bed I go.
Sunday ... 15 minutes later: I get another text message. This time Postini indicates that they've started spooling our email ... crud, this is not good. I stumble downstairs to my PC ... crud, I can't VPN ... crud, I can't get to our OWA (outlook web access) either. Nooooooooooooooooo!
Now my sleepy brain goes into panic mode ... what if our SBC connection at church is down? What do I have to do to enable our backup ISP connection? What if our backup is down also? I start to perspire. Ok brain, let's calm down and think this through. It's possible our ISA firewall has had a glitch and just needs rebooted. Any other time and I'd just hop in the car and drive to church ... but, having to dress-n-drag 2 sleepy kids with me isn't something I'd like to do.
So let's see if SBC is having problems. GRRRR, their website sucks if you're trying to find a phone number to call to get support. Since I cache all my GCC email at home and found my SBC docs and called up the 24hr support line. Troy answers and I ask him to check and see if he can find any issues with their router at our location. Troy has a good demeaner for talking my call at 2am.
He does several tests on the router and it all looks good. He goes a step further and checks the ARP tables and can see traffic trying to get to internal addresses with no luck. I thank him for his help.
Sunday 2:30am-ish: So the good news is that it's not an ISP problem. The bad news is that whatever the problem is will require a trip to the church. Hmm, go now or wait until 7am?? I'm the worst morning person so I decide to bite the bullet and go now since I'm already up and probably won't be able to sleep knowing there's a problem
I go out and start up the minivan so it warms up ... luckily the weather isn't bad at all. Aaron has some warm jammies on so I slap some shoes on him and carry him to the van. Jacob is next ... man, how much does this kid weigh?! I cover them with blankets and tell them to go to sleep.
Sunday 2:45am-ish: We arrive at the church ... I pull the van up on the sidewalks and right to the front door. The van is toasty warm so I kill the motor and let the kids sleep ... don't worry, I can see the van from just outside the room I'll be working in. I open the tech room door and see no power lights from any of the computers on top of the audio rack. After a few moments I conclude that the UPS powering our ISA firewall has failed...grrr. Here's the kicker, I just bought a new UPS last week to replace it. It's sitting right in front of me still in the box ... serenity now!!!! For a moment I contemplate unboxing and setting up the new unit ... then laugh for thinking that at 3am.
So I route all the stuff on that UPS over to non-UPS powerstrips. Power stuff on and waalaa things are working again ... a few moments later I get text messages from Postini that connection has been established and unspooling has begun. Back to the van ... back home ... haul kids back up to bed ... dang, I'm out of shape! ... test VPN and OWA from home PC ... life is good and child check-in is ready to go for the 8:45am service.
Sunday 3:10am: I lay down with a warm feeling that all is well in the world and look forward to sleeping in late.
Sunday 8:20am-ish: I think I hear my cellphone ringing? Once I actually awake I realize it's not a dream. My cellphone, which is now right beside the bed, is saying I've got a vmail from Shelley, our First Impressions leader ... A HUGE deflating feeling comes over me ... a call from Shelley can only mean that something is wrong with the check-in PC's. Noooooooooooooooooooo!
I give her a buzz and she explains the situation ... everything is fine except that no Sunday activities are available to select in the check-in app. They've rebooted with no change in results. Andy, our F1 guru, is out of town on his anniversary and thus should have his phone off...which he does. Unfortunately, I don't know enough about F1 to know how to get into the activity setup area ... but all signs seem to point to the activities not being setup or somesuch. I feel bad that I can't help, but tell Shelley I'm at a loss and starting thinking about plan B for check-in.
I head down to my PC, VPN into the church and start testing ... everything is normal from a network standpoint. By this time the 8:45am service has already begun ... luckily it's the lightest attended service we have. However, the upcoming 10:15am service is almost always are highest attended ... yikes! I finally remote control one of the checkin PC's (I should note that both boys are still asleep at this point) and when I launch the check-in app it won't take my activity code at all. I call Shelley and let her know I'm going to check with Fellowship One and see if they can help. I hit the F1 support page and see a red status indicater noting that they're having problems with activity codes...doh! I update Shelley. I send in a support ticket to F1 just to let them know our church is having issues as well as others. Within a couple minutes I get a reply back that they are working quickly to resolve the issue.
I keep checking the check-in app every couple minutes and around 10:07am I'm able to get in and see the Sunday activities. I quickly call Shelley and they've noticed the same and have started routing people from the backup handwritten system to the check-in PC's. Better late than never :-)
A minute later I get an email from F1 support that things are back up and running.
By now my kids are up and asking for breakfast ... they say they kinda remember riding in the van :-)
So what did I learn from this? Several things...but I'm tired and it'll have to wait for a later post :-)
Recent Comments