Jason Powell - Church IT and Other Musings

Indiana Time Zone change part 4

Well it's over.  Our PC's are all now on Eastern Standard Time and our users didn't have to do anything to their PC's or calendars ... a huge success in my book.

We did have 1 user that the calendar script would not fix ... for whatever reason the script would not run against her mailbox.  1 out of 135 mailboxes ... not bad at all!

We did end up manually updating the time zone on each PC via remote desktop and DameWare.  We used a time zone change script and ran it in a user logon script (we never got a machine logon script to work correctly) ... so all we had to do was log on then back off each PC.  Took 3 of us about a hour to hit all the boxes (thanks for the help Dustin!).  It was also a good time to prune Active Directory of domain machines that no longer exist.  I pulled away from the church at 2:12am ... we could have went home much earlier, but we all felt pretty peppy and pushed on.

I learned that Domain Controllers don't like it when you change their time zone.  We had a moment of panic when both DC's stopped replicating after zone change.  Reboot them both and their happy again.

We also discovered more C: drives nearly full ... when a P4 2.8GHz box with 1GB ram takes a while to boot up you know something's wrong.  Disk quotas are on the horizon.

The last 2 weeks have been overwhelming.  I'm just so glad to have this time zone crap behind us.

Indiana Time Zone change part 3

Just got an email from one of our volunteers that works in the IT department for a local hospital.  Tuesday night he was with us at GCC as we did testing on the calendar issue surrounding Indiana's time zone change.

He asked the guys in charge of their email system if they had a fix yet for the calendar issue.  They basically told him there will be no problem.  He tried again to raise some red flags, but was ignored.

Now that they changed all the PC's to Eastern Standard Time their help desk is flooded with people asking why all their appointments are off by an hour.  Now their IT dept has to scramble to find a fix.

This is pretty unsettling IMHO.  If a hospital missed the fact that this calendar issue was going to occur, how many other organizations across Indiana are going to have jacked up calendars over the next couple days.  Also makes me think what other time zone based systems are going to be effected ... yep, Indiana's own little Y2K :-)

What I still don't get is the fact that nobody seems to be addressing this issue ... and thus many organizations will have a rude surprise when they shift to EST.

Tonight at 9pm we start our migration to EST.  First we have to run the Purdue script against all 135 exchange mailboxes.  Then change the time zones on the DC's.  Then change the zones on all 150-ish PC's.  So far we have not been able to get Microsoft's hotfix, that is supposed to switch all the PC's for us, to work.  We have another script that's supposed to do the same thing, but of course we can't get that script to run either through logon...grrr.  So if needed our last resort is to just log onto each box remotely and change the zone manually ... ~ 45s per box won't be too bad.  2 of our volunteers have said they'll come in to help ... so 150 PC's divided by 4 should go rather quick should we have to go manual.

Indiana Time Zone Calendar Fix

If you read my last post on this you'll recall I was pretty bummed Microsoft did not offer any tools to fix the upcoming Indiana time zone change and it's effect on Outlook calendars...in fact they said you had to do it manually...uggh!  But alas!!  I think we found a winner ... through a series of contacts we were put in touch with Jeff Ello from Purdue University who wrote several scripts to "fix" the calendar issue people will face when changing their computers from Indiana Time Zone to Eastern Time Zone.  We've run the scripts in our test domain and on a few test accounts in our local domain ... so far so good!  Thursday night starting around 9pm we're running the script and changing the time zone on all PC's ... I'll post more details later, but wanted to get the below info posted ASAP...

I contacted Jeff and he sent me the following to post here in hopes it will be of help for others.

Attached are some server-side VBScripts for Exchange 2003 (2000 was not
tested) that effectively change appointments created in the Indiana Time
Zone to Eastern Time Zone.  It fixes both the time zone tags and start times
where appropriate.  Recurring meetings are dealt with properly, without
creating exceptions.  In fact, recurrences and exceptions adjust
automatically.  It also fixes single occurrence appointments for Apr 2 - Oct
29, 2006 ONLY.  All fixed meetings have their subject lines appended with
the original time and date of the item.

There are additional scripts for fixing Public folders, rolling back
changes, and for the situation where you may have already decided to go the
manual fix route.

These scripts were very hastily developed over the last two weeks and
deployed in a 9000 user environment, a 600 user environment, and most
recently, several 50-200 user environments.  The smaller environments
executed the script in bulk without any trouble.  The 9000 user environment
has some some difficulties, with some accounts missed in the first pass, and
some having an "unexpected error" with the script exiting early.  As to what
is causing that it could be anything from corruption in the database to any
odd behavior that the script or exoledb provider doesn't like.  So be aware
of this and test test test.... though, again, it appears smaller
environments seem to have fewer problems with the implementation.

IMPORTANT!  I provide these scripts for educational purposes only.  Use at
your own risk.  You assume all responsibility for any damage caused by their
usage.  Please watch the video and read all materials before doing anything.
I wish I could provide more support but I just don't have that kind of time.
Please test test test until you can't test any more before using ANY
solution on a live server.  I have little doubt that there are situations
which were not taken into account when writing these.  There is a readme
file and a number of additional notes in the headers of each script.

Demo Video
http://www.massiveelements.com/dstfixerdemo.wmv

Scripts
http://www.massiveelements.com/IndianaToEasternDST_for_Exch2003_v_3e.zip

Jeff Ello

Downloading at 8Mbps

Comcast dude just left my house so after activiting my account I immediately went to DSL reports and ran speed tests.

While I'm not happy with my upload speeds (should be close to 1Mbps), my download speed is just sick!

now compare that to my DSL "pro" speeds

I'll call tech support tomorrow and gripe about my upload speed.  Now to call SBC and tell them to cancel my overpriced and slow DSL connection.

Week in Review

A few tidbits from this week...as they spring forth out of me brain

Upgraded to SP2 on our ISA2004 box Tuesday night ... turns out it breaks the iTunes music store connection.  Several users were complaining they couldn't get their "fix" of new tunes the next day :-)
Of course my first reaction is ... well, we don't officially support iTunes ... but we're nice and this afternoon Ed finally googled and found the resolution.

I haven't posted about Yahoo Music Unlimited in a while ... they released the latest software version 2 months ago and it was a huge improvement.  I'm also quite pleased that Ed and I have been able to convince our musicians that Yahoo Music is the way to go.  They're all using it now :-)
Of course the puzzling part is that once they find stuff they really really want by browsing Yahoo ... they buy it from iTunes.  Why?  I have no clue.  Dustin? Dan??

I've been testing Winternals Enterprise Defrag Manager for a month now (trial ends today).  I have to say I'm very impressed.  It's fully Active Directory aware so creating groups of computers/servers with set defrag schedules and configurations was a snap.  It's defrag performance is much better than XP's built-in defrag ... and it's fast.  You can get detailed reports on each machine before/after defrags ... which led me to discover that many of our users C drives have little free space left!  Most all our PC's have 40GB drives ... so where has all that space gone considering that their My Documents sits on the servers? ... yup, music files.  I'm partly happy because we've asked people to move their iTunes off the servers and onto the C drive, but now we're running into space issues on the servers and local drives!  XP is NOT happy when there is little drive space left.  Even with almost no free space, Defrag Manager is able to do it's job whereas you must have 15% free to use XP's defrag.
So I'm contemplating if we need to block the saving of music files on the C drive and in essence require people to buy thier own USB drives to store music.  A few of our heavy iTunes users have already done this, but I forsee a coming need to do something across the board.  Where are your users storing their music files?

It's hard to believe but I actually programmed one of our VCR's to record The Apprentice this season since it overlaps with 24 which is Tivo'd.  I'm keeping an eye out for another cheap Tivo to stick up in our bedroom to record overlaps and allow us to watch stuff from the giant drive Tivo downstairs.  Anyone else running 2 Tivo units?  How long do you have to wait to watch a show streamed from one Tivo to another?

Brian Glass made his 3rd trek to GCC Monday this time for the Getting A Grip On Groups workshop.  During one of their breaks I gave him a quick backstage tour of some of the new AV gear.  Sadly we didn't get much time to hang out and talk shop this time 'round. 

That new UPS unit that would have kept me from 3am unhappiness is now up and running...grumble grumble

Finally got our first look at Server2003 R2 courtesy our TechNet subscription ... I'll have to post more on this later, but after seeing several of the new features I'm sold and we will be upgrading at least all fileservers to R2.

Comcast is now offering their upper level 8Mb down/1Mb up service for $29.99/mo to new subscribers ... even after the introductory time is over it's only $52/mo which is what I'm already dishin' out for my not very fast SBC DSL.  The online signup was simple and effortless AND someone will be here Tuesday morning to hook it all up...wow, I figured I'd have to wait at least a couple weeks.  I'm betting they'll have to run a new cable to the house as the current one is old and has at least one splice where my wife "discovered" it under her flower garden.
I don't really need the 8Mb down speed, but the 1Mb up will be helpful as we're looking to test offsite replication between my home and GCC.  It sure will stink the day I need something like a frame relay 3Mb up/down pipe at my home for offsite replication/disaster recovery :-)
The amazing thing is that this is not that far fetched.  I can easily forsee a small rack in my basement with a VMware server that is replicating most if not all the data and services at the church.  If the church burns down everyone can come over to my house and continue to work.  Of course I think I'll need a few more PC's and more than a 4 port switch :-)  We already do daily offsite ftp's of some of our critical data so this would just be a natural progression.  Here's where having more than one building in your organization would be a big plus.

Easter is right around the corner ... we could have 10,000 people attend this Easter so I'm working with Andy to get more check-in units ordered and operational.

Daylight Savings is next Sunday!  Monday and Tuesday we're ignoring all but critical issues to focus on testing Microsoft's Indiana hotfix.  Here's where having our test domain network is going to pay large dividends!

Since I most likely will not post again for quite some time I'd encourage you to check out the archives.  There's now 305 posts for you to chose from ... fun for the whole family!

Dell DataSafe

Dell now has a great new feature called DataSafe for their Dimension and XPS systems.  Basically they use Ghost, Intel Matrix Storage Technology, and a 2nd hard drive to create a not only a mirrored drive setup but also periodic drive snapshots.  So if a drive fails ... you're still up and running.  If you majorly corrupt some files are delete something important ... restore a prior snapshot and you're back up and running.

This sounds really great and the cost isn't bad at all.  To add DataSafe to a PC with 80GB drive was only $99.  $99 gets you the additional 80GB drive and Ghost licensing and everything is setup for you.  For home users and small offices, this is da bomb.  I just ordered a Dimension for an after-hours client with DataSafe and if it's as sweet as it sounds I'll be recommending it to everyone.  I've been wondering when we'd start seeing RAID on desktop motherboards ... it's finally here and affordable.  Check out the details on Intel's Matrix Storage Technology.

Speaking of Dell, check out the price I just got for this system for a client ...
Dell Dimension 5150 (intended props to Van Halen?), Pentium D 2.8GHz (dual core), 1GB ram, 80GB drive with DataSafe, 2yr on-site warranty, 17" flatpanel, 16x CD/DVD burner ... before tax and with free shipping $758!  That's nuts.

Indiana's Y2K is coming...

It took 'em long enough, but Microsoft has finally (yesterday) released the hotfix for Indiana's upcoming change to daylight savings time.

On April 2, 2006, Indiana will start using daylight saving time. Most counties in Indiana will use the Eastern Time zone. However, some counties will use the Central Time zone.

Microsoft Windows users in Indiana will have to change their Windows time zone settings from Indiana (East) to either Eastern Time (US & Canada) or Central Time (US & Canada). The Indiana (East) time zone setting will become obsolete and should not be used in the future.

After reading through the lengthy instructions I'm not pleased that this appears to be a not so simple ordeal...grrr, like we're not already busy enough.

The "fix" goes something like this ... Run around and change the timezone by hand on all 150 PC's (shaa right, and monkies might fly out my butt) ...OR... call Microsoft to get access to the hotfix download, create a custom script to deploy the hotfix (5 steps), create a Group Policy to run the script (11 steps), change the servers first, workstations second, instruct users on how to correct their calendars/events/meetings and correct OWA (outlook web access).

So come April 2nd all our Outlook calendar events will move up an hour ... and must be manually changed ... yes, you read that correct ... manually changed.  So if have a meeting scheduled for 10am on April 3rd, it will update to 11am April 2nd.  I then have to go and manually move it back to 9am.  Luckily, I don't have a lot of appointments scheduled in advance, but I know many of our staff do ... they will NOT be happy.

Here's a sample senario from Microsoft with emphasis added by yours truly...

Sample scenario

•	Jason is in the Indiana (East) time zone.
•	Ben is in the Central Time (US & Canada) zone.
•	Lynn is in the Eastern Time (US & Canada) zone.

On January 24, 2006, Jason scheduled a 9:00 A.M. Indiana (East) time zone appointment with Ben and with Lynn to occur April 4, 2006. The scheduled appointment automatically appears in Ben’s calendar as 8:00 A.M. and in Lynn’s calendar as 10:00 A.M. This behavior occurs because Ben's calendar program displays appointments based on the local time zone.

When the time changes on April 2, 2006, in Indiana, Jason’s network administrator uses Group Policy to change the local computer time to Eastern Time. Jason’s appointments that are scheduled after April 2, 2006, are automatically updated to reflect the new time zone. Therefore, the appointment on April 4, 2006, is automatically changed to 10:00 A.M. Although this behavior may appear confusing to Indiana residents, it is correct and shows the time that Lynn already had on her calendar.

If Jason wants the appointment to occur at 9:00 A.M. after the time zone change, he must manually move the appointment back in his calendar. After he does this, he can notify users that the appointment time has changed. However, Jason should consider that others who are invited to the appointment may have already been in the Eastern Time zone. These attendees may plan on attending the 10:00 A.M. appointment and may not be available at 9:00 A.M. Jason should examine the calendars of the other meeting attendees before he moves the appointment.

For more information about how to adjust repeating appointments in Microsoft Outlook, click the following article number to view the article in the Microsoft Knowledge Base:

915577 (http://support.microsoft.com/kb/915577/) How to incorporate Indiana's time zone change into Outlook

More from Microsoft:

How to control local calendars in Indiana

Typically, users have scheduled appointments with other users in Indiana. These appointments have to be moved manually. We recommend that you do not change local time zone appointments until after network administrators have changed the time setting on servers in Indiana.

You can make this change before April 2, 2006. As soon as the server times are changed, you can change the local time of the workstations. Users will now see that appointments that are scheduled after April 2, 2006, have moved up one hour. Users can now manually move the appointments as needed. Updates to the appointments will be sent to users and will appropriately reflect the correct time.

Meetings or events that repeat

Meetings or events that repeat in calendar programs may store the local time zone adjustment in each meeting or event. You may have to manually resend or re-create meetings or events that repeat after you change the server or workstation time zone settings.

Outlook Web Access

Outlook Web Access (OWA) also has a time zone setting. Each user must set this manually. Users must log on to OWA, click Options, locate the time zone setting, and then change the OWA time zone to either Central Time or Eastern Time. When OWA users log on, they may see an information bar that indicates that the local time differs from the OWA time zone setting if they do not update OWA.

This may warrant a few minutes of time at one of our upcoming weekly all staff meetings as I have a funny feeling an all staff email isn't going to cut it.

Though the world will not come to and end on April 2nd, I guarantee that there will be some disruptions caused by the time zone change...especially if this hotfix isn't so hot.

Random Stuff

Sorry for my lack in posting folks.  Major backlogged at work and my insomnia got really bad the last 2 weeks.  I think it's time I finally visited the doc to see about some sleep meds.

Here's some random thoughts/ideas/discoveries from the past 2 weeks...

Check out the insane upper body strength of this guy!

I'm digging CoComment again ... they've tweaked on it hard and so far I've not run into any blogs it won't work with.  It still doesn't track other's comments left on a post, but it's a great simple way to keep track of where you've left a comment.  I don't comment much, but here's where I've been commenting lately. 

My recent post about VMware was commented on and partially reposted on VMware's company blog ... that's pretty kewl and surprising exposure :-)

My recent post about VMware was commented on and a tiny bit reposted on OpManager's company blog and one of their developer's blogs ... hmm, I've not yet blogged about our use of OpManager yet.  It's coming guys :-)

I keep meaning to mention ChurchTechBlogs.com ... John Chandler has created a great blog aggregator that compiles several church tech related blogs into a one-stop shop.  I tried to convince my buddy Dave to create this exact thing several months ago ... snooze ya loose Davey ;-)

QUESTION:  What do you have to do so a Mac can read a freshly formatted NTFS drive?  They read FAT32 no prob.

Ran SpinRite on my first Mac formatted drives ... worked like a champ though 1 drive was so bad that it couldn't recover several sectors though it tried several times using several techniques.  SpinRite rules!

After over a year on our to-do list ... we now have 2 separate single function domain controllers for our network.  One is on it's own GX260 box while the other is in VMware.  Both have no other role than being a DC ... no fileserving, printing, etc.  Why do this?  Security mainly.  Prior we had DC roles on our main fileserver and also on our exchange server.  Best practices say you really want your DC's separate and stand-alone so there's an additional layer between user files where malicious code could infect and your DC's which control pretty much everything.  Each DC will also be very far from each other in the building in case an IDF drops out for some reason.

So that's some random stuff from the past 2 weeks.  I've got more great pics on my camera if I can find time to download and post.

Last 8 Hours ... Lessons and Thoughts

So after some sleep and contemplation here are a few of my take aways from Sunday's problems opportunities ...

Being a single parent in the 24/7/365 IT world would suck.  Thank you Jesus for a wonderful wife!

The ability to remote control almost everything on our network ... priceless!!

A reminder that UPS units can fail ... this is the 2nd old one I've seen fail and power off equipment tied to it!

Check the F1 support site sooner ... I would have saved 30mins troubleshooting if I'd checked sooner.

You can get right through to an SBC tech at 2am in the morning ... no wait time!

We need some remote controlled power strips ... we had these at my prior job ... give the power strip an outside IP and you can check the status via web interface of each item plugged into the strip and power cycle individual outlets on the strip should a PC/server be powered off or locked up hard.  A good one also has a modem for backup access.  Some popular ones are by Western Telematic, Dataprobe, and Micro Energetics Corp.  Of course if the UPS powering your remote power strips is down ... well, you're up the creek :-)

Living just 3 miles from where you work ... priceless!!

For us, I believe this is the first time in over a year that F1 has been down on a Sunday.  That's pretty impressive...of course we all want 100% uptime, but that's impossible.  Kudos to the F1 support team for the quick turnaround. 

Despite your best efforts ... problems will occur.  Always have a plan B ... and make sure it's tested before you need it.  Shelley made a comment that she was, in a way, glad that they had to actually put their check-in plan B to use.  I'm sure it helped them uncover inefficiencies that they'll use to revise plan B.  Some systems may require a plan C also.

Offsite monitoring tools rock!  Knowing there's a problem before your end users start calling you is golden.  Our Postini anti-spam/virus solution has SMTP monitoring built-in.  If you don't have something like this in place check out DynDNS business tools.  Their Network Monitoring service is only $99/yr and includes 12 different checks you can run.  They even now have an Email Backup service that will spool your email should your onsite email server be unavailable.

The fact that it was 50 degrees at 2am when there should be snow on the ground ... priceless!

After checking, our front firewall is no longer setup to auto fail-over to our backup ISP ... doh!  I'm betting at some point during our VLAN implementation several months ago this got broken.  The challenge is that our backup ISP comes in from the auditorium roof (wifi broadband) about as far away from our firewall as you can get in our facility.  Getting the connection to the firewall is a pain.
Wanna know what my plan B was if our SBC connection was dead?  Given my lack of hands-on knowledge of VLAN'ing, I thought of a quick-n-dirty solution.  Slap our spare/ghosting router between the backup ISP and our LAN.  Turn off DHCP on this router (it's on a different subnet) and give the check-in PC's static addresses to that router.  I'm guessing I could have the the router config'd and PC's remapped inside 30mins.
Obviously, we (aka Ed) now have a new task to get the auto fail-over working again :-)

The missing link I'd like to remedy is having someone network savvy onsite during services.  This weekend IT support role would be the first line of defense before I get called.  We identified this missing link back in June '05, but have still not filled it ... the good news is that we've not needed someone in this role for the past 9 months ... nor would they been able to fix the F1 issue Sunday.  We've had some discussion with our Tech Arts team about this in the past as they have some IT guys on their teams ... looks like it's time to rekindle that discussion.  The trick is finding someone network savvy already in a weekend role that has some flexibility to do IT support if needed. 

I was impressed with Shelley's troubleshooting ... before she called me they had rebooted more than once, verified they could get on the internet, and verified that the check-in app would launch ... not bad for someone non-techie :-)  She even commented she thought it was a problem on F1's end and not a network issue.

Explaining to a kid why you're going to church in the middle of the night is pretty humorous.  I could see the look of confusion on their half-awake faces :-)

I took Monday off and won't see anyone until 2pm today ... so I've not yet had a chance to chat with anyone about this so I'm sure some more thoughts are still to come...

I'd Rather Not Relive the Last 8 Hours

Sorry I haven't posted in while folks ... I'm soooo burried lately.  If you've sent me an email, please be patient.

Since I'm already sitting at my PC here at home I figured I'd writeup the events of the past 8 hrs ... get out your hankies! :-)

Here are a few back story items to keep in mind ...
My wife is out of town with several of her friends this weekend.
Ed is out of town with his wife on a couples retreat.

Sunday 2am-ish:  I awake to hear my cellphone indicating I have a text message.  The only text messages I get are either an alert that the power has been lost at church and our UPS systems are running on batt power OR that Postini, our offsite anti-spam/anti-virus solution, can't contact our email server.  I stumbled downstairs and read the message:  "unable to contact mail host".  Hmm, that's odd, but it's not a critical alert ... prob just a temp burp.  Back to bed I go.

Sunday ... 15 minutes later:  I get another text message.  This time Postini indicates that they've started spooling our email ... crud, this is not good.  I stumble downstairs to my PC ... crud, I can't VPN ... crud, I can't get to our OWA (outlook web access) either.  Nooooooooooooooooo!

Now my sleepy brain goes into panic mode ... what if our SBC connection at church is down?  What do I have to do to enable our backup ISP connection?  What if our backup is down also?  I start to perspire.  Ok brain, let's calm down and think this through.  It's possible our ISA firewall has had a glitch and just needs rebooted.  Any other time and I'd just hop in the car and drive to church ... but, having to dress-n-drag 2 sleepy kids with me isn't something I'd like to do.

So let's see if SBC is having problems.  GRRRR, their website sucks if you're trying to find a phone number to call to get support.  Since I cache all my GCC email at home and found my SBC docs and called up the 24hr support line.  Troy answers and I ask him to check and see if he can find any issues with their router at our location.  Troy has a good demeaner for talking my call at 2am.
He does several tests on the router and it all looks good.  He goes a step further and checks the ARP tables and can see traffic trying to get to internal addresses with no luck.  I thank him for his help.

Sunday 2:30am-ish:  So the good news is that it's not an ISP problem.  The bad news is that whatever the problem is will require a trip to the church.  Hmm, go now or wait until 7am??  I'm the worst morning person so I decide to bite the bullet and go now since I'm already up and probably won't be able to sleep knowing there's a problem

I go out and start up the minivan so it warms up ... luckily the weather isn't bad at all.  Aaron has some warm jammies on so I slap some shoes on him and carry him to the van.  Jacob is next ... man, how much does this kid weigh?!  I cover them with blankets and tell them to go to sleep.

Sunday 2:45am-ish:  We arrive at the church ... I pull the van up on the sidewalks and right to the front door.  The van is toasty warm so I kill the motor and let the kids sleep ... don't worry, I can see the van from just outside the room I'll be working in.  I open the tech room door and see no power lights from any of the computers on top of the audio rack.  After a few moments I conclude that the UPS powering our ISA firewall has failed...grrr.  Here's the kicker, I just bought a new UPS last week to replace it.  It's sitting right in front of me still in the box ... serenity now!!!!  For a moment I contemplate unboxing and setting up the new unit ... then laugh for thinking that at 3am.

So I route all the stuff on that UPS over to non-UPS powerstrips.  Power stuff on and waalaa things are working again ... a few moments later I get text messages from Postini that connection has been established and unspooling has begun.  Back to the van ... back home ... haul kids back up to bed ... dang, I'm out of shape! ... test VPN and OWA from home PC ... life is good and child check-in is ready to go for the 8:45am service.

Sunday 3:10am:  I lay down with a warm feeling that all is well in the world and look forward to sleeping in late.

Sunday 8:20am-ish:  I think I hear my cellphone ringing?  Once I actually awake I realize it's not a dream.  My cellphone, which is now right beside the bed, is saying I've got a vmail from Shelley, our First Impressions leader ... A HUGE deflating feeling comes over me ... a call from Shelley can only mean that something is wrong with the check-in PC's.  Noooooooooooooooooooo!

I give her a buzz and she explains the situation ... everything is fine except that no Sunday activities are available to select in the check-in app.  They've rebooted with no change in results.  Andy, our F1 guru, is out of town on his anniversary and thus should have his phone off...which he does.  Unfortunately, I don't know enough about F1 to know how to get into the activity setup area ... but all signs seem to point to the activities not being setup or somesuch.  I feel bad that I can't help, but tell Shelley I'm at a loss and starting thinking about plan B for check-in. 

I head down to my PC, VPN into the church and start testing ... everything is normal from a network standpoint.  By this time the 8:45am service has already begun ... luckily it's the lightest attended service we have.  However, the upcoming 10:15am service is almost always are highest attended ... yikes!  I finally remote control one of the checkin PC's (I should note that both boys are still asleep at this point) and when I launch the check-in app it won't take my activity code at all.  I call Shelley and let her know I'm going to check with Fellowship One and see if they can help.  I hit the F1 support page and see a red status indicater noting that they're having problems with activity codes...doh!  I update Shelley.  I send in a support ticket to F1 just to let them know our church is having issues as well as others.  Within a couple minutes I get a reply back that they are working quickly to resolve the issue.

I keep checking the check-in app every couple minutes and around 10:07am I'm able to get in and see the Sunday activities.  I quickly call Shelley and they've noticed the same and have started routing people from the backup handwritten system to the check-in PC's.  Better late than never :-)
A minute later I get an email from F1 support that things are back up and running.

By now my kids are up and asking for breakfast ... they say they kinda remember riding in the van :-)

So what did I learn from this?  Several things...but I'm tired and it'll have to wait for a later post :-)