Jason Powell - Church IT and Other Musings

Church IT Roundtable registration will open soon

Just a quick update...

Registration for the Sept 20th Church IT Roundtable will be opening soon.  At this point we will only be promoting the roundtable via this blog and not on the Innovative Church Conference site.  This is to give all of you first dibs :-)

Once registration is open I will email each of you who have asked to be notified with registration info.  Didn't email me yet?  Drop me a line at [email protected]

Registration cost will be in the $15-20 range to cover foodages (is that a real word?) ... yep, we'll have snacks, lunch, and some more snacks ... and of course lots of Mountain Dew for the caffeine addicts :-)

The roundtable will operate just like the other ICC workshops on Sept 20th.  So why not bring along some other staff/volunteers to the other workshops going on that day ... lots of great learning opportunities.  And finally why not just stay for the entire ICC conference? :-)

And Then There Were 2

I'm soooooo excited to share this news with all of you.  For almost 2 years Ed has been our part-time Network Admin.  We both have been longing for the day he would be moved up to full-time ... which I'm happy to announce will be this coming Monday!!  YES!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

So starting Monday GCC will officially have 2 "boxes and wires" guys.

Mucho thanks to our Senior Management Team for making this a reality.  I offered to give Tony a big hug for going to bat for me, but he declined ... in fact he threatened to axe the whole deal if I did hug him ;-)

Saddleback's CTO to present at TechEd this year!

What timing ... Tony Dye had a recent post on Church IT Visionaries and I mentioned Saddleback's Eric Busby.  However, I went on to demote Eric for not blogging great stuff for us to learn from...doh!

Our uber volunteer Dustin sent me this the other day as he was looking through his Microsoft Tech-Ed materials...

Do you have what it takes? Come join the head of technology for

Saddleback

Church

, Eric Busby, as he outlines how you can help build .NET software solutions for the biggest global effort in Christian history. Eric will be sharing a huge vision outlined by his boss, Rick Warren, author of Purpose Driven Life and influential global leader. This is an opportunity to use your skills to help solve the five most significant problems facing the world today: spiritual emptiness, corrupt leadership, extreme poverty, pandemic disease, and crippling illiteracy. YOU can make a difference. If you want to do something for God with your technology skills, you need to check this out. This group is moderated by ericbusby

Obviously, Eric is doing huge things!  Tech-Ed is only Microsoft's biggest technical conference of the year ... all the big dawgs go there.  I'd love to go, but it's very expensive ($2000+ per person!) and typically sold out.

So kudos to Eric and I hope he gets a ton of people involved. 

Goodbye Dell ... Hello IBM/Lenovo

I've been a hard core Dell dude for 7+ years.  I've easily purchased close to $2 million in Dell hardware since '99 (yes, mostly with my prior employer) ... with my biggest deal being a $960,000 lease to replace almost every PC in the high school.  Why Dell?  They have consistently given me great prices and very stable and reliable products over the years.  Except for our past grief with GX270's, it rare we need to call Dell for support issue.  Of course if you do need to call support, well, let's say that our experience over the past year has been not so good ... and we have Gold Level Support. 

At GCC we purchase Dell's Optiplex line for desktop PC's and for laptops either Latitute D600/610's or D800/810's ... oh and servers are all Dell PowerEdge ... oh and switches are Dell PowerConnect.

You may recall I did a short review 9 months ago comparing a D610, D810, and IBM/Lenovo T42.  I really liked the T42, but not enough to really pull me away from Dell.  Well, things have changed and I ordered my first non-Dell laptop last week.  In fact I ordered 3!  One X41 tablet and 2 T60's.  Why the change?

Most of our laptop users are very ... and I mean very hard on their laptops ... especially our Senior Management Team :-)  These poor laptops have been through the ringer ... dents, cracks, dings, scratches, screens so smudged I'm not sure how they actually see the cursor, gunk-n-crud in the various ports and crevices and keyboard ... I fire up the shop vac and use up a lot of Windex every time one rolls in for service/updates :-)  Compared to Dell's, IBM's Thinkpad series appear and feel much more rugged.  The added active hard drive protection is a nice feature as well ... gotta love those accelerometers.

The other thing I'm noticing is Dell's lack of laptop innovation.  I was hoping they'd add some welcome new features with their recently released D620/D820 series to keep pace with the Thinkpad's ... other than being dual-core there's nothing to write home about.  Having talked with Tony and gang over at Perimeter Church about their switch from Dell to IBM/Lenovo, I decided to research deeper into Lenovo.  It also helps that Tom and Dustin, 2 of our great volunteers, both work in large organizations that use the IBM/Lenovo Thinkpads almost exclusively.  In addition, Tom sells IBM gear ... typically large expensive server installations, but he also has channels for laptops :-)

A huge selling point for Thinkpad's is the 1 button driver updates.  Dell is way behind here.  You have to go to their website and download each individual driver and hand install one at a time ... with the Thinkpad it searches for new drivers for you and installs them ... even bios updates.  NICE!  I'm also looking forward to the fingerprint reader ... no more typing user/pass.  Tom also gave me a pretty in-depth tour of the profile manager ... wow, very impressive.  I should have had him do this with me when I reviewed the T42.  There is a little learning curve, but the stuff it can do is worth the time.

A great way to really know if you like something is to use it everyday ... so I decided to take the plunge.  Also around this time one of our SMT dudes was needing an upgrade and after given the option of Dell or Lenovo he wanted the X41 tablet ... who wouldn't?!  So Tom and I hammered out the options and got the order started.  I must say, ordering a Dell is much easier ... Lenovo needs to make their process much more logical.

Once the new stuff comes in I'll naturally post pics and other lovely details.  For the curious here's some specs on my T60p:  dual-core 2GHz, 14.1" (I don't want a big screen), 2GB ram, 100GB 7200rpm hard drive (extra ram and drive space so I can run virtual machines), 256MB video card, Centrino a/b/g wifi, Verizon EvDO wireless built-in, DVD burner, blah, blah, yada, yada ... it should rock! :-)

It's SAN time!

We are constantly fighting data growth on our servers.  I've sent out 5 all staff emails in the last 12 months begging people to remove/archive non-critical stuff off the fileserver.  A few do which frees up some space, then it fills back up in 2-3 months.  That's 400GB of expensive RAID5 SCSI drives maxed out. 
The exchange store (the database that holds everyone's email) on our email server has grown now to a whoppin' 55GB ... it was only 16GB about 2 years ago.  [We found out the hard way that Exchange2003 Standard had a 16GB store limit...2 days with little sleep to fix that fiasco].
The only server not gaspin' for space is our SQL server.

We're to the point where we can't just band-aid the problem by throwing more drives into a server ... mainly because the servers are already full.  Throwing up more fileservers only creates more stuff we have to manage ... and there's only Ed and I and Ed's part-time.  We also want to really take more advantage of server virtualization.  So we need a solution that is easy to scale and simple to manage.  Solution = Storage Area Network (SAN)

What's a SAN?  I like how IBM puts it...

A SAN is a specialized, high-speed network attaching servers and storage devices. It is sometimes called "the network behind the servers." A SAN allows "any-to-any" connection across the network, using interconnect elements such as routers, gateways, hubs, switches, and directors. It eliminates the traditional dedicated connection between a server and storage, and the concept that the server effectively "owns and manages" the storage devices. It also eliminates any restriction to the amount of data that a server can access, currently limited by the number of storage devices, which can be attached to the individual server. Instead, a SAN introduces the flexibility of networking to enable one server or many heterogeneous servers to share a common storage utility, which may comprise many storage devices, including disk, tape, and optical storage. And, the storage utility may be located far from the servers that use it. [link]

Problem is SAN's are expensive ... like stupid expensive if you ask me.  $20K for a 1TB (1000GB) SAN is commonplace.  But it's time to take the plunge...actually we're really past time.  So soon we'll begin to really dig into researching SAN's.  Luckily, 2 or our volunteers are very familiar with SAN technology so I'm pumped to work with them to find a best fit solution for GCC ... Dustin, Tom, put your thinkin' caps on!

Fellowship Church has a Lefthand Networks SAN, Perimeter Church has a SAN, who else has a SAN?  What vendor did you go with and why?  Who were your runner up vendors?  Juicy details please! :-)

Exchange moving to VMware environment

After almost 2 years of no problems our Exchange2003 server has gotten sick recently.  We've run diagnostic utilities on it since it acts like a hardware issue, but reports all come back green.  Reboot the server and all is well for a few days, but gradually things start to degrade.  People can still access their info as normal, but the response time becomes higher and higher ... until finally after a week or 2 the box stops responding.

We've determined two things that certainly aren't helping matters.  First is the size of our Store (the database that holds all the messages).  In almost 2 years we've gone from 16GB to 55GB!  For only having 135 mailboxes, that's a ridiculous size.  We have several users with 1GB+ mailboxes ... the largest is a whoppin' 5.5GB!!  Yes, I know ... we should be using quotas.  We will in the near future.  We also haven't run an offline defrag, but it's not going to reduce the size dramatically.

The exchange box is beefy ... dual Xeon 2GHz, 3GB ram, RAID5 10k rpm SCSI drives ... and the processors just sit a hair above idle all the time.  Store.exe eats up most of the ram, but that's what it's supposed to do.  Event logs don't show anything out of the ordinary and MOM (MS Operations Manager) isn't throwing any red flags either.  Obviously something is jacked up, but there's nothing you would typically look for indicating a problem.

The second thing we're pointing fingers at is the fact that several weeks ago we removed Domain Controller functions from the exchange box.  I've found several articles advising that if you place DC roles on an exchange box do don't remove them ... doh!  We've had more problems since removing the DC roles, but we also had some issues just prior to that as well.  So, yes a potential contributing factor, but not the smoking gun.

So now what?  Well, we could spend time trying to figure out this puzzle or rebuild the box.  I figure this is a great time to test running Exchange in a virtual environment.  It also gives us an opportunity to test moving virtual machines from one physical box to another and see how that works. 

So first a little info on our VMware experiment thus far.  We have a Dell PowerEdge 2650 that's now our VM server.  It's got dual Xeon 3GHz processors with 4GB ram ... and sadly only 137GB drive space (10k RAID5 SCSI).  The base OS is MS Server2003 SP1.  Then we have the free VMware server installed.  Inside of the VMware server we have the following separate virtual machines running ...
- MS Server2003 running as our secondary Domain Controller
- MS Server2003 running Terminal Services
- MS Server2003 running WSUS (windows update services)
- MS Server2003 running SpySweeper Ent Admin Console ... also slated to run Track-It, Symantec Corp Ed AV, and MOM2005
- Ubuntu Linux ... it's installed but I haven't had time to play with it yet.
- XP pro ... installed just as an XP test environment

So to put Exchange on the VM server required freeing up a TON of drive space ... remember the Store itself is 55GB.  So we shutdown all the virtual machines except the DC.  Then we moved each virtual machine's folder (in VMware everything sits in one folder and 1 file basically contains the entire server) over to another temp box.  It took a while to transfer 100GB.  This temp box is a GX260 with a 2GHz P4 proc and 2GB ram ... not beefy by any means, but it already had Server2003 R2 installed running our MOM2005.  After installing (which is simple) and launching VMware, I clicked file>open>and clicked on the folder containing the virtual machine I wanted to start up ... then clicked "start virtual machine".  Waalaa...Server2003 running Terminal Services was back up and running.  Same IP, same configs, same everything.  Yes, it still blows my mind how this technology works.  I moved a server from one physical box to another simply by moving a folder!!  I then did the same with the Server2003 for SpySweeper.  So our little GX260 was now supporting Server2003 R2 as the base OS along with 2 Server2003 virtual servers ... average CPU load was 90% spiking to 100% at times.  You could log into the virtual servers as normal though there was a tolerable lag associated.  That's just too kewl ... 3 separate installs of Server2003 running on a simple desktop PC.  This is not a good final solution, but in a pinch it works fine.

We have a plain jane Server2003 vmware template we created ... so making a new Server2003 virtual machine is quick and easy.  Ed's been working on getting Exchange installed on that virtual machine today.  If all goes as planned we'll start moving a few mailboxes over to this new virtual exchange box in the next day or 2.  Once everyone is moved to the new VM exchange, we'll completely wipe and rebuild the prior physical box.  I'm very curious to see how exchange runs in a virtual environment.  We're giving it 3GB of the 4 available on the physical server, so it's a close replica of the environment it had prior.  We do plan to move exchange from virtual back to it's prior physical box once it's rebuilt ... but if performance is good we might consider keeping it virtual.

In a way this is also testing our DR (disaster recovery) plans.  If our Exchange server suddenly burst into flames, how longer before we could have another Exchange box up and running?  This is helping us create procedures and timelines to handle such an event.  Another benefit to this rebuilding is we'll move to Exchange2003 SP2 which has built in Windows Mobile "Direct Push" technology.  So if you have a smart/PocketPC phone, you can continuously sync with exchange ala blackberry style.  Given the fact that Ed and I will be getting PocketPC phones soon this is certainly worth some of the pain involved in this ordeal :-)

I really wish the new version of Exchange was now available ... it's still in beta.  This would be a perfect time to upgrade.  Of course that would also mean buying a new 64 bit server as Exchange is 64 bit only from here out.  So in the end I'm looking at this through a positive lens.  It bites to have to mess with this right now, but we'll come out on the other end having learned a lot about stuff we've been wanting to test.  The hard part will be letting go of the fact we'll prob never know why we're having our current problems :-)

My Next Cell Phone?

It's finally happening.  We're dumping our GCC shared Nextel plan for our staff and revamping so that each individual manages their own phone/plan/overage/etc.  I'm excited because this means I don't have to manage cell phones anymore ... woot! woot!!  I will miss the chirp feature, but I'll gladly give it up for finally being able to get a smartphone.

Presently this is leading the pack to become my next cell phone...

I haven't owned a Palm device since '99 ... who would have thought 7 years later I'd be seriously pondering another Palm device ... and the most bizarre part is this Palm runs Windows Mobile!

The only thing it's missing so far is built-in 802.11 wireless ... though you can buy an SD card for it.  Of course my real question is can I access the Ev-DO network from inside my house?  We live right on the IN/MI state line and some carriers have very poor coverage at borders.  I hope Verizon is not one of them.  My ultimate goal for this (or any) smartphone is to be able to remote control any of our IT gear from almost anywhere.

Anyone got a Treo 700w?  How fast is the Ev-DO service in your area?  And can you bring it to my house for testing?? :-)

GCC Streaming Video Unveiled

A pet project of mine since coming to GCC in August 2003 has been to see our church offer video streams of the weekend service.  For many reasons it's never been made a priority so I've kept it on the bottom of my task list ... knowing someday it would become a reality.

We did our first live stream back in Sept 2005 using a NewTek TriCaster we had on loan.  Only a few select staff knew of it's existence.  The real value in streaming was brought to light, in my mind at least, when several of us were at the Catalyst Conference Oct 2005.  We were able to watch a weekend service live from my wireless laptop in a hotel room in Georgia.  This is when my boss, Tony Morgan (it's always smart to mention your boss whenever possible), mentioned he'd like to see some numbers on what it would look like to offer this to the general public.

We didn't own the TriCaster and we'd soon have to send it back or spend ~$5000 to keep it...doh!  So I started getting web-casting prices from various hosting vendors.  The pricing structures were all pretty confusing and seemed pretty expensive.  Then Paul Lara from NewTek left a comment on my blog saying to check out www.christianvideochannel.com .  I got in contact with Brad at CVC and his pricing was great ... only $75/mo for "unlimited" live or on-demand streams.  Brad was very helpful in explaining how all this streaming stuff worked ... I didn't know beans about it.  Brad also suggested just throwing the free Windows Media Encoder on an XP box with a good video capture card if we didn't keep the TriCaster.

For kicks I purchased an Osprey 300 capture card and slapped a fresh XP copy on an older Pentium 4 2.6GHz box we had available.  Downloaded/installed the free Windows Media Encoder, launched it and followed the wizard ... and we were streaming live.  Doh, if only I had know prior it was this simple to make happen.

A live stream link was sent out to a select few staff members for initial testing which was well received.  So from November through January we streamed each weekend and Thursday night service live.  The only problem was that nothing was automated.  Someone had press START on the encoder before service and STOP at the end.  Occasionally someone on the media team would forget to start it or it would be left streaming non-stop the whole weekend.  No biggie since it was just for internal use only.

I played with the encoder settings until I got what I thought was a good mix of quality and bandwidth.  The end result was to encode the audio at CD quality and the video at 30fps and 320x240 size ... yielding a 314kbps stream.  Even a slower DSL connection should be able to handle a 314k stream with no problem.  The audio has practically no artifacts and even at 200% the video looks very acceptable.

Recently I got the green light to get this ready for public consumption which meant it was time to get things automated.  Ed and I were swamped with other tasks so I employed John, one of our talented volunteers, to automate as much of the process as possible.

The end result is a process which involves no human interaction.  The encoder kicks off a few minutes before each service and broadcasts the live stream as well as captures the stream to the local drive.  A few minutes after each service the encoder turns off.  This happens for all 5 weekend services and Thursday night.  Each night after the encoder has run it ftp's the captured services up to christianvideochannel.com for on-demand viewing.  After that Kem's web team takes over to point the website links to the proper files ... which should take all of 15 seconds.

This weekend we're finally unveiling this to the public.  Sometime Monday you'll be able to watch the prior weekend service which will remain available for one week.  I'm very excited about this for 2 reasons... 
(1) I almost never get to do anything "public-facing" ... all the stuff I do is way back behind the scenes.  So having a forward role in something people will experience is quite a nice treat. 
(2)  Seeing this finally come to life after 2+ years is sweet.  People will be able to use this as a tool to introduce others to our church ... without having to ever step inside the doors.  Obviously, we want people to come to the church 'cause a web stream pales in comparison to being here live.  But, none-the-less, this tool will help reach people that may not otherwise ever consider coming ... that's just so cool to be part of.  And I can finally check this off my task list :-)

Another plus is that this solution is inexpensive...
P4 2.6GHz Dell Optiplex GX260 (~$300 off dfsdirectsales.com)
Osprey 300 video/audio capture card (~$300)
ChristianVideoChannel.com hosting for live and on-demand streaming ($75/mo)

Future enhancements?  I've already got a list which includes a dual core PC so we could encode at multiple bitrates rather than just one.  Trying to encode at both 300 and a higher rez 600kbps swamps our current PC.  We also need a separate audio mix just for the stream.  The current mix is straight off the audio board so the vocals during songs aren't as blended as I'd prefer.  Our TechArts guys have already had this on their to-do list for our WiredChurches.com audio downloads and sounds like it's still a good year out.

A big thanks to our Tech Arts guys for getting the video and audio feeds to our box.  Mega big thanks to John for his skillz in automating much of the encoding and file transfer process in a very short amount of time.  Also props to Kim V. for thinking about the internet stream as he's directing camera operators.  At times he sets up special shots just for web viewers like you :-)

We hope you all enjoy the stream.

10,262 in Attendance

I predicted an Easter attendance of 10,000 ... and Tony just released the final stats.  10,262 in attendance which blows the lid off of any prior records by about 2000 I believe.

Child check-in went flawless ... 466 kids were checked in at the 4:30pm service alone.  AND this weekend a newcomer check-in area was launched ... 3 kiosks completely wireless for our first time guests.  More on the wireless details later this week as it appears we finally have a winning wireless solution.

Exciting times!

F1 Check-in errors

Anyone else getting odd errors with F1 check-in lately?  We had intermittent check-in  problems Saturday night, Monday night, and again tonight (Weds).  I've not seen the errors personally and we don't have any screen shots :-(  One of the errors says something about SSL apparently. 

I've checked the event logs on the PC's...nothing.  Checked the firewalls...they look normal.  The other puzzling piece is that people can still surf and whatnot while the check-in app is having problems...heck, tonight I VPN'd into church while they were having problems so it's not our internet pipe dropping out.  And if you wait long enough, everything will start working normally again.

To make it even more of a brain scratcher, the PC's that had a problem Saturday night have a straight wired connection to our front firewall to their own wide open port on the firewall...no rules to get in the way...no traveling down any network switch...as close to being straight into our SBC pipe as you can get.

I've got a support ticket into F1 to see if they have an error log that the client generates locally to help in troubleshooting.  Obviously with Easter approaching this doesn't give me warm fuzzies...we could have 10,000 people show up this weekend and check-in must work flawlessly.

Guess I know what I'll being doing all day tomorrow ...

[update] Just got an email (at 12:30am EST!) from an SVP at Fellowship Tech.  He reads my blog (who would right?!), saw this entry, took the time to look at their error logs at FTech, and sent me an excel file with all the recent errors between GCC and FTech ... and asked me to call him tomorrow morning and they'll start troubleshooting this with us.  That's some incredible customer service in my book! 

[update #2] Just got an email (at 12:43am EST) from a Senior Network Engineer at FTech.  Gave me his direct line to call him in the morning.  Thanks DREZ :-)

[update final] After chatting with Daniel (aka DREZ) this morning the problem was a known issue they've been working on.  So all is well at this point for a flawless check-in this weekend.  My thanks to the FTech folks for their quick response and top class support.