Jason Powell - Church IT and Other Musings

Farewell Free OpenDNS For Businesses (and Churches)

Well, it's been a good run, but we recently got an email from OpenDNS that they are shutting down their free content filtering solution for businesses. They will still offer a free home solution, but a business (or church) will have to pay starting March 15. Here's a snipet from the email:

...OpenDNS Basic has been redesigned as a product for home use solely, to focus on delivering the best possible experience for individuals and families.  OpenDNS Enterprise, which launched two years ago and now has over 2,500 customers, is our product geared toward professional and commercial use of our Web content filtering and security services.  Though not free like the OpenDNS Basic service you use today, it includes significant additional functionality -- in particular, malware and botnet protection services vetted and chosen by Fortune 100 organizations who rely on OpenDNS Enterprise today.  And don't let the name fool you, while we are proud to have some of the largest companies in the world using OpenDNS, our business offering is competitively priced for businesses of all sizes.

If you choose not to migrate, you will continue to receive the same, best-in-class recursive DNS services for free without any usage limits, however all filtering functionality will be disabled on March 15th...

Here's the full announcement

WE started using OpenDNS at GCC on 7/30/08. Prior to that we were using the content filters on our Sonicwall firewall, but we had enough problems with it we went searching for a better solution … and thus started testing OpenDNS.

So I contacted OpenDNS to see if they offered a non-profit discount and to get pricing. Pricing is based on number of users you support … which is confusing if you're a church. If we include weekend attendees that's over 5000 people while our staff is about 80 full and part-timers. The quote I got back for 100 users was $2000 - $500 (non-profit discount) = $1500. While that's a descent discount, it's still way to much for my budget when our Sonicwall firewalls already have content filter licensing. We just haven't used the Sonicwall filters in forever! I'm told by other IT peers that the Sonicwall filters are far better than in years past … guess we'll soon find out ourselves :-)

I'm still a big fan of the free OpenDNS Family Shield solution for homes. Quick, easy, free and works very well.

[edit 2/8/12 12:58pm] After telling our OpenDNS rep we'll pass, he says he can get us down to $1000/yr.  Also had another church IT peer suggest we check out the free DynDNS filtering solution that sounds VERY similar to what OpenDNS has done for us. 

[edit 2/9/12] More info about the coming change in this forum

GCC's Servers & Storage have a new offsite home! Part 2

(Psst, hey you. You may want to go read Part 1 first)

After months and months of planning and prep work GCC's main IT production gear moved offsite to a real datacenter just a few weeks ago on Sunday Jan 22nd! w00t! We've always wished we could have our gear in sweet colocation (colo) space, but I never would have guessed we'd be there in early 2012 or even 2020 due to costs. How this came to be is a fun story, but it's not short … so that will be another blog post or 2 down the road :-)

Until then, here's a few details and pictures. Oh and monster props to Justin who did all the heavy lifting to plan, prep and execute the myriad of complexities of this move as there were a number of huge network changes and upgrades that proceeded the actual move. Thanks also to IT volunteers Ed Buford, Tom Templin, and Aaron Nush for their help Sunday to relocate the gear. Additional thanks to Ed Buford and Pinnacle for Exchange changes the week prior to the move to minimize email downtown.

Where is the gear now?
We're part of a non-profit co-op sharing a micro suite at Global Access Point (GAP) in downtown South Bend, IN. We share a 42U rack with our long time friend Josh from Center For Hospice Care for the price of … FREE! We're currently fully utilizing 20U of space (see below) and Josh says we can have a few of his U's if needed … right Josh? ;-)
The going rate for space at GAP is $50-70/U/month so we're getting $1000-$1400/mo of space at no cost. Booyah!

We have dedicated redundant 20A power feeds into our PDU in the rack so if one 20A feed goes down the other takes over without a power drop. The power feeds are also on UPS and generator backups … it's sweet to no longer worry when thunderstorms roll into the area wondering if power will go out at GCC. The downside? It's a salty $350/mo for said power. Wonder if any of the top dogs at GAP go to GCC? Would love to find a way to lower that cost.

How does the data get between GCC and the colo?
This is part of the long story mentioned above, but we have a sweet Metronet dark fiber link between GCC and our router at the colo at whatever speed we can light it. For now we're lighting it a 1Gig since 10Gig lasers aren't cheap. We'll monitor the link for a while to see how congested it gets. We're using Cogent as our ISP as they have a presence at GAP and were priced very well. We pay for 100Mbps up/down burstable to 400Mbps, but we've seen it hit as high as 500Mbps! The kicker? The cost is LESS than what we were paying for our ATT fiber at 10Mbps! Booyah! What's fun is when staff publicly cheer about how fast our internet speeds are like this tweet from Dustin :-)
Honestly, it's a bit strange to think about all our gear being downtown and accessed over a small optical fiber pair the size of a human hair. According to the Metronet folks, our optical path to the colo is 11.4 miles while the driving distance is only 7 miles. So, while it takes 20mins to get to the colo by car, it's a mere 90 microseconds for data traveling down the fiber :-)

And yes, we do have a cheap comcast business connection into GCC as our backup ISP. Should the fiber go down for some reason we can reroute most things over the comcast connection and limp along.

Guess I'll need a part 3 to go over a few more details like what gear is still at GCC, but until then here are some pics from the move.

Tom, Justin, Ed, Aaron ripping gear from the rack

Loading gear into cars. Thankfully the weather wasn't too bad considering the time of year.
All our junk in the Suite hallway. Tom is looking into our micro-suite. Yes, we brought a table and 2 chairs along with the gear since there's no place to sit or sit stuff :)
Justin and Ed hoist one of our Dell R710 ESXi hosts onto it's rails

Justin and Ed starting to wire up gear at the rear of the rack

Tom is working on the rails for our 2nd and oldest PS100E EqualLogic array

Tom and Ed working(?) at the rear of the rack. Yes, it's a bit cramped back there.

Front view of the rack with all gear in place. It's a bummer that the top R510 bezel is black and thus doesn't match the R710's and EqualLogic PS6000E :-(
Oh, we also need to take out the sliding shelf above the KVM at some point.

Rear rack view. Cables mainly nice and tidy and correctly color coded based on function. Also note our rear door can actually close unlike our neighbors. Maybe they'll be inspired to clean up their cabling after seeing ours? #doubtful

If you have question about any of this leave a comment and I'll try to address with a comment or include in the part 3 post.

Google Apps For Nonprofits AGAIN Open to Churches/Ministries

As many of you probably know, in March 2011, Google decided to no longer allow Churches/Ministry/Places of Worship access into their FREE Google For Nonprofits program (see article in Christianity Today). The result was churches/ministries would be required to pay $60/yr per user just like any other business. Yes, a number of Churches applying to the program were NOT happy and the fear was that churches already accepted and using Google For Nonprofits could be dropped from the free program without warning. Google never did give a reason why they decided to exclude Churches from the free program. Obviously, they can do whatever they want and given their history of not recognizing Christian holidays like Easter and Christmas this seemed like yet another jab at Christianity. So whatev's…

Then last week my buddy Derek Berg posted on twitter and his blog that he received an email from Google stating they were again changing their stance. He's the short and sweet "borrowed" from his blog:

"Organizations under the following categories may now be eligible for the Google for Nonprofits Program:

• Places or institutions of worship
• Programs requiring membership and/or providing benefit solely to members
• Websites with a primary focus on selling goods, products or services
• Car, boat, and real estate donation websites
• Consumer credit counseling

I went and checked Google's eligibility requirements and indeed it no longer excludes nor even mentions places or institutions of worship. They do have an FAQ that mentions a recent change in the eligibility guidelines, but doesn't list the changes. Link to the changes made

So, if you've been previously denied access to the Google For Nonprofits program because you're a church or ministry … go apply again :-)

And in case anyone is wondering … at GCC we have both Microsoft Exchange2010 and Google Apps, but have done very little with GApps thus far.

[edit 2/9/12] A Church IT peer just tweeted that their church has just been approved under the new eligibility change

[edit 2/20/12] Found a link where Google lists the eligibility changes https://groups.google.com/forum/#!topic/googlefornonprofits-discuss/8Gz64ixASWc

Registration Open for 2012 National Church IT Roundtable!

It's finally time to open registration for the 2012 National Church IT Roundtable April 18-20 in Dallas, TX! w00t w00t and a yeeeehawwww (Texas style)!

I'm soooooo pumped about this event … I can hardly stand it! This year we've added an optional 3rd day of training opportunities and other activities that we feel most won't want to miss. AND … get this, the 3rd day is no extra charge! All 3 days for only $75…thanks to our awesome sponsors/partners!

How about some tasty VMware and Network infrastructure design and best practices? Or Exchange2010 setup/migration/tuning? Or VoIP and Unified Communications? That's just a tease of what we're working on :-)

ALSO new this year, we're making a focused effort to involve Church Web/ChMS developers! In the past our events have focused more on the Network Admin/Help Desk side of IT … now we want our web/dev peeps to benefit from an event like this as much as we do. They'll have their own breakout sessions to geek out about code, APIs, ChMS integrations, tools and best practices as it pertains to their job roles. These breakouts will be lead by web/dev heads David Drinnon (Second Baptist Houston) and Chris Kehayias (Calvary Chapel Melbourne). So go now and invite your web/dev peeps on staff (assuming that's not your role too) to join you this spring.

SERIOUSLY - this is THE MUST ATTEND EVENT of 2012 for all church IT staff and volunteers! If you can only afford one event this year - this is THE ONE! I've been to every National CITRT event since 2006, helps that I started the 1st one heehee, and each year I continue to be amazed at how God uses this event to bring church IT folks together to sharpen one another and push the Kingdom forward. I guarantee if you come your brain will be filled to max capacity, you'll be throughly encouraged, and your spirit will be lifted towards almighty God. Tell me what other IT conference does all that! They don't :-)

Last year we had over 180 in attendance, this year we expect to surpass that easily! Training slots on Wednesday will be limited so register early! AND don't forget to spread the word. Reach out to churches near you and invite them too … oh, and don't forget your web/dev peeps as well!

Get details and register at http://churchITnetwork.com/spring2012/

GCC's Servers & Storage have a new offsite home! Part 1

This is part 1 of a multi-part series of blog posts about the COLO move we made last Sunday. Part 1 is just the all staff email I sent to our staff about the change. Normally my all staff emails are pretty brief and to the point, but I decided to be more verbose about this change since it was so monumental. Parts 2 and beyond will have more juicy details and specifics about the project. Happy reading!

---------------------------------------------

Beloved staff friends,
Just a heads up that tomorrow/Sunday afternoon we're making one of THE largest IT transitions in GCC's history … and hopefully you'll hardly notice it at all :-) What cha doin'? Well, we're moving the bulk of all our servers and storage gear to a state of the art Data Center/Colocation Center (aka the COLO) at Union Station in downtown South Bend! (yes, across the street from The Cove)
Uh, dude? Wha??? It's a long story that started last Spring and has some fun God stories :-) The short of it is we now have space in … get this … the same facility that houses the Notre Dame Supercomputer Cluster, some of the Amazon.com servers, and many other large companies local and US. We also have a dark fiber connection from Union Station to GCC (think of it as a really flippin' long network cable) which will make accessing GCC servers no different really than if they're housed at GCC. Cool eh? You've actually already been using the dark fiber and some of our gear already at the COLO. Our uber fast internet connection originates at Union Station and our email server was moved over there last Weds night. Booyah!
Why do this? We don't really have the type of space we'd like at GCC to house our critical gear. From security access, to backup power needs, to AC, to fire suppression (sprinklers and servers don't mix), etc. Union Station gives us all that and more … and we got a killer deal on it. Basically, we'll sleep better at night when storms roll through and tornado warnings go off knowing our gear and YOUR DATA is in a secure hardened building with lots of redundancies.
So after months of planning, late night work at GCC and Union Station (aka the COLO) we're finally ready to move most of our gear…and yes we are excited!!!
What stays at GCC? A few things will stay at GCC like the bookstore/café server, print server, and filezilla (aka the media drive). These, along with internet access, will remain available while we're moving stuff downtown.
So after the last expo is over (1pm-ish) we'll start powering down all the servers that are moving. You still get email access during this time.We then CAREFULLY unrack, box and load the gear for drive over to Union Station. Installing the gear at the COLO will be the longest part as we have to cable things compactly and neatly due to space constraints. During this time we'll have to drop email access periodically.Then we'll power things back up and start testing systems. Begin the office pool now on what time we actually leave the COLO and head home :-)
We'll be live streaming much of the activity (because we're geeky that way) here: http://www.ustream.tv/channel/jpowellcam

Follow @gcctechops on twitter with notifications turned ON if you want up to date info on when we're making changes, stuff that's broken, and when it's fixed :-) DON'T USE TWITTER? No problem! You can get TXT notifications without using twitter:

Just text FOLLOW GCCTECHOPS to 40404

If you have any questions about this don't hesitate to contact me directly. And we'd certainly covet your prayers tomorrow. We've tried to plan this process well and not disrupt activities Sunday afternoon, but there are so many complex moving pieces and we all know technology can be finicky at times.

Thanks,

Jason … No, COLO does NOT mean Colorado! … Powell

Dell Technology Symposium for Non-Profits in partnership with Joyce Meyer Ministries and the Church IT Roundtable

If you're not signed up for the CITRT enewsletter you missed seeing the below info ... and shame on you for not being signed up already!  Since this Dell deal is short notice I'm posting the enewsletter here for your viewing pleasure :)

UPDATE 7/7/11 ... if you filled out the survey you should have gotten this email from Dell..
As we draw closer to the Tech Summit, the excitement within Dell is growing!  We are making final preparation for your arrival and have a great agenda we will be finalizing this morning.  I wanted to take this opportunity to send out Dell’s preferred hotels.  We have negotiated rates for our colleagues and guests so please take advantage.  If you have any questions, please let me know.
 
Dell will be providing transportation so for logistical purposes, we recommend the following hotels (Price sheet below):

Candlewood Suites Austin-Round Rock
Springhill Suites Austin-Round Rock
Homewood Suites Austin-Round Rock
Marriott Austin North
 
Note: You will be receiving a formal invite in the next couple of days for this event.  This will be critical to fill out and return as it will capture your flight and hotel information so that we can provide transportation.

FWIW - I know a number of CITRT peeps, including myself, are going to be staying at Springhill Suites.

=======================

Hurry!  Limited time/space left for Dell's Free Technology Symposium for Christian Non-profits July 20-21!
This is a fun long story, but here's the short version - CITRT leadership recently learned of an upcoming meeting at Dell's HQ to foster peer learning among Christian non-proft orgs being spearheaded by Andrea Turek, Executive Director of IT at Joyce Meyer Ministries.  Since this sounded very similiar to CITRT, we reached out to see if collaborating might make sense.  Andrea hadn't heard of CITRT, but was very open to talk with us.  Several emails and conf calls later, we're excited to annouce our partnership with Andrea and Dell and to invite CITRT folks to Dell's campus for this free 2 day event.  Dell will be bringing in their engineering staff to talk with us about various topics many of us are wrestling with.  So while this is sorta last minute, it's an incredible opportunity to network with IT peers and learn about technology solutions to further ministry.  The event is free!  You just need to get yourself to Round Rock and find a hotel :-)

Here's the official invite from Dell:
You are invited to attend an exciting 2 day FREE event July 20-21 at the Dell Round Rock Executive Briefing Center focused on Christian non-profit organizations.  This event will provide content and best practices around the following topics:
Security – How to “plan” and create the best security roadmap solution
Storage -  Object Based Storage
Virtual Desktop Infrastructure – Save your budget by deploying VDI
Systems Management/Automation via KACE
Mobile Management-Apple, Android, Windows
DR and Backup – We all want it, very few have it
Digital Asset Management – Create, deploy reuse
Database /Application Integration- flow data back without hardcoding with Boomi
We hope you will be able join your peers for some exciting learning and fun.  If you would like to participate, please add your information the survey below so that we may send you a formal invite.  We currently have 17 registrations and will only have room for 30 total.  Registration closes July 11th.
http://www.surveymonkey.com/s/R8XHGF9 - to begin registration process (note: survey was created before we talked to Andrea & Dell so don't fret that there's no mention of CITRT)

OTHER NEWS...

Fall 2011 Regional Church IT RoundTables last week of October: 
The last week of October is our timeslot for all the regional sites to host their events with the exact day of the week up to the host site.  We have 2 sites confirmed but need many many more.  If you're interested in learning about becoming a regional site this fall check out this FAQ http://citrt.pbworks.com/w/page/7717466/hosting-a-regional-event-FAQ
You can also view the list of last year's host sites here http://citrt.pbworks.com/w/page/34816125/Past-CITRT-Events
If you're interested in being a host site this fall please send an email to info@citrt.org .. thanks!
Cost is $35 and once we firm up all the sites we'll send out registration info.

Spring 2012 National Church IT RoundTable:
It NOT too early to start planning for next spring's event!  Based on the attendance this past spring, we anticipate 200 IT peeps will be converging on Dallas next spring :-)
Put this on your calendar NOW and get it in your budget ASAP! 
Spring 2012 National Church IT RoundTable
April 18-20
$75 includes lunch/dinner both days
Watermark Community Church
Dallas, TX

As always, we welcome any suggestions/input you have on making CITRT better. What's working? What's not? How do we grow it? This is YOUR community.

Thanks,
CITRT Leadership Team
Geeks for Jesus!

Server Rack moving ... again!?

I just sent this email to All Staff ... which is pretty funny considering it was almost exactly 6yrs ago that we moved our server rack up to it's current location :-)

==================================

Howdy fellow STAFF-mates!

Tuesday July 12th starting at 6pm we're shutting the entire GCC computer network down … assume nothing will work … internet, printing, file access, nada.  Why???  Cause we're moving our server rack (seen below) from it's home up in the Tech Office space down to the lower level to the communications room … it's down the back hallway behind the farm and garden patch rooms.  Yes, that's 2 levels and NO we don't have an elevator to move a large heavy welded together rack and several hundreds pounds of expensive server gear from our area to the main level :-)

On the plus side we've made this move before, but in the opposite direction back on June 21, 2005 … read about it http://jpowell.blogs.com/jason_powell_church_it/2005/06/a_lift_and_a_wi.html

Why move??  Several years ago we needed easy access to our servers so having them in close proximity to our workspace was necessary.  In the past year or 2 we almost never look at them anymore … gotta love the advances in server virtualization and no longer doing tape backups!  Now, the bulk of our switching gear is down in the lower level comm room so moving the rack down there will make things easier from a cabling/networking perspective, we also believe the equipment down in it's new home won't require special cooling like it does today thus saving electrical costs.  Plus, having the gear on the lower level makes us feel better when there's a tornado warning for our area :)

So there ya have it!  We're moving servers around next Tuesday so expect nothing to work :-)

If you have any questions or concerns please let me know ASAP.

Thanks,
Jason … lift with your legs, not with your back … Powell

REMINDER: LIVE Interactive Church IT Podcast each month!

Being engaged on twitter so much I forget about my blog as a vehicle for sharing info ... note to self to do a better job with that :-)

Anyways, just a reminder to make sure you're scheduling time to join the LIVE Church IT Podcast each month!  So go now, open your calendar, and schedule in the 1st Friday of each month at 11am Eastern to join the phone conference bridge and chatroom.  Info on how to connect is on CITRT.org or quick link is http://citrt.pbworks.com/w/page/34565071/Podcast-Episodes

This Friday one topic of discussion is Google's recent stance on no longer allowing Churches access to the free EDU version of Google Apps. Other topics TBA and of course we'll have some open Q&A time so bring you questions/topics you'd like to discuss.

 

Revamping IT Volunteerism At GCC Part I - The Big Picture

Over the past couple months, my team has spent considerable time and energy around revamping our IT volunteer ministry.  Volunteering in our area has gone through many phases in my 7yrs at GCC, but this is our most focused effort to date to find new ways to engage people of all skill levels and build community.  It was a challenging, but rewarding process that has given us a great foundation from which to build upon in the months to come.

As we brainstormed items like new volunteer opportunities, we also went back and refined, “What is Tech Ops?” The goal: give a quick big picture overview of our department from technical facts to team culture to “set the stage” for people interested in volunteering.  Here's the result:

TECH OPS | BIG PICTURE

What we do:

The Tech Ops Department exists to equip the staff and members of GCC to be efficient and effective in the fulfillment of the Great Commission by providing relevant and strategic information, technological tools, technical support, and training.  In our service to the body of Christ, we seek to be positive, humble, approachable, responsive, and solution-oriented.

We care for:

• Network of approximately 200 computers both Dell and Apple
• 30+ virtual servers running on 2 Dell R710 VMware VSphere hosts tied to an EqualLogic PS6000E SAN
• Heavy Microsoft server and applications shop. Gotta love that non-profit discount!
• Server 2008 R2, Exchange 2010, SQL 2008 R2, SharePoint 2010 Foundations
• Windows Vista, Windows 7, OSX
• HP Gigabit to the desktop switching infrastructure
• ATT MetroE 100Mbps down/10Mbps up with backup internet through Comcast Business 50Mbps down/10Mbps up
• Veeam and DPM2010 for backups including offsite replication to DR site downtown
• Full building public and private WiFi access

Who we are:

• Group of Network ninjas, cable-monkeys, CIOs and everything in between who lives to empower people with smart and hard-working technology while simultaneously protecting those same people from technological pitfalls and vulnerabilities (including themselves).
• Technology concierges who serve people—we whole-heartedly understand that how people “feel” about technology is more important than the functionality it has the potential to deliver. At the end of the day we’ve done our job if we deliver a peaceful, easy feeling for users on all every point of the skill continuum.
• A unique and inspiring culture full of sled dogs that are energized, demanding, encouraging and a little weird—we have a deep appreciation for a serious sense of humor when under pressure.
• On-call “IT” doctors with no office hours or geographic boundaries who understand that a good techie has less to do with degrees, certifications, and experience than being smart, passionate, and fearless.
• Team oriented. No matter how great the talent, this is a no-ego fly zone.
• At the end of the day— the why trumps the what—and we work as one every time.
• Dual platform with love for both mac and PC users. We are a bi-partisan team with no annoying fan boys.
• Super talented in a general way, each bringing our own unique blend of skills to the table to get a job done.
• Leaders of people or projects—working together to rally the troops or the tasks.
• Servants committed to supporting and fulfilling the mission, vision, values and goals of the church.

The above is now included our our volunteer interest webform (see http://gccwired.com/volunteer) under Technical and Skilled.

In future posts I’ll go over some of the new volunteer roles we created and other nifty aspects of our volunteerism revamp. If you were interested in volunteering in IT at GCC, what else could we include in our "big picture" to help you determine if this would be a good fit for you?

[update: Part II - Building Community now posted]

Recommended Church IT Vendor/Partner List Ideas?

There are more and more churches looking for reputable IT service and product vendors that are ministry focused or at least very friendly towards churches/ministries.  There are also great ministry focused vendors looking to provide a great service/solution to churches/ministries at a great price.  The challenge is how to get the 2 sides together for a win-win?

I know from my own experience that I give a LOT of weight to vendors/products that I know other Church IT peers are using successfully.  Case in point: we’re looking very closely at Ruckus Wireless from Mirazon because of strong recommendations from Daryl Hunter.  I’m not alone, when I asked other’s in the CITRT IRC channel they all had a similar reply:

“I would be more likely to investigate those products and vendors ... especially if i hadn't heard of them or thought of them” – ccnash-scott

So … what’s the best way to create a list of Church/Ministry friendly IT partners/vendors … how/who would submit the vendor/partner … how could others then comment pro/con on their experience with said vendor/partner … maybe give it a star rating? … maybe something similar to Amazon and eBay comments/ratings?  How and how much do you moderate?  Allow anonymous feedback?  etc etc etc!

Does something like this already exist we might want to leverage?  May we can somehow create something inside our upcoming CITRT getshadetree release?  Someone in IRC mentioned this Wordpress plugin http://www.wpreviewsite.com/ Certainly looks interesting.

Already got some good feedback/ideas from a few IRC peeps like “allow anyone to submit a vendor/solution to the list, but require 2 or 3 people to vote “yes” before it gets approved”

Now it’s your turn.  What would be helpful to you from an IT role or vendor role?  Dream big and let’s see what we end up with :-)