Jason Powell - Church IT and Other Musings

GCC's Servers & Storage have a new offsite home! Part 2

(Psst, hey you. You may want to go read Part 1 first)

After months and months of planning and prep work GCC's main IT production gear moved offsite to a real datacenter just a few weeks ago on Sunday Jan 22nd! w00t! We've always wished we could have our gear in sweet colocation (colo) space, but I never would have guessed we'd be there in early 2012 or even 2020 due to costs. How this came to be is a fun story, but it's not short … so that will be another blog post or 2 down the road :-)

Until then, here's a few details and pictures. Oh and monster props to Justin who did all the heavy lifting to plan, prep and execute the myriad of complexities of this move as there were a number of huge network changes and upgrades that proceeded the actual move. Thanks also to IT volunteers Ed Buford, Tom Templin, and Aaron Nush for their help Sunday to relocate the gear. Additional thanks to Ed Buford and Pinnacle for Exchange changes the week prior to the move to minimize email downtown.

Where is the gear now?
We're part of a non-profit co-op sharing a micro suite at Global Access Point (GAP) in downtown South Bend, IN. We share a 42U rack with our long time friend Josh from Center For Hospice Care for the price of … FREE! We're currently fully utilizing 20U of space (see below) and Josh says we can have a few of his U's if needed … right Josh? ;-)
The going rate for space at GAP is $50-70/U/month so we're getting $1000-$1400/mo of space at no cost. Booyah!

We have dedicated redundant 20A power feeds into our PDU in the rack so if one 20A feed goes down the other takes over without a power drop. The power feeds are also on UPS and generator backups … it's sweet to no longer worry when thunderstorms roll into the area wondering if power will go out at GCC. The downside? It's a salty $350/mo for said power. Wonder if any of the top dogs at GAP go to GCC? Would love to find a way to lower that cost.

How does the data get between GCC and the colo?
This is part of the long story mentioned above, but we have a sweet Metronet dark fiber link between GCC and our router at the colo at whatever speed we can light it. For now we're lighting it a 1Gig since 10Gig lasers aren't cheap. We'll monitor the link for a while to see how congested it gets. We're using Cogent as our ISP as they have a presence at GAP and were priced very well. We pay for 100Mbps up/down burstable to 400Mbps, but we've seen it hit as high as 500Mbps! The kicker? The cost is LESS than what we were paying for our ATT fiber at 10Mbps! Booyah! What's fun is when staff publicly cheer about how fast our internet speeds are like this tweet from Dustin :-)
Honestly, it's a bit strange to think about all our gear being downtown and accessed over a small optical fiber pair the size of a human hair. According to the Metronet folks, our optical path to the colo is 11.4 miles while the driving distance is only 7 miles. So, while it takes 20mins to get to the colo by car, it's a mere 90 microseconds for data traveling down the fiber :-)

And yes, we do have a cheap comcast business connection into GCC as our backup ISP. Should the fiber go down for some reason we can reroute most things over the comcast connection and limp along.

Guess I'll need a part 3 to go over a few more details like what gear is still at GCC, but until then here are some pics from the move.

Tom, Justin, Ed, Aaron ripping gear from the rack

Loading gear into cars. Thankfully the weather wasn't too bad considering the time of year.
All our junk in the Suite hallway. Tom is looking into our micro-suite. Yes, we brought a table and 2 chairs along with the gear since there's no place to sit or sit stuff :)
Justin and Ed hoist one of our Dell R710 ESXi hosts onto it's rails

Justin and Ed starting to wire up gear at the rear of the rack

Tom is working on the rails for our 2nd and oldest PS100E EqualLogic array

Tom and Ed working(?) at the rear of the rack. Yes, it's a bit cramped back there.

Front view of the rack with all gear in place. It's a bummer that the top R510 bezel is black and thus doesn't match the R710's and EqualLogic PS6000E :-(
Oh, we also need to take out the sliding shelf above the KVM at some point.

Rear rack view. Cables mainly nice and tidy and correctly color coded based on function. Also note our rear door can actually close unlike our neighbors. Maybe they'll be inspired to clean up their cabling after seeing ours? #doubtful

If you have question about any of this leave a comment and I'll try to address with a comment or include in the part 3 post.

GCC's Servers & Storage have a new offsite home! Part 1

This is part 1 of a multi-part series of blog posts about the COLO move we made last Sunday. Part 1 is just the all staff email I sent to our staff about the change. Normally my all staff emails are pretty brief and to the point, but I decided to be more verbose about this change since it was so monumental. Parts 2 and beyond will have more juicy details and specifics about the project. Happy reading!

---------------------------------------------

Beloved staff friends,
Just a heads up that tomorrow/Sunday afternoon we're making one of THE largest IT transitions in GCC's history … and hopefully you'll hardly notice it at all :-) What cha doin'? Well, we're moving the bulk of all our servers and storage gear to a state of the art Data Center/Colocation Center (aka the COLO) at Union Station in downtown South Bend! (yes, across the street from The Cove)
Uh, dude? Wha??? It's a long story that started last Spring and has some fun God stories :-) The short of it is we now have space in … get this … the same facility that houses the Notre Dame Supercomputer Cluster, some of the Amazon.com servers, and many other large companies local and US. We also have a dark fiber connection from Union Station to GCC (think of it as a really flippin' long network cable) which will make accessing GCC servers no different really than if they're housed at GCC. Cool eh? You've actually already been using the dark fiber and some of our gear already at the COLO. Our uber fast internet connection originates at Union Station and our email server was moved over there last Weds night. Booyah!
Why do this? We don't really have the type of space we'd like at GCC to house our critical gear. From security access, to backup power needs, to AC, to fire suppression (sprinklers and servers don't mix), etc. Union Station gives us all that and more … and we got a killer deal on it. Basically, we'll sleep better at night when storms roll through and tornado warnings go off knowing our gear and YOUR DATA is in a secure hardened building with lots of redundancies.
So after months of planning, late night work at GCC and Union Station (aka the COLO) we're finally ready to move most of our gear…and yes we are excited!!!
What stays at GCC? A few things will stay at GCC like the bookstore/café server, print server, and filezilla (aka the media drive). These, along with internet access, will remain available while we're moving stuff downtown.
So after the last expo is over (1pm-ish) we'll start powering down all the servers that are moving. You still get email access during this time.We then CAREFULLY unrack, box and load the gear for drive over to Union Station. Installing the gear at the COLO will be the longest part as we have to cable things compactly and neatly due to space constraints. During this time we'll have to drop email access periodically.Then we'll power things back up and start testing systems. Begin the office pool now on what time we actually leave the COLO and head home :-)
We'll be live streaming much of the activity (because we're geeky that way) here: http://www.ustream.tv/channel/jpowellcam

Follow @gcctechops on twitter with notifications turned ON if you want up to date info on when we're making changes, stuff that's broken, and when it's fixed :-) DON'T USE TWITTER? No problem! You can get TXT notifications without using twitter:

Just text FOLLOW GCCTECHOPS to 40404

If you have any questions about this don't hesitate to contact me directly. And we'd certainly covet your prayers tomorrow. We've tried to plan this process well and not disrupt activities Sunday afternoon, but there are so many complex moving pieces and we all know technology can be finicky at times.

Thanks,

Jason … No, COLO does NOT mean Colorado! … Powell

Prayers Appreciated For Ed Buford As One Chapter Ends and Another Begins

Heads up to my friends and church IT peers ... below is a copy of the GCC allstaff communication that went out yesterday about a big change in our TechOps department ...

=============================================

Subject: A note from Jason and Ed

Jason here—I’ll go first.

On October 7th 2003, only 2 months after I started working at GCC, I had my first meeting with a fella named Ed Buford.  Ed had his own IT consulting gig and soon after our initial meeting he started volunteering, which lead to contract work here and there, then in June 2004 he joined our staff team as Network Admin.

Back in the day, our technology infrastructure had not kept pace with our growth and Ed walked in to help with a network as stable as a house of toothpicks. Many of you remember. You felt the pain of almost daily email outages. Crashed hard drives with no back-up. Print jobs that mysteriously vanished. Network speeds that made you want to scream. Ahh, the good ole days.

Things have been stable and secure for quite some time now, and much of that is due to Ed’s contribution. He was one of the hands-on gurus who took an extremely overextended, unstable network infrastructure and rebuilt it from scratch. He has brought countless DOA computers back to life and saved us all from impending lost data doom more times than I can count. The projects he has personally influenced over the past seven years has not only gained GCC a reputation of innovative technology leader with churches across the country, but also with marketplace companies. Ed is known personally for his expertise and integrity not only with the people of Granger, but with anyone who has interacted with the ministry here. Today, our network is more stable, secure and efficient than many organizations ever see in the lifetime of operations. We have Ed to thank for a network foundation that is not only serving us well now, but has us well-prepared for the future.

Which brings us to the purpose of this email. Partly due to Ed’s handiwork and partly due to a change in technology strategy, we have come to the difficult realization that our network administrator no longer requires a full-time staff position. This has been tough for all of us—Ed is a loved family member of GCC with a tremendous history. None of us ever pictured an end to Ed’s season on the staff team, yet in many ways, he has worked himself out of a job.

His last day will be this Wednesday, December 8.  If you don’t get a chance to wish him well Wednesday, don’t fret.  You’ll be able to offer a proper farewell and hug Ed yourself at the staff Christmas Party. He and Laura will both be there.

Please join me in praying for Ed, Laura and the boys as they take steps towards a new normal. The first few days and weeks will be the hardest—but the God story has already begun. Through this entire process, they both have exhibited trust in God and love for their church family that is both inspiring and humbling. This is not an easy goodbye, but then again—it’s not goodbye.

Jason

And here’s a note to everyone from Ed …

Friends,

Change is a difficult thing to witness and often it’s more difficult to be a part of.  When I look back to the past seven years, change has been my constant companion at GCC. Even the past year great changes have been a part of my life as I walked my daughter down the aisle and as I walked both of my sons down into the water to be baptized.  In the midst of all this change there has been a constant and that constant is the love that I’ve felt from each of you; it is palpable and it has touched me in so many ways. It is because of so many of you that my family and I are who we are today. Your influence and love are imprinted on the character of each member of my family and for that I cannot express my gratitude.

The changes at hand bring a mixture of sadness and joy, and I would not lie to you and tell you that these changes will be easy. Instead I will tell you that I’m looking at these past seven years with celebration in my heart for these years have been filled with much joy. I have great anticipation for what God has in store for my family. The sadness I speak of has nothing to do with leaving GCC, simply because I’m not leaving GCC. Although I’ll no longer be employed by GCC, my church is GCC and I’m still a part of this church as are my family and we will be here. No, my sadness is because I’ll miss serving each of you in the great adventure facing the staff in the upcoming months.

Please join me in prayer for my family as we embrace the changes God has placed on our horizon and know that I’m praying for each of you as you embrace the changes facing you. Let me also ask that you lift up Jason and Justin as they try to figure out how to move forward and best serve you.

-Ed

=============================================

Please join me in praying for this next chapter in Ed's life.  He's a great man and great friend.  If you'd like to ping him personally you can find him at ebuford@gmail, http://twitter.com/ebuford, and http://www.facebook.com/ebuford 

Recommended Church IT Vendor/Partner List Ideas?

There are more and more churches looking for reputable IT service and product vendors that are ministry focused or at least very friendly towards churches/ministries.  There are also great ministry focused vendors looking to provide a great service/solution to churches/ministries at a great price.  The challenge is how to get the 2 sides together for a win-win?

I know from my own experience that I give a LOT of weight to vendors/products that I know other Church IT peers are using successfully.  Case in point: we’re looking very closely at Ruckus Wireless from Mirazon because of strong recommendations from Daryl Hunter.  I’m not alone, when I asked other’s in the CITRT IRC channel they all had a similar reply:

“I would be more likely to investigate those products and vendors ... especially if i hadn't heard of them or thought of them” – ccnash-scott

So … what’s the best way to create a list of Church/Ministry friendly IT partners/vendors … how/who would submit the vendor/partner … how could others then comment pro/con on their experience with said vendor/partner … maybe give it a star rating? … maybe something similar to Amazon and eBay comments/ratings?  How and how much do you moderate?  Allow anonymous feedback?  etc etc etc!

Does something like this already exist we might want to leverage?  May we can somehow create something inside our upcoming CITRT getshadetree release?  Someone in IRC mentioned this Wordpress plugin http://www.wpreviewsite.com/ Certainly looks interesting.

Already got some good feedback/ideas from a few IRC peeps like “allow anyone to submit a vendor/solution to the list, but require 2 or 3 people to vote “yes” before it gets approved”

Now it’s your turn.  What would be helpful to you from an IT role or vendor role?  Dream big and let’s see what we end up with :-)

Video of Rick Warren Addressing the Church IT Roundtable Crowd

If you missed the recent Church IT Roundtable at Saddleback Church … well, you missed an AMAZING event! 

Just one part of what made it so incredible was having Rick Warren sit down (mostly unannounced) and spend 1.5hrs speaking to us, doing Q&A, and taking pictures if people wanted (and of course most everyone did!).

Rick’s talk was awesome…so much so that we’ve gotten TONS of people requesting copies of the live video we were streaming.  And thankfully the captured stream audio/video turned out great!  So hit the image above to hear Rick’s heart for the church and technology. 

Rick even tweeted about our CITRT event :-)
@RickWarren: We hosted 2 day #citrt nat'l roundtable for U.S. IT/Tech Pastors. Brilliant men & women! I love these guys! Pr.27:23

Pic of my wife and I hangin’ with Rick.

Did you know Rick’s a HUGE hot sauce fan? I did, so during the Q&A I asked him what his favorite was :-) ... I then followed up by asking how he kept/keeps his marriage strong.  He took several minutes to answer and did it in 3 parts: Divert Daily, Withdraw Weekly, Abandon Annually...I think many of us geeks really needed to hear that.  I sure did!

We also got to experience what a funny guy Rick is … he’s hilarious!! 
Thanks Rick for taking the time out of your busy schedule to pour into 80+ Church IT leaders.  You marked us all in a very positive way.  Geeks for Jesus!!!

Want a FREE WiredChurches.com Workshop/Practicum?!

That's right! I've been authorized to give the following deal to one lucky person: buy one workshop/practicum get one free!

Give the free one away as a gift or come with a friend/teammate and split the cost. You can use the buy one/get one on any workshop or practicum (includes the IT Roundtable) that's being hosted on June 25 or 26.

You interested? You should be! Just leave a comment below and I'll pick a random winner from the comments left next Tuesday.

How I use Priceline to get great hotel deals

A number of people have asked how I use Priceline to get crazy good deals on hotels ... most recently 3.5 star Hyatt Regency for $34 ($45 after tax and fees). Well, here's a video of the process I use combining biddingfortravel.com and priceline.com ... happy bidding!

Here's the old .swf video if you can't get vimeo to work

Workshops Coming to GCC

 

Wondering about practical events where you get tools and training for your team? WiredChurches.com workshops are teacher-led, classroom-style, one-day events. Boring they are not. You'll hear from Granger leaders about what works and what doesn’t. Practicums are facilitator-led, round-table discussions. It’s the sit-down, roll-up-your-sleeves conversation you’re craving. Not quite a support group but an informal, safe place where you can share and gain ideas from others like you.

NEW! Events grouped into two-day chunks. Make the most of your travel...come for one day or stay for two. NEW! Everyone gathers together for a half hour of worship on both days before breaking into individual workshops and practicums.

Events happening Thursday and Friday, March 5-6, 2009

Thursday
• First Impressions [Keep guests coming back]
• Kids [Creating safe, fun space for kids]
• Less Clutter. Less Noise. [How to get the word out]
• Pop Goes The Church [Leveraging pop culture in your church]
• Student Ministry Practicum [Reaching students]
• Worship Leaders Practicum [Leading worship]

Friday
• NEW! Lasting Impressions [Keeping guests engaged & growing]
• Maximizing the Arts [Unleashing the arts]
• Simply Strategic Volunteers [Finding and keeping volunteers]
• Web Strategy [Strategies to build an effective Web site]
• Technical Arts Practicum [Using lighting, sound & media]

One-price: Pre-register to get $20 discount rate of $99 per person, per day. Includes materials and lunch. Where & When: Granger Community Church campus, 90 miles east of Chicago,
9 a.m.-4 p.m. Thursday and/or Friday, March 5-6, 2009.

A Little Cable Management Tip

Ever have a cable fall behind your desk? ... then you have crawl under and try to push it back into position without it falling back down?

Yup, I hate that too!!

So one day a few years back I came up with a very simple solution ... binder clips.

Attach the clip to the end of your cable (power, network, etc) and never worry about loosing it behind your desk again!

I'm sure this is not a jpowell original design, but I'd not seen it before.  I was reminded of this simple solution while giving my office a major purge last week and figured I'd share with the group :-)

Belated Welcome Justin Moore To The GCC Team

It's a week late, but here's the all staff email I sent out this morning ... a week after Justin joined our IT team ... better late than never right?  Last week was pretty crazy with Leadership Summit so I have a better-than-lame excuse :-)

Ladies and Gentlemen, boys and girls...

If you missed the introduction at SWAT last week … it’s my most excellent joy to welcome the newest member of the GCC IT Team ... Justin Moore!  Justin joins our merry band of geeks as our new IT Specialist.  That means once he gets settled in, he'll be your initial contact for tech support issues as well as working on all sorts of uber kewl IT projects so amazing your head would explode if we told you about ‘em.  So I will protect your cranium by refraining from further details.  Yes, I’m just that thoughtful.

Here’s a few insider tidbits about Justin (and his wife Bonnie) ...

· Was born and raised “in the sticks” – Polk County, NC. And no, there is no town name – it’s just “Polk County”

(If you want to sound like a native, say “poke county”)

· Loves sweet tea and Sun Drop (if you don’t know the joys of Sun Drop, think “Mountain Dew, but sweeter and with more caffeine”)

· Always thought he’d marry a Southern Belle.

o Bonnie’s last address before us getting married was Brevard, NC.

o However, she was born and lived the first 11 years of her life in Lincoln Park, NJ.

o That’s right – married a Yankee!

· Justin and Bonnie have been to Russia 3 times for missions work … see http://www.wantmoore.com/archives/archives/category/missions/

· Strength Finder Top 5

• Belief
• Strategic
• Connectedness
• Self-Assurance
• Ideation

· Big Dave Ramsey fans … Justin lead a Financial Peace University small group this past Spring at their prior church

· All their pets are named after techie stuff: Two cats, Apache and Pixel and female Chocolate Lab named Java.

o Yes, we’re “one of those couples” who call them our children.   See http://www.wantmoore.com/archives/2007/11/19/happy-birthday-java/

Like the rest of the IT team, Justin has a blog and twitter … you can learn more about him via:

Blog: http://wantmoore.com

Twitter: http://twitter.com/wantmoore

So please welcome Justin (and Bonnie) to the team ... we're so excited to have them!  …and please help convince them winter is a fun time in Indiana ;-)

Jason ... pondering a career in Men’s synchronized swimming … NOT! ... Powell

Jason Powell
IT Director
Granger Community Church