Jason Powell - Church IT and Other Musings

GCC's Servers & Storage have a new offsite home! Part 2

(Psst, hey you. You may want to go read Part 1 first)

After months and months of planning and prep work GCC's main IT production gear moved offsite to a real datacenter just a few weeks ago on Sunday Jan 22nd! w00t! We've always wished we could have our gear in sweet colocation (colo) space, but I never would have guessed we'd be there in early 2012 or even 2020 due to costs. How this came to be is a fun story, but it's not short … so that will be another blog post or 2 down the road :-)

Until then, here's a few details and pictures. Oh and monster props to Justin who did all the heavy lifting to plan, prep and execute the myriad of complexities of this move as there were a number of huge network changes and upgrades that proceeded the actual move. Thanks also to IT volunteers Ed Buford, Tom Templin, and Aaron Nush for their help Sunday to relocate the gear. Additional thanks to Ed Buford and Pinnacle for Exchange changes the week prior to the move to minimize email downtown.

Where is the gear now?
We're part of a non-profit co-op sharing a micro suite at Global Access Point (GAP) in downtown South Bend, IN. We share a 42U rack with our long time friend Josh from Center For Hospice Care for the price of … FREE! We're currently fully utilizing 20U of space (see below) and Josh says we can have a few of his U's if needed … right Josh? ;-)
The going rate for space at GAP is $50-70/U/month so we're getting $1000-$1400/mo of space at no cost. Booyah!

We have dedicated redundant 20A power feeds into our PDU in the rack so if one 20A feed goes down the other takes over without a power drop. The power feeds are also on UPS and generator backups … it's sweet to no longer worry when thunderstorms roll into the area wondering if power will go out at GCC. The downside? It's a salty $350/mo for said power. Wonder if any of the top dogs at GAP go to GCC? Would love to find a way to lower that cost.

How does the data get between GCC and the colo?
This is part of the long story mentioned above, but we have a sweet Metronet dark fiber link between GCC and our router at the colo at whatever speed we can light it. For now we're lighting it a 1Gig since 10Gig lasers aren't cheap. We'll monitor the link for a while to see how congested it gets. We're using Cogent as our ISP as they have a presence at GAP and were priced very well. We pay for 100Mbps up/down burstable to 400Mbps, but we've seen it hit as high as 500Mbps! The kicker? The cost is LESS than what we were paying for our ATT fiber at 10Mbps! Booyah! What's fun is when staff publicly cheer about how fast our internet speeds are like this tweet from Dustin :-)
Honestly, it's a bit strange to think about all our gear being downtown and accessed over a small optical fiber pair the size of a human hair. According to the Metronet folks, our optical path to the colo is 11.4 miles while the driving distance is only 7 miles. So, while it takes 20mins to get to the colo by car, it's a mere 90 microseconds for data traveling down the fiber :-)

And yes, we do have a cheap comcast business connection into GCC as our backup ISP. Should the fiber go down for some reason we can reroute most things over the comcast connection and limp along.

Guess I'll need a part 3 to go over a few more details like what gear is still at GCC, but until then here are some pics from the move.

Tom, Justin, Ed, Aaron ripping gear from the rack

Loading gear into cars. Thankfully the weather wasn't too bad considering the time of year.
All our junk in the Suite hallway. Tom is looking into our micro-suite. Yes, we brought a table and 2 chairs along with the gear since there's no place to sit or sit stuff :)
Justin and Ed hoist one of our Dell R710 ESXi hosts onto it's rails

Justin and Ed starting to wire up gear at the rear of the rack

Tom is working on the rails for our 2nd and oldest PS100E EqualLogic array

Tom and Ed working(?) at the rear of the rack. Yes, it's a bit cramped back there.

Front view of the rack with all gear in place. It's a bummer that the top R510 bezel is black and thus doesn't match the R710's and EqualLogic PS6000E :-(
Oh, we also need to take out the sliding shelf above the KVM at some point.

Rear rack view. Cables mainly nice and tidy and correctly color coded based on function. Also note our rear door can actually close unlike our neighbors. Maybe they'll be inspired to clean up their cabling after seeing ours? #doubtful

If you have question about any of this leave a comment and I'll try to address with a comment or include in the part 3 post.

Veeam Virtualization Backup Software Discounts for Non-profits

Not heard of Veeam yet?  If your organization is invested in virtualization you really owe it to yourself to check ‘em out.  Veeam makes an award winning VM backup product, among other VM tools, and like a number of our IT peers we give it 2 thumbs up!

People continue to ask how do Churches get the best deal on veeam software.  Here’s what I have found so far … and yes, I got permission to post from both sources :)

From Mark at Veeam … find him on twitter @atxtraveler
I work specifically for the Dell relationship, and Dell has a dedicated discount program for non-profits, and Veeam has provided the "Public Sector" pricing which is 15% off list price on top of any of the products we have. So when it ships through Dell, they essentially take list -15% -Dell's Discount (which is completely based off the customer's spend).
So short answer is, all CITRT members can get a significant discount, but they all need to reach out to their Dell Software rep to get the best price from us.

From Craig at Mirazon … find them on twitter @mirazon
Jason, I just got word from Veeam that we can offer public sector pricing (15% off) to non-profits.  Mirazon will add another 5% for mentioning CITRT when requesting a quote if you want to mention that in your blog.

FWIW, we bought our Veeam Suite from Mirazon shortly after they participated and co-sponsored the Church IT Roundtable at GCC in 2009. 

Jungle Disk + Amazon S3 Online Backup Test #2

Continuing from my prior blog post on Jungle Disk Server Edition / Amazon S3 testing …

TEST #2 … see what Jungle Disk does with 67GB of user files.

I found this folder on our shared drive with mainly graphics files and pointed Jungle Disk at it to see how it would compress/dedup the data.  As you can see the bulk of the files are graphics related

Above screenshot is from TreeSizePro

RESULT:

Pretty sweet compression! 40.83GB of files but only 23.70GB pushed up to Amazon S3 storage.  For comparison I used 7zip on its maximum compression setting to see what it could do with the same files.  It didn’t do as well compressing down to 25.3GB vs 23.7GB.  So Jungle Disk is doing a great job with it’s compression engine.

So what about a folder with a mix of audio, video, and graphics files? 

RESULT:

Not nearly as good of compression, but I didn’t assume MP3 and .mov files would compress any so not really a bummer result wise.  Of course you could set file exclusions so Jungle Disk would skip those A/V files you don’t want/need to push them to S3.

You may wonder what the duration column is all about.  This shows how long a backup job has ran and thus how long it took to upload your backup to S3.  For my testing I’ve limited upload speeds to 7Mbps in the Jungle Disk configuration.  And yes, Jungle Disk will use whatever bandwidth you give it.  Amazon S3 is amazingly fast.

So here’s the result after a week of testing.  Jungle Disk is monitoring 113GB of files and has uploaded only 80GB of data to S3 for a 38% compression ratio.  Not bad at all!  Heck, even a 25% compression would enough to get excited about :-)

In Test #3 I’ll test restore functions, add a bunch more files to backup, and look at some of the reporting capabilities you get with Jungle Disk.

Jungle Disk + Amazon S3 Online Backup Test #1

I’ve been pondering recently ways to fill the gap in our current backup strategy.  In our current setup, if a disaster hit with the worst possible timing we could loose up to 4 weeks of data based on when tapes are taken offsite.  Yes, that would bite!

We do have a few measures in place to help fill in that possible 4 week window: 
1) Postini archives all our incoming/outgoing emails in their datacenter offsite.
2) We’ve been pushing our critical SQL backups offsite via MozyPro.  However, this has recently stopped working reliably so I cancelled this service.  And THUS my recent searching for a better replacement.

In a great luck of timing Chris Kehayias happened to tweet this week about his experiences with Jungle Disk as their online backup solution.  (side note: if you’re not following #citrt in twitter you’re missing some great Church IT info!)

I’ve not looked at Jungle Disk in over a year, so I was surprised to see how much it’s matured.  Basically Jungle Disk Server Edition is a backup application you install locally, but pushes your data to several cloud storage options like Amazon’s S3 service.

A 30day free trial of Jungle Disk and 10GB of free S3 storage and no transfer fees were enough to make me bite :-)

I installed the Jungle Disk server edition client on my virtual test server (Server2008R2) followed by the Jungle Disk Mangement console.  During the setup of the client it shoots you over to Amazon to setup an S3 account.  Configuring a backup is pretty similar to every other online backup tool I’ve tried (Mozy Home, Mozy Pro, Carbonite, Crashplan, etc).  Do note that Jungle Disk has no SQL or Exchange plug-ins … feed it files.

TEST #1 … give it our SQL.bak backup file we run nightly and see what happens.  It’s 2.54GB in size so I was curious to see how well it would compress/dedup before sending data out to S3.  The result:

Of 2.54GB it only sent 363MB up to S3 to store.  Nice!  Then I waited overnight to see how well it would handle the overall changes to the SQL.bak file.  The result:

Jungle Disk claims they track block level file changes and I concur.  Overnight the SQL.bak file was overwritten, but Jungle Disk was able to track and then upload to S3 only the block level changes … in this case only 20MB of changes.  That simple test was enough to sell me on using JungleDisk/S3 for our new offsite SQL solution.  So now to calc how much this should cost.

Normal pricing is $5/server for Jungle Disk and then $.15/GB/month for S3 storage space and upload transfer costs of $.10/GB/month … of course right now you get 10GB S3 space for free and no upload charges until June 30, 2010 … w00t!

In Test #2 I’ll go over my results from experimenting with about 75GB of various file types to see what we might expect if we pushed all our files up to S3.

Dell EqualLogic Auto-Replication and VMware Site Recovery Manager

Oooo, more new stuff for EqualLogic customers!  Now if only GCC were using VI3...

Dell EqualLogic Auto-Replication and VMware Site Recovery Manager

Simple, Affordable, Automated Disaster Recovery

New Solutions for Automated Site Recovery from Dell and VMware: The virtual infrastructure enabled by Dell EqualLogic and VMware VI3 technologies now automate disaster recovery planning and execution by natively integrating the PS Series Auto-Replication feature directly into VMware’s Site Recovery Manager software.
VMware Site Recovery Manager (SRM) is a new product that is tightly integrated with Dell EqualLogic PS Series to offer centralized disaster recovery management, automation, and testing for the entire virtualized datacenter. Using the PS Series’ native Auto-Replication feature integrated directly into Site Recovery Manager using specialized storage adapter software storage adapter software.  MORE

EqualLogic App-aware Snaps for Exchange

Tasty features coming for EqualLogic users!  I've been under NDA on this, but it's now free to air news :-)

Round Rock, Texas, April 28, 2008

• In a move that further enhances the Dell EqualLogic^TM  PS5000 Series as the midrange SAN of choice for Microsoft^®  Exchange Server, Dell is announcing Auto-Snapshot Manager Smart Copy for Exchange as the latest in a series of “application-aware” snapshot, cloning and data replication capabilities. This application awareness can simplify how IT application specialists can work with PS Series SANs to help improve data protection, disaster recovery and testing-and-development procedures. Smart Copy enhances the PS Series’ host integration capabilities to help reduce the amount of time IT administrators spend managing and protecting Windows^® -based business applications. In the case of Exchange, Smart Copy allows administrators to easily automate data validation checks and quickly recover from data loss or corruption. The first Smart Copy feature was announced for Microsoft SQL Server^®  in July 2007 (press release), furthering the Dell EqualLogic product line’s integration with Microsoft environments and helping to improve the speed and reliability of database and file recovery operations.
• “The Smart Copy feature set is yet another way the PS Series is simplifying storage and enhancing Dell’s overall vision of simplifying IT,” said Kevin Wittmer, senior manager, product marketing, for Dell EqualLogic storage solutions. “Evolving Smart Copy to include Exchange data protection and recovery is part of our initiative to make PS Series storage more application aware. This can empower application administrators by putting sophisticated data management and protection features at their fingertips – at no additional cost – in a safe and easy to-use package.”
• Smart Copy for Exchange will be available in May at no extra cost with all new purchases of Dell EqualLogic PS5000 Series iSCSI SAN storage arrays and for all existing PS Series customers under warranty or with a current support agreement.

Others related links:

EqualLogic and Exchange

EqualLogic and SQL

Want more info on EqualLogic Storage?  Contact VR6 Systems 

GCC Backup Cycle

A couple peeps along with our auditors recently asked about our data backup routines.  I created the following document last November as part of our budget process and figured I'd go ahead and share with everyone.  We're not backing up as thoroughly as we'd like, but we're doing what we can with our resources.  Like it or not, it's all a matter of acceptable risk.

Like many of you we're doing a Disk-to-Disk-to-Tape routine.

---------------------------------------------------

Subject: GCC Backup Summary

Here’s an overview of our current IT Backup Routine at GCC … note: there are no nightly backups being done on any mac stuff that I’m aware of.

· Monday nights we run full backups on our critical servers … email, files servers, bookstore server, Great Plains, etc. 

· Every night except for Monday we run differential backups on the same servers … this means we capture only the changes to the data since the full backup ran on Monday.  Reasoning is because the full backups can take up to 2.5 days (backing up to disk) to finish with our current backup hardware and the amount of data being backed up … the differentials complete in about 8hrs (backing up to disk).  In other words, we don’t have enough time to do full backups each night even when backing up to disk (backing up to tape takes considerably longer).

· All the above tapes are kept for 2 weeks in the server room.  Thus we can go back to any day in a 2 week window and get a file from that particular date.  After non-Monday tapes have aged over 2 weeks they’re put back into rotation and will be overwritten.

· We keep the Monday full backup tapes for a month.  So we can go back to any Monday in the current 30 day window.

· The last Monday of each month we tag as our monthly full backup.  These tapes are currently being kept for a quarter and I take them (there are many) home with me.  After these Monthly tapes age over 1 quarter they are brought back onsite and put back into rotation and will be overwritten.

· The last Monday of each quarter we tag as our quarterly full backup.  These tapes are kept for 1 year and I take them home with me.  After 1 year they are brought back onsite … yada yada yada

· The last Monday of each year we tag as our yearly full backup.  These tapes we plan to keep forever at my home currently.

Summary:

We can get back to a particular day in the current 2 week window.

We can only get back to a particular Monday after the current 2 week window and inside the current month window.

We can only get back to the last Monday of each month inside a quarter window.

We can only get back to the last Monday of each quarter inside a year window.

If we have a disaster at GCC we can only get back to the latest monthly Monday tapes at my home.  So depending on the timing of the disaster that means worst case we’d lose 4 weeks of data if we got hit at the end of the month window. 

As an example – had the recent Oct 11th tornado hit GCC … best we could do is get the tapes from Sept 24^th at my home.

Certainly, we’d like to have better backups, but this is our current reality based on resources available.

** update as of March 2008.  MozyPro is sending nightly SQL zips and trans logs offsite daily.

--------------------------------------------------------------

Some links to other parts of our backup solution:

Our backup server and D2D storage http://www.jasonpowell.net/jason_powell_church_it/2007/11/big-cheap-raid-.html

Our CommVault backup software http://www.jasonpowell.net/jason_powell_church_it/2007/02/goodbye_veritas.html

MozyPro Pricing Increase and MozyEnterprise

I got this email yesterday from a MozyPro salesdude about our account ... the gist is a HUGE price increase is coming :-(

"As a valued customer, I wanted to provide you with advance notice of a new MozyPro backup offering. March 2008, the MozyPro service will expand to include both a desktop and a server license.

The MozyPro desktop offering will run exclusively on desktop operating systems and will maintain the same low monthly price of $3.95 per license and 50 cents per GB.

The new MozyPro server offering will be available at a monthly price of $6.95 per license and $1.75 cents per GB. In addition to all the features found in the desktop offering, the robust MozyPro server product offers: 

• Desktop and Microsoft Windows server OS support
• Network share support (only available with server      license)
• VSS writer backup and restore to Exchange, SYSVOL, and      Active Directory  

Here's how it will affect you as a customer: 

New server and/or desktop licenses purchased including storage purchased and assigned to these licenses will fall under MozyPro's new pricing and product plan. 

Existing MozyPro licenses and storage on your account will be "grandfathered" and pricing will remain the same. Any additional storage purchased and assigned to a grandfathered license will retain MozyPro's existing pricing plan as well. Grandfathered licenses will retain existing features, functionality, and support. 

Take advantage of MozyPro's current product pricing:

Any licenses and storage you purchase before March 1, 2008 will be grandfathered. That means you'll lock in the current MozyPro license offering and price!

                                       

#   Server Licenses	#   GB	Grandfathered   Pricing per Month	New Server Pricing per   Month
1	10	$8.95	$24.45
5	20	$29.75	$69.75
10	100	$89.50	$244.50
20	500	$329.00	$1,014.00

If you anticipate needing additional licenses, purchasing now will save you money. If you switch your account to one of the discounted yearly or two-year plans, you will save at least 10% more as well.  Now is an excellent time to save money on our service. 

Please contact me with your questions, to purchase more resources, or to take advantage of one of our discounted plans."

Don't forget non-profits also get a 10% discount as well ... or at least they did.  Guess a good thing can only last so long.  An approx quadrupling in price seems ridiculous to me, but what do I know.  Thank goodness for the grandfather clause :-)

Also to note is the newer MozyEnterprise product line ... what caught my eye about this new offering is the local pre-population of data before you start backing up over the wire.

MozyEnterprise is a new offering based on the award-winning Mozy™; family of online backup SaaS solutions that became part of EMC with the October 2007 acquisition of Berkeley Data Systems. MozyEnterprise is an enhanced, highly scalable online backup and recovery solution that has been made enterprise-ready with several new features important to enterprise customers, including:

                   

• Enhanced security through RSA, the Security Division of EMC™ including integrated key management, authentication and authorization security features
• Rapid enterprise-wide deployment options such as assisted activation, auto-activation and proxy support as well as physical seeding up to 2 TB for larger devices to speed delivery of first backups and facilitate rapid rollouts in an enterprise environments
• Enterprise-class availability and support options tailored to meet the more demanding performance and availability needs required by large organizations
• Economical off-site data protection for remote servers, desktops and laptops located anywhere on a network—comparable solutions cost up to 10 times more than MozyEnterprise.

MozyEnterprise powered by EMC Fortress is available immediately in North America through EMC and its reseller partners. EMC plans to offer MozyEnterprise to its partners and customers outside of North America later in 2008. MozyEnterprise requires no minimum contract and imposes no fees for early service cancellation. Monthly subscription list prices are:

                   

• Desktop/laptop - $5.25/mo. per desktop/laptop plus $0.70/mo. per gigabyte protected
• Windows Server - $9.25/mo. per supported Windows server plus $2.35/mo. per gigabyte protected

Guess I'll be making some calls to Mozy this week and see what makes the most sense for GCC.

Postini Drops Their Pricing ... like Big Major Giant Drop

This is a snippet from Network World ...

" ... The Google Message Filtering service for filtering out spam and e-mail-carried malware costs $3 per user per year. At $12 per user per year, Google Message Security includes Google Message Filtering and added elements such as content-policy management to check for compliance violations.

Google Message Discovery builds off Google Message Security and adds a year of message data archiving, retention and discovery; it costs $25 per user per year for one year of archived data.

"Formerly, this would have been about $100 per user," Petry says. In general, Google is lowering the entry price for its Postini security services from about $30 per user to $3, according to Petry.

Google acquired Postini last year for approximately $625 million. "

GCC is a long time Postini user and lover ... I was excited when Google bought Postini last year and this looks like some of the first fruits from the acquisition.

IT GETS EVEN BETTER! ... Non-profits get a 66% discount on the above pricing!!!

That means the entry filtering service is $1/user/year!!
$4.08/user/year for the middle level service
$8.50/user/year for the highest level service which includes email archival for a year.

We currently pay around $22/user/year for the middle level service for 135 mailboxes ... which I pay happily because Postini rocks.  Now you're telling me I can save a bunch of money AND get message archiving to boot??  Pinch me someone :-)

Several of us are trying to get more info from Postini, you can read the response David got back here.

Hopefully we'll have more info to share during tomorrows podcast ... be there 2pm Eastern.

Big Cheap Raid Options Part III

Good grief it's taken forever to get around to posting our final solution to this thread I started way back in the spring ... post #1, and post #2 ...

So here's what we put together way back in June when we revamped our backup methods to include Disk-to-Disk before spinning backups off to tape.

I decided this time I didn't want to go completely ghetto-tastic like this, but I also didn't want to spend much on just D2D storage space.

Final solution is as follows:

Norco DS-1240 12 bay SATA Storage Array = $681.21 from mwave.com
12 Seagate 750GB Hard Drives = $2855.88 from Ciber.com
3Ware 9650SE-12ML 12 port SATA raid card = $714 from Ciber.com
4 (got one spare) meter long SFF8087 to SFF8470 cables (hard to find!) = $223 from Technical Cable Concepts

For a grand TOTAL of = $4,474.09 for ~8TB of RAID5 storage!  Not bad!

Of course when Ed went to install the RAID card we encountered our first snag.  The card is PCIe ... none of our servers had PCIe slots!  DOH!!  Since we had volunteers coming that evening and backups were part of the agenda, I headed off to Best Buy just down the street.  My objective - find the cheapest PC with a PCIe slot.  I landed a nice Gateway desktop for $484.95 ... Ed slapped Windows Server 2003 on it and then our CommVault Galaxy Express backup software.  Our volunteers had a bit of a struggle getting an old SCSI card to work in the Gateway, but after some firmware updates they were successful and got our tape autoloader connected to the box.

Yup, our backup solution consists of a cheap-o Gateway desktop hooked up to a sweet 22 tape IBM autoloader and the Norco storage chassis  The best part?  It all works way better than I expected ... so good that it's still on my radar as an option for our 50TB project.