Jason Powell - Church IT and Other Musings

Dell EqualLogic Auto-Replication and VMware Site Recovery Manager

Oooo, more new stuff for EqualLogic customers!  Now if only GCC were using VI3...

Dell EqualLogic Auto-Replication and VMware Site Recovery Manager

Simple, Affordable, Automated Disaster Recovery

New Solutions for Automated Site Recovery from Dell and VMware: The virtual infrastructure enabled by Dell EqualLogic and VMware VI3 technologies now automate disaster recovery planning and execution by natively integrating the PS Series Auto-Replication feature directly into VMware’s Site Recovery Manager software.
VMware Site Recovery Manager (SRM) is a new product that is tightly integrated with Dell EqualLogic PS Series to offer centralized disaster recovery management, automation, and testing for the entire virtualized datacenter. Using the PS Series’ native Auto-Replication feature integrated directly into Site Recovery Manager using specialized storage adapter software storage adapter software.  MORE

EqualLogic App-aware Snaps for Exchange

Tasty features coming for EqualLogic users!  I've been under NDA on this, but it's now free to air news :-)

Round Rock, Texas, April 28, 2008

• In a move that further enhances the Dell EqualLogic^TM  PS5000 Series as the midrange SAN of choice for Microsoft^®  Exchange Server, Dell is announcing Auto-Snapshot Manager Smart Copy for Exchange as the latest in a series of “application-aware” snapshot, cloning and data replication capabilities. This application awareness can simplify how IT application specialists can work with PS Series SANs to help improve data protection, disaster recovery and testing-and-development procedures. Smart Copy enhances the PS Series’ host integration capabilities to help reduce the amount of time IT administrators spend managing and protecting Windows^® -based business applications. In the case of Exchange, Smart Copy allows administrators to easily automate data validation checks and quickly recover from data loss or corruption. The first Smart Copy feature was announced for Microsoft SQL Server^®  in July 2007 (press release), furthering the Dell EqualLogic product line’s integration with Microsoft environments and helping to improve the speed and reliability of database and file recovery operations.
• “The Smart Copy feature set is yet another way the PS Series is simplifying storage and enhancing Dell’s overall vision of simplifying IT,” said Kevin Wittmer, senior manager, product marketing, for Dell EqualLogic storage solutions. “Evolving Smart Copy to include Exchange data protection and recovery is part of our initiative to make PS Series storage more application aware. This can empower application administrators by putting sophisticated data management and protection features at their fingertips – at no additional cost – in a safe and easy to-use package.”
• Smart Copy for Exchange will be available in May at no extra cost with all new purchases of Dell EqualLogic PS5000 Series iSCSI SAN storage arrays and for all existing PS Series customers under warranty or with a current support agreement.

Others related links:

EqualLogic and Exchange

EqualLogic and SQL

Want more info on EqualLogic Storage?  Contact VR6 Systems 

GCC Backup Cycle

A couple peeps along with our auditors recently asked about our data backup routines.  I created the following document last November as part of our budget process and figured I'd go ahead and share with everyone.  We're not backing up as thoroughly as we'd like, but we're doing what we can with our resources.  Like it or not, it's all a matter of acceptable risk.

Like many of you we're doing a Disk-to-Disk-to-Tape routine.

---------------------------------------------------

Subject: GCC Backup Summary

Here’s an overview of our current IT Backup Routine at GCC … note: there are no nightly backups being done on any mac stuff that I’m aware of.

· Monday nights we run full backups on our critical servers … email, files servers, bookstore server, Great Plains, etc. 

· Every night except for Monday we run differential backups on the same servers … this means we capture only the changes to the data since the full backup ran on Monday.  Reasoning is because the full backups can take up to 2.5 days (backing up to disk) to finish with our current backup hardware and the amount of data being backed up … the differentials complete in about 8hrs (backing up to disk).  In other words, we don’t have enough time to do full backups each night even when backing up to disk (backing up to tape takes considerably longer).

· All the above tapes are kept for 2 weeks in the server room.  Thus we can go back to any day in a 2 week window and get a file from that particular date.  After non-Monday tapes have aged over 2 weeks they’re put back into rotation and will be overwritten.

· We keep the Monday full backup tapes for a month.  So we can go back to any Monday in the current 30 day window.

· The last Monday of each month we tag as our monthly full backup.  These tapes are currently being kept for a quarter and I take them (there are many) home with me.  After these Monthly tapes age over 1 quarter they are brought back onsite and put back into rotation and will be overwritten.

· The last Monday of each quarter we tag as our quarterly full backup.  These tapes are kept for 1 year and I take them home with me.  After 1 year they are brought back onsite … yada yada yada

· The last Monday of each year we tag as our yearly full backup.  These tapes we plan to keep forever at my home currently.

Summary:

We can get back to a particular day in the current 2 week window.

We can only get back to a particular Monday after the current 2 week window and inside the current month window.

We can only get back to the last Monday of each month inside a quarter window.

We can only get back to the last Monday of each quarter inside a year window.

If we have a disaster at GCC we can only get back to the latest monthly Monday tapes at my home.  So depending on the timing of the disaster that means worst case we’d lose 4 weeks of data if we got hit at the end of the month window. 

As an example – had the recent Oct 11th tornado hit GCC … best we could do is get the tapes from Sept 24^th at my home.

Certainly, we’d like to have better backups, but this is our current reality based on resources available.

** update as of March 2008.  MozyPro is sending nightly SQL zips and trans logs offsite daily.

--------------------------------------------------------------

Some links to other parts of our backup solution:

Our backup server and D2D storage http://www.jasonpowell.net/jason_powell_church_it/2007/11/big-cheap-raid-.html

Our CommVault backup software http://www.jasonpowell.net/jason_powell_church_it/2007/02/goodbye_veritas.html

MozyPro Pricing Increase and MozyEnterprise

I got this email yesterday from a MozyPro salesdude about our account ... the gist is a HUGE price increase is coming :-(

"As a valued customer, I wanted to provide you with advance notice of a new MozyPro backup offering. March 2008, the MozyPro service will expand to include both a desktop and a server license.

The MozyPro desktop offering will run exclusively on desktop operating systems and will maintain the same low monthly price of $3.95 per license and 50 cents per GB.

The new MozyPro server offering will be available at a monthly price of $6.95 per license and $1.75 cents per GB. In addition to all the features found in the desktop offering, the robust MozyPro server product offers: 

• Desktop and Microsoft Windows server OS support
• Network share support (only available with server      license)
• VSS writer backup and restore to Exchange, SYSVOL, and      Active Directory  

Here's how it will affect you as a customer: 

New server and/or desktop licenses purchased including storage purchased and assigned to these licenses will fall under MozyPro's new pricing and product plan. 

Existing MozyPro licenses and storage on your account will be "grandfathered" and pricing will remain the same. Any additional storage purchased and assigned to a grandfathered license will retain MozyPro's existing pricing plan as well. Grandfathered licenses will retain existing features, functionality, and support. 

Take advantage of MozyPro's current product pricing:

Any licenses and storage you purchase before March 1, 2008 will be grandfathered. That means you'll lock in the current MozyPro license offering and price!

                                       

#   Server Licenses	#   GB	Grandfathered   Pricing per Month	New Server Pricing per   Month
1	10	$8.95	$24.45
5	20	$29.75	$69.75
10	100	$89.50	$244.50
20	500	$329.00	$1,014.00

If you anticipate needing additional licenses, purchasing now will save you money. If you switch your account to one of the discounted yearly or two-year plans, you will save at least 10% more as well.  Now is an excellent time to save money on our service. 

Please contact me with your questions, to purchase more resources, or to take advantage of one of our discounted plans."

Don't forget non-profits also get a 10% discount as well ... or at least they did.  Guess a good thing can only last so long.  An approx quadrupling in price seems ridiculous to me, but what do I know.  Thank goodness for the grandfather clause :-)

Also to note is the newer MozyEnterprise product line ... what caught my eye about this new offering is the local pre-population of data before you start backing up over the wire.

MozyEnterprise is a new offering based on the award-winning Mozy™; family of online backup SaaS solutions that became part of EMC with the October 2007 acquisition of Berkeley Data Systems. MozyEnterprise is an enhanced, highly scalable online backup and recovery solution that has been made enterprise-ready with several new features important to enterprise customers, including:

                   

• Enhanced security through RSA, the Security Division of EMC™ including integrated key management, authentication and authorization security features
• Rapid enterprise-wide deployment options such as assisted activation, auto-activation and proxy support as well as physical seeding up to 2 TB for larger devices to speed delivery of first backups and facilitate rapid rollouts in an enterprise environments
• Enterprise-class availability and support options tailored to meet the more demanding performance and availability needs required by large organizations
• Economical off-site data protection for remote servers, desktops and laptops located anywhere on a network—comparable solutions cost up to 10 times more than MozyEnterprise.

MozyEnterprise powered by EMC Fortress is available immediately in North America through EMC and its reseller partners. EMC plans to offer MozyEnterprise to its partners and customers outside of North America later in 2008. MozyEnterprise requires no minimum contract and imposes no fees for early service cancellation. Monthly subscription list prices are:

                   

• Desktop/laptop - $5.25/mo. per desktop/laptop plus $0.70/mo. per gigabyte protected
• Windows Server - $9.25/mo. per supported Windows server plus $2.35/mo. per gigabyte protected

Guess I'll be making some calls to Mozy this week and see what makes the most sense for GCC.

Postini Drops Their Pricing ... like Big Major Giant Drop

This is a snippet from Network World ...

" ... The Google Message Filtering service for filtering out spam and e-mail-carried malware costs $3 per user per year. At $12 per user per year, Google Message Security includes Google Message Filtering and added elements such as content-policy management to check for compliance violations.

Google Message Discovery builds off Google Message Security and adds a year of message data archiving, retention and discovery; it costs $25 per user per year for one year of archived data.

"Formerly, this would have been about $100 per user," Petry says. In general, Google is lowering the entry price for its Postini security services from about $30 per user to $3, according to Petry.

Google acquired Postini last year for approximately $625 million. "

GCC is a long time Postini user and lover ... I was excited when Google bought Postini last year and this looks like some of the first fruits from the acquisition.

IT GETS EVEN BETTER! ... Non-profits get a 66% discount on the above pricing!!!

That means the entry filtering service is $1/user/year!!
$4.08/user/year for the middle level service
$8.50/user/year for the highest level service which includes email archival for a year.

We currently pay around $22/user/year for the middle level service for 135 mailboxes ... which I pay happily because Postini rocks.  Now you're telling me I can save a bunch of money AND get message archiving to boot??  Pinch me someone :-)

Several of us are trying to get more info from Postini, you can read the response David got back here.

Hopefully we'll have more info to share during tomorrows podcast ... be there 2pm Eastern.

Big Cheap Raid Options Part III

Good grief it's taken forever to get around to posting our final solution to this thread I started way back in the spring ... post #1, and post #2 ...

So here's what we put together way back in June when we revamped our backup methods to include Disk-to-Disk before spinning backups off to tape.

I decided this time I didn't want to go completely ghetto-tastic like this, but I also didn't want to spend much on just D2D storage space.

Final solution is as follows:

Norco DS-1240 12 bay SATA Storage Array = $681.21 from mwave.com
12 Seagate 750GB Hard Drives = $2855.88 from Ciber.com
3Ware 9650SE-12ML 12 port SATA raid card = $714 from Ciber.com
4 (got one spare) meter long SFF8087 to SFF8470 cables (hard to find!) = $223 from Technical Cable Concepts

For a grand TOTAL of = $4,474.09 for ~8TB of RAID5 storage!  Not bad!

Of course when Ed went to install the RAID card we encountered our first snag.  The card is PCIe ... none of our servers had PCIe slots!  DOH!!  Since we had volunteers coming that evening and backups were part of the agenda, I headed off to Best Buy just down the street.  My objective - find the cheapest PC with a PCIe slot.  I landed a nice Gateway desktop for $484.95 ... Ed slapped Windows Server 2003 on it and then our CommVault Galaxy Express backup software.  Our volunteers had a bit of a struggle getting an old SCSI card to work in the Gateway, but after some firmware updates they were successful and got our tape autoloader connected to the box.

Yup, our backup solution consists of a cheap-o Gateway desktop hooked up to a sweet 22 tape IBM autoloader and the Norco storage chassis  The best part?  It all works way better than I expected ... so good that it's still on my radar as an option for our 50TB project.

MozyPro ... pre-populating media is coming

I had a question about MozyPro today so gave tech support a ring.  Brian answered my initial questions quickly and easily.  I also inquired into when or if they were going to develop a way to pre-populate media locally which would then be mailed to them before the initial online sync ... which can take weeks depending on how much data you want to backup.

He said that is huge in their development pipeline right now ... but couldn't give an ETA.

If Mozy can pull this off at a great pricepoint, they will have IMHO (at least for now) the killer online backup solution.  Load up my backup data onto disk locally, ship the disk back to Mozy, forget the really long initial upload and begin sync'ing block changes right away ... all for $.45/GB per month! 

It's still not a replacement (yet) for D2D nor tape for long term archival, but this is a great offsite DR solution.

MozyPro ... update

Thought I'd give everyone a quick update on our MozyPro online backup testing.

Well, we learned the hard way that Mozy needs to be installed on a drive that is NOT cramped for space.  Even after redirecting Mozy's temp files to a different drive we still had space issues as the app has a couple files in the Mozy folder that track file changes ... and thus these files can get huge!

Of course the day this becomes a critical issue both Ed and I are offsite.  So I just told Kyle to do an add/remove of Mozy and kill the Mozy folders until I have a chance to dork with it.  So a good week or more went by before I got around to reinstalling Mozy on a different drive.  I was pleased that it was smart enough to not need to re-upload all the data it had before the uninstall.  It was also now utilizing our full 3Mbps uplink.  Thinking it might impact our F1 checkin app for various events, I enabled the bandwidth throttle for 1Mbps max between 8am and 9pm ... and then several days past before I thought to check on it's progress tonight.

The bummer is that in given everything that's happened, I can't give an accurate measure of how long it actually took to sync our 46GB test bed.  It was averaging around 4GB/day when I started testing, but I believe that was majorly impacted by multiple variables (like many VM's running on the same box).  Once Mozy was clear of VM traffic it was able to hammer our uplink ... my best estimation is that it was then able to average 7.8GB/day.

Mozy's now had 4 days of just syncing the changes to our backup sets ... and it's doing this automatically every 2 hours.  It's very interesting to see what's changing on our file server in 2 hour increments ... as expected there's little changing over the weekend and Monday's are a slow day as well.  You can also tell who the night owl's are making changes in the wee hours of the morning :-)

Below is a screenshot of the history you can see in MozyPro.  I've highlighted the heaviest change period from Monday ... 138.5 MB of changes (should only be block level changes) ... taking 38mins (see prior post on what file types we're actually backing up).  The lower window has the details from the highlighted period sorted by transfer time.  The hoss of the changes was a whoppin' 91.3MB pdf file ... but it doesn't tell ya how much of the file actually changed ... only that it took 14s to encode and 17.5mins to upload those changes.

So aside from a few snags getting everything uploaded, syncs appear to be crankin' along.  As a cheap SMB DR solution I haven't seen anything yet that can touch MozyPro on features and price.  If only they would let you pre-populate some sort of media to mail to them prior they'd be golden Jerry, golden ;-)

Of course we may not need a solution like Mozy a few months down the road as we're going to get our wish of having an in town offsite location to backup/replicate to.  In the downtown community center that our church has purchased and currently renovating, we have our own server rack closet complete with ventilation, reinforced walls, etc (mad props to our volunteer Tom for making this happen!).  Our current thought is to use DoubleTake to replicate ALL our VM's over to this new location ... but I'll save that for another post :-)

MozyPro ... support and a call from a VP

I mentioned in my last MozyPro post that for some reason I couldn't get MozyPro to correctly run a SQL backup.  So rather than calling tech support I decided to just shoot an email to support with a link to my blog post ... thus they could see screen shots and whatnot.

Within 6 minutes I had an email reply back from Bryan asking to send me the log file.  About 10mins after I sent the log file Bryan said it looked like there wasn't enough disk space for Mozy to encode the files.  Sure enough, the C drive on our SQL box had about 1.5GB free space and this particular SQL database was 2.3GB ... doh!  By default MozyPro uses the C:\windows\temp directory for all it's file mojo ... not enough room and it won't run completely.  Makes perfect sense.

Bryan also sent directions (see below) on how to change the default directory ... which after following made the SQL backup work like a champ.  Kudos for rapid tech support!

That same day I also got a call from the VP of Product Development at Berkeley Data Systems ... the makers of Mozy/MozyPro.  Vance wanted to hear my experience, both good and bad, and seemed genuinely interested in my feedback.  He was catching a plane at the time so we had to cut the conversation off before we had a chance to fully dialog ... another chat is being planned.

While I feel they have a sweet solution for a sweet price, 2 major areas I feel they need to address are:

• Some mechanism to get your data to their data center BEFORE sync'ing.  At $.45/GB I'd be game to backup 500GB or so to them ... but that would take forever just to get the initial upload done.  Other online backup solutions will send you drives to populate and return, I strongly urge Berkeley to do likewise.  Shoot, just send me cheap-o USB Lacie drives or similar.  The Mozy client could encrypt and dump to the USB drive instead of uploading over the internet.  Then I can just send them the drive(s) back ... the data is encrypted so even if the drives get stolen during shipping I'm not going to sweat it that much.  A tech could contact me once they have the data ready for online sync and now I'm just sending block changes to their servers which shouldn't be that large of a data set per day (excluding large A/V files).  There should also be a way to get the data shipped back to me should disaster occur.  Should the church be wiped out, I can't wait weeks for our data to download.
• The MozyPro app must load faster.  When you open the app to do anything, you must wait for it to populate the backup sets you've flagged for backup.  For what I can tell, this means that the app must crawl the entire drive(s) looking for the files with the appropriate extensions.  On our 400GB file server this can take up to 5 minutes or more.  When you want to make a single quick configuration change, or say check the history log, this wait period just doesn't make me happy.  Perhaps make the app only do new backup set refreshes if you hit a refresh button or something ... don't make it scan drives during app startup ... please!

Address those 2 issues and they've got a killer solution for the price.  Granted, I've only been testing MozyPro for a week now and thus may uncover other issues with more testing.  For instance, I've not tested a single restore yet mainly 'cause I'm waiting for everything to get uploaded first.

Like most folks, I fully agree that online backups will not replace your D2D and tape backups ... upload speed is just not fast enough.  However, as a low cost supplement to your backup strategy, online backups have certainly gotten my attention and curiosity.  And I should also mention that our IT buddy Dean over at ACS is still pursuing online storage solutions in addition to their LiveStor product ... so something sweet may become of that as well.  Stay tuned ...

How to change the default temp directory for MozyPro:

1. Make sure you're running Mozy 1.8.2.3 or later.  (You can the latest version at http://mozy.com/downloads/mozysetup.exe)

2. Create a new folder for Mozy to build its files in. Most people use something like D:\mozytemp\
3. Open your registry editor by click the Start menu, then Run, and typing 'regedit'. Click OK.

4. Open HKEY_LOCAL_MACHINE\Software\Mozy\options

5. Right click the right pane and click New > String Value 6. Change the new value's name to 'tempdir'

7. Right click the value and select Edit.

8. Change the value to the name of the folder you created. (If you created D:\mozytemp\, make the value 'D:\mozytemp\')
9. Save the changes and close the registry editor.

10. Open a command prompt (click the Start menu, then Run, and type 'cmd'. Click OK.)
11. Type "net stop mozybackup" and hit Enter
12. Type "net start mozybackup" and hit Enter.

NOTE:  steps 11/12 didn't work for me so I just went into Admin Tools>services and restarted the mozy service.

MozyPro ... 5 days later

Continuing my online backup testing ... 5 days later and 20.2GB uploaded to MozyPro servers ... for an average of 4.04GB/day.

A smarter dude would have ran the MozyPro client on a different box rather than running it directly on our main fileserver which also hosts 4 or so VM's.  That certainly could have an impact on why I'm not seeing more of our uplink being utilized ;-)

I've also started testing SQL backups with MozyPro ... sadly it's not working for some reason.  The log files show that the app is encoding the database file, but the upload never starts.

Perhaps it's time to give tech support a ring and see how that goes.

I finally made a new blog category for backups so all these backup posts are nestled all together ... hmm, it's not often I use the word nestled.

The testing continues...

UPDATE:  I shot an email to MozyPro support about the SQL issue ... they responded back within 6 minutes and I've now sent them the log files for further investigation.  Gotta love prompt support :-)

UPDATE2: About 10mins later I got an email back with some steps to try ... and waalaa ... SQL backups are now crankin'.