Jason Powell - Church IT and Other Musings

GCC's Servers & Storage have a new offsite home! Part 2

(Psst, hey you. You may want to go read Part 1 first)

After months and months of planning and prep work GCC's main IT production gear moved offsite to a real datacenter just a few weeks ago on Sunday Jan 22nd! w00t! We've always wished we could have our gear in sweet colocation (colo) space, but I never would have guessed we'd be there in early 2012 or even 2020 due to costs. How this came to be is a fun story, but it's not short … so that will be another blog post or 2 down the road :-)

Until then, here's a few details and pictures. Oh and monster props to Justin who did all the heavy lifting to plan, prep and execute the myriad of complexities of this move as there were a number of huge network changes and upgrades that proceeded the actual move. Thanks also to IT volunteers Ed Buford, Tom Templin, and Aaron Nush for their help Sunday to relocate the gear. Additional thanks to Ed Buford and Pinnacle for Exchange changes the week prior to the move to minimize email downtown.

Where is the gear now?
We're part of a non-profit co-op sharing a micro suite at Global Access Point (GAP) in downtown South Bend, IN. We share a 42U rack with our long time friend Josh from Center For Hospice Care for the price of … FREE! We're currently fully utilizing 20U of space (see below) and Josh says we can have a few of his U's if needed … right Josh? ;-)
The going rate for space at GAP is $50-70/U/month so we're getting $1000-$1400/mo of space at no cost. Booyah!

We have dedicated redundant 20A power feeds into our PDU in the rack so if one 20A feed goes down the other takes over without a power drop. The power feeds are also on UPS and generator backups … it's sweet to no longer worry when thunderstorms roll into the area wondering if power will go out at GCC. The downside? It's a salty $350/mo for said power. Wonder if any of the top dogs at GAP go to GCC? Would love to find a way to lower that cost.

How does the data get between GCC and the colo?
This is part of the long story mentioned above, but we have a sweet Metronet dark fiber link between GCC and our router at the colo at whatever speed we can light it. For now we're lighting it a 1Gig since 10Gig lasers aren't cheap. We'll monitor the link for a while to see how congested it gets. We're using Cogent as our ISP as they have a presence at GAP and were priced very well. We pay for 100Mbps up/down burstable to 400Mbps, but we've seen it hit as high as 500Mbps! The kicker? The cost is LESS than what we were paying for our ATT fiber at 10Mbps! Booyah! What's fun is when staff publicly cheer about how fast our internet speeds are like this tweet from Dustin :-)
Honestly, it's a bit strange to think about all our gear being downtown and accessed over a small optical fiber pair the size of a human hair. According to the Metronet folks, our optical path to the colo is 11.4 miles while the driving distance is only 7 miles. So, while it takes 20mins to get to the colo by car, it's a mere 90 microseconds for data traveling down the fiber :-)

And yes, we do have a cheap comcast business connection into GCC as our backup ISP. Should the fiber go down for some reason we can reroute most things over the comcast connection and limp along.

Guess I'll need a part 3 to go over a few more details like what gear is still at GCC, but until then here are some pics from the move.

Tom, Justin, Ed, Aaron ripping gear from the rack

Loading gear into cars. Thankfully the weather wasn't too bad considering the time of year.
All our junk in the Suite hallway. Tom is looking into our micro-suite. Yes, we brought a table and 2 chairs along with the gear since there's no place to sit or sit stuff :)
Justin and Ed hoist one of our Dell R710 ESXi hosts onto it's rails

Justin and Ed starting to wire up gear at the rear of the rack

Tom is working on the rails for our 2nd and oldest PS100E EqualLogic array

Tom and Ed working(?) at the rear of the rack. Yes, it's a bit cramped back there.

Front view of the rack with all gear in place. It's a bummer that the top R510 bezel is black and thus doesn't match the R710's and EqualLogic PS6000E :-(
Oh, we also need to take out the sliding shelf above the KVM at some point.

Rear rack view. Cables mainly nice and tidy and correctly color coded based on function. Also note our rear door can actually close unlike our neighbors. Maybe they'll be inspired to clean up their cabling after seeing ours? #doubtful

If you have question about any of this leave a comment and I'll try to address with a comment or include in the part 3 post.

For Sale: Dell EqualLogic PS100E Storage Array

update >>> SOLD!

We just got a brand new 16TB EqualLogic PS6000E array to add to our EqualLogic SAN with the idea to “retire” one of our PS100E’s and use another PS100E for offsite replication.

We’d like to sell the retired array to a church/ministry that wouldn’t typically be able to afford an EqualLogic SAN.  This would be a great starter SAN for someone just getting into virtualization or someone needing higher performing iSCSI vs something like an MD3000i, Drobo, etc.

Specs
*EqualLogic PS100E
*14x250GB SATA 7200rpm = 3.5TB raw (who cares though) = ~2.2TB usable at RAID50 w/ 2 global hot spares (RAID50 is what most EQL customers run)
*Dual Controllers
*Here’s a blog post from when we 1st installed it http://jpowell.blogs.com/jason_powell_church_it/2006/07/san_project_upd.html ... Still has 5+ years of life left
*Just pulled from production environment and will be re-certified upon sale
*$5000 for the array +1 year of Dell NBD service at $2100 = $7100

I’d be happy to give anyone a demo of the interface and setup of the array if you’re interested … the beauty of EqualLogic is how easy/simple they are to setup and manage, especially for small IT shops.  We’ve been an EqualLogic shop since 2006 and still believe it’s the best enterprise iSCSI solution out there.

 

Happy Four Year Birthday to VMware Server! You Rocked For Us!

Since it didn't receive much, if any, news last week ... I thought I'd take a few moments to say Happy Birthday to VMware Server!  Hard to believe you're 4 years old already! 

While we're no longer using VMware Server, it had a monumental impact on our IT strategy and served us faithfully for over 3 1/2 years from Feb 2006 thru August 2009 ... forever changing the IT landscape at GCC.  In those years we were huge cheerleaders for the free VMware Server platform giving several public talks and demos ... culminating with VMware asking us to do a webinar for them :-)

So for fun here's a short look back at some of the VMware Server related "events" at GCC ...

Feb 2006 - GCC begins testing VMware Server Beta on a few lower priority applications like Terminal Services and WSUS.  We're so impressed that in the following weeks we start moving more and more applications into VM's.  Read the blog post

April 2006 - Our physical Exchange server pukes, so we rebuild it onto VMware Server.  We also start researching SAN solutions to take our virtualization environment to the next level.  Read the blog post

July 12th, 2006 - VMware Server 1.0.0 goes live and offers paid support options.  Of course by this time we're already sold on virtualization and VMware Server and have most everything running in a VM.  Read the blog post

July 2006 - I'm asked to give a presentation on our use of VMware Server at the Michiana IT Pro User Group.  Read the blog post

November 2007 - VMware contacts us and Ed does a webinar for them about our use of VMware Server.  Read the blog post

April 2008 - I present on Virtualization and VMware Server at MinistryTECH.  Read the blog post

April 2008 - Overview of our virtualization environment at the time. Read the blog post

August 19, 2009 - We begin saying farewell to VMware Server as we start our 1st Vsphere Install. Read the blog post

Why the switch from VMware Server?  Well, once you get over 20 VM's it becomes difficult to manage them all under the VMware Server platform.  You'll also get a performance increase by moving to a bare-metal hypervisor like ESX/ESXi.  So when I crunched the numbers, paying ~$1000/yr over 3 years to move us to Vsphere Standard was a no-brainer.

VMware Server is still available for free ... if you've not played with server virtualization (you need to be slapped mightly if that is true!) it's still one of the simplest ways to get your feet wet.

I will always have a special place in my heart for VMware Server ... thanks for all you gave to us at GCC ... and happy 4th BDAY!

Veeam Virtualization Backup Software Discounts for Non-profits

Not heard of Veeam yet?  If your organization is invested in virtualization you really owe it to yourself to check ‘em out.  Veeam makes an award winning VM backup product, among other VM tools, and like a number of our IT peers we give it 2 thumbs up!

People continue to ask how do Churches get the best deal on veeam software.  Here’s what I have found so far … and yes, I got permission to post from both sources :)

From Mark at Veeam … find him on twitter @atxtraveler
I work specifically for the Dell relationship, and Dell has a dedicated discount program for non-profits, and Veeam has provided the "Public Sector" pricing which is 15% off list price on top of any of the products we have. So when it ships through Dell, they essentially take list -15% -Dell's Discount (which is completely based off the customer's spend).
So short answer is, all CITRT members can get a significant discount, but they all need to reach out to their Dell Software rep to get the best price from us.

From Craig at Mirazon … find them on twitter @mirazon
Jason, I just got word from Veeam that we can offer public sector pricing (15% off) to non-profits.  Mirazon will add another 5% for mentioning CITRT when requesting a quote if you want to mention that in your blog.

FWIW, we bought our Veeam Suite from Mirazon shortly after they participated and co-sponsored the Church IT Roundtable at GCC in 2009. 

What Makes A Good iSCSI Switch Part 2

Continuing from my prior post on iSCSI switches … this time I found some great info searching through the EqualLogic technical whitepapers. 

iSCSI traffic is just standard TCP/IP, so there is nothing really special about the packets or
protocol. What is different about iSCSI traffic is the amount of it (usually there's a lot) and
the fact that it must be delivered with minimum delay (latency).

Most network traffic from desktops or small servers is low volume and intermittent, and
most switches are perfectly adequate to handle traffic like this. However, iSCSI tends to
generate lots of traffic, at high speeds, for long periods; for example, it is not unusual for an
EqualLogic array to be receiving and/or sending four simultaneous unbroken streams of
network packets, running at full wire speed (1 Gbps) for minutes or hours at a time. Many
network devices are simply not up to the challenge of delivering that volume of traffic
without having to discard packets.

Manufacturers generally rate their switches according to a category of use. Common terms
such as "workgroup", "wiring closet", "SOHO" and "unmanaged" are generally associated
with switches that may not be adequate for high-traffic networks such as iSCSI.

"Enterprise", "Layer 3" or "L3", "managed", and "datacenter/server farm" are associated
with higher-performance switches that may be acceptable for high-traffic applications.

We have found that the following are requirements for a network switch that will be used
in a high-traffic environment such as iSCSI:

1 - The switch should be able to process and forward a continuous stream of data at full
wire speed (1 Gbps) on all ports simultaneously. Many switches will not in fact do this
even though their advertising copy implies that they can.
Some switches are designed so that multiple Ethernet ports (usually 4 or 8) are assigned to
a "port group", and the amount of bandwidth that can be handled by that port group is
limited. This means that each port can handle the full 1 Gbps when used by itself, but as
soon as you start using two ports in that port group, you find that each one will only move
data at half of that speed (512 Kbps), and if you use four ports, you will only see one-fourth
of that speed (256 Kbps) on each port. This will give unacceptably poor performance in
high-traffic applications such as iSCSI.
In some switches, the internal data channels are not fast enough to handle the full flow of
data. The internal backplane or "fabric" should be rated for at least 1 Gbps per port. So, on
a 24-port switch, the internal backplane should be designed to handle 24 Gbps or more.

2 - The switch must minimally support "receive flow control", meaning that if it is sending
traffic to another device, and that device sends a "pause frame" back to the switch to tell it
to stop sending, the switch must stop sending traffic until the receiving device tells the
switch to proceed. This is necessary to prevent the receiving device from having to discard
packets because it is getting overwhelmed.
Some switches also support "send flow control"; that is, if the switch decides that it is being
overwhelmed by some sender, it can send "pause frames" to the sender to ask it to
momentarily stop sending traffic. This is a benefit if available but is not required.

3 - The switch must have adequate, dedicated buffer space *per port* to allow it to buffer
bursts of packets that it receives but momentarily cannot forward, or to buffer packets that
it is holding on behalf of a receiver that has used flow control to momentarily pause
transmission. We have found that a minimum of 256 KB *per port* is a desirable value.
Note that many switches have a buffer allocation *per port group* or *for the entire
switch*, which means that the entire group of ports (4 or 8), or every port on the switch,
shares the same buffer space. If this amount of buffer space, divided by the number of
ports in the port group or on the entire switch, does not give at least 256 KB per port, then
the switch will likely be a poor performer in a high-traffic application like iSCSI.

4 - If you are using your switch to route iSCSI traffic, you must ensure that it has adequate
speed and throughput to route the packets at wire speed with minimal delay (latency).
Many switches that advertise L3 routing capabilities cannot in fact perform the task
adequately for an iSCSI network.

So what switches are good?  Great question!  I heard an EqualLogic engineer in the past refer to the following:
Dell 54XX good
Dell 62XX best
HP 2800 bad
HP 2900 ok
HP 3500 better
How exactly he arrived at that list I don’t know and haven’t gone to investigate .. so you’ll have to do your own homework ;-)

We’ve been using a pair of Cisco 24port 2960G-24TC for our iSCSI network since installing our first EqualLogic array July 2006.  We were recommended this model by our EqualLogic SE when we purchased the array.
I was curious if it meet the above recommendations so I went googling…
*32 Gbps switching fabric and forwarding bandwidth 32Gbps..looks like that satisfies #1
*For sure we’re using flow control..but, nothing I could find stats if it does both directions
*64MB DRAM & 32MB Flash..again, nothing I could find specifically about buffer ports, but I did find this on a forum post “On 2960 switches each ASIC has shared buffers (ingress of 192KB and egress of 384KB).”  Not so helpful :-)
Although we’ve not done any performance testing, we’ve not noticed any speed issues of these switches for almost 4 years.  If we were looking for iSCSI switches today, we’d look at HP since we’re using HP switches for everything else.

There ya go.  Hope that’s helpful.  Maybe some switching gurus will chime in with additional info.  Of course this will all change when 10Gig becomes cheap enough to be mainstream..which I hope is very soon :-)

[ FYI - if you're investigating EqualLogic Storage I'd love to chat with you and provide quotes.  I've been moonlighting for a Dell Premier Partner since 2008 doing EqualLogic sales and implementation with customers large and small all across the country.  And yes, we beat Dell direct pricing :-)  Shoot an email to jason.powell at vr6systems dot com - thanks! ]

EqualLogic User Conference - What Makes a Good iSCSI Switch?

I hear this often when people are first getting into iSCSI storage, “What switch should I get?” Here’s what I jotted down from Dell EqualLogic engineers today at the EqualLogic User Conference

iSCSI Best Practices session

NOTES

port buffering ... want inbound and outbound

shared vs dedicated buffer architecture … dedicated obviously better but more $

make sure switches do flow control in BOTH directions in/out

minimize spanning  tree

use rapid spanning tree mode on interswitch links

Trunking - also called ISL, LAG, PaGP, LACP

Best Trunking is 1:1 network capacity
Acceptable trunking is 70-80% of network capacity
So if you have 2 arrays w/ 3x1Gb links, you should have 6Gb between switches (1:1)..or at min 6Gbx.7 (70%)=4.2Gb=5Gb between switches

Trunking < Stacking < Chassis Switch

What makes a good switch???

Buffers = 256K per port (guaranteed) [128-256K ok in 3 or less array setups]

flow control ... at min need receive … both directions best

Check for oversubscription in backplane

for non-stacking switches .. best if they have at least 2 10Gb "uplink" ports

continued on part 2

VMware Non-Profit Eligibility

To piggy-back on my recent post about VMware academic pricing now available to non-profits … here are the requirements, per VMware, on who’s eligible for academic pricing.  If you need VMware pricing you should contact Tom Templin at Ciber: ttemplin at ciber dot com. 

Tom's one of our long time uber IT volunteers at GCC and has gotten us amazing pricing on all sorts of software and hardware over the years.  Document courtesy of Tom.

Download VMware Non Profit Customer Definitions Jan 2010 FINAL

First Ever EqualLogic User Conference

If you’re an EqualLogic storage customer, like GCC, you really should try to make it to the first ever national EqualLogic User Conference March 1-3 in Dallas.  We purchased our 1st EqualLogic array back in July ‘06 and we’ve been a big cheerleader for the product ever since.

From the conf website:
The main conference on March 1-3 is packed with educational sessions where distinguished technical staff, EqualLogic product managers, tech support gurus, peer users and Dell executives will deliver:

• Primers on networking best practices for iSCSI SANs
• Tips on how to avoid common technical issues
• Deep dives into the EqualLogic software tools such as SAN HeadQuarters
• Data protection strategies for virtual environments and production applications
• Product and feature demonstrations
• Sessions on 10 GbE networking and SSD technology
• Case studies featuring your peers at leading organizations
• Lessons on how to leverage EqualLogic for cutting edge deployments

I’m also excited to hang out with several other church IT EqualLogic users from Watermark Church and Churches in Covenant just to name a few.  And this post wouldn’t be complete without the obligatory plug for VR6 Systems whom we all, and many other churches, have purchased EqualLogic gear from ;-)

I’m also hoping to get some tasty TX BBQ … I have found memories of Hard 8 BBQ from the Texas CITRT event last Feb … Mmmmm!

VMware Academic Pricing now available to Churches!

Boy, we’ve been lobbying for this for 3+ years … and yesterday VMware announced it’s finally giving non-profit organizations access to academic pricing! w00t!!!

Of course academic pricing is still no where near the crazy discounts we get from Microsoft, but it’s a great start!  Thank you VMware!!

This means on most products expect about 40% off the list price.

Here’s some pricing info for comparison:

VMware vSphere 4 Essentials Bundle for 3 hosts (Max 2 processors per host and 6 cores per processor)
List - $879 … 1yr support $116
Academic - $527 … 1yr support $70

VMware vSphere 4 Essentials Plus Bundle for 3 hosts
List - $2995 … Gold support $629 … Platinum support $749
Academic - $1797 … Gold $377 … Platinum $749

VMware vSphere 4 Standard for 1 processor
List - $795 … Gold $273 … Platinum $323
Acad - $477 … Gold $164 … Platinum $323

VMware vSphere 4 Advanced for 1 processor
List - $2245… Gold $472… Platinum $561
Acad - $1347 … Gold $283 … Platinum $561

VMware vSphere 4 Enterprise for 1 processor
List - $2875… Gold $604… Platinum $719
Acad - $1725… Gold $362… Platinum $719

VMware vCenter Server 4 Standard for vSphere (Includes Orchestrator and Linked Mode)
List - $4995… Gold $1049… Platinum $1249
Acad - $2997… Gold $629… Platinum $1249

Eligibility requirements and who we bought from

ESXi Officially Free and Available

As was rumored last week ... the latest VMware hypervisor is now available for free.

Features & Benefits

Virtualize processor, memory, storage and networking resources into multiple virtual machines with VMware ESXi, an enterprise-class hypervisor with a thin 32 MB footprint for added security and reliability.

VMware ESXi is the virtualization platform customers build upon for continuous application availability and infrastructure security. With VMware Infrastructure 3, customers can add the benefits of server consolidation, automatic load balancing and business continuity to ESXi.