Our Resurrection web sites are currently hosted on a pair of dedicated servers at Vine Hosting in Philadelphia.  (Vine Hosting is affiliated with Web Empowered Church, which has received major funding from the Methodist Foundation for Evangelism.)  Vine Hosting’s data center is colocated with XO Communications in a massive carrier hotel at 401 N. Broad.  From there they have direct access to bandwidth from XO, Verizon, Level 3, and Internap.  This is a true, world-class data center facility.  I don’t recall having a single outage or service interruption with Vine since they moved into the carrier hotel. 

Through our time with Vine Hosting, Glenn Kelley, its founder and high-energy leader, has become a trusted friend and partner in ministry.  No one is more passionate about sharing the gospel through the use of technology than Glenn.

So why are we moving?

Before I settled on Vine in 2005, I did an exhaustive search of hosting facilities in Kansas City.  I found a couple of decent ones, but nothing at the level I was seeking.  So we eventually hooked up with Vine and the ensuing partnership has been excellent as I mentioned.

Three years later, the situation has changed.  Kansas City now has a world-class carrier hotel of its own.  The 70 year-old, 26 story Bryant Building at 1102 Grand sits right on AFS’s KC metro fiber ring.  All of the major telecom companies in KC are connected to the metro fiber ring and all of them except Sprint have a presence in 1102 Grand.  Fiber comes into the building via diverse underground vaults.  The building gets power from two separate KCP&L substations, has a 2 MW diesel generator in the basement, and provides centralized UPS power for those tenants that want it.  To keep everything cool, the building has 1100 tons of cooling.  Physical security is provided by card and code access and monitored video surveillance.

 

Ian and I got a tour in March and knew right away we were seeing something special.  Our wheels began turning.  We immediately recognized the potential for 1102 Grand to be the hub of a regional network connecting all of our church facilities.  We already have staff located in Leawood, Overland Park, and Olathe.  Our next campus will be in downtown KC, just two blocks from 1102 Grand.  We have land in Cass County for a retreat center.  And we’re beginning to dream about a campus several miles south of our Leawood location.  Over the next few years we will need to come up with an elegant, cost-effective way to connect all of these locations.

Next, we thought about 1102 Grand as a potential disaster recovery site.  Business interruption/disaster recovery planning has been on my radar since I joined the staff in 2003.  Each year we have made incremental improvements in our DR posture.  For example, we now use Iron Mountain for offsite backup tape storage.  But I could never find an affordable, nearby DR site.  Until now.

The more we thought about it, this opportunity was simply too compelling to pass up even though we are extremely happy with Vine Hosting.

Move in

Yesterday we moved in.  Below is Ian setting up his web cam before starting the install.  The cage behind him is part of the carrier-neutral Meet Me room operated by the building owner where tenants, including carriers, ISPs and end users like us, connect with each other.  Our provider is KCNAP.  They have a patch panel in the Meet Me room.  From there it’s just a short cat 5 cable drop into our cabinet.  We have the bottom third of the cabinet in the background (they charge less for the bottom third and we’re very price-sensitive!).

Below is the front of our cabinet.  Note the broom in Ian’s hand to sweep out the dust and debris from the floor before installing any equipment.

Below is the back of our cabinet.  A 3U-tall power distribution/fan unit is at the top.  Fully redundant power (two city power grids, diesel generator, and UPS) comes into the cabinet in the outlet box at the left.

Below is what it looked like after physically installing the four 1U servers, network switch, KVM switch, and a flat-panel monitor.  The rack rails we got from Dell won’t work in this cabinet, so we’ll have to get the right ones in order to permanently install the Dells.  You can see them temporarily sitting on top of an Appro server donated to us by Tradebot Systems.  Note the jack hanging down in the upper right corner of the picture.  That’s the drop from KCNAP.  They are multihomed to multiple tier 1 providers via BGP.

We have some high-quality Cat 6 patch cords that are too short for our data center on campus so we brought them along.  Too short for our main data center and too long for this cabinet!  So Ian did what all good techs do: he broke out the tie wraps!

Although the physical installation went very smoothly (except the Dell rack rails), we had to leave yesterday afternoon before we could establish network connectivity.  Ian was having trouble with the pfSense firewall/router he configured.

We have until the end of June to move all of our existing sites and services from Vine into the new data center. We’re very excited about the new capability this represents for us.

Don’t let the mild-mannered appearance of this group fool you.

We’re in our last day of Arena training before go live on Tuesday next week. Chuck is teaching the Report Builder in SQL Reporting Services. With these eager students, his first exercise was to simply select first and last name and then run the report. This resulted in 9 simultaneous attempts to report all 60,000 people in our database. It soon became clear we had a problem.

We’re running SQL Server in an ESX guest VM. This has been our configuration for 9 months with no issues. The CPU on the host Dell 2950 server (total of 8 cores) spiked for a minute or two but then settled back down. Yet the server was still whimpering in the corner. No one could do anything. We had a class full of students trying to use a server that was dead for all practical purposes.

Turns out SQL Server itself was okay. IIS (running on the same ESX guest VM with SQL Server) was gulping enormous slurps of RAM to generate the multiple thousand-page reports. Windows responded by trying to expand its page file. Under ESX, this is not a good scenario. We had an unscheduled outage on our main database server in the middle of the work day. Clif not happy. The good thing is, you add RAM to a guest VM with just a few mouse clicks and a reboot, which Ian did. After a 20 minute delay, the class resumed. From then on, Chuck made sure to run the examples himself with everyone else just watching.

With the number of guest VMs we have running on that host, Ian thinks we need to add to the 16 GB of physical RAM we already have on that box. This is why IT is so FUN!