Life Church Internet Campus benchmarking day 1

I’m in Oklahoma City with Chuck Russell, Brian Slezak, and Andrew Conard to benchmark LifeChurch’s Internet Campus by observing them in action and by meeting with Terry Storch (Digerati Pastor) and Brandon Donaldson (Internet Campus Pastor).

Our day began with attending the 10:00 service (“experience” in LifeChurch parlance) at the Edmond campus where the Global Operations Center and Internet Campus offices are located.

worship at LifeChurch Edmond

After the 10:00 experience, Terry took us up to the Global Operations Center where we got a high-level view of what was happening at all Life Church campuses, including the Internet Campus, during the 11:30 experience.

LifeChurch Global Operations Center sign

Terry Storch discusses the Internet Campus while it displays on the lower monitor

We then went over to the Internet Campus offices to chat with Brandon Donaldson while the 11:30 experience was underway.

Brandon Donaldson (Internet Campus Pastor) with his DELL laptop!

Terry’s boss, Bobby Gruenewald, dropped by.

Bobby Gruenewald and Terry Storch

We learned many interesting things and I’m sure have much more to learn when we meet again on Monday.

After hanging out with Brandon, Terry, and Bobby, we grabbed some lunch and went downtown to see the Oklahoma City National Memorial and Museum. In this photo you can see the 9:01 Gate with First Methodist Church in the upper right background. Being right across the street to the SE of the Murrah Building, the church was heavily damaged in the blast and played a key role in the recovery following the events of April 19, 1995.

01 Gate and First United Methodist Church

My apologies to Jason Reynolds and David Helbig of Christ Fellowship, who hosted Chuck Russell and me back in mid-June as we benchmarked the Internet Campuses of Christ Fellowship and Flamingo Road. Due to my laptop being stolen the night we arrived (while we were enjoying the beach), I never posted a single blog entry about our trip or what we learned there. I hope to correct that soon!

Should a congregation use technology?

Pastor Andrew Conard of our congregational care team at Resurrection is in a series of posts about the “business side” of the church.  I started to compose a comment to his post on technology and ended up with a post of my own.  So here are my thoughts in response to Andrew.

A local congregation isn’t a “business” in the sense of an organization that is operated with the objective of making a profit from the sale of goods or services.  Though its mission is quite different from that of a business, a congregation is an organization that should be just as purposefully operated as the most efficient and effective business.  We rightfully use different terminology when discussing congregational strategy, yet the parallels to business strategy are as numerous as they are obvious.  I believe God wants us to use our brains fully in pursuit of the mission He gave us.  When that means borrowing great ideas from business, we should do that to the fullest extent consistent with our mission.

So, to answer Andrew’s question, technology is largely created by profit-motive-driven businesses for the purpose of becoming more efficient, more effective, and more profitable.  Why wouldn’t a kingdom-driven congregation borrow those tools and techniques from business in order to become more efficient and effective in pursuing our God-given mission?

Once having decided to use technology, good judgment is required to apply it in a ministry setting.  Not every technology advances our Kingdom cause.  Not every shiny new thing is appropriate, effective, or affordable.  Even technologies that make a huge positive contribution commonly have a dark side that must be carefully managed.  WIth care, skill, and a willingness to invest, technology can be a major accelerator.  That’s why I feel called of God to use my secular technology skills in pursuit of Resurrection’s mission.  For a congregation to make optimal use of technology, it needs people like me to select, acquire, manage, and support it.  I am honored God has called me to serve in this way.

You’d think Amazon would be five 9s

Five 9s is telcom and data center lingo meaning 99.999% availability. It’s equivalent to 5 minutes of downtime per year. The PSTN (public switched telephone network) is designed to five 9s going back to the days of fanatical engineers at Ma Bell. Remember when we all had AT&T and Bell Telephone? The phone simply always worked.

Last week Amazon was down for a while. According to the info in this PC World article, it seems they average a couple of hours per year of downtime. That’s approximately 99.98% availability. Not five 9s. In fact less than four 9s.

This raises an important question: what would we do if there was an AWS outage on any critical service we might have there? Who would I call? Would I be able to get timely and accurate status updates to inform my boss or ministries that rely upon it? The whole thing gives me pause.

Our new web home

Our Resurrection web sites are currently hosted on a pair of dedicated servers at Vine Hosting in Philadelphia.  (Vine Hosting is affiliated with Web Empowered Church, which has received major funding from the Methodist Foundation for Evangelism.)  Vine Hosting’s data center is colocated with XO Communications in a massive carrier hotel at 401 N. Broad.  From there they have direct access to bandwidth from XO, Verizon, Level 3, and Internap.  This is a true, world-class data center facility.  I don’t recall having a single outage or service interruption with Vine since they moved into the carrier hotel. 

Through our time with Vine Hosting, Glenn Kelley, its founder and high-energy leader, has become a trusted friend and partner in ministry.  No one is more passionate about sharing the gospel through the use of technology than Glenn.

So why are we moving?

Before I settled on Vine in 2005, I did an exhaustive search of hosting facilities in Kansas City.  I found a couple of decent ones, but nothing at the level I was seeking.  So we eventually hooked up with Vine and the ensuing partnership has been excellent as I mentioned.

Three years later, the situation has changed.  Kansas City now has a world-class carrier hotel of its own.  The 70 year-old, 26 story Bryant Building at 1102 Grand sits right on AFS’s KC metro fiber ring.  All of the major telecom companies in KC are connected to the metro fiber ring and all of them except Sprint have a presence in 1102 Grand.  Fiber comes into the building via diverse underground vaults.  The building gets power from two separate KCP&L substations, has a 2 MW diesel generator in the basement, and provides centralized UPS power for those tenants that want it.  To keep everything cool, the building has 1100 tons of cooling.  Physical security is provided by card and code access and monitored video surveillance.

 image

Ian and I got a tour in March and knew right away we were seeing something special.  Our wheels began turning.  We immediately recognized the potential for 1102 Grand to be the hub of a regional network connecting all of our church facilities.  We already have staff located in Leawood, Overland Park, and Olathe.  Our next campus will be in downtown KC, just two blocks from 1102 Grand.  We have land in Cass County for a retreat center.  And we’re beginning to dream about a campus several miles south of our Leawood location.  Over the next few years we will need to come up with an elegant, cost-effective way to connect all of these locations.

Next, we thought about 1102 Grand as a potential disaster recovery site.  Business interruption/disaster recovery planning has been on my radar since I joined the staff in 2003.  Each year we have made incremental improvements in our DR posture.  For example, we now use Iron Mountain for offsite backup tape storage.  But I could never find an affordable, nearby DR site.  Until now.

The more we thought about it, this opportunity was simply too compelling to pass up even though we are extremely happy with Vine Hosting.

Move in

Yesterday we moved in.  Below is Ian setting up his web cam before starting the install.  The cage behind him is part of the carrier-neutral Meet Me room operated by the building owner where tenants, including carriers, ISPs and end users like us, connect with each other.  Our provider is KCNAP.  They have a patch panel in the Meet Me room.  From there it’s just a short cat 5 cable drop into our cabinet.  We have the bottom third of the cabinet in the background (they charge less for the bottom third and we’re very price-sensitive!).

Ian preparing for install

Below is the front of our cabinet.  Note the broom in Ian’s hand to sweep out the dust and debris from the floor before installing any equipment.

Resurrection's cabinet

Below is the back of our cabinet.  A 3U-tall power distribution/fan unit is at the top.  Fully redundant power (two city power grids, diesel generator, and UPS) comes into the cabinet in the outlet box at the left.

Resurrection's cabinet

Below is what it looked like after physically installing the four 1U servers, network switch, KVM switch, and a flat-panel monitor.  The rack rails we got from Dell won’t work in this cabinet, so we’ll have to get the right ones in order to permanently install the Dells.  You can see them temporarily sitting on top of an Appro server donated to us by Tradebot Systems.  Note the jack hanging down in the upper right corner of the picture.  That’s the drop from KCNAP.  They are multihomed to multiple tier 1 providers via BGP.

Resurrection's cabinet with servers installed

We have some high-quality Cat 6 patch cords that are too short for our data center on campus so we brought them along.  Too short for our main data center and too long for this cabinet!  So Ian did what all good techs do: he broke out the tie wraps!

Resurrection's cabinet with servers and interconnect cables

Although the physical installation went very smoothly (except the Dell rack rails), we had to leave yesterday afternoon before we could establish network connectivity.  Ian was having trouble with the pfSense firewall/router he configured.

We have until the end of June to move all of our existing sites and services from Vine into the new data center. We’re very excited about the new capability this represents for us.

Technorati Tags:

Not keeping up with the price of gas

I bought gas at the Phillips 66 station in Parkville last night.  The van tank holds approximately 23 gallons and it was nearly empty.  Price was $3.799, so that’s an $85 fill up.  Towards the end of the fill up, I noticed the flow slow and then stop exactly on $75. Apparently the pump had authorized my card for $75 and it wouldn’t allow me to pump any more.  I had to complete that transaction and then initiate a second one to pump the last $10.

At $2/gal, $75 would fill any tank except the largest commercial vehicles.  At $3.799, $75 won’t fill an ordinary Ford Freestar.  You think they should change the pump software to authorize a higher amount?

Plan the celebration too

The next time you have a large project that takes months to plan and execute, be sure to plan the celebration too. For the Arena implementation project we recently completed, we planned two celebration events.

First, we invited all staff to an ice cream social in the Student Center on the afternoon of go live. We took Shelby down starting at noon. At 2:30 pm we brought Shelby back up (now in a limited role), turned on Arena for all staff, and started the party. At the party we used the Internet cafe computers for staff to log in and try a few simple things in Arena. The timing turned out to be really cool. By having the party exactly as the system was going live, we weren’t buried in help desk calls and it was too soon for anyone to have had a negative experience. Bonus!

For the two weeks immediately after go-live, we beefed up our help desk by increasing the hours of our contractors, Philip and Leo. This allowed us to have a “dispatcher” physically at the help desk at all times to take calls, e-mails, and work orders.

When this period was over, we had a second celebration. This time it was just the IT Department (Ian’s daughters crashed the party). Six months ago Brian moved to a house just 3 miles from the church. We decided to seize the opportunity to invite ourselves over to sit on his back deck and just relax. We all brought something to put on the barbecue grill. God provided a spectacular spring day and a good time was had by all. Here are a few pictures.

IMG_-095-web

IMG_-104-web

IMG_-031-web

Yours truly

IMG_-118-web

Travis

IMG_-077-web

Leo

IMG_-074-web

Brian

IMG_-042-web

Ian

IMG_-040-web

Philip and Jeremy

Technorati Tags:

Fall RoundTable, yeah baby!

image

Attention all church IT people.  The Fall 2008 Church IT RoundTable will be hosted by Trace Pupke at Seacoast Church in Mount Pleasant, South Carolina (Charleston area) October 8-10!

Seacoast is among the largest, fastest growing, and most innovative churches in the country. Don’t miss the opportunity to see and learn about this awesome church while connecting with your fellow church IT people.

I will be there along with Brian and Jeremy from my team at ResurrectionI strongly urge you to join us in Charleston to meet a bunch of amazing people and find out if this for you.  We learn a lot from each other and, more importantly, we draw inspiration from each other.  To understand what I mean by that, read my posts after the Fall 2007 RoundTable and the Spring 2008 RoundTable.

Trace has set up a separate blog just for the Fall 2008 CITRT. Check it out here and subscribe to the feed in order to keep up to date with all the details as they come out.

I’m jazzed!  Are you?

Technorati Tags:

IDF rewiring project

We’ve been working with our Facilities Dept. since March on a new office suite for our pastors of caring ministries who handle the traditional "pastoral" functions such as sacraments, weddings, funerals, hospital visits, counseling, and worship leadership.  Over the last 5 years this group outgrew one office and spilled over into two additional offices in separate parts of the building.  We hired a contractor to come in and remodel a set of classrooms into an office suite large enough to consolidate all three groups of staff. 

The part of the building where the new suite is located is served by IDF 2E. Like the other IDFs in the older East Bldg., 2E was originally wired by well-meaning volunteers who had no concept of professional cable installation.  Everything was a tangled, unlabeled, undocumented mess.  There was a wall-mounted half rack for the data patch panel and a traditional telecom mounting board with a 66 block for the voice.  There was no way to expand the existing design to accommodate the additional 40+ voice/data pairs for the new suite.  Plus, the long-term plan is to convert that entire wing of the building from classrooms into offices.  So it made sense to completely rebuild the IDF to make it neat, well-documented, and expandable.

The electrical contractor for the new wing installed all the jacks and pulled the new cable to the IDF.  They installed a standard 2-post rack and dropped the cable through the ceiling in a pair of large sleeves.

Unfortunately, I didn’t think of taking pictures until halfway through the project.  Here is a typical tangle of four different colors of existing voice wires in the ceiling above the IDF.  This is actually much better than it started because by the time I took this we had already cleaned up all the data cables.

IMG_1850-web 

Here is Ian working on the new rack.  All of the old data lines have been moved and he is almost finished with the new data lines.  You can see the fiber tray at the top, a rack-mounted power strip, two HP 2650 switches, and four rows of patch panels.  All the new cabling is Commscope UltraMedia Cat 6 – blue for data and white for voice.  The unterminated bundle of white voice lines is at the right.

IMG_1842-web

We used the beautiful (but expensive) Panduit Cat 6 RJ45 jacks and their companion patch panels.  You can see how Ian had carefully labeled each wire with the new numbering scheme we will use throughout the wing served by this IDF.

IMG_1843-web

The Panduit patch panel allows you to connect a jack to each wire separately and then snap the jacks into plastic brackets holding 4 jacks each.

IMG_1844-web 

This is the almost finished rack.  All of the new voice lines have been terminated.  We still need to move all of the old voice lines.  Notice the really cool floor plan pinned to the wall showing every jack location and its number.

IMG_1854-web 

IMG_1856-web

Technorati Tags: