Where do I begin?

Many of you no doubt have already heard that Kelsey Smith was found murdered today. My wife, Laura, was at the Smith home when the detectives came with the news that a body had been found and it was likely Kelsey. The identification was later confirmed. Laura and my daughter, Beth, are attending a prayer service at Hillcrest Covenant Church as I write this. Prayers are flooding in from across the country and around the globe. Please add your prayers to them. I will have much more to say about this in the coming days.

Quite aside from the unspeakable human tragedy, my technical team and I learned a great deal through this. The traffic to www.findkelsey.com (see mirror of original HTML site) was 160,000 page views yesterday and grew exponentially this morning. By 10:00 am we had serious performance impact, essentially taking down Vine Hosting, the ISP where our web servers are located. Searches for findkelsey.com reached #5 on Google as reported by Google Trends. You read that right. Our little hand-built quickie site was the #5 search on Google today. (Right now “Kelsey Smith” is #3, “find Kelsey” is # 7, and “findkelsey.com” is #9.) The flood of legitimate traffic was so great, Glenn – the director of operations for Vine Hosting – at first thought it was a DDOS. In reality, it was all legit traffic. Too much traffic. Glenn is still dealing with the aftermath.

The only people with enough resources to absorb this kind of traffic is, that’s right, Google. Realizing the disaster we were causing for Vine, I made the command decision to totally take down all traffic related to findkelsey.com at Vine, even the DNS. Then, as rapidly as possible, we created a new site on Blogger (which is owned by Google), changed to a high-volume DNS host, and pointed the DNS at Blogger. As soon as the DNS change propagated, we started getting comments on our posts. In fact, my first indication that we finally had things working was when a comment appeared in my inbox. It took almost another hour to figure out how to get most traffic coming to us as it should. In the following 2.5 hours, 754 comments have come in from around the country, directed to Kelsey’s family. That’s one comment every 10-15 seconds. Unbelievable.

Today I had the experience of building one of the highest traffic sites on the Internet. I was building it WHILE it was being hit.

My daughter, Beth, was in band at Shawnee Mission West High School with Kelsey Smith, the 18 year-old girl apparently abducted from a Target parking lot here in Overland Park on Saturday night. The band has a very active parents’ organization called Band Aides West. Within 12 hours of Kelsey’s disappearance, Band Aids was taking the lead to organize the search and support the family. Band Aids asked my wife, Laura, to go to the Smith home yesterday and take a shift answering the phone. After arriving there, Laura became aware of a need for a web site about Kelsey that could handle the load coming from national news exposure, including Greta Van Susteren and Nancy Grace, as well as extensive coverage from the local media.

Through the intense efforts of Brian, Leo, and Chuck http://www.findkelsey.com/ (see mirror of original HTML site) went live yesterday around 5 pm. Glenn Kelly and Vine Hosting are providing hosting at no charge. Glenn even missed part of his daughter’s second birthday party to help set up the site on multiple servers to handle the load.

The site is linked from the home page of the Kansas City Star, the story about Kelsey on Fox News, and many other sites. It is also on flyers being posted all over town. The site is now averaging 2.6+ page views per second during waking hours.

Please stop what you are doing right now and pray for Kelsey, for her family, for the police, and for all of the volunteers who are supporting them and helping search.

Our VMware ESX project is well underway. Ian, our server/network admin, and I have been spending a great deal of time communicating with our ESX consultants and documenting both our existing configuration and our intended configuration when the project is complete. A detailed plan is beginning to emerge.

While we’re at it, we decided to upgrade the power distribution and core network switching in our data center. We have temporarily stacked two APC 2200 UPSes, the new Dell/EMC AX150i SAN with its dedicated UPS, and both the existing and the new Dell 2950 servers. All of this will be permanently installed in our new Dell cabinet when it arrives this week. We also installed a second HP 4104 switch to provide redundancy for both the SAN and the primary network.

Our consultants have loaded ESX Server onto the new 2950 server. The next step is to install Virtual Center onto a desktop computer that will be used for management. Once that is done, we will avail ourselves of Dell’s remote install assistance they bundle with every AX150 sold.

Our consultants will be in all day Friday and Saturday to complete as many of the remaining tasks as possible. Our goal is to have at least one or two virtual servers running on the new 2950 by Saturday.

It’s finally time for me to upgrade to a smartphone. I’ve never had a smartphone before. I used a Palm Pilot for almost four years until it died a couple of months ago and my cell is a 5 year-old workhorse that’s nothing more than a phone. I definitely need the phone to have EVDO Rev. A for use as a modem for my laptop. I use Sprint and I’m not interested in Blackberry. I’ve read a lot of reviews, narrowing my short list down to the Treo 755p, the Motorola Q, or possibly even the PPC 6700.

I need real-world advice from smartphone users. Should I assimilate into to the Treo club? Should I go sleek with the Q? Or should I put a priority on WiFi and choose the PPC 6700? Should I wait for Windows Mobile 6? Should I stick with Palm OS? Help!

Many of you know that my wife is a church planter. I’ve been struggling with finding a good way to get high-speed Internet service at the church since we moved into the building more than two years ago. DSL service is not available in that semi-rural location and cable is too expensive.

In late January Sprint completed their roll out of EVDO Rev. A service in the Kansas City area. The first day it was available I got a Pantech card and started testing/using the service everywhere I went, including the church. Bottom line: it rocks. It gives a user experience very similar to DSL or even cable for everything except large file downloads. I’m seeing speeds in the neighborhood of 750+ Kb/s down and 250+ Kb/s up (though it does vary somewhat by location and time).

So having passed the performance and reliability test, I decided it would work well for the church until a faster, affordable land-line service becomes available. Yesterday I got a Linksys WRT54G3G-ST router that accepts an EVDO card and/or a wired broadband connection, routes, has a 4-port switch, and does WiFi. In other words, it’s a WiFi router that can use EVDO to get to the Internet. This thing is very cool and it works quite well, but it was a pain to set up. Turns out you need to upgrade the router firmware before it can see/use the Pantech card, but finding the upgraded firmware on the Linksys site is nearly impossible. (By the way, it’s here.)

Now … think about how this could fit in to your disaster recovery planning. This little guy is a completely mobile WiFi hotspot. All you need is electrical power and EVDO coverage. (If it can’t find a Rev A tower it will drop back to the the older EVDO Rev 0 or 1xRtt.) Admittedly, the bandwidth is pretty minimal for a subtantial number of users trying to share an Internet connection, but it could be set up instantly almost anywhere. One of these and some computers could provide an instant office with zero lead time to order DSL or T1 service to a location.

In the event of disaster, you could go to your local Sprint or Verizon store, sign up for service, and walk out within an hour with an activated card. Getting an EVDO router could be a bit tougher to find on zero notice. Stores like CompUSA are carrying some of these, but not all are compatible with all EVDO cards and services. You need to verify compatibility prior to purchase. We tried a D-Link router we bought at retail with no success and had to take it back.

Also … think about how this could be used to provide backup for a critical Internet link in a remote location such as a second campus. I configured the Linksys to give priority to the wired WAN link and connected that link to my cable modem at home. Then I connected my laptop to the router via WiFi and I was on the Internet through the cable. To test the failover, I pulled the plug on the cable. Instantly the Linksys connected to EVDO and continued my sessions without interruption. Neither my Skype nor my MSN IM connection dropped. Sweet!

I’m very excited to announce that our senior pastor, Adam Hamilton, will be coming to the Fall RoundTable for a 30-minute Q&A session. Adam is an amazing man who started this church in 1990 and has seen it grow to more than 12,000 adult members and average worship attendance in the 7,000 range. I am grateful he has accepted my invitation to spend some time with us during one of the busiest weeks of his year. It will be very interesting to get Adam’s insight into the value of IT in the church and how we IT people can better serve the mission, minisry, and senior leadership of our churches.

The next KC-area church IT lunch will be at Chili’s in the Merriam Town Center mall at Antioch and Johnson Drive in Merriam, Kansas on Thursday, May 31, at 12:30 pm. See you there!

Tony Dye hit the nail on the head when he said that the consistent sentiment of Church IT RoundTable participants is that we should “include more people and keep it small.” That paradox still makes me chuckle every time I think about it, but it does set out a vision that I believe (all humor aside) we should pursue. How could we do it?

In my original invitation to come to Church of the Resurrection for the Fall RoundTable, I explained a half-baked idea about how the RoundTable can grow while retaining the best aspects of being small. The essence of the idea is to have roundtables in cities across the country and then link them technologically a couple of times each year for a national roundtable of roundtables. Now I’d like to update you on how that half-baked idea is maturing into something that could be truly workable.

Our plan for the Fall RoundTable is to try this once with everyone located in our building. We think that is a good idea because it will allow us to test our facilitation technique and technology with all of us physically together. (Prototype? Cool!) If something goes horribly wrong, we can scramble and put everyone together in one large room and salvage the event. (Contingency plan? Cool!)

We are going to break up the large group into four separate classrooms, each of which will accommodate 20 participants and a few observers. With this idea we will be able to handle as many as 80 participants and 20 vendors/observers before we have to close registration. (I’ve been told this is similar to the approach taken by Leadership Network, but I’ve never been to their events so I have to take the word of others on that.)

Each classroom will have two projectors: one showing content from the conversation in that room, and the other showing content from all of the other rooms. Each participant will use their own laptop to join an Adobe Connect meeting with everyone else in their room. Through Connect, anyone in the room will be able to show content (web sites, pictures, diagrams, PowerPoints, etc.) on the primary projector. United Methodist Communications (UMCOM) has agreed to let us use their Adobe Connect system at no charge, and Sean McAtee from UMCOM Tech Shop will be here to assist us with his deep expertise in Connect.

So that we are not limited to interacting only with the participants in our own room, the secondary projector will show content from the conversations happening in the other rooms. We’re still in the process of figuring out exactly how this will work. We will run some tests within the next couple of weeks to evaluate various ideas. I’ll post once we have a clear picture of this in order to solicit your feedback.

When I suggested this idea to Tony, he said, “Sounds exciting, crazy, and even a bit prone to potential failure. Perfect! 🙂” Tony’s right. This is both exciting and risky. But I figure, if a bunch of techno-geeks can’t make something like this work, who can?

I’m very excited that so many of you have commented on my invitation post indicating your interest in coming to the Fall Roundtable, October 3-4, 2007. We’re actively working on detailed plans so we can open registration.

We will start on Wednesday around 9:00 am and end Thursday at 5:00 pm. We’re also planning a get-together for Tuesday (October 2) afternoon around 3:00 pm for those who are flying in on Tuesday and would like to get a facilities/IT tour of Church of the Resurrection. After the roundtable is Resurrection’s annual Leadership Institute, October 5-6. Leadership Institute is our biggest conference of the year. If you are able to stay, I know it will be a blessing to you as well.

For any vendors that would like to attend, sponsorship opportunities are available. Please e-mail me for info (clif.guy at cor dot org).

Watch this blog for more deets to follow soon. It’s going to be an AWESOME event!