My boss took me and my senior staff peers in the Operations Division bowling a couple of weeks ago. What a nerdy looking group! Turns out, Kelly was a bowler in high school, so she smoked us. I broke 100 both games, which isn’t bad considering it’s been a few years since I last bowled.

Left to right: Dick Cooper, Director of Facilities; yours truly; Susan Bell, Director of Guest Services; Kelly Williams, Director of Finance; Brent Messick, Managing Executive Director and CFO (our boss); Lisa Viehweg, Human Resources Director, Nancy Hines, Director of Resurrection Resources (book store manager); Travis Morgan, Director of Operations for Resurrection West

Er, I would have gotten your e-mail about SonicWALL acquiring Aventail, except my SonicWALL anti-spam software (MailFrontier) junked it!

The memorial service for Kelsey Smith was this afternoon.

Nick Kastelan, one of the guys in our media department (Saving Grace Productions – SGP), had worked in TV news in Kansas City before joining our staff. He and Peter Metz, our Director of Communications, handled media relations for the visitation last night and the service today. Working with our facilities and security teams, they did an awesome job with the media.

They selected a perfect location where the media could have the access they needed without being in the way. They blocked off the street between our two main buildings, making it available exclusively for the TV trucks and other media vehicles. This street is conveniently located outside our backstage entrance, right next to our video control room.

SGP generated a live audio/video feed from the output of our audio mixer and video switcher and hooked it to a distribution box in the control room. Each of the TV stations ran their own heavy-duty cables from their truck, along the ground, under the door and into the control room. The complete service video is here. You can see news reports using our video here, here, and here. The family allowed one local reporter inside the building. You can see his report here.

SGP also set up a media room backstage where print media people could watch the service on video monitors. We provided power strips and they had access to our guest WiFi.

This was an excellent way to handle a very delicate situation. The media were able to get the content they needed without being visible to mourners inside the building. I attended the service myself and never felt I was on display because the only video available to the media was determined by our video director in our control room.

Today I saw a very special example of the use of technology in ministry. Props to SGP!

I mentioned that my wife, Laura, was the only pastor present on Wednesday afternoon when Kelsey’s family learned that she had been found murdered. A few minutes later, pastors from the Smith’s church, Hillcrest Covenant, arrived. Laura quietly suggested to them that when it came time to begin discussing memorial service plans, the family should consider having the service at Church of the Resurrection. At the risk of sounding immodest, I truly believe Resurrection is the only church in the area with the capability to handle a funeral like this. Our main sanctuary seats 3,000+ and we have extensive video production capabilities to be able to generate a pool feed for the media while keeping them mostly out of sight.

Wednesday afternoon I let our upper management know that the suggestion had been made. On Thursday morning several of our staff came to me and said that all of the teams required to support the service were on board and eager to help, if asked.

Then Friday the word came that the family had decided to take our suggestion. Normally, the church has fees for outside funerals, but they were completely waived in this case, even though a huge effort will be required. This will be a major event in the middle of the work day (2 pm Tuesday) when volunteer availability will be low.

The planning began immediately Friday, showing great foresight and attention to detail. Our experience with the Ali Kemp funeral five years ago has been invaluable. From traffic, to parking, hospitality, facilities, security, and A/V we have vast expertise and resources to handle every detail so the family and friends can simply be in the moment and celebrate Kelsey’s life. This is the Body of Christ functioning as should (but too rarely does) – joining together across congregational and denominational boundaries to serve in a very special way in a time of great need. Although I will not have a personal role in any of this, I have never been prouder of Church of the Resurrection than I am right now. It is a special place to serve.

We built a site that went from launch to the #5 search term on Google and then to obsolete in three days. Here’s the (updated) story …

The first version of the Kelsey Smith website, made by a family friend on Monday and hosted at sitesled.com, was barely more than a web version of the “have you seen this person?” poster. As the friend added pictures to the site, it eventually did not perform well, even though it had only light traffic at the time.

Around 2:00 pm Monday my wife called me from the Smith home asking for help. We jumped right in and bought the domain findkelsey.com, quickly built a static HTML site (see mirror), and put it up on four servers at Vine Hosting, using DNS round robin for load balancing. The site went live around 5:00 pm and averaged 20+ page views per minute through the remaining waking hours of Monday. Not bad … and we were grateful to have the opportunity to help with the search for Kelsey in this small way. Little did we know we were about to experience an exponential explosion in site traffic unlike anything any of us had ever seen.

Through the day on Tuesday the traffic increased by a factor of 7 to an average of 140+ page views per minute. We got as many page views on findkelsey.com Tuesday, 160,000, as we get in an average month at www.cor.org. Our infrastructure held up very well and we felt that this level of interest in the site represented a tremendous return on our investment in ministry.

Then Wednesday morning traffic increased by another factor of 7 to 1,000+ page views per minute and we had a serious problem on our hands. As I mentioned, the traffic growth was so rapid, at first Vine Hosting thought we were experiencing a DDOS attack aimed at findkelsey.com. Every site they host was being slowed to a crawl. We quickly became concerned that the bandwidth consumption could result in a huge bill for overage charges. Looking back now, we think it’s likely the servers were just fine but Vine’s WAN link was saturated (which, by the way, seriously hampered our efforts to fix the problem as you can imagine). We discovered later that “findkelsey.com” had become the #5 search term on Google and that the site had 11,000+ other pages linking to it.

As a practical matter, findkelsey.com was very slow to display for most site visitors, if it displayed at all. Vine was in serious trouble and most people weren’t getting to the site anyway. Consequently, I felt the only good option was to take the site down from our servers at Vine and move it to Blogger, which is owned by Google. Only Google had the capacity to handle the traffic while being affordable (free). While we were moving the site, between 2:00 and 5:00 pm Wednesday, it was mostly inaccessible. Meanwhile, Vine had to do a great deal of work to restore full service for its customers while traffic slowly faded away as the DNS changes propagated. In an urgent situation like this, DNS propagation delays are terribly frustrating. There’s nothing like taking down a site, rebuilding it, and moving it WHILE it is getting bombarded with thousands of page requests per minute. It’s truly a mind boggling challenge.

During the three hours the site had been (mostly) down, Kelsey’s body was discovered and the news quickly spread through the media. Around 5:00 pm when we had the site rebuilt and the DNS redirected, immediately we again experienced the 1,000+ page views per minute we had seen in the morning. Just as immediately, expressions of love and support began pouring in through Blogger’s comment feature at the rate of 1 every 10 seconds. People came to the site from across the country and around the world to express their love, comfort, sorrow, and outrage. Those comments continued through Thursday and gradually tapered off to less than 1 every 5 minutes late Thursday night. In 30 hours the site received a total of 2,700+ comments that we hope will be a great encouragement to the family when they eventually have time to read them and reflect.

As the comments came flooding in, we realized that our ministry was not only to the Smith family to aid in the search and the immediate aftermath, but also to the public to provide a way for people to express themselves to the family. Other than the 1% we had to remove because they were inappropriate, the comments were overwhelmingly supportive. (Understandably, some expressed uncensored anger towards the perpetrator. Those we didn’t remove.)

Thursday afternoon we got word from the family that the term “Find Kelsey” no longer fit their situation. The volunteers during the search had nicknamed themselves “Kelsey’s Army” and that was the domain name the family wanted to use going forward. In light of the major challenges we had experienced with traffic, I advised against redirecting findkelsey.com to the new kelseysarmy.com. Instead, I posted a message at the top of the page referring people to the new site and indicating that comments would be closed.

And that’s how it ended. The site went live on Monday around 5:00 pm and was mothballed late Thursday night. It was in production just over three days, and yet the ministry value it created in that short time was immeasurable.

Where do I begin?

Many of you no doubt have already heard that Kelsey Smith was found murdered today. My wife, Laura, was at the Smith home when the detectives came with the news that a body had been found and it was likely Kelsey. The identification was later confirmed. Laura and my daughter, Beth, are attending a prayer service at Hillcrest Covenant Church as I write this. Prayers are flooding in from across the country and around the globe. Please add your prayers to them. I will have much more to say about this in the coming days.

Quite aside from the unspeakable human tragedy, my technical team and I learned a great deal through this. The traffic to www.findkelsey.com (see mirror of original HTML site) was 160,000 page views yesterday and grew exponentially this morning. By 10:00 am we had serious performance impact, essentially taking down Vine Hosting, the ISP where our web servers are located. Searches for findkelsey.com reached #5 on Google as reported by Google Trends. You read that right. Our little hand-built quickie site was the #5 search on Google today. (Right now “Kelsey Smith” is #3, “find Kelsey” is # 7, and “findkelsey.com” is #9.) The flood of legitimate traffic was so great, Glenn – the director of operations for Vine Hosting – at first thought it was a DDOS. In reality, it was all legit traffic. Too much traffic. Glenn is still dealing with the aftermath.

The only people with enough resources to absorb this kind of traffic is, that’s right, Google. Realizing the disaster we were causing for Vine, I made the command decision to totally take down all traffic related to findkelsey.com at Vine, even the DNS. Then, as rapidly as possible, we created a new site on Blogger (which is owned by Google), changed to a high-volume DNS host, and pointed the DNS at Blogger. As soon as the DNS change propagated, we started getting comments on our posts. In fact, my first indication that we finally had things working was when a comment appeared in my inbox. It took almost another hour to figure out how to get most traffic coming to us as it should. In the following 2.5 hours, 754 comments have come in from around the country, directed to Kelsey’s family. That’s one comment every 10-15 seconds. Unbelievable.

Today I had the experience of building one of the highest traffic sites on the Internet. I was building it WHILE it was being hit.

My daughter, Beth, was in band at Shawnee Mission West High School with Kelsey Smith, the 18 year-old girl apparently abducted from a Target parking lot here in Overland Park on Saturday night. The band has a very active parents’ organization called Band Aides West. Within 12 hours of Kelsey’s disappearance, Band Aids was taking the lead to organize the search and support the family. Band Aids asked my wife, Laura, to go to the Smith home yesterday and take a shift answering the phone. After arriving there, Laura became aware of a need for a web site about Kelsey that could handle the load coming from national news exposure, including Greta Van Susteren and Nancy Grace, as well as extensive coverage from the local media.

Through the intense efforts of Brian, Leo, and Chuck http://www.findkelsey.com/ (see mirror of original HTML site) went live yesterday around 5 pm. Glenn Kelly and Vine Hosting are providing hosting at no charge. Glenn even missed part of his daughter’s second birthday party to help set up the site on multiple servers to handle the load.

The site is linked from the home page of the Kansas City Star, the story about Kelsey on Fox News, and many other sites. It is also on flyers being posted all over town. The site is now averaging 2.6+ page views per second during waking hours.

Please stop what you are doing right now and pray for Kelsey, for her family, for the police, and for all of the volunteers who are supporting them and helping search.

Our VMware ESX project is well underway. Ian, our server/network admin, and I have been spending a great deal of time communicating with our ESX consultants and documenting both our existing configuration and our intended configuration when the project is complete. A detailed plan is beginning to emerge.

While we’re at it, we decided to upgrade the power distribution and core network switching in our data center. We have temporarily stacked two APC 2200 UPSes, the new Dell/EMC AX150i SAN with its dedicated UPS, and both the existing and the new Dell 2950 servers. All of this will be permanently installed in our new Dell cabinet when it arrives this week. We also installed a second HP 4104 switch to provide redundancy for both the SAN and the primary network.

Our consultants have loaded ESX Server onto the new 2950 server. The next step is to install Virtual Center onto a desktop computer that will be used for management. Once that is done, we will avail ourselves of Dell’s remote install assistance they bundle with every AX150 sold.

Our consultants will be in all day Friday and Saturday to complete as many of the remaining tasks as possible. Our goal is to have at least one or two virtual servers running on the new 2950 by Saturday.