At Church of the Resurrection we’re working to get better at visual communication. Visual communication is much more effective than communication with words alone. The problem is, most of us (me included) spent the majority of our education years learning how to communicate with words. And most of us (me included) have spent the majority of our careers communicating with words — memos, e-mails, talking with co-workers, speaking in front of groups, etc. We need to learn a new skill: visual communication.

Kathy Sierra is one of my favorite visual communicators. Her recent post on creating graphics illustrates why. Kathy, thanks for helping us with this. (Notice that even this post on visual communication has no visuals. See? I need to learn it too.)

Most, if not all, pastors I know have thought the same things Perry shares in his post today. They agree with Perry. Acting on those ideas is what they find difficult. (Everyone in my family is a pastor except me, so I have a lot of opportunity to know what’s on their minds.) Thanks, Perry.

In case the suspense has been killing you, yes the awful problems we had with SQL Server and our Shelby database are finally over. We’re new now running Shelby 5.6.2009 on SQL Server 2005. It’s rock stable and noticeably faster (though still not fast enough by a long shot). Praise the Lord!

Of course, we have had a number of issues with 5.6.2000 and 5.6.2009, which is hardly surprising given Shelby’s track record, but all of that seems trivial compared to the database pain that has been relieved. Can I get an amen?

We bought a new server (a shiny Dell 2950 with Windows 2003 x64 and SQL Server 2005) back in May with the idea of moving Shelby and our other SQL databases to it in a planned, smooth way in June or July. We needed to free up a server for our new Rez West office so we figured it was a great opportunity to upgrade the server hardware and maybe speed up Shelby a good bit.

Due to a number of delays in the Rez West office completion, we had plenty of time to play with the new server. As June melted into July, we systematically moved our non-Shelby databases to the new server and started configuring it for Shelby.

When we updated to Shelby 5.6 in early August, massive performance problems followed. After some investigation, we came to the conclusion that 5.6 placed enough additional resource demands on the server that the old guy, a Dell 2500, wasn’t getting the job done any more.

Before we could plan a cut over, the real adventure began. The old server crashed on a Wednesday morning, a time of the week when Shelby is typically under its heaviest load. In response, I divided our IT staff into two teams: one team would work on repairing the old server, while the second team would work on moving to the new server. We would go with whichever team finished first. In the course of that, we discovered that Shelby 5.6 still wasn’t compatible with SQL Server 2005, despite the fact that we had been told that the May 2006 update would be compatible. So we finished restoring the old server (the crash had been caused by Windows registry corruption) and after a 6-hour outage we had no choice but to stay there – as painfully slow as it was.

By early October, we were faced with a dilemma. We had to move off of the old server to free it up for Rez West (not to mention the fact that it was so slow it was nearly unusable), but we still didn’t have a SQL 2005-compatible version from Shelby and we already had databases running on SQL 2005 – there was no going back. That’s when we got the idea to use the free version of VMware to create a virtual server for Shelby that would be configured with SQL 2000 and run on the new 2950 server. Our thought was to run this configuration temporarily until Shelby’s October update, which was promised to be SQL 2005-compatible. Seemed like a brilliant idea at the time. Little did we know.

We have been on that configuration for the last five weeks, suffering from massive instability the entire time, including another server crash causing a 12-hour unplanned outage two weeks ago. The source of our grief was SQL 2000, not Shelby, but the user experience was that Shelby would randomly get disconnected from its database and start throwing errors. Shelby error handling is an awful mess of infinite loops, so the user’s only recourse was to bring up Task Manager, shut down the client, and restart it. We spent hours on the phone with Microsoft tech support. They had us apply all kinds of updates and patches, including an unreleased hotfix. We also tried a number of changes to our VMware configuration, some of which actually made the problem worse. Each time we and our user community hoped that stability would finally be achieved, only to be bitterly disappointed. Not only was this frustrating for our users, but it was embarrassing for us. Though they were characteristically gracious, the users must have been thinking, “Does our IT Dept. know what it’s doing?”

Our goal for this time was simply to hang on until we could install the SQL 2005-compatible version of Shelby, which was released last week. Every day the database was flaky and users were inconvenienced. Then we had something go bad last week in Bank Reconciliation, preventing our Finance folks from reconciling October. Shelby tech support concluded there was a damaged software component, which would be reinstalled by updating to the latest software version. So we scheduled the update for Sunday afternoon. Unfortunately, the Shelby update procedure didn’t work. We fought that all day Monday and finally got all desktops updated.

The last step was to install new backup software because the version of Veritas we were running doesn’t support Windows 2003 Server x64. To address that, we installed a trial version of ARCserv yesterday.

We finally had all the pieces in place to cut over to SQL Server 2005, running directly on our 2950. Then today the Bank Rec thing started crashing SQL 2000 every time we ran it. Since it was crashing and has been completely unstable for five weeks, we figured we would go ahead and cut over with yet another unplanned outage.

So there you have a tale of how NOT to move to a new database server. By the way, so far it seems stable and faster by 50-60%. Now we’re praying it’s really over.

Our Shelby system is down right now because of ongoing woes with SQL Server. When you are responsible for IT in a very large church that uses Shelby for every aspect of church operations, “Shelby down” is not a happy thing. This is an unplanned outage made necessary by the fact that every time we ran a bank reconcilliation this afternoon, it blew up SQL Server 2000 (errors in the log and the need to manually restart SQL Server every time).

In response, we’re moving our database to a new server running SQL Server 2005. This is something we’ve been working on for months. The final step before moving to 2005 was to install Shelby 5.6.2000, which we did earlier this week. (Since we’re down, we’re installing the 5.6.2009 brand new patch right now.)

We’re now 105 minutes into an unplanned 30 minute outage. Remind me again why I have made a 22 year career in IT???

I spent 75 minutes on the phone with Tony yesterday, which is great for me because I’m always smarter after I’ve talked with Tony than I was before. Perimeter is just ahead of us in the ChMS selection process. He has a deadline of this month to make a final decision. For us, our only looming deadline is to put in our budget request for 2007 this week. I met with our executive management on Monday. Based on that meeting, it seems very likely that our budget request for ChMS will be approved. If so, 2007 will be the year we make a change, but we still have a couple of months to decide what to do.

Here’s our short list:

MSCRM 3.0 plus add-ons from Microsoft CRM partners
Fellowship One
Shelby Arena
Build-our-own open source system in partnership with WEC

Interestingly, two of my finalists overlap with Tony’s three finalists.

Tony’s third finalist is Blackbaud. I’m intrigued by that and enjoyed the chance earlier this year to discuss it in detail with Shelley Hildebrand of Tony’s staff at Perimeter. Unfortunately, the cost of Blackbaud is up there near what it would cost to build our own, so that’s not a feasible option for us.

Also like Tony, I know there’s a chance our short list will get longer before it gets shorter. Did I mention this decision is risky and complicated?

Jason mentioned that Network World featured him extensively in an article on church IT. Jason also gave the reporter my name and I was quoted too (on pages 2 and 4). This is the first time I’ve been quoted in a national trade publication. Thankfully, I have Brian to keep me humble!

I had the privilege of spending the day Friday with Alfred Johnson, Steve Pruitt, and their teams at Shelby Systems. Frank Canady also joined us for lunch. We had deep discussions about the Church Management System (ChMS) market, and specifically the soon-to-be-released Arena ChMS.

I first met Alfred at the IT Roundtable. He arranged the day and introduced me to the senior management team of Shelby. Alfred, you are a gentleman and a great host. The warmth of your hospitality made my time there special.

Shelby has been around a long time and has been a leader in the ChMS market for many years. During my time there, I gained a lot of insight into Shelby’s history, how they see the marketplace, and their product plans going forward. Some of those things I was asked to keep to myself, but many of them are open and I will be commenting on them in the coming days.

We had a frank discussion about our concerns as a Shelby customer, and everyone was gracious enough to really listen and take my concerns seriously. At the same time, I was surprised to learn that my impressions of Shelby’s customer base and target market were incorrect in several respects. Like the Fellowship Tech team before them, they also indulged me an extensive discussion of strategic considerations without insisting on jumping right to a product demo. There will be plenty of time for that.

The Shelby senior management team is seasoned and humble. They have a good understanding of what the marketplace is asking of them. Regardless of whether we become an Arena customer, I’m glad to say I have new friends in Memphis who care about the local church as much as I do.

I will continue to post about my trip and my impressions as we make budget plans for 2007 over the next few weeks.

No, it’s not an obscure law firm. I’m referring to Jeff Hook and his team at Fellowship Technologies.

Today I flew to Dallas for a tour and discussions to help me decide whether Fellowship One will be our next church management system. Jeff, Jeff, Curtis, Curtis, and Scott were awesome hosts and I enjoyed my time with them very much. The team is impressive (even those whose names aren’t Jeff or Curtis!) and their passion for the local church came through loud and clear. I particularly appreciated the openness of our conversation. Church management system selection can be complex (it certainly is in our case) and the issues often can’t be articulated in bullet points or sound bites. So they indulged me by taking the time to fully air the issues and explore them from multiple angles. It definitely sharpened my thinking. I love serious, deep conversations with smart people.

Thanks again Jeff, Jeff, Curtis, Curtis, and Scott!

You’ve been warned. Thanks Jason!