Rain fade

I posted earlier about the wireless LAN bridge we installed from our campus to our new leased office a couple of miles away.

These radios operate at such a high frequency (80 GHz) that nearly everything — a tree, a dense flock of birds, an errant golf ball (okay, I just made that one up) — is opaque to the signal. Consequently, they suffer from signal attenuation due to rain. This is a well-known phenomenon called “rain fade” that the RF engineers have to take into account in their designs.

We’re having some trouble getting sufficient path clearance (getting the base signal high enough) to avoid a link drop during periods of intense rain. Right now the base level is approx. -44 dBm. We were expecting rain fade of approx. 16 dBm, which should make it bottom out around -63 dBm. According to Bridgewave, these radios are supposed to be able to maintain the 100 Mb/s link down to -72 dBm. Therefore, we should have approximately 9 dBm of safety margin.

But we have two problems. First, we’re seeing rain fade of 22 dBm. Second, we’re seeing the link drop when the signal goes below -65 dBm. With 6 dBm more rain fade than expected and the link dropping 7 dBm before it should, our 9 dBm of safety margin is shattered.

Here is a graph showing two times when we had brief periods of intense rain. Each time the link dropped for a few minutes and then came back up after the intense rain passed:

After the the second drop you can see other periods of rain fade that weren’t intense enough to cause a drop. (The graph shows signal level in 100ths of a dBm. i.e. -4400 is -44 dBm.)

I gave a requirement of four 9s of availability (99.99% up time). Clearly, with the current base signal level, the amount of fade we’re experiencing, and the link drop level, we’re not going to be close to four 9s. Thankfully, we’re installing this well ahead of staff moving into the office so we have time to improve the situation. Also, I’m grateful that we’re installing in the Spring when we have rain every couple of days. Each time we make a change, we don’t have to wait too long before a real-world test presents itself.

We’re out there on the bleeding edge with this one! Please pray with us that this wireless LAN bridge story has a happy ending.

GodTube mocked in Wired

Did anyone else notice this Wired blog post mocking GodTube?

There are many things swirling in my mind about this:

1. Does it make sense to have “Christian” versions of popular, secular web sites, or does it make more sense to proclaim the good news on the secular sites themselves? John Brownlee, the writer of the post, gets at question when he says, “What is disturbing about GodTube is that it is an observable microcosm of the way that fundamentalist Christians have shut themselves off from any outside perspective.” Where is the line between being “in” the world but not “of” it?

2. John does us a favor by saying what many others also must be thinking. Doesn’t this help us gain insight into the mindset of secular humanists and atheists? We can’t possibly present the good news persuasively without knowing the views of our hearers.

3. Also facinating are the comments at the end of the post. Clearly John struck a nerve because many people wanted to react to what he said — both echoes of John’s ideas as well as thoughtful Christian responses.

Network and server monitoring

We are in the process of taking our network management up to the next level. Our server and network infrastructure has gotten sufficiently large and complex that we need to keep a much closer eye on what is happening. We need to be able to monitor resources, applications, and errors. We need a way to alarm, for example, when a server, switch, or application is down or when a resource gets to a critical range.

We’re currently evaluating What’s Up Gold. I also recently heard about Splunk. Do any of you have any thoughts on either of these or something else we should evalutate?

Wireless LAN bridge

We are preparing to move 40 staff to an office building a couple of miles from the church. For the last several years those staff have worked out of a temporary building for which the temporary use permit is now expiring. So how do you connect 40 remote staff to your LAN? If you have a clear line-of-sight, a great option is to use a wireless LAN bridge.

Here are pictures taken yesterday as our vendor performed alignment of the integrated radio/antenna units. These are 80 GHz (millimeter wave) radios that are capable of 1 Gb/s full-duplex, but we got the version that is rate-limted to 100 Mb/s.

Radio on roof of office building, pointed towards the church. The steel wall in front is a “penthouse” that hides roof-top air handlers an other equipment. The mast is mounted to steel girders supporting the penthouse.

Close up of back of radio with volt meter showing signal strength.
Looking over the top of the radio toward the church building, 1.7 miles away.

Close up taken from the office building with black arrow showing the location of the radio on the roof of the church building.

Radio on roof of church building. The mast is held by a 4-point mount and weighed down by concrete blocks so roof penetration isn’t required. This location was chosen for maximum wind protection (air handling unit to the right; roof parapet to the left). It also happens to be right next to a roof access hatch.

Ian (our network admin) with his shiny new radio!

Jay, the lead installer for our vendor, tweaks the alignment by turning nuts on the mount while looking at his volt meter.

Jay relays meter readings to his co-worker on the roof of the other building.

On this end the radio is connected by fiber.

Close up taken from the church building with white arrow showing the location of the radio on the roof of the office building.

Is redundant more reliable?

Pete Whiting, in a guest post on Joel Dehlin’s blog, has some insights into the perils of complexity. This is good food for thought as we plan our server virtualization project at Resurrection, which we have scheduled for implementation in May. I particularly agree with Pete’s point about isolating/avoiding tight coupling between services. For example, it’s much better if an application is dependent only on one server, than if it requires two (or more) servers to be able to operate. With virtualization, we are making it a priority to gather everything a particular application needs onto a single virtual server, which will be able to run on either of two redundant physical servers. This architecture provides the benefits of redundancy without introducing unnecessary interdependence.

A Long Overdue Thank-You

Thanks Tony for your outstanding hospitality during my visit to Perimeter Church back in December with Mark Stephenson. Please offer my long overdue thanks to the rest of your team as well. Our tour of Perimeter Church’s facilities was amazing. God has blessed the church with an innovative spirit that is never satisfied with past accomplishments. The oldest part of the building is only 15 years old, but already many parts have been purposed and repurposed, designed and redesigned, modeled and remodeled numerous times as the church’s needs have grown and changed. That kind of passionate pursuit of ministry warms my heart and re-energizes me.

The DST Saga Continues (with a happy ending, we think?)

Tony has been blogging about various Daylight Savings Time issues at Perimeter Church here, here, and here. We, too, have had some serious headaches with this. I commented on one of Tony’s posts saying that we couldn’t get the Exchange Calendar Update tool to work. Tony generously put the question out to IT Discuss and we got a response from Jason Hand of Walk Through the Bible. Thanks Jason, but unfortunately it didn’t help.

So … Yesterday, when everything was going horribly wrong 😉 Ian, our network administrator, found someone at Microsoft who could help us. He posted a full explanation of what happend here. Turns out even the developer at Microsoft who wrote the tool couldn’t get it to work in our case. In the end he threw up his hands and had Ian do a manual workaround to the front-end of the process that allowed the rest of it to run successfully.

Things have finally settled down here. We have only a few, minor issues being reported today. DST – was it fun for you?

Fall IT Roundtable Invitation

I was part of a group of IT leaders from very large churches that met a couple of times a year for three years or so. The group (which never had a name other than “megachurch IT group”) included Mike Gold of Willow Creek, Terry Chapman of Fellowship Church, Vic Mollett of Southland Christian Church, Eric Busby of Saddleback Church, and others. When last we met in May of 2005 I told the group I would host the next meeting here at Church of the Resurrection in the Kansas City area in 2006. Unfortunately, I got slammed last year with workload and staff turnover issues so I wasn’t able to make it happen. I recently exchanged e-mail with Mike Gold about renewing my invitation to host the group here and suggested merging the group with the Church IT Roundtable. He was very supportive and hoped to be able to attend and invite the old gang to join the Roundtable.

Accordingly, we would like to invite you to come to Kansas City for the Fall Roundtable October 3-4, which is right before Leadership Institute, our biggest conference of the year. After the Roundtable you could stay for Leadership Institute if you’re so inclined. I spoke with Tony Dye about getting this on the agenda for discussion at the Spring Roundtable to determine the level of interest. To get the conversation started, please leave a comment here and let me know if this sounds good.

National Church IT Association?

If you read Jason Powell’s blog, you already know about his hope to create some sort of national church IT association. (Tony has arranged for Phill Martin of NACBA to come to the Spring Roundtable to speak about this.) Now as for me, I strongly prefer the roundtable format for meeting with and learning from other church IT leaders. I worry that formalizing and scaling up what we’re doing into a national association would result in losing the powerful, peer-to-peer sharing of the roundtable format. I also worry that we’ll discover even the 25-30 people expected to attend the Spring Roundtable will be too many for everyone to fully participate. So what to do?

Fall Roundtable Format

Now here’s where it gets really exciting (we think?!) or maybe even crazy. For the last couple of years I have been meeting informally with church IT leaders here in the Kansas City area such as Mike Mayfield of Pleasant Valley Baptist Church. I suspect many of you have had similar meetings with other church IT leaders in your own cities. Perhaps some of you have already held roundtables in your cities. This makes me wonder if the best way for us to scale up would be to have roundtables in cities across the country and then link them technologically.

What if we made a first effort at this idea for the Fall Roundtable? What if we limit attendance here in Kansas City to 15-20 people and find some great way to link in roundtable groups from other cities? This idea is partially inspired by the Willow Creek Leadership Summit that brings together church leaders from around the world. We’re imagining some kind of “un-conference” that retains the strengths of peer-to-peer sharing but is scalable to involve many more people than you can fit around one table. We would like for everyone to be able to see/hear each other and use their own laptops to annotate/live blog the discussion and have side conversations. We’re not even close to figuring out how to do this from either a facilitation or technical standpoint, but one idea is to use Adobe Connect (formerly Macromedia Breeze). Whatever the mechanism is, we would need to fully test both the technology and the method of facilitation prior to the meeting. And, no doubt, we’d learn a great deal from doing this once. At that point we would know either a) this works; b) this works but needs to be tweaked/improved; or c) it doesn’t/can’t work.

Please comment and let me know your thoughts or post on your own blog and link back to this post.