Moving Day 1

We’re working closely with our Facilities Department to move 50 staff people, starting yesterday through the end of next week. Yesterday was Move Day 1 when we moved 21 people from Central Campus to the leased office at Southcreek. Next Wednesday and Thursday we’ll be moving 11 people from one place to another within campus (including Andrew). And then Friday we’ll move another 18 people from campus to Southcreek. Whew!

Philip (foreground), Jeremy, and Brian working among the cubicles:

Movers from Fry-Wagner:

Dick Cooper, our Director of Facilities, holding forth:


Jeremy prepping phone cords:


Ian, concentrating on settting up phone extensions, amid the chaos:


Kelly Williams (center) and some of her staff from the Finance Department surprised us with lunch (what a blessing!). Uber voluteer Doug Blackwood, retired from management of the help desk at HP, is on the far right.

Outfitting the new office space

Here’s some of the work we’ve been doing over the last three weeks to prepare for 40 staff moving into the 9000 sq. ft. of office space we leased to replace our 13 year-old temporary building.

In the process of installing data and phone cables:
Ian and Jeremy begin installing equipment in the phone/network room:

Jerry and Kevin from WKT (our phone vendor) installing the phone system:

Ian installing a repurposed Dell 2650 for use as a file/print server (it does DFS replication with a server in our data center on Central Campus):

Rain fade update

I posted previously about the wireless LAN bridge we installed from our Central Campus to our new leased office space, and I mentioned the problems we’ve had with rain fade.

Since my last post I learned that the radios nominally use the 80 GHz band, but since it’s a full-duplex link, there are two diffferent frequencies in use: around 80 GHz in one direction and around 72 GHz in the other direction. Consequently, the signal strength in one direction isn’t the same as the signal strength in the other direction. The graph I posted previously was for the stronger of the two.

Since my last post our vendor has repositioned the radios again and wrung out another 3 dB, which is great. Unfortunately, the link is dropping at -66 dBm, whereas the spec says it won’t drop until somewhere in the -69 to -72 range. And according to Bridgewave, our radios tested to -69 in the factory before they were shipped. So we still have a problem.

Here is the signal strength graph from yesterday when in rained starting around noon and continued at varying intensities for the next 24 hours:

In all of that we had only one drop lasting around 30 seconds at 3:39 pm. You can’t see the excursion below -65 because the monitoring system is on the other end of the link. (The monitor needs the link to be up to grab the signal strength from the other end.)

Now, one 30 second drop doesn’t sound to bad considering it’s been raining for 24 hours. But the problem is, if we had staff working here yesterday, every phone call in progress at 3:39 pm would have been dropped. Most likely every Shelby session would also have been dropped. That doesn’t seem good enough to me. What do you think?

Three values

In reponse to Tony’s post about how much to serve users

This is good stuff, Tony, but twelve values is probably too many. I try to simplify things down to three if at all possible.

1. Our default answer is “yes”. We always hope to find a way to meet the user’s need, even if it isn’t in quite the way the user expressed the need or quite the solution the user proposed. See: http://appianway.blogspot.com/2006/01/our-default-answer-is-yes.html

2. Our mission is our user’s mission. That is, when we’re working with Children’s Ministry, our mission is to teach kids about Jesus; when we’re working with Congregational Care, our mission is to visit people in the hospital; etc. Rather than focusing on our own mission, we’re focused on the mission of those we serve.

3. The one mission that is uniquely our own is information security. If we don’t look after information security, no one will. We define information security as providing or denying information access as appropriate and ensuring that critical information can’t be lost. If #1 or #2 above conflict with security, then we make a tradeoff.

We don’t always achieve our internal IT Department goals (roll out the software update by Feb. 1, upgrade environmental monitoring in the data center by the end of the month, etc.), but we always get high marks from our user community. As far as I’m concerned that’s the correct priority. And of course, we have to work very hard behind the scenes to make sure we don’t work exclusively on short-term or user-visible things. Sometimes the best thing we can do for our user community is to work on something that takes a lot of resources and users never know about. But that’s nothing fancy, it’s just good, disciplined execution.

Fall Roundtable update

From the number of comments on my Fall IT Roundtable Invitation, it seems like we’ll have a great group here in Kansas City for the Fall Roundtable. For now I just wanted to acknowledge all the responses. There will be a lot more information and discussion about the Fall event during the Spring Roundtable in Houston. We want to get feedback from the Spring Roundtable before we start posting more specifics.

Have a blessed Good Friday and Easter.

Rain fade

I posted earlier about the wireless LAN bridge we installed from our campus to our new leased office a couple of miles away.

These radios operate at such a high frequency (80 GHz) that nearly everything — a tree, a dense flock of birds, an errant golf ball (okay, I just made that one up) — is opaque to the signal. Consequently, they suffer from signal attenuation due to rain. This is a well-known phenomenon called “rain fade” that the RF engineers have to take into account in their designs.

We’re having some trouble getting sufficient path clearance (getting the base signal high enough) to avoid a link drop during periods of intense rain. Right now the base level is approx. -44 dBm. We were expecting rain fade of approx. 16 dBm, which should make it bottom out around -63 dBm. According to Bridgewave, these radios are supposed to be able to maintain the 100 Mb/s link down to -72 dBm. Therefore, we should have approximately 9 dBm of safety margin.

But we have two problems. First, we’re seeing rain fade of 22 dBm. Second, we’re seeing the link drop when the signal goes below -65 dBm. With 6 dBm more rain fade than expected and the link dropping 7 dBm before it should, our 9 dBm of safety margin is shattered.

Here is a graph showing two times when we had brief periods of intense rain. Each time the link dropped for a few minutes and then came back up after the intense rain passed:

After the the second drop you can see other periods of rain fade that weren’t intense enough to cause a drop. (The graph shows signal level in 100ths of a dBm. i.e. -4400 is -44 dBm.)

I gave a requirement of four 9s of availability (99.99% up time). Clearly, with the current base signal level, the amount of fade we’re experiencing, and the link drop level, we’re not going to be close to four 9s. Thankfully, we’re installing this well ahead of staff moving into the office so we have time to improve the situation. Also, I’m grateful that we’re installing in the Spring when we have rain every couple of days. Each time we make a change, we don’t have to wait too long before a real-world test presents itself.

We’re out there on the bleeding edge with this one! Please pray with us that this wireless LAN bridge story has a happy ending.

GodTube mocked in Wired

Did anyone else notice this Wired blog post mocking GodTube?

There are many things swirling in my mind about this:

1. Does it make sense to have “Christian” versions of popular, secular web sites, or does it make more sense to proclaim the good news on the secular sites themselves? John Brownlee, the writer of the post, gets at question when he says, “What is disturbing about GodTube is that it is an observable microcosm of the way that fundamentalist Christians have shut themselves off from any outside perspective.” Where is the line between being “in” the world but not “of” it?

2. John does us a favor by saying what many others also must be thinking. Doesn’t this help us gain insight into the mindset of secular humanists and atheists? We can’t possibly present the good news persuasively without knowing the views of our hearers.

3. Also facinating are the comments at the end of the post. Clearly John struck a nerve because many people wanted to react to what he said — both echoes of John’s ideas as well as thoughtful Christian responses.

Network and server monitoring

We are in the process of taking our network management up to the next level. Our server and network infrastructure has gotten sufficiently large and complex that we need to keep a much closer eye on what is happening. We need to be able to monitor resources, applications, and errors. We need a way to alarm, for example, when a server, switch, or application is down or when a resource gets to a critical range.

We’re currently evaluating What’s Up Gold. I also recently heard about Splunk. Do any of you have any thoughts on either of these or something else we should evalutate?

Wireless LAN bridge

We are preparing to move 40 staff to an office building a couple of miles from the church. For the last several years those staff have worked out of a temporary building for which the temporary use permit is now expiring. So how do you connect 40 remote staff to your LAN? If you have a clear line-of-sight, a great option is to use a wireless LAN bridge.

Here are pictures taken yesterday as our vendor performed alignment of the integrated radio/antenna units. These are 80 GHz (millimeter wave) radios that are capable of 1 Gb/s full-duplex, but we got the version that is rate-limted to 100 Mb/s.

Radio on roof of office building, pointed towards the church. The steel wall in front is a “penthouse” that hides roof-top air handlers an other equipment. The mast is mounted to steel girders supporting the penthouse.

Close up of back of radio with volt meter showing signal strength.
Looking over the top of the radio toward the church building, 1.7 miles away.

Close up taken from the office building with black arrow showing the location of the radio on the roof of the church building.

Radio on roof of church building. The mast is held by a 4-point mount and weighed down by concrete blocks so roof penetration isn’t required. This location was chosen for maximum wind protection (air handling unit to the right; roof parapet to the left). It also happens to be right next to a roof access hatch.

Ian (our network admin) with his shiny new radio!

Jay, the lead installer for our vendor, tweaks the alignment by turning nuts on the mount while looking at his volt meter.

Jay relays meter readings to his co-worker on the roof of the other building.

On this end the radio is connected by fiber.

Close up taken from the church building with white arrow showing the location of the radio on the roof of the office building.