We have *both* kinds of transit!

We have *both* kinds of transit!

This is what it looks like when your provider pulls the rug out from underneath an entire data center:

a.a.a.a - - [11/Feb/2013:21:59:27 -0800] "GET ..."
b.b.b.b - - [12/Feb/2013:01:30:29 -0800] "GET ..."

Yep, that's 3.5 hours with absolutely no network connectivity. Thanks ServerBeach! Your choices for circuits are exquisite.

This is the first thing they said:

The event started at approximately 12:00am and our initial investigation showed that both of our transport circuits that connect our San Antonio Data Center to our Dallas POP were down and not able to pass traffic. We have escalated this issue to the Executive level at both providers and are working diligently to restore network services.

Note: "both providers"... as in, there are only two. That makes me think of the Blues Brothers: we have both kinds of music: country and western!

A later update explained a little more:

We have identified that both of our transport carriers have a maintenance ongoing that is affecting our network connectivity between San Antonio and Dallas. The work is to be completed by 6AM CST and we are escalating through the Executive level to complete the maintenance window sooner in order to restore your service.

So, yes, both providers went out for maintenance at the same time. Assuming it was scheduled, then that means nobody was paying attention and realized that was going to leave them high and dry.

Personally, I find it amazing that their idea of connectivity is two different providers back to the same city. Is San Antonio really just a little growth on the bottom of Dallas? Is there no other way to get transit other than going up I-35? What about Houston? It's actually closer.

Blue herons getting warm one final time, farmers burying cows, hunters plinking aerial runs in the middle of nowhere, and now this.

So yes, if you couldn't get your dose tonight, this is why. Sorry.

This should be an interesting post-mortem.