2

That time we had a network outage due to unexpected hunter activity

 2 years ago
source link: https://devblogs.microsoft.com/oldnewthing/20211019-00/?p=105811
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.

That time we had a network outage due to unexpected hunter activity

Raymond

October 19th, 2021

The current Office team were very much amused by my story of the The Magical Excel 97 Far East Language Build Screwdriver™, although the details couldn’t be verified by anyone still on the team, and the timeline and other details weren’t quite accurate.¹ Nevertheless, the old-timers did admit, “Stuff like that happened all the time.”

In appreciation, they shared their own funny story from 2015:

The team were informed of emergency network maintenance between Redmond and the Quincy data center. The maintenance would require the shutdown of most of the direct circuits between the two sites, and the remaining circuits weren’t enough to cover typical daily usage. However, the network operations team were able to acquire extra circuits to make up the difference. Unfortunately, those extra circuits took a longer route between the two sites, adding latency. The network operations team requested that data migration or other large operations be postponed if possible until after the repairs are complete. The repairs were expected to take four hours, but the proximity to the power lines required coordination with the local power authority and extra safety precautions, so the local power authority requested a 10-hour maintenance window.

So far, this is typical network outage email.

The interesting part came when they explained the cause of the outage.

“The fiber was damaged by a hunter shooting at a pheasant near the power lines.”

¹ Okay, so it was Building 36, not Building 37. Not that that really affects the story.


About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK