Hurricane Sandy hit the Eastern US hard this past week, leaving a trail of destruction and damage across multiple states. But in its devastating wake, there have been many well publicized cases of generosity and compassion shown to and by members of the affected communities. Wednesday afternoon I witnessed the combined power of young professionals and Google Hangouts to solve problems remotely. This is the story about how people came together to help Fog Creek Software avert disaster.
Fog Creek Software Tackles Hurricane Sandy
Fog Creek Software is a company that makes a handful of software services (including the bug tracker Fog Bugz, their oldest and most popular service). Their co-founder and chairman of the board, Joel Spolsky, is Internet famous. If you’re a software person, you’ve likely heard of Fog Creek and/or Joel – they compete with Amazon, Google, Microsoft, and Facebook for employees.
Fog Creek is located in Lower Manhattan, and their data centre is located in what became Evacuation Zone A during Hurricane Sandy. These servers house all of their customer data and are absolutely essential for the continued operation of the business. Despite the approaching storm, Fog Creek on Monday (just before Hurricane Sandy hit) maintained that: “Given the preparation work that’s gone into this, we are confident that all of our services will remain available to our customers throughout the weather.”
Save Fog Creek Data!
Perhaps unsurprisingly, their next post (on Tuesday) warned of imminent unplanned downtime. Hurricane Sandy had arrived in full force on Monday night. The power was out for close to one million people, and most tunnels were flooded resulting in continued subway closures and traffic delays. Lower Manhattan was hit the hardest; the data centre did not have any power besides a diesel generator. And by the wee hours on Wednesday morning, that fuel had almost run out.
Two Problems For Fog Creek
There were two problems. First, more fuel was needed. Second, the pump that pushed the fuel from the basement to the 17th floor where it was needed was damaged in the flood. Fog Creekers rushed to the scene. So did developers from Squarespace, another New York City software company whose servers were also in peril. Some bought fuel and brought it to the data centre. Others formed a bucket brigade to get the fuel to the 17th floor and keep the generator going.
The situation seemed to get worse every hour. The bucket brigade continued on Thursday until building management shut it down. As physical labour was no longer the central need, Thursday afternoon turned to contingency planning. The Fog Creek leadership team had a quick meeting just before noon and established that two plans needed to be worked on. One plan addressed the situation where the servers could be removed from the building (if moving them became necessary) while the other plan addressed the worst case scenario: the servers would have to stay in situ and the data would need to be retrieved remotely.
Contingency Planning For Hurricane Sandy
The Data Centre was not the only Fog Creek asset impacted by the hurricane. The building housing Fog Creek HQ was shut down for the week, so the contingency plans were worked on remotely using a combination of chat rooms, phone calls, text messages, emails, and Google Hangouts. Not everyone worked from home; some Creekers left their apartments due to the power outages. Three Creekers worked frantically from my living room. It was a veritable Kitchen Cabinet.
Lena, Fog Creek’s accountant, sat on the couch, monitoring and facilitating communications chiefly in the chat rooms and via email. Daniel, a developer on the Kiln team, worked on the server removal contingency plan from the floor (his choice). Aaron, the Fog Bugz team lead, sat at the table in a never ending Google Hangout. Having received his marching orders from Joel, he briefed his team, augmented by systems administrators from Fog Creek and Stack Exchange. The exchanges were efficient and effective. Everyone involved was focused and responsive.
Not Over For Fog Creek
The crisis is not yet over – the data centre is still running on the generator. But a new generator is on the way and power from the grid should be restored soon. Plus, Fog Creek HQ is now up and running and most Fog Creekers have power in their apartments once more. It appears that the worst is over.
It’s not everyday young professionals are faced with a stressful situation the likes of which hit Fog Creek. But in such extraordinary times certain qualities surface. If you want to work at Fog Creek, you have to be smart and get things done, as per Joel’s hiring philosophy. There were many smart people getting things done on Thursday afternoon.
Problem solving, creative and critical thinking, and working under pressure are all assets for young professionals. Seek out opportunities to practice these skills. If you’re a software student looking for a great job this summer, Fog Creek is currently accepting applications for their internship programme.
Apply now; I’ll see you in New York City this summer.