Power Down…A True Story

Power Down…A True Story

On January 12th 2016 we received a notice from the building management company that our annual planned power outage was to be scheduled for Sunday February 14th from 11am to 2pm. We knew from years of previous tests what to expect but we also knew that we had quite a lot of work to do to ensure that things would go as smoothly as possible.

Monday January 18th we had our initial planning meeting to run through key points from our 2015 test, talk about changes to our office environment from the previous year and any possible areas of concern for the upcoming office power down. The meeting yielded quite a lot of information that had been forgotten. For example we were reminded that the office firewall froze-up when the building’s power was restored and this caused most systems to be unavailable remotely. It also required a senior engineer’s Sunday time to be onsite to fix the firewall issue.

Monday January 25th meeting began with a review of IT systems which would need a manual reboot and those which should come on automatically once the power was restored. A special asterisk (*) was placed next to the firewall as a potential problem. Next, we talked through which systems should be backed up before the power shutdown, how they would be backed up and if recovery was required the steps to recover the systems.

Monday February 1st meeting began with sharing of detailed list of systems to be backed up on February 10th prior to building power down (engineer assigned to backup systems). Next a list of systems that should come on automatically and those that would require a manual restart was created.  A person was assigned to come to the office from 14:30 to manually start listed systems and check the firewall. An engineer would be monitoring systems remotely to confirm that systems responded correctly.

Monday February 8th final general planning meeting. All requirements and responsibilities were reviewed. It was also suggested that a short list of things to check after power is restored like printers, email, phones and website be added to the February 14th to do list. This was added to the onsite persons duties.

Sunday February 14th 14:30 – all systems that were supposed to auto restart did including the firewall. All systems requiring a manual restart were restarted. Remote engineer checked systems to ensure there we no issues – none found.  Person onsite confirmed that printers were working, phones were on, website was viewable and emails could be sent and received.

Monday February 15th review meeting. All reports were positive.  No problems to report and schedule went as planned.

As you can see from the information we started planning about 1 month in advance, held regular meetings, created detailed to-do-lists, assigned responsibilities and people to the project. Not all office power downs can be this successful but with planning and key decision maker’s involvement problems can be avoided or their negative affects lessened.