Annual Planned Power Outage

Annual Planned Power Outage

Planned power outage (PPO) are a common practice for all office buildings in Japan.  The power will be completely shut down during maintenance for a set period of time. Building tenants in turn need to plan for this power down to limit potential damage to network infrastructure. Specifically, computer systems can be damaged unless properly powered down prior to building maintenance.
Over the years Blueshift has seen a number of systems damaged during planned power outages which has resulted in lost time and revenue. In our experience many of these problems could have been avoided with some basic planning and preparation.
To start with there are three (3) main areas that companies should take a closer look at:

·         System Documentation

·         Data Backups

·         Hardware Verification
Let’s take a look at all of these in more detail.

What systems and applications are running in your IT environment?  All system documentation including operating systems, configurations, settings, licenses keys, etc. should be stored and accessible in a safe location.
Confirm what dependencies or boot order requirements are necessary in your network environment. In what order do machines need to be restarted? Document any special steps that need to be taken or special instructions that need to be followed when restarting applications.
It’s a good idea to maintain a list of internal contacts and vendors in case trouble occurs and support is required.  It also may be necessary to notify service providers of the planned power outage so it is not recognized as an incident.

For a typical planned power outage, backups are of extreme importance. There are two (2) types of backups that should be taken on all servers:

·         Image Backup – full backup of operating system (OS) and file system

·         Data Backup – files and folders

You should make absolutely certain that all backups are finished and that they ran successfully before powering down the servers.
It is recommended to run a test recovery before the planned power outage even if it is only restoring a single file.

Prior to shutting down any systems, all hardware should be checked to make sure there are no undetected failures. This is especially true of disk drives. If there is some form of RAID protection then it is possible that there have been disk failures that have gone undetected. If the failures go undetected for too long, you run the risk of data loss. If there are any failed disk drives in a disk array or in a server, they should be replaced and the data resynchronized.

If you have servers that have not been rebooted in a long time, it may be a good idea to go ahead and reboot the server prior to the power-down to make sure it can boot successfully.  Keep in mind that Window systems may have updates that are pending which can prolong the shutdown process.
A successful reboot of the server prior to the power-down should lessen the chances of something going wrong when you power-up the server after maintenance.