But doing the same with Windows Server 2012 is much more difficult, since Microsoft decided to remove the ability to run a program on a low battery event from its power management settings. To make things worse I discovered that a bug in Server 2008 and later meant that issuing a Shutdown command from the native power settings would not perform a clean shut down, instead killing the power in a few seconds. This is not good news for RAID arrays and data integrity.

Time for a new solution, and since Microsoft are of no use, help would need to come from a 3^rd party. After research and testing answer came from Shutter, a small program that runs as a trigger and event type program for a variety of different scenarios, with battery discharging status being one. Luckily two instances of the program could be run, one to shut down the remote servers and another for the host machine. Importantly the program can also be run as a Windows service, but more on this in the walk through.  here is how it is done:

Download the portable version of Shutter from den4b.com.

In the Windows directory, create a new folder, here it will be named ups.

In that folder create another 2 folders, one will handle the local server shutdown and the other for remote servers.

Copy the contents of the downloaded Shutter zip to both of the folders.

Before opening the program for the first time, create a .bat that will issue commands to shut down servers. Open Notepad and enter the following text:

shutdown /s /f /t 10 /c “Battery Backup Low”

This is for shutting down the host server. To save, select the folder to be used for host shutdown, in the file type menu select All files, and create a file name with it ending in .bat

Do the same for remote server shutdown using this script, and save to the remote folder:

shutdown /s /f /m \\SERVER /t 10 /c “Battery Backup Low”

Replace SERVER with the name or IP address of the other server running on the UPS.

Now go to the local shutdown folder and open Shutter.

In the events box, add battery and choose a trigger percentage and add. In the action box, choose run a program, and locate the .bat file you created for shutting down the host server.

Important: Go to the settings window and copy the following options:

Save settings and the program can now be closed. Repeat these steps to shut down the remote server, it’s essential to set the battery percentage trigger higher for remote server as they will not get the signal if the host server has already shut down, I recommend at least 5% higher as a safety net especially if the UPS battery drains quickly.

As a power outage is an unpredictable event, Shutter needs to be running at all times on the host server, therefore it needs to be ran as a system service. Microsoft did release a tool that could allow any program to run as a service, however it is very basic and will not restart a program if it happened to crash. Lucky a superior utility by the name of NSSM is available that takes this into account, it also has a more user friendly interface to set up services.

Download the latest release of NSSM and place in the main folder created previously, as it needs to be run alongside the Shutter instances.

From a Command Prompt, navigate to the folder holding the program, by using cd, then a space and the full path to the folder, e.g: cd C:\Windows\ups

Type nssm install and enter, and the new service setup screen will appear.

Choose the Shutter program to run, and create a name to the process, additional details can be made in the Description tab.

Move to the login tab and select Log on as, then use an account capable of administrator access and password details.

Click create service and its installed, repeat the process for the other shutter program, giving it a different service name.

By default, these newly created services will start when Windows starts, but need to be started manually when first installed. To start them, the simple option is to restart the machine, but much better would be to press the Windows key & R, then type services.msc and click Run. Scroll down the list to the created services, right click on each and choose start.

Testing

It’s worth testing to see if it works rather than just hoping it does. And to save the wait for the batteries to discharge its worth raising the trigger values on Shutter to shut down the machines earlier, this also gives the benefit more charge left if the shutdown does not work.

Before opening Shutter to change values, the services need to be stopped. Open services.msc like before and chose stop against the service.

Make the changes in the action box, its recommended to choose somewhere between 70 and 90 percent. Close the programs and start the services back up. Then testing is as simple as unplugging the UPS from the wall, for security the host machine can be left logged on to monitor the remaining battery percentage so you know when the trigger points are happening.

Unfortunately, the UPS in use only has one monitoring port which is connected to my original server, meaning during a spell of prolonged power outage, the new server will not know when to shut down safely and will continue to run until the UPS runs out of battery supply, leaving it vulnerable to data corruption that affected the original server prior to battery backup.

But with a simple script and some setup both servers can shut down safely before the batteries run out.

The Idea is to have a script sent from the host server with instructions to shut down the other server when the battery level runs low, luckily Windows gives the ability to run a program on low and critical power level notifications, with adjustable power level triggers.

The most important factor with this idea is that the host server is able to communicate and send the script to the other server, in my current setup both servers are connected to the main network switch that does not have battery backup, meaning that during a power outage the servers will be unable to communicate and therefore unable to shut down safely.

The simple solution was to introduce another switch close to the servers that has power provided by the UPS, for this I chose a NetgearProsafe GS105 and connected both servers to it, with an uplink to the main switch. This also has the advantage of removing traffic away from the main hub when backups between servers take place.

With the servers connected via a backed up power source, it is time to create the shutdown command. On the host server connected to the signal of the UPS, create a .bat file using Notepad with the following line:

shutdown /s /f /m \\MACHINENAME /t 10 /c “Battery Backup Low”

This script tells the computer MACHINENAME to force a shut down in 10 seconds regardless of who is logged in, for those that are the message “Battery Backup Low” will be displayed. The script was then saved to the Windows directory to help avoid accidental deletion.

From the host server, open the power settings from the Control Panel, on the Alarms tab choose Alarm Action on the Low battery alarm field. Check the Run program box and choose the .bat file created to run on the low battery alarm. I chose the Low power instead of Critical to allow the other server to shut down as the Critical would be reserved for the host server itself to shut down.

For testing I chose to put the alarm settings pretty high, which makes the process quicker and also allows more time for a safe shutdown when on battery power. Apart from that the only way to test is to kill the power from the wall and see what happens, look for the other server setting down first as it will hit the Low battery first, then wait for the host server to shut down later. Expect the host server to last a little longer than expected as with the other server shut down, the battery life doubles on two identical machines when one is off.

The post before that however had more gravitas, with my server being offline for close to a month all due to a one second power cut made me feel vulnerable to another downtime incident over something I couldn’t control. It was time to look into a Uninterruptable Power Supply to protect my server from power cuts that could knock my RAID out of sync.

My purchasing requirements were as cheap as possible,  but had to be an online system, meaning that the UPS would notify the server when it was running on batteries and shut down the system down before the juice ran out. An offline system would be less inexpensive, but a long term power outage would just postpone the problem, plus since I don’t intend to supply backup to the network hub, I would be powerless (on both levels) to do anything about it.

I settled on the APC CS350, a lead-acid unit with serial and USB connectivity. Making thing simple I connected the UPS to the server via USB and Windows 2003 found it without extra drivers, making it as familiar to use as a laptop with its battery management.

After allowing the battery to charge, testing is easy, switch off the mains and see how it does. Even from this modest unit my HP Microserver in idle gets 50 minutes of extra life before the battery runs out. I set Windows to shut down at 25% charge, more than enough to shutdown, given that during a power outage the router would be down so the server would be unable to communicate with anything, data preservation is the main game.

The UPS cost £46, my preferred WD Caviar Green 1TB drive costs £48, if another power cut occurs then my it would of paid for itself, another cut and its paid for itself. Yes?

Once I got down there and started to fathom out what cables were in use and what could be safely isolated without stopping operations, the small wafts of smoke could be seen drifting up from under the desk. At this point it was obvious that any timescales for diagnosing the issue was getting smaller along with the grace period before the smoke detectors trigger the fire alarms and clears the store.

On the initial look, I noticed that one of the Proliant servers had a flashing LED next to a power symbol, two and two went together and thought that a power supply had failed spectacularly, so chose to switch it off, knowing the server was just for redundancy.

A minute passed and no let up of the smoke, by this time a CO2 extinguisher, pin pulled, was close at hand. Out of ideas I pulled all plugs from the wall, the RS6000 UPS failed immediately, the Proliants carried on under battery juice with 105mins left according to their UPS display (1 was still powered off). I left it another minute to rule out a problem with an input to the UPS, and with nervous relief the smoke subsided, a few back office systems went down with the RS6000 but the customer end Proliant stayed online.

With the batteries keeping customer facing systems online for a further hour or so, it was a safe time to find the culprit. An extensive sniff test and the UPS for the RS6000 was pointed out as the source of the incident, possibly why it failed as soon as power was cut. It was taken out of commission and bypassed to get the IBM machine back online.

A rather eventful day compared to the normal, mundane non-IT job. I haven’t opened up the failed UPS to see what went wrong, nor would I want to thinking about what state the (probably) lead cells are in.