The arrival of the Raspberry Pi 2 meant the B+ previously used as a server was now ready for a new purpose. These little machines make great media players and is easy to setup so this would be its new role.
However, this Pi would be in my parents’ home, and miles away from where my videos are stored. I didn’t want to add a hard drive locally as syncing the library would be an issue. But thanks to the fibre internet connection on the server side I could upload at 10Mbps, enough to stream media across the net.
Therefore, the plan is to use an install of OSMC and add a VPN connection to my home server that starts when the Pi starts to make the operation seamless.
With the new server up and running it seemed fitting to connect it to my UPS, and thanks to the low power consumption of the HP Microservers I still get around 40 minutes of battery only time with both servers running before the UPS runs out of juice.
Unfortunately, the UPS in use only has one monitoring port which is connected to my original server, meaning during a spell of prolonged power outage, the new server will not know when to shut down safely and will continue to run until the UPS runs out of battery supply, leaving it vulnerable to data corruption that affected the original server prior to battery backup.
But with a simple script and some setup both servers can shut down safely before the batteries run out.
The Raspberry Pi Foundation has brought a new model of the Pi to the table, so with the same low price it was easy to add to my Pi family.
For those not yet familiar with the RPi 2, it keeps the small and simple form factor of its predecessors, but gives performance upgrades in the form of an 800mhz quad core processor, with overclocking options still available, and combined it with a whole Gigabyte of RAM. These upgrades, on paper at least sound that it could make the Pi into a useable desktop machine, with less freezing during full CPU load that was easy to achieve previously.
Fibre broadband has been available in my region for a couple of years now, and while it was tempting it was just too expensive for a fully unlimited package. Therefore I had to make do with ADSL, combined with being a few miles from the exchange it was pretty poor, even when pulling the ring wire.
ADSL Speedtest
But at last an offer from Sky for existing customers to get Fibre half price, so I had to jump on this bandwagon immediately. The Fibre To The Cabinet (FTTC) service is a 40Mbps down and 10Mbps up with no download cap. Compared to before this is a huge improvement.
My trusty HP Microserver N36L has been ticking along nicely for years with Windows 2003 at the helm, but with support for 2003 coming to an end an alternative was needed.
Recently I have been using Amazon EC2 cloud services for all my website hosting, new customers to the service get a 12 months free teir1.micro instance with myself opting for Windows Server 2012 R2 as my OS, and I’ve taken up this offer since October last year when an extended spell of server woes left me unable to serve websites.
A caveat with the free EC2 instance is the billing process, while the instance is free you have to pay attention to what is included as part of the offer, network usage, hard drive capacities and security keys are subject to charge over certain thresholds, so be aware.
With the trail due to expire in a few months I preferred to avoid an ongoing monthly cost and bring website hosting back to my own server, but not my current server as I didn’t want a box that served the web as well as store all my personal files. The logical conclusion was to get a new server.
I have been running Gigabit Ethernet on my network for a few years, the speed from my everyday laptop was good enough at 40MBps but wasn’t running close to capacity when moving large files to the server, I assumed that this was down to the 5400rpm Hard Drives on the laptop not being able to fill the bandwidth on the network.
Recently while shopping for network cables to connect my new server I though it time to get a new line for the laptop since the plug clip on the current one was broken. I decided to go for Cat 6 cables just to keep up to date and nowadays there is not such a price premium over Cat 5e.
The first transfer of data through the new cable and instantly got 70MBps, nearly double the 40MBps I was getting before. Wondering why there was such a difference I looked at the replaced cable:
The time came when my 1TB drives were full, having a data clearout reclaimed a bit of free space but it was time to upgrade.
The HP Microserver I am running has four HDD bays all populated with 1TB drives in RAID 1 configuration, so to increase capacity I had to replace two of the disks. I went for two Western Digital Green 2TB disks as the 1TB variants I currently had proved reliable.
Moving the data over to a new disk would be tricky, the simplest solution would be to put the old drives into a USB caddy and plug into the server, but I had previously found that USB drives don’t like RAID. But since RAID1 means redundancy, I could tackle this another way…
From the last post, the idea of having UPS in my home may put me off forever, but to put it into context the unit had been installed before I was employed over 12 years ago, and over the past 2 years it was beeping intermittently to indicate a fault that a convenient press on any button would silence.
The post before that however had more gravitas, with my server being offline for close to a month all due to a one second power cut made me feel vulnerable to another downtime incident over something I couldn’t control. It was time to look into a Uninterruptable Power Supply to protect my server from power cuts that could knock my RAID out of sync.
A bit off topic but I should document what happened in work today. Got called to investigate a burning smell in one of the offices that house all the servers and network head end. The request was placid enough not to cause alarm but when I got to the room the smell hit you as soon as the door opened. Narrowing the smell down, it was coming from a caged off area underneath a desk that held the servers: An ancient IBM RS600 with UPS and two HP Proliant ML350 G5 with a shared UPS in two modules, along with what seemed decades of dust, discarded cables and old computer hardware that had accumulated over the years.
Servers claimed by years of dust
Once I got down there and started to fathom out what cables were in use and what could be safely isolated without stopping operations, the small wafts of smoke could be seen drifting up from under the desk. At this point it was obvious that any timescales for diagnosing the issue was getting smaller along with the grace period before the smoke detectors trigger the fire alarms and clears the store.
On the initial look, I noticed that one of the Proliant servers had a flashing LED next to a power symbol, two and two went together and thought that a power supply had failed spectacularly, so chose to switch it off, knowing the server was just for redundancy.
A minute passed and no let up of the smoke, by this time a CO2 extinguisher, pin pulled, was close at hand. Out of ideas I pulled all plugs from the wall, the RS6000 UPS failed immediately, the Proliants carried on under battery juice with 105mins left according to their UPS display (1 was still powered off). I left it another minute to rule out a problem with an input to the UPS, and with nervous relief the smoke subsided, a few back office systems went down with the RS6000 but the customer end Proliant stayed online.
With the batteries keeping customer facing systems online for a further hour or so, it was a safe time to find the culprit. An extensive sniff test and the UPS for the RS6000 was pointed out as the source of the incident, possibly why it failed as soon as power was cut. It was taken out of commission and bypassed to get the IBM machine back online.
Failed UPS, I’m not so trusting of you anymore
A rather eventful day compared to the normal, mundane non-IT job. I haven’t opened up the failed UPS to see what went wrong, nor would I want to thinking about what state the (probably) lead cells are in.
It had to happen, after years of reliability (apart from an ISP related failure), I had my first hardware related downtime, caused by a power cut that lasted all of one second.
In the electric free event, only a few electricals switched off, my HP Microserver was one that lost power and restarted. Looking at the headless unit after boot, all lights were on, HDD light on full and the network lamp was flickering away as normal. However trying to access the server, even down to a simple ping, it was unresponsive.
Time for investigation, and it was ripped out of its kitchen cupboard home and connected to a TV along with a keyboard and mouse. From there it was painfully apparent that the RAID mirror had been corrupted and the BIOS couldn’t find the OS. The OS drive was in a RAID 1 mirror so I took out the primary master disk (first on the BIOS boot priority) and tried to boot the remaining mirror. This time it started Windows. All was back on track it seemed, waited for the other mirror holding data to re-sync then changed the boot priority in Windows (not BIOS) to use the good OS drive first. A restart to plug the un-synced HDD in and it booted fine, no SMART errors reported with the removed drive and it started to rebuild the system mirror.
Things then took an ugly turn, using the desktop would be as normal for around 90 seconds, then the system would freeze, apart from the mouse for minutes at a time, before coming back to life and displayed applications requested before the freeze. It seemed as if the system was having big problems trying to read from the disk, it would run fine simply moving the mouse around, but when selecting a program it would freeze, and depending on what you requested to load it could be up to 20 minutes. While in this state of freeze, the HDD lamp on the Microserver would be solid, so naturally it pointed to either a bad hard drive or the RAID mirror was having problems.
The HP Microserver in a state of repair
Not finding a solution, I admitted defeat and did a fresh install of windows, but still wanted to get the latest backup, the data mirror was easy to recover by just removing the drives as they can be imported on a new install. The OS drive was a bit trickier, the system would freeze if I tried to copy files as is. Luckily I had the old 250GB drive that came with the Microserver, it had Windows 2003 on it and ran on the system until more capacity was required, it was swapped out for a 1TB drive. Not so lucky was that the only software found to copy files from a Foreign RAID mirror cost £50, I shelled out this money as my data was more valuable than the asking fee. Along with an extra 1TB drive to hold the data while I juggle drives it ended up costing me a few quid.
All this from a 1 second power cut.
RAID 1 on a system disk:
Research says this is not a good idea. While it will run with no issues during normal operation, after an unexpected shutdown a RAID controller just can’t tell the difference between a good file and an un-synced, corrupt file, so the controller will either guess, which could restore an out of data file, or create a mismatch of current and out of date files that ultimately brings your OS to a halt.
