From time to time we come across legacy applications and deployments that you didn’t know exist until something goes wrong with it. This week it was an unbeknown to me Joomla website that had been ticking over since 2012. However recent visits to the site got this result:
A report came in on this issue and a few checks of the domain DNS revealed it was on a platform we use for domains and web hosting.
Not overly familiar with the hosting company from a website standpoint and even less with Joomla, it was time to first fathom out how it works, and them find the problem and fix
When running a website from a home server, viewing it locally will make it seem that the site is responding lightning fast and there are no issues. But what about the outsiders wanting a look at your content, are they getting the same performance? Chances are they are not, as a visitor’s machine needs to negotiate the internet and its equivalent of back streets and country roads to get to the home server’s location.
Where a home server can differentiate greatly from hosted solutions is the speed and relative location on the net. Visitors who view a website relies on the upstream connection at the server end to receive the content, and when this is via domestic internet connection the upstream can much smaller than the heavily advertised downstream connection. So it’s worth checking the theoretical upload speed to establish what kind of service and content can be served.
Viewing your website on a local network compared to the internet.
In terms of location, hosting companies are as close to the internet backbone as feasibly possible to get the best speeds and lower latency. The backbone of the net is handled by major operation companies that handle the bulk of all internet traffic between countries and continents, these in turn have datacentres where the traffic from countries are trunked to the different internet providers and down to the end user. As data makes its way from the backbone to the end user, it can hop between different servers as it meanders towards the final destination. For each hop the networking equipment has to read where to send it on, and route it on the right path. This all takes time, even though it is measured in milliseconds, an extended number of hops and the volume of data packets needed may produce a noticeable wait for a user to see the desired page.
All home user’s computers need deal with negotiating its way through the service providers’ local infrastructure to get to most sites, but when visiting a site hosted on a home server, data may need to navigate another service providers’ network to reach the site. This is where visitors may experience slower loading times compared to mainstream sites.
So how to tell if your home hosted website will be speedy when out in the wild? There’s a few different ways to check:
Let’s hope three servers is a charm, as its time for a new server. But this time I’m moving away from the HP Microserver. Why? Well the new server is destined to be a dedicated web server for my sites, ever concerned with security and protecting my network I thought it wise to separate the public facing websites physically from my data, adding an extra layer of security.
The choice was to go for a NUC based machine or Nettop, their small footprint allows them to be placed out of the way, plus they are in keeping with my low power requirements and often fan-less design keeps them quiet. As it’s to be a web server only, the restrictions on a device this size such as space for multiple hard drives, graphics performance and upgradability are not an issue.
In January I was given the opportunity to design and build a new website to help colleagues in the retail store where I worked. This website serves as a demonstration on how I took my website and server knowledge to create a low cost solution to an issue I was confronted with.
This occurred when working in a retail store but can be re-purposed to suit other needs.
What I came up with was Canton TV, a website written in ASP.NET C# and hosted on my home server, it served as a tool for colleagues of varying technological skills create messages and upload images for display in video format on screen in a communal area of the store.
My trusty HP Microserver N36L has been ticking along nicely for years with Windows 2003 at the helm, but with support for 2003 coming to an end an alternative was needed.
Recently I have been using Amazon EC2 cloud services for all my website hosting, new customers to the service get a 12 months free teir1.micro instance with myself opting for Windows Server 2012 R2 as my OS, and I’ve taken up this offer since October last year when an extended spell of server woes left me unable to serve websites.
A caveat with the free EC2 instance is the billing process, while the instance is free you have to pay attention to what is included as part of the offer, network usage, hard drive capacities and security keys are subject to charge over certain thresholds, so be aware.
With the trail due to expire in a few months I preferred to avoid an ongoing monthly cost and bring website hosting back to my own server, but not my current server as I didn’t want a box that served the web as well as store all my personal files. The logical conclusion was to get a new server.
From the last post, the idea of having UPS in my home may put me off forever, but to put it into context the unit had been installed before I was employed over 12 years ago, and over the past 2 years it was beeping intermittently to indicate a fault that a convenient press on any button would silence.
The post before that however had more gravitas, with my server being offline for close to a month all due to a one second power cut made me feel vulnerable to another downtime incident over something I couldn’t control. It was time to look into a Uninterruptable Power Supply to protect my server from power cuts that could knock my RAID out of sync.
It had to happen, after years of reliability (apart from an ISP related failure), I had my first hardware related downtime, caused by a power cut that lasted all of one second.
In the electric free event, only a few electricals switched off, my HP Microserver was one that lost power and restarted. Looking at the headless unit after boot, all lights were on, HDD light on full and the network lamp was flickering away as normal. However trying to access the server, even down to a simple ping, it was unresponsive.
Time for investigation, and it was ripped out of its kitchen cupboard home and connected to a TV along with a keyboard and mouse. From there it was painfully apparent that the RAID mirror had been corrupted and the BIOS couldn’t find the OS. The OS drive was in a RAID 1 mirror so I took out the primary master disk (first on the BIOS boot priority) and tried to boot the remaining mirror. This time it started Windows. All was back on track it seemed, waited for the other mirror holding data to re-sync then changed the boot priority in Windows (not BIOS) to use the good OS drive first. A restart to plug the un-synced HDD in and it booted fine, no SMART errors reported with the removed drive and it started to rebuild the system mirror.
Things then took an ugly turn, using the desktop would be as normal for around 90 seconds, then the system would freeze, apart from the mouse for minutes at a time, before coming back to life and displayed applications requested before the freeze. It seemed as if the system was having big problems trying to read from the disk, it would run fine simply moving the mouse around, but when selecting a program it would freeze, and depending on what you requested to load it could be up to 20 minutes. While in this state of freeze, the HDD lamp on the Microserver would be solid, so naturally it pointed to either a bad hard drive or the RAID mirror was having problems.
The HP Microserver in a state of repair
Not finding a solution, I admitted defeat and did a fresh install of windows, but still wanted to get the latest backup, the data mirror was easy to recover by just removing the drives as they can be imported on a new install. The OS drive was a bit trickier, the system would freeze if I tried to copy files as is. Luckily I had the old 250GB drive that came with the Microserver, it had Windows 2003 on it and ran on the system until more capacity was required, it was swapped out for a 1TB drive. Not so lucky was that the only software found to copy files from a Foreign RAID mirror cost £50, I shelled out this money as my data was more valuable than the asking fee. Along with an extra 1TB drive to hold the data while I juggle drives it ended up costing me a few quid.
All this from a 1 second power cut.
RAID 1 on a system disk:
Research says this is not a good idea. While it will run with no issues during normal operation, after an unexpected shutdown a RAID controller just can’t tell the difference between a good file and an un-synced, corrupt file, so the controller will either guess, which could restore an out of data file, or create a mismatch of current and out of date files that ultimately brings your OS to a halt.
It’s simple, instead of the process of installing PHP, MySQL, assigning permissions and all the debugging that many of us have gone through, Microsoft has come up with WPI, or Web Platform Installer.
WPI makes installing WordPress easy.
Amongst other web applications, WordPress is the most popular and it makes the install on Windows as simple as a few clicks.
All done!
So far I’ve tried it on Server 2003 R2 and Server 2012 R2, both with instant success.
Recently I had an attack on this website, as I run a WordPress site this is not an isolated incident. However, in this instance it was a rather aggressive attack compared to the bandwidth I have available. The attacker in this case was saturating my connection with POST commands to wp-login.php as apposed to the usual attacker who send requests every few seconds, in an attempt I presume to not be noticed.
Same server, same file.
Knowing that this was a sustained attack I first turned to enabling Windows authentication to wp-login.php, this made things worse as the data sent from a HTTP 401 error to this page used more data than WordPress denying access.
A 401 error transfers more data than a 200 error.
Failing this, I blocked the IP address from IIS so it will return a 403 error, this brought the data sent back down to normal but still the attack continued.
As a last resort, I performed a lookup on the attacking IP via the WHOIS database, and discovered the attacking computer was on a hosted server in Malaysia:
% [whois.apnic.net]
% Whois data copyright terms http://www.apnic.net/db/dbcopyright.html
% Information related to '183.81.162.0 - 183.81.162.255'
inetnum: 183.81.162.0 - 183.81.162.255
netname: IPSERVERONE-MY
descr: IPSERVERONE - Co-location - AIMS Data Center
country: MY
address: L7-13, Level 7, Brem Mall,
address: Jalan Kepong, 52000,
address: Kuala Lumpur
e-mail: abuse@ipserverone.com
abuse-mailbox: abuse@ipserverone.com
phone: +60-3-625-95-625
fax-no: +60-3-625-95-629
e-mail: ipnoc@ipserverone.com
% This query was served by the APNIC Whois Service version 1.69.1-APNICv1r0 (WHOIS4)
The website dawhois.com was the first best match and reveled that the hosting service for this site was ipserverone.com, and in desperation to solve this I contacted the abuse email listed above.
I have often contact ISPs in this method when receiving an attack, but none have come to fruition, not even a response, this time I got an answer:
Hi James,
We are sorry to hear that, could you please verify now, is still got attack to your site?
Checking that the attack is still going on, I replied that is is still happening:
Hi James,
Seems that I’ve disabled an access few sites that contain suspicious codes.
Is the issue persist?
Checking again, it stopped! Guessing from the fact that the attack continued after changing responses from my server to 401 and 403, it was a bot running, but many thanks to Mohd and the people at ipserverone for stopping this attack and giving me back my bandwidth!
Since I run this blog from from my home web server, I have quick access to the web logs to see how my site is doing, I check this on an infrequent basis but always revisit them to see who/what is looking at my site.
My normal routine is to simply look at the log folder to see the file size of the daily logs, I’m used to seeing spikes in my traffic, caused by bots trying to guess my WordPress admin password, however on 20th December the file size was abnormally high, a look at the log shows a barrage of requests for only one image file:
Many IPs with different meta data all wanting the same file.
A look at the referring page in the logs pointed to a forum for people looking for freebies, and a popular forum at that! A forum user trying to get their point across about what Sky Router they had Google searched it, found my image and linked it on the forum post.
I didn’t necessarily have a problem with this, since it was a “not for profit” point, but my beef was that the user decided to link the raw 4320 x 2880 image taken off my bridge camera, weighing in at 4.87MB!
On the forum a few grumbles over the image started to appear, all about it causing long page load times, No shit! At my current upload speed it takes 76 seconds to download, couple that with around 5 requests per min it would take much longer.
Lessons Learned:
Watch your logs for spikes, content used on more popular websites.
Be aware that when you upload an image to WordPress, it keeps the original size for the expanded view.
Make sure ALL the content you host on your home server is tailored for the bandwidth, as you never know what may be used!
FOOTNOTE:
The forum post that my image was linked to was about an offer to get a free Sky Wireless Booster. So I thought I may as well:
