While I’ve been self-hosting websites since 2002, it wasn’t until 2015 that I started making a conscious effort to backup and retain web logs during server reinstalls and migrations.

To create statistics from these logs, I’ve been using Analog since around 2004. A simple parser that dates back to the early days of the internet and remains wholly unchanged for many decades.

This milestone prompted a nostalgia dive into the archives to see how far back I can go, uncovering Analog output from 2004, and an historic look on how the internet has changed over the last 20 years…

History

Since getting online in the late 90’s, I had always found the concept of networks and being able to access servers and web content from anywhere in the world absolutely fascinating.

I dabbled in html in the early years, creating my stake on the internet using Yahoo GeoCities. So, when given the opportunity to build a website for my friend’s music band, I wanted to create something bespoke; something that included social interaction, but being fully integrated to the website and importantly, without adverts.

Back then, to do this for free would involve creating a guestbook on a 3^rd party site such as Lycos, among the other web tools that were rife at the time. Hosting dynamic content was relatively involved and not inexpensive, so providers such as Lycos would offer to do this for you, and support it with adverts. I’ve touched on this a while back following the end of Lycos. LINK

My solution, host the web server myself! The rational was that using my own server I could serve dynamic content and fully integrate it into the website. The catalyst was recently getting an “always on” internet connection in the form of cable internet in 2001, and that my first PC was made obsolete by the introduction of my second PC.

In 2002, using my first PC, Windows NT 4.0 Server, a dynamic DNS service, and purchasing a Netgear RT311 router, I started self-hosting.

Logging

The thrill of self-hosting came from the IIS logs, constantly reviewing them to see connections from around the world requesting files from the box in the corner of my room.

Analog, the web log parser came along as an evolution to this, allow consolidation of the logs into a single web page for easy reference and wonder. The more the logs grew, the more the statistics grew and the trends started to develop and reinforce themselves.

Analog has been around since 1995 and is largely unchanged, but still receives periodic maintenance updates from the C:Amie Edition.

10 Year Trend

Starting with the 10 year view, these are my points of interest for the modern internet, from this website’s perspective…

Traffic exploded during lockdown

Page requests and visits really took off in 2020, seeing over a five-fold increase in traffic from June 2020. Its unsure if it was a popular post, or the Google algorithm looked favourably on my site, but people sure had a lot of spare time around this period.

Self-hosting shielded bot access

I’ve recently moved away from hosting directly from home (more in a future post) and since moving presence to a commercial VPS service I’ve seen an uptick in visits. This should be good news, but a deeper look at the logs and the increased traffic is from malicious bots, targeting the common WordPress vulnerabilities. These bots must target IP ranges of established hosting providers.

Domains mean nothing anymore

Back when I started viewing Analog reports, the domain report was my way of gauging where in the world a visitor is from. Since then, the boundaries of domains have blurred drastically, and more used for vanity rather that identification. For example, have I really had 13722 visits from .vn (Vietnam)?

Cloudflare ruins logs

While hosting from home, I used Cloudflare as a proxy for a thin veil of security against exposing my IP. This had a detriment to the logs in the form of caching, where Cloudflare would serve a page rather than my server, and so not getting the log of the transaction. This caching also did register as a noticeable drop in traffic.

Bots, bots everywhere

Back in 2004, I only need worry about web crawlers scraping and indexing the site. Today, anyone can use a cheap VPS or commercial VPN service to anonymously attack and brute force a site it has become an epidemic. Now, my highest failure reports come from other hosting services.

Operating Systems

Finally, did someone really try to visit this site on a RISC OS system or even an Amiga?

20 Year Snapshot

Turning the clock back 20 years to my hosting infancy, I found an old report buried in a website backup, again, parsed by Analog.

Granted, this is a snapshot of from a much-reduced sample period, but in honesty, believe the real human visitor count is on par to what this website is now. Albeit catering to very localised content.

Hourly trends were what you expect.

This may be highly skewed by the localised content of the site at the time, but what struck me was the predictability of when users were visiting the site. With a sharp drop during the early hours and peaking in the evening. An indication that these were real visitors.

File requests were easier to understand.

A benefit of creating the website yourself. With the exception of forums, off the shelf applications such as WordPress were yet to materialise. Therefore, the file request reports were easier to decipher and to identify popular sections.

Data transfer was at a premium.

Self-hosting relied on using your home connection to serve pages to the world, and given that most home internet connections are asynchronous, you had to make the website as efficient as possible.

During this snapshot, while I had a modest 600kbps downstream bandwidth, the upstream utilised by visitors was just 128kbps.

Operating Systems

More nostalgia than a trend, but this complete list of OS visitors takes you back.

Summary

While a delightful look back, both at the web logs and the website files the discovery was bundled with. It makes you want to go back to those simpler times.

Where a home server can differentiate greatly from hosted solutions is the speed and relative location on the net. Visitors who view a website relies on the upstream connection at the server end to receive the content, and when this is via domestic internet connection the upstream can much smaller than the heavily advertised downstream connection. So it’s worth checking the theoretical upload speed to establish what kind of service and content can be served.

In terms of location, hosting companies are as close to the internet backbone as feasibly possible to get the best speeds and lower latency. The backbone of the net is handled by major operation companies that handle the bulk of all internet traffic between countries and continents, these in turn have datacentres where the traffic from countries are trunked to the different internet providers and down to the end user. As data makes its way from the backbone to the end user, it can hop between different servers as it meanders towards the final destination. For each hop the networking equipment has to read where to send it on, and route it on the right path. This all takes time, even though it is measured in milliseconds, an extended number of hops and the volume of data packets needed may produce a noticeable wait for a user to see the desired page.

All home user’s computers need deal with negotiating its way through the service providers’ local infrastructure to get to most sites, but when visiting a site hosted on a home server, data may need to navigate another service providers’ network to reach the site. This is where visitors may experience slower loading times compared to mainstream sites.
So how to tell if your home hosted website will be speedy when out in the wild? There’s a few different ways to check:

DIY Simulation

Firstly, let’s address if the upstream connection may be a cause of concern. Start by finding out the true upstream connection speed, this can be achieved by both running an online speed check and noting down the upload/upstream speed. Alternatively look around the modem’s setting pages to establish the theoretical maximum upload speed.
Then use the developer tools that come with most popular web browsers to simulate loading the site with the upstream bandwidth that may be available to visitors. In the browser, using Chrome for this example, press F12 to bring up the developer tools. Then click on the Network tab and choose the No Throttling button. There are a number of pre-sets available but to get the best scenario, create a custom profile.

For the new profile, your upload speed will become the profiles downstream speed, and your downstream speed can be the upload speed, latency is also important, and where the use of Dynamic DNS can really slow a connection down. To find out a realistic latency value, perform an online ping test to your own domain, and choose a latency value based on the results.

Hit F5 to refresh the page and watch how the site loads from an outside visitor’s perspective. The graph created in the developer tools gives insight into what elements of the page loaded and when. Interesting to note is the top row entry with the URL to the chosen page, as this is the time the visitor would have waited before seeing any text or loading action on their screen. It’s critical to get this quickly as with the increasing speeds of the internet, end users will quickly dismiss a site as being not available within a single second of not seeing any new content on screen.
This method is great when updating a site as it gives instant insight to how the page will load for visitors with any page change impacts can be viewed on the fly.

Third Party

The second option is to use a third party, a service that is already out in the internet wilderness and able to provide detailed insight to website loading times. This gives advantages over browser simulation as the virtual visitor is an actual machine on the internet, so any delays that a real visitor will have can be observed. But be aware that these services are servers themselves, and therefore may be closer to the internet backbone than the typical home visitor. Even though it’s a great tool to use periodically to gauge how the site is performing.
I prefer to use webpagetest.org to check my site, as along with bar graphs that display loading times for individual elements, it also gives the option to view a video that visually simulates the loading times just like a real visitor to the site would.

Get Mobile

A third and simpler option is to just find another computer away from the local network and visit the website to get a feel for loading times. I’ve used this in the past and it does indeed give a good indication of speed of load. However, be aware that the biggest skew point of this method is the choice of internet service provider, if the internet connection of your test computer and home server is via the same provider then the data would need only travel through its infrastructure creating a false speed advantage, not to say that it’s incorrect. If a site was to target users of the same ISP in the same locale then this would be considered an advantage, but does not provide an accurate sample of the wider user base. If the test computer was on a different ISP infrastructure, this is a better simulation as packets would still need to go to a datacentre to transfer between companies, much like it would for users over a country regardless of location. But be aware that this method is based on best case scenario.

And thanks to the steer to mobile devices, a public Wi-Fi or mobile data can be used to test a website. Cellular networks will definitely use a different infrastructure to the one the home server resides on, but less chance on public Wi-Fi. But with all networks and especially with public ones your test data may be competing with many other users locally, that can negatively affect the results.

For this project I recommend a RPi 2, as its powerful enough to perform the transfers up to its maximum 100Mbps network speed, and is a cheaper choice since the RPi 3 superseded it last year. To get started, you need the following hardware:

• Raspberry Pi 2 board.
• Micro USB wall adaptor – Needs to be capable of 2Amp output.
• Micro SD card – Minimum of 8GB, class 6 or above, plus SD adaptor for connecting to a PC.
• USB Portable Hard drive – USB powered is preferred, I use a Toshiba Canvio Basics 1TB.
• Ethernet Cable – And spare port on the modem/router for internet connection.
• A PC – On the same network as the Pi for connection and configuring.

The Pi was designed to as low cost as possible to the user, so apart from the Pi board you may already have everything to run a Pi, and if not these are cheap and easily available online.

With a RPi 2 at hand, let’s get started…

Setup

Before powering on, an operating system is needed on the micro SD card.

Download the latest version of Debian onto a PC and extract the ISO from the Zip file, then download Win32Imager that will transfer the image to a SD card.

Run Win32Imager as an Administrator by right clicking the icon, browse and choose the Debain ISO for the image, select the drive letter for the micro SD card, and click write. When complete, move the micro SD card to the Pi and its ready to boot for the first time.

Connecting

To make the Pi less obtrusive in the home, it will be configured and run as a headless server, therefore no monitor or Human Interface Devices needed.

Follow this guide I created earlier to get connected and into a terminal window.

Housekeeping

Once connected it’s a fully functional computer, but it’s worth the extra effort to perform some housekeeping to make it easier to connect to by assigning it a static IP, and changing the password for that extra layer of security.

First the essentials, after a fresh install of Debian the filesystem has to be expanded so it can make full use of the micro SD card’s capacity:

In the terminal window, type raspberrypi-config. Then from the menu choose expand filesystem. Reboot the Pi and connect again, now the full capacity is useable.

Next, and optionally, change the default password:

From the terminal, type sudo passwd, then follow the prompts to change it. No need to reboot.

Next is to assign the Pi a static IP address, allowing it to be found at the same location on the network every time, handy as connecting to the torrent client requires knowing the Pi’s IP address:

From the terminal, type sudonano /etc/network/interfaces to bring up the network config file. In the text editor that opens, find the string beginning eth0, and edit it to look like the following:

Where the gateway is the IP address of your router and the address will be the new home of the Pi, for those not familiar with IP addresses, copy the first three numbers from the routers IP address, and for the final number use a number between 150 and 253, this is usually high enough not to interfere with other devices on the network.

Use Ctrl + O to save the file, then Ctrl + X to exit.

Then type sudo reboot to restart the Pi, and now it can be found on the new chosen IP.

Connecting USB Hard Drive

By default the Pi does not supply enough power to the USB ports to power and mechanical drive, but with a little tweak of the settings it can deliver enough to fully power a USB hard drive.

In a terminal, type sudo nano /boot/config.txt

Scroll to the bottom of the file using the F8 key, then add the following new line:

Max_usb_current=1

Use Ctrl + O to save the file, then Ctrl + X to exit.

Reboot the Pi by typing sudo reboot.

From a new terminal window, check the drive is connected by typing
sudo blkid, it will appear as dev/sda1

With it recognised, use sudo mkdir /media/HDD to create a folder that will be used as a gateway to the new drive.

Then mount the drive by typing
sudo mount –t auto /dev/sda1  /media/HDD

The Pi can now write to the USB drive for the duration the Pi is powered on, but for simplicity it needs to be available if the Pi reboots…

Configuring & Sharing USB Hard Drive

To allow the drive to automatically mount upon startup, type
sudo nano /etc/fstab to open the file system config file.

Add the following line under the existing entries:

/dev/sda1 /media/USBHDD1 auto noatime 0 0

Use Ctrl + O to save the file, then Ctrl + X to exit.

Next up is to install samba, it allows the Pi to communicate with windows computers on the network, allowing easy access to downloaded files.

Type sudo apt-get install samba

When installed, open the config file by typing sudo nano /etc/samba/smb.conf

Enter the following information at the bottom of the file:

[Media]
comment = Mass Storage Share
path = /media/HDD
browsable = yes
read only = no
force user = root

Ctrl + O to save the file, then Ctrl + X to exit, then reboot with sudo reboot. When restart completes the Pi can be accessed by typing \\raspberrypi in a folder address bar, or will show as a network device in Windows explorer.

Installing Transmission

For torrent sharing, Transmission is used as a simple and quick program.

Update the Pi by using sudo apt-get update, followed by sudo apt-get upgrade. These may take a while to complete but ensures the latest software is running, including the latest version of Transmission.

Then type sudo apt-get install transmission-daemon and install the program, pressing Y when prompted.

Configuring Transmission

As you may have guessed, configuring anything in Linux means editing a text file, but first the Transmission service must be stopped to allow changes, by typing sudo server transmission-daemon stop.

Now open the settings by using
sudo nano /etc/transmission-daemon/settings.json

Here are all the settings, but for now scroll (F7&F8 keys) to locate rpc-whitelist, and edit it to include your local ip address range, like below:

Then change download-dir and incomplete-dir to point to a folder on the USB hard drive, using /media/HDD/Complete and /media/HDD/Downloading respectively. Be sure to create a Complete and Downloading folder on the hard drive before setting these, this can be done in Windows explorer by navigating to \\raspberrypi .

Also for convenience the password can be switched off by changing
rpc-authentication-required from true to false.

As always, Ctrl + O to save the file, then Ctrl + X to exit.

Using Transmission

The great feature of Transmission is its own web server and that all control is done via a webpage, allowing the Pi to function headless and the torrents managed by any internet capable device.

On another PC, open your favourite browser and navigate to
http://PI-IP:9091 replacing PI-IP with the static address you chose earlier.

From here it’s a fully functional, low cost, low power torrent box. An easy test is to revisit the RPi foundation’s website and download Debian via torrent, then seed it help others discover the wonderful world of Pi.

Unfortunately, the UPS in use only has one monitoring port which is connected to my original server, meaning during a spell of prolonged power outage, the new server will not know when to shut down safely and will continue to run until the UPS runs out of battery supply, leaving it vulnerable to data corruption that affected the original server prior to battery backup.

But with a simple script and some setup both servers can shut down safely before the batteries run out.

The Idea is to have a script sent from the host server with instructions to shut down the other server when the battery level runs low, luckily Windows gives the ability to run a program on low and critical power level notifications, with adjustable power level triggers.

The most important factor with this idea is that the host server is able to communicate and send the script to the other server, in my current setup both servers are connected to the main network switch that does not have battery backup, meaning that during a power outage the servers will be unable to communicate and therefore unable to shut down safely.

The simple solution was to introduce another switch close to the servers that has power provided by the UPS, for this I chose a NetgearProsafe GS105 and connected both servers to it, with an uplink to the main switch. This also has the advantage of removing traffic away from the main hub when backups between servers take place.

With the servers connected via a backed up power source, it is time to create the shutdown command. On the host server connected to the signal of the UPS, create a .bat file using Notepad with the following line:

shutdown /s /f /m \\MACHINENAME /t 10 /c “Battery Backup Low”

This script tells the computer MACHINENAME to force a shut down in 10 seconds regardless of who is logged in, for those that are the message “Battery Backup Low” will be displayed. The script was then saved to the Windows directory to help avoid accidental deletion.

From the host server, open the power settings from the Control Panel, on the Alarms tab choose Alarm Action on the Low battery alarm field. Check the Run program box and choose the .bat file created to run on the low battery alarm. I chose the Low power instead of Critical to allow the other server to shut down as the Critical would be reserved for the host server itself to shut down.

For testing I chose to put the alarm settings pretty high, which makes the process quicker and also allows more time for a safe shutdown when on battery power. Apart from that the only way to test is to kill the power from the wall and see what happens, look for the other server setting down first as it will hit the Low battery first, then wait for the host server to shut down later. Expect the host server to last a little longer than expected as with the other server shut down, the battery life doubles on two identical machines when one is off.

Recently I have been using Amazon EC2 cloud services for all my website hosting, new customers to the service get a 12 months free teir1.micro instance with myself opting for Windows Server 2012 R2 as my OS, and I’ve taken up this offer since October last year when an extended spell of server woes left me unable to serve websites.

A caveat with the free EC2 instance is the billing process, while the instance is free you have to pay attention to what is included as part of the offer, network usage, hard drive capacities and security keys are subject to charge over certain thresholds, so be aware.

With the trail due to expire in a few months I preferred to avoid an ongoing monthly cost and bring website hosting back to my own server, but not my current server as I didn’t want a box that served the web as well as store all my personal files. The logical conclusion was to get a new server.

HP has just launched the Gen 9 series servers so the Gen 8 are being offered with hefty cash back offers, tempting but the entry level Gen 8 moved to using Intel Celeron processors, while having slightly better performance, the power consumption also increased. Combine that with the expense on upgrading the basic offering with more memory and storage the cheapest solution was to go for a second hand “Gen 7”, something I’m familiar with.

I settled on a used HP Microserver N54L, ready kitted out with 4GB EEC RAM, 2 x 2TB Seagate Barracuda drives, DVD-RW drive and an extra gigabit Ethernet port, plus I can add 2 x 1TB WD Green drives spare from an earlier upgrade. Experience from a micro EC2 instance showed that this server would not be the bottleneck for a website, that would be firmly in my internet connections court.

Going forward the plan is to mothball the N36L server running Win 2003, keeping it essentially as a NAS device and closing incoming connections from the net, with the newer N54L replacing it in everyday tasks . But until then the EC2 instance is free until October, giving me chance to experiment with the best setup before going live, virtualisation will get a look in too.

Recently while shopping for network cables to connect my new server I though it time to get a new line for the laptop since the plug clip on the current one was broken. I decided to go for Cat 6 cables just to keep up to date and nowadays there is not such a price premium over Cat 5e.

The first transfer of data through the new cable and instantly got 70MBps, nearly double the 40MBps I was getting before. Wondering why there was such a difference I looked at the replaced cable:

The old cable was in fact the first Ethernet cable I ever owned, acquired in 2000 as part of a Netgear home network kit and since had a lot of use as an emergency and temporary cable, it was used for the laptop only due to it not being used and long enough to reach. It turned out to be Cat 5 and in poor condition, so no wonder it couldn’t cope with speeds it wasn’t designed for.

Getting the most out of a Gigabit home network can be as simple as swapping over cables.

My first venture into this came about before it was even called “the cloud”, I was setting up a guestbook system to work on my otherwise static only website, this meant signing up to one of those “guestbook websites” such as Html Gear, which later became part of Lycos. In those days of 2001+ my biggest bugbear was the adverts that always accompanied a free guestbook setup.

Even though the static web pages I created with links to guestbook sites have long past, I have enjoyed reminiscing the past posts on the guestbook of my creations, made possible only by looking through the backups of old websites and extracting the obscure URL from the HTML.

Alas, on my recent nostalgic trip of past websites I was met by this handy message:

Thanks I thought, my piece of data that I thought would carry me till my twilight years has been suddenly ripped from my heart. A bit dramatic, but that’s what I thought.

Travelling back to early 2001, I simply didn’t have the resource to host a guestbook locally, whether it be with hosted solutions that did not allow dynamic code (remember Geocities), or my own first home server that did not have the power to process code without time-outs.

Since then I quickly learnt to try my best at hosting forums (the then next step after guestbooks) locally on my own server, free to dispose of the data as I wish. In the years that follow I realise it may be a pain to find/write the code to display the data as it was, but I will always have that data close to me, free to read the raw database and exercise my nostalgia.

Furthermore, this event takes me back to when me and my peers used a service called MSN Groups, those of you that remember the acronym MSN assume that it is now doomed. It is, with all of our social commentary and all photos since deleted “cos it’s on MSN so we don’t need to keep our own local copy”.

Be warned, the cloud may be a quick, cheap and easy alternative to learning things properly, but rest assured the data you create WILL NOT be around forever, or even long enough for you to get bored of it.

If you run a website where you are dealing with data, guestbook, forums etc. Get your own server! Or least a service that allows you to pull all the data to a local copy.

I took delivery of a Netgear FVS318N router to replace a basic hub, installed it and did a bit of cable management which involved unplugging my Sagem F@st 2504 that I use as a modem.

However upon powering up the Sagem after tidying cables, it has no life, apart from this strange arrangement of light on the front:

I called Sky (my ISP) support who happily informed me that there is a common issue with the power supply to the Sagem router that caused them to fail. Wanting to get back on the net immediately and conversation about a replacement power supply giving vague delivery lead times, I opted to purchase the new Sky branded router (dubbed the Sky Hub):

The outage lasted 5 days as I waited for delivery of the new modem. An annoyance of this is that I had a spare, working ADSL modem but this could not be used as Sky does not give out the credentials to log on to their network, instead choosing to pre-load them on the modem before shipping.

Overall its an example of the unexpected issues that can arise when running a home server on a budget.

BOOTNOTE:

It has been mentioned in many Sky internet forums that using an unapproved Sky router, i.e. one not supplied by Sky, will be in breach of the Terms & Conditions. However whilst on the phone to Sky broadband technical support the representative told me that it was acceptable to use a 3rd party router if the user was confident and acknowledged that no support would be given unless a Sky provided router was used.

The case may be that you still need to hand over the cash to Sky for one of their routers and keep it to hand, but after that the choice is yours!

This blog is here mainly to accompany my other website over at diyhomeserver.co.uk. There you have a guide to building, setting up and enjoying your own personal home server, while any updates and new tricks I learn from the net will appear here.

Apart from that, this blog will also house my usual ramblings about the world and just what I find absolutely absurd about it.

Feel free to check in, maybe you will find something interesting and new, but probably, you will work out how different we really are.