This week I noticed a little milestone; my current iteration of website logs had clicked over to 10 years’ worth of accumulated statistics.
While I’ve been self-hosting websites since 2002, it wasn’t until 2015 that I started making a conscious effort to backup and retain web logs during server reinstalls and migrations.
To create statistics from these logs, I’ve been using Analog since around 2004. A simple parser that dates back to the early days of the internet and remains wholly unchanged for many decades.
This milestone prompted a nostalgia dive into the archives to see how far back I can go, uncovering Analog output from 2004, and an historic look on how the internet has changed over the last 20 years…
History
Since getting online in the late 90’s, I had always found the concept of networks and being able to access servers and web content from anywhere in the world absolutely fascinating.
I dabbled in html in the early years, creating my stake on the internet using Yahoo GeoCities. So, when given the opportunity to build a website for my friend’s music band, I wanted to create something bespoke; something that included social interaction, but being fully integrated to the website and importantly, without adverts.
Back then, to do this for free would involve creating a guestbook on a 3rd party site such as Lycos, among the other web tools that were rife at the time. Hosting dynamic content was relatively involved and not inexpensive, so providers such as Lycos would offer to do this for you, and support it with adverts. I’ve touched on this a while back following the end of Lycos. LINK
My solution, host the web server myself! The rational was that using my own server I could serve dynamic content and fully integrate it into the website. The catalyst was recently getting an “always on” internet connection in the form of cable internet in 2001, and that my first PC was made obsolete by the introduction of my second PC.
In 2002, using my first PC, Windows NT 4.0 Server, a dynamic DNS service, and purchasing a Netgear RT311 router, I started self-hosting.
Logging
The thrill of self-hosting came from the IIS logs, constantly reviewing them to see connections from around the world requesting files from the box in the corner of my room.
Analog, the web log parser came along as an evolution to this, allow consolidation of the logs into a single web page for easy reference and wonder. The more the logs grew, the more the statistics grew and the trends started to develop and reinforce themselves.
Analog has been around since 1995 and is largely unchanged, but still receives periodic maintenance updates from the C:Amie Edition.
10 Year Trend
Starting with the 10 year view, these are my points of interest for the modern internet, from this website’s perspective…
Traffic exploded during lockdown
Page requests and visits really took off in 2020, seeing over a five-fold increase in traffic from June 2020. Its unsure if it was a popular post, or the Google algorithm looked favourably on my site, but people sure had a lot of spare time around this period.
Self-hosting shielded bot access
I’ve recently moved away from hosting directly from home (more in a future post) and since moving presence to a commercial VPS service I’ve seen an uptick in visits. This should be good news, but a deeper look at the logs and the increased traffic is from malicious bots, targeting the common WordPress vulnerabilities. These bots must target IP ranges of established hosting providers.
Domains mean nothing anymore
Back when I started viewing Analog reports, the domain report was my way of gauging where in the world a visitor is from. Since then, the boundaries of domains have blurred drastically, and more used for vanity rather that identification. For example, have I really had 13722 visits from .vn (Vietnam)?
Cloudflare ruins logs
While hosting from home, I used Cloudflare as a proxy for a thin veil of security against exposing my IP. This had a detriment to the logs in the form of caching, where Cloudflare would serve a page rather than my server, and so not getting the log of the transaction. This caching also did register as a noticeable drop in traffic.
Bots, bots everywhere
Back in 2004, I only need worry about web crawlers scraping and indexing the site. Today, anyone can use a cheap VPS or commercial VPN service to anonymously attack and brute force a site it has become an epidemic. Now, my highest failure reports come from other hosting services.
Operating Systems
Finally, did someone really try to visit this site on a RISC OS system or even an Amiga?
20 Year Snapshot
Turning the clock back 20 years to my hosting infancy, I found an old report buried in a website backup, again, parsed by Analog.
Granted, this is a snapshot of from a much-reduced sample period, but in honesty, believe the real human visitor count is on par to what this website is now. Albeit catering to very localised content.
Hourly trends were what you expect.
This may be highly skewed by the localised content of the site at the time, but what struck me was the predictability of when users were visiting the site. With a sharp drop during the early hours and peaking in the evening. An indication that these were real visitors.
File requests were easier to understand.
A benefit of creating the website yourself. With the exception of forums, off the shelf applications such as WordPress were yet to materialise. Therefore, the file request reports were easier to decipher and to identify popular sections.
Data transfer was at a premium.
Self-hosting relied on using your home connection to serve pages to the world, and given that most home internet connections are asynchronous, you had to make the website as efficient as possible.
During this snapshot, while I had a modest 600kbps downstream bandwidth, the upstream utilised by visitors was just 128kbps.
Operating Systems
More nostalgia than a trend, but this complete list of OS visitors takes you back.
Summary
While a delightful look back, both at the web logs and the website files the discovery was bundled with. It makes you want to go back to those simpler times.