This created a new issue, since Grafana’s supported browser list moves with the times, my two “NOC” screens consisting each of a Raspberry Pi 3A running Raspbian Buster and Hyperpixel 4.0 screens no longer worked. Instead, the outdated Chromium browsers now displayed the “If you’re seeing this Grafana has failed to load its application files” page when visiting the Grafana web interface.

Bringing the OS on the Pi’s up to date is the logical resolution, however, there are a couple of issues with this:

• One of the Hyperpixel 4.0 screens is an early revision model, meaning the official and legacy drivers do not work.
• Even with working screen drivers, compatible browsers will not load on the 512MB RAM available to the Pi 3A.

Not wanting to obsolete a pair of Pi 3A’s with a now unique form factor, instead a workaround was found to continue displaying Grafana dashboards on this older hardware. The solution even gave a slight benefit over the original URL…

To achieve this, Grafana has released grafana-image-renderer, essentially a binary with a self-contained and headless Chromium instance that will load a dashboard and capture an image of the output.

Until recently, this was available as a Grafana plugin, however this plugin has since been depreciated in favour of it being a standalone application or docker container. A lot of online documentation still refers to it in its plugin form and so is now outdated.

The current iteration of grafana-image-renderer recommends using it within a docker container due to the potential resource usage it can generate. However, for this guide, only the standalone binary will be used as its only generating two images, and I find resource usage is at a minimum.

Installation

Firstly, grab the binary and store it in a easy but sensible location:

cd /opt
wget https://github.com/grafana/grafana-image-renderer/releases/download/v5.0.6/grafana-image-renderer-linux-amd64

Then make the file executable:

chmod +x grafana-image-renderer-linux-amd64

To have it run as a service, create a service file for it:

nano /etc/systemd/system/grafana-image-renderer.service

And populate the file with the following:

[Unit]
Description=Grafana Image Renderer service
After=network.target

[Service]
ExecStart=/opt/grafana-image-renderer-linux-amd64 server --
WorkingDirectory=/opt/
Restart=on-failure
User=root
Environment=GRAFANA_RENDERER_PLUGIN_STARTUP_TIMEOUT=30s 
[Install]
WantedBy=multi-user.target

Reload the service daemon, start the new service, and check it is running:

systemctl daemon-reload
systemctl start grafana-image-renderer.service
systemctl status grafana-image-renderer.service

You can check If the service is running by visiting the web page of the server on port 8081:

Authentication

A default Grafana server install requires login before being able to see any dashboards, this extends to the image renderer. As we can’t login to the web interface interactively within the service, a service account needs to be created.

In the main Grafana web interface, login and navigate to Home > Administration > Users and access > Service accounts, then click “Add service account”.

Give it a display name and set the role to viewer:

Now in the newly created service account, click “Add service token”, then on the popup dialog, click “Generate Token”.

A token with a “glsa_” prefix is displayed, make a copy of this as it will not be available to view after closing this dialog box:

This will be used as authentication when generating our dashboard images…

Generating Images

With the renderer installed, and authentication available, images can now be generated.

Images are generated by requesting a URL from the service with the dashboard within the variables, the syntax is:

http://{grafana-image-renderer}:8081/render?encoding=png&url=http://{grafana-dashboard}?{options}

Where:
Grafana-image-renderer – The IP of the machine the renderer is running on.
Grafana-dashboard – The URL of the dashboard you’d like to capture.
Options – Options to manipulate the output/how the dashboard is displayed.

However, the authentication header is needed to successfully access the dashboard. For this reason its easier to use a curl command to set the headers and grab the image

For example, if the renderer is running on the same machine as the main Grafana, a dashboard can be generated by executing:

curl -H "X-Auth-Token: {glsa_token}" "http://localhost:8081/render?encoding=png&url=http://localhost:3000/d/{id}/{dashboad}?kiosk&width=800&height=600" -o image.png

The options used here is kiosk; to remove the menus from the output, and height and width to match the intended displays.

Automation:

One minute updates are fine for this use case, so the easiest way automate this is via cron. Using crontab -e, add the following line:

/usr/bin/curl -H "X-Auth-Token: {glsa_token}" "http://localhost:8081/render?encoding=png&url=http://localhost:3000/d/{id}/{dashboard}?kiosk&width=800&height=600" -o /var/www/html/dash1.png

This line generates the dashboard every minute and stores it on the default web server directory, allowing it to be viewed on the network via an installed web server.

But this will only create a static image, to allow the image to update on a display, a simple webpage can be created and stored on the webserver for easier access and reference.

Create a new webpage:

nano /var/www/html/dashboard.html

And paste the following:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Grafana Dashboard</title>
  <style>
    body {
      margin: 0;
      background-color: black;
      display: flex;
      justify-content: center;
      align-items: center;
      height: 100vh;
    }
    img {
      max-width: 100%;
      max-height: 100%;
      object-fit: contain;
    }
  </style>
</head>
<body>
  <img id="liveImage" src="dash1.png" alt="Live Image">
  <script>
    const img = document.getElementById("liveImage");
    setInterval(() => {
      const timestamp = new Date().getTime(); // avoid browser cache
      img.src = "dash1.png?t=" + timestamp;
    }, 60000); // 60 seconds
  </script>
</body>
</html>

When visited, this webpage will update the same image (dash1.png) every 60 seconds to ensure the browser updates with the latest available image instead of caching.

Displays

Viewing this on the display is as simple as visiting the above webpage. As the html is served from the Grafana server and where the auto refresh is taken care of, as long as the display is able to load the page, all is taken care of at the Grafana server.

Summary

Since all that is required from a display is being able to load and render a static image, this allows the oldest and slowest machines to be able to display a Grafana dashboard, negating the need to keep the displays up to date with Grafana’s evolving browser support.

I wish I learnt of this sooner, as it gives more standardised control of the layout of a dashboard. The HyperPixel displays are quite high resolution for their size, and loading a dashboard natively often lead to frustrations with the layout of panels not matching that displyed on a laptop/desktop screen. Presumably due to scaling. Using the image render allows you to test and fix the layout before deploying to a screen, and with a higher panel density too.

Another benefit is that most processing is taken away from the Pi 3A’s, as it is now only responsible for displaying a simple webpage with a single image, rather than loading the full Grafana UI. The Pis have been running on this new method for over a month without any lockups, which was previously seen on a weekly basis.

Instead of a backup and restore of the VM, it seems the right time to migrate Zabbix and Grafana from an aging RHEL 8 instance to a new VM running a fresh copy of Debian 13. At the same time upgrading the applications to their latest versions…

Zabbix

New Server: Preparation

Begin by installing a LAMP stack on the new server:

apt install mariadb-server apache2 php php-mysql php-bcmath php-mbstring php-xml php-ldap php-json php-gd php-zip curl gnupg lsb-release

Then setup the basic configuration of MariaDB:

mysql -u root

ALTER USER 'root'@'localhost' IDENTIFIED BY 'password';
DELETE FROM mysql.user WHERE User='';
DELETE FROM mysql.user WHERE User='root' AND Host NOT IN ('localhost', '127.0.0.1', '::1');
DROP DATABASE IF EXISTS test;
DELETE FROM mysql.db WHERE Db='test' OR Db='test\\_%';
FLUSH PRIVILEGES;

To give it somewhere to migrate to, install a fresh, blank copy of Zabbix on the new server. The latest version can be used and is available on the Zabbix website, as the application will detect and auto upgrade your exisiting data (So long as a direct upgrade path between versions is supported).

As an example with version 7.4, first add the Zabbix repositories:

wget https://repo.zabbix.com/zabbix/7.4/release/debian/pool/main/z/zabbix-release/zabbix-release_latest_7.4+debian13_all.deb
dpkg -i zabbix-release_latest_7.4+debian13_all.deb
apt update

And install Zabbix:

apt install zabbix-server-mysql zabbix-frontend-php zabbix-apache-conf zabbix-sql-scripts zabbix-agent2

If following the install intructions from the Zabbix website, stop before starting the service. Instead, we’ll first pull and import the database from the old server…

Old Server: Export database

On the old/source server, temporarily stop Zabbix server:

systemctl stop zabbix-server

Now dump the database to a file. As this old server only ran Zabbix, we’ll dump the whole MySQL instance to preserve users and settings which will make migration easier.

If yours is more complex, dump just the Zabbix database and recreate the users manually on the new server.

Export the whole MySQL instance:

mysqldump -u root -p --all-databases > mysql.sql

When it completes, restart Zabbix on the old server, just incase you’d need another go at it later.

systemctl start zabbix-server

New Server: Import

Back on the new server, pull the exported database locally:

scp root@{old-server-ip}:/root/mysql.sql .

Also grab the configuration files of the old server, it’ll come in handy later:

scp -r root@{old-server-ip}:/etc/zabbix/* .

Import the copied database:

mysql -u root -p < mysql.sql

If importing an “–all-databases” export, the newly added users need to be commited before they can be used.

Note that after executing the below, the root credentials will be overwritten by those of the old server:

mysql -u root -p
FLUSH PRIVILEGES;

Next is the configuration file. I would recommend comparing the old and new /etc/zabbix/zabbix_server.conf files, and updating the new .conf to match the old rather than just copying the file.

nano /etc/zabbix/zabbix_server.conf

Now, its time to start Zabbix:

systemctl enable --now zabbix-server

Checking for any errors can be made by monitoring the log file:

tail -n 100 /var/log/zabbix/zabbix_server.log

Soon after starting the service you’ll notice a database upgrade completing in the file, this is Zabbix automatically upgrading your old data to the newly installed version, neat.

Visit the Zabbix web interface on the new server:

http://{new-server-ip}/zabbix/

You’ll be greeted with the setup wizard, follow the steps on screen to complete.

Don’t worry, Zabbix is running and logging now data to your existing Zabbix hosts, this wizard simply creates the /etc/zabbix/web/zabbix.conf.php file.

Once the wizard is completed, you’ll return to a familiar yet updated interface via the new server.

Zabbix Proxy

If the migration involves a major version change and you’re using proxies, the dashboard will quickly flag a problem with the proxy version being incompatible.

On the proxy, you’ll need to update the application by setting the associated version in the repositories and installing the updated version.

Using a Rocky 8 install as an example, add the updated repo:

rpm -Uvh https://repo.zabbix.com/zabbix/7.4/release/rocky/8/noarch/zabbix-release-latest-7.4.el8.noarch.rpm
dnf clean all

Then install the updated version:

Install
dnf install zabbix-proxy-mysql

Restart the service, and it will be up to date:

systemctl restart zabbix-proxy

Grafana

Migrating and updating Grafana is much the same process as Zabbix, if a little easier.

Install Grafana, on Debian 13 it is available from the main repository:

apt install grafana

Pull the required files direct from the old server to the new:

scp root@{old-server-ip}:/etc/grafana/grafana.ini .
scp root@{old-server-ip}:/var/lib/grafana/grafana.db .

Overwrite the installed files with the ones pulled from the old server, and set the correct permissions:

cp grafana.ini /etc/grafana/grafana.ini
chown root:grafana /etc/grafana/grafana.ini
cp grafana.db /var/lib/grafana/grafana.db
chown grafana:grafana /var/lib/grafana/grafana.db

Before starting, install any plugins used on the old server, for example:

grafana-cli plugins install alexanderzobnin-zabbix-app
grafana-cli plugins install grafana-clock-panel

Now Grafana can be started:

systemctl enable --now grafana-server

And visit the web interface:

http://{new-server-ip}:3000/

Where you can pick up at where you left on the old server.

While I’ve been self-hosting websites since 2002, it wasn’t until 2015 that I started making a conscious effort to backup and retain web logs during server reinstalls and migrations.

To create statistics from these logs, I’ve been using Analog since around 2004. A simple parser that dates back to the early days of the internet and remains wholly unchanged for many decades.

This milestone prompted a nostalgia dive into the archives to see how far back I can go, uncovering Analog output from 2004, and an historic look on how the internet has changed over the last 20 years…

History

Since getting online in the late 90’s, I had always found the concept of networks and being able to access servers and web content from anywhere in the world absolutely fascinating.

I dabbled in html in the early years, creating my stake on the internet using Yahoo GeoCities. So, when given the opportunity to build a website for my friend’s music band, I wanted to create something bespoke; something that included social interaction, but being fully integrated to the website and importantly, without adverts.

Back then, to do this for free would involve creating a guestbook on a 3^rd party site such as Lycos, among the other web tools that were rife at the time. Hosting dynamic content was relatively involved and not inexpensive, so providers such as Lycos would offer to do this for you, and support it with adverts. I’ve touched on this a while back following the end of Lycos. LINK

My solution, host the web server myself! The rational was that using my own server I could serve dynamic content and fully integrate it into the website. The catalyst was recently getting an “always on” internet connection in the form of cable internet in 2001, and that my first PC was made obsolete by the introduction of my second PC.

In 2002, using my first PC, Windows NT 4.0 Server, a dynamic DNS service, and purchasing a Netgear RT311 router, I started self-hosting.

Logging

The thrill of self-hosting came from the IIS logs, constantly reviewing them to see connections from around the world requesting files from the box in the corner of my room.

Analog, the web log parser came along as an evolution to this, allow consolidation of the logs into a single web page for easy reference and wonder. The more the logs grew, the more the statistics grew and the trends started to develop and reinforce themselves.

Analog has been around since 1995 and is largely unchanged, but still receives periodic maintenance updates from the C:Amie Edition.

10 Year Trend

Starting with the 10 year view, these are my points of interest for the modern internet, from this website’s perspective…

Traffic exploded during lockdown

Page requests and visits really took off in 2020, seeing over a five-fold increase in traffic from June 2020. Its unsure if it was a popular post, or the Google algorithm looked favourably on my site, but people sure had a lot of spare time around this period.

Self-hosting shielded bot access

I’ve recently moved away from hosting directly from home (more in a future post) and since moving presence to a commercial VPS service I’ve seen an uptick in visits. This should be good news, but a deeper look at the logs and the increased traffic is from malicious bots, targeting the common WordPress vulnerabilities. These bots must target IP ranges of established hosting providers.

Domains mean nothing anymore

Back when I started viewing Analog reports, the domain report was my way of gauging where in the world a visitor is from. Since then, the boundaries of domains have blurred drastically, and more used for vanity rather that identification. For example, have I really had 13722 visits from .vn (Vietnam)?

Cloudflare ruins logs

While hosting from home, I used Cloudflare as a proxy for a thin veil of security against exposing my IP. This had a detriment to the logs in the form of caching, where Cloudflare would serve a page rather than my server, and so not getting the log of the transaction. This caching also did register as a noticeable drop in traffic.

Bots, bots everywhere

Back in 2004, I only need worry about web crawlers scraping and indexing the site. Today, anyone can use a cheap VPS or commercial VPN service to anonymously attack and brute force a site it has become an epidemic. Now, my highest failure reports come from other hosting services.

Operating Systems

Finally, did someone really try to visit this site on a RISC OS system or even an Amiga?

20 Year Snapshot

Turning the clock back 20 years to my hosting infancy, I found an old report buried in a website backup, again, parsed by Analog.

Granted, this is a snapshot of from a much-reduced sample period, but in honesty, believe the real human visitor count is on par to what this website is now. Albeit catering to very localised content.

Hourly trends were what you expect.

This may be highly skewed by the localised content of the site at the time, but what struck me was the predictability of when users were visiting the site. With a sharp drop during the early hours and peaking in the evening. An indication that these were real visitors.

File requests were easier to understand.

A benefit of creating the website yourself. With the exception of forums, off the shelf applications such as WordPress were yet to materialise. Therefore, the file request reports were easier to decipher and to identify popular sections.

Data transfer was at a premium.

Self-hosting relied on using your home connection to serve pages to the world, and given that most home internet connections are asynchronous, you had to make the website as efficient as possible.

During this snapshot, while I had a modest 600kbps downstream bandwidth, the upstream utilised by visitors was just 128kbps.

Operating Systems

More nostalgia than a trend, but this complete list of OS visitors takes you back.

Summary

While a delightful look back, both at the web logs and the website files the discovery was bundled with. It makes you want to go back to those simpler times.

It wasn’t until my most recent visit for an upcoming project that things looked, well, a little different. It was mostly the same, but subtle differences meant it was no longer compatible with how I’d been familiar with setting it up.

Therefore this quick post is to capture the old method of setting it up for posterity.

The below steps are to replicate all MySQL databases on another server.

Prepare MySQL .conf files

To allow MySQL to replicate, the servers must be arranged in a hirarchy and logging enabled so each system can keep track and update any changes made.

The .conf files can be found in the following locations:

/etc/my.cnf.d/server.cnf

Or on newer distros:

/etc/mysql/mariadb.conf.d/50-server.cnf

Source/Master Server

On the source server (server you want to copy from), add the following lines below and within the [mysqld] section:

bind-address            = {local ip address},{remote ip address}
server-id               = 1
log_bin                 = /var/log/mariadb/mysql-bin.log

binlog-format = mixed
sync_binlog=1

Replica/Slave Server

On the replica server (server that databases are copied to), add the following lines below and within the [mysqld] section:

server-id               = 2
log_bin                 = /var/log/mysql/mysql-bin.log
relay-log               = /var/log/mysql/mysql-relay-bin.log
binlog-format           = mixed
read-only               = 1

On both servers, MySQL requires a restart to pick up the config changes:

systemctl restart mariadb

Create a replication user

A replication needs to be authenticated, rather than using the root user its preferred to create a user that is only able to perform the replication.

On the source server, login to MySQL:

mysql -u root -p

Create the user, and give it replication duties with the following commands:

CREATE USER '{replica_user}'@'{replica_ip}' IDENTIFIED BY '{password}';
GRANT REPLICATION SLAVE ON *.* TO '{replica_user}'@'{replica_ip}';
FLUSH PRIVILEGES;

Copy current databases

Before the replication can begin, we’ll need a copy of the existing data captured within a known point of time. This allows the replica to know where to pick up from where the imported databases left off.

Source Server

If not already, log into mysql :

mysql -u root -p

Lock the databases from being able to be written to:

FLUSH TABLES WITH READ LOCK;

Display the current state of the logfiles while the databases are frozen in time:

SHOW MASTER STATUS;

This will produce something similar to this:

MariaDB [(none)]> SHOW MASTER STATUS;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000016 |      342 |              |                  |
+------------------+----------+--------------+------------------+
1 row in set (0.000 sec)

Make a note of the File and Position values, we’ll need this as a reference point for the replica. I’d recommend saving it as a file for future reference.

Now, open up new SSH session to the source server. Its important that the current MySQL session is kept open and in its locked state.

Take a mysqldump of the entire database:

mysqldump -u root -p --all-databases > mysql_replication.sql

When this completes, return to the original session to return the databases to normal:

UNLOCK TABLES;

Transfer SQL file to Replica

Move the generated mysqldump file to the replica server, I usually transfer the master status txt file too for future reference:

scp mysql_replication.sql mysql_master.txt root@{replica_ip}:/root/

Setup Replication

Import the newly transferred file to the replica’s MySQL:

mysql -u root -p < mysql_replication.sql

Then log in to MySQL:

mysql -u root -p

Time to setup the replication, using the replica user setup previously, and the logfile positions captured during the mysqldump. The SHOW MASTER STATUS; output from above is referenced in this command example:

 CHANGE MASTER TO MASTER_HOST='{source_ip}', MASTER_USER='{replica_user}', MASTER_PASSWORD='{password}', MASTER_LOG_FILE='mysql-bin.000016', MASTER_LOG_POS=342;

Reference points now in place, start the replication process:

START SLAVE;

You can check it is running with:

SHOW SLAVE STATUS\G;

In the output you’re looking for Slave_IO_Running and Slave_SQL_Running to both be Yes:

With both reporting Yes, the replication is running. All changes made to the source server is automatically pushed to the replica.

Notes

Ports

The two databases communicate over its MySQL port, default is 3306. Ensure that both servers are able to communicate on this port.

Authentication

Immediately after setting up replication, you’ll be able to log in to the replica server as normal. However, if the service/server was to be restarted you may not be able to login as normal.

As this guide replicates everything on the source MySQL, it will also replicate the users and permissions.

Following a replication start, the “old” credentials are still usable But following a service restart, the credentials of the source are the new normal on the replica, and should be used for any future login.

This will also apply to any instances of phpMyAdmin.

Traditionally this service is provided via its supplied Fritzbox router, utilising its FXS port to supply telephony to analogue devices.

As the service is SIP based, it is possible to connect direct via SIP phones or Asterisk which is ideal should the Fritzbox not be suitable for your requirements. While this is permitted by Zen, it is unsupported.

Here are a couple of examples of getting connected to Zen DV without the use of the Fritzbox.

A Pre-Setup Warning

This process involves exposing a SIP server / device to the internet, this protocol is constantly targeted by hackers and fraudsters due to its reward for exposing a vulnerable system.

For example, hackers can exploit a system to generate high volumes of calls on your account for either their own benefit (calling a premium rate number they setup) or for the sheer fun of it, while fraudsters could use your “line” to make robocalls to other phone numbers.

Therefore, it is imperative to limit the service’s exposure to only the IP’s listed below, if your router is unable to open ports to specified addresses, consider upgrading or not proceeding.

Network

From a network point of view, VoIP is split into two sections; Signalling (SIP) that deals with the actions of a call, such as “start ringing” and “hangup now”. The Media (RTP) solely deals with the audio of the call and its setup is defined via the SIP signalling.

Uniquely For the DV service, Zen employs a separate server each for inbound and outbound calls:

SIP Inbound:
voip.zen.co.uk – 212.23.7.228

SIP Outbound:
voip2.zen.co.uk – 212.23.7.235

Media uses a common range:

RTP Media:

62.3.88.0/28
62.3.88.16/28

Ports used by the service are the standard:

SIP: 5060
RTP: 10000 – 20000

Setup

As we are trying to mimic the Fritzbox, which is the first point of contact from the internet, the above ports need to be passed onto your SIP device. As this could pass any internet traffic on these ports to the device it’s a good idea to limit these open ports to only the IPs above too.

This will differ depending on the router used, as an example this is how to set it up on a Draytek router:

Allowed Client List / Whitelist

In the router’s web interface, navigate to Objects Setting >> IP Object, add the IPs:

These need to be grouped together, in Objects Setting >> IP Groups, add these IPs into the group:

Open Ports

Now there is a list of allowed IPs, the ports can be opened, in NAT >> Open Ports, edit an index, setting:

WAN Interface: Port connected to Zen internet
Source IP: The group created above
Private IP: Local IP of SIP device
Ports: Open UDP ports 5060 and a range 10000-20000

Now VoIP traffic is forwarding to the device, it can be configured…

SIP

To authenticate with the service the phone number {number} (full number including area code, in national format, i.e starting 01 or 02) will be used as the username, and you’ll need the password {password}(secret). The password is available via the old Zen portal for legacy customer (https://portal.zen.co.uk/) or by contacting Zen and asking for “your VoIP password”.

Asterisk

As there are two servers in use for SIP signalling, two trunks are required on the PBX:

Trunk 1 – Inbound; This just needs to accept calls from Zen’s server so is pretty basic:

type=peer
trustrpid=yes
nat=yes
insecure=invite
host=voip.zen.co.uk
disallow=all
allow=alaw

Trunk 2 – Outbound; This contains authentication to login (register) to the service and make calls:

type=peer
secret={password}
host=voip2.zen.co.uk
fromuser={number}
disallow=all
authuser={number}
allow=alaw
nat=yes
insecure=invite
fromdomain=voip2.zen.co.uk
defaultuser={number}

The outbound trunk also requires the registration details, this allows Zen’s servers to recognise that your service is online and ready for calls:

{number}:{password}@voip2.zen.co.uk/{number}

If you’d like to monitor the connection, you’d see this in sngrep:

Note even when setup correctly, the initial registration attempt will fail as Asterisk attempts to register without authentication, upon the 401 error it will then attempt registration with authentication details and receive the 200 OK response and so registered to the service.

SIP Phone

To connect directly to SIP device, it’s mainly a case of just getting the phone to register, from there the Zen server will know it is logged in for calls and take it from there. The generic details needed are below, apply this best to the terminology used by the device:

Username / Register Name: {number}
Password / Secret: {password}

Server / Registrar / Host: voip2.zen.co.uk
Transport / Protocol: UDP
Port: 5060

For example, this is a setup of a Yealink phone:

If setup correctly, the registration status should state “Registered”.

This is for Yealink devices but will likely be applicable for other vendors, you will need to change the following settings in Features >> General Information:

Accept SIP Trust Server Only: This needs to be disabled as inbound calls come from a different server than the device is registered to.

Allow IP Call: Important to disable especially if there is no IP whitelisting on the router, this stops random or “ghost” calls generated by hackers trying to find a response from internet facing phones. This will not be an issue if you are able to restrict open WAN ports to allowed IP’s, but if not, this helps.

Conclusion

Setup, you should be able to use your Zen DV number with your VoIP / SIP devices.

For a single phone this is a simple solution, but for multiple phones around the home your either limited to only cordless phones from the likes of Yealink or Gigaset (where the base unit is the controller and SIP registration device).

But for a combination of landline and cordless, the VoIP revolution means the end of an era of running an extension cable to the next phone, you’ll need a local PBX of some sort to handle signalling.

Finale

This is also likely an end of an era for my VoIP based posts, as I have left my telecoms role for something different.

I may return sporadically to VoIP postings if I find a home use for it, or revive a post that didn’t quite get there. But for now, it’s a goodbye to SIP.

I’d like the KVM guests to access the network in bridge mode of the host for direct access to the network. However I faced the following issue:

• KVM host can ping gateway and internet.
• KVM host can ping the guest.
• Guest can ping the host.
• Guest cannot ping gateway or anything outside of the host.
• Guest is showing in router ARP table, with its IP address and own MAC

This one got me for more time than I wish to admit, and seems to have caught others out along the way, this is how I finally solved it…

This test setup has the following parameters:

• Network: 10.0.4.0/24
• Internet Gateway: 10.0.4.1
• KVM Host: 10.0.4.100
• GuestL 10.0.4.101

Install KVM and Cockpit

Install KVM and Cockpit (For easy access and testing of a guest):

apt install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils virtinst cockpit cockpit-machines

For Debain 12, Cockpit blocks login from root, remove it from the disallowed-users file:

nano /etc/cockpit/disallowed-users

Restart Cockpit to take changes:

systemctl restart cockpit

Cockpit’s web interface can be accessed via port 9090 of the host IP.

https://10.0.4.100:9090/

Host Network

Now bridge-utils is installed, the bridge can be configured on the host:

nano /etc/network/interfaces

Remove / comment out the physical interface config and add the new bridge br0:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug ens192
iface ens192 inet manual
        #address 10.0.4.100/24
        #gateway 10.0.4.1
        # dns-* options are implemented by the resolvconf package, if installed
        #dns-nameservers 10.0.4.1
        #dns-search debian-kvm.james-batchelor.local

auto br0
iface br0 inet static
        address 10.0.4.100
        netmask 255.255.255.0
        network 10.0.4.0
        broadcast 10.0.4.255
        gateway 10.0.4.1
        bridge_ports ens192
        bridge_stp off
        bridge_fd 0
        bridge_maxwait 0
        dns-nameservers 1.1.1.1
        hwaddress ether 00:0c:29:3a:bc:72
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward

NOTE: Add the hwaddress line and use the mac address of the physical interface, otherwise you’ll loose connection to the host when networking is restarted.
NOTE: For my solution, IP forwarding needs to be enabled.

Restart networking service, cross your fingers:

systemctl restart networking

Guest setup

Create folders for the guests:

mkdir /vm
mkdir /vm/iso
mkdir /vm/datastore

This issue got me for a while, so for testing I refined the process down to using a System Rescue cd for quick VM spin ups. Download the ISO:

wget https://fastly-cdn.system-rescue.org/releases/11.01/systemrescue-11.01-amd64.iso -P /vm/iso/

Now create the VM from the CLI:

virt-install --name vm-test --ram=8192 --vcpu=4 --cpu host-passthrough --disk path=/vm/datastore/vm-test,size=8 --cdrom /vm/iso/systemrescue-11.01-amd64.iso --os-variant linux2022 --network bridge=br0,model=virtio --graphics vnc

Now you can move to Cockpit to view and configure the guest from its console.

Guest Networking

Lets take a second to think about whats happening when a guest pings the network; It has to leave through the host in order to get to the gateway, as the ARP table has has the correct details for the guest (including guest’s actual MAC), the ping reply will be send to the guest but needs to go through the host first.

Therefore a static route is needed to steer traffic for the guest through the host. On the router, create a static route:

Via Cockpit, access the console of the System Rescue OS, enter the network config via:

nmtui

Navigate to edit a connection, and select the Wired Ethernet Connection, set IPv4 configuration to Manual and add the details:

• IP address: IP address you set as the destination of the static route.
• Gateway: IP address of host.

Select OK at the bottom of the screen to confirm config, go back to the main menu.

TIP: Navigate to Activate Connections and toggle the interface off and back on, as otherwise there is a lag before the config is applied to the network.

Test with pings from the guest and to it from other machines, they should now respond.

Last of the hurdles was to get data from Asterisk in order to send to Zabbix, however the traditional way of loading the res_snmp.so module in Asterisk was not available, as while the PBX in question utilises Asterisk, its buried under proprietary licensing and a non-standard api, therefore being unable to either add the SNMP module or to query it.

If you are experiencing a similar situation, here is how to extract some stats from Asterisk 16 using SNMP, but without the SNMP module…

Despite not being able edit Asterisk code directly, SSH access and the Asterisk CLI is available, so can run commands to return a snapshot of PBX activity.

The following commands can be run to output some valuable info:

asterisk -rx ‘pjsip show endpoints’ returns total number and current online endpoints, as well as reported user agent of device.

asterisk -rx ‘core show channels verbose’ gives us a list of active channels and current point in an inbound call flow, along with a counter of completed calls.

asterisk -rx ‘pjsip show channelstats’ is similar to ‘show channels verbose’, but includes the codec in use, helpful to determine if the system is transcoding.

Scripts

To get and output just the info required, a script can be created to parse the command output and list each value on a new line.

Create the three files below and paste the code in.
In this example I saved to /usr/local……

asterisk_chan.sh

command_output=$(asterisk -rx 'core show channels')

active=$(echo "$command_output" | grep -oE '[0-9]+ active channels' | grep -oE '[0-9]+')
processed=$(echo "$command_output" | grep -oE '[0-9]+ calls processed' | grep -oE '[0-9]+')

echo "$active"
echo "$processed"

asterisk_chanstat.sh

command_output=$(asterisk -rx 'pjsip show channelstats')

opus=$(echo "$command_output" | grep 'opus' | (wc -l))
g722=$(echo "$command_output" | grep 'g722' | (wc -l))
alaw=$(echo "$command_output" | grep 'alaw' | (wc -l))
ulaw=$(echo "$command_output" | grep 'ulaw' | (wc -l))
g729=$(echo "$command_output" | grep 'ulaw' | (wc -l))
total=$(echo "$command_output" | grep -oP 'Objects found: \K\d+' || echo 0)

echo "$opus"
echo "$g722"
echo "$alaw"
echo "$ulaw"
echo "$g729"
echo "$total"

asterisk_endpoint.sh

command_output=$(asterisk -rx 'pjsip show endpoints')

avail=$(echo "$command_output" | grep 'Avail' | (wc -l))
total=$(( $(echo "$command_output" | grep -c 'Endpoint') - 1 ))

echo "$avail"
echo "$total"

Saved, they need to be made executable, add the execute flag to each file…

chmod +x /usr/local/asterisk_chan.sh
chmod +x /usr/local/asterisk_chanstat.sh
chmod +x /usr/local/asterisk_endpoint.sh

SNMP extend_sh

SNMP has an extension feature to add and poll your own custom data, to take advantage of this the snmp.d.conf file need to include reference and path to the scripts. At the bottom of snmpd.conf, add the following lines…

extend-sh asterisk_chan /usr/local/share/snmp/rps/asterisk_chan.sh
extend-sh asterisk_chanstat /usr/local/share/snmp/rps/asterisk_chanstat.sh
extend-sh asterisk_endpoint /usr/local/share/snmp/rps/asterisk_endpoint.sh

Restart snmpd to load changes…

Gentoo:

/etc/init.d/snmpd restart

RHEL:

systemctl restart snmpd

Testing

Using snmpget command locally or on your collector machine, test that its working..

snmpget -c }your-community} -v 2c {ip-address} 'NET-SNMP-EXTEND-MIB::nsExtendOutLine."asterisk_chan".1'

Where “asterisk_chan” is the name of the extend_sh created in snmpd.conf file, and .1 on the end collates to the line of script output, .2 returns the second line and so on.

Add to monitoring

When adding to your monitoring system, in this example Zabbix, use the command listed in the testing phase to grab the values…

Logging and monitoring is now enabled for the Asterisk service…

Many of our PBX’s were deployed from the vendors ISO and so run atop of Gentoo, and it has a couple of issues:

1. We’ve been told not to “emerge” anything by the vendor, as the base OS on the image is not maintained.
2. Portage (Gentoo’s package manager) has fallen out of date, meaning even if emerge is attempted, it’ll fail as all repository links are broke.

If faced with the same issue, this is how to install net-snmpd from source, add it a startup service and be able to monitor via snmp…

Installation

On the Gentoo box, move to your home directory

cd ~

Download the source .tar.gz file

wget https://sourceforge.net/projects/net-snmp/files/net-snmp/5.9.4/net-snmp-5.9.4.tar.gz

Visit http://www.net-snmp.org/download.html for the latest version. When I was downloading SourceForge’s SSL cert had expired, if this happens add –no-check-certificate after wget in the command above.

Untar the file to your home directory, and change into it

tar -xzvf net-snmp-5.9.4.tar.gz
cd net-snmp-5.9.4

Run the configuration script

./configure

This allows you to set snmp version, file installation locations and snmp details, with exception of snmp version these can be left as default (Hit enter at the prompt). snmp details will look like this if left default…

Compile and install snmpd

make
make install

The make process is quite CPU intensive, may want to do this during a quiet period for the server.

Some libraries are not where they need to be, namely:
libnetsnmpagent.so.40
libnetsnmpmibs.so.40
libnetsnmp.so.40

Find where they were installed

find / -name libnetsnmpagent.so.40

Pick the result that is not in your current working folder (the install setup) and link it to /usr/lib

ln -s /usr/local/lib64/libnetsnmpagent.so.40 /usr/lib/

Repeat this process with the other two files

ln -s /usr/local/lib64/libnetsnmpmibs.so.40 /usr/lib/
ln -s /usr/local/lib64/libnetsnmp.so.40 /usr/lib/

Now can test to see if snmpd can run

snmpd -v

Configuration

Create a configuration file, the “snmpconf” command can be used but I found it easier to create from scratch

nano /usr/local/share/snmp/snmpd.conf

For my needs, only this line is needed

rocommunity {community} {ip address}

Save and exit the editor.

Testing

If you’d like to test your configuration, run snmpd

snmpd

Test on the snmp target machine, for quick results snmpwalk can be run

snmpwalk -c {community} -v 2c {ip address}

When run, snmpd automatically switches to a background process. To stop the process ID needs to be found and killed

ps aux | grep snmpd

Here the process ID (PID) is 5716, use this with the kill command

kill -9 {PID}

Add Service

Create the new service file

nano /etc/init.d/snmpd

Paste the following

#!/sbin/openrc-run
 
depend() {
    after modules
}
 
start() {
    ebegin "Starting snmpd"
    start-stop-daemon --background --start --exec /usr/local/sbin/snmpd --pidfile /var/run/snmpd.pid \
    -- -p /var/run/snmpd.pid -c /usr/local/share/snmp/snmpd.conf
    eend $?
}
 
stop() {
    ebegin "Stopping snmpd"
    start-stop-daemon --stop --exec /usr/local/sbin/snmpd \
    --pidfile /var/run/snmpd.pid
    eend $?
}
restart() {
    ebegin "Restarting snmpd"
    start-stop-daemon --stop --exec /usr/local/sbin/snmpd
    start-stop-daemon --background --start --exec /usr/local/sbin/snmpd --pidfile /var/run/snmpd.pid \
    -- -p /var/run/snmpd.pid -c /usr/local/share/snmp/snmpd.conf
    eend $?
}

Save and exit the text editor, then make the file executable

chmod +x /etc/init.d/snmpd

Start the service, and check its running

/etc/init.d/snmpd start

/etc/init.d/snmpd status

Finally, to make snmpd start with the system, run

rc-update add snmpd default

References

Help Creating init.d file

https://big-elephants.com/2013-01/writing-your-own-init-scripts/
https://tecadmin.net/startup-shutdown-script-on-gentoo/

Experienced issue where the process-id in the PID file was not matching the PID of process. With help of below found flag for snmpd to specify PID file, and match that to init.d

https://forums.gentoo.org/viewtopic-t-886230-start-0.html

No problem I thought, use the iDrac to log into the ESXi console and set a new IP as it is statically assigned to a different subnet to the office…

Trouble is, the iDrac is so out of date I can’t get to its web interface on any browser available to my Windows 10 machine.

This is how to get access to an outdated iDrac 6 web interface and remote console…

Upgrade iDRAC firmware

First is to get some sort of access the the web interface to power the server on, as mentioned all modern browsers don’t support the SSL version the iDrac offers.

For this I had no choice but to go back in time, finding and spinning up a VM instance of Windows 7 with IE 8. From here I was able to log into and bring the firmware from 1.95 to the most recent 2.80. (Source)

At this firmware level, modern Chrome is able to log in and so the server can be started remotely.

Remote Console

Round two was getting the remote console to display, this too has fallen too far out of date for even the latest 2.80 firmware.

The console is Java based, so first need to allow the IP address of the iDrac through:

In Windows control panel, click the Java icon and move to the Security tab:

Ensure “Enable Java .. Web Start applications is ticked, set the Security level to High and then click Edit Site List…

Enter the IP of the iDrac in the format https://nnn.nnn.nnn.nnn/ , click OK and OK again to save.

Chrome doesn’t do Java applications, and so when launching the remote console it just downloads a .jnlp file. This can be forced to open with javaws.exe located in the bin folder of Java in the Program Files (x86) folder of Windows.

A better way is to use this batch script created by xbb which automates the process:

To use, download the batch script (.bat file) and place in a folder.

Then login to the iDrac interface, then in the same window visit https://nnn.nnn.nnn.nnn:443/software/avctKVM.jar where nnn is the IP of the iDrac, this will download the avctKVM.jar file, place this in the same folder as the .bat file.

Next create a subfolder named “jre” next to the .bat file, and COPY the bin and lib folder from C:\Program Files (x86)\Java\jre1.8.0_171\ to the jre folder, so the .bat file references files within ./jre/bin/

You’ll now have a structure like this:

Run the .bat file and enter the details to connect.

I however had one more hurdle…

Connection Failed

When connecting I had the same error message coming up:

Thanks to this post from virtual.mvp , turns out there’s more out of date oddities to resolve.

In the ./jre/lib/security folder, use Notepad++ to open java.security file.

In the file, search for the string “jdk.tls.disabledAlgorithms=SSLv3” and comment it out (add a # before the string) as below:

Save, and try the .bat script again.

Virtual Console is now working…

Many of our systems deployed from OVA’s use Gentoo as it’s base OS for inexplicable reasons, here’s how to increase the drive capacity of a VM instance running Gentoo and make the extra space usable…

Prework

SSH into the machine and identify the volume to benefit from the increase by entering

df -h

In this scenario the /home mount point requires the extra space, this is served by the /dev/sda4 Filesystem.

Now discover which disk this is on and where it sits in the layout of the disk (partition):

fdisk -l

The virtual disk is detailed towards the end of the output, and is identified with sda, multiple disks will be lableled sdb, sdc etc.

Along with the partition table at the end:

/dev/sda4 is located towards the end of the disk as denoted by the start and end sectors, so will be able to increase the disk by adding extra storage onto what is effectively the end of the drive.

Increase Drive Capacity.

Shut down the VM.

At the hypervisor, edit the configuration of the VM and enter a new capacity.

IM

Start up the VM, restore any Replications (and wait for initial sync to complete)

Highly recommended to take a snapshot at this point.

Reconfigure Partitions

SSH back into the machine, fdisk -l can be used again to sanity check that the drive now has increased capacity:

Now to assign this extra capacity to the volume, edit the disk partitions by entering:

fdisk /dev/sda

To refresh yourself on the current partition layout, enter p command to view, it also helps to have this on screen for the next steps…

As mentioned, here the goal is to increase /dev/sda4. To achieve this we need to remove this partition then re-add it to overlay the current structure but including the extra space. Sounds scary, but these changes aren’t in place until we commit the changes later. Nevertheless a good time to check the recent snapshot has completed.

Delete the current partition using command “d”, then entering partition number:

Again, can you the “p“ command to check its gone.

Now create the new partition with the “n” command.

Prompted for the first sector, enter the same value recorded on the original partition

The last sector, enter the default value shown, this should be greater than the original partition as there’s more storage available.

Output will advise that partition is made, then ask if you want to remove the signature. Enter NO to keep it, as we want to keep the original partition intact.

Finally, enter “w” command to write and commit the changes.

Reboot the VM.

Assign new space.

On restart, using fdisk -l will confirm the partition is bigger:

However, df -h will not have increased:

To use the extra space, we need to tell Gentoo that its available to use by entering:

resize2fs /dev/sda4

Output should look like this:

Now df -h will reflect the extra space and is ready to use: