The most common way you will likely be regularly accessing and controlling your services is via SSH. SSH allows you to connect to your servers or virtual machines via command line, by default using a username and password combination that will be defined during the setup of the host. To enhance security, we will migrate the SSH authentication method to a private certificate, and disable password access.

SSH Certificates

By generating your certificate chain, you will be provided with a public and private key pair. The private key will remain on the device you generate it from and intend to access your other hosts, whilst the public key will need to be transferred to the host devices for authentication. GitHub provides both instructions on how to generate this pair, and the ability to host your public key through your GitHub profile for later access.

How to create your certificate chain

1. In a terminal window, execute the following command

   ssh-keygen -t ed25519 -C "your_email@example.com"
   

2. Follow the prompts, all can be left as default as desired
3. Start the ssh-agent

   $ eval "$(ssh-agent -s)"
   > Agent pid 59566
   

4. Add your private key to the SSH agent

   ssh-add ~/.ssh/id_ed25519
   

We now have a public and private key chain, and our private key loaded on our local SSH agent.

Getting your public key onto host devices

Next we need to ensure our public key is loaded on our host devices, to allow for certificate based authentication. One of the simplest ways to achieve this is to load your public key onto your GitHub profile, which can then be used to download to host machines. Steps to add your public SSH key to your GitHub profile are available here.

Once loaded, your key(s) will be available at the following endpoint:

https://github.com/username.keys

On your host machine, you can now download the public key to your authorized keys directory, updating this with any new keys you may have added to your profile.

curl https://github.com/username.keys >> ~/.ssh/authorized_keys

Enabling certificate authentication

Finally, we now want to enable certificate authentication, and disable password authentication, meaning that only users with a valid certificate pair can SSH into the host.

1. Use a text editor to modify the SSH config

   nano /etc/ssh/sshd_config
   

2. Ensure ‘PasswordAuthentication’ is set to ‘no’

   PasswordAuthentication no
   

3. Enable public key authentication by removing the # comment from the following line

   PubkeyAuthentication yes
   

4. Disable SSH access via the root account

   PermitRootLogin no
   

5. Finally, restart the SSH service

   systemctl restart ssh
   

You will now be able to login to your host devices without the need to enter a password, making the process far simpler, and also more secure.

Monitoring your infrastructure

In order to maintain the security of our environment, we need to ensure we have the ability to monitor it for consistency and change. To achieve this, we will deploy a monitoring service that can be used as a central point of reporting and visibility into our environment. In this case, I will be deploying the Checkmk monitoring solution, but other popular choices for homelab environments include Zabbix and Uptime Kuma.

To setup a checkmk monitoring environment, we require two parts - the host server for checkmk raw, and the checkmk agent which will be deployed on devices we want to monitor.

Installing the checkmk raw server

Checkmk server can be configured on a range of linux environments, for my purposes I will be using a Debian 12 based LXC.

1. Download the latest version of the checmk raw server install file onto your host device

   wget https://download.checkmk.com/checkmk/2.2.0p23/check-mk-raw-2.2.0p23_0.bookworm_amd64.deb
   

2. Run the installation file

   apt install check-mk-raw-2.2.0p23_0.bookworm_amd64.deb
   

3. Confirm the installation has been successfully by verifying Open Monitoring Distribution is running

   omd version
   OMD - Open Monitoring Distribution Version 2.2.0p1.cre
   

A comprehensive guide for configuration options for checmk can be found here.

Installing the checkmk agent

Now that our checmk server is up and running, we need to install the agent application on devices we want to monitor. We can start with the host of the checkmk server itself.

1. Download the latest version of the checmk agent from your checkmk server to your local device.
2. Copy the agent from your local device to the intended host

   scp /local/path/check-mk-agent_2.3.0b1-1_all.deb checkmk-server@host-ip /remote/path
   

3. Install the checmk agent by running the following command on the host machine

   apt install /local/path/check-mk-agent_2.3.0b1-1_all.deb
   

Adding a host to your checkmk server

1. Jump back into your checkmk server, and head to Setup > Hosts
2. Select ‘Add a Host’, and enter the required information, including a hostname and IP address for the host you wish to monitor
3. Click ‘Save & run service discovery’ - the checkmk server will connect to the agent on the host device and provide a range services that are available for monitoring.
4. Click ‘Accept All’, and then click on the yellow alert banner at the top of the screen which indicates a pending change.
5. Click ‘Activate on selected sites’, which will commence the monitoring activities.

You can repeat this process for as many hosts as you wish to monitor within your environment.

With checmk now configured, you will have access to the default dashboard highlighting any service problems or events that have taken place. Regularly checking in here and addressing any problems, or monitoring events for changes within your environment will ensure you are across what is happening in your homelab.