First,

I put some effort into writing down exactly how to run FerretDB, an open-source document database. While large parts of it are still useful to show how to setup a self-recovering database with CloudNativePG, FerretDB has been slowly disappearing since after legal issues and the Linux Foundation adopting a different open source document database project. I cannot claim to know the how or why, but of course I regret this, as they were a community-focused small company taking on a huge corporation and fighting for an open standard. That open standard is there now: unfortunately, FerretDB isn’t anymore.

The second time it happened,

is just this week, and is a happier story. I wrote a month ago that the Rockchip RK3588-based SBCs are the ideal low-power machines for network attached storage (NAS) and distributed storage systems, with their 2.5 Gb/s network interface:

Because ten Gigabit tends to be expensive and power-hungry

So guess what happened: 10 Gb/s is now available on an incredibly power-efficient single-board computer: the Odroid H5

Again, there’s still value in the article: the Odroid H5 does not come with 32GB of DDR5, it has to be purchased separately, so it is still a lot more expensive. And the tips on how to safely connect a bunch of SATA SSDs to a small computer that only has M.2 connectors, as well as how to handle volumes in Talos, are just as applicable to the Odroid H5 as to the Rockhip.

Still, I can’t help but feel like there is some negative predictive power to my blog. Every time I say “do X because Y”, Y ends up becoming untrue within weeks.

So now,

to better serve the community, let me state some more tips with reasons to do them, see if it happens again:

• because RAM and SSD prices will stay very high for a long time, sit on your hardware
• because the Steam Frame is not coming out for a while, get a used VR headset
• …
• …

I was going to list some more things that I hope won’t come true, related to European (digital) sovereignty, but it gets really dark, so I’ve left some space for you to fill that out yourself.

Disclaimer: This blog was written without the help of LLMs (although that should be obvious) or any kind of financial incentive. I bought all the hardware myself, don’t have stocks in any of these companies and don’t use affiliate links. The result is entirely my own, unprofessional rambling. Though no one can claim to be devoid of bias, this article is as honest as I dare to be with myself. Which may not be a whole lot after spending hundreds on a weird homelab, but it’s the best you’re going to get.

You’ve seen Kubernetes run on big EPYC servers in data centers, and on F-16 fighter jets. You’ve seen it run on Raspberry Pis and even on phones.

The big professional setups guzzle electricity, water (and jet fuel), but the smallest setups are limited, mostly due to storage and network bandwidth.

What if we could run a distributed, reasonably powerful Kubernetes cluster, including distributed storage, using efficient single-board computers (SBCs) without the limitations that they usually have?

With the help of some friends, I built a setup that is exactly that: the smallest enterprise homelab. I call the nodes kubes.

Let’s start with the why. Or just skip to a section you are interested in.

Why small?

You might want your homelab to:

• take little physical space
• need little electrical power
• make little noise

So you may think you can just buy a Raspberry Pi with Raspberry OS and run Docker Compose on it. Think again.¹

Why enterprise?

Kubernetes is cool. GitOps is cool. Not needing to run a number of scripts in a very specific order, or remember a bunch of sysctls you need to set, and just being able to copy from kubesearch to deploy an app without any commands except just adding code to a Git repo is really cool. But you can run Kubernetes anywhere, so why do I need specific hardware? Even Talos, which removes the traditional Linux administration layer and gives an OS that only runs Kubernetes, runs on any x86 hardware and a bunch of SBCs (single board computers). What is special about my hardware setup and why did I pick such a weird setup?

Enterprise SSDs

You probably want to use Enterprise SSDs in your homelab eventually. The property of the drive that matters is Power Loss Protection (PLP). Especially for databases like etcd (Kubernetes control plane), Postgresql, and distributed storage like Ceph. Why? The short version is: those databases are very slow on consumer SSDs and they destroy them as well, especially if they write to them often. Etcd does in a normal Kubernetes setup. My friend @uhthomas is the big Enterprise SSD evangelist in Home Operations, he’ll write a blog post on it soon with more details. Ask him when the blog post goes live on Discord.

And no, enterprise SSDs are actually not that expensive if you buy them used.

The problem with Raspberries and laptops

So why don’t we just add enterprise SSDs to a laptop or Raspbery Pi and use that as a node?

I started homelabbing with 3 laptops and that’s exactly what I did. The problem is form factor and connectivity:

• Most laptops have either mSATA or SATA via M2 connections
• Enterprise SSDs typically (exclusively?) come in 2.5" SATA, M2 NVMe or U.2 form factors

The solution seemed simple: a USB 3 port offers enough bandwidth and power for a SATA 2.5" SSD, so I’ll just connect an Enterprise drive to every laptop via USB 3 and run Ceph on that. This seems to work at first but there were hiccups. Apparently, the USB controller resets sometimess and this would cause a node to hang. After a while, it became more frequent, and I decided to switch to better and smaller hardware. Not sure if Longhorn would have fared better here.

The problem with the Raspberry Pi is similar: besides USB, the Pi 5 has a proper PCIe connection, which is used by most Pi-NAS solutions. Unfortunately, it only has a single PCIe 2.0 lane available, which caps bandwidth at 500MB/s. OK for a single drive, but expensive: at the same price, you can get a machine with much more storage bandwidth (and RAM).

Mini-PCs like the NUC are the obvious solution. But most people don’t know that there is a step in between the Pi and the mini-pc, using a more power-efficient SBC with plenty of cores, RAM, storage and ..

Networking

The other big thing, is network bandwidth. You will use networked storage: using NAS or a distributed storage solution like Rook-Ceph or Longhorn is a requirement for a multi-node cluster. Getting more than the default 1Gb of networking bandwidth will make your storage faster. Because ten Gigabit tends to be expensive and power-hungry, let’s go for 2.5 Gigabit: it’s the same as most gaming motherboards come with and is still 2.5 times faster than the normal 1 Gigabit you get on a Pi.

EDIT (2026-05-23): 10 Gb/s is now available on an incredibly power-efficient single-board computer: the Odroid H5

The requirements

So if you are looking to build a small multi-node Kubernetes homelab, you’d want:

• 8 cores per node
• 24-32 GB of RAM per node
• 2.5 Gb/s network bandwidth
• Much more than 500 MB/s of storage bandwidth

You could get a NUC, or a Mac Mini, especially if you want to run local AI, but then you can’t use your Enterprise SSDs.

So if you have a bunch of SATA (and maybe NVMe) Enterprise disks, and you want a small energy efficient homelab, this is it.

Parts list

This is the full parts list for a single kube, excluding the SSDs themselves, which you should buy used.

About the same as for a used Mac Mini, provided you already have some Enterprise drives you want to use.

I expect you have:

• a small microSD card as a boot disk (this won’t be written to except on talos config changes or upgrades)
• Enterprise SSDs and SATA data cables to put into this
• Wago fixture (5B / 5B+ only)
• Dupont cables (5B / 5B+ only)
• screws and inserts to finish the 3D-printed case

Build: hardware

I went for the Rock 5B+ with 24GB of RAM and equipped each node with a single Micron 7450 Pro NVMe SSD, and one or more of the various SATA Enterprise SSDs I had hoarded strategically gathered. You could also use an Orange Pi 5 Plus or Max, as they are based on the same RK3588 chip and have the same PCIe 3.0 x4 storage bandwidth available. The normal Orange Pi 5 is limited to PCIe 2.0 x1 which limits bandwidth to 500 MB/s, just like the Raspberry Pi 5.

After unboxing the SBC, I first added the CPU heatsink to it. With some force, the heatsink snapped on, and then it was time to add both units to the M.2 slot on the back.

This is where the fun begins.

The thick power resistor

From what I can gather, this seems specific to the Rock 5B Plus model. There is a big fat power resistor, which looks like it could get quite hot, sitting exactly where one of the two M.2 SSDs should be once plugged in. This means you have to bend one of the M.2 units over that resistor to fit it in. I would not do this with a Micron 7450 SSD (it’s not quite that cheap either), so the SATA adapter had to get bent.

An M.2 SSD is held in place by a screw at the end that keeps it pushed down. Fun detail: the M.2 screw is not included! The screw should go into a little pillar on the motherboard. It turns out that the cantilever force of the adapter PCB trying to straighten itself, combined with the force needed to pull a SATA data cable out of it, was more than the little pillar could handle. It was just barely clamped into the PCB, not soldered or attached in any way.

This is where the M.2 mounting kit comes in. It has screws, standoffs, nuts and bolts and a little screwdriver. This is enough to reconstruct the M.2 standoff in a more robust way. The standoff is a little taller than the one that the Rock 5 came with, which helps to ease the tension on the M.2 stick.

Micron 7450 NVMe SSD cooling

The Micron 7450 SSD is one of the few models of NVMe Enterprise PLP SSDs that comes in a 2280 form factor, and physically fits on boards like these. Most others come in either 22110 or U.2 form factor. But there is one issue with these drives. It’s probably related to the Power-Loss Protection circuitry, but they get very hot. If you don’t apply any cooling, they will throttle.

So I got myself some M2 coolers from Arctic that looked small and easy to fit. All M.2 SSD coolers however, use something that goes around the bottom of the SSD to clamp into place. This needs 1 or 2 mm of space on each side of the SSD to be able to wrap around. Unfortunately, on the Rock 5B+ specifically, there is no space between the 2 M.2 units when both slots are filled. Less than a mm in any case. This is of course not an issue if you are using only one M.2 slot, on the Rock 5B or Orange Pi 5 Max or Plus, and it even seems like there might be enough space between both M.2 slots on the bigger Rock 5T. Haven’t tried that one yet though.

So I asked my friend @bo0tzz, who was making a modular case design for these kubes, to add some hooks for tie-wraps to keep the M.2 cooler from falling off.

I discovered later that without any airflow over them, these drives can still get up to 70 degrees C, so that is why I bought some 90mm PWM fans and a single USB-powered PWM fan controller to control the fans on all nodes. If you buy Noctua, you get a nice vibration-damping piece of rubber with them. If you buy Arctic, it’s a lot cheaper, only slightly less silent but you do have to cut the vibration-damping rubber out of an old bike tyre yourself.

Now we have the NVMe in place, and cooled and we have the required SATA data connections to connect SATA drives. How do we power them?

Connecting externally-powered SATA drives to an SBC

The SBC does not have SATA power connector, or Molex cable sticking out, and it can’t deliver enough power on its GPIO pins either to power a few SATA drives.

We could use an external 12V barrel plug power supply, and connect it to a buck converter (you can buy these pre-made with up to 4 SATA power connectors: search for “DC 5525 SATA adapter”), so the drives get both 12V and 5V, as they would get from an ATX power supply.

However, we need to be careful not to fry the drives: if the negative side of the power supply is not connected to the SBC’s ground, they might end up at a different voltage. This voltage difference would cause a current to flow through the SSDs, from the power- pin to the data ground. That is not healthy, so we need to ensure that the external power supply shares a common ground with the SBC.

The easiest way to do that is to cut open the power supply’s negative wires (not while it is plugged in of course), strip the ends, and connect them in a 5-slot wire connector / fixture. Double check if you really cut the negative and not the positive side!

Then connect that to the SBC ground on the GPIO pins via Dupont wires. Maximum is 5 (check Radxa docs on powering via GPIO or the pinout to see which).

The pins marked in black on the outside of the Rock 5B+’s GPIO connector are the ground ones, but do check the official spec sheet before you connect the power.

The case: Making it beautiful

After making it work, I asked @bo0tzz to make it beautiful by designing and 3D-printing a modular case for each kube.

The design can be downloaded at https://www.printables.com/model/1659104-modular-sbc-case

There are 3 modules:

• On the left, the Rock 5B(+) module can be screwed in. On the right, a module that holds SATA 2.5" SSDs is placed.
• In the middle, a blank slot covers the space that is occupied by SATA data cables and/or the NVMe SSD.
• At the bottom, there is enough space for the SATA data cables and the buck converter.

Air can come in from the sides and back if it is pulled through by putting a 90 mm fan on top.

Here’s a view of the internals

Both M2 slots occupied for maximum storage

You can see why a blank panel in the middle was needed.

You can see the 2 SATA SSDs that I put in there now (there is room for 4)

The SSD module was adorned with a picture of @uhthomas’ dog holding a 2.5" SATA SSD. It is thanks to Thomas that I was convinced to use power-loss protected enterprise SSDs and he also helped me find them. And the dog is how I know him, since that is his profile picture. The blank slot where all the heat goes, has a line tracing of my old cat, who loved to lay on his back on warm days.

The SVGs are here, you can apply them to a 3D model in the slicer before printing:

• My cat: https://fhoekstra.eu/animal-linetraces/cat-1.svg
• Thomas’ dog: https://fhoekstra.eu/animal-linetraces/dog-1.svg

Build: software

If you came here for the code, not the pictures, strap in!

This is a kube, it’s going to run Kubernetes and nothing else, so we will choose the most declarative and secure way to run Kubernetes: Talos Linux.

It may seem like a bit of a hassle to set up if you’re not familiar with it, but once you have this set up (and you can start with cluster-template, then copy the rest from my repo), you will be able to manage your entire cluster, including node maintenance, by simply updating the YAML files in your Git repo. Talos completely removes the traditional Linux layer, is immutable, very small and very secure: it is designed to run only Kubernetes, nothing else, so there is no bloat and a very small attack surface. This is simpler, faster, and much more reliable than Ansible scripts. It makes it such that working with the OS feels just like working with Kubernetes manifests.

If you use the Orange Pi 5 Plus or the Rock 5B or Rock 5T, you can follow the regular process for installing Talos and just walk through the Talos Factory at https://factory.talos.dev.

If you are using a different board, like the 5B+, that does have a Talos overlay in https://github.com/siderolabs/sbc-rockchip but is not yet selectable in the Image Factory UI, you need to use the API directly. That is what I will be explaining here.

EDIT (2026-05-31): The rock 5B+ is now available in the Talos Image Factory UI, so you can skip the “Getting Talos Image Factory IDs” section and just get your raw disk image from https://factory.talos.dev and flash it

Initial boot off of the SD card

While it is theoretically possible to boot an RK3588 machine off of an SSD, it’s really not worth the hassle. We just don’t want write-heavy directories to be on the SD card, but for the immutable root filesystem, and the boot partition, it’s not an issue. Write-heavy directories on a Talos Kubernetes machine are: /var/log, /var/lib/etcd, /var/lib/rook, /var/lib/contained, /var/mnt/local-hostpath, etc. They all start with /var. And Talos has this concept of the EPHEMERAL system volume, which is a magic name that means: put /var on this partition. So we can just put that on an SSD via the config, and boot normally off of the SD card.

To flash the SD card, I mostly followed this doc initially, but later learned there is an easier way.

As part of your Talos config, you’ll need to put in a talosImageURL. You can fetch this from the Talos Image Factory using the Web GUI with most hardware, but if you have a Rock 5B+, it’s not too hard either, you just need to use the REST API. Follow along:

Image Factory REST API: Getting Talos Image Factory IDs:

Write a schematic-rock-5b-plus.yaml like this:

customization:
  systemExtensions:
    officialExtensions:
      - siderolabs/realtek-firmware
overlay:
  name: rock5b-plus
  image: ghcr.io/siderolabs/sbc-rockchip:v0.2.0

Then just POST it to the Talos Image Factory schematics endpoint, and you’ll get an ID back.

curl -X POST --data-binary @schematic-rock-5b-plus.yaml https://factory.talos.dev/schematics

Put this in an installer URL like this: factory.talos.dev/installer/id-goes-here and that is your talosImageURL for your config. This logic is also encoded in the task talos:fetch-schematic-id

Image Factory REST API: Bootstrapping Talos on SBC with SD card

For bootstrap, you’ll need a raw disk image to flash to an SD card, which you can get from this URL: https://factory.talos.dev/image/id-goes-here/v1.12.6/metal-arm64.raw.xz (obviously, replace the Talos version if you want a newer version)

Then uncompress it: xz -d metal-arm64.raw.xz

The above logic is also encoded in the task talos:fetch-raw-disk-image-for-sbc. It uses your talconfig to determine the Talos version and image ID.

Check which dev the SD card is on (check via KDE partition manager what the /dev/sdX path is) Then write to SD card with dd: replace sdX with the SD card’s path sudo dd if=./metal-arm64.raw of=/dev/sdX conv=fsync oflag=direct status=progress bs=4M

Insert it, connect the power and you should be able to connect to the Talos API server after a minute or 2 on the local IP address of your node:

talosctl get machinestatus --insecure -n <IP>

You can then follow the regular Talos bootstrap instructions, either from the Talos docs, or cluster-template, if you’re using it.

If this is your first time setting up a Gitops Talos cluster, I highly recommend using cluster-template.

Day 2: Upgrades

Talos version and Kubernetes version upgrades can be done according to the standard instructions in the cluster-template repo.

Upgrading the version of the siderolabs/sbc-rockchip overlay requires a different procedure. For each node:

• cordon and drain the node
• then remove the power
• Use tweezers to pull out the microSD card
• Get the new factory ID as per the instructions above, or use the task talos:fetch-schematic-id
• Update the talconfig.yaml with the new schematic ID.
• Fetch the new raw disk image using the instructions above, or use the task talos:fetch-raw-disk-image-for-sbc
• flash a new raw disk image as per instructions above
• Put the microSD back in and power up the node
• Run an adapted version of the talos:apply-node task, with –insecure and a different precondition. I named it talos:apply-node-from-maintenance

Creating the Talos config for different disk setups

So how do we configure all those disks in Talos? They have a great documentation site, with a chat bot that is actually helpful, on how to do it with raw Talos manifests. Since I started with cluster-template, I got Talhelper with that. Its main benefit is that it integrates SOPS-encrypted secrets into Talos configuration for a full GitOps setup.

So I’ll show you how to do it with Talhelper. For this demo, I’ve set up each node with a different storage/disk configuration, then tagged it with talos-different-storage-setups since I’ll probably change that later. Here’s the code:

• Talos setups: https://github.com/fhoekstra/home-ops/blob/talos-different-storage-setups/kubernetes/bootstrap/talos/talconfig.yaml
• Rook-Ceph setup: https://github.com/fhoekstra/home-ops/blob/talos-different-storage-setups/kubernetes/apps/rook-ceph/rook-ceph/cluster/helmrelease.yaml#L92

Single NVMe only

DISCLAIMER: In the 3 months I ran with these different node setups, I had issues with Rook-Ceph on a node twice, needing a wipe and re-bootstrap. Both times it was this node, where everything is on a single shared disk. If you can do multiple disks, please give rook-ceph its own disk for the OSD.

We can actually run Talos with Rook-Ceph on a single disk, provided it has PLP and is fast enough. To do this, I create 2 volumes:

• one EPHEMERAL volume: documented here, this is a magic name in Talos which means it gets mounted on /var and contains all the write-heavy directories underneath it: logs, container images, local storage
• one raw volume: documented here, for Ceph as it needs its own block device without a file system

In Talhelper’s talconfig, I created a 250 GiB EPHEMERAL volume on the Micron 7450 like this:

nodes:
  - hostname: "talos-rock-1"
    ipAddress: "192.168.1.21"
    installDisk: /dev/mmcblk1 # This is the microSD card, as there is no onboard MMC on my Rock 5B+
    volumes:
      - name: EPHEMERAL
        provisioning:
          diskSelector:
            match: "disk.model.startsWith('Micron_7450_')"
          minSize: 250GiB
          maxSize: 250GiB
          grow: false

Then I include the raw volume manifest by pointing at it in the node’s patches section:

nodes:
  - hostname: "talos-rock-1"
    here: go the rest of the keys, as shown above
    patches:
      # We also create a raw volume for Ceph on the same NVMe disk
      - "@./patches/talos-rock-1/ceph-raw-volume.yaml"
      # And bind-mount a path for local host storage to the kubelet
      - "@./patches/talos-rock-1/csi-local-hostpath.yaml"

The raw volume manifest creates a volume without a filesystem, which is what Ceph wants:

---
apiVersion: v1alpha1
kind: RawVolumeConfig
name: osd-data # Do not use "ceph" in this name, or rook will refuse to use it for an OSD!!
provisioning:
  diskSelector:
    match: "disk.model.startsWith('Micron_7450_')"
  minSize: 400GiB
  maxSize: 400GiB

Then we point the Ceph cluster at the raw volume by using the partition label, which for a raw volume in Talos is r-volumename, so r-osd-data for us:

cephClusterSpec:
  storage:
    useAllNodes: false
    useAllDevices: false
    nodes:
      - name: "talos-rock-1"
        devices:
          -  name: "/dev/disk/by-partlabel/r-osd-data"

You may be wondering: how did I know to select the Micron 7450 with the match: "disk.model.startsWith('Micron_7450_')" selector?

Remember how I said Talos works just like you’re used to in Kubernetes? You can see all the disks on a node with talosctl -n <NODE_IP> get disks (add --insecure if your node is still in maintenance mode and hasn’t had a Talos config applied yet). Then to inspect the properties of an individual disk, you just do: talosctl -n <NODE_IP> get disk <DISK_NAME> -o yaml --insecure. You can then use its spec to write a selector, exactly as you’re used to when dealing with Kubernetes manifests and kubectl.

Using NVMe and SATA disks together

The other nodes in my example are simpler, even though they use more disks. For both of them, I just used a full SATA disk for the Talos EPHEMERAL system volume, and gave the full NVMe to Ceph:

cephClusterSpec:
  storage:
        useAllNodes: false
        useAllDevices: false
        nodes:
          - name: "talos-rock-1"
            devices:
              - name: "/dev/disk/by-partlabel/r-osd-data"
          - name: "talos-rock-2"
            devices:
              - name: "/dev/nvme0n1" # Micron 7450 Pro 1TB
          - name: "talos-rock-3"
            devices:
              - name: "/dev/nvme0n1" # Micron 7450 Pro 1TB

For node 2, I used a second SATA disk for the local storage provisioner. This is done by democratic-csi in my cluster, though openebs is another popular solution.

This provisioner can create PVCs backed by local storage, from a volume that is mounted to the kubelet. On the other nodes, I just bind-mount a directory from the EPHEMERAL system volume.

For node 2, I instead provision a UserVolume (documented here, almost completing our tour of Talos volume provisioning docs). What you need to know is this:

A UserVolume gets a filesystem, XFS by default, and gets a partition label u-name. In addition, it gets mounted at /var/mnt/name.

nodes:
  - hostname: "talos-rock-2"
    ipAddress: "192.168.1.22"
    installDisk: /dev/mmcblk1 # This is the microSD card, as there is no onboard MMC on my Rock 5B+
    volumes:
      - name: EPHEMERAL
        provisioning:
          diskSelector:
            match: "disk.model.startsWith('SSDSC2KG480G8R')" # Intel DC S4610, 480 GB
          minSize: 250GiB
          grow: true
    userVolumes:
      - name: csi-local-hostpath
        provisioning:
          diskSelector:
            match: "disk.model.startsWith('INTEL SSDSC2BX40')"
          minSize: "50GiB"
          grow: true
    patches:
      # Here we bind the local hostpath from the separate device so we have a different patch than in talos-rock-1
      - "@./patches/talos-rock-2/csi-local-hostpath.yaml"

And then the kubelet patch for the local host path uses the /var/mnt/name path for the UserVolume:

machine:
  kubelet:
    extraMounts:
      - destination: /var/mnt/csi-local-hostpath  # In other nodes, this was /var/lib/csi-local-hostpath
        type: bind
        source: /var/lib/csi-local-hostpath
        options: ["bind", "rshared", "rw"]

If you lost track of what goes where, the full code is here:

• Talos setup: https://github.com/fhoekstra/home-ops/blob/talos-different-storage-setups/kubernetes/bootstrap/talos/talconfig.yaml
• Rook-Ceph setup: https://github.com/fhoekstra/home-ops/blob/talos-different-storage-setups/kubernetes/apps/rook-ceph/rook-ceph/cluster/helmrelease.yaml#L92

Addendum: numbers

Power use

I measured the power use of my entire homelab by plugging it into a power monitoring Zigbee outlet.

Just my micro-NAS in the form of a Raspberry Pi 4B with a single consumer 1TB SATA SSD connected via USB, and an old 1GbE router: 8W after power-on, settle on 6W after 2 minutes.

Add in the three Radxa Rock 5B+ nodes, each with a Micro 7450 Pro NVMe SSD, and two of them with a SATA-M.2 adapter and two 400GB Enterprise SATA drives, for a total of 4 SATA and 3 NVMe drives, all Enterprise, PLP-cache equipped. The Rocks themselves are powered by a single UGREEN 100W USB-PD GaN charger each, while the SATA drives are powered by one 12V barrel plug PSU per node. (2 PSUs for 4 drives)

Plugging in the 2 12V barrel plug PSUs (I didn’t get the Radxa ones, but adjustable voltage supplies which are probably less efficient) alone pushes the total power use to just above 11W. So they use about 3 W a piece even when the SSDs are not in use.

Finally, plugging in the Rocks and thus starting them bumps power use first to 65W, then settles around 50W. 16W per Rock node with around 30% CPU usage. This can probably be done more efficiently, with less power supplies, or more appropriately sized ones. The easiest way to improve this would be to get a Rock 5T, which is a slightly larger board with the same chipset, but powered via DC barrel plug, so you could easily use the same power supply for the SBC and the SSDs.

Disk performance

I measured the raw disk performance of the Rock 5B+ under Radxa OS (an older Debian with KDE that is provided by the manufacturer) with a Micron 7450 Pro NVMe drive at 1638 MB/s read, and 1441 MB/s write with the other M.2 slot empty. This is more than 3x the storage bandwidth you get with a Raspberry Pi 5 or a Rock 5C and the NAS solutions for them, no matter how many disks you connect.

Are you still using Ingress NGINX? Do you want a complete example of how to migrate to the future-proof Gateway API? Wondering how to migrate forward auth?

I did the migration on my homelab only this month (I know, I’m late), but I made sure to have a clean string of commits with no other changes or dependency bumps between them. I wanted to make it easy to view the full extent of the migration from nginx-ingress to Gateway API with everything it entails: forward auth, Anubis bot protection, etc.

I’ve tagged them for your convenience. This should be especially useful if your networking setup is from cluster-template 1 to 2 years ago, or if you’re using forward auth with Authelia or Authentik, or if you have some public endpoints protected by the Anubis bot blocker.

• git clone https://github.com/fhoekstra/home-ops
• Or compare the tags right here in your browser by choosing Compare on one of the following pages:
  • before-gateway-migration
  • first-app-working-with-gateway
  • after-gateway-migration

Do you also find yourself wanting to use your custom keyboard while travelling with your Linux laptop? Without triggering keypresses through the built-in keyboard underneath of course!

I’ll first show you how to disable the built-in laptop keyboard with a bash script. And then automatically disable the laptop keyboard when, and only when, your custom keyboard is plugged in, and enable it again when your custom keyboard is unplugged.

How to disable the built-in keyboard

You can disable the keyboard dynamically by writing 1 to the inhibited property in sys/devices/.../input/input3. Enabling is a matter of writing 0 to the same path.

First we need to find the precise SysFs path of the internal keyboard:

cat /proc/bus/input/devices | less

If necessary, type / then keyboard to search the output.

[...]
N: Name="AT Translated Set 2 keyboard"
P: Phys=isa0060/serio0/input0
S: Sysfs=/devices/platform/i8042/serio0/input/input3
[...]

Fill in the SYSFS_PATH in the following script, which I save to /home/freek/Scripts/laptop-kb.sh (and make it executable with chmod +x <filepath>):

#!/usr/bin/env bash

SYSFS_PATH=/devices/platform/i8042/serio0/input/input3
DEV_PATH="/sys/${SYSFS_PATH}/inhibited"

case "$1" in
    disable)
        echo 1 | sudo tee "$DEV_PATH"
        ;;
    enable)
        echo 0 | sudo tee "$DEV_PATH"
        ;;
    toggle)
        cur=$(cat "$DEV_PATH")
        new="1"
        if [ "$cur" = "1" ]; then $new="0"; fi
        echo $new | sudo tee "$DEV_PATH"
        ;;
    *)
        echo "Usage: $0 {enable|disable|toggle}" >&2
        exit 1
        ;;
esac

Trigger on plug / unplug external keyboard

To only disable it when a specific external keyboard is plugged in, write a udev rule that will trigger different actions on plug and unplug of the external keyboard. udev is part of systemd and included in most modern Linux distributions, including Arch Linux, by default. Even distributions that ship without systemd, such as Void Linux, often have a compatible implementation that also accepts udev rules, such as eudev.

First find the external keyboard’s PRODUCT id by watching the output of this command:

sudo udevadm monitor --kernel --property --subsystem-match=usb

While plugging and unplugging the external keyboard.

For my keyboard, it is: PRODUCT=a8f8/1836/200

Then write a udev rules file with the following rules. Replace your PRODUCT id and script path and save this to /etc/udev/rules.d/99-disable-internal-keyboard.rules

SUBSYSTEM=="usb", ACTION=="add", ENV{DEVTYPE}=="usb_device", ENV{PRODUCT}=="a8f8/1836/200", RUN+="/home/freek/Scripts/laptop-kb.sh disable"
SUBSYSTEM=="usb", ACTION=="remove", ENV{DEVTYPE}=="usb_device", ENV{PRODUCT}=="a8f8/1836/200", RUN+="/home/freek/Scripts/laptop-kb.sh enable"

Then reload the udev rules by rebooting or run: udevadm control --reload-rules && udevadm trigger

Have fun working on your Linux laptop anywhere with your custom keyboard!

Thanks to Koen Vervloesem for figuring this stuff out for webcams.

Yay! I got my ‘CKS - Certified Kubernetes Security Specialist’ certification.

It was quite a ride, as I had almost forgotten about it. You see, I had 2 goals this year: the GitOps homelab and CKS. I have done and learned a lot in the homelab, so much so that I had neglected the CKS a bit. I was under the impression that I had some time left when I was reminded by The Linux Foundation that I only had 4 weeks left to plan the exam (and potential retake). After those 4 weeks, the money my employer generously paid for me to get certified would be down the drain. So I applied myself and studied as hard as I could.

I knew CKA had taken me some practice, not because it is so hard, but because you need to be fast to get through all the assignments. CKA, CKS and CKAD are hands on exams where you get a terminal, vim kubectl (and the k alias luckily), and a Firefox browser that can only browse the official documentation pages.

In order to get through all the assignments in the allotted time, you have to know your way around the documentation very well.

Practice, practice, practice

For CKA, I prepared using the Linux Foundation course. I did some of the labs multiple times to ensure I could do an “open” assignment unguided using just the documentation. Being able to do the labs where every step is instructed in detail is not enough.

For CKS, I heard from friends that there is a serious Linux sysadmin component to it, that happens on the Linux (Debian on the exam) nodes. As a former developer turned Kubernetes Engineer, I felt that the Linux Foundation course labs did not prepare me sufficiently for that. I am not affiliated with Kodekloud, but their course is great, and the labs being prepared for you in the cloud is a huge productivity booster. So I highly recommend taking a Kodekloud subscription for a month to do their labs.

And remember to stay focused in your practice: are you just following instructions, or can you find these instructions and example configs to edit in the official documentation? If what you are doing in a lab is not obvious to you, try doing it again using just the official documentation.

Embrace Vim

I spent the last 2 years (since the CKA exam) using (neo)vim as my main IDE, but if you don’t yet, spend some time getting used to vim. It really helps with speed on the exam to be able to indent the next 5 lines at once (^, Ctrl+V, 5j, space space, Esc), move or replace an entire line(dd, then P to paste above current line, p to paste below, cc to replace, Shift+V to select multiple lines, then d for delete or c for change), or change text within "" (simply ci", change inside ").

You will never get this fast with nano. Embrace vim, leave the mouse alone as much as possible. Your shoulders, your wrists and the children in Uganda will thank you.

Practice exams

The Linux Foundation gives you 2 Killer.sh practice exams, and kodekloud also provides 3 practice exams. I found the kodekloud exams had a lot of overlap (I did 2 and felt like most of it was the exact same assignment), and their environment is more helpful/easier: zsh with better autocomplete, whereas the real exam has a bare bash shell where autocomplete only happens with some delay after you press tab, not with greyed out text appearing as you’re typing (like zsh).

So keep that in mind and use your killer.sh practice exams when you feel you’ve mastered the labs!

aliases and shortcuts

Don’t. For CKA 2 years ago, I had trained myself to write out some 3-letter aliases of the oh-my-zsh kubectl plugin, as well as kubectl config set-context --current --namespace and --dry-run -oyaml as a var. Now, there was a different node you ssh’ed into for every assignment, so those aliases wouldn’t stick around anyway, and the assignments and files were set up such that you usually didn’t need to k get deploy name -o yaml. I still did often, purely out of habit, before noticing the yaml was already on the filesystem.

yq / jq chops

Like vim, these skills are generally good to have, not just for the exam. I spent the night before the exam practicing them a bit, because I noticed I was scrolling through YAML and searching through json in less (with /, n, N) for things that could have been simple queries. In my quest to save any minute I could off of the time I need, I decided to practice my jq/yq query writing.

In the end, none of the assignments in the actual exam required these skills, but I am still grateful to finally understand when I need to do .someArraywithbrackets[] | select .key == "value" and when I need to do .someArraywithoutbrackets | group_by(.key).

Enjoy the process!

If you follow these tips, you should have no trouble learning all the required tools and skills for CKS, or any other Kubernetes exam, such as CKA or CKAD. Then the last tip is to enjoy it!

I find these hands-on exams very satisfying to do: they give me a feeling of mastery and confidence that no multiple choice or pen and paper exam has ever given me. Engineers and tinkerers who love doing and fixing things should really try these hands-on exams. Coming out of the exam, I felt like a Kubernetes God: “I can fix any cluster, write any policy manifest and find any misbehaving container now!” It sounds stupid, but if you know, you know. It’s a really good feeling and one of the best ways to demonstrate your technical ability.

At my current job, we maintain a package that is used by developers in their own codebases to use a database service on the platform. Like any package or artifact that is released to then be used by application developers, this requires a clear release with a semantic version that communicates what users can expect from it: Does it contain only fixes (patch), or are there new features in the release (minor)? Most importantly, do users (potentially) need to change how they use the package when upgrading (major)?

Ideally, the version communicates the category of the release: patch (fixes only), minor (new features), or major (breaking changes), and the details are found in a changelog. Without a changelog, your users have a hard time figuring out what to expect from a new release.

Can’t I just write a changelog myself?

If your users are normal end users, you can have the product and/or UX people write a changelog that is aimed at the end-user experience. But when your users are developers who integrate your package with their own application, completeness is key! You cannot know all the ways and context in which developers use your software, so it is important to be careful and complete and mark changes that are potentially breaking as such. If you forget to mention a change, that could waste a lot of developer time on debugging.

The reason why you could forget relevant changes, is because the changelog is fundamentally a duplication. In a changelog, you describe the changes to the software since the previous release. But as a platform engineer, you should already have this information somewhere else: in the git log!

If you follow conventional commits, your commits already contain this information:

feat(api)!: Make retentionPeriod a required property on orderDetails
fix: OOM on submission of large orderList
chore: Bump Python to 3.14.0

Thus, I needed to generate a changelog (and a next version number) based on the conventional commits in the git log between the last release and now. Seems like it should be a common problem.

What if you’re not on Github?

To my surprise, I saw that the most commonly used tool for this problem does not use plain Git: release-please is built around the Github API. We use a less commonly used forge, so this did not work for us. The alternative release-it has bindings for the GitLab and GitLab APIs, but also has a way to generate a changelog from pure Git. Since we do not directly depend on any NodeJS projects in the team I work, I looked a bit further for a self-contained binary that really only does changelog and version generation from git.

Git Cliff

Git Cliff is precisely what I was looking for:

• only does changelog and semver generation from git
• self-contained binary
• sensible defaults, works out of the box
• very customizable
• great documentation
• written in Rust and available from many repositories

To generate a configuration file with the defaults and a lot of helpful comments, run git-cliff --init.

The cliff.toml default configuration has a section that looks like:

commit_parsers = [
    { message = "^feat", group = "<!-- 0 -->🚀 Features" },
    { message = "^fix", group = "<!-- 1 -->🐛 Bug Fixes" },
    { message = "^doc", group = "<!-- 3 -->📚 Documentation" },
    ...
]

You can add or remove parsers easily: the message field is the regex pattern that is matched to the commit message, and the number in the group determines the order of the sections in the generated changelog.

Some useful commands:

# Generate the changelog file by running:
git-cliff -o CHANGELOG.md

# Get the last released (tagged) version:
git-cliff --context | jq '.[0].version'

# Get the version for the next release (based on what has been committed since)
git-cliff --bumped-version

# Generate the changelog for that next release
git-cliff --bump -o CHANGELOG.md

Whichever Git Forge you use, these commands should allow you to create a changelog and release process that works for your platform! Happy crafting!

I recently replaced the batteries on two of my Zigbee Tradfri remotes and had to pair them again to ZHA in Home Assistant because they had been offline for a while.

I noticed the same thing happening for both devices: they got stuck on the Interview complete - Configuring step. They are different models, so this made me suspect a software problem. I had a look and found this topic on the Home Assistant community forum.

In it, 2 potential root causes with their respective workarounds are proposed:

1. Cause: The Zigbee device is falling asleep.

• Workaround: Keep pressing one of its regular buttons to keep it awake during pairing

2. Cause: The ZHA controller is missing a step or edgecase or the device is not sending the appropriate signal when it is paired.

• Workaround: When you get to the Interview complete step, go to Integrations -> ZHA -> "your coordinator", press the 3 dots next to it and select: Reload. Afterwards, you’ll see your device added to ZHA.

Number 1 didn’t work for my Tradfri remotes, but number 2 did. I hope writing this helps someone else who runs into this.

FerretDB, the company, seems to have disappeared. This blog is still useful as a reference, as most of it applies to CloudnativePG and the Postgres DocumentDB extension in general.

Why?

FerretDB is an open source compatibility layer that serves a MongoDB-compatible database server, and stores the data in PostgreSQL. I think this is amazing, but I will save the “Why FerretDB?” talk for another blog.

This is a technical guide that will walk you through how to get the following:

1. FerretDB deployment backed by highly available and secured cloudnative-pg postgres cluster.
2. Barman-cloud plugin backups for point-in-time restores
3. Automatic recovery on redeploy

In those 3 steps. What this means for me is that I can get a completely open source MongoDB-compatible database that behaves like the rest of my Kubernetes homelab: if I delete the whole cluster and redeploy it from manifests using GitOps, all my data automatically comes back from backups, without needing any manual intervention. This allows me to play with my homelab without worrying about data restore: everything will come back up on a fresh bootstrap, no manual intervention required.

Step 0: Prerequisites

• An S3-compatible object storage. Either get one from a cloud provider near you, or set up something like Garage or Versity Gateway on your NAS. If you get a cloud-provided object storage, make sure to get one that is not too expensive per write transaction, as postgres will be writing there constantly.
• A Kubernetes cluster (I’m using Kubernetes 1.34 on Talos 1.11 at the time of writing)
• cloudnative-pg operator installed on it: one of these commands should get you going (I’m running 1.27 at the moment)
• barman-cloud plugin for backups of cloudnative-pg provisioned databases. Check the instructions here. I tested this with 0.6.0.
• Both cloudnative-pg and barman-cloud require cert-manager to be installed on the cluster
• In addition, to run postgresql databases, you need some kind of provisioner to provision volumes, preferably on local host storage. Check out democratic-csi or openebs-local

With that out of the way, let’s get started.

Step 1: FerretDB backed by cloudnative-pg

I will be sharing YAML throughout this tutorial. You can copy it to your machine, and then run kubectl apply -f filename.yaml to apply it to your kubernetes cluster, or put it in your GitOps repo, whatever you want.

The following YAML gives you a highly available cloudnative-pg cluster of 3 nodes, with 2 FerretDB servers in front to serve your application with MongoDB-compatible requests. Feel free to change these numbers, or add a resources block, if you prefer.

The comments explain some of the how and why.

---
# yaml-language-server: $schema=https://github.com/datreeio/CRDs-catalog/raw/refs/heads/main/postgresql.cnpg.io/cluster_v1.json
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: ferretdb-pg-cluster
  namespace: ferretdb
spec:
  # Keep this in sync with the correct image for FerretDB itself
  #  read FerretDB release notes and upgrade them together
  imageName: ghcr.io/ferretdb/postgres-documentdb:17-0.106.0-ferretdb-2.5.0

  instances: 3

  # postgres-documentdb needs these IDs
  postgresUID: 999
  postgresGID: 999

  storage:
    size: 10Gi
    # This should be the name of the storage class 
    # as configured in your provisioner
    storageClass: local-hostpath

  postgresql:
    shared_preload_libraries:
      - pg_cron
      - pg_documentdb_core
      - pg_documentdb

    parameters:
      # pg_cron needs to know which database FerretDB uses
      cron.database_name: ferretDB
      # These parameters are necessary to run ferretdb without superuser access
      # Copied from https://github.com/FerretDB/documentdb/blob/ferretdb/packaging/10-preload.sh
      documentdb.enableCompact: "true"
      documentdb.enableLetAndCollationForQueryMatch: "true"
      documentdb.enableNowSystemVariable: "true"
      documentdb.enableSortbyIdPushDownToPrimaryKey: "true"
      documentdb.enableSchemaValidation: "true"
      documentdb.enableBypassDocumentValidation: "true"
      documentdb.enableUserCrud: "true"
      documentdb.maxUserLimit: "100"
    pg_hba:
      # pg_cron always runs as `postgres`
      - host ferretDB postgres localhost trust
      # This is needed to prevent fe_sendauth error
      - host ferretDB ferret localhost trust
  bootstrap:
    initdb:
      database: ferretDB
      owner: ferret
      postInitApplicationSQL:
        - create extension if not exists pg_cron;
        - create extension if not exists documentdb cascade;
        - grant documentdb_admin_role to ferret;
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ferretdb
  namespace: ferretdb
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ferretdb
  template:
    metadata:
      labels:
        app: ferretdb
    spec:
      containers:
        - name: ferretdb
          # Keep this in sync with the correct image for the postgresql cluster,
          #  always read FerretDB release notes and upgrade them together
          image: ghcr.io/ferretdb/ferretdb:2.5.0
          ports:
            - containerPort: 27017
          env:
            - name: FERRETDB_POSTGRESQL_URL
              valueFrom:
                secretKeyRef:
                  # This secret gets automatically generated by cloudnative-pg
                  name: ferretdb-pg-cluster-app 
                  key: uri
---
apiVersion: v1
kind: Service
metadata:
  name: ferretdb-service
  namespace: ferretdb
spec:
  selector:
    app: ferretdb
  ports:
    - protocol: TCP
      port: 27017
      targetPort: 27017
  type: NodePort

This is a bit longer than the basic example that was on FerretDB’s blog earlier this year, but that is because we need to run postgres without connecting as the postgres superuser.

After applying, watch the pods come up one by one:

kubectl -n ferretdb get pods -w

After startup is done, check it all works by running:

kubectl get cluster.postgresql.cnpg.io -n ferretdb

And check that it says: Cluster in healthy state under STATUS like in the below example output:

NAMESPACE   NAME                  AGE     INSTANCES   READY   STATUS                     PRIMARY
ferretdb    ferretdb-pg-cluster   33h     3           3       Cluster in healthy state   ferretdb-pg-cluster-1

This is great for security, and is necessary to work with the way cloudnative-pg handles backups in our case.

Try out your FerretDB service now: just connect to the node it is running on (since we used NodePort), or if you’re using a cloud Kubernetes outside your home network, change that NodePort to ClusterIP and use kubectl port-forward. You could use any Mongo client you wish, this is what it looks like with mongosh: mongosh 'mongodb://ferret:the-password-from-the-cloudnative-pg-secret@ip-of-the-node-it-is-running-on-or-localhost-if-port-forwarding:27017'

Then run some Mongo commands.

Step 2: Enable backups

We will use the barman-cloud plugin to take backups. You may read about a legacy way of taking backups with cloudnative-pg, but there are 2 reasons why we use the plugin instead: the legacy way will be removed from the next cloudnative-pg release; and the legacy way requires barman-cloud (the backup tool) to be a part of the postgres image used. Our FerretDB provided image does not have barman-cloud. Luckily the new way with the plugin creates a separate container with barman-cloud and adds it to each database pod, so that it works regardless of which postgres container is running.

Postgres backups come in multiple flavors. Today we’ll look at the 2 that barman provides: base backups are like “save points” that can be restored from scratch in full. Write-Ahead-Logs (WALs) backup is a log of all the database transactions that can be replayed. These can be used to replay all the way to the latest state, or up to some point in time between the earliest base backup and the latest WAL.

So the combination of WALs and base backups give us both point in time restore, and restore up to the latest state, with no or very little data loss in case of an accidental (or intentional) cluster-wide outage. How these 2 should be used, is entirely handled by the tooling. All we need to do, is ensure backups are automatically made, and to specify a point in time to restore to.

So let’s get those backups rolling. First we connect the postgres cluster to the S3 bucket using a BarmanObjectStore object:

---
apiVersion: v1
kind: Secret
metadata:
  name: s3-creds
  namespace: ferretdb
stringData:
  ACCESS_KEY_ID: your S3 access key ID goes here
  SECRET_KEY: your S3 secret key goes here
---
# yaml-language-server: $schema=https://github.com/datreeio/CRDs-catalog/raw/refs/heads/main/barmancloud.cnpg.io/objectstore_v1.json
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: ferretdb-backupstore
  namespace: ferretdb
spec:
  retentionPolicy: 14d
  configuration:
    destinationPath: s3://bucketname/optionalsubfolder/ # Change this
    # Change the endpoint URL to whatever your cloud provider told you to use
    # NOTE: should be https if using cloud bucket
    endpointURL: http://versity.storage.svc.cluster.local:7070 
    s3Credentials:
      accessKeyId:
        name: s3-creds
        key: ACCESS_KEY_ID
      secretAccessKey:
        name: s3-creds
        key: SECRET_KEY

Fill in the following S3 connection data in the above YAML before applying:

• credentials:
  • access key ID
  • secret key
• bucket name (and optional subfolder) NOTE: ensure bucket exists first
• endpoint URL

And never commit unencrypted secrets to your Git repo (the ObjectStore is safe but the Secret should be encrypted with SOPS or filled using external-secrets if you are doing GitOps). Alternatively, .gitignore it.

Then add this plugins section to the spec of your postgresql cluster:

---
# yaml-language-server: $schema=https://github.com/datreeio/CRDs-catalog/raw/refs/heads/main/postgresql.cnpg.io/cluster_v1.json
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: ferretdb-pg-cluster
  namespace: ferretdb
spec:
  [...]

  plugins:
    - name: barman-cloud.cloudnative-pg.io
      isWALArchiver: true
      parameters:
        barmanObjectName: ferretdb-backupstore
        serverName: pg

Check that the barman-cloud pods are all started after a few seconds. You should see 2/2 for every pod that is part of the cluster under the READY column of the output of the following command:

kubectl -n ferretdb get pods

Like the ferretdb-pg-cluster pods here:

NAME                              READY   STATUS    RESTARTS        AGE
ferretdb-79dccb7d5d-8kbnq         1/1     Running   0               34h
ferretdb-79dccb7d5d-wnlc9         1/1     Running   0               34h
ferretdb-pg-cluster-1             2/2     Running   0               33h
ferretdb-pg-cluster-2             2/2     Running   0               33h
ferretdb-pg-cluster-3             2/2     Running   0               33h

You can also check your S3 storage to see if the WALs are actually ending up in there. For example, you could install rclone and connect it to your cloud storage by running rclone config. Then afterwards, you could run rclone ls name-of-your-remote: to see all the files in there. You should see files in a wals/ subdirectory of the path you configured in your ObjectStore.

So now we have the barman-cloud containers streaming the WALs to your cloud object storage. But in order to be able to restore, we also need a “save point” or base backup. I like to set this up in such a way that it is taken periodically, for example every night, and cloudnative-pg makes this very easy. No cronjob needed, just this:

---
# yaml-language-server: $schema=https://raw.githubusercontent.com/datreeio/CRDs-catalog/refs/heads/main/postgresql.cnpg.io/scheduledbackup_v1.json
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: ferretdb
spec:
  backupOwnerReference: self
  cluster:
    name: ferretdb-pg-cluster # Should be the name of your postgres cluster
  schedule: "0 0 1 * * *" # This is a slightly unusual cron format
  immediate: true # Means: take a snapshot right now as well
  method: plugin
  pluginConfiguration:
    name: barman-cloud.cloudnative-pg.io

Check that it is applied:

kubectl -n ferretdb get scheduledbackup

Check that it spawns a regular Backup object:

kubectl -n ferretdb get scheduledbackup

And check the phase. You can watch it to not have to keep refreshing by adding -w. Eventually the PHASE should be completed.

NAMESPACE   NAME                        AGE     CLUSTER               METHOD              PHASE       ERROR
ferretdb    ferretdb-20250924091153     33h     ferretdb-pg-cluster   plugin              completed   

And you should see a recovery window in the status of your ObjectStore: kubectl -n ferretdb describe objectstore ferretdb-backupstore On the bottom of the output you should see something like:

Status:
  Server Recovery Window:
    Pg:
      First Recoverability Point:   2025-09-24T09:12:01Z
      Last Successful Backup Time:  2025-09-25T01:00:07Z

And that’s it. Now you could leave it running for a while, write some data, delete some data and try out point in time restore using cloudnative-pg documentation

But for this tutorial, we will go to the initially promised final step.

Step 3. Automatic recovery

It is important that your backups are properly wired up and you have a recovery window in your ObjectStore before you start this step.

The usecase is the following: imagine your whole kubernetes cluster is wiped. Maybe there’s water damage to your house, a lightning hit, or (more common in the homelab) you have so seriously messed up your cluster that it is easier to start over and re-apply all the YAML files you have been keeping track of, than to actually fix the problem. Or if you are using a proper GitOps solution like FluxCD, applying all the YAMLs is the same as just deploying Flux into a fresh cluster.

Let’s set up our postgres cluster configuration to automatically restore from the ObjectStore. Add the lines marked with + to the postgres cluster document (without the +), comment out the initdb section and re-apply it:

---
# yaml-language-server: $schema=https://github.com/datreeio/CRDs-catalog/raw/refs/heads/main/postgresql.cnpg.io/cluster_v1.json
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: ferretdb-pg-cluster
  namespace: ferretdb
+ annotations:
+   # required for seamless bootstrap: https://github.com/cloudnative-pg/cloudnative-pg/issues/5778#issuecomment-2783417464
+   cnpg.io/skipEmptyWalArchiveCheck: "enabled"
spec:
  # Keep this in sync with the correct image for FerretDB itself
  #  read FerretDB release notes and upgrade them together
  imageName: ghcr.io/ferretdb/postgres-documentdb:17-0.106.0-ferretdb-2.5.0

  instances: 3

  # postgres-documentdb needs these IDs
  postgresUID: 999
  postgresGID: 999

  storage:
    size: 10Gi
    # This should be the name of the storage class as configured in your provisioner
    storageClass: local-hostpath

  postgresql:
    shared_preload_libraries:
      - pg_cron
      - pg_documentdb_core
      - pg_documentdb

    parameters:
      # pg_cron needs to know which database FerretDB uses
      cron.database_name: ferretDB
      # The parameters below are necessary to run ferretdb without superuser access
      # Copied from https://github.com/FerretDB/documentdb/blob/ferretdb/packaging/10-preload.sh
      documentdb.enableCompact: "true"
      documentdb.enableLetAndCollationForQueryMatch: "true"
      documentdb.enableNowSystemVariable: "true"
      documentdb.enableSortbyIdPushDownToPrimaryKey: "true"
      documentdb.enableSchemaValidation: "true"
      documentdb.enableBypassDocumentValidation: "true"
      documentdb.enableUserCrud: "true"
      documentdb.maxUserLimit: "100"
    pg_hba:
      # pg_cron always runs as `postgres`
      - host ferretDB postgres localhost trust
      # This is needed to prevent fe_sendauth error
      - host ferretDB ferret localhost trust
  bootstrap:
+   recovery:
+     source: &source pg
#    initdb:
#      database: ferretDB
#      owner: ferret
#      postInitApplicationSQL:
#        - create extension if not exists pg_cron;
#        - create extension if not exists documentdb cascade;
#        - grant documentdb_admin_role to ferret;
  plugins:
    - name: barman-cloud.cloudnative-pg.io
      isWALArchiver: true
      parameters:
        barmanObjectName: ferretdb-backupstore
        serverName: pg

+ externalClusters:
+   - name: pg
+     plugin:
+       name: barman-cloud.cloudnative-pg.io
+       parameters:
+         barmanObjectName: ferretdb-backupstore
+         serverName: pg

Don’t forget the annotation! Without it, barman-cloud will refuse to continue writing backups to the same ObjectStore and serverName after recovery. It is generally safe to use this annotation¹.

Now check this has applied successfully:

kubectl get cluster.postgresql.cnpg.io -n ferretdb

kubectl describe cluster.postgresql.cnpg.io -n ferretdb

(you can also shorten these to just use cluster if you have no other resources named cluster except the cnpg cluster.)

If it has, go ahead and delete your entire postgres cluster:

kubectl delete -n ferretdb cluster.postgresql.cnpg.io ferretdb-pg-cluster --force

Check that all the postgres cluster pods and PVCs are gone (you might have to kubectl delete --force some of them):

kubectl -n ferretdb get pods

kubectl -n ferretdb get pvc

If they are all gone, here comes the test: re-apply the postgresql YAML and it should start a full-recovery:

kubectl apply -f postgres-cluster.yaml

Check the pods:

kubectl -n ferretdb get pods -w Check the logs:

kubectl -n ferretdb logs name-of-the-pod

And soon, your cluster should be back and healthy. Connect to it as in Step 1 and check that your data is restored. Make a happy dance, then go to bed and sleep soundly, knowing your data is safe, no manual intervention is required for restore, and you are using a completely open source document database solution.

If things change and you want to check how I am running FerretDB, go to github.com/fhoekstra/home-ops, press t and type ferretdb.

If you found an error, I would love to accept your PR to this blog at https://codeberg.org/fhoekstra/blog

If you need help setting this up at home, come to the Home Operations Discord.

Attribution

I did not come up with any of this myself, I just combined the ideas from some amazing people around me, tested it and wrote it down. Special thanks go to @eaglesemanation for figuring out how to run FerretDB against cloudnative-pg without superuser access. That made this setup possible.

I should also thank @tholinka and @Phycoforce from the Home-Operations community for their help with barman-cloud backups. Their home-ops repos are a great resource.

You can browse many more repos using cloudnative-pg and barman-cloud by searching kubesearch.dev to get ideas of other ways to set this up.

Footnotes