Are you still using Ingress NGINX? Do you want a complete example of how to migrate to the future-proof Gateway API? Wondering how to migrate forward auth?

I did the migration on my homelab only this month (I know, I’m late), but I made sure to have a clean string of commits with no other changes or dependency bumps between them. I wanted to make it easy to view the full extent of the migration from nginx-ingress to Gateway API with everything it entails: forward auth, Anubis bot protection, etc.

I’ve tagged them for your convenience. This should be especially useful if your networking setup is from cluster-template 1 to 2 years ago, or if you’re using forward auth with Authelia or Authentik, or if you have some public endpoints protected by the Anubis bot blocker.

• git clone https://github.com/fhoekstra/home-ops
• Or compare the tags right here in your browser by choosing Compare on one of the following pages:
  • before-gateway-migration
  • first-app-working-with-gateway
  • after-gateway-migration

Do you also find yourself wanting to use your custom keyboard while travelling with your Linux laptop? Without triggering keypresses through the built-in keyboard underneath of course!

I’ll first show you how to disable the built-in laptop keyboard with a bash script. And then automatically disable the laptop keyboard when, and only when, your custom keyboard is plugged in, and enable it again when your custom keyboard is unplugged.

How to disable the built-in keyboard

You can disable the keyboard dynamically by writing 1 to the inhibited property in sys/devices/.../input/input3. Enabling is a matter of writing 0 to the same path.

First we need to find the precise SysFs path of the internal keyboard:

cat /proc/bus/input/devices | less

If necessary, type / then keyboard to search the output.

[...]
N: Name="AT Translated Set 2 keyboard"
P: Phys=isa0060/serio0/input0
S: Sysfs=/devices/platform/i8042/serio0/input/input3
[...]

Fill in the SYSFS_PATH in the following script, which I save to /home/freek/Scripts/laptop-kb.sh (and make it executable with chmod +x <filepath>):

#!/usr/bin/env bash

SYSFS_PATH=/devices/platform/i8042/serio0/input/input3
DEV_PATH="/sys/${SYSFS_PATH}/inhibited"

case "$1" in
    disable)
        echo 1 | sudo tee "$DEV_PATH"
        ;;
    enable)
        echo 0 | sudo tee "$DEV_PATH"
        ;;
    toggle)
        cur=$(cat "$DEV_PATH")
        new="1"
        if [ "$cur" = "1" ]; then $new="0"; fi
        echo $new | sudo tee "$DEV_PATH"
        ;;
    *)
        echo "Usage: $0 {enable|disable|toggle}" >&2
        exit 1
        ;;
esac

Trigger on plug / unplug external keyboard

To only disable it when a specific external keyboard is plugged in, write a udev rule that will trigger different actions on plug and unplug of the external keyboard. udev is part of systemd and included in most modern Linux distributions, including Arch Linux, by default. Even distributions that ship without systemd, such as Void Linux, often have a compatible implementation that also accepts udev rules, such as eudev.

First find the external keyboard’s PRODUCT id by watching the output of this command:

sudo udevadm monitor --kernel --property --subsystem-match=usb

While plugging and unplugging the external keyboard.

For my keyboard, it is: PRODUCT=a8f8/1836/200

Then write a udev rules file with the following rules. Replace your PRODUCT id and script path and save this to /etc/udev/rules.d/99-disable-internal-keyboard.rules

SUBSYSTEM=="usb", ACTION=="add", ENV{DEVTYPE}=="usb_device", ENV{PRODUCT}=="a8f8/1836/200", RUN+="/home/freek/Scripts/laptop-kb.sh disable"
SUBSYSTEM=="usb", ACTION=="remove", ENV{DEVTYPE}=="usb_device", ENV{PRODUCT}=="a8f8/1836/200", RUN+="/home/freek/Scripts/laptop-kb.sh enable"

Then reload the udev rules by rebooting or run: udevadm control --reload-rules && udevadm trigger

Have fun working on your Linux laptop anywhere with your custom keyboard!

Thanks to Koen Vervloesem for figuring this stuff out for webcams.

Yay! I got my ‘CKS - Certified Kubernetes Security Specialist’ certification.

It was quite a ride, as I had almost forgotten about it. You see, I had 2 goals this year: the GitOps homelab and CKS. I have done and learned a lot in the homelab, so much so that I had neglected the CKS a bit. I was under the impression that I had some time left when I was reminded by The Linux Foundation that I only had 4 weeks left to plan the exam (and potential retake). After those 4 weeks, the money my employer generously paid for me to get certified would be down the drain. So I applied myself and studied as hard as I could.

I knew CKA had taken me some practice, not because it is so hard, but because you need to be fast to get through all the assignments. CKA, CKS and CKAD are hands on exams where you get a terminal, vim kubectl (and the k alias luckily), and a Firefox browser that can only browse the official documentation pages.

In order to get through all the assignments in the allotted time, you have to know your way around the documentation very well.

Practice, practice, practice

For CKA, I prepared using the Linux Foundation course. I did some of the labs multiple times to ensure I could do an “open” assignment unguided using just the documentation. Being able to do the labs where every step is instructed in detail is not enough.

For CKS, I heard from friends that there is a serious Linux sysadmin component to it, that happens on the Linux (Debian on the exam) nodes. As a former developer turned Kubernetes Engineer, I felt that the Linux Foundation course labs did not prepare me sufficiently for that. I am not affiliated with Kodekloud, but their course is great, and the labs being prepared for you in the cloud is a huge productivity booster. So I highly recommend taking a Kodekloud subscription for a month to do their labs.

And remember to stay focused in your practice: are you just following instructions, or can you find these instructions and example configs to edit in the official documentation? If what you are doing in a lab is not obvious to you, try doing it again using just the official documentation.

Embrace Vim

I spent the last 2 years (since the CKA exam) using (neo)vim as my main IDE, but if you don’t yet, spend some time getting used to vim. It really helps with speed on the exam to be able to indent the next 5 lines at once (^, Ctrl+V, 5j, space space, Esc), move or replace an entire line(dd, then P to paste above current line, p to paste below, cc to replace, Shift+V to select multiple lines, then d for delete or c for change), or change text within "" (simply ci", change inside ").

You will never get this fast with nano. Embrace vim, leave the mouse alone as much as possible. Your shoulders, your wrists and the children in Uganda will thank you.

Practice exams

The Linux Foundation gives you 2 Killer.sh practice exams, and kodekloud also provides 3 practice exams. I found the kodekloud exams had a lot of overlap (I did 2 and felt like most of it was the exact same assignment), and their environment is more helpful/easier: zsh with better autocomplete, whereas the real exam has a bare bash shell where autocomplete only happens with some delay after you press tab, not with greyed out text appearing as you’re typing (like zsh).

So keep that in mind and use your killer.sh practice exams when you feel you’ve mastered the labs!

aliases and shortcuts

Don’t. For CKA 2 years ago, I had trained myself to write out some 3-letter aliases of the oh-my-zsh kubectl plugin, as well as kubectl config set-context --current --namespace and --dry-run -oyaml as a var. Now, there was a different node you ssh’ed into for every assignment, so those aliases wouldn’t stick around anyway, and the assignments and files were set up such that you usually didn’t need to k get deploy name -o yaml. I still did often, purely out of habit, before noticing the yaml was already on the filesystem.

yq / jq chops

Like vim, these skills are generally good to have, not just for the exam. I spent the night before the exam practicing them a bit, because I noticed I was scrolling through YAML and searching through json in less (with /, n, N) for things that could have been simple queries. In my quest to save any minute I could off of the time I need, I decided to practice my jq/yq query writing.

In the end, none of the assignments in the actual exam required these skills, but I am still grateful to finally understand when I need to do .someArraywithbrackets[] | select .key == "value" and when I need to do .someArraywithoutbrackets | group_by(.key).

Enjoy the process!

If you follow these tips, you should have no trouble learning all the required tools and skills for CKS, or any other Kubernetes exam, such as CKA or CKAD. Then the last tip is to enjoy it!

I find these hands-on exams very satisfying to do: they give me a feeling of mastery and confidence that no multiple choice or pen and paper exam has ever given me. Engineers and tinkerers who love doing and fixing things should really try these hands-on exams. Coming out of the exam, I felt like a Kubernetes God: “I can fix any cluster, write any policy manifest and find any misbehaving container now!” It sounds stupid, but if you know, you know. It’s a really good feeling and one of the best ways to demonstrate your technical ability.

At my current job, we maintain a package that is used by developers in their own codebases to use a database service on the platform. Like any package or artifact that is released to then be used by application developers, this requires a clear release with a semantic version that communicates what users can expect from it: Does it contain only fixes (patch), or are there new features in the release (minor)? Most importantly, do users (potentially) need to change how they use the package when upgrading (major)?

Ideally, the version communicates the category of the release: patch (fixes only), minor (new features), or major (breaking changes), and the details are found in a changelog. Without a changelog, your users have a hard time figuring out what to expect from a new release.

Can’t I just write a changelog myself?

If your users are normal end users, you can have the product and/or UX people write a changelog that is aimed at the end-user experience. But when your users are developers who integrate your package with their own application, completeness is key! You cannot know all the ways and context in which developers use your software, so it is important to be careful and complete and mark changes that are potentially breaking as such. If you forget to mention a change, that could waste a lot of developer time on debugging.

The reason why you could forget relevant changes, is because the changelog is fundamentally a duplication. In a changelog, you describe the changes to the software since the previous release. But as a platform engineer, you should already have this information somewhere else: in the git log!

If you follow conventional commits, your commits already contain this information:

feat(api)!: Make retentionPeriod a required property on orderDetails
fix: OOM on submission of large orderList
chore: Bump Python to 3.14.0

Thus, I needed to generate a changelog (and a next version number) based on the conventional commits in the git log between the last release and now. Seems like it should be a common problem.

What if you’re not on Github?

To my surprise, I saw that the most commonly used tool for this problem does not use plain Git: release-please is built around the Github API. We use a less commonly used forge, so this did not work for us. The alternative release-it has bindings for the GitLab and GitLab APIs, but also has a way to generate a changelog from pure Git. Since we do not directly depend on any NodeJS projects in the team I work, I looked a bit further for a self-contained binary that really only does changelog and version generation from git.

Git Cliff

Git Cliff is precisely what I was looking for:

• only does changelog and semver generation from git
• self-contained binary
• sensible defaults, works out of the box
• very customizable
• great documentation
• written in Rust and available from many repositories

To generate a configuration file with the defaults and a lot of helpful comments, run git-cliff --init.

The cliff.toml default configuration has a section that looks like:

commit_parsers = [
    { message = "^feat", group = "<!-- 0 -->🚀 Features" },
    { message = "^fix", group = "<!-- 1 -->🐛 Bug Fixes" },
    { message = "^doc", group = "<!-- 3 -->📚 Documentation" },
    ...
]

You can add or remove parsers easily: the message field is the regex pattern that is matched to the commit message, and the number in the group determines the order of the sections in the generated changelog.

Some useful commands:

# Generate the changelog file by running:
git-cliff -o CHANGELOG.md

# Get the last released (tagged) version:
git-cliff --context | jq '.[0].version'

# Get the version for the next release (based on what has been committed since)
git-cliff --bumped-version

# Generate the changelog for that next release
git-cliff --bump -o CHANGELOG.md

Whichever Git Forge you use, these commands should allow you to create a changelog and release process that works for your platform! Happy crafting!

I recently replaced the batteries on two of my Zigbee Tradfri remotes and had to pair them again to ZHA in Home Assistant because they had been offline for a while.

I noticed the same thing happening for both devices: they got stuck on the Interview complete - Configuring step. They are different models, so this made me suspect a software problem. I had a look and found this topic on the Home Assistant community forum.

In it, 2 potential root causes with their respective workarounds are proposed:

1. Cause: The Zigbee device is falling asleep.

• Workaround: Keep pressing one of its regular buttons to keep it awake during pairing

2. Cause: The ZHA controller is missing a step or edgecase or the device is not sending the appropriate signal when it is paired.

• Workaround: When you get to the Interview complete step, go to Integrations -> ZHA -> "your coordinator", press the 3 dots next to it and select: Reload. Afterwards, you’ll see your device added to ZHA.

Number 1 didn’t work for my Tradfri remotes, but number 2 did. I hope writing this helps someone else who runs into this.

FerretDB is an open source compatibility layer that serves a MongoDB-compatible database server, and stores the data in PostgreSQL. I think this is amazing, but I will save the “Why FerretDB?” talk for another blog.

This is a technical guide that will walk you through how to get the following:

1. FerretDB deployment backed by highly available and secured cloudnative-pg postgres cluster.
2. Barman-cloud plugin backups for point-in-time restores
3. Automatic recovery on redeploy

In those 3 steps. What this means for me is that I can get a completely open source MongoDB-compatible database that behaves like the rest of my Kubernetes homelab: if I delete the whole cluster and redeploy it from manifests using GitOps, all my data automatically comes back from backups, without needing any manual intervention. This allows me to play with my homelab without worrying about data restore: everything will come back up on a fresh bootstrap, no manual intervention required.

Step 0: Prerequisites

• An S3-compatible object storage. Either get one from a cloud provider near you, or set up something like Garage or Versity Gateway on your NAS. If you get a cloud-provided object storage, make sure to get one that is not too expensive per write transaction, as postgres will be writing there constantly.
• A Kubernetes cluster (I’m using Kubernetes 1.34 on Talos 1.11 at the time of writing)
• cloudnative-pg operator installed on it: one of these commands should get you going (I’m running 1.27 at the moment)
• barman-cloud plugin for backups of cloudnative-pg provisioned databases. Check the instructions here. I tested this with 0.6.0.
• Both cloudnative-pg and barman-cloud require cert-manager to be installed on the cluster
• In addition, to run postgresql databases, you need some kind of provisioner to provision volumes, preferably on local host storage. Check out democratic-csi or openebs-local

With that out of the way, let’s get started.

Step 1: FerretDB backed by cloudnative-pg

I will be sharing YAML throughout this tutorial. You can copy it to your machine, and then run kubectl apply -f filename.yaml to apply it to your kubernetes cluster, or put it in your GitOps repo, whatever you want.

The following YAML gives you a highly available cloudnative-pg cluster of 3 nodes, with 2 FerretDB servers in front to serve your application with MongoDB-compatible requests. Feel free to change these numbers, or add a resources block, if you prefer.

The comments explain some of the how and why.

---
# yaml-language-server: $schema=https://github.com/datreeio/CRDs-catalog/raw/refs/heads/main/postgresql.cnpg.io/cluster_v1.json
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: ferretdb-pg-cluster
  namespace: ferretdb
spec:
  # Keep this in sync with the correct image for FerretDB itself
  #  read FerretDB release notes and upgrade them together
  imageName: ghcr.io/ferretdb/postgres-documentdb:17-0.106.0-ferretdb-2.5.0

  instances: 3

  # postgres-documentdb needs these IDs
  postgresUID: 999
  postgresGID: 999

  storage:
    size: 10Gi
    # This should be the name of the storage class 
    # as configured in your provisioner
    storageClass: local-hostpath

  postgresql:
    shared_preload_libraries:
      - pg_cron
      - pg_documentdb_core
      - pg_documentdb

    parameters:
      # pg_cron needs to know which database FerretDB uses
      cron.database_name: ferretDB
      # These parameters are necessary to run ferretdb without superuser access
      # Copied from https://github.com/FerretDB/documentdb/blob/ferretdb/packaging/10-preload.sh
      documentdb.enableCompact: "true"
      documentdb.enableLetAndCollationForQueryMatch: "true"
      documentdb.enableNowSystemVariable: "true"
      documentdb.enableSortbyIdPushDownToPrimaryKey: "true"
      documentdb.enableSchemaValidation: "true"
      documentdb.enableBypassDocumentValidation: "true"
      documentdb.enableUserCrud: "true"
      documentdb.maxUserLimit: "100"
    pg_hba:
      # pg_cron always runs as `postgres`
      - host ferretDB postgres localhost trust
      # This is needed to prevent fe_sendauth error
      - host ferretDB ferret localhost trust
  bootstrap:
    initdb:
      database: ferretDB
      owner: ferret
      postInitApplicationSQL:
        - create extension if not exists pg_cron;
        - create extension if not exists documentdb cascade;
        - grant documentdb_admin_role to ferret;
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ferretdb
  namespace: ferretdb
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ferretdb
  template:
    metadata:
      labels:
        app: ferretdb
    spec:
      containers:
        - name: ferretdb
          # Keep this in sync with the correct image for the postgresql cluster,
          #  always read FerretDB release notes and upgrade them together
          image: ghcr.io/ferretdb/ferretdb:2.5.0
          ports:
            - containerPort: 27017
          env:
            - name: FERRETDB_POSTGRESQL_URL
              valueFrom:
                secretKeyRef:
                  # This secret gets automatically generated by cloudnative-pg
                  name: ferretdb-pg-cluster-app 
                  key: uri
---
apiVersion: v1
kind: Service
metadata:
  name: ferretdb-service
  namespace: ferretdb
spec:
  selector:
    app: ferretdb
  ports:
    - protocol: TCP
      port: 27017
      targetPort: 27017
  type: NodePort

This is a bit longer than the basic example that was on FerretDB’s blog earlier this year, but that is because we need to run postgres without connecting as the postgres superuser.

After applying, watch the pods come up one by one:

kubectl -n ferretdb get pods -w

After startup is done, check it all works by running:

kubectl get cluster.postgresql.cnpg.io -n ferretdb

And check that it says: Cluster in healthy state under STATUS like in the below example output:

NAMESPACE   NAME                  AGE     INSTANCES   READY   STATUS                     PRIMARY
ferretdb    ferretdb-pg-cluster   33h     3           3       Cluster in healthy state   ferretdb-pg-cluster-1

This is great for security, and is necessary to work with the way cloudnative-pg handles backups in our case.

Try out your FerretDB service now: just connect to the node it is running on (since we used NodePort), or if you’re using a cloud Kubernetes outside your home network, change that NodePort to ClusterIP and use kubectl port-forward. You could use any Mongo client you wish, this is what it looks like with mongosh: mongosh 'mongodb://ferret:the-password-from-the-cloudnative-pg-secret@ip-of-the-node-it-is-running-on-or-localhost-if-port-forwarding:27017'

Then run some Mongo commands.

Step 2: Enable backups

We will use the barman-cloud plugin to take backups. You may read about a legacy way of taking backups with cloudnative-pg, but there are 2 reasons why we use the plugin instead: the legacy way will be removed from the next cloudnative-pg release; and the legacy way requires barman-cloud (the backup tool) to be a part of the postgres image used. Our FerretDB provided image does not have barman-cloud. Luckily the new way with the plugin creates a separate container with barman-cloud and adds it to each database pod, so that it works regardless of which postgres container is running.

Postgres backups come in multiple flavors. Today we’ll look at the 2 that barman provides: base backups are like “save points” that can be restored from scratch in full. Write-Ahead-Logs (WALs) backup is a log of all the database transactions that can be replayed. These can be used to replay all the way to the latest state, or up to some point in time between the earliest base backup and the latest WAL.

So the combination of WALs and base backups give us both point in time restore, and restore up to the latest state, with no or very little data loss in case of an accidental (or intentional) cluster-wide outage. How these 2 should be used, is entirely handled by the tooling. All we need to do, is ensure backups are automatically made, and to specify a point in time to restore to.

So let’s get those backups rolling. First we connect the postgres cluster to the S3 bucket using a BarmanObjectStore object:

---
apiVersion: v1
kind: Secret
metadata:
  name: s3-creds
  namespace: ferretdb
stringData:
  ACCESS_KEY_ID: your S3 access key ID goes here
  SECRET_KEY: your S3 secret key goes here
---
# yaml-language-server: $schema=https://github.com/datreeio/CRDs-catalog/raw/refs/heads/main/barmancloud.cnpg.io/objectstore_v1.json
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
  name: ferretdb-backupstore
  namespace: ferretdb
spec:
  retentionPolicy: 14d
  configuration:
    destinationPath: s3://bucketname/optionalsubfolder/ # Change this
    # Change the endpoint URL to whatever your cloud provider told you to use
    # NOTE: should be https if using cloud bucket
    endpointURL: http://versity.storage.svc.cluster.local:7070 
    s3Credentials:
      accessKeyId:
        name: s3-creds
        key: ACCESS_KEY_ID
      secretAccessKey:
        name: s3-creds
        key: SECRET_KEY

Fill in the following S3 connection data in the above YAML before applying:

• credentials:
  • access key ID
  • secret key
• bucket name (and optional subfolder) NOTE: ensure bucket exists first
• endpoint URL

And never commit unencrypted secrets to your Git repo (the ObjectStore is safe but the Secret should be encrypted with SOPS or filled using external-secrets if you are doing GitOps). Alternatively, .gitignore it.

Then add this plugins section to the spec of your postgresql cluster:

---
# yaml-language-server: $schema=https://github.com/datreeio/CRDs-catalog/raw/refs/heads/main/postgresql.cnpg.io/cluster_v1.json
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: ferretdb-pg-cluster
  namespace: ferretdb
spec:
  [...]

  plugins:
    - name: barman-cloud.cloudnative-pg.io
      isWALArchiver: true
      parameters:
        barmanObjectName: ferretdb-backupstore
        serverName: pg

Check that the barman-cloud pods are all started after a few seconds. You should see 2/2 for every pod that is part of the cluster under the READY column of the output of the following command:

kubectl -n ferretdb get pods

Like the ferretdb-pg-cluster pods here:

NAME                              READY   STATUS    RESTARTS        AGE
ferretdb-79dccb7d5d-8kbnq         1/1     Running   0               34h
ferretdb-79dccb7d5d-wnlc9         1/1     Running   0               34h
ferretdb-pg-cluster-1             2/2     Running   0               33h
ferretdb-pg-cluster-2             2/2     Running   0               33h
ferretdb-pg-cluster-3             2/2     Running   0               33h

You can also check your S3 storage to see if the WALs are actually ending up in there. For example, you could install rclone and connect it to your cloud storage by running rclone config. Then afterwards, you could run rclone ls name-of-your-remote: to see all the files in there. You should see files in a wals/ subdirectory of the path you configured in your ObjectStore.

So now we have the barman-cloud containers streaming the WALs to your cloud object storage. But in order to be able to restore, we also need a “save point” or base backup. I like to set this up in such a way that it is taken periodically, for example every night, and cloudnative-pg makes this very easy. No cronjob needed, just this:

---
# yaml-language-server: $schema=https://raw.githubusercontent.com/datreeio/CRDs-catalog/refs/heads/main/postgresql.cnpg.io/scheduledbackup_v1.json
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: ferretdb
spec:
  backupOwnerReference: self
  cluster:
    name: ferretdb-pg-cluster # Should be the name of your postgres cluster
  schedule: "0 0 1 * * *" # This is a slightly unusual cron format
  immediate: true # Means: take a snapshot right now as well
  method: plugin
  pluginConfiguration:
    name: barman-cloud.cloudnative-pg.io

Check that it is applied:

kubectl -n ferretdb get scheduledbackup

Check that it spawns a regular Backup object:

kubectl -n ferretdb get scheduledbackup

And check the phase. You can watch it to not have to keep refreshing by adding -w. Eventually the PHASE should be completed.

NAMESPACE   NAME                        AGE     CLUSTER               METHOD              PHASE       ERROR
ferretdb    ferretdb-20250924091153     33h     ferretdb-pg-cluster   plugin              completed   

And you should see a recovery window in the status of your ObjectStore: kubectl -n ferretdb describe objectstore ferretdb-backupstore On the bottom of the output you should see something like:

Status:
  Server Recovery Window:
    Pg:
      First Recoverability Point:   2025-09-24T09:12:01Z
      Last Successful Backup Time:  2025-09-25T01:00:07Z

And that’s it. Now you could leave it running for a while, write some data, delete some data and try out point in time restore using cloudnative-pg documentation

But for this tutorial, we will go to the initially promised final step.

Step 3. Automatic recovery

It is important that your backups are properly wired up and you have a recovery window in your ObjectStore before you start this step.

The usecase is the following: imagine your whole kubernetes cluster is wiped. Maybe there’s water damage to your house, a lightning hit, or (more common in the homelab) you have so seriously messed up your cluster that it is easier to start over and re-apply all the YAML files you have been keeping track of, than to actually fix the problem. Or if you are using a proper GitOps solution like FluxCD, applying all the YAMLs is the same as just deploying Flux into a fresh cluster.

Let’s set up our postgres cluster configuration to automatically restore from the ObjectStore. Add the lines marked with + to the postgres cluster document (without the +), comment out the initdb section and re-apply it:

---
# yaml-language-server: $schema=https://github.com/datreeio/CRDs-catalog/raw/refs/heads/main/postgresql.cnpg.io/cluster_v1.json
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: ferretdb-pg-cluster
  namespace: ferretdb
+ annotations:
+   # required for seamless bootstrap: https://github.com/cloudnative-pg/cloudnative-pg/issues/5778#issuecomment-2783417464
+   cnpg.io/skipEmptyWalArchiveCheck: "enabled"
spec:
  # Keep this in sync with the correct image for FerretDB itself
  #  read FerretDB release notes and upgrade them together
  imageName: ghcr.io/ferretdb/postgres-documentdb:17-0.106.0-ferretdb-2.5.0

  instances: 3

  # postgres-documentdb needs these IDs
  postgresUID: 999
  postgresGID: 999

  storage:
    size: 10Gi
    # This should be the name of the storage class as configured in your provisioner
    storageClass: local-hostpath

  postgresql:
    shared_preload_libraries:
      - pg_cron
      - pg_documentdb_core
      - pg_documentdb

    parameters:
      # pg_cron needs to know which database FerretDB uses
      cron.database_name: ferretDB
      # The parameters below are necessary to run ferretdb without superuser access
      # Copied from https://github.com/FerretDB/documentdb/blob/ferretdb/packaging/10-preload.sh
      documentdb.enableCompact: "true"
      documentdb.enableLetAndCollationForQueryMatch: "true"
      documentdb.enableNowSystemVariable: "true"
      documentdb.enableSortbyIdPushDownToPrimaryKey: "true"
      documentdb.enableSchemaValidation: "true"
      documentdb.enableBypassDocumentValidation: "true"
      documentdb.enableUserCrud: "true"
      documentdb.maxUserLimit: "100"
    pg_hba:
      # pg_cron always runs as `postgres`
      - host ferretDB postgres localhost trust
      # This is needed to prevent fe_sendauth error
      - host ferretDB ferret localhost trust
  bootstrap:
+   recovery:
+     source: &source pg
#    initdb:
#      database: ferretDB
#      owner: ferret
#      postInitApplicationSQL:
#        - create extension if not exists pg_cron;
#        - create extension if not exists documentdb cascade;
#        - grant documentdb_admin_role to ferret;
  plugins:
    - name: barman-cloud.cloudnative-pg.io
      isWALArchiver: true
      parameters:
        barmanObjectName: ferretdb-backupstore
        serverName: pg

+ externalClusters:
+   - name: pg
+     plugin:
+       name: barman-cloud.cloudnative-pg.io
+       parameters:
+         barmanObjectName: ferretdb-backupstore
+         serverName: pg

Don’t forget the annotation! Without it, barman-cloud will refuse to continue writing backups to the same ObjectStore and serverName after recovery. It is generally safe to use this annotation¹.

Now check this has applied successfully:

kubectl get cluster.postgresql.cnpg.io -n ferretdb

kubectl describe cluster.postgresql.cnpg.io -n ferretdb

(you can also shorten these to just use cluster if you have no other resources named cluster except the cnpg cluster.)

If it has, go ahead and delete your entire postgres cluster:

kubectl delete -n ferretdb cluster.postgresql.cnpg.io ferretdb-pg-cluster --force

Check that all the postgres cluster pods and PVCs are gone (you might have to kubectl delete --force some of them):

kubectl -n ferretdb get pods

kubectl -n ferretdb get pvc

If they are all gone, here comes the test: re-apply the postgresql YAML and it should start a full-recovery:

kubectl apply -f postgres-cluster.yaml

Check the pods:

kubectl -n ferretdb get pods -w Check the logs:

kubectl -n ferretdb logs name-of-the-pod

And soon, your cluster should be back and healthy. Connect to it as in Step 1 and check that your data is restored. Make a happy dance, then go to bed and sleep soundly, knowing your data is safe, no manual intervention is required for restore, and you are using a completely open source document database solution.

If things change and you want to check how I am running FerretDB, go to github.com/fhoekstra/home-ops, press t and type ferretdb.

If you found an error, I would love to accept your PR to this blog at https://codeberg.org/fhoekstra/blog

If you need help setting this up at home, come to the Home Operations Discord.

Attribution

I did not come up with any of this myself, I just combined the ideas from some amazing people around me, tested it and wrote it down. Special thanks go to @eaglesemanation for figuring out how to run FerretDB against cloudnative-pg without superuser access. That made this setup possible.

I should also thank @tholinka and @Phycoforce from the Home-Operations community for their help with barman-cloud backups. Their home-ops repos are a great resource.

You can browse many more repos using cloudnative-pg and barman-cloud by searching kubesearch.dev to get ideas of other ways to set this up.

Footnotes