How a Fedora Update Crashed My Server and the Fix

Fedora Server Update Troubleshooting

Running OpenZFS on Fedora Server provides a modern, fast container host with robust ZFS storage. However, an automatic Fedora update can unexpectedly pull in a new Linux kernel, causing compatibility issues.

The Moment of Realization

On a Friday evening, I tried to sync photos using the Immich app on my iPhone, but it wouldn’t connect to my server. Switching to my desktop, I found Immich still wouldn’t load. Something was definitely wrong. I opened Cockpit and discovered my Fedora Server homelab had failed significantly since I last checked. I turned to Gemini and my Gemini Notebook to start troubleshooting, and it was like unraveling a complex puzzle.

Leveraging Gemini for Troubleshooting

Gemini was crucial in guiding me through the troubleshooting process. It helped document each step, making it easy to track progress and identify what needed fixing. Gemini also suggested preventive measures to avoid future issues, which was reassuring since I often rely on experimentation and past experiences with my homelab.

With Gemini, I could:

  • Log each step: Keep a clear record of actions and results to see what worked.
  • Identify preventative steps: Get suggestions for avoiding similar issues in the future.
  • Stay organized: Have all information in one place to ensure nothing was overlooked.

Gemini simplified the process, acting like a digital assistant to ensure every detail was captured.

Balancing Act with Tools

Although I often write about de-googling and privacy-focused solutions, I use tools that meet my needs. While I aim to reduce reliance on big tech, Gemini has been incredibly helpful for managing my homelab, especially during troubleshooting.

This balance is part of finding tools that align with my values while ensuring efficiency. I hope my experience encourages others to choose tools that suit their needs, even if it means occasionally stepping outside their usual preferences.

Understanding the Key Concepts

  • OpenZFS: A high-performance file system and logical volume manager designed for data integrity and efficient storage management.
  • ZFS Storage: ZFS, short for Zettabyte File System, is designed to handle large amounts of data with features like data compression, snapshots, and error correction.
  • Cockpit: A web-based interface that simplifies managing Linux servers, providing a dashboard for monitoring and operations.
  • DKMS (Dynamic Kernel Module Support): A framework that automatically recompiles kernel modules when a new kernel is installed, ensuring driver compatibility.
  • /tank: A ZFS storage pool on my server, serving as a virtual storage space for important data and applications.

Layer 1: The Kernel Jump vs. Out-of-Tree ZFS

OpenZFS is an out-of-tree kernel module built via DKMS on Fedora, meaning every kernel update triggers a module rebuild. When Fedora’s automatic updates pulled in kernel 7.3.3, OpenZFS (2.4.2 / 2.4.4) only supported up to the 7.2 kernel series. On reboot:

  • DKMS failed to compile the ZFS module for 7.3.3.
  • Without the kernel module loaded, ZFS couldn’t import my mirrored pool (/tank), resulting in zpool status reporting no pools available.

The Fix

First, I rolled back and locked the kernel. Booting back into kernel 7.2.15-100.fc43.x86_64, I retrieved the matching archived headers from Fedora’s updates-archive repo (kernel-devel-7.2.15-100.fc43.x86_64) and updated OpenZFS to 2.4.4. Rolling back the kernel felt like stepping into a familiar old pair of shoes—comfortable and reliable.

Next, I imported the pool and re-enabled boot units. After DKMS built zfs/2.4.4 cleanly for 7.2.15, I brought the pool back and ensured the ZFS systemd targets were enabled for future boots:

sudo zpool import -a

sudo systemctl enable zfs-import-cache zfs-mount zfs.target

Layer 2: The Silent Podman + ZFS Boot Race Condition

Once /tank was back online, Cockpit’s Podman view showed my Immich containers (immich_postgres, immich_redis, immich_machine_learning) as Up and healthy. Yet, Immich wouldn’t load in the browser. Why? My rootless Podman user services (immich-stack.service and pod-paperless-stack.service) started at boot before /tank was imported and mounted. When containers start before their underlying ZFS dataset mounts:

  • Volume mounts bind to empty underlying directories on the root OS drive (or fail to initialize the application server properly).
  • Everything appears operational at the container runtime level while the app itself is broken.

The Fix

Once /tank was mounted, restarting the user-level systemd services forced Podman to tear down the stale mounts and cleanly bind to the real ZFS datasets:

systemctl --userrestart pod-paperless-stack.service
systemctl --userrestart immich-stack.service

(Bonus cleanup: an earlier service trigger had also left behind an empty default pod shell named pod_paperless, which I cleared out with podman pod rm -f pod_paperless.)

Layer 3: The “Ghost” DKMS Artifact

Even after kernel 7.2.15 was running, zfs/2.4.4 was loaded, and all containers were healthy, Cockpit kept flagging a boot failure: Failed to start dkms.service. Checking systemctl status dkms.service revealed the culprit wasn’t the active ZFS installation at all:

Autoinstall of module zfs/2.4.2 for kernel 7.2.15-100.fc43.x86_64 (x86_64)
(bad exit status: 1) Consult /var/lib/dkms/zfs/2.4.2/build/make.log for more information.

During the upgrade from OpenZFS 2.4.2 to 2.4.4, an orphaned /var/lib/dkms/zfs/2.4.2 directory was left behind. Because the package metadata was gone, dkms remove zfs/2.4.2 --all returned Error! The module/version combo: zfs/2.4.2 is not located in the DKMS tree—yet dkms.service still tried to build it on every boot.

The Fix

Removing the stale build folder directly and restarting dkms.service cleared the boot error immediately:

sudorm-rf /var/lib/dkms/zfs/2.4.2
sudo systemctl restart dkms.service

Hardening the Setup Against Future Surprises

To keep Fedora’s unattended security updates running without letting a surprise kernel bump break ZFS again, I made a couple of adjustments.

First, I pinned kernels in DNF. Adding excludepkgs=kernel* to /etc/dnf/dnf.conf allows normal userspace and security packages to auto-update while holding the kernel at 7.2.15 until I’m ready to manually verify OpenZFS compatibility. It felt like putting a lock on a door that had been swinging open at inconvenient times.

Next, I locked in a clean recovery snapshot. With /tank healthy and both stacks verified, I took a named recursive ZFS snapshot as a clean baseline:

sudo zfs snapshot -r tank@post-kernel-recovery-20261113

Taking a snapshot was like saving the game before a challenging level, reassuring to know there’s a point to return to if things go awry again.

Throughout this process, Gemini was a crucial tool, helping me keep track of technical details and suggesting steps to prevent future issues. While I don’t fully understand every technical aspect, Gemini provided invaluable guidance. I hope sharing my experience helps others facing similar challenges. The main lesson from managing a DIY Linux + ZFS + Podman server is that understanding how all the components work together makes solving problems much easier. This experience has underscored the importance of knowing the ins and outs of my homelab setup.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Scroll to Top

Discover more from Rob McBryde

Subscribe now to keep reading and get access to the full archive.

Continue reading