All Systems Operational With Podman

After a frustrating couple of days trying to recover data after inadvertently deleting all of my Docker containers on the Nukbox, which included my 300GB Immich photo collection, I am happy to report that all systems are operational again. This time with persistent storage and backups in place!

It’s unfortunate that some data was lost (an undetermined number of Immich photos from the last couple of years as well as less than 30 PDFs that were uploaded to Paperless). But like I said in the last post, failure is a great teacher. I’m happier with my homelab set up now than I’ve ever been, and my mistake last Sunday of deleting all my containers catapulted me forward with learning Cockpit and Podman, both of which I’ve become quite familiar.

🏛️ My New Architecture

====================================================================
 🛡️ THE TAILSCALE MESH (Private & Encrypted Peer-to-Peer Network)
====================================================================

[ 🚀 MISSION CONTROL ]
THINKSTATION P310 (Fedora Server 43)
Specs: Intel i7-6700 | 48GB RAM | 512GB SSD (Boot)
——————————————————————–
|
├── 🗄️ ZFS POOL (The “Tank”)
│ └── 3TB Mirrored Storage (The Single Source of Truth)
|
└── 🐳 PODMAN / COCKPIT (The Production Fleet)
    ├── Homarr (The Dashboard)
    ├── Immich (Photo Management)
    ├── Paperless-ngx (Document Archive)
    ├── Joplin Server (Notes Backend)
    ├── Calibre-web (The Reader Interface)
    └── Syncthing (Data Synchronization)

====================================================================

[ 💻 PRIMARY WORKSTATION ]
NUKBOX K10 (Pop!_OS)
Specs: Intel i9-13900HK | 98GB RAM | 4TB NVMe SSD
——————————————————————–
|
└── 🖥️ Usage: Daily Work, High-Performance Lab Administration
    └── (Zero containers running for now; pure workstation power)

====================================================================

[ 📚 AI & SOURCE LIBRARY ]
2020 MAC MINI (macOS)
Specs: Apple M1 | 16GB RAM | 1TB SSD | 2TB SSD
——————————————————————–
|
├── 📚 CALIBRE DESKTOP (Master Library)
│ └── (Local ebook files & metadata.db managed here)
|
└── 🧠 OLLAMA (Containerized via Docker Desktop)
    └── (Local LLM & Generative AI services)

====================================================================

The Mesh: A Private Cloud with Tailscale

Tailscale has effectively transformed my home server from a simple local project into a fully functional private cloud. It acts as a secure, private tunnel that allows me to access my dashboard and applications like Homarr and Immich from my phone or laptop regardless of where I am. The beauty of this connectivity is that it requires no complex router configurations or risky port forwarding, providing a seamless and encrypted bridge to my data from anywhere in the world.

Photography and Media: My Personal Vault

My application ecosystem is now powered by rootless Podman, with every service backed by the reliability of a mirrored ZFS tank. At the center of my media management is Immich, a high-performance Google Photos replacement that handles intensive AI and machine learning tasks to organize massive photo libraries. Using the Immich mobile app over Tailscale, my photos automatically sync to my personal vault the moment I take them, ensuring I never have to rely on big tech cloud providers again.

Organization: The Digital Filing Cabinet

For document management and personal organization, I rely on Paperless and Joplin Server. Paperless serves as a digital filing cabinet, utilizing OCR text recognition to make my entire PDF archive searchable and organized. Meanwhile, Joplin Server acts as my private note-taking backend, a service that replaced Apple Notes years ago, providing an encrypted, database-backed sync that keeps my thoughts and records consistent across all my devices.

Mission Control: Library and Logistics

The rest of the ecosystem is rounded out by specialized tools for media, data movement, and oversight. Calibre-Web provides a beautiful e-book library interface, giving me anywhere access to my entire digital book collection. To keep data flowing between my NukBox workstation and the ZFS storage, I use Syncthing, which bypasses traditional cloud storage by syncing files directly between hardware. Finally, everything is tied together by Homarr, my Mission Control dashboard, which provides a centralized starting point for all these services so I never have to memorize individual port numbers again.

The Next Chapter: Backups

Now that my services are stable, I’m going to focus my energy into the most critical part of any home lab: making sure I never have to do this from scratch again. My next topic will dive deep into backup configurations, where I’ll be exploring the 3-2-1 backup rule using ZFS snapshots and automated replication. I’m looking forward to sharing how I’ve set up a system that not only protects my files but also makes the entire server resilient enough to survive a total hardware failure.

PS – Fool Me Once, Shame On You

I tried and tried to configure Calibre on the server (not just Calibre-web), but kept hitting walls that I couldn’t overcome. At one point, Gemini suggested a “hard reset” for Podman.

The “Hard Reset” for Podman

Since system renumber failed, we’ll use the “shutdown” command which is more aggressive at clearing the internal state.

Wait! Read this first: podman system reset will remove all your containers and images (it won’t touch your books/files in /tank). Since we want to start fresh and your Calibre-Web is also there, you’ll have to run the podman run command for Calibre-Web again afterward. This is the only way to clear the “layer is in use” deadlock.

To which I replied:

What?

robmcbryde@fedoraserver:~$ podman system reset

WARNING! This will remove:

– all containers

What happens to all my other containers for Immich, Paperless, Joplin Server, etc?

Glad I asked! Gemini replied:

If you hit y, they will be deleted.

I’m glad you paused. Since you have a whole ecosystem running (Immich, Paperless, etc.), do not run podman system reset yet. When Podman says “all containers,” it means the instances and the images (the downloaded apps). While your data (the photos in Immich, the documents in Paperless) is safe on your ZFS drive, you would have to manually run the podman run commands or docker-compose up for every single one of those apps to get them back.

After several back and forth messages with Gemini, I decided to delete the Calibre image from the Cockpit GUI.

Gemini was impressed:

That was a smart move. Deleting it from the Cockpit GUI helps ensure that any residual metadata or “half-pulled” image layers are cleared out of the Podman database. Since you’re using Cockpit on Fedora, it sometimes handles the background cleanup better than a forced CLI command when things get “ghosted.”

That was a close one! Phew!

This goes to show you can teach an old dog new tricks. Less than a week ago, I made the mistake of blindly following Gemini’s guidance. This time I paused and asked, “Wait a minute, are you sure?” It was a blunt reminder that while these AI tools are great for troubleshooting maybe most of the time, you have to manually verify every command before hitting enter on your primary server. A measure twice, cut once type thing.