6bfdf2c674
Narrative account of the August 2026 migration: why fourteen small agents cost more than seven large ones, the rebuild failures that made the first pair take three times its budget, and the faulty Ethernet cable behind them that had been silently downshifting a host to 100 Mbit for nine days. Also records the operational gotchas worth knowing: Longhorn evicts at disk level rather than node level, draining does not preserve replicas, rebuilds can report progress while transferring nothing, GitOps selfHeal reverts temporary drain workarounds, and cloud images panic on the first boot after a resize. Updates node counts in README and storage.md, which the post would otherwise contradict, and adds a deep-dives index so the docs/ pages are reachable from the README at all.
2.6 KiB
Storage
Overview
Three storage tiers, each doing a different job:
| Tier | System | Access | Used by |
|---|---|---|---|
| Distributed block | Longhorn | RWO + RWX | All stateful K8s workloads |
| Relational | CloudNativePG | In-cluster Postgres | Immich |
| Network file | NFS (bare-metal) | NFS mount | Jellyfin media library |
Longhorn
Longhorn gives distributed block storage across all 7 agent nodes. Each volume is replicated (default: 3 replicas) across different nodes, using the local disk on each agent (256 GB each), with soft anti-affinity and auto-balance enabled so replicas actively spread out rather than pile up.
RWO (ReadWriteOnce) covers most services. RWX (ReadWriteMany) is used where multiple pods need access to the same volume. Snapshots and backups are available through the Longhorn UI — backups target a Hetzner Storage Box over CIFS, so a full node-loss scenario doesn't also mean a data-loss scenario.
Control plane nodes are tainted NoSchedule — Longhorn manager tolerates this and runs everywhere, but user workloads stay on agent nodes.
A recurring annoyance: containerd's image layer cache on /var/lib/rancher/k3s/agent/containerd/ grows unbounded and can eat tens of GB, which Longhorn then reads as node-level disk pressure and stops scheduling. A systemd timer (node_cleanup Ansible role) prunes it daily with crictl rmi --prune and vacuums journal logs down to 500 MB, staggered by a random delay so it doesn't hit every node at once.
CloudNativePG
CloudNativePG manages HA PostgreSQL clusters as Kubernetes resources — one Cluster per app (Immich, Vaultwarden, N8n, Paperless, LiteLLM), each backed by Longhorn PVCs. Vaultwarden runs a 3-instance HA cluster with automatic failover; the rest run single-instance. CNPG handles streaming replication, failover, and scheduled backups via Barman Cloud to S3-compatible storage, with WAL compression and point-in-time recovery.
NFS
aya01 is a dedicated bare-metal NFS server. Jellyfin mounts the share from docker-host11 to access movies, TV shows, and music. Keeping the media library on a separate host means the Jellyfin VM can be rebuilt without touching the data.
NFS is not used for K8s workloads — Longhorn handles all PVC-backed storage.
Secrets
Kubernetes secrets go through Sealed Secrets (Bitnami). The workflow: create a regular Secret, encrypt it with kubeseal using the cluster's public key into a SealedSecret, then commit that to Git. Only the in-cluster controller can decrypt it.
Ansible secrets (VM credentials, API tokens) are encrypted with Ansible Vault and live in vars/group_vars/*/secrets_*.yaml.