1.9 KiB
Storage
Overview
Three storage tiers, each doing a different job:
| Tier | System | Access | Used by |
|---|---|---|---|
| Distributed block | Longhorn | RWO + RWX | All stateful K8s workloads |
| Relational | CloudNativePG | In-cluster Postgres | Immich |
| Network file | NFS (bare-metal) | NFS mount | Jellyfin media library |
Longhorn
Longhorn gives distributed block storage across all 14 agent nodes. Each volume is replicated (default: 3 replicas) across different nodes, using the local disk on each agent (128 GB each).
RWO (ReadWriteOnce) covers most services. RWX (ReadWriteMany) is used where multiple pods need access to the same volume. Snapshots and backups are available through the Longhorn UI.
Control plane nodes are tainted NoSchedule — Longhorn manager tolerates this and runs everywhere, but user workloads stay on agent nodes.
CloudNativePG
CloudNativePG manages HA PostgreSQL clusters as Kubernetes resources. Immich uses it for its primary database (photos, albums, users, ML embeddings). CNPG handles streaming replication, failover, and scheduled backups, with data stored on Longhorn PVCs.
NFS
aya01 is a dedicated bare-metal NFS server. Jellyfin mounts the share from docker-host11 to access movies, TV shows, and music. Keeping the media library on a separate host means the Jellyfin VM can be rebuilt without touching the data.
NFS is not used for K8s workloads — Longhorn handles all PVC-backed storage.
Secrets
Kubernetes secrets go through Sealed Secrets (Bitnami). The workflow: create a regular Secret, encrypt it with kubeseal using the cluster's public key into a SealedSecret, then commit that to Git. Only the in-cluster controller can decrypt it.
Ansible secrets (VM credentials, API tokens) are encrypted with Ansible Vault and live in vars/group_vars/*/secrets_*.yaml.