60 lines
2.3 KiB
Markdown
60 lines
2.3 KiB
Markdown
# Storage
|
|
|
|
## Overview
|
|
|
|
Three storage tiers serve different workloads:
|
|
|
|
| Tier | System | Access | Used by |
|
|
|------|--------|--------|---------|
|
|
| Distributed block | Longhorn | RWO + RWX | All stateful K8s workloads |
|
|
| Relational | CloudNativePG | In-cluster Postgres | Immich |
|
|
| Network file | NFS (bare-metal) | NFS mount | Jellyfin media library |
|
|
|
|
---
|
|
|
|
## Longhorn
|
|
|
|
Longhorn provides distributed block storage across all 14 agent nodes. Each volume is replicated (default: 3 replicas) across different nodes.
|
|
|
|
- **RWO** (ReadWriteOnce) — used for most services (Vaultwarden, Paperless, etc.)
|
|
- **RWX** (ReadWriteMany) — used where multiple pods need shared access
|
|
- Volumes are backed by the local disk on each agent node (128 GB each)
|
|
- Longhorn manager runs as a DaemonSet; the CSI plugin integrates with the K8s storage layer
|
|
- Snapshots and backups are supported via the Longhorn UI
|
|
|
|
Control plane nodes (`k3s-server-*`) are tainted `NoSchedule` — Longhorn manager tolerates this taint and runs everywhere, but user workloads are pushed to agent nodes only.
|
|
|
|
---
|
|
|
|
## CloudNativePG
|
|
|
|
The CNPG operator manages HA PostgreSQL clusters as first-class Kubernetes resources. Currently used by:
|
|
|
|
- **Immich** — primary database (photos, albums, users, ML embeddings)
|
|
|
|
CNPG handles streaming replication, failover, and scheduled backups. Data is stored on Longhorn PVCs.
|
|
|
|
---
|
|
|
|
## NFS
|
|
|
|
A dedicated physical node (`aya01`) runs a bare-metal NFS server. This serves the media library to Jellyfin.
|
|
|
|
- Movies, TV shows, and music live on `aya01`
|
|
- `docker-host11` (where Jellyfin runs) mounts the NFS share
|
|
- Separating media storage from the compute host means the Jellyfin VM can be rebuilt without touching the library
|
|
- NFS is not used for K8s workloads — Longhorn handles all PVC-backed storage
|
|
|
|
---
|
|
|
|
## Secret Storage
|
|
|
|
Kubernetes secrets are managed with **Sealed Secrets** (Bitnami). The workflow:
|
|
|
|
1. Create a regular K8s `Secret`
|
|
2. Encrypt it with `kubeseal` using the cluster's public key → produces a `SealedSecret`
|
|
3. Commit the `SealedSecret` to Git — it is safe to store publicly
|
|
4. The in-cluster Sealed Secrets controller decrypts it into a regular `Secret` at apply time
|
|
|
|
Ansible secrets (VM credentials, API tokens) are encrypted with **Ansible Vault** and stored in `vars/group_vars/*/secrets_*.yaml`.
|