6936c3f82a
Corrected stale claims found by cross-referencing git history and live kubectl/SSH state: - Drop Terraform provisioning claim (VMs are created by the ansible-homelab proxmox role via community.proxmox.proxmox_kvm; terraform/ is empty) - Bump k3s badge to the version actually running (v1.36) - Drop Elastic Stack as a live logging path - it was disabled in favor of agentmemory and has no namespace or ArgoCD Application anymore - Drop zeroclaw from the app list - manifests exist but it was never wired into ArgoCD's app registry, so it has never actually run Added detail confirmed against live state: Longhorn's Hetzner backup target, node_cleanup containerd-pruning timer, MikroTik static DHCP leases, Prometheus cardinality/false-positive-alert tuning, actual CNPG cluster inventory, corrected MetalLB pool size, aya01's real NFS capacity, k3s-server11's post-rebuild spec drift, and the naruto01 etcd quorum concentration.
46 lines
2.6 KiB
Markdown
46 lines
2.6 KiB
Markdown
# Storage
|
|
|
|
## Overview
|
|
|
|
Three storage tiers, each doing a different job:
|
|
|
|
| Tier | System | Access | Used by |
|
|
|------|--------|--------|---------|
|
|
| Distributed block | Longhorn | RWO + RWX | All stateful K8s workloads |
|
|
| Relational | CloudNativePG | In-cluster Postgres | Immich |
|
|
| Network file | NFS (bare-metal) | NFS mount | Jellyfin media library |
|
|
|
|
---
|
|
|
|
## Longhorn
|
|
|
|
Longhorn gives distributed block storage across all 14 agent nodes. Each volume is replicated (default: 3 replicas) across different nodes, using the local disk on each agent (128 GB each), with soft anti-affinity and auto-balance enabled so replicas actively spread out rather than pile up.
|
|
|
|
RWO (ReadWriteOnce) covers most services. RWX (ReadWriteMany) is used where multiple pods need access to the same volume. Snapshots and backups are available through the Longhorn UI — backups target a Hetzner Storage Box over CIFS, so a full node-loss scenario doesn't also mean a data-loss scenario.
|
|
|
|
Control plane nodes are tainted `NoSchedule` — Longhorn manager tolerates this and runs everywhere, but user workloads stay on agent nodes.
|
|
|
|
A recurring annoyance: containerd's image layer cache on `/var/lib/rancher/k3s/agent/containerd/` grows unbounded and can eat tens of GB, which Longhorn then reads as node-level disk pressure and stops scheduling. A systemd timer (`node_cleanup` Ansible role) prunes it daily with `crictl rmi --prune` and vacuums journal logs down to 500 MB, staggered by a random delay so it doesn't hit every node at once.
|
|
|
|
---
|
|
|
|
## CloudNativePG
|
|
|
|
CloudNativePG manages HA PostgreSQL clusters as Kubernetes resources — one `Cluster` per app (Immich, Vaultwarden, N8n, Paperless, LiteLLM), each backed by Longhorn PVCs. Vaultwarden runs a 3-instance HA cluster with automatic failover; the rest run single-instance. CNPG handles streaming replication, failover, and scheduled backups via Barman Cloud to S3-compatible storage, with WAL compression and point-in-time recovery.
|
|
|
|
---
|
|
|
|
## NFS
|
|
|
|
`aya01` is a dedicated bare-metal NFS server. Jellyfin mounts the share from `docker-host11` to access movies, TV shows, and music. Keeping the media library on a separate host means the Jellyfin VM can be rebuilt without touching the data.
|
|
|
|
NFS is not used for K8s workloads — Longhorn handles all PVC-backed storage.
|
|
|
|
---
|
|
|
|
## Secrets
|
|
|
|
Kubernetes secrets go through Sealed Secrets (Bitnami). The workflow: create a regular `Secret`, encrypt it with `kubeseal` using the cluster's public key into a `SealedSecret`, then commit that to Git. Only the in-cluster controller can decrypt it.
|
|
|
|
Ansible secrets (VM credentials, API tokens) are encrypted with Ansible Vault and live in `vars/group_vars/*/secrets_*.yaml`.
|