Files

Tuan-Dat Tran 6936c3f82a Fact-check docs against live cluster and repo history

Corrected stale claims found by cross-referencing git history and live
kubectl/SSH state:
- Drop Terraform provisioning claim (VMs are created by the ansible-homelab
  proxmox role via community.proxmox.proxmox_kvm; terraform/ is empty)
- Bump k3s badge to the version actually running (v1.36)
- Drop Elastic Stack as a live logging path - it was disabled in favor of
  agentmemory and has no namespace or ArgoCD Application anymore
- Drop zeroclaw from the app list - manifests exist but it was never wired
  into ArgoCD's app registry, so it has never actually run

Added detail confirmed against live state: Longhorn's Hetzner backup
target, node_cleanup containerd-pruning timer, MikroTik static DHCP
leases, Prometheus cardinality/false-positive-alert tuning, actual
CNPG cluster inventory, corrected MetalLB pool size, aya01's real NFS
capacity, k3s-server11's post-rebuild spec drift, and the naruto01
etcd quorum concentration.

2026-07-28 01:43:07 +02:00

2.6 KiB

Raw Blame History

Storage

Overview

Three storage tiers, each doing a different job:

Tier	System	Access	Used by
Distributed block	Longhorn	RWO + RWX	All stateful K8s workloads
Relational	CloudNativePG	In-cluster Postgres	Immich
Network file	NFS (bare-metal)	NFS mount	Jellyfin media library

Longhorn

Longhorn gives distributed block storage across all 14 agent nodes. Each volume is replicated (default: 3 replicas) across different nodes, using the local disk on each agent (128 GB each), with soft anti-affinity and auto-balance enabled so replicas actively spread out rather than pile up.

RWO (ReadWriteOnce) covers most services. RWX (ReadWriteMany) is used where multiple pods need access to the same volume. Snapshots and backups are available through the Longhorn UI — backups target a Hetzner Storage Box over CIFS, so a full node-loss scenario doesn't also mean a data-loss scenario.

Control plane nodes are tainted NoSchedule — Longhorn manager tolerates this and runs everywhere, but user workloads stay on agent nodes.

A recurring annoyance: containerd's image layer cache on /var/lib/rancher/k3s/agent/containerd/ grows unbounded and can eat tens of GB, which Longhorn then reads as node-level disk pressure and stops scheduling. A systemd timer (node_cleanup Ansible role) prunes it daily with crictl rmi --prune and vacuums journal logs down to 500 MB, staggered by a random delay so it doesn't hit every node at once.

CloudNativePG

CloudNativePG manages HA PostgreSQL clusters as Kubernetes resources — one Cluster per app (Immich, Vaultwarden, N8n, Paperless, LiteLLM), each backed by Longhorn PVCs. Vaultwarden runs a 3-instance HA cluster with automatic failover; the rest run single-instance. CNPG handles streaming replication, failover, and scheduled backups via Barman Cloud to S3-compatible storage, with WAL compression and point-in-time recovery.

NFS

aya01 is a dedicated bare-metal NFS server. Jellyfin mounts the share from docker-host11 to access movies, TV shows, and music. Keeping the media library on a separate host means the Jellyfin VM can be rebuilt without touching the data.

NFS is not used for K8s workloads — Longhorn handles all PVC-backed storage.

Secrets

Kubernetes secrets go through Sealed Secrets (Bitnami). The workflow: create a regular Secret, encrypt it with kubeseal using the cluster's public key into a SealedSecret, then commit that to Git. Only the in-cluster controller can decrypt it.

Ansible secrets (VM credentials, API tokens) are encrypted with Ansible Vault and live in vars/group_vars/*/secrets_*.yaml.

2.6 KiB Raw Blame History