tudattr/homelab-docs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fact-check docs against live cluster and repo history

Browse Source 
Corrected stale claims found by cross-referencing git history and live
kubectl/SSH state:
- Drop Terraform provisioning claim (VMs are created by the ansible-homelab
  proxmox role via community.proxmox.proxmox_kvm; terraform/ is empty)
- Bump k3s badge to the version actually running (v1.36)
- Drop Elastic Stack as a live logging path - it was disabled in favor of
  agentmemory and has no namespace or ArgoCD Application anymore
- Drop zeroclaw from the app list - manifests exist but it was never wired
  into ArgoCD's app registry, so it has never actually run

Added detail confirmed against live state: Longhorn's Hetzner backup
target, node_cleanup containerd-pruning timer, MikroTik static DHCP
leases, Prometheus cardinality/false-positive-alert tuning, actual
CNPG cluster inventory, corrected MetalLB pool size, aya01's real NFS
capacity, k3s-server11's post-rebuild spec drift, and the naruto01
etcd quorum concentration.
This commit is contained in:
Tuan-Dat Tran
2026-07-28 01:43:07 +02:00
parent 15fe8665eb
commit 6936c3f82a
4 changed files with 34 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/networking.md
View File

@@ -5,11 +5,13 @@
| Segment | Range | Purpose |
|---------|-------|---------|
| LAN | `192.168.20.0/24` | All VMs — flat layer 2 |
| MetalLB pool | Reserved /28 within LAN | LoadBalancer services in Kubernetes |
| MetalLB pool | `192.168.20.240–250` (11 IPs) within LAN | LoadBalancer services in Kubernetes |
| K8s service CIDR | `10.43.0.0/16` | In-cluster service IPs |
| K8s pod CIDR | `10.42.0.0/16` | Pod networking (Flannel) |
| WireGuard | `10.133.7.0/24` | VPN tunnel: cluster ↔ edge VPS |
A handful of k3s nodes have static DHCP leases pinned on the MikroTik router (managed via Ansible, config backed up to Git on every run) — cheap insurance against a lease renewal quietly moving a node's IP out from under kube-vip or the inventory.
---
## Traffic flows

8
docs/observability.md
View File

@@ -6,7 +6,9 @@ Metrics and logs both surface in Grafana, keeping dashboards and log queries in
## Metrics
kube-prometheus-stack runs in the `prometheus` namespace (ArgoCD-managed). Prometheus scrapes all nodes, pods, and control plane components. Grafana has dashboards for cluster overview, node resources, Longhorn, ArgoCD, and Traefik.
kube-prometheus-stack runs in the `prometheus` namespace (ArgoCD-managed). Prometheus scrapes all nodes, pods, and control plane components, on a 60s scrape/eval interval rather than the more common 30s — halves CPU and WAL overhead at the cost of resolution. Grafana has dashboards for cluster overview, node resources, Longhorn, ArgoCD, and Traefik.
Because k3s embeds the scheduler and controller-manager (no separate scrape endpoint) and doesn't run kube-proxy, the stock `KubeControllerManagerDown` / `KubeSchedulerDown` / `KubeProxyDown` alerts are permanent false positives on this cluster and are disabled. Cardinality also got out of hand at one point — ~470k active series, largely duplicate apiserver/etcd metrics scraped twice (once directly, once via kubelet) plus high-cardinality histograms with no alerts attached. Relabeling rules drop those now; live series count sits around 350-400k.
Node Exporter is deployed via Ansible on every VM including `docker-host11` and the edge VPS, so coverage isn't limited to what's inside Kubernetes.
@@ -18,7 +20,9 @@ Alertmanager routes alerts to Ntfy via a custom webhook bridge.
## Logs
Loki runs in the `loki` namespace (single-node, filesystem storage). Promtail runs as a DaemonSet and ships pod logs from every cluster node to Loki. Logs are queryable in Grafana alongside metrics.
Loki runs in the `loki` namespace (single-node, filesystem storage, 14-day retention). Promtail runs as a DaemonSet and ships pod logs from every cluster node to Loki. Logs are queryable in Grafana alongside metrics.
An Elastic stack (Elasticsearch + Kibana via ECK) lived here for a while — manifests are still in the repo — but it was disabled in favor of running agentmemory instead, so Loki is currently the only log path.
---

8
docs/storage.md
View File

@@ -14,17 +14,19 @@ Three storage tiers, each doing a different job:
## Longhorn
Longhorn gives distributed block storage across all 14 agent nodes. Each volume is replicated (default: 3 replicas) across different nodes, using the local disk on each agent (128 GB each).
Longhorn gives distributed block storage across all 14 agent nodes. Each volume is replicated (default: 3 replicas) across different nodes, using the local disk on each agent (128 GB each), with soft anti-affinity and auto-balance enabled so replicas actively spread out rather than pile up.
RWO (ReadWriteOnce) covers most services. RWX (ReadWriteMany) is used where multiple pods need access to the same volume. Snapshots and backups are available through the Longhorn UI.
RWO (ReadWriteOnce) covers most services. RWX (ReadWriteMany) is used where multiple pods need access to the same volume. Snapshots and backups are available through the Longhorn UI — backups target a Hetzner Storage Box over CIFS, so a full node-loss scenario doesn't also mean a data-loss scenario.
Control plane nodes are tainted `NoSchedule` — Longhorn manager tolerates this and runs everywhere, but user workloads stay on agent nodes.
A recurring annoyance: containerd's image layer cache on `/var/lib/rancher/k3s/agent/containerd/` grows unbounded and can eat tens of GB, which Longhorn then reads as node-level disk pressure and stops scheduling. A systemd timer (`node_cleanup` Ansible role) prunes it daily with `crictl rmi --prune` and vacuums journal logs down to 500 MB, staggered by a random delay so it doesn't hit every node at once.
---
## CloudNativePG
CloudNativePG manages HA PostgreSQL clusters as Kubernetes resources. Immich uses it for its primary database (photos, albums, users, ML embeddings). CNPG handles streaming replication, failover, and scheduled backups, with data stored on Longhorn PVCs.
CloudNativePG manages HA PostgreSQL clusters as Kubernetes resources — one `Cluster` per app (Immich, Vaultwarden, N8n, Paperless, LiteLLM), each backed by Longhorn PVCs. Vaultwarden runs a 3-instance HA cluster with automatic failover; the rest run single-instance. CNPG handles streaming replication, failover, and scheduled backups via Barman Cloud to S3-compatible storage, with WAL compression and point-in-time recovery.
---