Fact-check docs against live cluster and repo history
Corrected stale claims found by cross-referencing git history and live kubectl/SSH state: - Drop Terraform provisioning claim (VMs are created by the ansible-homelab proxmox role via community.proxmox.proxmox_kvm; terraform/ is empty) - Bump k3s badge to the version actually running (v1.36) - Drop Elastic Stack as a live logging path - it was disabled in favor of agentmemory and has no namespace or ArgoCD Application anymore - Drop zeroclaw from the app list - manifests exist but it was never wired into ArgoCD's app registry, so it has never actually run Added detail confirmed against live state: Longhorn's Hetzner backup target, node_cleanup containerd-pruning timer, MikroTik static DHCP leases, Prometheus cardinality/false-positive-alert tuning, actual CNPG cluster inventory, corrected MetalLB pool size, aya01's real NFS capacity, k3s-server11's post-rebuild spec drift, and the naruto01 etcd quorum concentration.
This commit is contained in:
Tuan-Dat Tran
36
README.md
36
README.md
@@ -1,8 +1,8 @@
|
||||
# Homelab
|
||||
|
||||
17-node Kubernetes cluster on four bare-metal Proxmox hosts, provisioned with Terraform and Ansible, managed through ArgoCD GitOps. Runs my home automation, media stack, photo backup, documents, and a few side projects.
|
||||
17-node Kubernetes cluster on four bare-metal Proxmox hosts, provisioned and configured with Ansible, managed through ArgoCD GitOps. Runs my home automation, media stack, photo backup, documents, and a few side projects.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -77,11 +77,13 @@ Secrets are committed to Git too, encrypted via Sealed Secrets. Only the in-clus
|
||||
|
||||
No ports are open on the home router. Internal load balancing goes through MetalLB + Traefik. External access uses Cloudflare tunnels or a WireGuard VPN routed through the edge VPS.
|
||||
|
||||
Longhorn handles block storage by replicating volumes across all 14 agent nodes. The media library lives on a dedicated NFS host instead — latency matters when Jellyfin is reading large video files, and NFS is simpler for that.
|
||||
Longhorn handles block storage by replicating volumes across all 14 agent nodes, with backup snapshots shipped off-site to a Hetzner Storage Box over CIFS. The media library lives on a dedicated NFS host instead — latency matters when Jellyfin is reading large video files, and NFS is simpler for that.
|
||||
|
||||
Metrics go to Prometheus + Grafana. Logs go to Loki, also surfaced in Grafana — Promtail runs as a DaemonSet and ships pod logs from every node.
|
||||
|
||||
All VMs are provisioned with Terraform and configured by Ansible. Rebuilding from scratch doesn't require remembering anything.
|
||||
Not every HA guarantee holds up under scrutiny — two of the three etcd control-plane VMs currently run on the same physical Proxmox host (`naruto01`), a single point of failure for cluster quorum that surfaced during an incident and is still on the list to rebalance.
|
||||
|
||||
All VMs are provisioned and configured by Ansible — the `proxmox` role calls the Proxmox API directly (`community.proxmox.proxmox_kvm`) to clone, resize, and boot VMs from cloud-init templates, then the rest of the playbooks configure them. Rebuilding from scratch doesn't require remembering anything.
|
||||
|
||||
---
|
||||
|
||||
@@ -89,18 +91,18 @@ All VMs are provisioned with Terraform and configured by Ansible. Rebuilding fro
|
||||
|
||||
| Layer | Host | Role | Resources |
|
||||
|-------|------|------|-----------|
|
||||
| Physical | `aya01` | Proxmox node + NFS server | Dedicated storage — no VMs |
|
||||
| Physical | `aya01` | Proxmox node + NFS server (49 TB share, ~60% used) | Dedicated storage — no VMs |
|
||||
| Physical | `lulu` | Proxmox node | k3s agents |
|
||||
| Physical | `naruto01` | Proxmox node | k3s server + agents |
|
||||
| Physical | `mii01` | Proxmox node | k3s server + agents + docker host |
|
||||
| VM | `k3s-server-{10,11,12}` | K3s control plane (HA etcd + kube-vip VIP) | 2 vCPU · 4 GB RAM · 64 GB |
|
||||
| VM | `k3s-server-{10,11,12}` | K3s control plane (HA etcd + kube-vip VIP) | 2 vCPU · 4 GB RAM · 64 GB (`k3s-server11` runs 4 vCPU · 6 GB after a rebuild — not yet reconciled with the other two) |
|
||||
| VM | `k3s-agent-{10…23}` | K3s worker nodes ×14 | 2 vCPU · 4 GB RAM · 128 GB |
|
||||
| VM | `docker-host11` | Docker host | 2 vCPU · 4 GB RAM · 192 GB |
|
||||
| VM | `docker-host11` | Docker host (AMD Ryzen 7 5700U — CPU-only, no iGPU passthrough for Jellyfin) | 2 vCPU · 4 GB RAM · 192 GB |
|
||||
| VM | `docker-lb` | Caddy reverse proxy (LAN only) | 1 vCPU · 2 GB RAM |
|
||||
| VPS | `mii` | Edge node (Netcup) | WireGuard · Traefik · Pangolin |
|
||||
| SBC | `naruto` | Raspberry Pi | Zigbee2MQTT · Mosquitto MQTT broker |
|
||||
|
||||
All VMs run Debian 12 on `virtio` network bridges, provisioned from cloud-init templates via Terraform + Ansible. The Raspberry Pi runs Docker Compose, managed by the same Ansible setup.
|
||||
All VMs run Debian 12 on `virtio` network bridges, provisioned from cloud-init templates via Ansible. The Raspberry Pi runs Docker Compose, managed by the same Ansible setup. A MikroTik router is also under Ansible's control — it backs up its RouterOS config to Git on every run and gets static DHCP leases pinned for the k3s nodes, so a lease renewal can't silently move a node's IP.
|
||||
|
||||
---
|
||||
|
||||
@@ -116,9 +118,9 @@ All VMs run Debian 12 on `virtio` network bridges, provisioned from cloud-init t
|
||||
| **Cert-Manager** | Helm (ArgoCD) | Automated TLS via Let's Encrypt DNS-01 (Cloudflare API) |
|
||||
| **Sealed Secrets** | Helm (ArgoCD) | Encrypts secrets for safe storage in Git |
|
||||
| **Longhorn** | Helm (ArgoCD) | Distributed block storage (RWO + RWX) across all 14 agents |
|
||||
| **CloudNativePG** | Operator (ArgoCD) | HA PostgreSQL — used by Immich and LiteLLM |
|
||||
| **CloudNativePG** | Operator (ArgoCD) | HA PostgreSQL — one cluster per app (Immich, Vaultwarden, N8n, Paperless, LiteLLM); Vaultwarden's is 3-instance HA, the rest single-instance |
|
||||
| **Kube-Prometheus-Stack** | Helm (ArgoCD) | Prometheus + Grafana monitoring |
|
||||
| **Loki + Promtail** | Helm (ArgoCD) | Log aggregation — Promtail DaemonSet ships pod logs to Loki |
|
||||
| **Loki + Promtail** | Helm (ArgoCD) | Log aggregation — Promtail DaemonSet ships pod logs to Loki (14-day retention) |
|
||||
| **Goldilocks + VPA** | Helm (ArgoCD) | Resource usage analysis and request/limit rightsizing |
|
||||
| **Istio (Ambient)** | Helm (ArgoCD) | Service mesh — ztunnel DaemonSet on all nodes (L4); no Waypoint proxies yet |
|
||||
| **K3s Upgrade Controller** | Operator (ArgoCD) | Automated rolling K3s version upgrades |
|
||||
@@ -141,12 +143,12 @@ All VMs run Debian 12 on `virtio` network bridges, provisioned from cloud-init t
|
||||
| **Stirling PDF** | PDF tools | – |
|
||||
| **Karakeep** | Bookmark manager | – |
|
||||
| **Gitea** | Self-hosted Git (source of truth for ArgoCD) | Kubernetes · Longhorn PVC · SSH via MetalLB · Pangolin tunnel (git.tudattr.dev) |
|
||||
| **Gitea Runner** | CI/CD runner | – |
|
||||
| **Arr Stack** | Media automation suite | Prowlarr · Sonarr · Radarr · Unpackarr |
|
||||
| **Download clients** | VPN-isolated download clients (×2) | Gluetun sidecar |
|
||||
| **Gitea Runner** | CI/CD runner, ×2 (one for the in-cluster Gitea, one for the external instance) | Docker-in-Docker, rootless, shared RWX Longhorn volume |
|
||||
| **Arr Stack** | Media automation suite | Prowlarr · Sonarr · Radarr · Recyclarr · Unpackarr |
|
||||
| **Download clients** | VPN-isolated + clearnet qBittorrent pair, run side by side | Gluetun sidecar (ProtonVPN) on the VPN-isolated one |
|
||||
| **Jellyfin** | Media server, CPU-only transcoding | Docker |
|
||||
| **LiteLLM** | OpenAI-compatible LLM proxy — routes to Gemini, DeepSeek, and other providers | CloudNativePG |
|
||||
| **agentmemory** | Persistent memory server for AI agents | Longhorn PVC · private Gitea registry |
|
||||
| **LiteLLM** | OpenAI-compatible LLM proxy — routes to Gemini 2.5 Flash and DeepSeek | CloudNativePG |
|
||||
| **agentmemory** | Persistent memory server for AI agents — consolidation, graph extraction, reflection | Longhorn PVC · private Gitea registry · local embeddings |
|
||||
| **athome** | Personal homepage | – |
|
||||
|
||||
---
|
||||
@@ -163,7 +165,9 @@ ansible-homelab/
|
||||
│ ├── docker_host/ # Docker + Compose services
|
||||
│ ├── proxmox/ # Proxmox node setup
|
||||
│ ├── edge_vps/ # VPS: WireGuard, Traefik, Pangolin
|
||||
│ └── raspberry_pi/ # Raspberry Pi: Zigbee2MQTT + Mosquitto via Docker Compose
|
||||
│ ├── raspberry_pi/ # Raspberry Pi: Zigbee2MQTT + Mosquitto via Docker Compose
|
||||
│ ├── mikrotik/ # router config backup + static DHCP leases
|
||||
│ └── node_cleanup/ # systemd timer: prunes containerd images, vacuums journal logs
|
||||
└── playbooks/
|
||||
|
||||
argocd-homelab/
|
||||
|
||||
@@ -5,11 +5,13 @@
|
||||
| Segment | Range | Purpose |
|
||||
|---------|-------|---------|
|
||||
| LAN | `192.168.20.0/24` | All VMs — flat layer 2 |
|
||||
| MetalLB pool | Reserved /28 within LAN | LoadBalancer services in Kubernetes |
|
||||
| MetalLB pool | `192.168.20.240–250` (11 IPs) within LAN | LoadBalancer services in Kubernetes |
|
||||
| K8s service CIDR | `10.43.0.0/16` | In-cluster service IPs |
|
||||
| K8s pod CIDR | `10.42.0.0/16` | Pod networking (Flannel) |
|
||||
| WireGuard | `10.133.7.0/24` | VPN tunnel: cluster ↔ edge VPS |
|
||||
|
||||
A handful of k3s nodes have static DHCP leases pinned on the MikroTik router (managed via Ansible, config backed up to Git on every run) — cheap insurance against a lease renewal quietly moving a node's IP out from under kube-vip or the inventory.
|
||||
|
||||
---
|
||||
|
||||
## Traffic flows
|
||||
|
||||
@@ -6,7 +6,9 @@ Metrics and logs both surface in Grafana, keeping dashboards and log queries in
|
||||
|
||||
## Metrics
|
||||
|
||||
kube-prometheus-stack runs in the `prometheus` namespace (ArgoCD-managed). Prometheus scrapes all nodes, pods, and control plane components. Grafana has dashboards for cluster overview, node resources, Longhorn, ArgoCD, and Traefik.
|
||||
kube-prometheus-stack runs in the `prometheus` namespace (ArgoCD-managed). Prometheus scrapes all nodes, pods, and control plane components, on a 60s scrape/eval interval rather than the more common 30s — halves CPU and WAL overhead at the cost of resolution. Grafana has dashboards for cluster overview, node resources, Longhorn, ArgoCD, and Traefik.
|
||||
|
||||
Because k3s embeds the scheduler and controller-manager (no separate scrape endpoint) and doesn't run kube-proxy, the stock `KubeControllerManagerDown` / `KubeSchedulerDown` / `KubeProxyDown` alerts are permanent false positives on this cluster and are disabled. Cardinality also got out of hand at one point — ~470k active series, largely duplicate apiserver/etcd metrics scraped twice (once directly, once via kubelet) plus high-cardinality histograms with no alerts attached. Relabeling rules drop those now; live series count sits around 350-400k.
|
||||
|
||||
Node Exporter is deployed via Ansible on every VM including `docker-host11` and the edge VPS, so coverage isn't limited to what's inside Kubernetes.
|
||||
|
||||
@@ -18,7 +20,9 @@ Alertmanager routes alerts to Ntfy via a custom webhook bridge.
|
||||
|
||||
## Logs
|
||||
|
||||
Loki runs in the `loki` namespace (single-node, filesystem storage). Promtail runs as a DaemonSet and ships pod logs from every cluster node to Loki. Logs are queryable in Grafana alongside metrics.
|
||||
Loki runs in the `loki` namespace (single-node, filesystem storage, 14-day retention). Promtail runs as a DaemonSet and ships pod logs from every cluster node to Loki. Logs are queryable in Grafana alongside metrics.
|
||||
|
||||
An Elastic stack (Elasticsearch + Kibana via ECK) lived here for a while — manifests are still in the repo — but it was disabled in favor of running agentmemory instead, so Loki is currently the only log path.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -14,17 +14,19 @@ Three storage tiers, each doing a different job:
|
||||
|
||||
## Longhorn
|
||||
|
||||
Longhorn gives distributed block storage across all 14 agent nodes. Each volume is replicated (default: 3 replicas) across different nodes, using the local disk on each agent (128 GB each).
|
||||
Longhorn gives distributed block storage across all 14 agent nodes. Each volume is replicated (default: 3 replicas) across different nodes, using the local disk on each agent (128 GB each), with soft anti-affinity and auto-balance enabled so replicas actively spread out rather than pile up.
|
||||
|
||||
RWO (ReadWriteOnce) covers most services. RWX (ReadWriteMany) is used where multiple pods need access to the same volume. Snapshots and backups are available through the Longhorn UI.
|
||||
RWO (ReadWriteOnce) covers most services. RWX (ReadWriteMany) is used where multiple pods need access to the same volume. Snapshots and backups are available through the Longhorn UI — backups target a Hetzner Storage Box over CIFS, so a full node-loss scenario doesn't also mean a data-loss scenario.
|
||||
|
||||
Control plane nodes are tainted `NoSchedule` — Longhorn manager tolerates this and runs everywhere, but user workloads stay on agent nodes.
|
||||
|
||||
A recurring annoyance: containerd's image layer cache on `/var/lib/rancher/k3s/agent/containerd/` grows unbounded and can eat tens of GB, which Longhorn then reads as node-level disk pressure and stops scheduling. A systemd timer (`node_cleanup` Ansible role) prunes it daily with `crictl rmi --prune` and vacuums journal logs down to 500 MB, staggered by a random delay so it doesn't hit every node at once.
|
||||
|
||||
---
|
||||
|
||||
## CloudNativePG
|
||||
|
||||
CloudNativePG manages HA PostgreSQL clusters as Kubernetes resources. Immich uses it for its primary database (photos, albums, users, ML embeddings). CNPG handles streaming replication, failover, and scheduled backups, with data stored on Longhorn PVCs.
|
||||
CloudNativePG manages HA PostgreSQL clusters as Kubernetes resources — one `Cluster` per app (Immich, Vaultwarden, N8n, Paperless, LiteLLM), each backed by Longhorn PVCs. Vaultwarden runs a 3-instance HA cluster with automatic failover; the rest run single-instance. CNPG handles streaming replication, failover, and scheduled backups via Barman Cloud to S3-compatible storage, with WAL compression and point-in-time recovery.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user