6936c3f82a
Corrected stale claims found by cross-referencing git history and live kubectl/SSH state: - Drop Terraform provisioning claim (VMs are created by the ansible-homelab proxmox role via community.proxmox.proxmox_kvm; terraform/ is empty) - Bump k3s badge to the version actually running (v1.36) - Drop Elastic Stack as a live logging path - it was disabled in favor of agentmemory and has no namespace or ArgoCD Application anymore - Drop zeroclaw from the app list - manifests exist but it was never wired into ArgoCD's app registry, so it has never actually run Added detail confirmed against live state: Longhorn's Hetzner backup target, node_cleanup containerd-pruning timer, MikroTik static DHCP leases, Prometheus cardinality/false-positive-alert tuning, actual CNPG cluster inventory, corrected MetalLB pool size, aya01's real NFS capacity, k3s-server11's post-rebuild spec drift, and the naruto01 etcd quorum concentration.
178 lines
9.1 KiB
Markdown
178 lines
9.1 KiB
Markdown
# Homelab
|
||
|
||
17-node Kubernetes cluster on four bare-metal Proxmox hosts, provisioned and configured with Ansible, managed through ArgoCD GitOps. Runs my home automation, media stack, photo backup, documents, and a few side projects.
|
||
|
||
|
||
|
||
|
||
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
```mermaid
|
||
graph TB
|
||
subgraph ext[" External"]
|
||
CF["Cloudflare CDN"]
|
||
Admin["Remote Admin"]
|
||
end
|
||
|
||
subgraph vps["Edge VPS"]
|
||
WG["WireGuard VPN Gateway"]
|
||
TraefikVPS["Traefik"]
|
||
Pangolin["Pangolin Tunnel Server"]
|
||
end
|
||
|
||
subgraph proxmox["Proxmox Cluster — 4 physical nodes"]
|
||
subgraph cp["Control Plane x3 — HA etcd + kube-vip"]
|
||
S["k3s-server"]
|
||
end
|
||
subgraph workers["Worker Nodes x14"]
|
||
W["k3s-agent"]
|
||
end
|
||
DH["docker-host"]
|
||
NFS["NFS Server — dedicated storage node"]
|
||
end
|
||
|
||
subgraph k8s["Kubernetes"]
|
||
subgraph platform["Platform"]
|
||
direction LR
|
||
MetalLB
|
||
Traefik
|
||
Longhorn
|
||
ArgoCD
|
||
Prometheus
|
||
Loki
|
||
Istio["Istio Ambient"]
|
||
end
|
||
subgraph apps["Applications"]
|
||
direction LR
|
||
Immich
|
||
VW["Vaultwarden"]
|
||
HA["Home Assistant"]
|
||
Media["Arr Stack + Jellyfin"]
|
||
Other["Paperless, N8n, Ntfy ..."]
|
||
end
|
||
end
|
||
|
||
Admin -->|WireGuard VPN| WG
|
||
WG -->|tunnel| k8s
|
||
CF -->|Cloudflare tunnel| k8s
|
||
TraefikVPS --> Pangolin
|
||
Pangolin -->|Newt client| k8s
|
||
cp --- workers
|
||
workers --- Longhorn
|
||
NFS -->|NFS mount| Media
|
||
DH -->|Docker| Media
|
||
```
|
||
|
||
---
|
||
|
||
## Design notes
|
||
|
||
Everything goes through Git. ArgoCD owns the cluster state; nothing gets `kubectl apply`'d directly. ArgoCD Image Updater handles the image update loop: when a new tag appears in the registry, it commits the change back to Git and ArgoCD picks it up from there.
|
||
|
||
Secrets are committed to Git too, encrypted via Sealed Secrets. Only the in-cluster controller holds the decryption key.
|
||
|
||
No ports are open on the home router. Internal load balancing goes through MetalLB + Traefik. External access uses Cloudflare tunnels or a WireGuard VPN routed through the edge VPS.
|
||
|
||
Longhorn handles block storage by replicating volumes across all 14 agent nodes, with backup snapshots shipped off-site to a Hetzner Storage Box over CIFS. The media library lives on a dedicated NFS host instead — latency matters when Jellyfin is reading large video files, and NFS is simpler for that.
|
||
|
||
Metrics go to Prometheus + Grafana. Logs go to Loki, also surfaced in Grafana — Promtail runs as a DaemonSet and ships pod logs from every node.
|
||
|
||
Not every HA guarantee holds up under scrutiny — two of the three etcd control-plane VMs currently run on the same physical Proxmox host (`naruto01`), a single point of failure for cluster quorum that surfaced during an incident and is still on the list to rebalance.
|
||
|
||
All VMs are provisioned and configured by Ansible — the `proxmox` role calls the Proxmox API directly (`community.proxmox.proxmox_kvm`) to clone, resize, and boot VMs from cloud-init templates, then the rest of the playbooks configure them. Rebuilding from scratch doesn't require remembering anything.
|
||
|
||
---
|
||
|
||
## Hardware
|
||
|
||
| Layer | Host | Role | Resources |
|
||
|-------|------|------|-----------|
|
||
| Physical | `aya01` | Proxmox node + NFS server (49 TB share, ~60% used) | Dedicated storage — no VMs |
|
||
| Physical | `lulu` | Proxmox node | k3s agents |
|
||
| Physical | `naruto01` | Proxmox node | k3s server + agents |
|
||
| Physical | `mii01` | Proxmox node | k3s server + agents + docker host |
|
||
| VM | `k3s-server-{10,11,12}` | K3s control plane (HA etcd + kube-vip VIP) | 2 vCPU · 4 GB RAM · 64 GB (`k3s-server11` runs 4 vCPU · 6 GB after a rebuild — not yet reconciled with the other two) |
|
||
| VM | `k3s-agent-{10…23}` | K3s worker nodes ×14 | 2 vCPU · 4 GB RAM · 128 GB |
|
||
| VM | `docker-host11` | Docker host (AMD Ryzen 7 5700U — CPU-only, no iGPU passthrough for Jellyfin) | 2 vCPU · 4 GB RAM · 192 GB |
|
||
| VM | `docker-lb` | Caddy reverse proxy (LAN only) | 1 vCPU · 2 GB RAM |
|
||
| VPS | `mii` | Edge node (Netcup) | WireGuard · Traefik · Pangolin |
|
||
| SBC | `naruto` | Raspberry Pi | Zigbee2MQTT · Mosquitto MQTT broker |
|
||
|
||
All VMs run Debian 12 on `virtio` network bridges, provisioned from cloud-init templates via Ansible. The Raspberry Pi runs Docker Compose, managed by the same Ansible setup. A MikroTik router is also under Ansible's control — it backs up its RouterOS config to Git on every run and gets static DHCP leases pinned for the k3s nodes, so a lease renewal can't silently move a node's IP.
|
||
|
||
---
|
||
|
||
## Platform Stack
|
||
|
||
| Component | How deployed | Purpose |
|
||
|-----------|-------------|---------|
|
||
| **ArgoCD** | Helm (App-of-Apps) | GitOps CD — all cluster state driven from Git |
|
||
| **ArgoCD Image Updater** | Helm | Watches registries, commits updated image tags back to Git |
|
||
| **kube-vip** | DaemonSet on control plane | HA VIP for the K8s API server |
|
||
| **Traefik** | k3s built-in | Ingress controller, fronted by MetalLB |
|
||
| **MetalLB** | Helm (ArgoCD) | Bare-metal load balancer, assigns IPs from reserved pool |
|
||
| **Cert-Manager** | Helm (ArgoCD) | Automated TLS via Let's Encrypt DNS-01 (Cloudflare API) |
|
||
| **Sealed Secrets** | Helm (ArgoCD) | Encrypts secrets for safe storage in Git |
|
||
| **Longhorn** | Helm (ArgoCD) | Distributed block storage (RWO + RWX) across all 14 agents |
|
||
| **CloudNativePG** | Operator (ArgoCD) | HA PostgreSQL — one cluster per app (Immich, Vaultwarden, N8n, Paperless, LiteLLM); Vaultwarden's is 3-instance HA, the rest single-instance |
|
||
| **Kube-Prometheus-Stack** | Helm (ArgoCD) | Prometheus + Grafana monitoring |
|
||
| **Loki + Promtail** | Helm (ArgoCD) | Log aggregation — Promtail DaemonSet ships pod logs to Loki (14-day retention) |
|
||
| **Goldilocks + VPA** | Helm (ArgoCD) | Resource usage analysis and request/limit rightsizing |
|
||
| **Istio (Ambient)** | Helm (ArgoCD) | Service mesh — ztunnel DaemonSet on all nodes (L4); no Waypoint proxies yet |
|
||
| **K3s Upgrade Controller** | Operator (ArgoCD) | Automated rolling K3s version upgrades |
|
||
| **mii-wireguard** | Manifest (ArgoCD) | WireGuard pod — connects cluster to edge VPS, masquerades service CIDR |
|
||
| **Newt** | Deployment (ArgoCD) | Pangolin tunnel client for VPS-proxied services |
|
||
| **Cloudflared** | Deployment ×2 (ArgoCD) | Cloudflare tunnel — exposes selected services to the internet |
|
||
|
||
---
|
||
|
||
## Applications
|
||
|
||
| Service | Description | Notable tech |
|
||
|---------|-------------|--------------|
|
||
| **Immich** | Photo & video backup (self-hosted Google Photos) | CloudNativePG · Redis · ML pod |
|
||
| **Vaultwarden** | Bitwarden-compatible password manager | – |
|
||
| **Paperless-ngx** | Document management + OCR | – |
|
||
| **Home Assistant** | Home automation hub | – |
|
||
| **N8n** | Workflow automation | – |
|
||
| **Ntfy** | Self-hosted push notifications | – |
|
||
| **Stirling PDF** | PDF tools | – |
|
||
| **Karakeep** | Bookmark manager | – |
|
||
| **Gitea** | Self-hosted Git (source of truth for ArgoCD) | Kubernetes · Longhorn PVC · SSH via MetalLB · Pangolin tunnel (git.tudattr.dev) |
|
||
| **Gitea Runner** | CI/CD runner, ×2 (one for the in-cluster Gitea, one for the external instance) | Docker-in-Docker, rootless, shared RWX Longhorn volume |
|
||
| **Arr Stack** | Media automation suite | Prowlarr · Sonarr · Radarr · Recyclarr · Unpackarr |
|
||
| **Download clients** | VPN-isolated + clearnet qBittorrent pair, run side by side | Gluetun sidecar (ProtonVPN) on the VPN-isolated one |
|
||
| **Jellyfin** | Media server, CPU-only transcoding | Docker |
|
||
| **LiteLLM** | OpenAI-compatible LLM proxy — routes to Gemini 2.5 Flash and DeepSeek | CloudNativePG |
|
||
| **agentmemory** | Persistent memory server for AI agents — consolidation, graph extraction, reflection | Longhorn PVC · private Gitea registry · local embeddings |
|
||
| **athome** | Personal homepage | – |
|
||
|
||
---
|
||
|
||
## Repo layout
|
||
|
||
```
|
||
ansible-homelab/
|
||
├── roles/
|
||
│ ├── common/ # base OS config, SSH hardening, node-exporter
|
||
│ ├── k3s_server/ # control plane install + NoSchedule taint
|
||
│ ├── k3s_agent/ # worker node install
|
||
│ ├── kube_vip/ # kube-vip DaemonSet + TLS SAN config
|
||
│ ├── docker_host/ # Docker + Compose services
|
||
│ ├── proxmox/ # Proxmox node setup
|
||
│ ├── edge_vps/ # VPS: WireGuard, Traefik, Pangolin
|
||
│ ├── raspberry_pi/ # Raspberry Pi: Zigbee2MQTT + Mosquitto via Docker Compose
|
||
│ ├── mikrotik/ # router config backup + static DHCP leases
|
||
│ └── node_cleanup/ # systemd timer: prunes containerd images, vacuums journal logs
|
||
└── playbooks/
|
||
|
||
argocd-homelab/
|
||
├── infrastructure/ # MetalLB, Longhorn, Cert-Manager, Istio, Prometheus, ...
|
||
├── services/ # Immich, Vaultwarden, arr-stack, Home Assistant, ...
|
||
└── cluster-apps/ # ArgoCD App-of-Apps root + ApplicationSets
|
||
```
|