homelab-docs/README.md

Homelab

17-node Kubernetes cluster on four bare-metal Proxmox hosts, provisioned and configured with Ansible, managed through ArgoCD GitOps. Runs my home automation, media stack, photo backup, documents, and a few side projects.

---

Architecture

graph TB
    subgraph ext[" External"]
        CF["Cloudflare CDN"]
        Admin["Remote Admin"]
    end

    subgraph vps["Edge VPS"]
        WG["WireGuard VPN Gateway"]
        TraefikVPS["Traefik"]
        Pangolin["Pangolin Tunnel Server"]
    end

    subgraph proxmox["Proxmox Cluster — 4 physical nodes"]
        subgraph cp["Control Plane x3  —  HA etcd + kube-vip"]
            S["k3s-server"]
        end
        subgraph workers["Worker Nodes x14"]
            W["k3s-agent"]
        end
        DH["docker-host"]
        NFS["NFS Server — dedicated storage node"]
    end

    subgraph k8s["Kubernetes"]
        subgraph platform["Platform"]
            direction LR
            MetalLB
            Traefik
            Longhorn
            ArgoCD
            Prometheus
            Loki
            Istio["Istio Ambient"]
        end
        subgraph apps["Applications"]
            direction LR
            Immich
            VW["Vaultwarden"]
            HA["Home Assistant"]
            Media["Arr Stack + Jellyfin"]
            Other["Paperless, N8n, Ntfy ..."]
        end
    end

    Admin -->|WireGuard VPN| WG
    WG -->|tunnel| k8s
    CF -->|Cloudflare tunnel| k8s
    TraefikVPS --> Pangolin
    Pangolin -->|Newt client| k8s
    cp --- workers
    workers --- Longhorn
    NFS -->|NFS mount| Media
    DH -->|Docker| Media

---

Design notes

Everything goes through Git. ArgoCD owns the cluster state; nothing gets kubectl apply'd directly. ArgoCD Image Updater handles the image update loop: when a new tag appears in the registry, it commits the change back to Git and ArgoCD picks it up from there.

Secrets are committed to Git too, encrypted via Sealed Secrets. Only the in-cluster controller holds the decryption key.

No ports are open on the home router. Internal load balancing goes through MetalLB + Traefik. External access uses Cloudflare tunnels or a WireGuard VPN routed through the edge VPS.

Longhorn handles block storage by replicating volumes across all 14 agent nodes, with backup snapshots shipped off-site to a Hetzner Storage Box over CIFS. The media library lives on a dedicated NFS host instead — latency matters when Jellyfin is reading large video files, and NFS is simpler for that.

Metrics go to Prometheus + Grafana. Logs go to Loki, also surfaced in Grafana — Promtail runs as a DaemonSet and ships pod logs from every node.

Not every HA guarantee holds up under scrutiny — two of the three etcd control-plane VMs currently run on the same physical Proxmox host (naruto01), a single point of failure for cluster quorum that surfaced during an incident and is still on the list to rebalance.

All VMs are provisioned and configured by Ansible — the proxmox role calls the Proxmox API directly (community.proxmox.proxmox_kvm) to clone, resize, and boot VMs from cloud-init templates, then the rest of the playbooks configure them. Rebuilding from scratch doesn't require remembering anything.

---

Hardware

Layer	Host	Role	Resources
Physical	aya01	Proxmox node + NFS server (49 TB share, ~60% used)	Dedicated storage — no VMs
Physical	lulu	Proxmox node	k3s agents
Physical	naruto01	Proxmox node	k3s server + agents
Physical	mii01	Proxmox node	k3s server + agents + docker host
VM	k3s-server-{10,11,12}	K3s control plane (HA etcd + kube-vip VIP)	2 vCPU · 4 GB RAM · 64 GB (k3s-server11 runs 4 vCPU · 6 GB after a rebuild — not yet reconciled with the other two)
VM	k3s-agent-{10…23}	K3s worker nodes ×14	2 vCPU · 4 GB RAM · 128 GB
VM	docker-host11	Docker host (AMD Ryzen 7 5700U — CPU-only, no iGPU passthrough for Jellyfin)	2 vCPU · 4 GB RAM · 192 GB
VM	docker-lb	Caddy reverse proxy (LAN only)	1 vCPU · 2 GB RAM
VPS	mii	Edge node (Netcup)	WireGuard · Traefik · Pangolin
SBC	naruto	Raspberry Pi	Zigbee2MQTT · Mosquitto MQTT broker

All VMs run Debian 12 on virtio network bridges, provisioned from cloud-init templates via Ansible. The Raspberry Pi runs Docker Compose, managed by the same Ansible setup. A MikroTik router is also under Ansible's control — it backs up its RouterOS config to Git on every run and gets static DHCP leases pinned for the k3s nodes, so a lease renewal can't silently move a node's IP.

---

Platform Stack

Component	How deployed	Purpose
ArgoCD	Helm (App-of-Apps)	GitOps CD — all cluster state driven from Git
ArgoCD Image Updater	Helm	Watches registries, commits updated image tags back to Git
kube-vip	DaemonSet on control plane	HA VIP for the K8s API server
Traefik	k3s built-in	Ingress controller, fronted by MetalLB
MetalLB	Helm (ArgoCD)	Bare-metal load balancer, assigns IPs from reserved pool
Cert-Manager	Helm (ArgoCD)	Automated TLS via Let's Encrypt DNS-01 (Cloudflare API)
Sealed Secrets	Helm (ArgoCD)	Encrypts secrets for safe storage in Git
Longhorn	Helm (ArgoCD)	Distributed block storage (RWO + RWX) across all 14 agents
CloudNativePG	Operator (ArgoCD)	HA PostgreSQL — one cluster per app (Immich, Vaultwarden, N8n, Paperless, LiteLLM); Vaultwarden's is 3-instance HA, the rest single-instance
Kube-Prometheus-Stack	Helm (ArgoCD)	Prometheus + Grafana monitoring
Loki + Promtail	Helm (ArgoCD)	Log aggregation — Promtail DaemonSet ships pod logs to Loki (14-day retention)
Goldilocks + VPA	Helm (ArgoCD)	Resource usage analysis and request/limit rightsizing
Istio (Ambient)	Helm (ArgoCD)	Service mesh — ztunnel DaemonSet on all nodes (L4); no Waypoint proxies yet
K3s Upgrade Controller	Operator (ArgoCD)	Automated rolling K3s version upgrades
mii-wireguard	Manifest (ArgoCD)	WireGuard pod — connects cluster to edge VPS, masquerades service CIDR
Newt	Deployment (ArgoCD)	Pangolin tunnel client for VPS-proxied services
Cloudflared	Deployment ×2 (ArgoCD)	Cloudflare tunnel — exposes selected services to the internet

---

Applications

Service	Description	Notable tech
Immich	Photo & video backup (self-hosted Google Photos)	CloudNativePG · Redis · ML pod
Vaultwarden	Bitwarden-compatible password manager	–
Paperless-ngx	Document management + OCR	–
Home Assistant	Home automation hub	–
N8n	Workflow automation	–
Ntfy	Self-hosted push notifications	–
Stirling PDF	PDF tools	–
Karakeep	Bookmark manager	–
Gitea	Self-hosted Git (source of truth for ArgoCD)	Kubernetes · Longhorn PVC · SSH via MetalLB · Pangolin tunnel (git.tudattr.dev)
Gitea Runner	CI/CD runner, ×2 (one for the in-cluster Gitea, one for the external instance)	Docker-in-Docker, rootless, shared RWX Longhorn volume
Arr Stack	Media automation suite	Prowlarr · Sonarr · Radarr · Recyclarr · Unpackarr
Download clients	VPN-isolated + clearnet qBittorrent pair, run side by side	Gluetun sidecar (ProtonVPN) on the VPN-isolated one
Jellyfin	Media server, CPU-only transcoding	Docker
LiteLLM	OpenAI-compatible LLM proxy — routes to Gemini 2.5 Flash and DeepSeek	CloudNativePG
agentmemory	Persistent memory server for AI agents — consolidation, graph extraction, reflection	Longhorn PVC · private Gitea registry · local embeddings
athome	Personal homepage	–

---

Repo layout

ansible-homelab/
├── roles/
│   ├── common/           # base OS config, SSH hardening, node-exporter
│   ├── k3s_server/       # control plane install + NoSchedule taint
│   ├── k3s_agent/        # worker node install
│   ├── kube_vip/         # kube-vip DaemonSet + TLS SAN config
│   ├── docker_host/      # Docker + Compose services
│   ├── proxmox/          # Proxmox node setup
│   ├── edge_vps/         # VPS: WireGuard, Traefik, Pangolin
│   ├── raspberry_pi/     # Raspberry Pi: Zigbee2MQTT + Mosquitto via Docker Compose
│   ├── mikrotik/         # router config backup + static DHCP leases
│   └── node_cleanup/     # systemd timer: prunes containerd images, vacuums journal logs
└── playbooks/

argocd-homelab/
├── infrastructure/       # MetalLB, Longhorn, Cert-Manager, Istio, Prometheus, ...
├── services/             # Immich, Vaultwarden, arr-stack, Home Assistant, ...
└── cluster-apps/         # ArgoCD App-of-Apps root + ApplicationSets