From 1c2eb6688be52c090f86e20bdac2edadd5932b40 Mon Sep 17 00:00:00 2001 From: Tuan-Dat Tran Date: Sun, 21 Jun 2026 21:39:40 +0200 Subject: [PATCH] docs: add Loki/Promtail, LiteLLM, agentmemory, Raspberry Pi; drop Elastic MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Replace Elastic Stack with Loki + Promtail throughout - Add LiteLLM (LLM proxy), agentmemory, athome (personal homepage) to applications - Add Raspberry Pi (naruto) to hardware table — Zigbee2MQTT + Mosquitto via Ansible - Update CloudNativePG note to include LiteLLM as consumer - Add raspberry_pi role to repo layout --- README.md | 17 +++++++++++------ docs/observability.md | 22 +++------------------- 2 files changed, 14 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index d41250e..d9ac55e 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ graph TB Longhorn ArgoCD Prometheus - ECK["Elastic Stack"] + Loki Istio["Istio Ambient"] end subgraph apps["Applications"] @@ -83,8 +83,9 @@ graph TB | VM | `docker-host11` | Docker host w/ GPU passthrough | 2 vCPU · 4 GB RAM · 192 GB · Intel QuickSync | | VM | `docker-lb` | Caddy reverse proxy (LAN only) | 1 vCPU · 2 GB RAM | | VPS | `mii` | Edge node (Netcup) | WireGuard · Traefik · Pangolin | +| SBC | `naruto` | Raspberry Pi | Zigbee2MQTT · Mosquitto MQTT broker | -All VMs run Debian 12 on `virtio` network bridges, provisioned from cloud-init templates via Terraform + Ansible. +All VMs run Debian 12 on `virtio` network bridges, provisioned from cloud-init templates via Terraform + Ansible. The Raspberry Pi runs Docker Compose, managed by the same Ansible setup. --- @@ -100,9 +101,9 @@ All VMs run Debian 12 on `virtio` network bridges, provisioned from cloud-init t | **Cert-Manager** | Helm (ArgoCD) | Automated TLS via Let's Encrypt DNS-01 (Cloudflare API) | | **Sealed Secrets** | Helm (ArgoCD) | Encrypts secrets for safe storage in Git | | **Longhorn** | Helm (ArgoCD) | Distributed block storage (RWO + RWX) across all 14 agents | -| **CloudNativePG** | Operator (ArgoCD) | HA PostgreSQL — used by Immich | -| **Elastic Stack (ECK)** | Operator (ArgoCD) | Elasticsearch + Kibana + Fleet + Elastic Agents for observability | +| **CloudNativePG** | Operator (ArgoCD) | HA PostgreSQL — used by Immich and LiteLLM | | **Kube-Prometheus-Stack** | Helm (ArgoCD) | Prometheus + Grafana monitoring | +| **Loki + Promtail** | Helm (ArgoCD) | Log aggregation — Promtail DaemonSet ships pod logs to Loki | | **Goldilocks + VPA** | Helm (ArgoCD) | Resource usage analysis and request/limit rightsizing | | **Istio (Ambient)** | Helm (ArgoCD) | Service mesh — ztunnel DaemonSet on all nodes (L4); no Waypoint proxies yet | | **K3s Upgrade Controller** | Operator (ArgoCD) | Automated rolling K3s version upgrades | @@ -130,6 +131,9 @@ All VMs run Debian 12 on `virtio` network bridges, provisioned from cloud-init t | **Arr Stack** | Media automation suite | Prowlarr · Sonarr · Radarr · Unpackarr | | **Download clients** | VPN-isolated download clients (×2) | Gluetun sidecar | | **Jellyfin** | Media server with hardware transcoding | Docker · Intel QuickSync | +| **LiteLLM** | OpenAI-compatible LLM proxy — routes to Gemini, DeepSeek, and other providers | CloudNativePG | +| **agentmemory** | Persistent memory server for AI agents | Longhorn PVC · private Gitea registry | +| **athome** | Personal homepage | – | --- @@ -143,7 +147,7 @@ No ports are open on the home router. Internal load balancing goes through Metal Longhorn handles block storage by replicating volumes across all 14 agent nodes. The media library lives on a dedicated NFS host instead — latency matters when Jellyfin is reading large video files, and NFS is simpler for that. -Metrics go to Prometheus + Grafana. Logs and fleet management go to Elastic Stack via the ECK operator, with Elastic Agents running as a DaemonSet so every node is covered. +Metrics go to Prometheus + Grafana. Logs go to Loki, also surfaced in Grafana — Promtail runs as a DaemonSet and ships pod logs from every node. All VMs are provisioned with Terraform and configured by Ansible. Rebuilding from scratch doesn't require remembering anything. @@ -160,7 +164,8 @@ ansible-homelab/ │ ├── kube_vip/ # kube-vip DaemonSet + TLS SAN config │ ├── docker_host/ # Docker + Intel QuickSync GPU passthrough │ ├── proxmox/ # Proxmox node setup -│ └── edge_vps/ # VPS: WireGuard, Traefik, Pangolin, Elastic Agent +│ ├── edge_vps/ # VPS: WireGuard, Traefik, Pangolin, Elastic Agent +│ └── raspberry_pi/ # Raspberry Pi: Zigbee2MQTT + Mosquitto via Docker Compose └── playbooks/ argocd-homelab/ diff --git a/docs/observability.md b/docs/observability.md index 542b65f..dc60cbd 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -1,6 +1,6 @@ # Observability -Two parallel stacks: Prometheus for metrics, Elastic for logs. +Metrics and logs both surface in Grafana, keeping dashboards and log queries in one place. --- @@ -16,21 +16,9 @@ Alertmanager routes alerts to Ntfy via a custom webhook bridge. --- -## Logs and fleet management +## Logs -The ECK operator (Elastic Cloud on Kubernetes) manages the Elastic stack in the `elastic-system` namespace: - -| Component | Purpose | -|-----------|---------| -| Elasticsearch | Log storage and search (single-node, 15 Gi heap) | -| Kibana | Log exploration and dashboards | -| Fleet Server | Manages Elastic Agent enrollment and policies | -| Elastic Agent (DaemonSet) | Ships logs and metrics from every cluster node | -| Elastic Agent (standalone) | Runs on docker-host11 and the edge VPS | - -The DaemonSet tolerates the control-plane `NoSchedule` taint so server nodes are covered too. - -Elastic alert rules are bridged to Ntfy via `elastic-ntfy-bridge`, a small CronJob that polls the Elasticsearch alerts API and forwards new alerts as push notifications. +Loki runs in the `loki` namespace (single-node, filesystem storage). Promtail runs as a DaemonSet and ships pod logs from every cluster node to Loki. Logs are queryable in Grafana alongside metrics. --- @@ -38,8 +26,4 @@ Elastic alert rules are bridged to Ntfy via `elastic-ntfy-bridge`, a small CronJ ``` Prometheus Alertmanager ──► Ntfy (push notification) - ▲ -Elasticsearch alert rule ──► elastic-ntfy-bridge CronJob ─┘ ``` - -Both sources land in the same Ntfy topic.