6936c3f82a
Corrected stale claims found by cross-referencing git history and live kubectl/SSH state: - Drop Terraform provisioning claim (VMs are created by the ansible-homelab proxmox role via community.proxmox.proxmox_kvm; terraform/ is empty) - Bump k3s badge to the version actually running (v1.36) - Drop Elastic Stack as a live logging path - it was disabled in favor of agentmemory and has no namespace or ArgoCD Application anymore - Drop zeroclaw from the app list - manifests exist but it was never wired into ArgoCD's app registry, so it has never actually run Added detail confirmed against live state: Longhorn's Hetzner backup target, node_cleanup containerd-pruning timer, MikroTik static DHCP leases, Prometheus cardinality/false-positive-alert tuning, actual CNPG cluster inventory, corrected MetalLB pool size, aya01's real NFS capacity, k3s-server11's post-rebuild spec drift, and the naruto01 etcd quorum concentration.
34 lines
1.9 KiB
Markdown
34 lines
1.9 KiB
Markdown
# Observability
|
|
|
|
Metrics and logs both surface in Grafana, keeping dashboards and log queries in one place.
|
|
|
|
---
|
|
|
|
## Metrics
|
|
|
|
kube-prometheus-stack runs in the `prometheus` namespace (ArgoCD-managed). Prometheus scrapes all nodes, pods, and control plane components, on a 60s scrape/eval interval rather than the more common 30s — halves CPU and WAL overhead at the cost of resolution. Grafana has dashboards for cluster overview, node resources, Longhorn, ArgoCD, and Traefik.
|
|
|
|
Because k3s embeds the scheduler and controller-manager (no separate scrape endpoint) and doesn't run kube-proxy, the stock `KubeControllerManagerDown` / `KubeSchedulerDown` / `KubeProxyDown` alerts are permanent false positives on this cluster and are disabled. Cardinality also got out of hand at one point — ~470k active series, largely duplicate apiserver/etcd metrics scraped twice (once directly, once via kubelet) plus high-cardinality histograms with no alerts attached. Relabeling rules drop those now; live series count sits around 350-400k.
|
|
|
|
Node Exporter is deployed via Ansible on every VM including `docker-host11` and the edge VPS, so coverage isn't limited to what's inside Kubernetes.
|
|
|
|
Goldilocks and VPA run alongside and analyze actual resource usage to suggest better request/limit values.
|
|
|
|
Alertmanager routes alerts to Ntfy via a custom webhook bridge.
|
|
|
|
---
|
|
|
|
## Logs
|
|
|
|
Loki runs in the `loki` namespace (single-node, filesystem storage, 14-day retention). Promtail runs as a DaemonSet and ships pod logs from every cluster node to Loki. Logs are queryable in Grafana alongside metrics.
|
|
|
|
An Elastic stack (Elasticsearch + Kibana via ECK) lived here for a while — manifests are still in the repo — but it was disabled in favor of running agentmemory instead, so Loki is currently the only log path.
|
|
|
|
---
|
|
|
|
## Alerting flow
|
|
|
|
```
|
|
Prometheus Alertmanager ──► Ntfy (push notification)
|
|
```
|