# Observability

Metrics and logs both surface in Grafana, keeping dashboards and log queries in one place.

---

## Metrics

kube-prometheus-stack runs in the `prometheus` namespace (ArgoCD-managed). Prometheus scrapes all nodes, pods, and control plane components, on a 60s scrape/eval interval rather than the more common 30s — halves CPU and WAL overhead at the cost of resolution. Grafana has dashboards for cluster overview, node resources, Longhorn, ArgoCD, and Traefik.

Because k3s embeds the scheduler and controller-manager (no separate scrape endpoint) and doesn't run kube-proxy, the stock `KubeControllerManagerDown` / `KubeSchedulerDown` / `KubeProxyDown` alerts are permanent false positives on this cluster and are disabled. Cardinality also got out of hand at one point — ~470k active series, largely duplicate apiserver/etcd metrics scraped twice (once directly, once via kubelet) plus high-cardinality histograms with no alerts attached. Relabeling rules drop those now; live series count sits around 350-400k.

Node Exporter is deployed via Ansible on every VM including `docker-host11` and the edge VPS, so coverage isn't limited to what's inside Kubernetes.

Goldilocks and VPA run alongside and analyze actual resource usage to suggest better request/limit values.

Alertmanager routes alerts to Ntfy via a custom webhook bridge.

---

## Logs

Loki runs in the `loki` namespace (single-node, filesystem storage, 14-day retention). Promtail runs as a DaemonSet and ships pod logs from every cluster node to Loki. Logs are queryable in Grafana alongside metrics.

An Elastic stack (Elasticsearch + Kibana via ECK) lived here for a while — manifests are still in the repo — but it was disabled in favor of running agentmemory instead, so Loki is currently the only log path.

---

## Alerting flow

```
Prometheus Alertmanager ──► Ntfy (push notification)
```