# Consolidating 14 Worker Nodes Into 7 *August 2026* The cluster ran 14 K3s agents at 2 vCPU / 4 GiB / 128 GB each. That sounded reasonable when it was built and turned out to be wrong in three separate ways, so over three days I merged them down to 7 nodes at 4 vCPU / 8 GiB / 256 GB. The migration itself was mechanical. Almost everything interesting that happened was something I had not planned for — including a fault that had been silently degrading the cluster for nine days before I noticed it. --- ## Why bother Fourteen small nodes cost more than seven big ones, in ways that aren't obvious until you're running them. **Per-node overhead is fixed.** Every agent runs a kubelet, containerd, a Longhorn instance-manager, an engine-image pod, promtail, node-exporter, a MetalLB speaker, Istio's CNI and ztunnel. On a 4 GiB node that's a meaningful fraction of the box before a single workload lands. Doubling the node size doesn't double that overhead — it halves it per unit of capacity. **128 GB disks were too small.** Longhorn replicates every volume three ways, and with 14 candidate nodes the replicas scattered. Nodes kept hitting disk pressure, at which point Longhorn stops scheduling to them, which concentrates replicas further on whatever's left. **RAM was oversubscribed at the host level.** Two of the three Proxmox hosts had more RAM allocated to VMs than they physically had, papered over with KSM deduplication. Fewer, larger VMs don't fix that arithmetic, but they make it easier to reason about. The plan was seven destroy/grow pairs — kill one agent, grow another into the space — run strictly one at a time, so the cluster never lost more than one node's worth of capacity at once. --- ## The part that went wrong Pair one was budgeted at 1.5 to 3 hours. It took 4.5. The symptom: Longhorn replica rebuilds would climb steadily, then die with `connection reset by peer` and restart from zero. Small volumes finished because they slipped in between failures. Large ones never finished at all — a 50 GiB volume looped for over two hours without completing once. I worked through the obvious suspects and ruled out every one: | Suspect | Verdict | |---|---| | Overlay network MTU | Uniform 1500/1450; fragmentation behaved correctly | | conntrack exhaustion | 7k of 131k | | OOM kills | None anywhere | | Node memory pressure | Clear | | Host saturation | Load 2.67, 0% iowait | | NIC error counters | Zero | | Snapshot auto-cleanup | Disabling it made failures *worse* — 30 to 36 | Meanwhile I raised Longhorn's rebuild concurrency to unblock a queue. Degraded volumes went from 2 to 9. Put it back to 1 and the cluster recovered — a useful reminder that concurrency on a saturated link makes everything slower without moving more bytes. --- ## The actual cause One host was negotiating gigabit and then falling back: ``` Generic FE-GE Realtek PHY: Downshift occurred from negotiated speed 1Gbps to actual speed 100Mbps, check cabling! ``` A **faulty Ethernet cable.** The NIC was gigabit-capable, the switch advertised gigabit, and the PHY was failing 1000BASE-T signal training and quietly dropping to 100 Mbit — where 2-pair signalling still worked fine. Error counters stayed clean, because nothing was erroring. It was negotiating exactly as designed. The host had held 1 Gbps until a link flap nine days earlier. Every boot since had re-downshifted immediately. That single fact explained the entire mystery. A 100 Mbit link ceilings at ~12.5 MB/s; measured rebuild throughput was 8–10 MB/s. The link was saturated and working as well as it could. A 50 GiB volume needed ~95 minutes of *uninterrupted* transfer, and it never got that long a window. I had written a note to myself days earlier saying this host's 100 Mbit was accepted and not worth re-investigating. That note was wrong, and it nearly cost me the diagnosis. **A number you've decided to stop questioning is a good place to look when nothing else fits.** Cable swapped. Link came up at 1 Gbps. Cross-host round-trip went from 10.7 ms to 0.31 ms. --- ## Before and after The clearest measure of what that cable was costing: | | Before cable fix | After | |---|---|---| | Evacuating a 50 GiB volume | Never completed in 2+ hours | **11 minutes** | | Time per pair | 4.5 hours | 20–35 minutes | | Degraded volumes during a pair | Up to 9 | 0 | Same procedure, same commands, same cluster. --- ## Things I learned the hard way **Longhorn evicts at disk level, not node level.** Marking a node as `evictionRequested` does nothing on its own — replicas sit there indefinitely. You have to set the flag on the *disk* inside the node spec too. I watched 25 replicas ignore a node-level eviction for ten minutes, then five clear within 45 seconds of adding the disk patch. **Draining a node does not preserve its replicas.** I'd assumed draining would leave replica data on disk to reattach after a reboot, skipping a rebuild. It doesn't: the drain evicts Longhorn's instance-manager, every replica on that node is marked failed, and they get rebuilt elsewhere regardless. There is no fast path. Budget for the rebuild. **Eviction time isn't governed by the evicting node's link speed.** I deferred the riskiest pair for days because that node sits behind a USB 100 Mbit adapter that can't be upgraded, and I'd budgeted bytes ÷ 12 MB/s. It ran at ~21 MB/s and finished in 13 minutes. Rebuilds read from *whichever* healthy replica is convenient, and that's usually not the copy on the node you're draining. The slow link only binds when it holds the only usable source. **A rebuild can report progress while transferring nothing.** One sat at 99% for fifteen minutes with `isRebuilding: true`, no error, and silent logs — while moving 0.65 MB/s. Another froze at exactly 81% for four hours. Percentage is not liveness; check actual throughput. Deleting the stalled replica restarted it at 120 MB/s. **GitOps will undo your temporary fixes.** The standard trick for a PodDisruptionBudget blocking a drain is to scale a single-replica Deployment to 2. That failed twice, for different reasons: once because the Deployment mounted a ReadWriteOnce volume, so the second pod could never start; once because the app synced with `selfHeal: true` and ArgoCD reverted the scale mid-drain. Cordon the node and delete the pod instead. **Verify the live object, not the file in Git.** I disabled automated sync on the Application that manages node objects, committed it, and moved on. The commit never applied — that Application manages its own manifest, so the only thing that could have applied "disable sync" was the sync I was disabling. It had been dormant for months. The live object still had `selfHeal: true` the whole time. **Deleting a VM isn't done until it's out of the inventory.** Five destroyed VMs sat in the Ansible host vars for days. The module that creates them is create-only — a no-op for VMs that exist, but perfectly happy to create one that doesn't. Any routine playbook run would have resurrected all five. **Watchdogs fight you during maintenance.** Each Proxmox host runs a cron job that force-restarts VMs failing a liveness probe. Since CPU and memory changes only apply on a cold boot, a watchdog restart landing between `qm shutdown` and `qm set` gives you a VM back at its old size while every command reports success. Silent spec drift. Pause them first — and remember to restore them. **Collapsing a failure domain has side effects.** One Proxmox host went from two agents to one, which left its Longhorn zone containing a single node. Zone-based anti-affinity spreads across *zones*, so that one node started attracting a full replica set — one copy of every volume in the cluster. Capacity guardrails stopped it from filling the disk, but it generated constant rebuild traffic across the slowest link in the cluster. **Cloud images panic on first boot after a resize.** Seven resizes, seven kernel panics at ~2 seconds — `Attempted to kill init!` — every one cleared by a plain restart. Filesystems clean, partition UUIDs intact, nothing actually wrong. Also worth knowing: the VM reports `running` while the guest kernel is dead, so the tell is the node going NotReady rather than anything Proxmox says. --- ## Where it landed | Host | Agents | Spec | |---|---|---| | Host A | 3 | 4 vCPU / 8 GiB / 256 GB | | Host B | 3 | 4 vCPU / 8 GiB / 256 GB | | Host C | 1 | 3 vCPU / 6 GiB / 256 GB | Seven agents plus three control-plane nodes. All 35 Longhorn volumes healthy, no data lost, no unplanned downtime beyond the seconds of pod rescheduling each drain caused. The third host gets a smaller agent because it's a 4-thread, 7 GiB machine whose only network interface is a USB adapter capped at 100 Mbit — a genuine silicon limit rather than a bad cable, and the reason it's on the retirement list. --- ## What I'd do differently **Check the physical layer first.** I spent hours on MTU, conntrack, memory pressure and Longhorn tunables before running `ethtool`. The cheapest checks are the ones you skip because they feel too obvious. **Don't write "do not investigate" notes.** Recording a conclusion is useful. Recording it in a way that discourages your future self from re-examining it is not, especially when the conclusion was a guess. **Measure before choosing.** The plan named which node to destroy in each pair. Three times the measured replica distribution said the opposite, and following the plan blindly would have moved far more data than necessary. In one pair both candidate nodes turned out to hold zero replicas — no eviction needed at all. The consolidation was worth doing. The cable was worth finding more.