docs(runbook): add arr-stack downloads cleanup investigation and scripts
~16T freed on aya01 (92% → 57% mergerfs pool). Documents root cause (no hardlinks across mergerfs due to cross-device mounts), cleanup passes via Sonarr/Radarr API verification, and pending decisions (Bleach remux, 111 skipped Sonarr entries).
This commit is contained in:
Tuan-Dat Tran
146
docs/runbooks/arr-cleanup/findings.md
Normal file
146
docs/runbooks/arr-cleanup/findings.md
Normal file
@@ -0,0 +1,146 @@
|
||||
# arr-stack Downloads Cleanup — Investigation Findings
|
||||
|
||||
## Storage Layout (aya01)
|
||||
|
||||
| Device | FS | Size | Used | Mount |
|
||||
|--------|----|------|------|-------|
|
||||
| `/dev/sdc3` | btrfs | 1.9T | 177G (10%) | `/` (system) |
|
||||
| `/dev/sda1` | btrfs `proxmox` | 2.8T | 1.3T (48%) | `/opt` |
|
||||
| `/dev/sdd1` | ext4 | 17T | 15T (92%) | `/mnt/hdd0` |
|
||||
| `/dev/sde1` | ext4 | 17T | 15T (92%) | `/mnt/hdd2` |
|
||||
| `/dev/sdf1` | ext4 | 17T | 15T (92%) | `/mnt/hdd1` |
|
||||
| `mergerfs` | fuse | 49T | 43T (92%) | `/media` |
|
||||
|
||||
`/media` is a mergerfs union of hdd0 + hdd1 + hdd2. All three HDDs were at ~92% capacity before cleanup.
|
||||
|
||||
**After cleanup (2026-04-23):**
|
||||
|
||||
| Device | Used | Avail | Use% |
|
||||
|--------|------|-------|------|
|
||||
| `/dev/sdd1` (hdd0) | 9.4T | 6.2T | 61% |
|
||||
| `/dev/sdf1` (hdd1) | 9.3T | 6.3T | 60% |
|
||||
| `/dev/sde1` (hdd2) | 7.8T | 7.8T | 51% |
|
||||
| `mergerfs /media` | 27T | 21T | 57% |
|
||||
|
||||
**~16T freed total** (92% → 57% on the mergerfs pool).
|
||||
|
||||
## /media Breakdown (before cleanup)
|
||||
|
||||
| Directory | Size |
|
||||
|-----------|------|
|
||||
| `downloads` | **22T** |
|
||||
| `series` | 16T |
|
||||
| `movies` | 5T |
|
||||
|
||||
## Root Cause: No Hardlinks → All Imports Are Copies
|
||||
|
||||
Zero hardlinked files exist anywhere across all three HDDs. Confirmed by inspecting the Kubernetes manifests in `argocd-homelab/services/arr-stack/` and by inode comparison of 1365 download/media file pairs (0 shared inodes found).
|
||||
|
||||
**All three services mount the mergerfs `/media/` path via NFS:**
|
||||
|
||||
```
|
||||
sonarr: NFS 192.168.20.12:/media/downloads → /downloads
|
||||
NFS 192.168.20.12:/media/series → /tv
|
||||
radarr: NFS 192.168.20.12:/media/downloads → /downloads
|
||||
NFS 192.168.20.12:/media/movies → /movies
|
||||
qbit: NFS 192.168.20.12:/media/downloads → /downloads
|
||||
```
|
||||
|
||||
mergerfs does not support hardlinks across underlying filesystems. When qBit downloads to `/media/downloads/sonarr/` (lands on e.g. hdd1) and Sonarr imports to `/media/series/` (lands on e.g. hdd0), the hardlink attempt crosses a physical disk boundary → falls back to copy. Every import doubles the data.
|
||||
|
||||
## Cleanup Performed (2026-04-23)
|
||||
|
||||
Three passes using the scripts in this directory:
|
||||
|
||||
### Pass 1 — Orphans (not in Sonarr at all)
|
||||
Script: `cleanup-orphans.py`
|
||||
|
||||
Deleted 49 entries totalling **461.6G** — downloads with no matching Sonarr series and no series directory on disk. Includes Game of Thrones (all 8 seasons), Sex Education (all 4 seasons), Love Death & Robots (multiple duplicate copies), and various anime episode files.
|
||||
|
||||
111 entries were SKIPPED (series dir found on disk, needs manual review) — includes Bleach, House, Lucifer, You, Detective Conan episodes, What If, etc. See cleanup.log for full list.
|
||||
|
||||
### Pass 2 — Confirmed-imported Sonarr downloads
|
||||
Script: `cleanup.py --arr sonarr`
|
||||
|
||||
Deleted **1106 entries**, 0 failed. These were downloads where Sonarr confirmed `episodeFileCount > 0` AND the series directory was verified to exist on disk at the time of `verify.py` run.
|
||||
|
||||
### Pass 3 — Confirmed-imported Radarr downloads
|
||||
Script: `cleanup.py --arr radarr`
|
||||
|
||||
Deleted **259 entries**, 0 failed. These were downloads where Radarr confirmed `hasFile=True` AND the file/directory path was verified to exist on disk.
|
||||
|
||||
### Totals
|
||||
| Pass | Entries | Space |
|
||||
|------|---------|-------|
|
||||
| Orphans (cleanup-orphans.py) | 49 | ~461G |
|
||||
| Sonarr imports (cleanup.py) | 1106 | ~12T (estimated) |
|
||||
| Radarr imports (cleanup.py) | 259 | ~4T (estimated) |
|
||||
| **Total** | **1414** | **~16T freed** |
|
||||
|
||||
All deletions logged to `cleanup.log` with UTC timestamp, size, title, path, outcome.
|
||||
|
||||
## Verification Results (via API + disk path check)
|
||||
|
||||
API keys stored in `../sonarr.api.env` and `../radarr.api.env`.
|
||||
Access via `kubectl -n arr-stack port-forward svc/sonarr 8989:8989` and `svc/radarr 7878:7878`.
|
||||
|
||||
Container path mappings:
|
||||
- Sonarr `/tv/` → `/media/series/`
|
||||
- Radarr `/movies/` → `/media/movies/`
|
||||
|
||||
| | Safe to delete | Orphans (not in arr) | Keep |
|
||||
|---|---|---|---|
|
||||
| **Radarr** (289 items, ~5.2T) | **265** | 25 | 0 |
|
||||
| **Sonarr** (1439 items, ~17T) | **1106** | 333 | 0 |
|
||||
|
||||
"Safe to delete" = API confirms `hasFile=True` (Radarr) or `episodeFileCount > 0` (Sonarr), AND the reported file/directory path was verified to exist on disk via SSH.
|
||||
|
||||
### Radarr Orphans (25) — not matched in Radarr, not deleted
|
||||
- Constantine (2005)
|
||||
- Cowboy Bebop: Knockin' on Heaven's Door (2001)
|
||||
- Les Misérables (2012)
|
||||
- Pokémon Detective Pikachu (2019)
|
||||
- Code Geass: Fukkatsu no Lelouch (2019)
|
||||
- Eiga Go-Toubun no Hanayome (2022)
|
||||
- Gisaengchung / Parasite (Korean title — matching failure)
|
||||
- Dune: Part One (2021) — matching failure, is in Radarr
|
||||
- Harry Potter (older/duplicate copies — matching failure)
|
||||
- Porco Rosso / Kurenai no buta — matching failure
|
||||
- Castle in the Sky / Laputa — matching failure
|
||||
- Steins;Gate: The Movie — matching failure
|
||||
- Project Silence / Talchul — matching failure
|
||||
- Digimon: Frontier & Savers films
|
||||
- One Piece films (several)
|
||||
- Paripi Koumei movie
|
||||
- Fantastic Four (2025) extra copies (3)
|
||||
- JJK DCP trailer file
|
||||
|
||||
### 6 Radarr "path mismatch" entries (all confirmed safe, deleted)
|
||||
Flagged due to path comparison artifacts, manually verified on disk:
|
||||
- Star Wars Episode IV/V/VI/IX — each is a separate Radarr entry; all directories exist
|
||||
- WALL·E — `·` middle-dot character caused comparison failure; file exists
|
||||
|
||||
## Pending Decisions
|
||||
|
||||
### Bleach USBD Remux TL (1.8T)
|
||||
`/media/downloads/sonarr/Bleach USBD Remux TL` — full lossless Bluray remux S00–S16 (-ZR- group).
|
||||
Currently in SKIPPED (series dir `/media/series/Bleach (2004) {imdb-tt0434665}/` exists, 310G imported).
|
||||
Most seasons were imported from x265 Bluray packs (-iVy group) rather than from this remux.
|
||||
S11 has no imported content at all. S13, S14 partially imported.
|
||||
Decision: keep (for quality imports once disk freed) or delete (free 1.8T, accept x265 quality).
|
||||
See memory file for full per-season breakdown.
|
||||
|
||||
### SKIPPED downloads (111 Sonarr entries)
|
||||
Downloads where the series directory exists on disk but the series is not currently in Sonarr.
|
||||
Likely removed series (House, Lucifer, You, Black Clover, etc.) or ongoing shows with stale episodes.
|
||||
These need manual review — series may have been intentionally removed from Sonarr.
|
||||
|
||||
## Fix (not applied — future reference)
|
||||
Mount per-HDD NFS paths instead of the mergerfs path, so downloads and media share the same physical filesystem and hardlinks work:
|
||||
```yaml
|
||||
# sonarr/radarr/qtun deployments — change NFS path from:
|
||||
path: /media/downloads → path: /mnt/hdd0/downloads
|
||||
path: /media/series → path: /mnt/hdd0/series
|
||||
path: /media/movies → path: /mnt/hdd0/movies
|
||||
```
|
||||
Jellyfin/Plex continue reading from `/media/` (mergerfs). New imports hardlink within hdd0.
|
||||
Reference in New Issue
Block a user