8239988a70
~16T freed on aya01 (92% → 57% mergerfs pool). Documents root cause (no hardlinks across mergerfs due to cross-device mounts), cleanup passes via Sonarr/Radarr API verification, and pending decisions (Bleach remux, 111 skipped Sonarr entries).
147 lines
6.6 KiB
Markdown
147 lines
6.6 KiB
Markdown
# arr-stack Downloads Cleanup — Investigation Findings
|
||
|
||
## Storage Layout (aya01)
|
||
|
||
| Device | FS | Size | Used | Mount |
|
||
|--------|----|------|------|-------|
|
||
| `/dev/sdc3` | btrfs | 1.9T | 177G (10%) | `/` (system) |
|
||
| `/dev/sda1` | btrfs `proxmox` | 2.8T | 1.3T (48%) | `/opt` |
|
||
| `/dev/sdd1` | ext4 | 17T | 15T (92%) | `/mnt/hdd0` |
|
||
| `/dev/sde1` | ext4 | 17T | 15T (92%) | `/mnt/hdd2` |
|
||
| `/dev/sdf1` | ext4 | 17T | 15T (92%) | `/mnt/hdd1` |
|
||
| `mergerfs` | fuse | 49T | 43T (92%) | `/media` |
|
||
|
||
`/media` is a mergerfs union of hdd0 + hdd1 + hdd2. All three HDDs were at ~92% capacity before cleanup.
|
||
|
||
**After cleanup (2026-04-23):**
|
||
|
||
| Device | Used | Avail | Use% |
|
||
|--------|------|-------|------|
|
||
| `/dev/sdd1` (hdd0) | 9.4T | 6.2T | 61% |
|
||
| `/dev/sdf1` (hdd1) | 9.3T | 6.3T | 60% |
|
||
| `/dev/sde1` (hdd2) | 7.8T | 7.8T | 51% |
|
||
| `mergerfs /media` | 27T | 21T | 57% |
|
||
|
||
**~16T freed total** (92% → 57% on the mergerfs pool).
|
||
|
||
## /media Breakdown (before cleanup)
|
||
|
||
| Directory | Size |
|
||
|-----------|------|
|
||
| `downloads` | **22T** |
|
||
| `series` | 16T |
|
||
| `movies` | 5T |
|
||
|
||
## Root Cause: No Hardlinks → All Imports Are Copies
|
||
|
||
Zero hardlinked files exist anywhere across all three HDDs. Confirmed by inspecting the Kubernetes manifests in `argocd-homelab/services/arr-stack/` and by inode comparison of 1365 download/media file pairs (0 shared inodes found).
|
||
|
||
**All three services mount the mergerfs `/media/` path via NFS:**
|
||
|
||
```
|
||
sonarr: NFS 192.168.20.12:/media/downloads → /downloads
|
||
NFS 192.168.20.12:/media/series → /tv
|
||
radarr: NFS 192.168.20.12:/media/downloads → /downloads
|
||
NFS 192.168.20.12:/media/movies → /movies
|
||
qbit: NFS 192.168.20.12:/media/downloads → /downloads
|
||
```
|
||
|
||
mergerfs does not support hardlinks across underlying filesystems. When qBit downloads to `/media/downloads/sonarr/` (lands on e.g. hdd1) and Sonarr imports to `/media/series/` (lands on e.g. hdd0), the hardlink attempt crosses a physical disk boundary → falls back to copy. Every import doubles the data.
|
||
|
||
## Cleanup Performed (2026-04-23)
|
||
|
||
Three passes using the scripts in this directory:
|
||
|
||
### Pass 1 — Orphans (not in Sonarr at all)
|
||
Script: `cleanup-orphans.py`
|
||
|
||
Deleted 49 entries totalling **461.6G** — downloads with no matching Sonarr series and no series directory on disk. Includes Game of Thrones (all 8 seasons), Sex Education (all 4 seasons), Love Death & Robots (multiple duplicate copies), and various anime episode files.
|
||
|
||
111 entries were SKIPPED (series dir found on disk, needs manual review) — includes Bleach, House, Lucifer, You, Detective Conan episodes, What If, etc. See cleanup.log for full list.
|
||
|
||
### Pass 2 — Confirmed-imported Sonarr downloads
|
||
Script: `cleanup.py --arr sonarr`
|
||
|
||
Deleted **1106 entries**, 0 failed. These were downloads where Sonarr confirmed `episodeFileCount > 0` AND the series directory was verified to exist on disk at the time of `verify.py` run.
|
||
|
||
### Pass 3 — Confirmed-imported Radarr downloads
|
||
Script: `cleanup.py --arr radarr`
|
||
|
||
Deleted **259 entries**, 0 failed. These were downloads where Radarr confirmed `hasFile=True` AND the file/directory path was verified to exist on disk.
|
||
|
||
### Totals
|
||
| Pass | Entries | Space |
|
||
|------|---------|-------|
|
||
| Orphans (cleanup-orphans.py) | 49 | ~461G |
|
||
| Sonarr imports (cleanup.py) | 1106 | ~12T (estimated) |
|
||
| Radarr imports (cleanup.py) | 259 | ~4T (estimated) |
|
||
| **Total** | **1414** | **~16T freed** |
|
||
|
||
All deletions logged to `cleanup.log` with UTC timestamp, size, title, path, outcome.
|
||
|
||
## Verification Results (via API + disk path check)
|
||
|
||
API keys stored in `../sonarr.api.env` and `../radarr.api.env`.
|
||
Access via `kubectl -n arr-stack port-forward svc/sonarr 8989:8989` and `svc/radarr 7878:7878`.
|
||
|
||
Container path mappings:
|
||
- Sonarr `/tv/` → `/media/series/`
|
||
- Radarr `/movies/` → `/media/movies/`
|
||
|
||
| | Safe to delete | Orphans (not in arr) | Keep |
|
||
|---|---|---|---|
|
||
| **Radarr** (289 items, ~5.2T) | **265** | 25 | 0 |
|
||
| **Sonarr** (1439 items, ~17T) | **1106** | 333 | 0 |
|
||
|
||
"Safe to delete" = API confirms `hasFile=True` (Radarr) or `episodeFileCount > 0` (Sonarr), AND the reported file/directory path was verified to exist on disk via SSH.
|
||
|
||
### Radarr Orphans (25) — not matched in Radarr, not deleted
|
||
- Constantine (2005)
|
||
- Cowboy Bebop: Knockin' on Heaven's Door (2001)
|
||
- Les Misérables (2012)
|
||
- Pokémon Detective Pikachu (2019)
|
||
- Code Geass: Fukkatsu no Lelouch (2019)
|
||
- Eiga Go-Toubun no Hanayome (2022)
|
||
- Gisaengchung / Parasite (Korean title — matching failure)
|
||
- Dune: Part One (2021) — matching failure, is in Radarr
|
||
- Harry Potter (older/duplicate copies — matching failure)
|
||
- Porco Rosso / Kurenai no buta — matching failure
|
||
- Castle in the Sky / Laputa — matching failure
|
||
- Steins;Gate: The Movie — matching failure
|
||
- Project Silence / Talchul — matching failure
|
||
- Digimon: Frontier & Savers films
|
||
- One Piece films (several)
|
||
- Paripi Koumei movie
|
||
- Fantastic Four (2025) extra copies (3)
|
||
- JJK DCP trailer file
|
||
|
||
### 6 Radarr "path mismatch" entries (all confirmed safe, deleted)
|
||
Flagged due to path comparison artifacts, manually verified on disk:
|
||
- Star Wars Episode IV/V/VI/IX — each is a separate Radarr entry; all directories exist
|
||
- WALL·E — `·` middle-dot character caused comparison failure; file exists
|
||
|
||
## Pending Decisions
|
||
|
||
### Bleach USBD Remux TL (1.8T)
|
||
`/media/downloads/sonarr/Bleach USBD Remux TL` — full lossless Bluray remux S00–S16 (-ZR- group).
|
||
Currently in SKIPPED (series dir `/media/series/Bleach (2004) {imdb-tt0434665}/` exists, 310G imported).
|
||
Most seasons were imported from x265 Bluray packs (-iVy group) rather than from this remux.
|
||
S11 has no imported content at all. S13, S14 partially imported.
|
||
Decision: keep (for quality imports once disk freed) or delete (free 1.8T, accept x265 quality).
|
||
See memory file for full per-season breakdown.
|
||
|
||
### SKIPPED downloads (111 Sonarr entries)
|
||
Downloads where the series directory exists on disk but the series is not currently in Sonarr.
|
||
Likely removed series (House, Lucifer, You, Black Clover, etc.) or ongoing shows with stale episodes.
|
||
These need manual review — series may have been intentionally removed from Sonarr.
|
||
|
||
## Fix (not applied — future reference)
|
||
Mount per-HDD NFS paths instead of the mergerfs path, so downloads and media share the same physical filesystem and hardlinks work:
|
||
```yaml
|
||
# sonarr/radarr/qtun deployments — change NFS path from:
|
||
path: /media/downloads → path: /mnt/hdd0/downloads
|
||
path: /media/series → path: /mnt/hdd0/series
|
||
path: /media/movies → path: /mnt/hdd0/movies
|
||
```
|
||
Jellyfin/Plex continue reading from `/media/` (mergerfs). New imports hardlink within hdd0.
|