feat(cli): add hyperframes clean to find and remove what HyperFrames left on disk - #4701
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
…left on disk `hyperframes clean` lists leftover render work dirs, debug renders, the history of projects that no longer exist, idle video proxies and extracted video frames, with sizes, and removes them. `--dry-run` removes nothing, `--json` is for agents, `--snapshots` also removes snapshot folders (otherwise listed as reclaimable). It never touches outputs, sources or anything a running render or preview still owns.
bb999d2 to
8120b4e
Compare
…manifest is regenerated
…scoder named It removed any idle .mp4 or .webm in .transcode-cache, so a user file placed there, or reached through a symlinked cache folder, could go. Proxies are always <sha256>.<ext> and their temp files .tmp-<uuid>-<proxy name>; nothing else is swept.
somanshreddy
left a comment
There was a problem hiding this comment.
Verdict: APPROVE.
Reviewed at bfb3184e. This is a real irreversible-file-deletion feature, so I gave it the highest scrutiny I've applied today: the failure mode isn't "wrong output," it's "deleted a user's actual file."
The core safety property: can this ever delete something it shouldn't?
Traced isAbandoned() (renderDirOwner.ts) line by line. The check that matters most: !lstatSync(dir).isDirectory() returns false-abandoned immediately — and critically this uses lstatSync, not statSync, so a symlink is never even considered for removal, regardless of what it points to (a symlink's own lstat type is never "directory"). This is stronger than just "rmSync doesn't follow symlinks" (which is also true) — the scanner refuses to walk into the decision logic for a symlinked entry at all. Confirms exactly what the PR's own sandboxed watcher-based test table shows: every one of six symlink-to-outside-the-scratch-root cases kept its target intact, including a link literally named like a render temp dir.
Other layers I verified by reading, not just trusting the summary:
existsSync(join(dir, TRANSACTION_BACKUP))unconditionally protects a staging dir mid-swap, regardless of owner state — a render that died between swapping in a new output and clearing the backup can't lose the previous good output.ownerState()requires host+boot+pidns match and a numeric pid and the process being dead — every ambiguous case (no owner file, wrong scope, malformed pid) returns"unknown"/"none", which only becomes removable via the separate, much more conservative 6-hour-idle fallback — never immediately.newestWriteMs()'s recursive walk useslstatSyncper entry too, so a symlink nested inside an otherwise-abandoned dir doesn't get followed to compute idle time off some unrelated external target.
The bug the author found and fixed during their own testing — verified the fix, not just the changelog line
proxyCache.ts's sweep previously matched any file by extension alone in the cache folder; if .transcode-cache were symlinked somewhere holding an unrelated holiday.mp4, it would sweep that too. The fix adds TRANSCODER_PROXY_NAME (^[0-9a-f]{64}\.\w+$, the transcoder's own SHA256-hash naming) and TRANSCODER_TEMP_NAME as hard gates before a file is even considered a candidate. Read the regexes myself and confirmed holiday.mp4 genuinely fails TRANSCODER_PROXY_NAME while a real <hash>.mp4 passes.
Also traced a second, subtler fix in the same file: the old budget-eviction pass re-iterated the full entries array (already-removed entries included), only guarded by existsSync. In dry-run mode nothing is ever actually unlinked, so existsSync can't distinguish "already logically removed by the idle pass" from "still there" — meaning a dry-run's reported list could double-count the same file. The new code restricts the budget pass to kept (entries the idle pass didn't already claim), which closes exactly that gap. Confirmed by reasoning through the dry-run code path myself, not just from the PR's own "the budget pass seeing only what the idle pass kept" note.
Tests
Hit two of today's now-familiar environment gaps and worked through both: clean.test.ts's one failure ("reports a leftover it cannot remove") turned out to be my stale @hyperframes/studio-server/@hyperframes/producer/@hyperframes/engine dist from an earlier PR's build this session — added a temporary debug print, confirmed the genuine intended EACCES error was present alongside three mock-resolution errors caused by the stale dist, rebuilt all three packages at this commit, reran clean, removed the debug print: 7/7 pass. extractionCache.test.ts + videoFrameInjector.test.ts: 53/53 pass. proxyCache.test.ts hits the already-documented vitest/node: builtin loader issue on this devbox (not this PR's fault, same as core/src/projects.test.ts earlier today).
CI: all failing/blocking checks are none; the Windows suite and 10 regression shards were still IN_PROGRESS (not failed) when I checked — flagging honestly rather than claiming a green I didn't observe.
No findings. This is a genuinely careful piece of work — the author's own sandboxed watcher-based testing caught a real bug before I ever saw the PR, and everything I independently traced held up.
Follows #4694 (renders leave no captured frames behind), which records the owner each render temp dir uses here.
What this adds
hyperframes cleanreports what HyperFrames left on this machine, with sizes, and removes it.What it removes:
renders/), and in the temp dir. A dir counts as abandoned only when its recorded owner (fix(producer): renders leave no captured frames behind when interrupted #4694) has exited in this pid scope, or, for dirs made before owners were recorded, when nothing in it has been written for 6 hours. Only render temp dir names are considered.PRODUCER_RENDERS_DIRpoints it somewhere shared.pruneGoneProjectHistories).dryRunoption, so the listing and the removal run the same code.Snapshots are listed as "also reclaimable" and removed only with
--snapshots. Proxies and snapshots are only swept inside a HyperFrames project (a folder whoseindex.htmlholds a composition root). Outputs, sources, the browser download and any dir whose owner is still running are never touched. A leftover that cannot be removed, or an unreadable history folder, is reported undererrors, the rest of the clean still runs, and the command exits 1.Engine fix this needed
A render renewed its lease on an extracted-frame cache entry only while that clip was on screen. A clip that first appears more than an hour into a long capture looked abandoned to any sweep in that window (the cache's own size-cap sweep before this PR, and
cleannow). The injector now renews every entry the render holds on every frame, throttled per directory as before (FrameLookupTable.frameDirs()).Evidence
On a shared Linux dev machine:
clean --dry-runclean --dry-run --jsonSandboxed run of every delete path
The built CLI ran with HOME, TMPDIR, XDG dirs,
PRODUCER_RENDERS_DIRand the frame cache redirected into a scratch root, while a watcher listed every pre-existing file outside it that disappeared. Result: 29 of 29 checks pass, nothing outside the scratch root removed.backup,work-in-progressuser folder, the output andindex.html.build-id,my-notesholiday.mp4in the cache folder--snapshots.mp4and a framesnapshotslink (only the link goes),.transcode-cachelink, temp-dir link, frame-cache linkThe
.transcode-cachecase failed on the first run: the proxy sweep removed any idle.mp4or.webmin that folder, so an old video in a folder linked as the cache went. The sweep now removes only names the transcoder writes (<sha256>.<ext>and.tmp-<uuid>-<proxy name>), which also protects the Studio's own sweep.Tests
clean.test.tsremoves what dead renders left and keeps what a running render uses (live owner, fresh ownerless dir, user folder, output, non-job debug dir)clean.test.tsreports a leftover it cannot remove and still removes the restclean.test.tsleaves snapshots alone in a folder that is not a HyperFrames project (a plain website)clean.test.tsstill cleans when the history folder cannot be readclean.test.tslists a leftover once when the project folder is a link to the temp dirclean.test.tsexits 1 when something could not be read or removed (through the command)clean.test.tslists snapshots as reclaimable and removes them only when askedvideoFrameInjector.test.tsrenews the entry of a clip that is not on screen yetproxyCache.test.tsreports the same removals in a dry run and removes nothingproxyCache.test.tsnever removes a file the transcoder did not name, even from an idle cache over budgetextractionCache.test.tscounts the same evictions in a dry run and removes nothingEach "fails without" was checked by reverting that piece and watching the test go red.
Not covered here
-oelsewhere) are found only whencleanruns in that folder.