Skip to content

feat(cli): add hyperframes clean to find and remove what HyperFrames left on disk - #4701

Merged
miguel-heygen merged 5 commits into
mainfrom
feat/cli-clean
Sep 29, 2026
Merged

miguel-heygen merged 5 commits into
mainfrom
feat/cli-clean

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

Follows #4694 (renders leave no captured frames behind), which records the owner each render temp dir uses here.

What this adds

hyperframes clean reports what HyperFrames left on this machine, with sizes, and removes it.

hyperframes clean --dry-run    # list, remove nothing
hyperframes clean              # remove
hyperframes clean --snapshots  # also remove snapshots/ folders
hyperframes clean --json       # for agents

What it removes:

  • Render leftovers: work and staging dirs of renders that are no longer running, in the current folder, in every project the history cache knows (and their renders/), and in the temp dir. A dir counts as abandoned only when its recorded owner (fix(producer): renders leave no captured frames behind when interrupted #4694) has exited in this pid scope, or, for dirs made before owners were recorded, when nothing in it has been written for 6 hours. Only render temp dir names are considered.
  • Debug renders: job-id-named work dirs under the render debug dir, by the same rule. Nothing else in that dir is touched, even when PRODUCER_RENDERS_DIR points it somewhere shared.
  • Project history of projects that no longer exist, using the studio server's own rule (pruneGoneProjectHistories).
  • Video proxies nobody has used for an hour and extracted video frames, both regenerable. Each cache's existing sweep gains a dryRun option, so the listing and the removal run the same code.

Snapshots are listed as "also reclaimable" and removed only with --snapshots. Proxies and snapshots are only swept inside a HyperFrames project (a folder whose index.html holds a composition root). Outputs, sources, the browser download and any dir whose owner is still running are never touched. A leftover that cannot be removed, or an unreadable history folder, is reported under errors, the rest of the clean still runs, and the command exits 1.

Engine fix this needed

A render renewed its lease on an extracted-frame cache entry only while that clip was on screen. A clip that first appears more than an hour into a long capture looked abandoned to any sweep in that window (the cache's own size-cap sweep before this PR, and clean now). The injector now renews every entry the render holds on every frame, throttled per directory as before (FrameLookupTable.frameDirs()).

Evidence

On a shared Linux dev machine:

Run Result
clean --dry-run 1.9 s, 11 rows, 5.8 GB, 0 errors
clean --dry-run --json home folder redacted in every path

Sandboxed run of every delete path

The built CLI ran with HOME, TMPDIR, XDG dirs, PRODUCER_RENDERS_DIR and the frame cache redirected into a scratch root, while a watcher listed every pre-existing file outside it that disappeared. Result: 29 of 29 checks pass, nothing outside the scratch root removed.

Path Removed Kept
Render leftovers dead-owner work dir, dead staging dir, ownerless dir idle 7 h, dead-owner dir in the project root, temp-dir leftover live render's dir, fresh ownerless dir, staging dir holding backup, work-in-progress user folder, the output and index.html
Debug renders job-id dir idle 7 h .build-id, my-notes
Project history history of a project whose folder is gone histories of existing projects, one of them reached through a symlinked path
Video proxies idle proxy, stale temp file recent proxy, a user's holiday.mp4 in the cache folder
Extracted frames entry unused for 2 h, aged partial fresh entry, live partial
--snapshots project snapshots snapshots of a folder that is not a HyperFrames project (a plain website)
Ctrl+C in the middle of removing a 60,000-file leftover exits 130, nothing on the keep list touched; the next run finishes the removal
Symlinks to folders outside the scratch root, each holding a user file, an .mp4 and a frame all six targets intact: render-named link, link inside a removed dir, snapshots link (only the link goes), .transcode-cache link, temp-dir link, frame-cache link

The .transcode-cache case failed on the first run: the proxy sweep removed any idle .mp4 or .webm in that folder, so an old video in a folder linked as the cache went. The sweep now removes only names the transcoder writes (<sha256>.<ext> and .tmp-<uuid>-<proxy name>), which also protects the Studio's own sweep.

Tests

Test Fails without
clean.test.ts removes what dead renders left and keeps what a running render uses (live owner, fresh ownerless dir, user folder, output, non-job debug dir) job-id matching for debug dirs
clean.test.ts reports a leftover it cannot remove and still removes the rest per-dir error handling
clean.test.ts leaves snapshots alone in a folder that is not a HyperFrames project (a plain website) the composition-root project check
clean.test.ts still cleans when the history folder cannot be read the history error handling
clean.test.ts lists a leftover once when the project folder is a link to the temp dir comparing real paths
clean.test.ts exits 1 when something could not be read or removed (through the command) reporting the exit code through the CLI's command result
clean.test.ts lists snapshots as reclaimable and removes them only when asked
videoFrameInjector.test.ts renews the entry of a clip that is not on screen yet renewing every held entry
proxyCache.test.ts reports the same removals in a dry run and removes nothing the budget pass seeing only what the idle pass kept
proxyCache.test.ts never removes a file the transcoder did not name, even from an idle cache over budget matching transcoder names
extractionCache.test.ts counts the same evictions in a dry run and removes nothing

Each "fails without" was checked by reverting that piece and watching the test go red.

Not covered here

  • Sizes can under-report slightly: aged partial frame-cache dirs, stale proxy temp files and history dirs already mid-prune are removed but not counted.
  • A render that hits the frame cache and then spends more than an hour extracting other, uncached videos renews nothing until its first captured frame, so a sweep in that window can remove the hit entry. This predates the PR (the cache's own sweep has the same window); closing it means renewing during extraction.
  • Leftovers next to an output written outside any known project (-o elsewhere) are found only when clean runs in that folder.

Base automatically changed from fix/render-leaves-no-frames to main September 28, 2026 22:10
@mintlify

mintlify Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
hyperframes 🟢 Ready View Preview Sep 28, 2026, 11:12 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

…left on disk

`hyperframes clean` lists leftover render work dirs, debug renders, the history of projects
that no longer exist, idle video proxies and extracted video frames, with sizes, and removes
them. `--dry-run` removes nothing, `--json` is for agents, `--snapshots` also removes
snapshot folders (otherwise listed as reclaimable). It never touches outputs, sources or
anything a running render or preview still owns.
…scoder named

It removed any idle .mp4 or .webm in .transcode-cache, so a user file placed there, or reached through
a symlinked cache folder, could go. Proxies are always <sha256>.<ext> and their temp files
.tmp-<uuid>-<proxy name>; nothing else is swept.

@somanshreddy somanshreddy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE.

Reviewed at bfb3184e. This is a real irreversible-file-deletion feature, so I gave it the highest scrutiny I've applied today: the failure mode isn't "wrong output," it's "deleted a user's actual file."

The core safety property: can this ever delete something it shouldn't?

Traced isAbandoned() (renderDirOwner.ts) line by line. The check that matters most: !lstatSync(dir).isDirectory() returns false-abandoned immediately — and critically this uses lstatSync, not statSync, so a symlink is never even considered for removal, regardless of what it points to (a symlink's own lstat type is never "directory"). This is stronger than just "rmSync doesn't follow symlinks" (which is also true) — the scanner refuses to walk into the decision logic for a symlinked entry at all. Confirms exactly what the PR's own sandboxed watcher-based test table shows: every one of six symlink-to-outside-the-scratch-root cases kept its target intact, including a link literally named like a render temp dir.

Other layers I verified by reading, not just trusting the summary:

  • existsSync(join(dir, TRANSACTION_BACKUP)) unconditionally protects a staging dir mid-swap, regardless of owner state — a render that died between swapping in a new output and clearing the backup can't lose the previous good output.
  • ownerState() requires host+boot+pidns match and a numeric pid and the process being dead — every ambiguous case (no owner file, wrong scope, malformed pid) returns "unknown"/"none", which only becomes removable via the separate, much more conservative 6-hour-idle fallback — never immediately.
  • newestWriteMs()'s recursive walk uses lstatSync per entry too, so a symlink nested inside an otherwise-abandoned dir doesn't get followed to compute idle time off some unrelated external target.

The bug the author found and fixed during their own testing — verified the fix, not just the changelog line

proxyCache.ts's sweep previously matched any file by extension alone in the cache folder; if .transcode-cache were symlinked somewhere holding an unrelated holiday.mp4, it would sweep that too. The fix adds TRANSCODER_PROXY_NAME (^[0-9a-f]{64}\.\w+$, the transcoder's own SHA256-hash naming) and TRANSCODER_TEMP_NAME as hard gates before a file is even considered a candidate. Read the regexes myself and confirmed holiday.mp4 genuinely fails TRANSCODER_PROXY_NAME while a real <hash>.mp4 passes.

Also traced a second, subtler fix in the same file: the old budget-eviction pass re-iterated the full entries array (already-removed entries included), only guarded by existsSync. In dry-run mode nothing is ever actually unlinked, so existsSync can't distinguish "already logically removed by the idle pass" from "still there" — meaning a dry-run's reported list could double-count the same file. The new code restricts the budget pass to kept (entries the idle pass didn't already claim), which closes exactly that gap. Confirmed by reasoning through the dry-run code path myself, not just from the PR's own "the budget pass seeing only what the idle pass kept" note.

Tests

Hit two of today's now-familiar environment gaps and worked through both: clean.test.ts's one failure ("reports a leftover it cannot remove") turned out to be my stale @hyperframes/studio-server/@hyperframes/producer/@hyperframes/engine dist from an earlier PR's build this session — added a temporary debug print, confirmed the genuine intended EACCES error was present alongside three mock-resolution errors caused by the stale dist, rebuilt all three packages at this commit, reran clean, removed the debug print: 7/7 pass. extractionCache.test.ts + videoFrameInjector.test.ts: 53/53 pass. proxyCache.test.ts hits the already-documented vitest/node: builtin loader issue on this devbox (not this PR's fault, same as core/src/projects.test.ts earlier today).

CI: all failing/blocking checks are none; the Windows suite and 10 regression shards were still IN_PROGRESS (not failed) when I checked — flagging honestly rather than claiming a green I didn't observe.

No findings. This is a genuinely careful piece of work — the author's own sandboxed watcher-based testing caught a real bug before I ever saw the PR, and everything I independently traced held up.

@miguel-heygen
miguel-heygen added this pull request to the merge queue Sep 29, 2026
Merged via the queue into main with commit 663b022 Sep 29, 2026
75 checks passed
@miguel-heygen
miguel-heygen deleted the feat/cli-clean branch September 29, 2026 00:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants