Conversation
rsync returns 24 ("some files vanished before they could be transferred")
when a file disappears between building the file list and transferring it.
Everything else is still copied, so it is a warning rather than a failure,
but execAsync rejects on any non-zero exit code and the whole web server
backup aborts.
This fires routinely on hosts where a compose stack keeps a live database
directory under BASE_PATH: a Postgres checkpoint purges
pg_logical/snapshots/*.snap in one batch, and any rsync run overlapping that
purge dies with code 24. --ignore-errors does not help, it only applies to
errors reported during --delete.
Tolerate exit code 24, log it, and let the backup finish.
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| if (error instanceof ExecError && error.exitCode === 24) { | ||
| writeStream.write( | ||
| "Some files vanished while copying the filesystem (rsync exit 24), continuing\n", | ||
| ); |
There was a problem hiding this comment.
Treating every rsync exit 24 under the entire BASE_PATH as benign is broader than the transient-snapshot case described here. If a control-plane path such as application configuration, certificates, or SSH data is deleted and recreated during this pass, rsync can omit it and return 24. This branch then uploads the incomplete archive and marks the backup successful, and a later restore replaces the current data path with that incomplete tree. Retry the copy or restrict suppression to omissions known to be disposable before reporting success.
Knowledge Base Used: Backups and restore
There was a problem hiding this comment.
Addressed in a4b4a84 by taking the retry option you suggested.
The copy now runs again once when rsync reports vanished files, and every vanished path is written to the backup log. A retry is an incremental pass over an almost complete copy, so it is cheap, and it picks up anything that was deleted and recreated while the first pass ran, which is the control-plane case you describe. Only a second exit 24 is tolerated, and that means the source tree keeps churning during the copy rather than a one-off rewrite: in practice a live database directory inside BASE_PATH, for example a Postgres checkpoint purging pg_logical/snapshots/*.snap every five minutes.
One thing worth separating out: exit 24 is not what makes the archive non-atomic. rsync over a live tree is never point in time consistent, whatever it exits with. A file created after the file list was built is silently absent with exit 0, and a file rewritten in place is copied in whatever state it is in when rsync opens it. Exit 24 only reports paths that existed when the list was built and were gone when rsync tried to open them, so failing the whole backup on it does not buy consistency, it just loses the backup. The remaining omissions are now visible in the log instead of being silent.
There was a problem hiding this comment.
The retry addresses the original blocking concern: a first exit 24 now gets an incremental second pass, while non-24 failures still propagate and a second exit 24 is explicitly logged before continuing. That substantially reduces the chance that a control-plane file deleted and recreated during the first pass is omitted.
One small mismatch remains in packages/server/src/utils/backups/rsync.ts: MAX_LOGGED_VANISHED_PATHS limits the log to 10 paths per pass, so the statement that “every vanished path is written” is not accurate—paths after the tenth are only represented by the ...and N more summary. Either describe the behavior as logging up to 10 paths, or remove the cap if a complete audit trail is required. Aside from that wording/logging detail, the original concern is resolved.
Tip: You can customize Greptile's behavior for this repo with .greptile/rules.md and .greptile/config.json.
There was a problem hiding this comment.
Good catch, the doc comment was overstating it. Reworded in 934118d in the same commit style: each pass logs the first MAX_LOGGED_VANISHED_PATHS paths plus a count of the rest. Kept the cap deliberately - this goes into the deployment log that the UI streams, and a churning directory can produce hundreds of entries per pass, so a bounded sample plus the count is more useful there than a full dump. The complete list stays available in rsync's own stderr on the ExecError.
| } catch (error) { | ||
| if (error instanceof ExecError && error.exitCode === 24) { | ||
| writeStream.write( | ||
| "Some files vanished while copying the filesystem (rsync exit 24), continuing\n", | ||
| ); | ||
| } else { | ||
| throw error; | ||
| } |
There was a problem hiding this comment.
The new selective recovery behavior has no automated regression coverage. Please add tests proving that an ExecError with exit code 24 continues through upload and success reporting, while code 23 and non-ExecError failures still take the error path. Without these tests, a later change could silently reintroduce the original backup failure or suppress genuine copy errors.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
Added in a4b4a84. The behaviour moved into packages/server/src/utils/backups/rsync.ts so it can be tested without pulling the whole backup pipeline in, and apps/dokploy/__test__/backups/rsync-vanished-retry.test.ts covers: a plain success runs the command once and logs nothing; exit 24 followed by success runs it twice and logs the vanished path plus the retry; exit 24 twice resolves and logs that it continues without those files; long vanished lists are truncated in the log; exit 23 and a non-ExecError failure both propagate without a retry.
Verified by mutation as well: removing the retry fails 5 of the 6 tests.
Log every vanished path and retry the rsync once instead of suppressing exit 24 outright, so a one-off rewrite under BASE_PATH is picked up by the second, incremental pass and only a continuously churning source tree ends up with omissions. Covered by tests for the retry, the repeated warning and the untouched error paths.
1 participant
What is this PR about?
The web server backup aborts whenever
rsyncreturns exit code 24.Code 24 means "some files vanished before they could be transferred" — a file disappeared between the moment rsync built its file list and the moment it tried to transfer it. Everything else is still copied, which is why rsync classes it as a warning rather than an error. But
execAsyncrejects on any non-zero exit code, so the whole backup dies right after the database dump:Note that
--ignore-errorsdoes not cover this — perman rsyncit only applies to errors reported during--delete.Why this is not an exotic edge case
Any compose stack that keeps a live database directory under
BASE_PATHtriggers it. The concrete case here is a Supabase stack whose compose bind-mounts./volumes/db/data, with logical replication enabled (realtime). Postgres accumulatespg_logical/snapshots/*.snapfiles and then purges the whole batch at each checkpoint — measured on a live host:With
checkpoint_timeout = 5minand an rsync pass overBASE_PATHtaking ~60s, roughly one backup run in five overlaps a purge and fails. On the affected host: 2 failures out of the last 11 scheduled runs, with nothing wrong with the backup itself.The fix
Catch the
ExecError, and ifexitCode === 24, write a line to the deployment log and continue. Every other exit code still propagates unchanged, so genuine failures (e.g. code 23 from unreadable files, as handled in #3853) keep failing loudly.Checklist
canarybranch.Verification: with the equivalent change applied to a running v0.30.6 instance, I ran the exact backup rsync inside the dokploy container while forcing
CHECKPOINTon the Supabase database to trigger the purge deterministically. rsync reported the vanished file and code 24, and the command returned exit 0 instead of aborting the backup:biome checkis clean on the changed file.Issues related (if applicable)
Follow-up to #3853, which replaced
cpwithrsyncin this code path.This PR is not safe to merge until exit 24 is handled without allowing required control-plane files to be silently omitted from a successful backup.
Summary
This PR changes self-hosted web-server backups to catch rsync failures and continue when the process exits with status 24.
ExecErrorfrom the existing process wrapper.Reviews (1) · Last reviewed commit: "fix(backups): treat rsync exit 24 as suc..."