Repository navigation
Exit air run --watch when the streamed node succeeds - #7031
Open
stevenchen-db wants to merge 4 commits into
Open
stevenchen-db wants to merge 4 commits into
stevenchen-db wants to merge 4 commits into
Conversation
Jobs reports an AI Runtime run terminal tens of seconds after the user command exits, so `air run --watch` kept running after the work was done. The stream now watches for the entry script's success summary. Once the streamed node prints it, every node's newest lines are checked for the same block, pinned to the current attempt, and the command exits 0 with a SUCCESS closing envelope. A failure summary, a missing summary, an unknown node count, or an API error keeps the existing wait for Jobs, since a failed attempt may be retried. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
Integration test reportCommit: e6328c8
Top 6 slowest tests (at least 2 minutes):
|
Once the streamed node's summary triggered the check, every poll re-fetched every node up to the first unfinished one, so a failing node late in a large run cost one request per node per second until Jobs reported the run terminal. A node that succeeded stays succeeded, so the check now resumes at the first unconfirmed node, and a retry resets that progress. Also pin with tests that a missing node count blocks the early exit and that each node is confirmed again on a new attempt. Co-authored-by: Isaac <no-reply@databricks.com>
--watch follows one node, so stop when that node's command succeeds instead of first confirming every node through extra per-node log requests. This drops the node count, the per-attempt checks, and the confirmed-node tracking. A failure summary still waits for Jobs, so a retried attempt keeps being followed and a failed run still exits non-zero. Co-authored-by: Isaac <no-reply@databricks.com>
stevenchen-db
changed the title
air run --watch when the command succeeds on every nodeair run --watch when the streamed node succeeds
| log.Infof(st.ctx, "air logs: run %d finished in state %s", st.req.runID, st.status.displayState()) | ||
| return st.status.succeeded(), nil | ||
| } | ||
| if st.req.exitOnCompletion && st.completion.completed { |
Contributor
There was a problem hiding this comment.
In a multi-node run, node 0 can succeed while another worker fails. This returns success immediately, so --watch exits 0 and emits SUCCESS without observing the failure or retries. Could we restrict this shortcut to single-node runs or confirm all nodes first?
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
air run --watchnow stops as soon as the command succeeds on the node it streams, instead of waiting for Jobs to mark the run terminal.While streaming, the CLI watches the streamed node's new lines for the entry script's "Execution Summary" block ending in
Completed at, which is printed only on exit code 0. When it appears, the command exits 0, and in-o jsonthe closing envelope reportsSUCCESSeven though Jobs may still sayRUNNING. A failure summary is ignored and the CLI waits for Jobs as today, so a retried attempt keeps being followed and a failed run still exits non-zero. Only--watchopts in;air logsis unchanged.Why
Jobs reports the run terminal well after the user command exits. On a 2-node H100 run, the summary printed at 21:46:19 but the run was marked terminal at 21:46:51, so
--watchlooked hung for ~30s after the code finished.--watchfollows one node, so it now stops when that node's command stops.Tests
--watchpaths. Removing the opt-in gate, the success-only match, the matcher's state across polls, or the JSONSUCCESSstatus each makes a test fail.go test ./cmd/air/...andTestAccept/air/pass;./task fmtand./task checksare clean.unusedcould not run locally (Go 1.27 toolchain crash), so CI is their first run.This pull request and its description were written by Isaac.