Conversation
…d errors checkAndRetryJob had no error handling around isJobQueued: any error from it (a transient GitHub API failure, a rate limit, an unsupported event type) propagated out of the function. The lambda handler catches and only logs that error, so the SQS message is still marked processed and never redelivered - the job's retry is silently dropped for good, and the RetryJob metric for that attempt is never recorded either. isJobQueued now runs in a try/catch that mirrors the same check in scale-up.ts: on an UnsupportedEventError the retry is skipped (that error can never resolve itself), on any other error the job is assumed still queued and the retry is published anyway, since a transient error is not evidence the job stopped needing a runner.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5423
Summary
checkAndRetryJobhad no error handling around its call toisJobQueued. Any error from it (a transient GitHub API error, a rate limit, an unsupported event type) propagated out of the function, and the lambda handler only logs and swallows that error — the SQS message is marked processed and never redelivered, so the job's retry is dropped for good, and theRetryJobmetric for that attempt is never recorded either.Changes
isJobQueuednow runs inside a try/catch, mirroring the equivalent check already inscale-up.ts:UnsupportedEventError, the retry is skipped (that error can never resolve itself, retrying won't help).Note on overlap with #5422
This touches the same lines as #5422 (which makes the retry lambda respect
ENABLE_JOB_QUEUED_CHECK) — they're different bugs (a missing config check vs. missing error handling) found independently, so kept as separate PRs rather than bundled. Whichever merges first, the other will likely need a small rebase.Test plan
yarn test— two new cases injob-retry.test.ts: a transientisJobQueuederror still publishes the retry and records the metric (fail-open), and an unsupported event type skips the retry without publishing. Full control-plane suite passes (358 tests).yarn lint/yarn formattsc --noEmit