Skip to content

fix(job-retry): fail open instead of dropping the job when isJobQueued errors - #5424

Open
atsikham wants to merge 4 commits into
github-aws-runners:mainfrom
atsikham:fix/job-retry-fail-open-on-error
Open

atsikham wants to merge 4 commits into
github-aws-runners:mainfrom
atsikham:fix/job-retry-fail-open-on-error

Conversation

@atsikham

Copy link
Copy Markdown
Contributor

Fixes #5423

Summary

checkAndRetryJob had no error handling around its call to isJobQueued. Any error from it (a transient GitHub API error, a rate limit, an unsupported event type) propagated out of the function, and the lambda handler only logs and swallows that error — the SQS message is marked processed and never redelivered, so the job's retry is dropped for good, and the RetryJob metric for that attempt is never recorded either.

Changes

  • isJobQueued now runs inside a try/catch, mirroring the equivalent check already in scale-up.ts:
    • On UnsupportedEventError, the retry is skipped (that error can never resolve itself, retrying won't help).
    • On any other error, the job is assumed to still be queued and the retry is published anyway — a transient error is not evidence the job stopped needing a runner.

Note on overlap with #5422

This touches the same lines as #5422 (which makes the retry lambda respect ENABLE_JOB_QUEUED_CHECK) — they're different bugs (a missing config check vs. missing error handling) found independently, so kept as separate PRs rather than bundled. Whichever merges first, the other will likely need a small rebase.

Test plan

  • yarn test — two new cases in job-retry.test.ts: a transient isJobQueued error still publishes the retry and records the metric (fail-open), and an unsupported event type skips the retry without publishing. Full control-plane suite passes (358 tests).
  • yarn lint / yarn format
  • tsc --noEmit

…d errors

checkAndRetryJob had no error handling around isJobQueued: any error
from it (a transient GitHub API failure, a rate limit, an unsupported
event type) propagated out of the function. The lambda handler catches
and only logs that error, so the SQS message is still marked processed
and never redelivered - the job's retry is silently dropped for good,
and the RetryJob metric for that attempt is never recorded either.

isJobQueued now runs in a try/catch that mirrors the same check in
scale-up.ts: on an UnsupportedEventError the retry is skipped (that
error can never resolve itself), on any other error the job is assumed
still queued and the retry is published anyway, since a transient
error is not evidence the job stopped needing a runner.
@atsikham
atsikham requested a review from a team as a code owner September 12, 2026 19:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Job retry silently drops the job (and its metric) on a transient GitHub API error

1 participant