Skip to content

fix(endpoint): Endpoint.run_sync drops the error of a FAILED job - #699

Open
deanq wants to merge 6 commits into
mainfrom
deanq/sls-1003-run-sync-failed-error
Open

deanq wants to merge 6 commits into
mainfrom
deanq/sls-1003-run-sync-failed-error

Conversation

@deanq

@deanq deanq commented Nov 7, 2026

Copy link
Copy Markdown
Member

What

Endpoint.run_sync and Job.output returned None when a job ended FAILED, TIMED_OUT or CANCELLED, and the job's error was dropped. A failed job looked the same as a handler that returned None:

endpoint.run_sync({"input": {...}})   # handler raised ValueError
# before: None
# after:  JobError: job <id> FAILED: {"error_type": "<class 'ValueError'>", "error_message": "...", ...}

Changes:

  • New runpod.error.JobError(RunPodError) with job_id, status and error.
  • Endpoint.run_sync, Job.output and the asyncio Job.output raise it for any final state other than COMPLETED. The asyncio docstring already said output() raises when the job fails.
  • COMPLETED jobs return their output as before, including None.
  • CANCELLED is now handled the same as the other final states in run_sync, which previously only checked COMPLETED, FAILED and TIMED_OUT.

Behavior change

Code that checked result is None to detect a failure now gets a JobError instead. Raising was chosen over returning the error so a failure can't be silently treated as a result. It matches runpod.apps, which raises RemoteExecutionError for failed jobs.

Test plan

  • New tests/test_endpoint/test_failed_jobs.py:
    • run_sync raises with status and error for FAILED, TIMED_OUT and CANCELLED
    • run_sync raises when a polled job fails
    • a COMPLETED job with no output still returns None
    • Job.output raises on every call after a failure
    • a failure without an error field still names the status
    • asyncio Job.output raises on failure
  • The raise tests failed before the change
  • make quality-check (2781 passed)
  • Live check against a deployed queue endpoint whose handler raises: on unfixed code run_sync returned None; with this change run_sync and run().output() raise JobError with status FAILED and the handler's error message. A successful call still returns its output.

Endpoint.run_sync and Job.output returned None for a job that ended
FAILED, TIMED_OUT or CANCELLED, and dropped the job's error. A failed
job looked the same as a handler that returned None.

- New runpod.error.JobError carries job_id, status and error.
- Endpoint.run_sync, Job.output and the asyncio Job.output raise it
  for any final state other than COMPLETED.
- COMPLETED jobs return their output as before, including None.

SLS-1003
@celine-runpod celine-runpod changed the title fix(endpoint): raise JobError when a job fails fix(endpoint): raise JobError when a job fails (SLS-1003) Nov 9, 2026
@celine-runpod celine-runpod changed the title fix(endpoint): raise JobError when a job fails (SLS-1003) fix(endpoint): Endpoint.run_sync drops the error of a FAILED job (SLS-1003) Nov 9, 2026
@celine-runpod

Copy link
Copy Markdown
Contributor

bugbot run

@deanq
deanq requested a balanced review from Copilot November 9, 2026 18:02
@deanq deanq changed the title fix(endpoint): Endpoint.run_sync drops the error of a FAILED job (SLS-1003) fix(endpoint): Endpoint.run_sync drops the error of a FAILED job Nov 9, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Both synchronous and asynchronous runners return None after streaming completes because the cached completed status bypasses fetching the available aggregate output from /status.

2 open issues
What changed in this PR

This PR makes endpoint job failures distinguishable from successful jobs that return None, exposing failure details through JobError.

Changes:

  • Adds JobError with job ID, status, and error details.
  • Raises it for failed, timed-out, and cancelled jobs in synchronous and asynchronous output retrieval.
  • Adds regression tests for failure handling and successful None output.
File Description
tests/​test_endpoint/​test_failed_jobs.py Tests terminal failures, repeated error reporting, and successful None output.
runpod/​error.py Defines the new JobError exception.
runpod/​endpoint/​runner.py Caches failure details and raises errors from synchronous job results.
runpod/​endpoint/​helpers.py Adds shared terminal-state error handling.
runpod/​endpoint/​asyncio/​asyncio_runner.py Applies failure handling to asynchronous job output.

Review effort: Balanced

Comment thread runpod/endpoint/runner.py Resolved
deanq and others added 3 commits November 9, 2026 10:11
stream() caches a terminal COMPLETED status with no aggregate output,
because the stream response carries chunks rather than a result. The
cached-status fast path in output() then returned None without querying
/status, dropping an available result.

When the cached status is COMPLETED with no cached output, fetch it from
/status before returning. Cached failures still raise without an extra
fetch. Covers both the sync and async runners.
…rror' into deanq/sls-1003-run-sync-failed-error

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants