Repository navigation
Use a separate retry timeout for tensor tests in CI - #3132
Merged
Merged
Conversation
The tensor tests run in a few minutes but shared the main suite's retry timeout, so a sporadic hang there burned the whole timeout before a retry. Add a dedicated rerun-tensor-tests-timeout in conda-package.yml (15 min) and check-onemath.yaml (10 min). In check-onemath.yaml also raise the main suite retry timeout to 30 min, since healthy runs take up to ~19 min, and the oneMath develop job timeout from 60 to 120 min to match the tag job, so a retry can still fit.
antonwolfy
requested review from
ndgrigorian and
vlad-perevezentsev
as code owners
October 29, 2026 16:01
Contributor
|
Array API standard conformance tests for dpnp=0.22.0dev0=np2py314h8d9cdd5_6 ran successfully. |
Contributor
|
View rendered docs @ https://intelpython.github.io/dpnp/index.html |
Collaborator
There was a problem hiding this comment.
🟢 Approval recommended
There are no active review issues for the CI timeout changes.
No open issues 🎉
What changed in this PR
Separates tensor-test retry timeouts from the main test-suite timeouts so sporadic tensor hangs trigger retries sooner.
Changes:
- Sets tensor-test timeouts to 15 minutes in conda CI and 10 minutes in oneMath CI.
- Raises the oneMath main-suite timeout from 20 to 30 minutes; leaves conda’s unchanged.
- Extends the oneMath develop-branch job limit from 60 to 120 minutes to allow retries to run.
| File | Description |
|---|---|
.github/workflows/conda-package.yml |
Adds a dedicated 15-minute tensor timeout for Linux and Windows retry steps. |
.github/workflows/check-onemath.yaml |
Adds a 10-minute tensor timeout and increases main-suite and develop-branch job limits. |
Review effort: Balanced
Healthy Windows tensor runs take up to ~19 min, so a 15 min timeout could kill a slow but passing run. Add rerun-tensor-tests-timeout-win (20 min) and use it for the Windows tensor step in conda-package.yml. Also use the dedicated tensor timeouts (15 min on Linux, 20 min on Windows) in the reusable conda-build-test.yml, since conda-package-cf.yml hits the same tensor hang.
With the shorter tensor timeouts, three attempts cost less than the old two, and a hang in both attempts does happen. Add rerun-tensor-tests-max-attempts (3) and use it in the tensor retry steps. Raise the conda-package.yml Linux test job timeout from 100 to 120 min, the same as Windows, so a main suite hang plus two tensor hangs still fit.
github-actions Bot
added a commit
that referenced
this pull request
Oct 30, 2026
The `dpnp.tests.tensor` retry step shared `rerun-tests-timeout` with the main `dpnp.tests` suite, although the tensor tests normally finish in a few minutes. The tensor tests sporadically hang (a deadlock in the OpenCL CPU runtime), and each hang used up the whole shared timeout before the retry started. This PR adds a dedicated `rerun-tensor-tests-timeout` (plus `rerun-tensor-tests-timeout-win` for Windows) and uses it in the tensor retry steps: | Workflow | Healthy tensor run | Tensor timeout (before → after) | Main suite timeout (before → after) | |---|---|---|---| | `conda-package.yml`, Linux | up to 5.8 min | 45 → 15 min | 45 min (unchanged, healthy runs up to ~30 min) | | `conda-package.yml`, Windows | mostly under 12 min, up to 19.2 min | 45 → 20 min | 45 min (unchanged, healthy runs up to ~34 min) | | `conda-package-cf.yml` (reusable `conda-build-test.yml`) | as above, Windows up to 17.6 min | 45 → 15 min (Linux), 20 min (Windows) | 45 min (unchanged, healthy runs up to ~37 min) | | `check-onemath.yaml` | 2.4–5.4 min | 20 → 10 min | 20 → 30 min (healthy runs up to ~19.3 min) | Windows gets a longer tensor timeout because its healthy runs have a slow tail: in October, 18 of 688 runs took over 10.5 min and the slowest took 19.2 min, though none took over 11.7 min since October 16. In `check-onemath.yaml`, the oneMath develop branch job timeout is also raised from 60 to 120 min, the same as the oneMath tag job. That job spends a median of ~31 min building (up to ~37) and ~16 min in the main suite, so the tensor tests start around minute 51 (up to 58) and even healthy jobs finish close to the limit (median 54.5 min, p95 59.2 min). A hung tensor run was therefore cancelled at the job limit before the retry could fire. Since October 1, 35 develop branch jobs were cancelled at 60 min: 17 hung at 77–82% in `test_usm_ndarray_linalg.py`, 14 ran out of time during the tensor tests, and 4 during the main suite. The hang became frequent from about October 19 on the develop job and October 25 on the tag job, and 6 of the last 80 passing oneMath jobs passed only because the retry killed a hung tensor run. In `conda-package.yml`, 68 of 1462 tensor steps in October (4.7%) hit the hang and took 48–52 min, and `conda-package-cf.yml` hits it at the same rate (57 of 1102). With this change, a hang costs the timeout plus a healthy rerun, ~18–21 min on Linux, instead of ~48–52 min. 8461ec8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
dpnp.tests.tensorretry step sharedrerun-tests-timeoutwith the maindpnp.testssuite, although the tensor tests normally finish in a few minutes. The tensor tests sporadically hang (a deadlock in the OpenCL CPU runtime), and each hang used up the whole shared timeout before the retry started.This PR adds a dedicated
rerun-tensor-tests-timeout(plusrerun-tensor-tests-timeout-winfor Windows) and uses it in the tensor retry steps:conda-package.yml, Linuxconda-package.yml, Windowsconda-package-cf.yml(reusableconda-build-test.yml)check-onemath.yamlWindows gets a longer tensor timeout because its healthy runs have a slow tail: in October, 18 of 688 runs took over 10.5 min and the slowest took 19.2 min, though none took over 11.7 min since October 16.
In
check-onemath.yaml, the oneMath develop branch job timeout is also raised from 60 to 120 min, the same as the oneMath tag job. That job spends a median of ~31 min building (up to ~37) and ~16 min in the main suite, so the tensor tests start around minute 51 (up to 58) and even healthy jobs finish close to the limit (median 54.5 min, p95 59.2 min). A hung tensor run was therefore cancelled at the job limit before the retry could fire. Since October 1, 35 develop branch jobs were cancelled at 60 min: 17 hung at 77–82% intest_usm_ndarray_linalg.py, 14 ran out of time during the tensor tests, and 4 during the main suite. The hang became frequent from about October 19 on the develop job and October 25 on the tag job, and 6 of the last 80 passing oneMath jobs passed only because the retry killed a hung tensor run.In
conda-package.yml, 68 of 1462 tensor steps in October (4.7%) hit the hang and took 48–52 min, andconda-package-cf.ymlhits it at the same rate (57 of 1102). With this change, a hang costs the timeout plus a healthy rerun, ~18–21 min on Linux, instead of ~48–52 min.