Skip to content

Use a separate retry timeout for tensor tests in CI - #3132

Merged
antonwolfy merged 4 commits into
masterfrom
split-tensor-tests-retry-timeout
Oct 30, 2026
Merged

antonwolfy merged 4 commits into
masterfrom
split-tensor-tests-retry-timeout

Conversation

@antonwolfy

@antonwolfy antonwolfy commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

The dpnp.tests.tensor retry step shared rerun-tests-timeout with the main dpnp.tests suite, although the tensor tests normally finish in a few minutes. The tensor tests sporadically hang (a deadlock in the OpenCL CPU runtime), and each hang used up the whole shared timeout before the retry started.

This PR adds a dedicated rerun-tensor-tests-timeout (plus rerun-tensor-tests-timeout-win for Windows) and uses it in the tensor retry steps:

Workflow Healthy tensor run Tensor timeout (before → after) Main suite timeout (before → after)
conda-package.yml, Linux up to 5.8 min 45 → 15 min 45 min (unchanged, healthy runs up to ~30 min)
conda-package.yml, Windows mostly under 12 min, up to 19.2 min 45 → 20 min 45 min (unchanged, healthy runs up to ~34 min)
conda-package-cf.yml (reusable conda-build-test.yml) as above, Windows up to 17.6 min 45 → 15 min (Linux), 20 min (Windows) 45 min (unchanged, healthy runs up to ~37 min)
check-onemath.yaml 2.4–5.4 min 20 → 10 min 20 → 30 min (healthy runs up to ~19.3 min)

Windows gets a longer tensor timeout because its healthy runs have a slow tail: in October, 18 of 688 runs took over 10.5 min and the slowest took 19.2 min, though none took over 11.7 min since October 16.

In check-onemath.yaml, the oneMath develop branch job timeout is also raised from 60 to 120 min, the same as the oneMath tag job. That job spends a median of ~31 min building (up to ~37) and ~16 min in the main suite, so the tensor tests start around minute 51 (up to 58) and even healthy jobs finish close to the limit (median 54.5 min, p95 59.2 min). A hung tensor run was therefore cancelled at the job limit before the retry could fire. Since October 1, 35 develop branch jobs were cancelled at 60 min: 17 hung at 77–82% in test_usm_ndarray_linalg.py, 14 ran out of time during the tensor tests, and 4 during the main suite. The hang became frequent from about October 19 on the develop job and October 25 on the tag job, and 6 of the last 80 passing oneMath jobs passed only because the retry killed a hung tensor run.

In conda-package.yml, 68 of 1462 tensor steps in October (4.7%) hit the hang and took 48–52 min, and conda-package-cf.yml hits it at the same rate (57 of 1102). With this change, a hang costs the timeout plus a healthy rerun, ~18–21 min on Linux, instead of ~48–52 min.

  • Have you provided a meaningful PR description?
  • Have you added a test, reproducer or referred to an issue with a reproducer?
  • Have you tested your changes locally for CPU and GPU devices?
  • Have you made sure that new changes do not introduce compiler warnings?
  • Have you checked performance impact of proposed changes?
  • Have you added documentation for your changes, if necessary?
  • Have you added your changes to the changelog?

The tensor tests run in a few minutes but shared the main suite's retry
timeout, so a sporadic hang there burned the whole timeout before a retry.
Add a dedicated rerun-tensor-tests-timeout in conda-package.yml (15 min)
and check-onemath.yaml (10 min).

In check-onemath.yaml also raise the main suite retry timeout to 30 min,
since healthy runs take up to ~19 min, and the oneMath develop job timeout
from 60 to 120 min to match the tag job, so a retry can still fit.
@antonwolfy antonwolfy self-assigned this Oct 29, 2026
@antonwolfy antonwolfy added this to the 0.22.0 release milestone Oct 29, 2026
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Array API standard conformance tests for dpnp=0.22.0dev0=np2py314h8d9cdd5_6 ran successfully.
Passed: 1491
Failed: 0
Skipped: 6

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

View rendered docs @ https://intelpython.github.io/dpnp/index.html

@coveralls

coveralls commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Coverage Status

No base build to compare — split-tensor-tests-retry-timeout into master

@ndgrigorian ndgrigorian left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

There are no active review issues for the CI timeout changes.

No open issues 🎉

What changed in this PR

Separates tensor-test retry timeouts from the main test-suite timeouts so sporadic tensor hangs trigger retries sooner.

Changes:

  • Sets tensor-test timeouts to 15 minutes in conda CI and 10 minutes in oneMath CI.
  • Raises the oneMath main-suite timeout from 20 to 30 minutes; leaves conda’s unchanged.
  • Extends the oneMath develop-branch job limit from 60 to 120 minutes to allow retries to run.
File Description
.github/​workflows/​conda-package.yml Adds a dedicated 15-minute tensor timeout for Linux and Windows retry steps.
.github/​workflows/​check-onemath.yaml Adds a 10-minute tensor timeout and increases main-suite and develop-branch job limits.

Review effort: Balanced

Healthy Windows tensor runs take up to ~19 min, so a 15 min timeout
could kill a slow but passing run. Add rerun-tensor-tests-timeout-win
(20 min) and use it for the Windows tensor step in conda-package.yml.

Also use the dedicated tensor timeouts (15 min on Linux, 20 min on
Windows) in the reusable conda-build-test.yml, since conda-package-cf.yml
hits the same tensor hang.
With the shorter tensor timeouts, three attempts cost less than the old
two, and a hang in both attempts does happen. Add
rerun-tensor-tests-max-attempts (3) and use it in the tensor retry
steps.

Raise the conda-package.yml Linux test job timeout from 100 to 120 min,
the same as Windows, so a main suite hang plus two tensor hangs still
fit.
@antonwolfy
antonwolfy merged commit 8461ec8 into master Oct 30, 2026
108 of 111 checks passed
@antonwolfy
antonwolfy deleted the split-tensor-tests-retry-timeout branch October 30, 2026 13:42
github-actions Bot added a commit that referenced this pull request Oct 30, 2026
The `dpnp.tests.tensor` retry step shared `rerun-tests-timeout` with the
main `dpnp.tests` suite, although the tensor tests normally finish in a
few minutes. The tensor tests sporadically hang (a deadlock in the
OpenCL CPU runtime), and each hang used up the whole shared timeout
before the retry started.

This PR adds a dedicated `rerun-tensor-tests-timeout` (plus
`rerun-tensor-tests-timeout-win` for Windows) and uses it in the tensor
retry steps:

| Workflow | Healthy tensor run | Tensor timeout (before → after) | Main
suite timeout (before → after) |
|---|---|---|---|
| `conda-package.yml`, Linux | up to 5.8 min | 45 → 15 min | 45 min
(unchanged, healthy runs up to ~30 min) |
| `conda-package.yml`, Windows | mostly under 12 min, up to 19.2 min |
45 → 20 min | 45 min (unchanged, healthy runs up to ~34 min) |
| `conda-package-cf.yml` (reusable `conda-build-test.yml`) | as above,
Windows up to 17.6 min | 45 → 15 min (Linux), 20 min (Windows) | 45 min
(unchanged, healthy runs up to ~37 min) |
| `check-onemath.yaml` | 2.4–5.4 min | 20 → 10 min | 20 → 30 min
(healthy runs up to ~19.3 min) |

Windows gets a longer tensor timeout because its healthy runs have a
slow tail: in October, 18 of 688 runs took over 10.5 min and the slowest
took 19.2 min, though none took over 11.7 min since October 16.

In `check-onemath.yaml`, the oneMath develop branch job timeout is also
raised from 60 to 120 min, the same as the oneMath tag job. That job
spends a median of ~31 min building (up to ~37) and ~16 min in the main
suite, so the tensor tests start around minute 51 (up to 58) and even
healthy jobs finish close to the limit (median 54.5 min, p95 59.2 min).
A hung tensor run was therefore cancelled at the job limit before the
retry could fire. Since October 1, 35 develop branch jobs were cancelled
at 60 min: 17 hung at 77–82% in `test_usm_ndarray_linalg.py`, 14 ran out
of time during the tensor tests, and 4 during the main suite. The hang
became frequent from about October 19 on the develop job and October 25
on the tag job, and 6 of the last 80 passing oneMath jobs passed only
because the retry killed a hung tensor run.

In `conda-package.yml`, 68 of 1462 tensor steps in October (4.7%) hit
the hang and took 48–52 min, and `conda-package-cf.yml` hits it at the
same rate (57 of 1102). With this change, a hang costs the timeout plus
a healthy rerun, ~18–21 min on Linux, instead of ~48–52 min. 8461ec8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants