Skip to content

feat(cuda.core): add cluster scheduling policy to LaunchConfig - #2703

Open
atiaomar1978-hub wants to merge 12 commits into
NVIDIA:mainfrom
atiaomar1978-hub:feat/launch-config-cluster-scheduling-policy-2629-signed
Open

feat(cuda.core): add cluster scheduling policy to LaunchConfig#2703
atiaomar1978-hub wants to merge 12 commits into
NVIDIA:mainfrom
atiaomar1978-hub:feat/launch-config-cluster-scheduling-policy-2629-signed

Conversation

@atiaomar1978-hub

Copy link
Copy Markdown
Contributor

Description

closes #2629

Add cluster_scheduling_policy_preference to LaunchConfig, mapping to
CU_LAUNCH_ATTRIBUTE_CLUSTER_SCHEDULING_POLICY_PREFERENCE via
ClusterSchedulingPolicyType (DEFAULT, SPREAD, LOAD_BALANCING).

Unlike stream synchronization policy (#2628), cluster scheduling policy is
documented as valid for graph nodes and kernel launches, so LaunchConfig is
the correct surface.

This is a signed re-issue of #2702 (same change, linear history on current
main). Commits are SSH-signed. GitHub would not reopen #2668.

Checklist

  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

Test plan

  • Native attribute mapping for DEFAULT / SPREAD / LOAD_BALANCING
  • Getter/setter round-trip for each policy
  • Driver enum acceptance; invalid value rejection; CC < 9.0 gate
  • Combined cluster + policy attributes
  • Hopper+ launch() smoke for all three policies
  • RunPod H100 80GB HBM3 / driver 580 / CTK 13.3 — 13 passed, 0 skipped
  • pre-commit.ci stubgen-pyx stubs regenerated

@copy-pr-bot

copy-pr-bot Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the cuda.core Everything related to the cuda.core module label Aug 26, 2026
atiaomar1978-hub and others added 4 commits August 25, 2026 17:19
Expose CU_LAUNCH_ATTRIBUTE_CLUSTER_SCHEDULING_POLICY_PREFERENCE on
LaunchConfig via ClusterSchedulingPolicyType, with validation, CC >= 9.0
gating, and tests mapping to the native launch attribute.

Closes NVIDIA#2629

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Omar Atie <atiaomar1978@gmail.com>
Assert LaunchConfig emits both CU_LAUNCH_ATTRIBUTE_CLUSTER_DIMENSION
and CU_LAUNCH_ATTRIBUTE_CLUSTER_SCHEDULING_POLICY_PREFERENCE when set.

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Omar Atie <atiaomar1978@gmail.com>
Use init_cuda with an inline noop kernel so test_launcher.py does not
depend on the get_saxpy_kernel_cubin fixture from test_module.py.

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Omar Atie <atiaomar1978@gmail.com>
…rough

Align NVIDIA#2629 tests with reviewer guidance: round-trip each policy on
LaunchConfig and exercise launch() for DEFAULT/SPREAD/LOAD_BALANCING.

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Omar Atie <atiaomar1978@gmail.com>
@atiaomar1978-hub

atiaomar1978-hub commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

Hi @lijinf2 — when you have a moment, could you please review this PR?

This is the signed re-issue of #2702 (GitHub would not reopen #2668). It implements the plan we aligned on in #2629: LaunchConfig.cluster_scheduling_policy_preference maps to CU_LAUNCH_ATTRIBUTE_CLUSTER_SCHEDULING_POLICY_PREFERENCE (DEFAULT / SPREAD / LOAD_BALANCING), with CC >= 9.0 gating.

Tests follow your guidance (getter/setter plus a launch() run-through; no attempt to observe the live scheduler). On RunPod H100 80GB HBM3 (CC 9.0, driver 580, CTK 13.3) the cluster-policy suite is 13 passed, 0 skipped.

I do not have permission to set assignee or milestone on this account; cuda.core 1.3.0 would match the related work if that is still the right milestone.

Sorry for the force-push on this branch. I know rewriting published history is against the project rules. I did it only so the commits would be SSH-signed and show as Verified on GitHub. I will not force-push again.

@lijinf2

lijinf2 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

/ok to test c6c283b

@lijinf2 lijinf2 added the feature New feature or request label Aug 27, 2026
@lijinf2 lijinf2 added this to the cuda.core next milestone Aug 27, 2026
@github-actions

Copy link
Copy Markdown

CI test_subpackage_exports_are_documented[typing] failed because the
new typing export was missing from docs/source/*.rst.

Signed-off-by: Omar Atie <atiaomar1978@gmail.com>
@atiaomar1978-hub

Copy link
Copy Markdown
Contributor Author

Posted a docs fix for the CI failure on test_subpackage_exports_are_documented[typing].

ClusterSchedulingPolicyType was exported from cuda.core.typing but missing from docs/source/*.rst. Commit 39c9fc0843 adds it to api_private.rst (SSH-signed / Verified).

RunPod (RTX 2000 Ada, CC 8.9 — L4 community was out of stock; same compute capability as the failing L4 jobs):

  • test_subpackage_exports_are_documented[typing] passed
  • cluster launcher suite: 10 passed, 3 skipped (launch smokes require CC >= 9.0)

PR metadata is now set (cuda.core, feature, assignee, milestone).

@atiaomar1978-hub

atiaomar1978-hub commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

Hi @lijinf2 — could you please trigger CI again on 39c9fc0843?

The previous /ok to test covered c6c283bb5d3f. That matrix failed; this SHA has the docs fix and has not gotten a new GPU run yet.

Why CI failed

Every GPU job died on the same assertion — not on LaunchConfig or Hopper launch:

tests/test_api_docs_consistency.py::test_subpackage_exports_are_documented[typing] FAILED
E   AssertionError: public by cuda.core.typing.__all__ but missing from docs/source/*.rst:
E       ['ClusterSchedulingPolicyType']

ClusterSchedulingPolicyType was added to cuda.core.typing.__all__ but was not listed in cuda_core/docs/source/api_private.rst (the autosummary the consistency test reads). Example: linux-64 / py3.10 / CUDA 13.3 wheels / L4 — 1 failed, launcher cluster tests passed or skipped by CC.

Why it works again

39c9fc0843 adds the missing autosummary line:

typing.ClusterSchedulingPolicyType

That name is now present in docs/source/*.rst, so the export-vs-docs check passes. Runtime tests were already green.

RunPod replay (pod 7xp9vqqo44kk1s, RTX 2000 Ada, CC 8.9)

L4 community was out of stock; Ada is the same CC 8.9 as the failing L4 jobs. HEAD on the pod was 39c9fc0.

############ CI FAILURE: docs export consistency (typing) ############
tests/test_api_docs_consistency.py::test_subpackage_exports_are_documented[typing] PASSED [100%]
======================= 1 passed, 5 deselected in 1.10s ========================

############ CLUSTER SCHEDULING POLICY TESTS (CI-like GPU) ############
compute_capability ComputeCapability(major=8, minor=9)
tests/test_launcher.py::test_to_native_launch_config_cluster_scheduling_policy[0] PASSED
tests/test_launcher.py::test_to_native_launch_config_cluster_scheduling_policy[1] PASSED
tests/test_launcher.py::test_to_native_launch_config_cluster_scheduling_policy[2] PASSED
tests/test_launcher.py::test_to_native_launch_config_cluster_scheduling_policy_accepts_driver_enum PASSED
tests/test_launcher.py::test_launch_config_cluster_scheduling_policy_invalid PASSED
tests/test_launcher.py::test_launch_config_cluster_scheduling_policy_rejects_pre_hopper_cc PASSED
tests/test_launcher.py::test_to_native_launch_config_cluster_and_policy PASSED
tests/test_launcher.py::test_launch_config_cluster_scheduling_policy_getter_setter[0] PASSED
tests/test_launcher.py::test_launch_config_cluster_scheduling_policy_getter_setter[1] PASSED
tests/test_launcher.py::test_launch_config_cluster_scheduling_policy_getter_setter[2] PASSED
tests/test_launcher.py::test_launch_cluster_scheduling_policy_smoke[0] SKIPPED
tests/test_launcher.py::test_launch_cluster_scheduling_policy_smoke[1] SKIPPED
tests/test_launcher.py::test_launch_cluster_scheduling_policy_smoke[2] SKIPPED
=========== 10 passed, 3 skipped, 61 deselected, 1 warning in 0.74s ============

Launch smokes skip on CC < 9.0, same as the L4 CI jobs. Pod has been deleted.

Full pod log (nvidia-smi + pytest; apt/pip noise omitted)
Thu Aug 27 21:56:33 2026
NVIDIA-SMI 570.195.03   Driver Version: 570.195.03   CUDA Version: 12.8
GPU: NVIDIA RTX 2000 Ada Generation

HEAD: 39c9fc0 docs(cuda.core): document ClusterSchedulingPolicyType in API RST
Fetching mini CUDA toolkit 13.3.0...
#define CUDA_VERSION 13030
cuda.core 0.0.0

tests/test_api_docs_consistency.py::test_subpackage_exports_are_documented[typing] PASSED
======================= 1 passed, 5 deselected in 1.10s ========================

compute_capability ComputeCapability(major=8, minor=9)
10 passed, 3 skipped, 61 deselected, 1 warning in 0.74s

Thank you.

@lijinf2

lijinf2 commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

/ok to test 39c9fc0

@lijinf2

lijinf2 commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

@atiaomar1978-hub Hey, there are 7 test cases, and it looks like some are redundant. Could we remove redundancy and merge them into at most 3 cases to reduce code maintenance overhead going forward?

@atiaomar1978-hub

Copy link
Copy Markdown
Contributor Author

I will do now @lijinf2

Merge seven overlapping LaunchConfig cluster-scheduling tests into
mapping, rejection, and Hopper launch smoke to cut maintenance.

Signed-off-by: Omar Atie <atiaomar1978@gmail.com>
@atiaomar1978-hub

Copy link
Copy Markdown
Contributor Author

@lijinf2 folded the seven cluster-policy tests into three in 1a78ad5cd1 (Verified):

  1. test_launch_config_cluster_scheduling_policy — ctor / getter-setter / driver enum / native attrs for DEFAULT, SPREAD, LOAD_BALANCING (including cluster + policy together)
  2. test_launch_config_cluster_scheduling_policy_rejected — invalid value and CC < 9.0
  3. test_launch_cluster_scheduling_policy_smokelaunch() for all three policies on Hopper+

Could you /ok to test 1a78ad5cd1cf0294d91994b0214ec71bdf9bdd2f when you have a moment?

@lijinf2

lijinf2 commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

/ok to test 1a78ad5

CUDA 13.0.2's CUlaunchAttributeValue setter reads .value, so a raw int
breaks native LaunchConfig conversion in CI.

Signed-off-by: Omar Atie <atiaomar1978@gmail.com>
@atiaomar1978-hub

Copy link
Copy Markdown
Contributor Author

@lijinf2 CUDA 13.0.2 jobs failed on test_launch_config_cluster_scheduling_policy (AttributeError — that setter reads .value, so a raw int breaks). 3a90298eb1 passes CUclusterSchedulingPolicy instead.

RunPod replay (RTX 2000 Ada, bindings 13.0.3): mapping + reject PASSED; smoke SKIPPED on CC 8.9 (already passed on CI H100).

Could you please /ok to test 3a90298eb1e788047e702076b81dcdb28073e699?

…-2629-signed

Signed-off-by: Omar Atie <atiaomar1978@gmail.com>
@lijinf2

lijinf2 commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

/ok to test 3a90298

@copy-pr-bot

copy-pr-bot Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

/ok to test 3a90298

@lijinf2, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

Comment thread cuda_core/cuda/core/_launch_config.pxd Outdated
Comment thread cuda_core/cuda/core/_launch_config.pyi Outdated
Comment thread cuda_core/cuda/core/_launch_config.pyi Outdated
Comment thread cuda_core/cuda/core/_launch_config.pyi Outdated
Drop ClusterSchedulingPolicyType so callers pass DEFAULT, SPREAD, or
LOAD_BALANCING without importing a new enum.

Signed-off-by: Omar Atie <atiaomar1978@gmail.com>
Cython types the argument as str, so only DEFAULT/SPREAD/LOAD_BALANCING
are accepted.

Signed-off-by: Omar Atie <atiaomar1978@gmail.com>
@atiaomar1978-hub

Copy link
Copy Markdown
Contributor Author

@lijinf2 addressed the review on c15bf56d51 (Verified): dropped ClusterSchedulingPolicyType, LaunchConfig.cluster_scheduling_policy_preference is now str | None ("DEFAULT" / "SPREAD" / "LOAD_BALANCING").

RunPod H100 SXM (CC 9.0), HEAD c15bf56:

  • test_launch_config_cluster_scheduling_policy PASSED
  • test_launch_config_cluster_scheduling_policy_rejected PASSED
  • test_launch_cluster_scheduling_policy_smoke PASSED

3 passed, 0 skipped. Could you please /ok to test c15bf56d51c1826b48c31608bae9e13fd52ca84d?

Signed-off-by: Omar Atie <atiaomar1978@gmail.com>
@lijinf2

lijinf2 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

/ok to test c15bf56

@copy-pr-bot

copy-pr-bot Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

/ok to test c15bf56

@lijinf2, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

@lijinf2

lijinf2 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

/ok to test d185a86

@atiaomar1978-hub

Copy link
Copy Markdown
Contributor Author

Thanks @lijinf2 for triggering the CI builds and tests 👍

Comment thread cuda_core/cuda/core/_launch_config.pyx Outdated
'cluster_scheduling_policy_preference',
)

_CLUSTER_SCHED_POLICY_NAMES = (

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove this variable and reuse _CLUSTER_SCHED_POLICY_TO_DRIVER dictionary keys. In future maintainance we hope less variables.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e85fafe — removed _CLUSTER_SCHED_POLICY_NAMES; validation now uses _CLUSTER_SCHED_POLICY_TO_DRIVER.keys().

Comment thread cuda_core/cuda/core/_launch_config.pyx Outdated
)


def _cluster_sched_policy_to_driver(value):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

delete this function to use _CLUSTER_SCHED_POLICY_TO_DRIVER[value] directly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e85fafe — removed _cluster_sched_policy_to_driver; lookup is _CLUSTER_SCHED_POLICY_TO_DRIVER[value] via LaunchConfig._cluster_sched_policy_driver_value().

"LOAD_BALANCING",
)

_CLUSTER_SCHED_POLICY_TO_DRIVER = {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's combine _CLUSTER_SCHED_POLICY_TO_DRIVER, and _validate_cluster_scheduling_policy_preference into a unified function (maybe _validate_cluster_scheduling_policy_preference or better name)?

Also, please put the unified function under LaunchConfig as a member function not global.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e85fafe — merged validation into LaunchConfig._validate_cluster_scheduling_policy_preference (member method).

"cluster launch attributes are not supported on devices with "
f"compute capability < 9.0 (got {cc})"
)
if cluster is not None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's keep lines 142-150 unchanged and add compute capability checking to the unified function.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e85fafe — lines 142–150 unchanged; CC check for policy is also in the member validator.

attr.value.clusterSchedulingPolicyPreference = _cluster_sched_policy_to_driver(
config.cluster_scheduling_policy_preference
)
attrs.append(attr)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we set twice? one at line 217-223, and the other at line 285-292?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The attribute is appended in two conversion paths until all modules are cythonized: the cdef _to_native_launch_config (cydriver structs) and the cpdef module helper (Python driver.CUlaunchAttribute objects). e85fafe factors shared driver lookup into LaunchConfig._cluster_sched_policy_driver_value().

Comment thread cuda_core/tests/test_launcher.py Outdated
assert (attr.value.clusterDim.x, attr.value.clusterDim.y, attr.value.clusterDim.z) == (2, 2, 2)


_CLUSTER_SCHED_POLICIES = (

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove _CLUSTER_SCHED_POLICIES and import.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e85fafe — tests iterate _CLUSTER_SCHED_POLICY_TO_DRIVER keys instead of a local tuple.

@lijinf2

lijinf2 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

atiaomar1978-hub

@atiaomar1978-hub Hey, are you a human contributor or a code agent contributor? Your replies and code style read like automatic agent-generated content, with little human review/refinement.

@atiaomar1978-hub

Copy link
Copy Markdown
Contributor Author

human + agent together and integration with RunPod to Test @lijinf2

…onfig

Move validation and driver lookup onto LaunchConfig members, drop duplicate
policy name constants, and reuse _CLUSTER_SCHED_POLICY_TO_DRIVER in tests.

Signed-off-by: Omar Atie <atiaomar1978@gmail.com>
@atiaomar1978-hub

Copy link
Copy Markdown
Contributor Author

Addressed latest review feedback in e85fafe:

  • Moved cluster policy validation onto LaunchConfig._validate_cluster_scheduling_policy_preference (uses _CLUSTER_SCHED_POLICY_TO_DRIVER keys; CC check included).
  • Removed _CLUSTER_SCHED_POLICY_NAMES, _cluster_sched_policy_to_driver, and test-local _CLUSTER_SCHED_POLICIES.
  • Shared driver lookup via LaunchConfig._cluster_sched_policy_driver_value() for both native conversion paths.

RunPod (H100, CC 9.0, cuda.bindings 13.3.1): 3/3 cluster scheduling policy tests passed.

Ready for /ok to test e85fafe when convenient.

@lijinf2

lijinf2 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

/ok to test e85fafe

@atiaomar1978-hub

Copy link
Copy Markdown
Contributor Author

The failing job Test win-64 / Python 3.14, CUDA 13.3.0 (wheels), GPU h100 (x2) (MCDM) (run) does not look related to this PR's code changes.

It failed during Configure driver mode (ci/tools/configure_driver_mode.ps1) while switching the Windows runner to MCDM and cycling the dual H100 devices. The job never reached artifact download or Run cuda.core tests.

On the same commit (e85fafe), builds, pre-commit, API checks, and the other Windows/Linux GPU test jobs passed. This matrix entry is the only dual-H100 + MCDM Windows job in the PR matrix, so this looks like a runner/infra flake rather than a regression from the cluster scheduling policy changes.

Happy to re-run that job if helpful. Cluster policy coverage was verified separately on RunPod H100 (3/3 tests passed).

@atiaomar1978-hub

Copy link
Copy Markdown
Contributor Author

@lijinf2 When you have a moment, could you please take another look at the latest review feedback addressed in e85fafe?

The only failing check on that commit was the Windows dual-H100 (MCDM) job, which failed during Configure driver mode before any cuda.core tests ran. That looks like a runner/infra issue rather than a regression from this PR. If you agree, a re-run of that job (or a fresh /ok to test) would be appreciated.

Cluster scheduling policy coverage was verified on RunPod H100 (3/3 tests passed). Thanks again for the detailed review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cuda.core Everything related to the cuda.core module feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEA]: Support CU_LAUNCH_ATTRIBUTE_CLUSTER_SCHEDULING_POLICY_PREFERENCE in LaunchConfig

2 participants