Skip to content

cuda.core: reject register() on a non-IPC memory resource - #2569

Open
fedonman wants to merge 5 commits into
NVIDIA:mainfrom
fedonman:fix-ipc-register-non-ipc-guard
Open

cuda.core: reject register() on a non-IPC memory resource#2569
fedonman wants to merge 5 commits into
NVIDIA:mainfrom
fedonman:fix-ipc-register-non-ipc-guard

Conversation

@fedonman

@fedonman fedonman commented Aug 9, 2026

Copy link
Copy Markdown

Summary

Closes #2568.

DeviceMemoryResource.register() and PinnedMemoryResource.register() terminated the process with a segmentation fault when the memory resource did not have IPC enabled, including the default resource from Device().memory_resource. Both route through MP_register, which read self._ipc_data unconditionally although it is None unless IPC is enabled.

Changes

  • _ipc.pyx: check is_ipc_enabled in MP_register and raise RuntimeError("Memory resource is not IPC-enabled"), the wording the allocation_handle property already uses. The check runs before the registry insertion, so a rejected registration no longer leaves an entry behind.
  • tests/memory_ipc/test_errors.py: regression test covering the raise and the registry.
  • docs/source/release/1.2.0-notes.rst: release note. The defect is present in released 1.0.1, 1.1.0 and 1.1.1, so this is user-visible rather than confined to the current cycle.

Verification

Without the production change the new test does not merely fail, it takes the interpreter down (exit=139, SIGSEGV). With the change:

$ python -m pytest tests/memory_ipc/test_errors.py -k test_register_rejects_non_ipc -v
tests/memory_ipc/test_errors.py::test_register_rejects_non_ipc_memory_resource PASSED [100%]
======================= 1 passed, 14 deselected in 0.28s =======================

No regressions in the surrounding suites:

$ python -m pytest tests/test_memory.py tests/memory_ipc/ -q
192 passed, 133 skipped in 4.48s

MP_register read self._ipc_data._alloc_handle._uuid unconditionally.
_ipc_data is None whenever IPC is not enabled, and Cython compiles the
chained access without a none check, so the call terminated the process
with a segmentation fault instead of raising. No exception reached the
caller, so the crash could not be guarded with try.

Check is_ipc_enabled first and raise RuntimeError, matching the wording
the allocation_handle property already uses. The check runs before the
registry insertion, so a rejected registration no longer leaves an entry
behind.

Closes NVIDIA#2568

Signed-off-by: Vyron Vasileiadis <hi@fedonman.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the cuda.core Everything related to the cuda.core module label Aug 9, 2026

@lijinf2 lijinf2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution — this turns a hard segfault into a clear RuntimeError that callers can actually handle.

@lijinf2

lijinf2 commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

/ok to test a7da9e2

@lijinf2

lijinf2 commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

@fedonman You may select cuda core next 1.2.0 under "Milestone" which should resolve the failing CI checks.

@github-actions

Copy link
Copy Markdown

@fedonman

Copy link
Copy Markdown
Author

@fedonman You may select cuda core next 1.2.0 under "Milestone" which should resolve the failing CI checks.

@lijinf2 I think I don't have permissions to set milestone.

@lijinf2 lijinf2 added this to the cuda.core 1.2.0 milestone Aug 10, 2026
@lijinf2

lijinf2 commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

@fedonman You may select cuda core next 1.2.0 under "Milestone" which should resolve the failing CI checks.

@lijinf2 I think I don't have permissions to set milestone.

Sorry, I was not aware of that. Label was added.

@fedonman
fedonman requested a review from lijinf2 August 15, 2026 10:33
@lijinf2

lijinf2 commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

/ok to test ecb4ecf

@lijinf2 lijinf2 added the bug Something isn't working label Aug 17, 2026
@Andy-Jost Andy-Jost self-assigned this Aug 17, 2026
@Andy-Jost
Andy-Jost self-requested a review August 17, 2026 18:23
@Andy-Jost

Copy link
Copy Markdown
Contributor

The new test needs to be gated on mempool availability:

def test_register_rejects_non_ipc_memory_resource(mempool_device):
    mr = DeviceMemoryResource(mempool_device)

@Andy-Jost Andy-Jost left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new test needs to be gated on mempool availability.

@fedonman

fedonman commented Aug 18, 2026

Copy link
Copy Markdown
Author

@Andy-Jost Thanks. Done.

test_register_rejects_non_ipc_memory_resource constructed a
DeviceMemoryResource from a bare Device(), which raises
CUDA_ERROR_NOT_SUPPORTED on devices without memory pool support. Every
win-64 TCC job in CI failed there before reaching the assertion under
test.

Take the mempool_device fixture instead, matching how the other
mempool-dependent tests are gated. The fixture skips when
memory_pools_supported is false and already makes the device current,
so the manual Device()/set_current() pair is dropped.
@fedonman
fedonman force-pushed the fix-ipc-register-non-ipc-guard branch from f76c70f to ff819a6 Compare August 18, 2026 08:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cuda.core Everything related to the cuda.core module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: DeviceMemoryResource.register() segfaults when IPC is not enabled

3 participants