Expose safe per-pool peer access for configured GPU pairs

[kevkrist]

Problem

Applications using cuCascade across multiple GPUs need to configure access to the actual memory pools backing their allocations and know whether that configuration succeeded. The current public API offers a best-effort operation across all visible GPUs, but no operation with an observable result for one pool and one accessing GPU.

CUDA distinguishes device peer capability from memory-pool permissions. cudaDeviceCanAccessPeer reports whether a device can access a peer, but neither that result nor cudaDeviceEnablePeerAccess grants access to allocations from a CUDA asynchronous memory pool. Pool access is configured separately with cudaMemPoolSetAccess and inspected with cudaMemPoolGetAccess. A grant on a device's default or currently selected pool does not grant access to another pool on that device. See CUDA memory-pool accessibility.

Why this is needed

Consider a process that sees four GPUs but uses only GPUs 0 and 2 for one execution group. Each GPU has an application-owned asynchronous pool, possibly alongside pools owned by other libraries. Before sharing buffers, the application needs to:

• Find the pool behind its cuCascade reservation-aware resource adaptor.
• Grant GPU 2 access to GPU 0's pool, and separately configure the reverse direction where needed, without granting pool access to every visible GPU.
• Distinguish a successful grant from an unsupported pair, a failed byte-transfer check, or a CUDA runtime error.

Without that information, callers must duplicate allocator-specific discovery and peer-access logic, or select a transfer strategy without establishing its prerequisites. CUDA can stage peer copies through the host, so a successful call alone is not proof that the intended direct path was used. This issue does not claim that every copy without pool grants fails or corrupts data. See CUDA peer-copy behavior.

Current API gap

In the current helper implementation, enable_pool_peer_access_for_all_visible_devices(pool, owner):

• Visits all visible peers and also processes the owner's currently selected pool, not just the supplied pool.
• Returns void and discards individual grant/query errors, so callers cannot distinguish success, lack of support, and failure.
• Checks only one direction of the existing byte-verifying peer-DMA probe before granting read/write access.

The reservation-aware adaptor implementation already caches a pool handle for OOM diagnostics, using an explicit constructor handle or discovery of supported RMM resources. That handle is not publicly exposed. Accessing the type-erased upstream resource is not equivalent: it requires callers to repeat discovery and does not recover an explicit constructor override.

Proposed solution

1. Expose the existing borrowed pool handle. Add reservation_aware_resource_adaptor::pool_handle() through its implementation wrapper. Return the cached handle, or null when the backing pool is unknown. Do not create, replace, or transfer ownership of a pool. Document its lifetime and that an explicitly supplied handle must describe the actual allocation pool when used as access evidence.

2. Add a per-pool, per-peer grant operation. Provide an API such as grant_pool_peer_access(pool, owner_device, accessing_device). Scope the pool-permission change to that pool and accessor, and return a typed outcome distinguishing granted access, unsupported peer capability, rejected byte-transfer verification, and CUDA runtime failure. Preserve underlying CUDA errors, including probe errors; a boolean failure must not hide invalid execution state. API spelling is open to library review.

3. Reuse one implementation. Have the all-visible convenience helper delegate to the same grant logic. Require the relevant byte-transfer checks in both directions before granting peer read/write access. Keep this device-pair evidence separate from the permissions of each actual pool; neither establishes a bandwidth guarantee or proves a later copy's route.

4. Make initialization and failure behavior explicit. Restore the caller's current device and document synchronization and legacy peer-state changes caused by the existing first-use probe. A per-pair pool grant does not imply that those probe side effects are pair-local. A failed grant does not prove an existing permission is absent or guarantee a safe host-staged fallback. Callers must verify the relevant permissions and establish their fallback independently.

This is a small memory-resource API extension, not an allocator-policy redesign, pool-ownership change, or driver-compatibility layer.

Acceptance criteria

• Accessor tests cover explicit handles, supported discovered resources, unknown/non-pool resources, and borrowed-handle lifetime.
• Grant tests cover self-access, repeated requests, and independent permissions for two pools on the same device.
• Multi-GPU tests inspect the actual pool permissions and verify transferred bytes in both directions.
• Tests distinguish unsupported peers, asymmetric/failed byte-transfer checks, and CUDA query/grant/probe errors; use a narrow deterministic test seam where necessary.
• Caller-device restoration and initialization side effects are documented and tested.
• The all-visible helper shares the implementation, with its broader behavior documented.
• Record the tested hardware, build/test results, and skipped cases. This issue proposes an API; it does not claim implementation or completed validation.

Background and downstream tracking

This need was identified during the Sirius dynamic-filter optimization campaign. Historical API prototypes are represented by commits 2d484a0 and 31155d6; they are reference material, not a branch to replay wholesale. Any implementation should target current cuCascade main and exclude the unrelated campaign driver shim and ancestry.

• Sirius planning issue: Plan: complete-build dynamic filters and six-optimization delivery stack sirius-db/sirius#1818
• Design and delivery plan: docs: define the complete-build dynamic-filter design and delivery stack sirius-db/sirius#1819
• Downstream integration: S4: Integrate pool peer access and pipeline Bloom publication sirius-db/sirius#1823

The downstream task is blocked by this issue. It requires the cuCascade implementation PR to merge upstream before consuming the API and validating its integration; closing this issue alone does not satisfy that requirement. The planned PR head is kevkrist/cuCascade:memory/pool-peer-access, targeting NVIDIA/cuCascade:main.