Dockerfile - Add cuda13.3.dockerfile on pytorch:26.07-py3

[gusui-msft]

Description
Adds dockerfile/cuda13.3.dockerfile on nvcr.io/nvidia/pytorch:26.07-py3 and rotates the CI matrix from cuda13.0 (25.08-py3) to it. This is the usual one-Dockerfile-per-CUDA-version rotation, same shape as #716 and #819.

The security side is worth calling out, because it decides what else had to change here. Trivy 0.72.0 on main-cuda13.0 reports 58 Critical occurrences from exactly two places: 18 inherited from the NVIDIA base image, 40 from the Docker 20.10.8 static bundle we install. The base bump closes the first group; the second is why DOCKER_VERSION moves in the same PR rather than being left for later. Supersedes #845, now closed.

Major Revision

• Add dockerfile/cuda13.3.dockerfile on 26.07-py3: CUDA 13.3.1, PyTorch 2.13.0a0, cuDNN 9.24, NCCL 2.30.7, OpenMPI 5.0.10. Clears the 18 inherited findings (linux-libc-dev, jupyter_server, the Nsight Go plugin).
• DOCKER_VERSION 20.10.8 -> 29.7.2. The other 40 are Go stdlib CVEs compiled into the binaries the RUN unpacks, so apt and pip can't reach them and the bundle has to be replaced. We only use the Docker client against the host socket, and 29.3.0 lowered the daemon API floor back to v1.40 (Docker 19.03), so older hosts keep working.
• Point the docker-build and docker-merge matrices at cuda13.3.
• third_party/Makefile: add a CUDA_VER >= 13.3 branch selecting CUTLASS v4.7.0. The >= 12.9 branch would otherwise pick v4.1.0, which predates CUDA 13.3.

Minor Revision

• HPCX_VERSION -> v2.50 and UCX_VERSION -> 1.21.0, both matching the base. Not cosmetic: the HPC-X block does rm -rf /opt/hpcx and /usr/local/lib comes first in LD_LIBRARY_PATH, so an older pin would shadow part of the stack PyTorch was built against.
• Drop the setuptools pin instead of carrying it over. cuda13.0 pins 78.1.0, which against this base is both a downgrade and CVE-2025-47273.
• Refresh the version comment header for 26.07-py3 and fill in the MOFED version, which was blank.
• Replace the stale comment above cuda_cutlass; it claimed 12.9+ builds v3.9 while the recipe already cloned v4.1.0.

Verification
Both legs build: arm64 36 min, amd64 35 min on 9c8274f9 — the only change since is the removed setuptools pin, which arm64 has already rebuilt with. Scanning the result has to wait for merge, since docker-build sets push: false on pull requests.

Open questions

• Copilot flags the unverified Docker tarball download and the PermitRootLogin/PermitUserEnvironment lines. Both are real but inherited: no Dockerfile here verifies a download, and both sed lines come from Setup: Add Feature - Add CUDA dockerfile for superbench #43 (2021) across all 14 sshd-bearing images. Worth noting Docker publishes no checksums for the static bundles, so any check would be a self-computed pin rather than upstream verification. I'd rather sweep dockerfile/*.dockerfile once than have cuda13.3 diverge; happy to fold it in here instead.
• cuda13.0.dockerfile stays in place, following CI/CD - Clean up image builds, remove cuda 12.8/12.4/12.2 and add cuda13.0 merge #819. cuda12.9, cuda11.1.1 and rocm6.3.x still carry Docker 20.10.8/27.5.1, roughly 40 and 8 Critical — separate PR, or fold in here?

[Copilot]

Pull request overview

This PR adds a new CUDA 13.3 container image definition based on nvcr.io/nvidia/pytorch:26.07-py3, updates the GitHub Actions image build/merge matrices to build/publish cuda13.3 instead of cuda13.0, and adjusts third-party build logic to select a newer CUTLASS for CUDA 13.3+.

Changes:

• Add dockerfile/cuda13.3.dockerfile (new base image + updated HPC-X/UCX/Docker client handling).
• Update .github/workflows/build-image.yml matrix entries to build/merge/publish cuda13.3 tags.
• Update third_party/Makefile CUTLASS selection to use v4.7.0 when CUDA_VER >= 13.3.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
third_party/Makefile	Adds a CUDA 13.3+ conditional branch to select CUTLASS v4.7.0.
dockerfile/cuda13.3.dockerfile	Introduces the CUDA 13.3 image based on nvcr.io/nvidia/pytorch:26.07-py3 with updated dependency versions.
.github/workflows/build-image.yml	Switches CI build/merge matrices from cuda13.0 to cuda13.3 tags/dockerfile.

Suppressed comments (1)

third_party/Makefile:47

• git clone ... && cd cutlass is misleading/no-op here: each recipe line runs in its own shell, and the later build uses -S ./cutlass anyway, so the cd cutlass has no effect on subsequent steps. Consider removing the trailing && cd cutlass to avoid implying later commands run from inside the repo.

	$(eval ARCHS := "100;103")
	if [ -d cutlass ]; then rm -rf cutlass; fi
	git clone --branch v4.7.0 --depth 1 https://github.com/NVIDIA/cutlass.git && cd cutlass

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (2)

third_party/Makefile:48

• git clone ... && cd cutlass is misleading here: Make runs each recipe line in its own shell, so the cd cutlass does not affect subsequent commands (which already reference ./cutlass). Consider dropping the && cd cutlass suffix to avoid implying that later steps run inside the repo directory.

ifeq ($(shell echo $(CUDA_VER)">=13.3" | bc -l), 1)
	$(eval ARCHS := "100;103")
	if [ -d cutlass ]; then rm -rf cutlass; fi
	git clone --branch v4.7.0 --depth 1 https://github.com/NVIDIA/cutlass.git && cd cutlass
else ifeq ($(shell echo $(CUDA_VER)">=12.9" | bc -l), 1)

dockerfile/cuda13.3.dockerfile:162

• This Dockerfile changes the established pattern of pinning setuptools to an exact version (e.g., cuda13.0.dockerfile and cuda12.9.dockerfile use setuptools==...). Using a lower-bound specifier (setuptools>=...) makes image builds non-reproducible because the resolved version can change over time. Consider pinning to a specific known-good version (e.g., the base image’s current setuptools) and bump it deliberately when needed.

RUN python3 -m pip install --upgrade "setuptools>=78.1.1" && \
    python3 -m pip install --no-cache-dir .[nvworker] && \

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.05%. Comparing base (28fe025) to head (01ca815).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #846      +/-   ##
==========================================
+ Coverage   86.02%   86.05%   +0.03%     
==========================================
  Files         103      103              
  Lines        7950     7996      +46     
==========================================
+ Hits         6839     6881      +42     
- Misses       1111     1115       +4     

Flag	Coverage Δ
cpu-python3.10-unit-test	70.99% <ø> (+0.11%)	⬆️
cpu-python3.12-unit-test	70.88% <ø> (ø)
cpu-python3.7-unit-test	70.31% <ø> (ø)
cuda-unit-test	83.95% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

third_party/Makefile:48

• The new CUDA_VER>=13.3 branch duplicates the same ARCHS/rm/clone recipe used for the >=12.9 branch, which increases maintenance cost and makes future tag/arch updates easy to miss. Consider setting CUTLASS_TAG (and ARCHS) inside the conditional branches, then running the shared rm/clone logic once after the conditional (or factoring a small helper target).

# The CUTLASS tag is picked per CUDA_VER because each release only supports the SM archs of its own CUDA generation.
cuda_cutlass:
ifeq ($(shell echo $(CUDA_VER)">=13.3" | bc -l), 1)
	$(eval ARCHS := "100;103")
	if [ -d cutlass ]; then rm -rf cutlass; fi
	git clone --branch v4.7.0 --depth 1 https://github.com/NVIDIA/cutlass.git && cd cutlass
else ifeq ($(shell echo $(CUDA_VER)">=12.9" | bc -l), 1)

[gusui-msft]

@microsoft-github-policy-service agree company="microsoft"

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

third_party/Makefile:47

• The && cd cutlass at the end of this recipe line is misleading/no-op: each Make recipe line runs in its own shell (no .ONESHELL here), and the later build already uses -S ./cutlass. Dropping the cd avoids implying subsequent lines run inside the repo directory.

	git clone --branch v4.7.0 --depth 1 https://github.com/NVIDIA/cutlass.git && cd cutlass

dockerfile/cuda13.3.dockerfile:138

• This UCX build leaves the downloaded tarball and extracted source tree under /tmp, which increases final image size and can make layer caching less effective. Consider removing the UCX tarball/source directory in the same RUN layer after make install.

    tar xzf ucx-${UCX_VERSION}.tar.gz && \
    cd ucx-${UCX_VERSION} && \
    ./contrib/configure-release-mt --prefix=/usr/local && \
    make -j ${NUM_MAKE_JOBS} && \
    make install

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (1)

.github/workflows/build-image.yml:34

• These entries remove the last CUDA 13.0 build and make the unvalidated CUDA 13.3 image the only CUDA 13 artifact, while the PR's own checklist still leaves both architecture builds, representative runs, and rescans unchecked. If the simultaneous CUDA/PyTorch/OpenMPI/UCX changes fail, this replaces the known-good CI artifact rather than introducing a canary. Please keep cuda13.0 in the matrix or gate this replacement on successful validation of both architectures and representative scans/runs.

        - name: cuda13.3-arm64
          dockerfile: cuda13.3
          tags: superbench/main:cuda13.3-arm64

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.