Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .github/workflows/modal-deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,13 @@ jobs:
env:
MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
# Set to 0 for rollback; ordinary submissions use the shared cache automatically.
KERNELBOT_NVCC_TOOLCHAIN: ${{ vars.KERNELBOT_NVCC_TOOLCHAIN || '1' }}
KERNELBOT_NVCC_PCH: ${{ vars.KERNELBOT_NVCC_PCH || '1' }}
KERNELBOT_NVCC_PCH_PROFILES: ${{ vars.KERNELBOT_NVCC_PCH_PROFILES }}
KERNELBOT_NVCC_TOOLCHAIN_VOLUME: ${{ vars.KERNELBOT_NVCC_TOOLCHAIN_VOLUME || 'kernelbot-nvcc-toolchains' }}
KERNELBOT_NVCC_PCH_VOLUME: ${{ vars.KERNELBOT_NVCC_PCH_VOLUME || 'kernelbot-nvcc-pch' }}
KERNELBOT_PCH_VOLUME: ${{ vars.KERNELBOT_PCH_VOLUME || 'kernelbot-torch-pch' }}

steps:
- name: Checkout Repository
Expand All @@ -33,6 +40,11 @@ jobs:
run: |
PYTHONPATH=${{ github.workspace }}/src:${{ github.workspace }}/src/runners modal run src/runners/modal_runner.py::warm_pch

- name: Warm NVCC framework header cache
if: env.KERNELBOT_NVCC_TOOLCHAIN == '1'
run: |
PYTHONPATH=${{ github.workspace }}/src:${{ github.workspace }}/src/runners modal run src/runners/modal_runner.py::warm_nvcc_pch --profiles "$KERNELBOT_NVCC_PCH_PROFILES"

- name: Deploy job
run: |
# TODO: Remove pythonpath here
Expand Down
34 changes: 33 additions & 1 deletion docs/modal-pch-cache.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@ The runner sets `CXX` to a GCC wrapper that loads a precompiled
changes. Leave `use_pch=False`; PyTorch's separate native PCH option is unnecessary.
The submission path assumes `load_inline()`.

Only the Torch host header is cached. CUDA/CUTLASS templates, user headers, kernel
NVCC also caches the implicit CUDA-side framework headers by default (see below).

The host cache only precompiles the Torch host header. CUDA/CUTLASS templates, user headers, kernel
objects, and shared libraries still compile in each submission container.
Different sources and module names share a PCH when their host flags match;
changing a user header after the Torch include still changes the compiled code.
Expand All @@ -28,6 +30,36 @@ speedup. Set `CXX=/usr/bin/g++` before building to bypass the wrapper entirely.

## Deployment

### CUDA-side framework cache

The runner uses NVIDIA's checksum-pinned NVCC 13.4.99 drop with the complete CUDA
13.4 toolkit. `PYTORCH_NVCC` automatically reuses `torch/types.h`, `cuda.h`, and
`cuda_runtime.h` from `kernelbot-nvcc-pch`. CUTLASS and user headers stay outside
this cache. The trusted warmer builds 17 profiles from five observed flag tuples;
submission workers mount them read-only. Unknown flags, custom compiler inputs,
and sources without the implicit header prefix compile normally. Corrupt caches
fall back to normal compilation. CPU artifact replay checks compiler identity
and user-header dependencies.

Upload the private compiler drop once, without committing binaries:

```sh
PYTHONPATH=src/runners uv run python scripts/upload_nvcc_toolchain.py --source /private/path/to/binaries
```

CI warms both caches before deployment. `KERNELBOT_NVCC_PCH_PROFILES` selects a
comma-separated subset; empty warms all 17. Manual warmup uses
`modal run src/runners/modal_runner.py::warm_nvcc_pch --profiles ...`.
Set `KERNELBOT_NVCC_PCH=0` to disable device PCH. Set both
`KERNELBOT_NVCC_TOOLCHAIN=0` and `KERNELBOT_NVCC_PCH=0` to restore CUDA 13.3.
`KERNELBOT_NVCC_PCH_DISABLE=1` also disables lookup within a submission.
The corresponding `KERNELBOT_NVCC_TOOLCHAIN_VOLUME`, `KERNELBOT_NVCC_PCH_VOLUME`,
and `KERNELBOT_PCH_VOLUME` variables override the private input, device, and host
volumes. Retain old cache generations for rollback and replace workers after
warming so they see the new Volume snapshot.

### Host cache

The main/dev workflow warms profiles before deploying GPU runners. Manually,
run both commands in the same Modal profile/environment:

Expand Down
52 changes: 52 additions & 0 deletions scripts/upload_nvcc_toolchain.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
"""Upload the checksum-pinned NVIDIA drop to KernelBot's private Modal volume.

PYTHONPATH=src/runners python scripts/upload_nvcc_toolchain.py --source /path/to/binaries
This does not deploy a runner or publish the binaries outside the Modal workspace.
"""

import argparse
import hashlib
import json
import os
import tempfile
from pathlib import Path

import modal
from nvcc_toolchain import BINARY_SHA256, RELEASE


def upload(source: Path, volume_name: str):
for name, expected in BINARY_SHA256.items():
if hashlib.sha256((source / name).read_bytes()).hexdigest() != expected:
raise ValueError(f"Compiler checksum mismatch: {name}")
manifest = json.dumps({"release": RELEASE, "sha256": BINARY_SHA256}, sort_keys=True) + "\n"
volume = modal.Volume.from_name(volume_name, create_if_missing=True)
manifest_path = f"/{RELEASE}/manifest.json"
try:
existing = b"".join(volume.read_file(manifest_path)).decode()
except FileNotFoundError:
existing = None
if existing is not None:
if existing != manifest:
raise ValueError("Existing compiler release has a different manifest")
print(f"{RELEASE} is already present in {volume_name}")
return
with tempfile.TemporaryDirectory() as directory:
metadata = Path(directory) / "manifest.json"
metadata.write_text(manifest)
with volume.batch_upload(force=False) as batch:
for name in BINARY_SHA256:
batch.put_file(source / name, f"/{RELEASE}/{name}")
batch.put_file(metadata, manifest_path)
print(f"Uploaded verified {RELEASE} to private volume {volume_name}")


if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--source", type=Path, required=True)
parser.add_argument(
"--volume",
default=os.environ.get("KERNELBOT_NVCC_TOOLCHAIN_VOLUME", "kernelbot-nvcc-toolchains"),
)
args = parser.parse_args()
upload(args.source, args.volume)
97 changes: 95 additions & 2 deletions src/libkernelbot/inline_artifacts.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,96 @@
import json
import os
import platform
import re
import subprocess
import sysconfig
from pathlib import Path

NVCC_FINGERPRINT = Path("/opt/kernelbot-pch/nvcc-fingerprint")
NVCC_PCH_ROOT = Path("/kernelbot-nvcc-pch")
MAX_CACHE_METADATA_BYTES = 64 * 1024


def _digest(value: bytes) -> str:
return hashlib.sha256(value).hexdigest()


def nvcc_fingerprint() -> str | None:
"""Bind artifacts to the immutable compiler/wrapper manifest, when installed."""
try:
contents = NVCC_FINGERPRINT.read_bytes()
except FileNotFoundError:
return None
if not contents.strip():
raise RuntimeError("NVCC toolchain fingerprint is empty")
return _digest(contents)


def _small_cache_file(path: Path) -> bytes:
if path.is_symlink() or path.stat().st_size > MAX_CACHE_METADATA_BYTES:
raise RuntimeError("Invalid NVCC PCH cache metadata")
return path.read_bytes()


def _pch_dependency_digest(path: Path) -> str:
"""Use a trusted, immutable cache manifest instead of rehashing GB-sized PCHs."""
path = path.resolve()
root = NVCC_PCH_ROOT.resolve()
if not path.is_relative_to(root):
raise RuntimeError("PCH dependency is outside the trusted NVCC cache")
parts = path.relative_to(root).parts
if (
len(parts) != 4
or not all(re.fullmatch(r"[0-9a-f]{64}", part) for part in parts[:2])
or parts[2] != "pch"
):
raise RuntimeError("Invalid NVCC PCH dependency path")
generation, key, _, _ = parts
if generation != NVCC_FINGERPRINT.read_text().strip():
raise RuntimeError("NVCC PCH dependency belongs to a different toolchain")
entry = root / generation / key
manifest_bytes = _small_cache_file(entry / "manifest.json")
manifest = json.loads(manifest_bytes)
if (
manifest.get("version") != 1
or manifest.get("generation") != generation
or manifest.get("key") != key
or manifest.get("header_sha256") != _digest(_small_cache_file(entry / "gpumode.cuh"))
):
raise RuntimeError("NVCC PCH dependency manifest does not match its cache entry")
files = manifest.get("files", [])
if len(files) < 2:
raise RuntimeError("NVCC PCH dependency manifest is incomplete")
frontends, names = set(), set()
for item in files:
name, size = item.get("path", ""), item.get("bytes")
match = re.fullmatch(r"pch/(cicc|cudafe)_[A-Za-z0-9_]+\.pch", name)
if not match or name in names or type(size) is not int or size <= 0:
raise RuntimeError("Invalid NVCC PCH dependency file metadata")
cache_file = entry / name
if (
cache_file.resolve() != cache_file
or not cache_file.is_file()
or cache_file.stat().st_size != size
):
raise RuntimeError("NVCC PCH dependency file changed")
names.add(name)
frontends.add(match[1])
if frontends != {"cicc", "cudafe"} or path.relative_to(entry).as_posix() not in names:
raise RuntimeError("NVCC PCH dependency is not covered by the cache manifest")
return f"nvcc-pch-v1:{_digest(manifest_bytes)}:{path.stat().st_size}"


def _dependency_digest(path: Path) -> str:
if path.suffix == ".pch":
return _pch_dependency_digest(path)
digest = hashlib.sha256()
with path.open("rb") as source:
for block in iter(lambda: source.read(1024 * 1024), b""):
digest.update(block)
return digest.hexdigest()


def _save_module(directory: Path, module, work: Path) -> None:
directory.mkdir(exist_ok=True)
binary = Path(module.__file__).read_bytes()
Expand Down Expand Up @@ -64,7 +145,7 @@ def _dependencies(build: Path, work: Path) -> dict[str, str]:
if path.parent == build.resolve() and path.name in {"main.cpp", "cuda.cu", "sycl.sycl"}:
continue # Generated from the source strings already in the request key.
name = str(path.relative_to(work)) if path.is_relative_to(work) else str(path)
dependencies[name] = _digest(path.read_bytes())
dependencies[name] = _dependency_digest(path)
return dependencies


Expand All @@ -73,7 +154,7 @@ def _check_dependencies(dependencies: dict[str, str] | None, work: Path) -> None
raise RuntimeError("Artifact has no compiler dependency information")
for name, digest in dependencies.items():
path = work / name
if _digest(path.read_bytes()) != digest:
if _dependency_digest(path) != digest:
raise RuntimeError(f"Build dependency changed: {path}")


Expand Down Expand Up @@ -124,18 +205,30 @@ def dispatch(*args, **kwargs):
current_runtime = {
**runtime,
"cuda_home": getattr(cpp_extension, "CUDA_HOME", None),
"nvcc_toolchain": nvcc_fingerprint(),
"build_environment": {
name: os.environ.get(name)
for name in (
"TORCH_CUDA_ARCH_LIST",
"CXX",
"CC",
"PYTORCH_NVCC",
"CUDA_HOME",
"CUDA_PATH",
"CUDA_ROOT",
"CPATH",
"CPLUS_INCLUDE_PATH",
"C_INCLUDE_PATH",
"GCC_EXEC_PREFIX",
"COMPILER_PATH",
"PATH",
"NVCC_CCBIN",
"NVVMIR_LIBRARY_DIR",
"NVCC_PREPEND_FLAGS",
"NVCC_APPEND_FLAGS",
"KERNELBOT_NVCC_PCH",
"KERNELBOT_NVCC_PCH_DISABLE",
"KERNELBOT_NVCC_PCH_ROOT",
"LIBRARY_PATH",
"LD_LIBRARY_PATH",
)
Expand Down
5 changes: 3 additions & 2 deletions src/libkernelbot/python_precompile.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
from pathlib import Path

from libkernelbot.consts import Timeout
from libkernelbot.inline_artifacts import prepare_environment
from libkernelbot.inline_artifacts import nvcc_fingerprint, prepare_environment
from libkernelbot.run_eval import CPUCompileInfo

MAX_ARTIFACT_BYTES = 64 * 1024 * 1024
Expand Down Expand Up @@ -67,11 +67,12 @@ def should_precompile(config: dict) -> bool:

def build_identity(config: dict) -> dict:
return {
"version": 2,
"version": 3,
"sources": hashlib.sha256(
json.dumps(config["sources"], sort_keys=True).encode()
).hexdigest(),
"image": Path("/opt/kernelbot-pch/fingerprint").read_text().strip(),
"nvcc_toolchain": nvcc_fingerprint(),
}


Expand Down
Loading
Loading