CollectiveX MVP v0.1 by Oseltamivir · Pull Request #2004 · SemiAnalysisAI/InferenceX

Oseltamivir · 2026-07-03T17:38:00Z

Summary

CollectiveX v1 expert-parallel communication benchmark and its isolated GitHub Actions publication path.

Benchmark and publication contract

Covers eight SKUs, six backend families, EP8/EP16, normal and genuine low-latency semantics.
Locks 58 shards, 608 requested cases / 1,600 points, including 364 runnable and 244 explicit unsupported cases.
Standardizes every measured component at 32 synchronized full-roundtrip warmups and 64 x 8 timed iterations: 512 cross-rank-max observations.
Defaults every sweep to release_tag=unversioned; only an explicit v1 tag with the locked full-matrix digest emits a release marker.
Uses no results server, Vercel storage, GCP, Neon, managed database, or managed object store.

Validation

151 Python contract/unit tests pass.
Matrix SHA-256: f1ca85f9689922b90edd5767b9ff2a902f6b896f32f68b2ca086dde3fd2157d0.
Case catalog SHA-256: 8e262178f770b0cdde12b7ec71604afd87251fa55685d4594f29717153ad6bbd.
Regeneration confirms 58 shards, 364 runnable cases, 244 unsupported cases, and 1,600 points.
Actionlint, ShellCheck, and git diff --check pass.

中文说明

本 PR 完成 CollectiveX v1 专家并行通信基准测试及隔离式 GitHub Actions 发布链路。代码已准备执行三轮完整、无 canary 的 V1 运行；本 PR 目前不声称已有完成 promotion 的实测结果。

基准测试与发布约定

覆盖八种 SKU、六类后端、EP8/EP16，以及常规与真正低延迟语义。
固定为 58 个 shard、608 个请求 case / 1,600 个点位，其中包含 364 个可运行 case 和 244 个显式 unsupported case。
每个被测组件统一执行 32 次同步完整往返预热和 64 x 8 次计时迭代，共 512 个跨 rank 最大值观测。
所有扫描默认使用 release_tag=unversioned；只有显式选择 v1 且匹配固定完整矩阵摘要时才生成 release marker。
发布必须使用来自同一 source SHA 的三个成功且带 tag 的运行。手动 publisher 会重新校验运行 metadata 与精确 marker，在 runner 临时存储中完成 promotion，并仅保留清理后的 cxpublication-v1-* NDJSON。
不使用 results server、Vercel storage、GCP、Neon、managed database 或 managed object store。

验证

151 个 Python 约定/单元测试通过。
Matrix SHA-256：f1ca85f9689922b90edd5767b9ff2a902f6b896f32f68b2ca086dde3fd2157d0。
Case catalog SHA-256：8e262178f770b0cdde12b7ec71604afd87251fa55685d4594f29717153ad6bbd。
重生成确认 58 个 shard、364 个可运行 case、244 个 unsupported case 和 1,600 个点位。
Actionlint、ShellCheck 与 git diff --check 通过。

mi325x-mori legs failed at ep_mori.py: 'MoRI has no pinned GPU architecture for runner mi325x'. The re-add (36b0830) restored the capability/config/ launcher/common.sh groupings but missed the per-runner MoRI GPU-arch pin — mi325x is gfx942 (like mi300x), so add it to that branch. Comment updated to note MI300X/MI325X both use the AsyncLL XGMI kernel.

…perator network overlay)

…EP16 Track per-SKU scale-out RDMA fabric selectors in configs/network-config.json (un-gitignored) and overlay them onto the base operator config via config.py:_network_overlay — SKUs absent there fall back to the network secret, a missing/invalid file is a no-op. mi355x internode selectors (rdma0-7, gid 1, eno0) were hardware-validated (16/16 ranks, 4 QP/PE), so drop the mi355x×mori×16 override — its EP16 leg is now schedulable. mi300x/mi325x EP16 stay walled (no validated internode selectors: mi300x debug cluster node-boot broken, mi325x GHA nodes on an unreachable pod cluster). 26 tests green.

…twork-config.json) mi355x internode selectors now ship in configs/network-config.json and are overlaid by config.py, so the mi355x×mori×16 override is no longer needed — its EP16 leg is schedulable. mi300x/mi325x EP16 remain walled (no validated internode selectors).

Replace deepep-hybrid with a vendor-neutral nccl-ep backend and finish the backend-registry consolidation into configs/backends.json. nccl-ep (bench/ep_nccl_ep.py): the collective-library EP baseline on RCCL (AMD) / NCCL (NVIDIA). Not the naive equal-split all_to_all_single — a routing-aware token-rank variable-split all-to-all that satisfies the oracle contract: dispatch sends each token once per distinct destination rank (top-k row masked to that rank), combine is the reverse variable all-to-all + unweighted rank-sum. No source pin or build; runs on any image collective. Schedulable on both vendors, all SKUs, EP8/EP16. deepep-hybrid removed entirely: adapter, capability entry + b200 EP16 override, run_ep choices/import, backends.json entry, config.py registry emit, common.sh (registry names, probe, source-pin, canonical unset), prepare_backend.sh (both build fns + dispatch + persisted env), and the ep_harness label map + tests. Backend source pins + images now live in configs/backends.json, validated by config.py backend-registry and consumed by common.sh (fail-closed). 30 tests + 6 subtests green; no hybrid refs remain in code.

The per-SKU launchers rejected nccl-ep at setup (unsupported EP backend): add nccl-ep to the allowlists (single-slurm + gb-nv: deepep-v2|nccl-ep; mi-amds: mori|nccl-ep) and keep pinned-source staging deepep-v2-only (nccl-ep has no source build).

… provenance The commit uniquely determines its tree and submodule gitlinks, so rev-parse HEAD at source fetch already enforces them — the separate tree pin was redundant. The nccl-check upstream_commit was provenance-only (the patch is applied by stage.py rewrite-deepep-v2, which needs no hash). Remove both from configs/backends.json, the config.py registry emit, common.sh (required names + source-pin format repo|commit|fmt), prepare_backend.sh (cache key + DEEPEP_TREE export/persist), and the registry test. 30 tests + 6 subtests green.

Rip out the nccl-ep backend entirely (adapter, capability entry, run_ep choices/import + dead dist-init branch, launcher gates, common.sh + prepare_backend probes, ep_harness label map, workflow choice, tests). The matrix is back to deepep-v2 + mori. Plus dead-code/slop removal (reviewed, no remaining consumers): drop unused EPBackend oracle_layout/payload_unit + redundant WorkloadSpec fields (routing/seed/hidden/topk/experts/num_logical) + RankInputs.global_tokens; mori dead label/tuning vars; routing SOURCE_ID_CONTRACT + token_offset; probe verify_cache_mount; numpy from requirements; README/methodology trim. 30 tests + 6 subtests green; matrix generates deepep-v2 (9) + mori (4).

Move the timing profile, routing seed, and measured + conditioning token ladders out of ep_harness constants into configs/suites.yaml as the sole source; the matrix bakes them into every scheduled case and run_ep receives them only through required CLI args (no module defaults, no cross-check guards). Decode ladder extended to 512 tok/rank; prefill densified to 256..2048. Dedupe the oracle expert-transform coefficients and column pattern between transform and expectation; unify the oracle report shape.

…e operator-config merge Drop artifact fields nothing consumes: the top-level case object, outcome.validity, shape, realized_placement/device_count, oracle ordering_contract/order_stable, and the scheduled-case canonical/timing/samples/warmup_semantics duplicates (all still required fields survive; the frontend keys off identity.case_factors.case). Move the operator-config overlay merge from the workflow's inline python into config.py merge-operator-config (one tested codec); drop backends.json documentation-only schema_version/notes/platforms/patches and the stage ownership tag. Update methodology.md to the emitted shape.

…step The merge-operator-config step runs before the launcher cd's into $COLLX_SOURCE_ROOT, in a cwd where the relative experimental/CollectiveX path does not exist (the sweep source is staged under $COLLX_SOURCE_ROOT, not checked out into the workspace). The prior inline heredoc needed no file path; the extracted subcommand did. Reference config.py by its absolute staged path, matching the Extract step.

…ioning ladder The untimed warm-up now walks the measured token ladder ascending (8 rounds per shape, cold-jump-safe) and reuses those problems in Pass 1, instead of a distinct conditioning ladder. Removes conditioning_ladders from suites.yaml, the --conditioning-ladder argv (config.py both codecs), the WorkloadSpec conditioning fields, and retain_global (every shape retains its global trace). create_buffer sizes from the measured maximum (the old value folded in the ramp but never exceeded that max, so the JIT dir is stable). Docs + tests updated.

… h100 deepep-v2 EP8 Consolidate per-SKU platform identity, placement, operator-field requirements, canonical policy, and scale-out RDMA selectors into one tracked configs/platform_config.json. capability.py, runtime/config.py, the launchers, and prepare_backend.sh all derive from it; suites.yaml 'platforms: all' resolves to the registry. Removes the separate network-config.json (selectors move to the mi355x 'network' block) and the gitignored platforms.yaml indirection. Falsify the h100-dgxc deepep-v2 wall for EP8 (on-node ElasticBuffer + dispatch/ combine validated 2026-07-11): schedule EP8 (LSA), keep EP16 walled pending a GIN-over-RoCE 2-node test. MoRI kernel type is now derived by the adapter from (arch, scope) rather than an env var; block/warp tuples unchanged. Routing counter uses keyed BLAKE2b to match the documented oracle. Shared launcher prologue/tail + registry-driven topology validation.

Adding .shards to stage.py EXCLUDES (in 2424fb4) dropped the per-leg control JSON (COLLX_SHARD_FILE=.shards/<leg>.json) from the compute-visible staged tree. The cross-node preflight then failed 'test -r shard' -> exit 11 -> repository-stage failure on EVERY leg (all SKUs, both backends). Remove .shards from EXCLUDES; keep .venv/.pytest_cache (local cruft, absent on runners). Extend StageTests to assert .shards/<file> is staged.

…platform_config.json Operator decision (2026-07-12): move runner-local operator config (partition, account, qos, squash_dir, exclude_nodes) and scale-out RDMA selectors from the GHA secrets into the tracked platform_config.json 'operator'/'network' blocks, derived from live cluster state + a recovered manual config. An operator config document, when present, still overrides per field; path '-' (no document, known SKU) uses the registry baseline. collx_load_operator_config falls back to the registry-only sentinel when no operator file exists but the SKU is known. Complete baselines (run without secrets): h100, h200, b200, b300, mi355x. Still secret-fed: gb200 (no path), mi300x/mi325x (unreachable at derivation), gb300 (squash_dir/enroot_cache_path not recoverable). audit_salt stays secret.

The 'run without secrets' path requires this: deleting the base secret makes COLLECTIVEX_OPERATOR_CONFIG_CONTENT empty, and json.loads('') crashed merge before the registry-baseline fallback was reached, failing every leg at merge. Treat an empty/blank base as {"runners":{}} so the merged file is still written (launcher finds a config -> uses the tracked platform_config.json operator baseline). A present base is still parsed strictly; overlays already skip empty. Verified: complete-baseline SKUs emit full config; gb200/gb300 fail-closed with validation-missing markers.

…implification Add tracked operator baselines (discovered on-cluster via SSH) so these run without secrets: gb200 partition=batch, account=benchmark, storage_roots=[/mnt/lustre01/users-public/sa-shared] gb300 + squash_dir=/data/home/sa-shared/collectivex/containers, enroot_cache_path=/data/home/sa-shared/collectivex/enroot-cache mi300x partition=compute, squash_dir=/home/bshan/collectivex/containers mi325x stays secret-fed (no SSH access target; runners offline). Runtime simplification (staged-tree cross-node preflight, canonical_policy, and registry run_timeout/master_port removed; run_timeout/master_port now defaulted in launchers/common.sh; shard control read from source, so .shards is excluded from the staged copy). Docs/tests reconciled: README documents the operator baseline + per-field secret overlay; skips_secret_fed_sku test points at mi325x.

…300); revert mi300x baseline b300: the runtime refactor dropped the per-path 'git -c safe.directory' guard the old collx_git wrapper applied, so on b300's UID-mapped Actions shell (NFS realizes the staged source dir as UID 0) git aborts the DeepEP fetch with 'dubious ownership'. Restore a job-scoped 'git config --global --add safe.directory *' in collx_prepare_deepep_source (ephemeral per-job HOME; reaches the fmt submodule child git; runner UID is inside the trusted cluster boundary). mi300x: revert the operator baseline. The reconstructed squash_dir was wrong (compute nodes have root-owned /home; only the submit host had /home/bshan), and the cluster's node-boot blocker ('nodes still not ready') prevents probing a compute node for the correct path. mi300x stays secret-fed/blocked like mi325x.

…anual-run fallbacks Every case now arrives from the matrix with fully-specified argv, so make --mode/--phase/--routing/--case-id/--suite/--workload-name/--gpus-per-node/ --scale-up-domain required and remove the getattr/'or ep_size'/'manual' fallbacks in run_ep/ep_harness/ep_deepep_v2/ep_mori/ep_backend (drop --timestamp). Simplify collx_run_shard/collx_launcher_cleanup control flow (equivalent). Verified: 28-case emit->parse seam all valid with non-zero placement; 32 tests.

…atform_config backends field Each SKU now declares its runnable {backend: [ep_sizes]} in platform_config.json; sweep_matrix derives topology from placement and resolves runnable/unsupported from that map, replacing capability.py (deleted) and the separate backends.json (deleted; DeepEP source pins + image tags now inline constants in common.sh). Drop vendor/machine/operator_fields (unused after this); launchers self-declare the operator fields they require via collx_require_vars; ep_mori derives arch from platform_config directly. Verified: 28 tests; py_compile/bash -n; all collx_* + config.py subcommands resolve; matrix scheduling matches each SKU backends map (h100/b300/mi300x/mi325x EP8-only, rest full); 28-case emit->parse argv seam valid; operator emit + launcher require_vars guard cleanly. Docs (README/methodology) repointed off backends.json.

…/point Fully de-secrete the runner config: drop the 7 COLLECTIVEX_*_CONFIG_CONTENT secret env vars + COLLECTIVEX_OPERATOR_CONFIG_REQUIRED and the whole merge-operator-config machinery (workflow step, config.py merge_operator_config + OPERATOR_ENV, common.sh secret-content/ephemeral handling). collx_load_operator_config now falls straight to the tracked platform_config.json baseline ("-" sentinel) under canonical GHA instead of requiring a merged secret file; a baseline-less SKU emits nothing and fails cleanly at the launcher's require_vars. Verified end-to-end by simulating a real leg (h200 -> baseline emitted, no die; mi325x -> clean miss). suites.yaml timing trials_per_point 128->256 (2048 obs/component, tighter p99); README + methodology updated to match. NOTE: version left at 1 (operator's call). prepare_backend.sh: cosmetic reformatting only. Frontend (separate repo, already pushed) verified to ingest this artifact contract: tsc clean, 46 tests pass, all required fields present.

b300 nodes have the gdrdrv module + nvidia_peermem loaded and a complete RoCE network block (bond0 + RoCE rdma_devices + gid 3), so NVSHMEM-IBGDA can register GPU memory via peermem; the prior '/dev/gdrdrv absent' exclusion may be stale. Un-wall b300 EP16 (backends [8,16], drop its CELL_EXCLUSIONS entry) so the sweep validates it on the metal. h100/mi300x/mi325x EP16 stay walled (no network block / infra-blocked); mi355x EP16 remains a real ionic MR-cap hardware wall.

…ll, BF16-faithful combine oracle Config/runtime: - Move container image pins out of common.sh into per-SKU image/image_platform in platform_config.json, plumbed via config.py PLATFORM_FIELDS and required by every launcher; registry-only SKUs (mi325x) still receive image config. - collx_apply_network_profile pins NCCL_IB_MERGE_NICS=0 on every scale-out (dual- port fusion disables NCCL GIN -> deep_ep EP16 asserts railedGinType==NONE) and NCCL_CROSS_NIC=0 when rail_isolated=1. Un-walls h100 + b300 deepep-v2 EP16 (validated 16/16 on-metal). h100 gains RoCE selectors + network block; b300 gains rail_isolated. - Bump container-import lock wait 900s->2700s with an explicit timeout message (a concurrent leg holds it ~18min for the b300 squash). - CELL_EXCLUSIONS: only mi300x/mi325x mori EP16 remain (deleted network block + cluster/access), with detailed reasons and a walled-vs-red doc. Correctness oracle: - Derive the expected combine from the values actually communicated: cast each destination rank's FP32 aggregate to BF16 before summing, exactly as the adapter does, so the gate models real BF16 accumulation instead of hiding it in a wide allclose. Tighten to max elementwise relative error < 8*2^-8 (mag floor 0.02); drop the loose rtol=0.05/atol=0.02 allclose and the global max_rel. - benchmark_stage drains dispatch with an untimed combine when dispatch_needs_combine_cleanup (MoRI), matching benchmark_dispatch. - Multi-point run summary (ladder ends + interior) instead of one mid-ladder row. - methodology.md + tests updated (27 pass).

…EP16 passes the tight gate The tightened combine oracle (2433548) cast each destination rank's aggregate to BF16 and summed in FP32, but omitted the per-scale-up-domain BF16 cast the backend applies to each node's partial before the scale-out send. Single-domain groups (every EP8 case + MNNVL EP16, where ep_size <= scale_up_domain) were unaffected, but multi-node RoCE EP16 (ep_size 16 > scale_up_domain 8, two domains) landed a deterministic 0.0478 off the expectation -- above the 8*2^-8=0.03125 gate -- reddening h100/h200/b200/b300 EP16 that are otherwise correct (run 29203408512: all four failed 'semantic correctness', identical max_rel, while gb200/gb300 EP16 over MNNVL passed at ~0.005). _expected_transformed_combine now reproduces the two-level reduction: per-rank BF16 output, FP32 accumulation within a scale-up domain, a BF16 cast of each domain aggregate for the scale-out send, then FP32 summation of the domain partials. Thread scale_up_domain through _run_expert_oracle. A numpy BF16 simulation confirms the multi-domain residual drops ~7x below the gate while single-domain cases become near-exact; the bound is unchanged. Docs updated.

… wall model, MoRI IntraNode-only on CDNA Config: replace configs/suites.yaml (YAML) with a flattened configs/sweep.json (single suite/workload), dropping the PyYAML dependency (requirements.txt gone) and the workflow's install step and 'suites' input. All readers are stdlib json. Prefill ladder moves to [1024,2048,4096,8192] (docs updated to match); version stays 1. Matrix: rewrite sweep_matrix.py (~400->223 lines). Walls are now purely the per-SKU 'backends' map -- a cell not in the map is 'unsupported' (never scheduled), a cell in the map schedules and is allowed to red. Removes the CELL_EXCLUSIONS table; test_matrix.py now asserts unsupported == exactly the (sku,backend,ep) absent from backends. config.py derives COLLX_* env names by convention (behavior-preserving) and splits OPERATOR_FIELDS/NETWORK_FIELDS. MoRI: drop the AsyncLL path; every CDNA scale-up cell uses the direct IntraNode kernel, scale-out EP16 uses InterNodeV1 (no low-latency-family kernel on either vendor). Size the communicator buffer to max(512, max_tokens_per_rank) instead of asserting <=512, so the larger prefill ladder runs. prepare_backend.sh: restructure into discovery/lifecycle/container-boundary helpers; same persisted rank-env contract (RANK_ENV_VARS), querying package roots via the venv interpreter, and persisting the deepep-v2 EP_SUPPRESS_NCCL_CHECK unset into the node env. 27 tests pass.

…tic heap The mi355x EP16 failure was misclassified as a physical ~1 GiB single-MR Pollara/Ionic wall. Falsified: the NICs advertise multi-GiB max_mr_size, direct uncached GPU-mem registration succeeds through 6-8 GiB, and Ionic multiplexes several firmware-command failures onto EINVAL (so the log proves only that the 6 GiB *static* registration failed during InterNodeV1 init, not an MR-size violation). MoRI already ships the supported path: VMM_HEAP backs the heap with on-demand 64 MiB DMA-BUF chunks (PR #155, validated on MI355X + AI NIC). Set MORI_SHMEM_MODE=VMM_HEAP in create_buffer, before shmem_torch_process_group_init, gated on self._inter_node — scale-out EP16 only, so scale-up EP8 (intranode, no RDMA registration) keeps the default static heap and its measurement basis is unchanged. The prefill ladder never enlarged the MR (MORI_SHMEM_HEAP_SIZE stays 6G, a virtual reservation under VMM). 27 tests pass.

…r this PR)

…the registry Addresses functionstackx's PR review: capture the NIC and switch per SKU so same-GPU clusters on different fabrics stay distinct registry entries (e.g. a second b200 cluster on a different NIC becomes its own entry with its own fabric + network block; the matrix already keys shards by SKU). Each platform now has a fabric {nic, switch} block, sourced from the gpu-specs page and cross-checked against the existing network selectors (h200 = Quantum-2 InfiniBand -> no RoCE gid; b200/b300 = Ethernet -> gid 3; mi355x = Pollara). test_runtime asserts every entry carries a non-empty nic and switch.

Oseltamivir requested a review from a team July 3, 2026 17:38

github-project-automation Bot added this to InferenceMAX Board Jul 3, 2026

claude Bot reviewed Jul 3, 2026

View reviewed changes

Comment thread experimental/CollectiveX/runtime/common.sh Outdated

Comment thread experimental/CollectiveX/env_capture.py Outdated

Comment thread experimental/CollectiveX/runtime/common.sh Outdated

Oseltamivir force-pushed the collectivex branch 4 times, most recently from 758fa52 to 1c5b901 Compare July 4, 2026 01:11

github-advanced-security AI found potential problems Jul 4, 2026

View reviewed changes

Comment thread experimental/CollectiveX/tests/test_sampling_contract.py Fixed

Oseltamivir force-pushed the collectivex branch 3 times, most recently from 7e5f80a to 28cbac4 Compare July 4, 2026 03:21

functionstackx changed the title ~~CollectiveX v1: cross-vendor EP benchmark suite~~ CollectiveX v1: cross-vendor EP benchmark suite / CollectiveX v1：跨厂商 EP 基准测试套件 Jul 4, 2026

Oseltamivir force-pushed the collectivex branch from 28cbac4 to aa318f7 Compare July 4, 2026 06:58

Oseltamivir force-pushed the collectivex branch 15 times, most recently from 4ff5841 to bffd8d6 Compare July 4, 2026 15:05

Oseltamivir added 28 commits July 14, 2026 15:04

chore(collectivex): gitignore configs/network-config.v1.json (local o…

d8526a9

…perator network overlay)

ci(collectivex): sweep backend choices — add nccl-ep, drop deepep-hybrid

5a990dd

chore(collectivex): drop AGENTS.md language override (out of scope fo…

03fc762

…r this PR)

Oseltamivir force-pushed the collectivex branch from 5a92dbd to 03fc762 Compare July 14, 2026 07:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CollectiveX MVP v0.1#2004

CollectiveX MVP v0.1#2004
Oseltamivir wants to merge 262 commits into
mainfrom
collectivex

Oseltamivir commented Jul 3, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Oseltamivir commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Benchmark and publication contract

Validation

中文说明

基准测试与发布约定

验证

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Oseltamivir commented Jul 3, 2026 •

edited

Loading