Repository navigation
packaging: the linux x86-64 rocm engine on pypi, npm, homebrew and install.sh - #92
Conversation
…stall.sh From v0.5.0 a release can carry basert-engine-linux-x64-rocm-<v>.tar.gz, the engine for AMD Strix Halo (gfx1151) and Strix Point (gfx1150) on ROCm 10. Without this, publish.yml broke on it: platform_tag.py tagged every ELF engine aarch64, so the ROCm wheel became a second manylinux_2_43_aarch64 wheel, test-linux's `pip install *manylinux*.whl` installed both and failed, and PyPI, npm and crates.io stopped for the whole release. The ROCm bundle is optional (release.yml ships without it when its AMD runner is off), so every channel follows what the release carries: - platform_tag.py reads the CPU from each ELF header (aarch64 / x86_64), skips GPU code objects, and refuses a folder built for two CPUs. - npm: @basecompute/basert-linux-x64-rocm (linux, x64, glibc), built only when the bundle is there; the main package's os/cpu follow the engines the release has; linux-x64 in both ENGINE_PACKAGES tables. - Homebrew: an on_linux Intel branch with the ROCm bundle and an apt caveat per chip; macOS stays arm64-only, and a release without the bundle keeps Linux arm64-only too. - publish.yml: the plan reads whether the release has the bundle; a ROCm test job on strix-halo-1 (pip, npm, a linked program) runs only then; PyPI/npm/crates accept it skipped only for a release without the bundle; test-linux installs only the aarch64 wheel; the wheel and npm lists gain the x86_64 wheel and the ROCm package. - install.sh: Linux x86_64 requires an amdgpu Strix GPU (/dev/kfd and a topology node with gfx_target_version 110501 or 110500; another AMD GPU is named and refused), warns with the chip's apt line when ROCm 10 or its BLAS kernels are missing, then takes the ROCm bundle. - README and the PyPI/npm READMEs name the AMD platform. Tests: platform tags per CPU; npm and Homebrew with and without the ROCm bundle; a real wheel build of a three-bundle release; install.sh on a fake / with stub uname/curl, sh -n and shellcheck; publish.yml's wheel and npm globs each match one package, its engine list matches build-packages.mjs, and the lab runners' labels stay out of workflows pull requests can start.
The engine's release pipeline now builds the ROCm bundle in an Ubuntu 24.04 + ROCm 10 image: libbaseRT needs GLIBC_2.38 and basert GLIBC_2.39, so its wheel is manylinux_2_39_x86_64 (platform_tag.py reads that from the binaries; no code change). The README asks for Ubuntu 24.04 or newer and a kernel AMD supports for RDNA3.5 (Ubuntu 24.04 HWE 6.17.0-19.19~24.04.2+, OEM 6.14.0-1018+, Ubuntu 26.04's own, elsewhere 6.18.4+). The test fixtures carry the bundle's real glibc versions.
IsuraManchanayake
left a comment
IsuraManchanayake
left a comment
There was a problem hiding this comment.
Codex Review
Reviewed 03f7754097cae6aa07a5da331f5f6fb6dc1a35ef against main at 78c88f05c065de9b0cb7acbfa319b67c05c2bf80, including all 14 changed files and the surrounding installation, library discovery, package staging and publishing paths. No existing reviews were present.
One P2 finding is attached to install.sh: mixed-AMD detection does not ensure the detected Strix GPU is the device the engine will use.
The CPU-specific wheel tags, ROCm npm package/discovery mapping, optional-bundle handling, Homebrew formula branches, and publish-job dependency conditions otherwise look consistent. The ten shared packaging/bindings/test files also match the corresponding engine-side changes byte-for-byte.
Validation:
- The local packaging suite already run on this exact, unchanged head passed 47/47, including actual wheel builds from synthetic bundles, npm package construction, formula generation and installer fixtures; I retained those results rather than repeating the unchanged suite.
- Additional mixed-AMD installer fixture (
gfx1100plusgfx1151, with the latter's runtime installed): exits 0, downloads the ROCm bundle, prints normal chat/serve instructions, and emits no device-selection warning. - Actionlint passes with the repository's custom runner labels configured and the pre-existing SC2254 warning excluded.
git diff --checkpasses; CI run 135 succeeded.- The README's kernel minimums agree with AMD's documented requirements.
Limits: no real ROCm inference or registry publication was run, and I did not verify the organization's runner-group permissions. The mixed-GPU runtime consequence follows from inspection of the engine's device-0 selection; the fixture verifies the installer's behavior, not a physical multi-GPU execution. The release-time runner-access and source-sync prerequisites in the PR description still apply.
| 110501) echo gfx1151; return 0 ;; | ||
| 110500) echo gfx1150; return 0 ;; |
There was a problem hiding this comment.
Codex Review — [P2] Surface the device-selection requirement on mixed-AMD hosts. These early returns find a supported Strix node anywhere in KFD, but they do not select that device for the installed engine. If HIP device 0 is an unsupported AMD GPU (for example gfx1100) and a later device is gfx1150/gfx1151, installation succeeds and prints ordinary basert chat/serve instructions, while the engine still opens device 0 and rejects it. I confirmed that the mixed-GPU installer fixture exits 0 with no warning; its test_a_strix_gpu_beside_another_is_used assertion only proves the bundle was downloaded. Preserve enough device identity to detect this case and either arrange supported-device selection or print actionable instructions for selecting it, respecting existing visibility settings. AMD documents the relevant visibility controls here. Finding a later KFD node alone should not be treated as proof that the engine will use it.
There was a problem hiding this comment.
Accepted, fixed in c8eaf36. install.sh now lets HIP device 0 decide, not any KFD node:
- GPU order.
kfd_gpusreads the topology nodes in number order, so node 10 no longer sorts before node 2 as it did with the glob.visiblethen narrows that list the way the ROCm runtime does:ROCR_VISIBLE_DEVICESfirst, thenHIP_VISIBLE_DEVICES(orCUDA_VISIBLE_DEVICES), following AMD's GPU isolation rules. A repeated index is dropped, an index past the end ends the list, and a UUID list is left unread. - Device 0 decides. If device 0 is a Strix GPU, nothing changes. If a Strix GPU sits behind another GPU, it is still installed for (the CLI works either way), with a warning:
the engine runs on HIP device 0, which is gfx1100 here; the gfx1151 is HIP device 1. make it device 0 wherever you run basert: export HIP_VISIBLE_DEVICES=1. The closing instructions repeat theexportline. If no Strix GPU is visible, it is refused and the message names device 0. - Tests.
test_a_strix_gpu_beside_another_is_usedbecametest_a_strix_gpu_behind_another_is_named_with_its_device_number, which checks the warning and the closing line. On the previous script it failed:'HIP device 0, which is gfx1100 here; the gfx1151 is HIP device 1' not found in ''.test_the_visibility_variables_choose_device_0covers the visibility variables:HIP_VISIBLE_DEVICES=1,ROCR_VISIBLE_DEVICES=1andCUDA_VISIBLE_DEVICES=1,0give no warning.HIP_VISIBLE_DEVICES=0is refused, because it leaves only the gfx1100; on the previous script this failed with0 != 1.- A UUID keeps the node order.
- Two more tests.
test_device_0_decidesandtest_topology_nodes_are_read_in_number_order(gfx1151 at node 2, gfx1100 at node 10).
tests/scripts/test_packaging.py is synced, so baseRT-internal#877 has the same bytes (112b71b10). I didn't choose the device automatically: install.sh doesn't run the engine, and an export written into shell profiles would change what other ROCm programs see.
…a Strix GPU behind another Review fix (Codex P2). The engine opens HIP device 0, but install.sh accepted a Strix GPU found anywhere in the KFD topology: with a gfx1100 as device 0 and a gfx1151 behind it, it installed and printed the usual instructions, and the engine then refused device 0. It now lists the AMD GPUs as HIP numbers them (topology nodes in number order, narrowed by ROCR_VISIBLE_DEVICES and then HIP_VISIBLE_DEVICES or CUDA_VISIBLE_DEVICES; a UUID list is left unread) and lets device 0 decide. A Strix GPU behind another one is still installed for, with a warning that names its device number, and the closing instructions repeat 'export HIP_VISIBLE_DEVICES=<n>'. Node directories are read in number order (node 10 sorts before node 2 as text).
|
Codex review of 03f7754: I accepted the one finding, and it's fixed in c8eaf36; details are in the thread. Checks run on macOS:
baseRT-internal#877 carries the same |
IsuraManchanayake
left a comment
IsuraManchanayake
left a comment
There was a problem hiding this comment.
Codex Review
Re-reviewed c8eaf366940ee859e8f242add3cf583a1b28e627, including the reply to the previous review and the related Superfluid changes. The new warning fixes the original unfiltered mixed-AMD example, but two P2 findings remain in the new visibility handling and recovery instructions, attached inline.
Validation:
- All 50 packaging tests passed across the initial run and the wheel-test retry. The isolated wheel build initially could not download its build dependencies under the network sandbox; with that access allowed, its three actual wheel builds passed.
- Added independent fixture probes for UUID selection, an existing reversed HIP list, applying the emitted recovery command, empty ROCR visibility, and disabled HIP visibility. These expose the cases described inline despite the suite being green.
- All ten shared packaging/bindings/test files still match their source counterpart in bytes and modes.
- Shellcheck, actionlint (custom runner labels configured; pre-existing SC2254 excluded), and the complete PR's
git diff --checkpass. - CI run 136 passed.
The probes run the actual installer against synthetic KFD/runtime fixtures. ROCm's expected selection semantics were checked against AMD's documentation and runtime implementation; I did not run physical multi-GPU inference or a live publish.
| lines="$(cat)" | ||
| list="$(printf '%s' "$1" | tr -d ' ')" | ||
| case "$list" in | ||
| "" | *[!0-9,]*) [ -z "$lines" ] || printf '%s\n' "$lines"; return 0 ;; |
There was a problem hiding this comment.
Codex Review — [P2] Honor UUID and zero-device masks before deciding which GPU is visible. Returning the unfiltered list here makes supported configurations fail the new admission check. With raw GPUs [gfx1100, gfx1151], ROCR_VISIBLE_DEVICES=<UUID of gfx1151> and HIP_VISIBLE_DEVICES=0, ROCm exposes the Strix GPU as device 0, but the actual installer fixture exits 1 with “not gfx1100”. With the UUID alone it instead recommends HIP index 1 although ROCR has exposed only index 0. The opposite error occurs for ROCR_VISIBLE_DEVICES="" or HIP_VISIBLE_DEVICES=-1: the fixture succeeds without a warning although no GPU is visible. AMD supports UUID selectors; ROCr's empty-list handling and HIP's parser also differ from this fallback. Preserve device identity and distinguish unset, empty, valid UUID and invalid/terminating selectors, or treat unresolved visibility as unknown rather than asserting that raw KFD order is the effective device list. The new test that expects a UUID to leave node order unchanged currently locks in this mismatch.
There was a problem hiding this comment.
Accepted, fixed in 778091c. I reproduced each case on c8eaf36 with the installer fixture:
ROCR_VISIBLE_DEVICES=<UUID>withHIP_VISIBLE_DEVICES=0exited 1 with "not gfx1100".- The UUID alone offered
HIP_VISIBLE_DEVICES=1. ROCR_VISIBLE_DEVICES=""andHIP_VISIBLE_DEVICES=-1both installed and exited 0.
install.sh now reads each list the way its runtime does, per the sources you linked:
rocr_visible: unset and set-but-empty now differ; empty shows no GPU. Entries are trimmed and read up to the number of GPUs, each parsed withstrtolin base 0 (strtol0handles hex, octal and decimal). An entry that doesn't parse, an index out of range, or a repeat ends the list.hip_visible: readsHIP_VISIBLE_DEVICES, orCUDA_VISIBLE_DEVICESwhen that is empty. Only plain decimal entries count, a repeat is skipped, any other entry ends the list, and the indices count what ROCr left.- No GPU left: a list that leaves none is refused with
ROCR_VISIBLE_DEVICES / HIP_VISIBLE_DEVICES leave no AMD GPU visible. - UUIDs: a UUID entry that either runtime would actually reach makes the order unknown. The installer then installs for a Strix GPU, offers no selector, and says device 0 is checked by the engine when it starts. That's the "treat unresolved visibility as unknown" option: ROCr and HIP match UUIDs differently (ROCr uppercases and matches a prefix of
GPU-%016X; HIP looks for a case-sensitive substring ofGPU-%016x, orGPU-XX), and the installer reproduces neither.
Tests:
test_a_uuid_selector_leaves_device_0_to_the_enginecovers your UUID +HIP_VISIBLE_DEVICES=0case, a ROCr UUID alone, and a HIP UUID. All three failed on c8eaf36; the first with1 != 0 : ... not gfx1100.test_lists_that_leave_no_gpu_are_refusedcoversROCR_VISIBLE_DEVICES="",ROCR_VISIBLE_DEVICES=5,HIP_VISIBLE_DEVICES=-1andHIP_VISIBLE_DEVICES=01. All four failed on c8eaf36.- The test that expected a UUID to keep node order is gone.
- I ran the shell functions under dash on the same 21-case table as superfluid#118's Rust model; every result and HIP index matched.
The same rules are in superfluid#118 (06e8072), so the installer and superfluid's planner agree.
| if [ "$device" -ne 0 ]; then | ||
| device_hint="export HIP_VISIBLE_DEVICES=$device" | ||
| warn "the engine runs on HIP device 0, which is $first here; the $gpu is HIP device $device." | ||
| warn "make it device 0 wherever you run basert: $device_hint" |
There was a problem hiding this comment.
Codex Review — [P2] Generate the recovery command from the pre-HIP device index. $device indexes gpus after the existing HIP/CUDA visibility filter has already reordered it, but replacing HIP_VISIBLE_DEVICES is interpreted against the list before that filter. For raw GPUs [gfx1151, gfx1100] with HIP_VISIBLE_DEVICES=1,0, I reproduced the closing instruction export HIP_VISIBLE_DEVICES=1; applying it leaves only gfx1100 and the installer then refuses the machine. The correct replacement is HIP_VISIBLE_DEVICES=0. The parenthetical advice to reorder an existing list does not make the explicit copyable command correct. Retain the selected GPU's identity/index before HIP filtering (while accounting for ROCR filtering), and test that applying the emitted command actually makes that same Strix GPU device 0.
There was a problem hiding this comment.
Accepted, fixed in 778091c. You were right: the offered value was the GPU's place after the HIP list, and on c8eaf36 raw [gfx1151, gfx1100] with HIP_VISIBLE_DEVICES=1,0 printed export HIP_VISIBLE_DEVICES=1.
hip_visible now outputs index gpu lines, where index is the number HIP_VISIBLE_DEVICES names that GPU by, counted among the GPUs ROCr leaves. The warning and the closing line offer that index. The old note about listing the GPU first in an existing list is gone, since the command is now exact.
As you suggested, test_the_offered_selector_makes_the_strix_gpu_device_0 runs the installer, takes the export HIP_VISIBLE_DEVICES=<n> it printed, and runs it again with that value. The Strix GPU must then be device 0, with no warning. It covers three cases:
| Raw GPUs | Variables | Offered |
|---|---|---|
[gfx1100, gfx1151] |
none | 1 |
[gfx1151, gfx1100] |
HIP_VISIBLE_DEVICES=1,0 |
0 |
[gfx1151, gfx1100, gfx1150] |
ROCR_VISIBLE_DEVICES=1,0,2, HIP_VISIBLE_DEVICES=0,1 |
1, the gfx1151's ROCr index |
On c8eaf36 the second case failed with Lists differ: ['1'] != ['0']. ROCR_VISIBLE_DEVICES is left as it is, because the index is counted after it.
…er the selector HIP reads Review fixes (Codex P2 x2 on c8eaf36). Visibility: an empty, UUID or invalid list was treated as no filter, so ROCR_VISIBLE_DEVICES=<UUID of the gfx1151> with HIP_VISIBLE_DEVICES=0 was refused (it saw only the gfx1100), and ROCR_VISIBLE_DEVICES="" or HIP_VISIBLE_DEVICES=-1 installed although ROCm shows no GPU. rocr_visible and hip_visible now read each list as its runtime does (ROCr: set-but-empty shows no GPU, entries trimmed, strtol base 0, a bad, out-of-range or repeated entry ends the list; HIP: HIP_VISIBLE_DEVICES or else CUDA_VISIBLE_DEVICES, plain decimals only, a repeat skipped, a bad entry ends the list, counting what ROCr left). A list with no GPU left is refused as such. A UUID entry the runtime would reach makes the order unknown: the installer then installs for a Strix GPU and says device 0 is the engine's to check, instead of guessing. Selector: the export offered was the GPU's place after the HIP list, but HIP counts the list before it: raw [gfx1151, gfx1100] with HIP_VISIBLE_DEVICES=1,0 was offered 1, which selects the gfx1100. It now offers the GPU's index among the GPUs ROCr leaves, and the tests run the installer again with that value and check the Strix GPU is device 0.
|
Codex re-review of c8eaf36: I accepted both P2 findings, and they're fixed in 778091c; details are in the threads.
superfluid#118 got the same rules in 06e8072. baseRT-internal#877 carries the same Checks run on macOS:
Not run: a real machine with several AMD GPUs or UUID selectors. |
`sudo apt install amdrocm-blas10.0-<gfx>` fails on a machine that has never had AMD's ROCm 10 repository, which is most of them. The README now gives the whole sequence (key, source line, apt update, the package, the render group), and install.sh prints the repository steps before the apt line when apt has no source for stable.repo.amd.com/rocm/core, with the dist for the machine's Ubuntu (ubuntu2404 or ubuntu2604, the two AMD serves). On another release it says which ones AMD serves instead of guessing. The key goes in as the ASCII-armoured .asc AMD publishes, so no gpg is needed.
IsuraManchanayake
left a comment
IsuraManchanayake
left a comment
There was a problem hiding this comment.
Codex Review
Re-reviewed 778091ce905494588cd46cf33f93cdd030ef3015, including the new commit, both replies, and the related Superfluid and source-packaging updates.
Both previous P2 findings are addressed. No new actionable finding in this PR.
- UUID selectors no longer produce a false device-0 assertion or an incorrect selector: the installer explicitly reports that selection is unverified and leaves it to the engine. Empty ROCR visibility and invalid/disabled HIP selectors are refused instead of silently using the raw GPU list.
- Recovery instructions now retain the pre-HIP index after ROCr filtering. The reversed-HIP example offers
HIP_VISIBLE_DEVICES=0, and the tests apply the offered command and confirm the supported GPU becomes device 0, including with ROCr reordering. - Reviewed the distinct numeric-list rules, duplicate handling, invalid-entry termination, CUDA fallback, and unknown-order path together with Superfluid's implementation.
Validation:
- Complete packaging suite: 53/53 passed, including actual wheel builds from the three synthetic bundles and the new installer regressions.
- Re-ran the previous independent installer probes: UUID + HIP 0 now installs with the explicit uncertainty warning; empty ROCR, HIP -1 and duplicate-terminated ROCR/HIP combinations refuse correctly; the recovery instruction uses the corrected index.
- All ten shared files match their source counterpart byte-for-byte and mode-for-mode.
- Shellcheck, actionlint (custom labels; pre-existing SC2254 excluded), and full-PR
git diff --checkpassed. - CI run 137 passed.
Limits: synthetic filesystem/package fixtures, not physical multi-GPU inference or a live publish. UUID resolution itself remains intentionally unverified by the installer. The release-time runner-access and source-sync prerequisites in the description still apply.
From v0.5.0 a release can carry
basert-engine-linux-x64-rocm-<v>.tar.gz: one engine for AMD Strix Halo (gfx1151) and Strix Point (gfx1150) on the ROCm 10 runtime, built on Ubuntu 24.04 (glibc 2.39: libbaseRT needs 2.38, basert 2.39). This makes the publish workflow package and test it, and install.sh install it.This must be merged before the v0.5.0 tag. Without it, the ROCm bundle breaks the publish run:
platform_tag.pytagged every ELF engineaarch64, so the ROCm wheel became a secondmanylinux_2_39_aarch64wheel. test-linux'spip install *manylinux*.whlthen installed both and failed, which stops PyPI, npm and crates.io for the whole release.The ROCm bundle is optional
The engine's release pipeline ships a release without the ROCm bundle when its AMD runner is off. A 0.4.x patch release would also have no ROCm bundle. So every channel follows what the release carries: the plan job reads the release's assets, and everything ROCm runs only when the bundle is there. A release without it publishes as before and does not wait for the Strix Halo runner.
Changes
packaging/python/platform_tag.py: reads the CPU from each ELF header (e_machine0xB7aarch64,0x3Ex86_64). It skips GPU code objects and refuses a folder built for two CPUs. The ROCm wheel ismanylinux_2_39_x86_64; the script reads the glibc floor from the binaries, so it follows the bundle's build image.@basecompute/basert-linux-x64-rocm(linux, x64, glibc), built only when the bundle is in the release. The main package'sos/cpufollow the engines the release has (["arm64", "x64"]with ROCm).linux-x64is added toENGINE_PACKAGESinpackaging/npm/basert.jsandbindings/node/src/index.ts.update-formula.sh):depends_on arch: :arm64moves intoon_macos, so macOS stays Apple Silicon only.on_linuxgains an Intel branch with the ROCm bundle.apt installline for each chip.brew stylepasses on both shapes.publish.yml:rocmand the engine list.[self-hosted, Linux, X64, rocm-gfx1151]. It runs the same pip / npm / linked-program checks as the GB10 job, and only whenrocmis true.*manylinux*_aarch64.whl.build/wheelbefore the script tests, because the tests build wheels now.install.sh: newLinux x86_64case./dev/kfdand a kfd topology node withgfx_target_version110501 (gfx1151) or 110500 (gfx1150). Another AMD GPU is named and refused, for example "not gfx1100"./dev/kfdcan't be opened (render group)./opt/rocm/core-10.0/lib/libamdhip64.so*and the chip'srocblas/library/<gfx>kernels. If either is missing, it printssudo apt install amdrocm-blas10.0-<gfx>and installs anyway (the CLI works without ROCm). When apt has no source for AMD's ROCm 10 repository, the apt line comes after the steps that add it: the key, the source line with the machine'subuntu2404orubuntu2604, andapt update. On another release it says those are the two AMD serves. Then it takesbasert-engine-linux-x64-rocm.BASERT_INSTALL_TEST_ROOT(tests only) points those checks at a fake/.6.17.0-19.19~24.04.2or later or the OEM kernel6.14.0-1018or later, Ubuntu 26.04's own kernel, and elsewhere Linux 6.18.4 or later. It gives the commands that add AMD's ROCm 10 repository and install the chip's package.ci.yml: the packaging job installsbuild/wheeland shellcheck, so the new tests run there.Before the v0.5.0 publish run
rocm-gfx1151) available to this repository. In its org runner group, allowbasecompute/baseRT; for a public repository the group also needs "Allow public repositories". Without this, the ROCm test job queues and PyPI/npm/crates wait for it.publish.ymluses the label, and it runs on releases and by hand only. A new test fails if a workflow that pull requests can start ever namesrocm-gfx1151orgb10.packaging/,bindings/andtests/scripts/test_packaging.pymust land in the engine repository too, with identical bytes. That repository owns these paths, and the next source sync copies its version over them. A patch for it is ready, and it applies to the 0.5.0 train.Testing
Tests ran locally on macOS with Python 3.9 and 3.14: 47 pass. Before this change, 27 of the new tests failed and 1 errored against
main's scripts.build-wheel.shrun over a fake three-bundle release givesmacosx_15_0_arm64,manylinux_2_38_aarch64andmanylinux_2_39_x86_64.uname/curlon a fake/for each GPU and runtime case, plussh -n,dash -n,bash -nand shellcheck.Also checked:
actionlint: clean apart from the SC2254 that main already has.brew style: passes in a throwaway tap.ubuntu:24.04andubuntu:26.04containers on a Strix Point with no AMD repository and no ROCm: they add the repository and install ROCm 10. The installed basert then answers a chat prompt on the GPU, and a second install.sh run prints no hint.