Sled-agent: detect switch backend (tofino, softnpu, et al) at runtime

[sion42x]

Replace the switch-asic, switch-stub, and switch-softnpu cargo features with startup detection. A Tofino node in the device tree selects the Tofino ASIC; otherwise a virtio 9p device answering 9P2000.P4 selects the propolis SoftNPU backend, and with neither the existing auto Tofino path is used. Optional switch_backend config added for the stub and SoftNPU zone backends. pumpkind is gated on Oxide hardware.

This PR is a proposed solution to #11202 based on feedback from the control plane team. Prior proposed/attempted solutions:

1. Don't detect, just publish the softnpu sled-agent with CI as a downloadable artifact: Publish the omicron-sled-agent package as a separate artifact for dev labs #11133 (half measure and doesn't solve the real issue)
2. Extending SMBIOS Type 1 table to label the device (results in a new propolis API version, overkill)
3. Creation of a named pci device in propolis for softnpu (probably unnecessary, read on)
4. Detection of the existing vio9p device based on a new subsystem ID (breaks driver loading due to illumos device binding and scadm hardcoding, likewise Mode::Modern requires newer stlouis changes and would be the first modern-only device)

Options 2-4 would have also required additional tooling in omicron to detect these things, making it a pretty sizeable cross-repo change. However, @rcgoodfellow proposed a detection method scadm already uses:

• look for p9fs devices using devinfo
• ask the p9 server if it is 9P2000.P4
• profit

So rather than changing propolis, this PR uses that method to detect softnpu devices. A test on a racklette vs on a voxel lab showed:

Rack	Sleds	Detection log	pumpkind	Dendrite
madrid	14, 16	found tofino node	imported, online	/dev/tofino/1
madrid	15, 17	none	imported, online	n/a
voxel	g0, g2	found SoftNPU 9p device	skipped, no instance	switch0 online
voxel	g1	none	skipped, no instance	n/a

Lastly, follow ups:

• sled_mode, sidecar_revision, and switch_backend overlap, and legal combinations are enforced by runtime checks. It would probably be good to get the latter two into one switch enum so illegal configs aren't representable.
• Switch zone contents are still chosen at build time, but dendrite has an issue for that: One build multiple ASICs dendrite#355

[sion42x]

Note for reviewers: I realized detect_switch_hardware should just use get_tofino_snapshot. The latter requires node.has_asic() and they both resolve identically downstream (TofinoView::Real either way), but folding it into one pre-existing surface makes more sense. I'm working on that now.

[dancrossnyc]

This loop kinda-sorta feels like a hand-rolled version of Iterator::find:
https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.find

? NodeWalk implements Iterator, so it may be useful to pull the bulk of logic here out into a predicate function. Dunno.

[sion42x]

Hmm... I can pull out the per-node logic and into a probe_node predicate. The catch with find itself is that each step is fallible and needs ? to propagate. try_find would make a good fit but I believe it's not stable. So I could do something like:

devinfo
    .walk_node()
    .map(|node| probe_node(log, &node.map_err(SwitchDetectError::DevInfo)?))
    .find_map(Result::transpose)
    .transpose()

but that feels harder to read than a loop. Not to mention doing a double transpose just to make it work which feels icky. Let me know which you think is clearer.

[dancrossnyc]

yeah, that's a lot uglier than a loop. It may still be worthwhile to extract the test into a dedicated function, but if you need fallibility during the search, then a loop is the way to go.

[sion42x]

Tests post c6b42fc on Madrid:

[madrid]root@oxz_switch0:~# pilot host exec -c 'grep -h -E "found tofino asic|found SoftNPU 9p device|probe failed|skipping pumpkind" /var/svc/log/oxide-sled-agent:default.log /pool/ext/*/crypt/debug/global/oxide-sled-agent* 2>/dev/null | cut -c1-150' 14-17
14  BRM42220081        ok: {"msg":"found tofino asic","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:00:44.501031762Z","hostname":"BRM42220081","pid":673,"component":
15  BRM42220046        ok: 
16  BRM42220007        ok: {"msg":"found tofino asic","v":0,"name":"SledAgent","level":30,"time":"1986-12-28T00:00:43.850669382Z","hostname":"BRM42220007","pid":674,"component":
17  BRM42220004        ok: 

On voxel:

root@helios1:~# for i in g0 g1 g2; do
> voxel host exec -c 'grep -h -E "found tofino asic|found SoftNPU 9p device|probe failed|skipping pumpkind" /var/svc/log/oxide-sled-agent:default.log /pool/ext/*/crypt/debug/global/oxide-sled-agent* 2>/dev/null | cut -c1-150' $i
> done
Sep 05 10:26:40.842 INFO set propolis binary to /root/propolis-9pexp/target/release/propolis-server
{"msg":"not an Oxide sled; skipping pumpkind","v":0,"name":"SledAgent","level":30,"time":"2026-09-04T17:35:22.830647805Z","hostname":"g0","pid":792,"c
{"msg":"found SoftNPU 9p device","v":0,"name":"SledAgent","level":30,"time":"2026-09-04T17:35:23.647422767Z","hostname":"g0","pid":792,"component":"Bo
Sep 05 10:26:59.634 INFO set propolis binary to /root/propolis-9pexp/target/release/propolis-server
{"msg":"not an Oxide sled; skipping pumpkind","v":0,"name":"SledAgent","level":30,"time":"2026-09-04T17:35:08.525600963Z","hostname":"g1","pid":741,"c
Sep 05 10:27:14.049 INFO set propolis binary to /root/propolis-9pexp/target/release/propolis-server
{"msg":"not an Oxide sled; skipping pumpkind","v":0,"name":"SledAgent","level":30,"time":"2026-09-04T17:36:12.543212179Z","hostname":"g2","pid":851,"c
{"msg":"found SoftNPU 9p device","v":0,"name":"SledAgent","level":30,"time":"2026-09-04T17:36:13.375344975Z","hostname":"g2","pid":851,"component":"Bo

Working as expected on all scrimlet/non-scrimlet + rack/voxel combinations.

[jmpesp]

We chatted via DM a bunch, and I'm summarizing it here: I am 100% on board with removing the sled-agent compile time features, but there's a lot of duplication with SwitchBackend, SwitchHardware, SidecarRevision, DendriteAsic, and kinda even SledMode. I'm not 100% sure what the sled-agent config changes should look like, but was hoping we could find a shape that elides some of those.

What about shaping the config around the deployment type? Something like:

• deployment = production would imply sled mode auto, where the actual mode would depend on sled-agent detecting the tofino or not, and if it was found the sidecar revision is always hard coded as "b" (as it is now in the config file)

• deployment = voxel would also imply sled mode auto, and similarly the actual mode would depend on sled-agent detecting the softnpu device or not.

• deployment = standalone would imply sled mode = scrimlet (as there's just one helios box) and the switch hardware would be set to use a softnpu zone

I originally talked myself out of this shape of config because the production and voxel branches look very similar (THAT'S THE POINT haha), and this isn't the granular config that exists today, which would support any type of deployment. Maybe then to support that we add a deployment = custom, which requires explicitly setting both the sled mode and the switch hardware?

But at a higher level, I don't like throwing away information if we can prevent it, and the intended deployment type is information that is thrown away when setting a bunch of very granular options. Said another way, there may be things we want to do only for production, and things we may want to do only when deployed for voxel / a4x2, etc. There may not be but shaping the config this way gives us the choice in the future.

I'm also looking to upstream patches that get my Canada Region config to work, and putting that behind a different deployment type also makes sense.

My only bike-shed-type comment would be to change SwitchBackend::Detect to something like SwitchBackend::ProbeHardware, as that's explicitly what's going on :)

[sion42x]

Thanks James. I like the "don't throw away information" thing and agree with it (versus in our chat when I mentioned using mechanism as the key).

I'm going to work on this as another PR on top of this one and will add you. Basically I'm thinking:

1. Ditch SwitchBackend and SidecarRevision config types
2. Merge sled_mode, sidecar_revision, and switch_backend into a tagged deployment enum
3. Possible options look like this:

• deployment = { kind = "production" }
• deployment = { kind = "virtual", front_port_count = 2, rear_port_count = 4 }
• deployment = { kind = "standalone", front_port_count = 1, rear_port_count = 1 }
• deployment = { kind = "custom", sled_mode = "scrimlet", switch = "tofino_stub" }

I'll work on the PR (and hope it passes CI since the buildomat scripts rely on some config settings).

[hawkw]

This comment is a bit hard to understand, IMO. Perhaps we can express the decision tree as more of an if-else chain?

[sion42x]

This comment will be unnecessary soon given the changes James proposed that I'm working on. Agreed, the whole shebang is a bit hard to understand compared to just... selecting what it is.

[hawkw]

"else the Tofino ASIC" feels like a sentence fragment?

[sion42x]

I can see that. I'll fix it, this will likely go away anyways.