Embed sush in sled-agent by plotnick · Pull Request #11252 · oxidecomputer/omicron

plotnick · 2026-09-06T23:11:23Z

This begins the integration of the Support Shell (RFD 620) service into the wider control plane, starting with sled-agent. Replaces initial integration attempt #11034.

Sled-agent now builds the sush job manager during bootstrap as a long-running task and serves its HTTP API on the bootstrap network from the start. Once the sled has an underlay address, it serves the same API there, too.

In addition to its client-facing API, sush also gossips sessions, jobs, and events over sprockets on the bootstrap network using rumors. Gossip peers are discovered using the bootstore's existing DDM polling loop, and a map polled from MGS resolves job targets given as cubby numbers.

The records sush must trust across reboots (its gossip identity and its execution boundary) live on the M.2s' cluster datasets. Job output is recorded on the ramdisk until an encrypted debug dataset is mounted, then moves there with a raised size limit. The signatures on all jobs must chain to one of the configured certs; the cert here is a sandbox root which must be replaced with a production cert before shipping (see oxidecomputer/sush#46).

This begins the integration of the [Support Shell](https://github.com/oxidecomputer/sush) ([RFD 620](https://rfd.shared.oxide.computer/rfd/0620)) service into the wider control plane, starting with sled-agent. Sled-agent now builds the sush job manager during bootstrap as a long-running task and serves its HTTP API on the bootstrap network from the start. Once the sled has an underlay address, it serves the same API there, too. In addition to its client-facing API, sush also gossips sessions, jobs, and events over sprockets on the bootstrap network using [rumors](https://github.com/oxidecomputer/rumors). Gossip peers are discovered using the bootstore's existing DDM polling loop, and a map polled from MGS resolves job targets given as cubby numbers. The records sush must trust across reboots (its gossip identity and its execution boundary) live on the M.2s' cluster datasets. Job output is recorded on the ramdisk until an encrypted debug dataset is mounted, then moves there with a raised size limit. The signatures on all jobs must chain to one of the configured certs; the cert here is a sandbox root which **must be replaced with a production cert before shipping** (see oxidecomputer/sush#46). Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

hawkw · 2026-09-08T19:46:10Z

+                // An error means the reconciler task is gone, in which case the
+                // set can never change again.
+                if receiver.changed().await.is_err() {
+                    std::future::pending().await
+                }


Hmm...are we sure this is the right behavior if the reconciler task has ended for some reason? Would that mean we are shutting down or something, and the caller should also bail out? Should we at least log something?

I do agree that this behavior makes sense for the test variants that will never actually change, FWIW.

Thank you, warning added in 4e9c78e.

hawkw · 2026-09-08T19:46:55Z

 }

-pub async fn poll_ddmd_for_bootstore_and_tq_peer_update(
+pub async fn poll_ddmd_for_peer_updates(


now that this has a more generic name, perhaps it deserves a comment explaining what the "peer updates" it polls for (i.e. the bootstore data, trust quorum, and sush gossip proto)

also, hmm...i kinda wonder if this really still belongs in a module called bootstore_setup, now that it does two other things that are not actually directly related to setting up the bootstore. arguably, that maybe should have been done when the TQ stuff was stuffed in here, though. and, maybe disentangling it from the surrounding code is more effort than it's really worth going to just because i complained about the module's name!

I do not disagree, but think it should be done as a follow-up in conjunction with someone that has (much) more sled agent experience than I do. Documented in d44a838, thanks.

hawkw · 2026-09-08T19:58:12Z

+    // TODO: check that this is the right default. `/var/run/oxide` is tmpfs
+    // and is created during bootstrap, but it exists to hold ZFS key files
+    // (see `illumos_utils::zfs::KEYPATH_ROOT`), not bulk job output.


does this TODO deserve to become TO-done before this merges?

Indeed, I am hoping this very review will settle the question!

What properties are you looking for? That the path will exist and be mounted and be a tmpfs? I think that's done by the filesystem/minimal svc:

https://github.com/oxidecomputer/illumos-gate/blob/0a8e320ff3c6ef999a764d0d07f26c92f6205ccf/usr/src/cmd/svc/milestone/fs-minimal#L66-L69

which our images are built with.

Thank you very much, that is exactly what I was looking for! Noted in 7481505.

hawkw · 2026-09-08T21:14:18Z

+    /// This is on the ramdisk, so it costs global zone memory and does not
+    /// survive a reboot.


hm, IIUC, this seems like it's describing a property that is true of the default value but is not actually being enforced? perhaps the comment should be reworded to instead say that the path provided here should be on the ramdisk?

Yes, corrected in 1b65c0f. We can enforce this, but it's somewhat ugly and (as far as I or Claude can tell) probably requires unsafe (warning: unfiltered Claude code ahead):

/// Whether dir sits on a tmpfs. Job output recorded before an /// encrypted dataset exists must never persist on a disk. #[cfg(target_os = "illumos")] fn is_on_tmpfs(dir: &Utf8Path) -> bool { use std::ffi::{CStr, CString}; use std::mem::MaybeUninit; let Ok(path) = CString::new(dir.as_str()) else { return false; }; let mut vfs = MaybeUninit::<libc::statvfs>::uninit(); // SAFETY: statvfs fills vfs on success, and it is read only then. if unsafe { libc::statvfs(path.as_ptr(), vfs.as_mut_ptr()) } != 0 { return false; } let vfs = unsafe { vfs.assume_init() }; // SAFETY: statvfs nul-terminates f_basetype. let basetype = unsafe { CStr::from_ptr(vfs.f_basetype.as_ptr()) }; basetype.to_str() == Ok("tmpfs") }

That seems pretty grotty to me, but if you have better suggestions or think it's just grotty enough, I'd be happy to add it or something like it.

Eh, I agree that the tmpfs check is probably not necessary here unless you think "never persisted unencrypted" is an improtant enough security property to enforce (I wouldn't know). The important part is that the comment now makes it clear that this is something required of the config rather than something enforced by this code, which is what I was really hoping to clear up!

hawkw · 2026-09-08T22:06:56Z

+//! Gossip runs over sprockets on the bootstrap network, so jobs and
+//! sessions are shared across sleds. A universe is a shared gossip
+//! history; peers that meet merge into one by a dominance rule.
+//! A restarted sled re-seeds its universe, rejoins the rack's, then
+//! replays its history without re-executing it. Its gossip identity
+//! is stored in a _bookmark_.


are "universe" and "dominance rule" terms of art being defined here, or is this just like, something Claude thought made sense?

(because i have no idea what this is trying to say)

@hawkw you asked me to comment here; the reason you have no idea what this is trying to say is because this doesn't make sense. With all due disrespect to Claude:

"Gossip runs over sprockets on the bootstrap network": true.

"jobs and sessions are shared across sleds": true in a sense, badly worded; more accurate would be to say "messages relating to jobs and sessions are communicated between sleds" — a job may be addressed to only one sled, but it will become known everywhere.

"A universe is a shared gossip history": true in a sense, misleading; more accurate would be to say "A universe refers to the causally-related set of messages sent between all peers in the gossip network. Peers which belong to the same universe may gossip, so they learn what other peers know, and they forget what other peers have redacted. Peers which belong to different gossip networks are said to be part of a different causal universe; their messages are not causally related, and they cannot gossip."

"peers that meet merge into one by a dominance rule": false; peers that meet exchange messages so that their set of known messages converge, but the peers retain distinct identities and do not merge in any sense I can think of.

"A restarted sled re-seeds its universe": mechanically true, misleading; the mechanism for leaderless consensus on a single causal universe (which must, due to interval tree clocks, derive from a sequence of forks of a single distinguished seed) is for peers to each declare themselves the seed, then resolve conflicts by a deterministic anti-symmetric rule until a single winner emerges, who becomes the seed node for the causal universe — then all other nodes in the gossip network bootstrap from it in a random tree shape, forking the identity of the node they bootstrap from, so that the entire universe shares a single seed, which ensures every subsequent versioned message is causally related. However, this mechanism is largely invisible; a restarted sled is observed to merely rejoin the rack's gossip network, since that universe resolution rule always causes it to drop its own seed in favor of the one that already exists in a peer it connects to.

"replays its history without re-executing it": true in code not yet merged; session replay prevention is enforced by a mechanism we haven't landed yet, see Jobs should execute at most once: simple, right? sush#79.

"Its gossip identity is stored in a bookmark": true, but unexplained. A bookmark is a persistence mechanism which ensures that after a crash, a peer can reclaim its slice of "identity space" so that the representation of causal versions doesn't expand under restarts.

the reason you have no idea what this is trying to say is because this doesn't make sense.

I mean, it would also help if the terms were defined. :)

They are in the module doc as of ecaf96d, and lockers are now described (briefly) in RFD 620 as well. But I agree with the sentiment that all of this should be better documented.

hawkw · 2026-09-08T22:21:31Z

+//! replays its history without re-executing it. Its gossip identity
+//! is stored in a _bookmark_.
+//!
+//! The bookmark, like every record sush must trust across reboots,
+//! lives in the [sush locker](https://github.com/oxidecomputer/sush/blob/main/server/src/locker.rs).


here we say that the gossip identity is stored in a bookmark, and that the bookmark is a locker record, but we don't really explain what a bookmark actually is --- is this something that changes during the gossip protocol or is it fixed?

It changes during the gossip protocol. Peers that bootstrap from you cause your own identity to fork in half; you hand one half to the bootstrapping peer, and keep the remainder. Every time this happens, your bookmark file is altered. Normal gossip does not change your identity, but may also modify the bookmark — it is only safe to merge back an old identity after you learn everything that prior identity once published to the gossip network, so the moment you do, you slurp it back from the bookmark, which may simplify it. Finally, peers may retire into another peer, proactively giving up their identity and merging it into the peer's — this is unexercised in sush presently.

hawkw · 2026-09-08T22:22:08Z

+//! Each locker record is one file on each M.2. Stores write every
+//! copy, and loads adopt a record only when the copies show it cannot
+//! be stale. The locker is not a small bootstore: the bootstore holds


what does "loads adopt a record only when the copies show it cannot be stale" mean?

My rewrite: "When you write a file to a locker record, you actually write 2 copies, one to each M.2. When you read back from a locker record, this only succeeds if both copies are exact matches, or if one of them is entirely missing (the latter to handle the case of M.2 hardware replacement in the field). Reads are rejected if they mismatch, which means that a torn write or corruption on one drive cannot induce the reader to load stale or invalid information from the other drive. This effectively turns the pair of M.2 drives into a single mirrored storage container which fails closed if either drive fails. We accept this because an M.2 failure is considered a non-user-replaceable part failure, for which the solution is an RMA. If this occurs in the field, sush will become inoperable on the sled containing the failed M.2 (even if only one of them fails); we report this error to the user, who should replace the sled."

Thank you, applied with light edits as ecaf96d.

there is a crate in Omicron at ledger/src/lib.rs with the explicit purpose of "helping store configs as mirrors to two M.2s, with tools for accessing the most recent one". We use this heavily for sled agent configurations. Might be worth taking a look at

Yes, the ledger was an inspiration for the locker, but they differ in some important ways. I have documented this a bit in the storage section of RFD 620, and am happy to add more color as needed.

hawkw · 2026-09-08T22:23:57Z

+//! The bookmark, like every record sush must trust across reboots,
+//! lives in the [sush locker](https://github.com/oxidecomputer/sush/blob/main/server/src/locker.rs).
+//! Each locker record is one file on each M.2. Stores write every
+//! copy, and loads adopt a record only when the copies show it cannot
+//! be stale. The locker is not a small bootstore: the bootstore holds
+//! rack-wide facts that sleds may recover from their peers, but a locker
+//! record says what this sled itself did or committed to do. These


it is unclear to me from this what locker records are, beyond "a file on each M.2"; are they gossipped to other sleds? or are they just something the sled stores locally? is the contents of locker records sensitive and/or security-critical? are they encrypted or authenticated in some way?

are they gossipped to other sleds?

No.

are they just something the sled stores locally?

Yes.

is the contents of locker records sensitive and/or security-critical?

In the sense of confidentiality: no, they are only meant to store metadata which does not contain job information. This includes causal version information, and job IDs (which are cryptographic hashes).

In the sense of integrity: yes, if someone writes a new, valid-but-wrong value to both M.2 drives, they can mess with the integrity of the gossip network as a whole, and sush's semantics specifically. In particular, mucking around with the Rumors bookmark can induce Rumors to violate causality, which can arbitrarily corrupt the ordering of messages delivered to sush, or contagiously delete messages throughout the network. Mucking around with the "execution boundary" file described in oxidecomputer/sush#79 can induce sush to re-execute previously executed jobs on the sled, or fail to execute jobs it has not yet executed (though it cannot induce it to execute jobs it was not already authorized to execute).

are they encrypted or authenticated in some way?

Encryption: no, because we don't have somewhere to get the encryption key from. Trust quorum may not be available, and sealing the entire M.2 drive using the RoT is something that I've heard discussed (@lzrd has thoughts about this I believe?), it's out-of-scope here. But there's nothing sensitive in these files.

Integrity to corruption: not by the locker mechanism, as I understand it, but rumors and sush use a hash checksum to check the integrity of the files they store in the locker, on a per-record-type basis.

Authentication: no, there's no signing of the files. It's not clear to me who would sign them; they need to be updated all the time during normal operation of sush, and it's not clear to me that signatures meaningfully foreclose the attacks possible if you can change the contents of the M.2. For instance, merely rolling back to an older version of any of the files stored would cause bad things (arbitrary gossip set corruption and job replay).

hawkw · 2026-09-08T22:25:21Z

+    pub fn start_api(
+        &self,
+        ip: Ipv6Addr,
+    ) -> Result<HttpServer<Arc<JobManager>>, String> {


i feel like it might be a bit nicer to return an anyhow::Error or something here although maybe the String is a constraint of other sled-agent code I haven't looked at in a while.

Thank you, fixed in ae91da0.

hawkw · 2026-09-08T22:44:10Z

+/// What gossip needs from the sled: its sprockets identity, its reference
+/// measurements, the bootstrap address to listen on, and where to find its
+/// peers.


this feels like it's basically restating the field names. i think that if this is going to have a doc comment on it, it should explain why gossip needs these things; otherwise, this doesn't really provide any useful information and may as well not have a doc comment.

thank you, cut in 301e9d5.

hawkw · 2026-09-08T22:44:41Z

+/// client sends is a signed job request or a certificate, both small.
+const REQUEST_MAX_BODY_BYTES: usize = 0xFFFF;
+
+/// Handles to the Support Shell server's tasks.


this mostly just shuts it down, right?

yes, and owns it while it's running.

hawkw · 2026-09-08T22:45:22Z

+/// Maximum size of a request body the API will accept. The largest thing a
+/// client sends is a signed job request or a certificate, both small.
+const REQUEST_MAX_BODY_BYTES: usize = 0xFFFF;


what determines that both of these small things will be < 65535 bytes? are there limits that are determined things defined elsewhere? or are we just deciding "eh, they shouldn't be bigger than 64kb" (which is reasonable, but would like to write that down explicitly)?

This cap was based on data at hand plus some overhead and a plausible-sounding power of 2, for sure. but I'm not sure how we would guarantee this or where else we would document it? Open to suggestions as always.

🤷‍♀️ mainly, I was wondering if there were defined max sizes for the protocol messages (the job request or cert) that we might determine this from. If not, that's fine, the comment just felt a bit unclear as to whether it was saying "this was determined because these have well known sizes" or if it was just saying "this is probably big enough for anything reasonable".

Fair enough, clarified in e50003d.

hawkw · 2026-09-08T23:15:28Z

+        }
+    });
+    let listen_addr = SocketAddrV6::new(bootstrap_ip, SUSH_GOSSIP_PORT, 0, 0);
+    let (universe, linked) = match spawn_gossip(


https://www.youtube.com/watch?v=ZRcpnM26nJM

hawkw · 2026-09-08T23:29:37Z

+/// as one is mounted, with a raised size limit. Output already recorded
+/// on the ramdisk stays there, readable until reboot.


might we wish to have something that copies old job output from the ramdisk onto the debug dataset once it's mounted? is that planned, or does it happen elsewhere?

It is planned, yes, but not yet implemented.

Is it tracked somewhere?

Is it tracked somewhere?

yes, it's oxidecomputer/sush#69.

maybe there should be a TODO about that here?

It's a sush todo, 2640954 points at the issue, thanks.

hawkw · 2026-09-09T00:01:06Z

+/// DDM's advertised subnets are only candidates (see
+/// [`DdmClient::derive_underlay_subnets_from_prefixes`]), so we must
+/// probe for MGS. Each round's answers merge into the existing map,
+/// so a probe outage never erases it.


i understand what this is saying but i really do not care for the claudelish...

thank you, rewritten in 7f810f4.

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Also rephrase a `bool.then()` as `if`. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Co-authored-by: Eliza Weisman <eliza@elizas.website>

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Co-Authored-By: finch <finch@oxidecomputer.com> Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

jmpesp · 2026-09-09T13:41:32Z

+    /// Returns immediately if the set has changed since this receiver last
+    /// observed it. The test variants' datasets never change, so for them this
+    /// never returns.
+    pub async fn changed(&mut self, log: &Logger) {


What about

fn changed(...) -> Result<..., ...>

instead? The caller can then decide what to do with the RecvError.

Excellent suggestion, thank you; applied in 898c661.

jmpesp · 2026-09-09T13:46:24Z

+        // Now that this sled knows its underlay address, sush can serve
+        // its API there, too.


Is there any hazard to having multiple sush servers, each accepting different requests? Can they conflict with each other?

We should not have multiple sush servers running jobs on the same sled.

Good question. There's no hazard in what's here, which is just starting multiple dropshot servers. Those are fine because they call into a shared job manager, with one locker. But if you were to launch two job managers (the "actual sush server") on one sled, it would be bad, because their lockers would currently try to clobber each other. I'm not sure we could easily prevent that structurally, or would even want to, as it's useful during testing. But we can definitely guard against the hazard, e.g., by taking an advisory lock on the locker files (which should be easy).

Addendum: it was indeed pretty easy, added as sush fa405c2. I'll handle the possible Locked error on the sled-agent side in the next re-pin.

Follow-up: 5096283 bumps the sush pin to include an exclusive lock held by each process. Multiple sush servers therefore fail to start rather than clobbering each other.

jmpesp · 2026-09-09T14:07:02Z

+        .build()
+        .expect("failed to build an HTTP client");
+    loop {
+        match ddm.derive_underlay_subnets_from_prefixes().await {


Worth confirming: this returns every prefix DDM has learned about - for multiple racks in a regional cluster, I think this still means only the sleds in the current rack, because the transit/transit peers will advertise/exchange at the /56 level. Otherwise this code (and related bootstore and trust quorum code probably!) would have to filter the results.

Yes, I believe that is right. If that changes, we can just filter here.

jmpesp · 2026-09-09T14:08:38Z

+/// Periodically ask MGS which baseboard sits in each cubby, and
+/// publish the map. A job may name its target sled by cubby;
+/// this map resolves it. MGS answers at a fixed address within its
+/// switch zone's subnet, so each round we ask at that address in every
+/// /64 ddmd has learned, and take answers from any that respond. The
+/// answers merge into the map, so a round that goes unanswered never
+/// erases it.


My assumption reading this is that we don't want to use internal DNS (and filter the results for this rack!) because that may not be up yet, and we want to be able to assign jobs to specific sleds before that?

Exactly correct. We don't even technically depend on DDM or MGS being up, though without the former we wouldn't find any peers with which to gossip, and without the latter we'd be unable to resolve cubby numbers as job targets. But jobs can still be executed safely (but possibly only locally) in all of those scenarios.

jmpesp · 2026-09-09T14:09:53Z

+/// answers merge into the map, so a round that goes unanswered never
+/// erases it.
+async fn poll_mgs_for_cubbies(log: Logger, cubbies: watch::Sender<Cubbies>) {
+    let ddm = match DdmClient::localhost(&log) {


What if this code loses the race with the global zone's DDM starting?

Should be fine. Losing the race just means a couple of failed polls, which are harmless. And with new polling behavior in e544dcd (see below), an empty answer leaves the map alone.

Sorry, what I meant is if this function returns before entering the loop, because there would be nothing that would update the cubby map.

Yeah, should we be retrying this on failures to construct the DDM client?

We should not, because the constructor doesn't do any I/O, it can only fail to build the actual client, which would indicate a config error and so fail every time. But that .expect() was not great, so f62cd42 replaces it with a log & bail.

ah, hmm, if constructing the client failing is truly a programmer error such as us passing in a bad config to reqwest, I feel like the panic is better than just not working?

jmpesp · 2026-09-09T14:14:48Z

+                        }
+                    }
+                }
+                cubbies.send_modify(|current| current.extend(map));


This doesn't clear the previous cubby map, extend for a BTreeMap overwrites the value previously at a key. I think it's possible for a not-fully-populated rack to have a sled moved from cubby X to an empty cubby Y, and for this to mean that a particular baseboard ID appears for both X and Y. You could store the baseboard as the key instead to prevent this, but why not clobber the map each update?

Thanks, clobbered in e544dcd. We now ignore empty responses (which can't be right, since we're in a cubby), merge multiple answers, and overwrite in every non-trivial round. Receiving less information in one round than a previous one could cause the map to shrink, but that fails safe, since we'd reject jobs rather than running them on the wrong sled.

sorry what do you mean by non-trivial here?

Not empty. An empty reply would be very weird, because we know we're on a sled, and it's got to live somewhere, so why doesn't MGS know about it? but I can imagine it might have an empty map while starting up or whatever, so it seemed best to ignore it rather than throw an error. If it will never be empty, we can remove the test and the hedge both.

jmpesp · 2026-09-09T14:16:32Z

+    #[serde(deserialize_with = "SushConfig::nonempty_roots")]
+    pub roots: Vec<Utf8PathBuf>,


take it or leave it suggestion: alternatively, this could be

pub root: Utf8PathBuf, pub additional_roots: Vec<Utf8PathBuf>,

That is a nice way to make the bad case unrepresentable, but I think failing at parse time is good enough for now.

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

hawkw

i have gone through and rewritten a bunch of Claudeprose comments that made me feel irrationally angry. sorry, this is just kind of an occupational hazard when one asks me to review one's PRs.

hawkw · 2026-09-09T21:26:40Z

+//! related messages is called a _universe_: peers in the same universe
+//! can gossip and converge on its contents; peers in different
+//! universes cannot gossip at all. Every identity in a universe
+//! descends from a single seed. When sleds meet, a deterministic rule


"when sleds meet" is cute but it could maybe be clearer that this is the gossip handshake?

Maybe clarified in f65febb.

hawkw · 2026-09-09T21:31:08Z

+//! write two copies, one to each M.2. When we load a record, it only
+//! succeeds if both copies are exact matches, or if one of them is
+//! entirely missing (the latter to handle the case of M.2 hardware
+//! replacement in the field). A load fails on mismatched copies, so that
+//! a torn write or corruption on one drive cannot induce the reader to
+//! load stale or invalid information from the other drive. This turns the
+//! pair of M.2 drives into a single mirrored storage container that fails
+//! closed on any disagreement. We accept this because an M.2 failure is
+//! considered a non-user-replaceable part failure, for which the solution
+//! is an RMA. If this occurs in the field, sush may refuse to run jobs on
+//! the sled containing the failed M.2; we report this error to the user,
+//! who should replace the sled.


it feels a bit contradictory here that we say we allow one copy to be missing so that we can handle M.2 replacement in the field, but then later on we say that we require both M.2s because they are not user replaceable and the sled must be RMA'd. I can maybe see how both can be true but I find it a bit confusing here. Can we expand a bit on what, exactly, the relationship with M.2 failures is?

Ok, I was getting lost myself trying to maintain this documentation in three (rapidly changing) places, so I've started consolidating. 9e058e7 now refers to the storage section of RFD 620, which should now answer your question in a canonical place. (If it does not, please let me know here or there or anywhere.)

hawkw · 2026-09-09T21:36:30Z

+/// as one is mounted, with a raised size limit. Output already recorded
+/// on the ramdisk stays there, readable until reboot.


maybe there should be a TODO about that here?

hawkw · 2026-09-09T21:57:29Z

+/// answers merge into the map, so a round that goes unanswered never
+/// erases it.
+async fn poll_mgs_for_cubbies(log: Logger, cubbies: watch::Sender<Cubbies>) {
+    let ddm = match DdmClient::localhost(&log) {


Yeah, should we be retrying this on failures to construct the DDM client?

Co-authored-by: Eliza Weisman <eliza@elizas.website>

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Failure to build an HTTP client would be weird, but shouldn't take down sled-agent. Log an error and bail instead. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Co-authored-by: Eliza Weisman <eliza@elizas.website>

plotnick · 2026-09-10T20:16:29Z

sorry, this is just kind of an occupational hazard when one asks me to review one's PRs.

Please do not be sorry; I am sorry that I inflicted such things upon well-meaning reviewers. I am still learning to drive these crazy tools, and in particular to find an acceptable balance between rigor and urgency. Thank you very much for your patience and feedback as I stumble my way through this new terrain, they are greatly appreciated.

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com> # Conflicts: # Cargo.lock # workspace-hack/Cargo.toml

Locker construction now fails if another process is using it. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com> # Conflicts: # Cargo.lock # workspace-hack/Cargo.toml

plotnick mentioned this pull request Sep 6, 2026

Tunnel sush through Nexus #11254

Draft

plotnick force-pushed the sush-sled-agent branch from e7b2c72 to 01f98ba Compare September 6, 2026 23:53

plotnick mentioned this pull request Sep 7, 2026

Need production key & certificate oxidecomputer/sush#46

Open

leftwo reviewed Sep 8, 2026

View reviewed changes

Comment thread sled-agent/src/long_running_tasks.rs

Comment thread sled-agent/src/long_running_tasks.rs

rmustacc requested review from hawkw and jmpesp September 8, 2026 17:30

hawkw reviewed Sep 8, 2026

View reviewed changes

plotnick mentioned this pull request Sep 8, 2026

Add the sush proxy to the switch zone #11253

Draft

hawkw reviewed Sep 8, 2026

View reviewed changes

hawkw reviewed Sep 9, 2026

View reviewed changes

plotnick and others added 17 commits September 8, 2026 18:01

Log the start of sush tasks

552801f

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Fix SushConfig::ramdisk_dir doc

1b65c0f

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Fix DDM subnet and cubby map docs

7f810f4

Also rephrase a `bool.then()` as `if`. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Document poll_ddmd_for_peer_updates

d44a838

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Use ByteCount for byte counts

3dc9c68

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Return an anyhow::Result from SushHandles::start_api

ae91da0

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Kill useless GossipInputs doc

301e9d5

Wrap long log line

59f30ff

Warn if the config reconciler task goes away

4e9c78e

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

fix locker link

5a65227

Co-authored-by: Eliza Weisman <eliza@elizas.website>

Declaude

3c7792c

Co-authored-by: Eliza Weisman <eliza@elizas.website>

better corpus closure

b1eefa9

Co-authored-by: Eliza Weisman <eliza@elizas.website>

Fix up applied suggestions

4c7ccb1

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Improve sush module doc

ecaf96d

Co-Authored-By: finch <finch@oxidecomputer.com> Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Update sush: isolated & lonely are now associated functions

6ef1b06

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Improve REQUEST_MAX_BODY_BYTES doc

e50003d

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Reject empty roots at config parsing time

48ada88

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

jmpesp requested changes Sep 9, 2026

View reviewed changes

plotnick and others added 3 commits September 9, 2026 11:36

Replace the cubby map each round instead of merging

e544dcd

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Return a result from AvailableDatasetsReceiver::changed

898c661

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

TODONE: check default_ramdisk_dir

7481505

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

hawkw reviewed Sep 9, 2026

View reviewed changes

plotnick and others added 11 commits September 10, 2026 08:57

Update sush config doc

3179577

Co-authored-by: Eliza Weisman <eliza@elizas.website>

More declauding

37e4597

Co-authored-by: Eliza Weisman <eliza@elizas.website>

Declaude poll_mgs_for_cubbies doc

05b110c

Co-authored-by: Eliza Weisman <eliza@elizas.website>

Declaude comment

a2b8a97

Co-authored-by: Eliza Weisman <eliza@elizas.website>

Declaude comment

b0eef7f

Co-authored-by: Eliza Weisman <eliza@elizas.website>

Clarify universe dominance rule

f65febb

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Point to RFD 620 for locker docs

9e058e7

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Clarify encrypted output dir upgrade

2640954

fmt

32741d6

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Don't panic!

f62cd42

Failure to build an HTTP client would be weird, but shouldn't take down sled-agent. Log an error and bail instead. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

TODO: cleanup poll_ddm_for_peer_updates

e9ff322

Co-authored-by: Eliza Weisman <eliza@elizas.website>

plotnick and others added 3 commits September 10, 2026 15:44

Merge remote-tracking branch 'github/main' into sush-sled-agent

e3068d5

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com> # Conflicts: # Cargo.lock # workspace-hack/Cargo.toml

Bump sush pin

5096283

Locker construction now fails if another process is using it. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>

Merge remote-tracking branch 'github/main' into sush-sled-agent

c494933

Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com> # Conflicts: # Cargo.lock # workspace-hack/Cargo.toml

		/// This is on the ramdisk, so it costs global zone memory and does not
		/// survive a reboot.

		/// as one is mounted, with a raised size limit. Output already recorded
		/// on the ramdisk stays there, readable until reboot.

		// Now that this sled knows its underlay address, sush can serve
		// its API there, too.

		#[serde(deserialize_with = "SushConfig::nonempty_roots")]
		pub roots: Vec<Utf8PathBuf>,

Uh oh!

Conversation

plotnick commented Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

plaidfinch Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

plotnick commented Sep 6, 2026 •

edited

Loading

plaidfinch Sep 8, 2026 •

edited

Loading

plotnick Sep 9, 2026 •

edited

Loading