diff --git a/issues/design-debt/getsockname-answers-an-address-nobody-asked-netd-for.md b/issues/design-debt/getsockname-answers-an-address-nobody-asked-netd-for.md new file mode 100644 index 00000000000..e06d0c01afd --- /dev/null +++ b/issues/design-debt/getsockname-answers-an-address-nobody-asked-netd-for.md @@ -0,0 +1,26 @@ +--- +status: open +kind: finding +opened: 2026-09-08 +--- + +# `getsockname` answers an address nobody asked netd for + +`userland/libc/src/socket.rs`'s `getsockname()` fills the caller's `sockaddr` +with `[10, 0, 2, 15]` and the socket's local port. The address is a literal in +that function; netd is not asked, and the SDK has nothing to ask it with — +`toyos::net` carries `tcp_connect`, `tcp_bind`, `tcp_accept`, the UDP calls and +`dns_lookup`, and no call that answers "what is this machine's address". + +It was true by coincidence until now: netd carried the same literal, so the +shim and the stack agreed. netd takes its address from DHCP as of the change +that filed this, so the two agree only on a machine whose server happens to +lease `10.0.2.15` — QEMU's user-mode backend does, and the bench's router does +not. Every C program that asks what address it is bound to is told the wrong +one there. + +What it costs to fix is a message type on netd's protocol and one plumbed +answer; what it costs to leave is that the one caller of `getsockname` in a +POSIX program is the one that then advertises an address nothing can reach. +Nothing in the tree reads it today, which is why this is a finding and not a +defect. diff --git a/issues/hardware/the-t14-answers-only-through-a-usb-stick.md b/issues/hardware/the-t14-answers-only-through-a-usb-stick.md index 467be96774d..38430a6fb3d 100644 --- a/issues/hardware/the-t14-answers-only-through-a-usb-stick.md +++ b/issues/hardware/the-t14-answers-only-through-a-usb-stick.md @@ -11,14 +11,14 @@ is on a cable on the same LAN as the development Mac and its NIC is the onboard Intel I219 at `00:1f.6`, `8086:15fc`, which the kernel enumerates and nothing claims. The track is to make that cable the answer path. -The substrate a process needs to drive a PCI function itself is built -(`kernel/src/pcidev/mod.rs`, `userland/netd/src/virtio_net.rs`). What is left is -the I219 driver in netd, with DHCP under the hostname `toyos-t14` and a first -ping and ssh from the Mac; a record stream from logd to a listener in the -harness, so a boot's log arrives while it is booting; command execution, file -transfer both ways and key auth in sshd, with the harness running userland tests -over ssh through a russh client; and a netboot spike in which the firmware -fetches the loader over HTTP so the stick leaves the boot path. +Built and green under QEMU: the substrate (`kernel/src/pcidev/mod.rs`), the +I219 driver (`toyos-i219/`, `userland/netd/src/i219.rs`), netd's address from +DHCP (`userland/netd/src/dhcp.rs`), the record stream (`toyos-logstream/`, +`userland/logd/src/stream.rs`) and sshd's exec, transfer and key auth. What is +left is the laptop — the claim on its own card (`tests/lancase`), the stream and +the ssh from the Mac over the cable (`tests/ssh-client-host`), and a netboot +spike that takes the stick out of the boot path — and all of it waits on +`issues/kernel/a-32-bit-bar-needs-the-host-bridges-aperture.md`. Constraints a reader would otherwise pay to re-derive: @@ -34,12 +34,20 @@ Constraints a reader would otherwise pay to re-derive: before suspecting the driver. - **ssh is the bench's transport and a real feature**: sshd is built on russh and the harness's client is russh too. No host ssh binary, no fork. -- **Addressing is DHCP with a hostname**, resolved through the router's DNS. The - T14's MAC is the same under ToyOS and Ubuntu, so the lease is the one `t14` - already resolves to. Wi-Fi is out — the AX210 needs a firmware image. -- The I219 has **32-bit BARs**, and `pcidev`'s window allocator has only ever - placed a 64-bit one: `Refusal::NoWindow` on that machine means nothing was - found above everything firmware described and below the platform's fixed MMIO. +- **Addressing is DHCP with a hostname**, and netd sends `toyos-t14` as the + host-name option — but **the name resolves to nothing on this LAN**, measured: + the T14's DHCP-served resolvers are the ISP's, and on the development Mac + `t14` resolves to the Tailscale address `100.92.92.12`, which only Ubuntu ever + holds. The address is read off the claimed PCI function instead + (`Driver::wire`): `enp0s31f6` at `192.168.1.46/24`, the Mac on `192.168.1.47`. + Wi-Fi is out — the AX210 needs a firmware image. +- **The I219 is an MSI part**, measured: `/proc/interrupts` names its interrupt + `IR-PCI-MSI-0000:00:1f.6` and `msi_irqs/162` reads `mode=msi`. +- The I219 has a **32-bit BAR** (`bar0=0xbcf00000`): `pcidev`'s window allocator + places a BAR above everything firmware described, and below 4 GiB there is no + above — the platform's fixed MMIO is at `0xFEC00000`. Leaving the BAR where it + sits is not a way out either: the internal NVMe's `0xbce00000` is in the same + 2 MiB page, which is the only page size this kernel maps. - **QEMU's `virtio-net-pci-non-transitional` on `q35` advertises no PCIe function-level reset** — measured, not assumed: `pcidev`'s refusal on that ground reddened every netd registration at once. So a re-claim is made safe by @@ -49,9 +57,8 @@ Constraints a reader would otherwise pay to re-derive: one; the I219 does, so on the T14 both hold. - **The record stream is `logstream=:` on the parameter line**, copied by the kernel into `/system/bin/init`'s environment and read from there - by `logd` (`toyos-logstream`'s `PARAM` and `ENV`). What is left to build is the - metal half: arming the flashed image with the Mac's address and listening while - the T14 boots. A boot that dies before `logd` runs still needs the stick. + by `logd` (`toyos-logstream`'s `PARAM` and `ENV`). A boot that dies before + `logd` runs still needs the stick. - **A stalled peer's backpressure reaches `logd`'s queue only after megabytes.** Between them stand a 2 MiB kernel pipe (`kernel/src/pipe.rs`'s `PIPE_SIZE`) and netd's 64 KiB send buffer, and a `log-storm` at `--smp 8` produces 4,213 lines diff --git a/issues/kernel/a-32-bit-bar-needs-the-host-bridges-aperture.md b/issues/kernel/a-32-bit-bar-needs-the-host-bridges-aperture.md new file mode 100644 index 00000000000..7ed5b17a4bb --- /dev/null +++ b/issues/kernel/a-32-bit-bar-needs-the-host-bridges-aperture.md @@ -0,0 +1,64 @@ +--- +status: open +kind: defect +opened: 2026-09-08 +--- + +# A 32-bit BAR cannot be handed to a process, because nothing here reads the host bridge's aperture + +`kernel/src/pcidev`'s window allocator places a claimed function's BARs on +2 MiB pages **above everything firmware described**. In 64 bits that is always +possible. Below 4 GiB it never is: the platform's fixed MMIO sits at +`0xFEC00000` and the UEFI map reaches it, so `window(narrow_end, PLATFORM_MMIO)` +answers `0x0..0x0` on every machine. Read off the ThinkPad T14, run 29: + +``` +pcidev: 24 functions; a 32-bit window comes from 0x0..0x0, a 64-bit one from 0x603dc00000..0x6040c00000 +pcidev: PCI 00:1f.6 NOT HANDED OVER — this machine has no 2 MiB-aligned address space above what firmware assigned to put a BAR in +``` + +That function is the bench's own NIC, and its BAR is 32-bit +(`bar0=0xbcf00000`). So the cable this project's test bench answers on cannot be +driven by a process at all until this is built. + +**Leaving the BAR where firmware put it is not the fix.** The kernel maps 2 MiB +pages, and on this machine the I219's BAR shares its page with the internal +NVMe's (`0xbcf00000` and `0xbce00000` are both inside `0xbce00000..0xbd000000`): +handing it over unmoved would put a disk controller's registers inside a +network daemon's mapping. `alone_in_its_page` is the assertion that says so, and +the refusal is right. + +**What is missing is a free run, and what is missing to find one is ACPI.** A +32-bit window is a run *between* things rather than a span above them, and three +of the four things it has to miss are readable already — the firmware map, the +BARs this bus assigned, and every range a PCI-to-PCI bridge forwards to a +secondary bus (`toyos_pci::bridge`, and `survey_low_space` prints all three on +the machine where the window comes out empty). The fourth is the host bridge's +own aperture: which addresses below 4 GiB the root complex decodes and forwards +to PCI at all. That is the `_CRS` of the `PNP0A08` device, an AML method, and +this kernel runs no AML. An address outside the aperture is not free space — it +is unrouted, and a read of it answers ones, which `Refusal::Dead` cannot tell +from a device that is simply not there. + +Linux's own answer is the same one: `acpi_pci_probe_root_resources` reads +`_CRS`, and the per-chipset fallbacks it keeps are quirks for firmware that gets +`_CRS` wrong, not an alternative to it. Reading a host bridge register such as +Intel's `TOLUD` instead would be chipset-specific and is not a road this project +takes. + +So the work is one of: + +- an AML interpreter far enough to evaluate `_CRS` on the host bridge, which is + a large thing to want for one method; or +- a 4 KiB mapping for a claimed BAR, which removes the need to move a BAR whose + page is shared and is a change in `mm` rather than here; or +- the owner ruling that some other source of the aperture is admissible. + +The survey is committed and prints the candidates; nothing hands one out. The +line it ends with says why: + +``` +pcidev: a run above is a candidate and not a claim — what says whether an address below 4 GiB +reaches this bus at all is the host bridge's own aperture, which is ACPI's `_CRS`, and this +kernel runs no AML +``` diff --git a/kernel/src/drivers/pci.rs b/kernel/src/drivers/pci.rs index 407c5ab1d8c..8141855934c 100644 --- a/kernel/src/drivers/pci.rs +++ b/kernel/src/drivers/pci.rs @@ -1,6 +1,6 @@ use alloc::vec::Vec; -use toyos_pci::{bar, caps, msi, msix}; +use toyos_pci::{bar, bridge, caps, msi, msix}; use crate::mm::Mmio; use crate::mm::paging::MmioPolicy; @@ -326,7 +326,7 @@ impl PciDevice { } cap.write_u16(msi.data(), data as u16); if let Some(mask) = msi.mask() { - cap.write_u32(mask, 0); + cap.write_u32(mask, msi::UNMASKED); } cap.write_u16(msi::MESSAGE_CONTROL, msi::Msi::enabled(control)); self.report_message( @@ -337,6 +337,41 @@ impl PciDevice { true } + /// Put MSI back off: the counterpart of [`Self::disable_msix`], and what a + /// claimed MSI function has in place of masking a table entry. + /// + /// The mask first where the function implements one, because that is the + /// per-vector lever and the one MSI-X's own hand-back uses; the enable bit + /// after, because every function has that one. A function still delivering + /// once its holder is gone writes its message into a slot with no reader. + pub fn disable_msi(&self) { + let Some(cap) = self.capabilities().find(|c| c.id() == msi::CAP_ID) else { return }; + let control = cap.read_u16(msi::MESSAGE_CONTROL); + if let Some(mask) = msi::Msi::decode(control).mask() { + cap.write_u32(mask, msi::MASKED); + } + cap.write_u16(msi::MESSAGE_CONTROL, msi::Msi::disabled(control)); + } + + /// Every memory range this function forwards to its secondary bus, below + /// 4 GiB. Empty on a function that is not a bridge. + /// + /// **Read, never probed**: these are the ranges nothing above this bridge + /// may hand out, and reading them costs the machine nothing — unlike + /// `bar_size`, which takes memory decode off for the length of its probe. + pub fn forwarded_below_4g(&self) -> Vec { + if self.read_config_u8(HEADER_TYPE) & !MULTI_FUNCTION != bridge::HEADER_TYPE_BRIDGE { + return Vec::new(); + } + let mut out = Vec::new(); + out.extend(bridge::window(self.read_config_u32(bridge::MEMORY_BASE))); + out.extend(bridge::prefetch_below_4g( + self.read_config_u32(bridge::PREFETCH_BASE), + self.read_config_u32(bridge::PREFETCH_BASE_UPPER), + )); + out + } + pub fn capabilities(&self) -> CapabilityIter<'_> { let first = self.mmio.read_u8(CAPABILITIES_PTR); CapabilityIter { device: self, walk: caps::CapWalk::new(), next: first } diff --git a/kernel/src/pcidev/mod.rs b/kernel/src/pcidev/mod.rs index ea8d9f58135..ecb3602c648 100644 --- a/kernel/src/pcidev/mod.rs +++ b/kernel/src/pcidev/mod.rs @@ -3,8 +3,9 @@ //! The line through the device is **who can name an address**. This module //! keeps config space — there is no write path to it from userland — puts the //! function in an address space of its own at the unit *before* it enables bus -//! mastering, programs the interrupt vector into its MSI-X table, and hands out -//! every device address a descriptor may carry. Nothing the holder writes into +//! mastering, programs the interrupt vector into whichever of the function's +//! two message mechanisms it has, and hands out every device address a +//! descriptor may carry. Nothing the holder writes into //! a descriptor can make the device touch memory the kernel did not grant it: //! the domain maps the grants and nothing else, and an address outside them is //! refused at the unit and recorded against that claim. @@ -16,7 +17,9 @@ //! //! **The BAR holding the MSI-X table or PBA is never mapped**: a holder that //! could rewrite the table could point the device's message at any address the -//! LAPIC decodes. +//! LAPIC decodes. An MSI function's message is in config space instead, which +//! is read-only from userland, so it needs no BAR withheld — the same rule +//! reaching a different register file, not a weaker one. //! //! **A function with no address space of its own is not handed over**, because //! every grant would answer with a physical address and a descriptor holding @@ -32,7 +35,7 @@ //! above is the mechanism and the reset is the belt. //! //! **What is read back, and what is not.** `Owned` is -//! `pci_function_is_exclusive`, `NoMsix` is `virtio_net_no_msix`, +//! `pci_function_is_exclusive`, `NoInterrupt` is `virtio_net_no_msix`, //! `Untranslated` is `iommu_virtio_platform`'s no-unit arm, the domain is //! `userdev_dma_fault`, and `SYS_DEVICE_REG_READ`'s bound is netd's own //! `config_space_is_bounded`. `Ambiguous`, `KernelDriven`, `Exhausted`, every @@ -121,12 +124,69 @@ struct Grant { bytes: u64, } +/// How a claimed function was made to speak, and what it takes to silence it. +/// +/// **The driver above the boundary cannot tell which one it got, and does not +/// care**: both deliver [`VECTORS`]`[slot]` into the same [`Interrupt`], and the +/// claim answers the same handle either way. What differs is where the message +/// lives — an MSI-X table entry in a mapping this kernel keeps for itself, or a +/// word of config space, which has no write path from userland at all — and so +/// what a hand-over back has to write to stop it. +enum Armed { + /// This function's one MSI-X table entry, mapped for the kernel alone. + Msix(Mmio), + /// MSI: the message is in this function's own config space and nothing was + /// mapped for it. + Msi, +} + +impl Armed { + fn name(&self) -> &'static str { + match self { + Self::Msix(_) => "MSI-X", + Self::Msi => "MSI", + } + } + + /// Stop this function delivering, for a holder that is gone. + fn silence(&self, pci: &PciDevice) { + match self { + Self::Msix(entry) => entry.write_u32(msix::ENTRY_VECTOR_CONTROL, msix::ENTRY_MASKED), + Self::Msi => pci.disable_msi(), + } + } + + /// Put the capability itself back off, for a hand-over that armed a vector + /// and was then refused: a function left enabled at a vector nobody holds + /// delivers into a slot with no reader. + fn undo(&self, pci: &PciDevice) { + match self { + Self::Msix(_) => pci.disable_msix(), + Self::Msi => pci.disable_msi(), + } + } +} + +/// MSI-X first, then MSI, and neither is somewhere a holder can write. +/// +/// **MSI-X first because it is the one this kernel can mask per vector**, and +/// because a function that publishes it is one whose table BAR is then kept out +/// of what the holder maps. MSI is not a lesser mechanism — the device performs +/// the same write to the same address — and the parts that have only it are not +/// rare: the ThinkPad's onboard I219 is one, which is where this arm came from. +fn arm(pci: &PciDevice, vector: u8) -> Option { + if let Some(entry) = pci.enable_msix(vector) { + return Some(Armed::Msix(entry)); + } + pci.enable_msi(vector).then_some(Armed::Msi) +} + /// What a live slot drives. The ISR never reads this. struct Bound { pci: PciDevice, space: DeviceSpace, - /// This function's one MSI-X table entry, mapped for the kernel alone. - entry: Mmio, + /// How this function was made to speak, and what silences it. + armed: Armed, id: PciId, /// Where each mappable BAR was put, and how much of it the function /// advertises; 0 bytes is a slot with no BAR this claim may map. @@ -308,6 +368,91 @@ pub fn publish(devices: &[PciDevice], maps: &[MemoryMapEntry]) { machine.wide.0, machine.wide.1, ); + let empty = machine.narrow.0 == 0; + let taken = machine.decoded.clone(); + drop(machine); + // Only on the machine where it is owed. A boot whose low space has room + // says nothing about it, and a survey printed every time would be thirty + // lines of a log that has one channel off this bench. + if empty { + survey_low_space(devices, maps, &taken); + } +} + +/// What is left below 4 GiB, said out loud on the machine where nothing is. +/// +/// **A refusal that says "no room" where the truth is "this module only ever +/// looks above everything" sends a reader to the wrong place**, and it sent one +/// there: the ThinkPad's I219 has a 32-bit BAR, [`window`] answered `0x0..0x0` +/// for the low space, and the refusal read as a full machine. Below 4 GiB there +/// is nothing above everything — the platform's fixed MMIO is at +/// [`PLATFORM_MMIO`] — so a 32-bit window is a free run *between* things rather +/// than a span above them. +/// +/// This prints the runs, and it accounts for exactly three things and names +/// them, because what it does not account for is the point: the firmware map, +/// the BARs this bus has assigned, and every range a bridge forwards to a +/// secondary bus. **It does not account for the host bridge's own aperture**, +/// which is what says whether an address below 4 GiB reaches this bus at all, +/// and which is ACPI's `_CRS` — an AML method this kernel does not run. So a +/// run below is a candidate for whoever reads the log, never a claim by this +/// module, and nothing here hands one out. +fn survey_low_space(devices: &[PciDevice], maps: &[MemoryMapEntry], decoded: &[(u16, u64, u64)]) { + let mut taken: Vec<(u64, u64)> = Vec::new(); + let mut note = |start: u64, end: u64| { + let (start, end) = (start.min(PLATFORM_MMIO), end.min(PLATFORM_MMIO)); + if start < end { + taken.push((start, end)); + } + }; + for entry in maps { + note(entry.start, entry.end); + } + for (_, start, end) in decoded { + note(*start, *end); + } + let mut bridges = 0usize; + for device in devices { + for forwarded in device.forwarded_below_4g() { + bridges += 1; + log!( + "pcidev: PCI {:02x}:{:02x}.{} forwards {:#x}..{:#x} to its secondary bus", + device.bus, + device.dev, + device.func, + forwarded.start, + forwarded.end, + ); + note(forwarded.start, forwarded.end); + } + } + taken.sort_unstable(); + let mut free: Vec<(u64, u64)> = Vec::new(); + let mut at = 0u64; + for (start, end) in taken { + if start > at { + free.push((at, start)); + } + at = at.max(end); + } + if at < PLATFORM_MMIO { + free.push((at, PLATFORM_MMIO)); + } + free.retain(|(start, end)| end - start >= PAGE_2M); + log!( + "pcidev: no 32-bit window. Below {PLATFORM_MMIO:#x} the firmware map, this bus's \ + assigned BARs and {bridges} forwarded bridge window(s) leave {} run(s) of 2 MiB or \ + more:", + free.len(), + ); + for (start, end) in free.iter() { + log!("pcidev: {start:#x}..{end:#x} ({} MiB)", (end - start) / (1024 * 1024)); + } + log!( + "pcidev: a run above is a candidate and not a claim — what says whether an address \ + below 4 GiB reaches this bus at all is the host bridge's own aperture, which is \ + ACPI's `_CRS`, and this kernel runs no AML" + ); } /// The span above `assigned` this module may hand out, or an empty one where @@ -329,9 +474,18 @@ fn window(assigned: u64, ceiling: u64) -> (u64, u64) { /// place. #[derive(Clone, Copy, PartialEq, Eq, Debug)] enum Refusal { - NoMsix, + NoInterrupt, Untranslated(IommuError), - NoWindow, + /// This machine published no window of that width at all. **Not the same + /// fact as a window that filled up**, and on the 32-bit side not the same + /// fact as a full machine either: `survey_low_space` is what says what is + /// actually left below 4 GiB. + NoWindow { wide: bool }, + /// The window exists and every page of it is already cut. + WindowFull { wide: bool }, + /// The function publishes nothing this claim may map — no memory BAR, or + /// only the one holding its own MSI-X table. + NoMappableBar, BarUnsizable(u8), BarUnplaceable(u8), BarResized(u8), @@ -341,21 +495,45 @@ enum Refusal { impl core::fmt::Display for Refusal { fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { match self { - Self::NoMsix => write!( + Self::NoInterrupt => write!( f, - "its MSI-X could not be armed, and a claim with no interrupt is a driver \ - that would never be told anything" + "neither its MSI-X nor its MSI could be armed, and a claim with no interrupt \ + is a driver that would never be told anything" ), Self::Untranslated(why) => write!( f, "it would have no address space of its own — {why} — and a process driving \ it would be given physical addresses to put in descriptors" ), - Self::NoWindow => write!( + // **Two sentences, because a 32-bit window is a different problem + // from a 64-bit one.** Above the highest address firmware described + // there is always room in 64 bits and never any in 32: the + // platform's fixed MMIO is up there. So the low answer names what + // would actually settle it, and the survey beside it in this log + // says what the machine has left. + Self::NoWindow { wide: true } => write!( f, - "this machine has no 2 MiB-aligned address space above what firmware \ + "this machine has no 2 MiB-aligned 64-bit address space above what firmware \ assigned to put a BAR in" ), + Self::NoWindow { wide: false } => write!( + f, + "its BAR is 32-bit and this module has no window below 4 GiB to put one in: \ + there is nothing above everything firmware described down there, so a window \ + has to be a free run between things — and what says a run is reachable is the \ + host bridge's aperture, which is ACPI's `_CRS` and which this kernel does not \ + read" + ), + Self::WindowFull { wide } => write!( + f, + "the {}-bit window this module cut is full", + if *wide { 64 } else { 32 } + ), + Self::NoMappableBar => write!( + f, + "it publishes no memory BAR this claim may map, so its holder would have no \ + registers to drive it through" + ), Self::BarUnsizable(i) => write!(f, "BAR {i} answers no size to bound a window by"), Self::BarUnplaceable(i) => write!(f, "BAR {i} did not take the address it was given"), Self::BarResized(i) => write!( @@ -422,12 +600,17 @@ pub fn claim(id: PciId) -> Result<(PciFunctionInfo, u8, Claim), ClaimError> { func: pci.func, _pad: [0; 5], }; + // Read off the hand-over rather than restated: which mechanism a + // function was armed on is the first thing a machine that never + // heard from its device is asked, and it is not a thing the driver + // above the boundary can see. + let armed = bound.armed.name(); *BOUND[slot].lock() = Some(bound); IRQ[slot].clear(); crate::iommu::note_user_owned(pci.bus, pci.dev, pci.func, Some(slot)); log!( "pcidev: PCI {:02x}:{:02x}.{} [{:04x}:{:04x}] handed over on slot {slot}, \ - vector {:#x}", + vector {:#x} on {armed}", pci.bus, pci.dev, pci.func, @@ -475,11 +658,11 @@ fn bring_up(pci: PciDevice, id: PciId, slot: usize) -> Result { // Decode must be on for a BAR to answer, and off across each move. pci.enable_memory_space(); - // Then the interrupt, still before a window is cut: a function whose MSI-X - // cannot be armed is one no holder could ever be told anything about, and - // `virtio_net_no_msix` reads that refusal off the console *and* the absence - // of any BAR line after it. - let entry = pci.enable_msix(VECTORS[slot]).ok_or(Refusal::NoMsix)?; + // Then the interrupt, still before a window is cut: a function neither + // mechanism can be armed on is one no holder could ever be told anything + // about, and `virtio_net_no_msix` reads that refusal off the console *and* + // the absence of any BAR line after it. + let armed = arm(&pci, VECTORS[slot]).ok_or(Refusal::NoInterrupt)?; // From here a refusal has to undo: a vector is armed, and the arms below // move the function's BARs. @@ -489,7 +672,7 @@ fn bring_up(pci: PciDevice, id: PciId, slot: usize) -> Result { Ok(Bound { pci, space, - entry, + armed, id, bar_at, bar_bytes, @@ -499,7 +682,7 @@ fn bring_up(pci: PciDevice, id: PciId, slot: usize) -> Result { }) } Err(why) => { - pci.disable_msix(); + armed.undo(&pci); Err(why) } } @@ -529,8 +712,11 @@ fn place_bars(pci: &PciDevice, table_bar: Option) -> Result<([u64; BARS], [u bar_bytes[index as usize] = size; index += step; } + // Its own refusal and not a window one: nothing about this machine's + // address space is wrong, and a reader sent to the window allocator would + // find it healthy. if bar_bytes.iter().all(|bytes| *bytes == 0) { - return Err(Refusal::NoWindow); + return Err(Refusal::NoMappableBar); } Ok((bar_at, bar_bytes)) } @@ -649,19 +835,22 @@ fn take_window(pci: &PciDevice, index: u8, wide: bool, span: u64) -> Result *top { - return Err(Refusal::NoWindow); + return Err(Refusal::WindowFull { wide }); } *next = end; at @@ -732,7 +921,7 @@ pub fn release(slot: usize) { fn tear_down(slot: usize, bound: Bound) { bound.pci.disable_bus_master(); - bound.entry.write_u32(msix::ENTRY_VECTOR_CONTROL, msix::ENTRY_MASKED); + bound.armed.silence(&bound.pci); crate::iommu::note_user_owned(bound.pci.bus, bound.pci.dev, bound.pci.func, None); for grant in bound.grants.iter() { if let Err(why) = bound.space.unmap(grant.at, grant.bytes) { diff --git a/src/bootlog.rs b/src/bootlog.rs index 4354bd85ecc..cf03b404033 100644 --- a/src/bootlog.rs +++ b/src/bootlog.rs @@ -226,6 +226,91 @@ pub fn record_millis(line: &str) -> Option { secs.checked_mul(1_000)?.checked_add(millis) } +/// The UTC second one record line carries, as seconds since the epoch. +/// +/// **The only field in a log a host clock can be held against.** Everything +/// else a record says is measured from this boot's own start, and a host that +/// wants to know whether something it saw happened *while this boot was up* has +/// nothing to compare that with. `logd` writes the wall clock; the panel writes +/// none, and this answers `None` for those lines rather than reading the +/// milliseconds field as a date. +pub fn record_unix_secs(line: &str) -> Option { + const EPOCH: &str = "1970-01-01"; + let mut fields = line.strip_prefix('[')?.split_whitespace(); + let day = crate::day::Day::parse(fields.next()?)?; + let days = crate::day::Day::parse(EPOCH).expect("the epoch is a date").until(day); + let (hours, rest) = fields.next()?.split_once(':')?; + let (minutes, seconds) = rest.split_once(':')?; + let (hours, minutes, seconds): (i64, i64, i64) = + (hours.parse().ok()?, minutes.parse().ok()?, seconds.parse().ok()?); + // A leap second is the one value past the ordinary range that is a time. + if !(0..24).contains(&hours) || !(0..60).contains(&minutes) || !(0..=60).contains(&seconds) { + return None; + } + u64::try_from(days * 86_400 + hours * 3_600 + minutes * 60 + seconds).ok() +} + +/// The span this boot's own records bracket: its first wall clock, and the one +/// on [`REBOOTING`] where the boot got that far. +fn record_unix_span(log: &str) -> Option<(u64, u64)> { + let first = log.lines().find_map(record_unix_secs)?; + let last = log.lines().rev().find_map(record_unix_secs)?; + let ended = log.lines().rfind(|l| l.contains(REBOOTING)).and_then(record_unix_secs); + Some((first, ended.unwrap_or(last))) +} + +/// Whether a second on the *host's* clock fell inside the boot this log is of, +/// at or after the record `after` names. +/// +/// **The records are the one place a host clock and a boot's clock meet.** +/// `window` is the host's own clock at the two ends of the span in which the +/// machine was running neither of its operating systems; this boot's records +/// have to fall inside it, which bounds the two clocks' disagreement against +/// the run's own data instead of assuming a bound. How far into that window an +/// observation came separates nothing: the window holds the operating system +/// that left and the one that came back as well as this boot. +pub fn host_second_inside_this_boot( + log: &str, + window: (u64, u64), + after: &str, + at: u64, +) -> Result<(), String> { + let (first, ended) = record_unix_span(log).ok_or_else(|| { + "this log carries no record with a wall clock on it, so there is nothing to hold the \ + host's own clock against" + .to_string() + })?; + let (from, to) = window; + if first < from || ended > to { + return Err(format!( + "this boot's own records run {first}..{ended} and the host watched the machine over \ + {from}..{to}: the two clocks disagree by more than the window is wide, so nothing \ + the host saw can be placed inside this boot" + )); + } + if at < first || at > ended { + return Err(format!( + "the host saw it at {at}, outside the {first}..{ended} this boot's own records \ + bracket: it came {} s {} the boot, so it belongs to the operating system on the \ + other side of it", + if at < first { first - at } else { at - ended }, + if at < first { "before" } else { "after" }, + )); + } + let after_at = log + .lines() + .find(|l| l.contains(after)) + .and_then(record_unix_secs) + .ok_or_else(|| format!("this boot has no {after:?} record carrying a wall clock"))?; + if at < after_at { + return Err(format!( + "the host saw it at {at}, {} s before this boot's {after:?} record at {after_at}", + after_at - at + )); + } + Ok(()) +} + /// When the last record in `log` was written, in milliseconds since boot. pub fn last_record_millis(log: &str) -> Option { log.lines().rev().find_map(record_millis) @@ -432,4 +517,82 @@ mod record_time_tests { assert_eq!(last_record_millis(log), Some(2_500)); assert_eq!(last_record_millis("nothing\n"), None); } + + /// One boot's records, verbatim from a stick the T14 wrote + /// (`lancase-run31/kernel.log` lines 1, 279 and 379). + const BOOT: &str = concat!( + "[2026-09-08 16:08:21 0.000 cpu0 boot] panic console: armed 1920x1080 stride=1920 \ + format=1 at 0x4000000000\n", + "[2026-09-08 16:08:22 1.258 cpu0] Boot: complete (1258ms)\n", + "[2026-09-08 16:08:44 23.340 cpu1] Rebooting.\n", + ); + + /// The whole window a host watches the machine over, wider than the boot at + /// both ends because firmware runs inside it. + fn window() -> (u64, u64) { + let (first, ended) = record_unix_span(BOOT).expect("a span"); + (first - 4, ended + 36) + } + + #[test] + fn a_second_inside_the_boot_and_after_the_named_record_is_this_boots() { + let (first, ended) = record_unix_span(BOOT).expect("a span"); + assert_eq!(ended - first, 23); + assert_eq!( + host_second_inside_this_boot(BOOT, window(), "Boot: complete", first + 2), + Ok(()) + ); + } + + /// **A reply after the boot handed the machine back is the next operating + /// system's, however early in the host's window it fell.** The window opens + /// no later than the boot's first record, so a reply 57 s into it came at + /// least 34 s after this boot's `Rebooting.` + #[test] + fn a_second_past_the_reboot_record_is_the_next_operating_systems() { + let (first, ended) = record_unix_span(BOOT).expect("a span"); + let why = host_second_inside_this_boot(BOOT, window(), "Boot: complete", first + 57) + .expect_err("57 s past the window's opening is past this boot"); + assert!(why.contains(&format!("{first}..{ended}")), "{why}"); + assert!(why.contains("34 s after the boot"), "{why}"); + } + + #[test] + fn a_second_before_the_named_record_is_refused_by_that_record() { + let (first, _) = record_unix_span(BOOT).expect("a span"); + let why = host_second_inside_this_boot(BOOT, window(), "Boot: complete", first) + .expect_err("the boot had not completed yet"); + assert!(why.contains("1 s before"), "{why}"); + let why = host_second_inside_this_boot(BOOT, window(), "netd: DHCP: lease ", first + 2) + .expect_err("this boot took no lease"); + assert!(why.contains("no \"netd: DHCP: lease \" record"), "{why}"); + } + + /// **The window is what bounds the two clocks' disagreement.** A boot whose + /// records fall outside the span the host watched it over is a boot whose + /// clock cannot be held against the host's at all, and the numbers are + /// printed rather than the conclusion. + #[test] + fn records_outside_the_hosts_own_window_place_nothing() { + let (first, ended) = record_unix_span(BOOT).expect("a span"); + let why = host_second_inside_this_boot(BOOT, (first + 5, ended + 36), "Rebooting.", ended) + .expect_err("the boot began before the host started watching"); + assert!(why.contains(&format!("{first}..{ended}")), "{why}"); + assert!(why.contains("disagree by more than the window"), "{why}"); + } + + /// The panel writes no wall clock, and its milliseconds field must not be + /// read as one: `[1.000 cpu0]` would otherwise parse `1.000` as a date and + /// answer some second in 1970. + #[test] + fn a_line_with_no_wall_clock_answers_none() { + assert_eq!(record_unix_secs("[1.000 cpu0] first"), None); + assert_eq!(record_unix_secs("not a record"), None); + assert_eq!(record_unix_secs("[2026-09-08 25:00:00 0.000 cpu0] x"), None); + assert_eq!(record_unix_secs("[2026-02-31 10:00:00 0.000 cpu0] x"), None); + assert_eq!(record_unix_span("[1.000 cpu0] first\n"), None); + let why = host_second_inside_this_boot("[1.000 cpu0] first\n", (0, 1), "x", 0) + .expect_err("a panel log carries no wall clock"); + assert!(why.contains("no record with a wall clock"), "{why}"); + } } diff --git a/src/build.rs b/src/build.rs index 56497284a93..b344adec72e 100644 --- a/src/build.rs +++ b/src/build.rs @@ -2607,6 +2607,7 @@ mod tests { "tests/e1000case/system.toml", "tests/jobcase/system.toml", "tests/jobdeadlinecase/system.toml", + "tests/lancase/system.toml", "tests/latencycase/system.toml", "tests/logrotatecase/system.toml", "tests/metalcase/system.toml", diff --git a/src/metal.rs b/src/metal.rs index 432d8860870..ae0bc6c184e 100644 --- a/src/metal.rs +++ b/src/metal.rs @@ -58,6 +58,25 @@ pub fn return_secs() -> u64 { const POLL_SECS: u64 = 5; +/// How often the machine's own address is pinged while it is not answering +/// `ssh`. +const PING_EVERY_SECS: u64 = 1; + +/// How long one probe waits for its reply. +const PING_WAIT_MS: u64 = 1_000; + +/// How long the address has to answer *nothing* before a reply counts as this +/// boot's. +/// +/// **`ssh` stops answering before the network does.** `reboot` takes `sshd` +/// down first and the interface some seconds later, so the machine that is +/// already "down" by this loop's reckoning still answers ICMP for a moment — +/// and a reply counted there would be the operating system that is leaving, +/// never the one being flashed. What is looked for is a reply *after* the +/// address went quiet, which is the window in which the only thing that can be +/// running is the image this loop wrote. +const PING_SILENCE_SECS: u64 = 5; + /// How long the boot stick gets to be there again once Ubuntu is up. /// /// **The bench's own device is the one judge there is of whether a reset left a @@ -129,6 +148,11 @@ pub enum Refusal { /// A lid key that no longer reads `ignore`, which is what keeps the machine up. Lid { key: &'static str, got: String }, Remote { what: String, status: String, stderr: String }, + /// The machine could not say what address it holds on the function the + /// flashed image claims, so the boot could not be reached over the cable. + Wire { nic: String, why: String }, + /// This host could not run the probe, which is a fact about the host. + Probe { why: String }, /// The machine did not go down, or did not come back. Silent { what: &'static str, secs: u64 }, /// The machine came back and the boot stick did not: the boot before this @@ -254,6 +278,18 @@ impl fmt::Display for Refusal { Self::Remote { what, status, stderr } => { write!(f, "{what} on the machine {status}: {stderr}") } + Self::Wire { nic, why } => write!( + f, + "the machine says nothing usable about PCI function {nic}: {why}. That is the \ + function the flashed image claims, and its address is the only one a boot of \ + that image could answer on" + ), + Self::Probe { why } => write!( + f, + "this host could not run `ping`: {why}. It is the only question this loop can \ + ask a boot while that boot is still up, so a run that cannot ask it \ + establishes nothing about the cable" + ), Self::Silent { what, secs } => write!( f, "the machine did not {what} within {secs} s, which is longer than every watchdog \ @@ -936,6 +972,160 @@ fn lid_policy(text: &str) -> Result<(), Refusal> { Ok(()) } +/// What the machine holds on the PCI function the flashed image claims. +/// +/// **Read off that function and not off a name.** The address the loop pings +/// has to be one a boot of this image could answer on, and the two operating +/// systems agree about it for exactly one reason: the function's MAC is the +/// same under both, so a DHCP server ordinarily hands both the same lease. The +/// MAC is carried out beside the address so the boot's own driver record can be +/// held to it. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Wire { + pub iface: String, + pub addr: std::net::Ipv4Addr, + /// Lower case, colon separated, as `/sys/class/net//address` writes it. + pub mac: String, +} + +/// `ip -4 -brief addr show `'s one line, as `Wire` needs it. +/// +/// The brief form is ` ...`, and an interface with no +/// address has no third field at all — which is the machine saying the cable is +/// out, and it is refused by name rather than read as some other interface's. +fn brief_address(iface: &str, text: &str) -> Result { + let line = text + .lines() + .find(|l| l.split_whitespace().next() == Some(iface)) + .ok_or_else(|| format!("`ip -4 -brief addr show {iface}` said {text:?}"))?; + let cidr = line + .split_whitespace() + .nth(2) + .ok_or_else(|| format!("{iface} holds no IPv4 address: {line:?}"))?; + cidr.split('/') + .next() + .unwrap_or(cidr) + .parse() + .map_err(|_| format!("{iface}'s address reads {cidr:?}")) +} + +/// The first reply after the silence: how far into the window it came, and when +/// it came on this host's clock. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Reply { + pub secs: u64, + /// Seconds since the epoch, UTC, taken when the probe answered. The probe + /// waits up to a second for its reply, so this is late by at most that. + pub at: u64, +} + +/// What this host saw across the window in which the machine was running +/// neither of its operating systems. +/// +/// **The two ends of the window are the host's own clock, and they are what a +/// judge holds the boot's records against.** Nothing else this loop reads can +/// place a host-side observation inside a boot: how far into the window a reply +/// came says only that it was in the window, which both operating systems are. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Watch { + pub from: u64, + pub to: u64, + pub reply: Option, +} + +/// The probe, running while the loop is inside `ssh` waiting for the machine. +/// +/// **The one thing this loop can ask a boot that is still running.** Everything +/// else it reads is on the stick, and the stick is read minutes later, from +/// Ubuntu. The probe is the host's own `ping`, an implementation of ICMP this +/// repository did not write. +struct Ping { + first: std::sync::Arc, String>>>, + stop: std::sync::Arc, + thread: std::thread::JoinHandle<()>, +} + +impl Ping { + /// Begin, now: the caller has just watched the machine stop answering + /// `ssh`, and the window this measures starts there. + fn start(addr: std::net::Ipv4Addr) -> Self { + let first = std::sync::Arc::new(std::sync::Mutex::new(Ok(None))); + let stop = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)); + let (mine, theirs) = (std::sync::Arc::clone(&first), std::sync::Arc::clone(&stop)); + let thread = std::thread::Builder::new() + .name("metal-ping".into()) + .spawn(move || { + let began = std::time::Instant::now(); + let mut quiet_since: Option = None; + let silence = std::time::Duration::from_secs(PING_SILENCE_SECS); + while !theirs.load(std::sync::atomic::Ordering::SeqCst) { + let answered = match ping_once(addr) { + Ok(answered) => answered, + Err(why) => { + *mine.lock().expect("the ping's answer") = Err(why); + return; + } + }; + if answered { + // A reply before the address has been quiet is the + // operating system that is going down, whose `sshd` + // stops before its interface does. + if quiet_since.is_some_and(|at| at.elapsed() >= silence) { + *mine.lock().expect("the ping's answer") = + Ok(Some(Reply { secs: began.elapsed().as_secs(), at: unix_now() })); + return; + } + quiet_since = None; + } else if quiet_since.is_none() { + quiet_since = Some(std::time::Instant::now()); + } + std::thread::sleep(std::time::Duration::from_secs(PING_EVERY_SECS)); + } + }) + .expect("the metal loop's ping probe could not be started"); + Self { first, stop, thread } + } + + /// Stop probing, and answer what the first reply after the silence was. + fn end(self) -> Result, Refusal> { + self.stop.store(true, std::sync::atomic::Ordering::SeqCst); + let _ = self.thread.join(); + let answer = self.first.lock().expect("the ping's answer").clone(); + answer.map_err(|why| Refusal::Probe { why }) + } +} + +/// This host's clock, as seconds since the epoch in UTC. +fn unix_now() -> u64 { + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .expect("a host clock before 1970 is a host to fix") + .as_secs() +} + +/// One probe, whose whole answer is whether the address replied. +/// +/// **A host with no `ping` and a cable with nothing on it are separate +/// answers.** A spawn that fails is this host's failing, and reporting it as +/// silence would red the boot for a binary the host does not have. +/// +/// **`-W` is milliseconds on this host and seconds on Linux**, and the two +/// spellings are three orders of magnitude apart: a bound written for one is a +/// probe that hangs for a quarter of an hour on the other. +fn ping_once(addr: std::net::Ipv4Addr) -> Result { + let wait = if cfg!(target_os = "macos") { + PING_WAIT_MS.to_string() + } else { + PING_WAIT_MS.div_ceil(1_000).to_string() + }; + Command::new("ping") + .args(["-n", "-c", "1", "-W", &wait, &addr.to_string()]) + .stdin(Stdio::null()) + .output() + .map(|out| out.status.success()) + .map_err(|e| e.to_string()) +} + /// The loop, over one target. struct Driver { target: Target, @@ -986,6 +1176,44 @@ impl Driver { answer(what, out).map(Some) } + /// What this machine holds on the function the flashed image claims: the + /// interface Ubuntu gave it, its address, and its MAC. + /// + /// Three reads and not one, so a machine that answers oddly is refused with + /// the read that was odd. None of them is a root command and none of them + /// writes. + fn wire(&self, nic: &str) -> Result { + let bad = |why: String| Refusal::Wire { nic: nic.to_string(), why }; + let at = shell_word(&format!("/sys/bus/pci/devices/{nic}/net")); + let listing = self + .ssh("listing the claimed function's interfaces", &format!("ls {at}")) + .map_err(|e| bad(e.to_string()))?; + let names: Vec<&str> = listing.split_whitespace().collect(); + // Exactly one, refused rather than resolved to the first: a function + // this loop cannot name one interface for is one whose address it would + // be guessing at. + let [iface] = names[..] else { + return Err(bad(format!("it answers {names:?} interface(s), and one is needed"))); + }; + let mac = self + .ssh( + "reading the claimed function's MAC", + &format!("cat {}", shell_word(&format!("/sys/class/net/{iface}/address"))), + ) + .map_err(|e| bad(e.to_string()))?; + let brief = self + .ssh( + "reading the claimed function's address", + &format!("ip -4 -brief addr show {}", shell_word(iface)), + ) + .map_err(|e| bad(e.to_string()))?; + Ok(Wire { + iface: iface.to_string(), + addr: brief_address(iface, &brief).map_err(bad)?, + mac: mac.trim().to_ascii_lowercase(), + }) + } + /// The loop refuses to run at all until the rule is on the machine. fn require_sudo(&self) -> Result<(), Refusal> { let probe = self.target.remote(Job::Probe, None)?; @@ -1083,9 +1311,25 @@ impl Driver { /// **`reboot` is `systemctl` and returns before the machine goes down**, so /// the machine is watched down before it is watched back up: a probe that /// caught dying Ubuntu would read a stick ToyOS had never booted. - fn ride_the_reboot(&self, secs: u64) -> Result { + /// + /// The window between the two is the only span in which the machine is + /// running the image this loop wrote, and [`Ping`] is what asks the cable + /// about it while it lasts — on the boots that name a function to ask it + /// over, and on no other. + fn ride_the_reboot( + &self, + secs: u64, + addr: Option, + ) -> Result<(u64, Option), Refusal> { self.wait(GOING_DOWN_SECS, "go down", false)?; - self.wait(secs, "come back", true) + let Some(addr) = addr else { + return Ok((self.wait(secs, "come back", true)?, None)); + }; + let from = unix_now(); + let ping = Ping::start(addr); + let back = self.wait(secs, "come back", true); + let reply = ping.end()?; + Ok((back?, Some(Watch { from, to: unix_now(), reply }))) } /// Wait for the log partition's device node, and say how long it took. @@ -1249,6 +1493,13 @@ pub struct Args { /// `toyos-fat32-check`. The outside judge, and the only reader of that /// volume in this tree that is not the family of code that wrote it. fat32_check: bool, + /// The PCI function this boot's image claims, in `/sys/bus/pci/devices`'s + /// spelling, for the boots this loop reaches over the cable. + /// + /// **A boot names it or the cable is not asked at all.** The reads are + /// three `ssh` round trips and the probe is a host binary, and a boot that + /// claims no NIC would be refused for a fact none of its judges reads. + nic: Option, } impl Args { @@ -1262,6 +1513,7 @@ impl Args { wait_secs: return_secs(), readback: None, fat32_check: false, + nic: None, }; let mut at = 0; while at < args.len() { @@ -1314,6 +1566,11 @@ impl Args { out.fat32_check = true; 1 } + "--nic" => { + out.about_a_boot.push("--nic"); + out.nic = Some(value()?); + 2 + } "--wait-secs" => { out.about_a_boot.push("--wait-secs"); let secs = value()?; @@ -1467,6 +1724,19 @@ pub fn run(args: &Args) -> Result, Refusal> { identity.vendor, identity.model ); + // Before the flash, because the address a boot of this image could answer + // on is one only the operating system that is still up can be asked for. + let wire = match &args.nic { + Some(nic) => { + let wire = driver.wire(nic)?; + println!( + "the claimed function {nic} is {} at {}, MAC {}", + wire.iface, wire.addr, wire.mac + ); + Some(wire) + } + None => None, + }; driver.flash(&image)?; let entry = driver.boot_entry(&image.esp)?; @@ -1480,8 +1750,22 @@ pub fn run(args: &Args) -> Result, Refusal> { return Ok(None); } - let back = driver.ride_the_reboot(args.wait_secs)?; + let (back, watched) = driver.ride_the_reboot(args.wait_secs, wire.as_ref().map(|w| w.addr))?; println!("the machine answered ssh again after {back} s"); + if let (Some(wire), Some(watch)) = (&wire, &watched) { + // **Something, and which something is not this loop's to say.** The + // machine's own wire comes back before its `sshd` does, so a reply in + // this window may be either operating system; the wall clock beside it + // is what a judge holds against the boot's own records. + match watch.reply { + Some(reply) => println!( + "{} answered a ping {} s into the window, after {PING_SILENCE_SECS} s of \ + silence, at {} UTC seconds", + wire.addr, reply.secs, reply.at + ), + None => println!("nothing answered a ping at {} while the machine was down", wire.addr), + } + } // Before the mount, so the stick's own answer is a number rather than // the reason a mount failed. let stick = driver.wait_for_the_stick()?; @@ -1508,7 +1792,7 @@ pub fn run(args: &Args) -> Result, Refusal> { println!("toyos-fat32-check: the log partition's {} bytes check out", bytes.len()); } if let Some(dir) = &args.readback { - write_readback(dir, &loader, &log, back, stick)?; + write_readback(dir, &loader, &log, back, stick, wire.as_ref(), watched)?; println!("readback written to {}", dir.display()); } // **Named by evidence, before the boot record is missed.** A boot that @@ -1626,10 +1910,27 @@ pub const READBACK_BOOT: &str = "boot.txt"; /// the outside judge read, so a complaint can be looked at rather than retold. pub const READBACK_VOLUME: &str = "log-partition.img"; -/// The two keys [`READBACK_BOOT`] carries, one ` ` per line. +/// The keys [`READBACK_BOOT`] carries, one ` ` per line. pub const BACK_SECS: &str = "back_secs"; pub const STICK_SECS_KEY: &str = "stick_secs"; +/// The cable: the address this loop pinged, the MAC of the function holding +/// it, and the host's own clock at the two ends of the window it asked across. +/// +/// **All four together or none of them.** They are written only by a boot that +/// named a function to ask over, and a judge that read three of them would be +/// placing an observation against a window it could not see. +pub const PING_ADDR_KEY: &str = "ping_addr"; +pub const WIRE_MAC_KEY: &str = "wire_mac"; +pub const WINDOW_FROM_KEY: &str = "window_from"; +pub const WINDOW_TO_KEY: &str = "window_to"; + +/// How far into that window the first reply came, and when it came on this +/// host's clock. **Both or neither**: an absent pair is `no` said where a zero +/// would be a reply in the first second, and the seconds alone place nothing. +pub const PING_SECS_KEY: &str = "ping_secs"; +pub const PING_AT_KEY: &str = "ping_at"; + /// Every file a readback directory carries, so a run that writes none of them /// leaves none of the last run's behind. pub const READBACK_FILES: &[&str] = @@ -1666,6 +1967,8 @@ fn write_readback( log: &str, back: u64, stick: u64, + wire: Option<&Wire>, + watched: Option, ) -> Result<(), Refusal> { let wrote = |path: &Path, text: &str| -> Result<(), Refusal> { std::fs::write(path, text) @@ -1678,7 +1981,17 @@ fn write_readback( // The boot's own millisecond count is in the kernel log and read from // there; this file carries only what the *host* clock measured, which no // log can. - wrote(&dir.join(READBACK_BOOT), &format!("{BACK_SECS} {back}\n{STICK_SECS_KEY} {stick}\n")) + let mut boot = format!("{BACK_SECS} {back}\n{STICK_SECS_KEY} {stick}\n"); + if let (Some(wire), Some(watch)) = (wire, watched) { + boot.push_str(&format!( + "{PING_ADDR_KEY} {}\n{WIRE_MAC_KEY} {}\n{WINDOW_FROM_KEY} {}\n{WINDOW_TO_KEY} {}\n", + wire.addr, wire.mac, watch.from, watch.to + )); + if let Some(reply) = watch.reply { + boot.push_str(&format!("{PING_SECS_KEY} {}\n{PING_AT_KEY} {}\n", reply.secs, reply.at)); + } + } + wrote(&dir.join(READBACK_BOOT), &boot) } /// Whether this boot was a loader pass that reported a record and booted no @@ -1725,6 +2038,70 @@ pub fn stick_secs(text: &str) -> Option { key(text, STICK_SECS_KEY) } +/// What one boot's readback says about the cable, or `None` where the loop was +/// not asked to reach one. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Cable { + pub addr: String, + /// The MAC of the function that held that address, as the operating system + /// before this boot reported it. + pub mac: String, + /// The host's own clock at the two ends of the window it asked across. + pub window: (u64, u64), + pub reply: Option, +} + +/// The cable a readback carries, **refusing every partial set by name**. +/// +/// A readback naming a reply and no window, or a window and no address, is one +/// this loop wrote in a shape no judge can read; answering `None` for it would +/// report a boot that answered as a boot nothing answered, which is the +/// opposite of what the run recorded. +pub fn cable(text: &str) -> Result, String> { + let addr = word(text, PING_ADDR_KEY); + let mac = word(text, WIRE_MAC_KEY); + let from = key(text, WINDOW_FROM_KEY); + let to = key(text, WINDOW_TO_KEY); + let secs = key(text, PING_SECS_KEY); + let at = key(text, PING_AT_KEY); + let named: Vec<&str> = [ + (addr.is_some(), PING_ADDR_KEY), + (mac.is_some(), WIRE_MAC_KEY), + (from.is_some(), WINDOW_FROM_KEY), + (to.is_some(), WINDOW_TO_KEY), + (secs.is_some(), PING_SECS_KEY), + (at.is_some(), PING_AT_KEY), + ] + .iter() + .filter_map(|(has, name)| has.then_some(*name)) + .collect(); + if named.is_empty() { + return Ok(None); + } + let (Some(addr), Some(mac), Some(from), Some(to)) = (addr, mac, from, to) else { + return Err(format!( + "this readback names {named:?} and a cable is {PING_ADDR_KEY}, {WIRE_MAC_KEY}, {WINDOW_FROM_KEY} and {WINDOW_TO_KEY} together" + )); + }; + let reply = match (secs, at) { + (Some(secs), Some(at)) => Some(Reply { secs, at }), + (None, None) => None, + _ => { + return Err(format!( + "this readback names {named:?}: a reply is {PING_SECS_KEY} and {PING_AT_KEY} together, and the seconds alone place it in neither operating system" + )); + } + }; + Ok(Some(Cable { addr, mac, window: (from, to), reply })) +} + +fn word(text: &str, name: &str) -> Option { + text.lines() + .find_map(|line| line.strip_prefix(name)) + .map(|rest| rest.trim().to_string()) + .filter(|got| !got.is_empty()) +} + fn key(text: &str, name: &str) -> Option { text.lines() .find_map(|line| line.strip_prefix(name)) @@ -1822,6 +2199,7 @@ mod tests { vec!["--fat32-check"], vec!["--dry-run"], vec!["--wait-secs", "60"], + vec!["--nic", "0000:00:1f.6"], ] { let mut words = vec!["--install-sudoers".to_string(), "/tmp/pw".to_string()]; words.extend(flag.iter().map(|w| (*w).to_string())); @@ -1933,6 +2311,73 @@ mod tests { assert_eq!(back_secs("back_secs later\n"), None); } + /// **A boot the cable did not answer is not a boot that answered in the + /// first second, and neither is a boot that was never asked.** + #[test] + fn a_ping_nothing_answered_is_written_as_no_answer() { + let asked = "back_secs 61\nstick_secs 2\nping_addr 192.168.1.46\n\ + wire_mac 8c:8c:aa:bb:cc:dd\nwindow_from 1757347650\nwindow_to 1757347711\n"; + let answered = format!("{asked}ping_secs 17\nping_at 1757347715\n"); + let answered_cable = cable(&answered).expect("a whole cable").expect("a cable"); + assert_eq!(answered_cable.addr, "192.168.1.46"); + assert_eq!(answered_cable.mac, "8c:8c:aa:bb:cc:dd"); + assert_eq!(answered_cable.window, (1_757_347_650, 1_757_347_711)); + assert_eq!(answered_cable.reply, Some(Reply { secs: 17, at: 1_757_347_715 })); + // A window nothing answered carries neither number. + let silent = cable(asked).expect("a whole cable").expect("a cable"); + assert_eq!(silent.reply, None); + // A boot that named no function to ask over carries none of it. + assert_eq!(cable("back_secs 46\nstick_secs 2\n"), Ok(None)); + assert_eq!(back_secs(&answered), Some(61)); + assert_eq!(stick_secs(&answered), Some(2)); + } + + /// **Every partial set is refused by name**, and the seconds without their + /// wall clock are the one that would otherwise read as no answer at all. + #[test] + fn half_a_cable_is_refused_rather_than_read_as_none() { + let whole = "ping_addr 192.168.1.46\nwire_mac 8c:8c:aa:bb:cc:dd\n\ + window_from 1757347650\nwindow_to 1757347711\n"; + for text in [format!("{whole}ping_secs 57\n"), format!("{whole}ping_at 1757347715\n")] { + let why = cable(&text).expect_err("half a reply is not a reply"); + assert!(why.contains("place it in neither operating system"), "{why}"); + } + for text in [ + "ping_addr 1.2.3.4\nwire_mac aa:bb\nwindow_from 1\n", + "ping_addr 1.2.3.4\nwindow_from 1\nwindow_to 2\n", + "ping_secs 57\nping_at 1757347715\n", + "ping_addr 192.168.1.46\n", + ] { + let why = cable(text).expect_err("half a cable is not a cable"); + assert!(why.contains("together"), "{why}"); + } + } + + /// **The address is the one on the function the image claims, and an + /// interface with none is refused rather than read as the next one's.** + /// `ip -4 -brief` prints the name, the state and then the addresses, and an + /// interface whose cable is out prints the first two and stops — which is + /// exactly the machine this loop must not go on to flash and then ping. + #[test] + fn an_interface_with_no_address_is_refused_by_name() { + let up = "enp0s31f6 UP 192.168.1.46/24 \n"; + assert_eq!(brief_address("enp0s31f6", up), Ok("192.168.1.46".parse().unwrap())); + + let down = "enp0s31f6 DOWN \n"; + assert!(brief_address("enp0s31f6", down).unwrap_err().contains("no IPv4 address")); + + // Another interface's line is not this one's answer, however many are + // printed: the T14 holds a Wi-Fi address and a Tailscale one, and a + // ping aimed at either is a ping only Ubuntu ever answers. + let many = "lo UNKNOWN 127.0.0.1/8\n\ + enp0s31f6 UP 192.168.1.46/24\n\ + wlp9s0 UP 192.168.1.244/24\n\ + tailscale0 UNKNOWN 100.92.92.12/32\n"; + assert_eq!(brief_address("enp0s31f6", many), Ok("192.168.1.46".parse().unwrap())); + assert_eq!(brief_address("wlp9s0", many), Ok("192.168.1.244".parse().unwrap())); + assert!(brief_address("enp0s31f7", many).unwrap_err().contains("ip -4 -brief")); + } + #[test] fn an_nvme_node_cannot_be_written_down() { for name in ["/dev/nvme0n1", "/dev/nvme0n1p3", "/dev/sda1", "/dev/sdaa", "/dev/SDA", "sda"] diff --git a/src/sourcegate.rs b/src/sourcegate.rs index c1393ca450a..285660803e4 100644 --- a/src/sourcegate.rs +++ b/src/sourcegate.rs @@ -559,6 +559,18 @@ const HOST_SPAWNS: &[Spawn] = &[ nothing else — no build, boot or gate reaches it, and the metal loop runs only \ when it is asked for", }, + Spawn { + arg: "\"ping\"", + sites: &[], + why: "the host's own ICMP client, in `src/metal.rs` alone and only across the window in \ + which the T14 is running neither of its operating systems. It is the one question \ + this repository can ask a metal boot while that boot is still up — everything \ + else it reads is on a stick, read minutes later from Ubuntu — and it is an \ + implementation of ICMP nobody here wrote, which is what makes it an oracle for the \ + stack under test rather than a second opinion from it. Outside the bar and \ + declared by nothing else: no build, boot or gate reaches it, and the metal loop \ + runs only when it is asked for", + }, Spawn { arg: "\"/sbin/newfs_msdos\"", sites: &[], diff --git a/tests/common/faults.rs b/tests/common/faults.rs index c1338bad029..1189e99aebb 100644 --- a/tests/common/faults.rs +++ b/tests/common/faults.rs @@ -285,7 +285,12 @@ pub fn virtio_net_no_msix() -> Result<(), String> { // whose holder would never be told anything, and handing it over anyway // would be handing out a device that looks alive and never speaks. log.must_say("pcidev: PCI 00:03.0 NOT HANDED OVER")?; - log.must_say("its MSI-X could not be armed")?; + // **Both mechanisms, named.** `pcidev` arms MSI-X and falls back to MSI, so + // this refusal is owed only by a function that has neither — and a virtio + // function stripped of its MSI-X table publishes no MSI capability to fall + // back to. A predicate naming one of them alone would be satisfied on a + // machine that armed the other and handed the function over. + log.must_say("neither its MSI-X nor its MSI could be armed")?; log.must_not_say("[1af4:1041] handed over")?; // And the refusal is the *whole* of it: no BAR moved for a function nobody // can be given one. diff --git a/tests/common/iommu.rs b/tests/common/iommu.rs index c6768648b90..66df45e8d8e 100644 --- a/tests/common/iommu.rs +++ b/tests/common/iommu.rs @@ -794,11 +794,12 @@ fn no_unit_is_no_claim(log: &Serial) -> Result<(), String> { log.must_not_say("handed over on slot")?; // **And the refusal spent nothing on the way out.** No BAR was moved, so // this function's BARs are still where firmware put them, and no vector was - // programmed into its MSI-X table — which is what says `bring_up` asks for - // the address space *before* it touches the function. `slot_space` put back - // below `place_bars` reds here. + // programmed into *either* of the two mechanisms `bring_up` may arm — which + // is what says it asks for the address space before it touches the function. + // `slot_space` put back below `place_bars` reds here. log.must_not_say(BAR_MOVED)?; log.must_not_say(MSIX_ARMED)?; + log.must_not_say(MSI_ARMED)?; log.must_say("init: netd: pci:1af4:1041 is on this machine and could not be handed over")?; // netd's own exit is not read here: it speaks after the ready marker this // capture ends at. It is the same endowment-is-empty path @@ -815,6 +816,13 @@ fn no_unit_is_no_claim(log: &Serial) -> Result<(), String> { const BAR_MOVED: &str = "pcidev: PCI 00:03.0 BAR"; const MSIX_ARMED: &str = "PCI 00:03.0: msix address="; +/// The other mechanism a hand-over may arm, whose absence the no-unit arm owes +/// as well. `bring_up` falls back to MSI, so "no MSI-X message was written" +/// stopped being the whole of "no vector was programmed" the moment it did — +/// and the two lines differ by one character, which is why each is spelled once +/// here rather than at its use. +const MSI_ARMED: &str = "PCI 00:03.0: msi address="; + /// The control that makes the two arms above mean something: a guest that /// declines the feature its host offered gets no device, not a bypassing one. /// `virtio_validate_features` returns `-EFAULT` and `virtio_set_status` returns diff --git a/tests/common/lan.rs b/tests/common/lan.rs new file mode 100644 index 00000000000..402480105ae --- /dev/null +++ b/tests/common/lan.rs @@ -0,0 +1,349 @@ +//! The cable: netd taking this machine's address from the network, and the T14 +//! answering the development host on it. +//! +//! Every line read here is a record. On the T14 a userland `println!` reaches +//! `Backend::None`, so what crosses to the stick is the kernel's log — into +//! which netd's `say!` writes, being a `write` to a console object. + +use std::path::Path; + +use toyos_build::bootlog; +use toyos_build::metalprofile::Profile; + +use super::metal; +use super::qemu::{self, BootOptions, QemuInstance}; +use super::serial; + +/// The boot config the T14 arm flashes, and the name every profile row for that +/// boot is under. +pub const CONFIG: &str = "tests/lancase"; +pub const BOOT: &str = "lancase"; + +/// The one job on that boot: it holds the machine up while the host pings it. +pub const JOBS: &[&str] = &["test_rs_lan_hold"]; + +/// The config the QEMU arm boots — the Intel driver in front of the user-mode +/// backend, which is the same driver the T14 arm runs and the only DHCP server +/// this host can put in front of it. +const QEMU_CONFIG: &str = "tests/e1000case"; + +/// What QEMU's user-mode backend leases, and what it says about the network it +/// leases on. Its own defaults, not this repository's: they are the oracle. +const SLIRP_ADDRESS: &str = "10.0.2.15"; +const SLIRP_PREFIX: u8 = 24; +const SLIRP_ROUTER: &str = "10.0.2.2"; +const SLIRP_DNS: &str = "10.0.2.3"; + +/// The card the T14 arm claims, as the kernel and the manifest spell it. +const ID: &str = "8086:15fc"; + +/// The PCI function that card is, as `/sys/bus/pci/devices` spells it: the +/// cable the metal loop reaches this boot over while it runs. +pub const NIC: &str = "0000:00:1f.6"; + +/// The records this pair of arms is written against, spelled once. +/// +/// They are netd's own `say!` lines, and netd is another crate: what holds the +/// two spellings together is that a boot missing any of these fails here by +/// name rather than passing quietly. +const MAC: &str = "netd: MAC "; +const LEASE: &str = "netd: DHCP: lease "; +const LINK_UP: &str = "netd: I219: link up at "; +const READY: &str = "netd: ready, at most "; +const NO_LEASE: &str = "netd: DHCP: no lease as toyos-t14 in "; + +/// netd's own `dhcp::LEASE_BOUND`: how long it waits before saying it has no +/// address. +const LEASE_BOUND_SECS: u64 = 20; + +/// The host-name option (RFC 2132 §3.14) as it goes out on the wire: the kind, +/// the length, and the name netd asks its network to record it under. +const HOST_NAME_OPTION: &[u8] = b"\x0c\x09toyos-t14"; + +/// One lease, as the record carries it. +#[derive(Debug, PartialEq, Eq)] +pub struct Lease { + pub address: String, + pub prefix: u8, + pub server: String, + pub gateway: String, + pub dns: Vec, + /// Milliseconds between netd starting and the lease landing. + pub ms: u64, +} + +/// The lease record, read out of a boot's log. +/// +/// Anchored on the record's own words rather than on positions, so a line that +/// grows a field still reads and one that loses a field is refused by name. +pub fn lease_in(text: &str) -> Result { + let line = text + .lines() + .find(|l| l.contains(LEASE)) + .ok_or_else(|| format!("no {LEASE:?} record: this boot took no address from its network"))?; + let unreadable = |what: &str| format!("{line:?} carries no {what}"); + let after = |head: &str, tail: &str| -> Result { + let (_, rest) = line.split_once(head).ok_or_else(|| unreadable(head))?; + let (got, _) = rest.split_once(tail).ok_or_else(|| unreadable(tail))?; + Ok(got.to_string()) + }; + let cidr = after(LEASE, " from ")?; + let (address, prefix) = cidr.split_once('/').ok_or_else(|| unreadable("an address/prefix"))?; + let dns = after(", dns [", "]")?; + Ok(Lease { + address: address.to_string(), + prefix: prefix.parse().map_err(|_| unreadable("a prefix length"))?, + server: after(" from ", ",")?, + gateway: after(", gateway ", ",")?, + dns: dns.split_whitespace().map(str::to_string).collect(), + ms: after("], ", " ms after netd came up")? + .parse() + .map_err(|_| unreadable("a millisecond count"))?, + }) +} + +/// How long after the driver came up the link did, out of the driver's own +/// record. +pub fn link_up_ms(text: &str) -> Result { + let line = text.lines().find(|l| l.contains(LINK_UP)).ok_or_else(|| { + format!("no {LINK_UP:?} record: this boot's card never reported a link") + })?; + let (_, rest) = line.split_once(", ").ok_or_else(|| { + format!("{line:?} says nothing about when the link came up, so the card was already up") + })?; + rest.split_once(" ms after the driver came up") + .ok_or_else(|| format!("{line:?} carries no link-up time"))? + .0 + .parse() + .map_err(|_| format!("{line:?} carries no readable link-up time")) +} + +/// The T14's judge: the claim, the card, the lease, and the host's own ping. +pub fn on_metal(back: &metal::Readback) -> Result<(), String> { + let profile = Profile::load(&super::compile::repo_root()).map_err(|why| why.to_string())?; + let kernel = back.kernel(); + let text = kernel.text(); + let mut bad: Vec = Vec::new(); + let cable = back.cable.as_ref().ok_or_else(|| { + format!( + "{}'s readback carries no cable: this boot was driven by a loop that was not asked \ + to reach it over one, so nothing here is about the network", + back.label + ) + })?; + + // The kernel's own account of the hand-over, which is where an interrupt + // mechanism the substrate cannot arm is refused by name. A boot with no + // hand-over line carries the refusal instead, and quoting it is the whole + // diagnosis. + let handed = format!("[{}] handed over on slot", ID); + match text.lines().find(|l| l.contains(&handed)) { + Some(line) => eprintln!(" [lan] {}", line.trim()), + None => bad.push(match text.lines().find(|l| l.contains("NOT HANDED OVER")) { + Some(line) => format!("the kernel refused this function: {}", line.trim()), + None => format!( + "no `{handed}` record and no refusal either: nothing on this machine claimed \ + {ID}, so `tests/lancase` was flashed onto a machine that has no such card" + ), + }), + } + + for owed in [MAC, LINK_UP, READY] { + if !text.contains(owed) { + bad.push(format!("no {owed:?} record")); + } + } + + // **The MAC is what makes an answered ping this boot's.** The address was + // read off the same PCI function under the operating system before this + // one, and a MAC does not change with the operating system — so a driver + // reporting this MAC is the driver holding that address, and a reply from + // anything else at it is some other interface's. + let mac = format!("{MAC}{}", cable.mac); + if !text.contains(&mac) { + bad.push(format!( + "no {mac:?} record: the card this boot brought up is not the one that held {} \ + before it", + cable.addr + )); + } + + match link_up_ms(text) { + Ok(ms) => { + eprintln!(" [lan] the link came up {ms} ms after the driver did"); + if let Err(why) = profile.judge(&format!("lan.{}.link_up_ms", back.label), ms) { + bad.push(why.to_string()); + } + } + Err(why) => bad.push(why), + } + + match lease_in(text) { + Ok(lease) => { + eprintln!( + " [lan] leased {}/{} from {} in {} ms, gateway {}, dns {:?}", + lease.address, lease.prefix, lease.server, lease.ms, lease.gateway, lease.dns + ); + if let Err(why) = profile.judge(&format!("lan.{}.lease_ms", back.label), lease.ms) { + bad.push(why.to_string()); + } + if lease.address != cable.addr { + bad.push(format!( + "this boot leased {} and the host pinged {}, which the router hands this \ + MAC under the operating system before it — so either something else \ + answered or that server does not repeat a lease across the two", + lease.address, cable.addr + )); + } + } + Err(why) => bad.push(why), + } + + match cable.reply { + Some(reply) => { + eprintln!( + " [lan] {} answered the host's ping {} s into the window", + cable.addr, reply.secs + ); + // The cost of the reply, priced. What says the reply was this + // boot's is the wall clock beside it, never how far into the window + // it came: the window holds both of this machine's operating + // systems. + if let Err(why) = profile.judge(&format!("boot.{}.ping_secs", back.label), reply.secs) { + bad.push(why.to_string()); + } + if let Err(why) = + bootlog::host_second_inside_this_boot(text, cable.window, LEASE, reply.at) + { + bad.push(why); + } + } + None => bad.push(format!( + "nothing answered a ping at {} while this machine was between its two operating \ + systems", + cable.addr + )), + } + + if let Err(why) = back.job_passed(JOBS[0]) { + bad.push(why); + } + + if bad.is_empty() { + return Ok(()); + } + Err(format!("{} finding(s):\n {}", bad.len(), bad.join("\n "))) +} + +/// The QEMU arm: the client, against a DHCP server this repository did not +/// write. +/// +/// Every field of the lease is checked against what the user-mode backend +/// serves, because a client that dropped the router option or read the mask off +/// the wrong option would otherwise pass on a machine where the answers happen +/// to agree. And the readiness line is checked to come *after* the lease: every +/// other arm in this suite waits for that line and then connects, so a netd that +/// announced itself before it had an address would hand those arms a stack with +/// none. +pub fn lan_dhcp_lease( + _test_config: &Path, + _c_bins: &[(String, Vec)], + _rust_bins: &[(String, Vec)], +) -> Result<(), String> { + let case = super::compile::repo_root().join(QEMU_CONFIG); + let dump = wire_dump("lease"); + let options = BootOptions { + profile: qemu::Profile::E1000e, + wire_dump: Some(dump.clone()), + ..Default::default() + }; + if !qemu::profile_argv(&options).iter().any(|a| a.contains("e1000e")) { + return Err("this test needs an Intel NIC and the profile has none".to_string()); + } + let mut guest = QemuInstance::boot_with_options(&case, &[], &[], options); + let mut console = guest.boot_log().to_string(); + qemu::await_marker(&mut guest, &mut console, READY, "netd to take an address")?; + console.push_str(&guest.drain_serial(std::time::Duration::from_millis(500))); + let log = serial::Serial::named("the lan boot", console.as_str()); + + let lease = lease_in(log.text())?; + let want = Lease { + address: SLIRP_ADDRESS.to_string(), + prefix: SLIRP_PREFIX, + server: SLIRP_ROUTER.to_string(), + gateway: SLIRP_ROUTER.to_string(), + dns: vec![SLIRP_DNS.to_string()], + ms: lease.ms, + }; + if lease != want { + return Err(format!( + "the client read this lease as {lease:?} and the backend serves {want:?}" + )); + } + // The order, and not merely the presence of both. + log.must_say_after(LEASE, READY)?; + log.must_say(LINK_UP)?; + let ms = link_up_ms(log.text())?; + eprintln!( + " [lan] the emulated link came up in {ms} ms and the lease landed {} ms after netd \ + started", + lease.ms + ); + log.must_be_clean()?; + asked_under_its_own_name(&dump) +} + +/// The client on a wire with nothing at the other end. +/// +/// **The refusal the lease boot cannot reach.** A machine whose network never +/// answers still has to announce itself, because every other arm in this suite +/// waits for that line and connects after it — a netd that stayed silent would +/// hang each of them instead of refusing their connects one at a time. +pub fn lan_no_lease( + _test_config: &Path, + _c_bins: &[(String, Vec)], + _rust_bins: &[(String, Vec)], +) -> Result<(), String> { + let case = super::compile::repo_root().join(QEMU_CONFIG); + let options = + BootOptions { profile: qemu::Profile::E1000eNoServer, ..Default::default() }; + let mut guest = QemuInstance::boot_with_options(&case, &[], &[], options); + let mut console = guest.boot_log().to_string(); + // Drained rather than waited on: netd owes its line inside its own bound + // and the guest says nothing at all until then, which every wait in this + // harness reads as a machine that stopped. + console.push_str(&guest.drain_serial(std::time::Duration::from_secs(LEASE_BOUND_SECS + 10))); + let log = serial::Serial::named("the lan boot with no server", console.as_str()); + if let Ok(lease) = lease_in(log.text()) { + return Err(format!("a wire with no server leased {lease:?}")); + } + log.must_say_after(NO_LEASE, READY)?; + eprintln!(" [lan] no server answered and netd said so, then served anyway"); + Ok(()) +} + +/// Where this process writes the frames one boot put on its wire. +fn wire_dump(which: &str) -> std::path::PathBuf { + let at = std::env::temp_dir() + .join(format!("toyos-lan-{which}-{}.pcap", std::process::id())); + let _ = std::fs::remove_file(&at); + at +} + +/// **The one place the host-name option can be read.** A server that ignores it +/// writes nothing about it and answers the same lease either way, so the frames +/// the client sent are the only evidence that it asked at all. +fn asked_under_its_own_name(dump: &Path) -> Result<(), String> { + let frames = std::fs::read(dump).map_err(|e| format!("{}: {e}", dump.display()))?; + let asked = frames.windows(HOST_NAME_OPTION.len()).any(|w| w == HOST_NAME_OPTION); + let _ = std::fs::remove_file(dump); + if !asked { + return Err(format!( + "none of the {} bytes this client put on the wire carries the host-name option \ + {HOST_NAME_OPTION:?}", + frames.len() + )); + } + eprintln!(" [lan] the client asked under its own name on the wire"); + Ok(()) +} diff --git a/tests/common/metal.rs b/tests/common/metal.rs index 43b310ccd16..453b47d0be5 100644 --- a/tests/common/metal.rs +++ b/tests/common/metal.rs @@ -62,6 +62,14 @@ pub struct Arm { /// kernel, and that is what most of the suite wants: it is the artifact the /// owner flashes. pub features: &'static [&'static str], + /// The PCI function this boot's image claims, where the loop is to reach + /// the boot over its cable while it runs. + /// + /// **`None` on every boot that does not ask.** Reading it costs three `ssh` + /// round trips before the flash and the probe costs a host binary, and a + /// boot whose judges read no cable would be refused for a fact none of them + /// looks at. + pub nic: Option<&'static str>, } /// The ordinary arm: one boot, and the fields a caller must still say. @@ -75,7 +83,7 @@ pub const fn once( params: &'static [&'static str], jobs: &'static [&'static str], ) -> Arm { - Arm { boot, config, params, jobs, features: &[] } + Arm { boot, config, params, jobs, features: &[], nic: None } } /// One boot carrying members that are **discovered rather than registered**. @@ -204,6 +212,10 @@ pub struct Readback { /// this machine holds, and it is a row rather than the reason a mount /// happened to work. pub stick_secs: u64, + /// What the host asked the cable while the machine was between its two + /// operating systems, and `None` on every boot that named no function to + /// ask over. + pub cable: Option, } impl Readback { @@ -424,6 +436,8 @@ struct Batch { jobs: Vec, files: Vec<(String, Vec)>, links: Vec<(String, String)>, + /// [`Arm::nic`], carried to the invocation that drives this boot. + nic: Option<&'static str>, } impl Batch { @@ -468,6 +482,7 @@ fn batches( jobs: boot.jobs.clone(), files: boot.files.clone(), links: boot.links.clone(), + nic: None, }, ); if was.is_some() { @@ -484,21 +499,25 @@ fn batches( jobs: Vec::new(), files: Vec::new(), links: Vec::new(), + nic: arm.nic, }); if batch.config != arm.config || batch.params != arm.params || batch.features != arm.features + || batch.nic != arm.nic { return Err(format!( - "{name} rides the boot {:?} as ({}, {:?}, {:?}) and another row rides it \ - as ({}, {:?}, {:?}); one boot is one image", + "{name} rides the boot {:?} as ({}, {:?}, {:?}, {:?}) and another row rides \ + it as ({}, {:?}, {:?}, {:?}); one boot is one image", arm.boot, arm.config, arm.params, arm.features, + arm.nic, batch.config, batch.params, - batch.features + batch.features, + batch.nic )); } batch.add(arm.jobs.iter().map(|j| (*j).to_string())); @@ -662,8 +681,8 @@ fn fingerprint(text: &str) -> u64 { /// The invocation that turns one image into one readback. Written down in the /// staged request and run by [`Mode::Drive`], so the two cannot differ. -fn invocation(image: &Path, home: &Path) -> Vec { - vec![ +fn invocation(image: &Path, home: &Path, nic: Option<&str>) -> Vec { + let mut words = vec![ "run".to_string(), "--bin".to_string(), "toyos-metal".to_string(), @@ -677,7 +696,12 @@ fn invocation(image: &Path, home: &Path) -> Vec { // `/log` has no reader of those bytes that is not the family of code // that wrote them. "--fat32-check".to_string(), - ] + ]; + if let Some(nic) = nic { + words.push("--nic".to_string()); + words.push(nic.to_string()); + } + words } fn read_readback(dir: &Path, label: &str) -> Result { @@ -695,6 +719,7 @@ fn read_readback(dir: &Path, label: &str) -> Result { .ok_or_else(|| format!("{label}'s boot file names no `back_secs`: {boot:?}"))?; let stick_secs = toyos_build::metal::stick_secs(&boot) .ok_or_else(|| format!("{label}'s boot file names no `stick_secs`: {boot:?}"))?; + let cable = toyos_build::metal::cable(&boot).map_err(|why| format!("{label}: {why}"))?; Ok(Readback { label: label.to_string(), boot_ms: bootlog::boot_millis(&kernel), @@ -702,6 +727,7 @@ fn read_readback(dir: &Path, label: &str) -> Result { kernel, back_secs, stick_secs, + cable, }) } @@ -818,7 +844,7 @@ pub fn run( request.push_str(&format!( "\n{label}\n image: {}\n cargo {}\n", image.display(), - invocation(image, &at(dir, label)).join(" ") + invocation(image, &at(dir, label), batches[*label].nic).join(" ") )); } let path = dir.join("request.txt"); @@ -845,7 +871,7 @@ pub fn run( let mut refused: BTreeMap<&str, String> = BTreeMap::new(); if mode == Mode::Drive { for (label, image) in &images { - let words = invocation(image, &at(dir, label)); + let words = invocation(image, &at(dir, label), batches[*label].nic); eprintln!("[metal] {label}: cargo {}", words.join(" ")); match Command::new("cargo").args(&words).current_dir(&root).status() { Ok(status) if status.success() => {} diff --git a/tests/common/mod.rs b/tests/common/mod.rs index a832cc19e5c..e9a3cb85e6f 100644 --- a/tests/common/mod.rs +++ b/tests/common/mod.rs @@ -25,6 +25,9 @@ pub mod https; pub mod iommu; #[allow(dead_code)] pub mod irqcensus; +/// The cable: netd's address, and the T14 answering the host on it. +#[allow(dead_code)] +pub mod lan; #[allow(dead_code)] pub mod logread; #[allow(dead_code)] diff --git a/tests/common/qemu.rs b/tests/common/qemu.rs index 136a3dc5cd8..509405f0eb4 100644 --- a/tests/common/qemu.rs +++ b/tests/common/qemu.rs @@ -1093,6 +1093,13 @@ pub enum Profile { /// NIC, and everything else — console, sound, disks — unchanged. The only /// machine in reach on which netd's Intel driver runs at all. E1000e, + /// [`Profile::E1000e`] with its cable plugged into nothing. + /// + /// The one machine in this suite on which a DHCP client gets no answer: + /// the user-mode backend serves a lease whatever else it is told to + /// restrict, so no profile that has one can ask what a boot does on a + /// network that never replies. + E1000eNoServer, Gop, /// A virtio-gpu function and no VGA: the owner's own desktop, and the one /// machine where a mode change can succeed rather than answering @@ -1480,6 +1487,10 @@ enum Nic { /// QEMU's `e1000e`, which is the 82574L at `8086:10d3`: the same register /// file the ThinkPad T14's onboard I219 has. E1000e, + /// The same card on a hub nothing else is plugged into: a link the guest + /// brings up and puts frames onto, with no host, router or server at the + /// other end. + E1000eNoServer, } /// Everything a profile decides about the machine, in one table. A new @@ -1663,6 +1674,7 @@ impl Profile { }, Self::HeadlessNoIommu => Shape { iommu: None, ..Self::Headless.shape() }, Self::E1000e => Shape { nic: Nic::E1000e, ..Self::Headless.shape() }, + Self::E1000eNoServer => Shape { nic: Nic::E1000eNoServer, ..Self::Headless.shape() }, Self::VirtioNetNoMsix => Shape { vga: "none", panel: None, @@ -2305,6 +2317,10 @@ pub struct BootOptions { /// carries no `-netdev` for it to reach, which [`ssh_forward_argv`] is /// what a test refuses before it boots. pub ssh_port: Option, + /// Write every frame this machine's NIC sends or receives to this file, in + /// pcap. **The only way to read what the guest asked for**: a request the + /// server ignores reaches no log on either side. + pub wire_dump: Option, } /// Where the guest sees the host under QEMU's user-mode networking, and where @@ -2387,6 +2403,7 @@ impl Default for BootOptions { extra_root_files: Vec::new(), log_stream: None, ssh_port: None, + wire_dump: None, } } } @@ -4205,6 +4222,16 @@ fn qemu_command( .arg("-device") .arg("e1000e,netdev=net0"); } + Nic::E1000eNoServer => { + qemu.arg("-netdev") + .arg("hubport,id=net0,hubid=0") + .arg("-device") + .arg("e1000e,netdev=net0"); + } + } + if let Some(at) = &options.wire_dump { + qemu.arg("-object") + .arg(format!("filter-dump,id=wire,netdev=net0,file={}", at.display())); } if shape.virtio.present() { diff --git a/tests/lancase/system.toml b/tests/lancase/system.toml new file mode 100644 index 00000000000..170c2ce8c97 --- /dev/null +++ b/tests/lancase/system.toml @@ -0,0 +1,27 @@ +# The one boot that runs netd in front of the ThinkPad T14's own NIC. +# +# A directory of its own for the reason `tests/e1000case` is one: a program that +# names a card the machine does not have costs an `init:` refusal line on every +# boot of the config that does, and no machine has both. +# +# Nothing here reaches the internal NVMe. There is no `[disks]` row, and the +# only device any program on this boot claims is the PCI function named below. + +[boot] +start = ["logd", "netd", "test-runner"] + +[programs.logd] +syscap = ["logread"] +receives = ["netd"] + +# netd holds the NIC's PCI function and drives it: the descriptor rings, the +# register window and the interrupt are its own, and the kernel keeps only the +# claim. Named by vendor and device rather than by slot, so one row finds the +# card wherever firmware put it. +[programs.netd] +serves = ["netd"] +devices = ["pci:8086:15fc"] + +[programs.test-runner] +receives = ["netd"] +syscap = ["logread"] diff --git a/tests/metal-profile.toml b/tests/metal-profile.toml index e99f0df0f1e..901a30e8c5b 100644 --- a/tests/metal-profile.toml +++ b/tests/metal-profile.toml @@ -609,3 +609,54 @@ name = "boot.ccorpus-2.stick_secs" unit = "s" ceiling = 30 ceiling_from = "as boot.testcases.stick_secs" + +# --- the cable: the boot that runs netd in front of the T14's own I219 --- +# The three boot facts below are the machine's own, off the last boot of this +# config; its claim was refused at the 32-bit BAR, which makes them facts about +# the boot rather than about the claim. The two `lan.` rows have none: netd +# never came up on any boot of it. + +[[number]] +name = "boot.lancase.complete_ms" +unit = "ms" +ceiling = 60000 +ceiling_from = "toyos_tco::JOB_BOUND_MS — as boot.testcases.complete_ms" +measured = 1258 + +[[number]] +name = "boot.lancase.back_secs" +unit = "s" +ceiling = 420 +ceiling_from = "toyos_build::metal::return_secs" +measured = 59 + +[[number]] +name = "boot.lancase.stick_secs" +unit = "s" +ceiling = 30 +ceiling_from = "as boot.testcases.stick_secs" +measured = 0 + +[[number]] +name = "boot.lancase.ping_secs" +unit = "s" +ceiling = 420 +ceiling_from = "toyos_build::metal::return_secs — the loop's own wait for the machine to answer ssh again, which bounds the window a reply can fall in and nothing narrower has been read. It is a cost and not a verdict: the window holds both of this machine's operating systems, so no ceiling on this number separates them, and what does is the wall-clock bracket in bootlog::host_second_inside_this_boot. A green reading tightens this to what a boot that answers actually costs" + +[[number]] +name = "list.lancase.job_ms" +unit = "ms" +ceiling = 22000 +ceiling_from = "lan_hold's own twenty-second window plus two seconds for the spawn and the exit record around it; the one member of this list is that window and nothing else" + +[[number]] +name = "lan.lancase.link_up_ms" +unit = "ms" +ceiling = 20000 +ceiling_from = "lan_hold's twenty-second window: a link that comes up later than that is a link this boot never had, because the machine has already handed itself back" + +[[number]] +name = "lan.lancase.lease_ms" +unit = "ms" +ceiling = 20000 +ceiling_from = "as lan.lancase.link_up_ms, and netd's own dhcp::LEASE_BOUND is the same twenty seconds: a boot with no lease by then has already said so in its log" diff --git a/tests/test-durations b/tests/test-durations index 5dd3b24c612..a339ae1d83b 100644 --- a/tests/test-durations +++ b/tests/test-durations @@ -254,6 +254,8 @@ keyboard_claim_close_spares_stdin 4580 shards=12 kill_while_blocked 44 shards=12 klogd_hosted 5674 shards=12 klogd_panic_halts 16658 shards=12 +lan_dhcp_lease 18446744073709551615 shards=none +lan_no_lease 18446744073709551615 shards=none lapic_spurious_vector 6829 shards=12 late_storage_connect 6455 shards=12 latency_wake 8790 shards=12 diff --git a/tests/toyos-rust-tests/src/bin/lan_hold.rs b/tests/toyos-rust-tests/src/bin/lan_hold.rs new file mode 100644 index 00000000000..6f783a37f64 --- /dev/null +++ b/tests/toyos-rust-tests/src/bin/lan_hold.rs @@ -0,0 +1,19 @@ +//! Hold the boot open for as long as the host needs to reach this machine over +//! the cable, and exit. +//! +//! It asserts nothing: what it is evidence *for* is judged on the host, out of +//! the records netd wrote inside this window and out of whether the host's own +//! `ping` was answered while it was open. + +use std::thread::sleep; +use std::time::Duration; + +/// How long this machine stays up for the host. `list.lancase.job_ms` in +/// `tests/metal-profile.toml` is this number plus what a spawn costs, and +/// moving one without the other is a job list the runner's own deadline cuts +/// short. +const HOLD: Duration = Duration::from_secs(20); + +fn main() { + sleep(HOLD); +} diff --git a/tests/toyos.rs b/tests/toyos.rs index 8a1a880e73d..e1b35f04913 100644 --- a/tests/toyos.rs +++ b/tests/toyos.rs @@ -12,8 +12,8 @@ use common::qemu::{ STALLED, }; use common::{ - audio, compile, devices, faults, hostload, metal, pkg, power, screen, serial, stats, storage, - usb, + audio, compile, devices, faults, hostload, lan, metal, pkg, power, screen, serial, stats, + storage, usb, }; use toyos_build::day::Day; use toyos_build::bootlog::{self, boot_millis}; @@ -231,6 +231,11 @@ const RUST_SKIP: &[&str] = &[ // Needs a NIC in front of netd; only `tests/netcase` has one. // `netd_listener_forgery` runs it there. "netd_listener_forgery", + // It asserts nothing at all: it holds a `tests/lancase` boot open for + // twenty seconds so the host can reach this machine over the cable, and + // `lan_dhcp_lease`'s metal arm is the only job list that names it. On a + // shared boot it would be twenty seconds of nothing. + "lan_hold", // Needs SYS_DEBUG, which the shipping kernel has no arm of at all. // `heap_ceiling_recovery` boots the `test-actuators` kernel on one CPU, // which is also what makes its claim about *the recovered CPU* precise. @@ -643,6 +648,18 @@ const MACHINE_TESTS: &[(&str, Sched, Tier)] = &[ // is the buffers' business and no arm's to demand; what it may never cost // is half a line, and that is what this one judges. ("log_stream_stalled_peer_delivers_whole_records", Sched::Parallel, Tier::Nightly), + // netd taking this machine's address from the network instead of carrying + // one written down. The DHCP server it is judged against is QEMU's own, an + // implementation of RFC 2131 this repository did not write, and its lease + // is known field by field. The verdicts are records and a lease's fields; + // no clock in it. Fast with the UNMEASURED bootstrap marker until priced. + ("lan_dhcp_lease", Sched::Parallel, Tier::Fast), + // The same client on a wire with no server: it says it has no address and + // announces itself anyway. Its cost is netd's own twenty-second lease bound + // waited out in real time, so it is `Why::TimerAnchored` and belongs + // Nightly; a new name is bootstrapped Fast with the UNMEASURED marker + // because only the fast tier can replace one. + ("lan_no_lease", Sched::Parallel, Tier::Fast), ("netd_connection_caps", Sched::Parallel, Tier::Fast), // The netcase boot again: netd must not abort a listener on a ring flag its // own client forged. Its verdict is a kernel-reported EOF or its absence; @@ -1295,6 +1312,16 @@ const METAL: &[(&str, metal::Metal)] = &[ "metal_device_probe", metal::Metal::Runs { arms: METALDEVICECASE, judge: |b| devices::on_metal(b[0]) }, ), + ( + // The cable. Under QEMU this name judges netd's DHCP client against the + // user-mode backend's server; here it judges the whole path — the + // kernel handing netd the T14's own I219, the link, a lease from the + // bench's router, and the development host's `ping` answered at the + // leased address in the window where the machine is running nothing but + // this image. + "lan_dhcp_lease", + metal::Metal::Runs { arms: LANCASE, judge: |b| lan::on_metal(b[0]) }, + ), // ---- one image: tests/testcases, no parameters, one job list ---- ( "blackbox_unclaimed_page", @@ -1665,6 +1692,12 @@ const USB_RESET_BOOTS: &[metal::Arm] = &[ const METALCASE: &[metal::Arm] = &[metal::once("metalcase", "tests/metalcase", &[], &[])]; +/// The cable's own boot: netd in front of the T14's I219, and one job that +/// holds the machine up long enough for the host to reach it. The one arm in +/// this suite that names a PCI function for the loop to reach the boot over. +const LANCASE: &[metal::Arm] = + &[metal::Arm { nic: Some(lan::NIC), ..metal::once(lan::BOOT, lan::CONFIG, &[], lan::JOBS) }]; + /// One boot for every in-kernel self-test that logs its verdict at init and /// does nothing else. /// @@ -13604,6 +13637,8 @@ fn run_machine_test( ); Ok(()) } + "lan_dhcp_lease" => lan::lan_dhcp_lease(test_config, c_bins, rust_bins), + "lan_no_lease" => lan::lan_no_lease(test_config, c_bins, rust_bins), "https_tls13" => common::https::tls13_judge(rust_bins, common::https::VIRTIO), "https_tls13_e1000e" => common::https::tls13_judge(rust_bins, common::https::E1000E), "log_stream" => common::logstream::stream(common::logstream::VIRTIO, c_bins, rust_bins), diff --git a/toyos-pci/src/bridge.rs b/toyos-pci/src/bridge.rs new file mode 100644 index 00000000000..15c21841c28 --- /dev/null +++ b/toyos-pci/src/bridge.rs @@ -0,0 +1,152 @@ +//! A PCI-to-PCI bridge's forwarded memory windows (PCI-to-PCI Bridge +//! Architecture Specification §3.2.5.6-3.2.5.8). +//! +//! **What a bridge forwards, nothing above it may hand out.** An address inside +//! a bridge's window is routed to that bridge's secondary bus and answered by +//! whatever is on it — or by nothing, which reads as ones and is not +//! distinguishable from unrouted space. So a module placing a window below +//! 4 GiB has to know these ranges before it can call any address free, and they +//! are readable from config space alone: no interpreter, no table. +//! +//! The registers hold address bits 31:20 in their top twelve bits and hardwire +//! the rest, so every window is a whole number of megabytes and a *limit* names +//! the last byte rather than the first free one. A bridge forwarding nothing +//! writes a base above its limit, which is the encoding for "disabled" and the +//! one this decode has to get right: read literally it is a range that wraps. + +/// Byte offsets in a Type 1 header. +pub const MEMORY_BASE: u64 = 0x20; +pub const PREFETCH_BASE: u64 = 0x24; + +/// The Type 1 header, as `HEADER_TYPE` reports it with the multi-function bit +/// removed. +pub const HEADER_TYPE_BRIDGE: u8 = 1; + +/// The granularity both windows are expressed in: bits 31:20. +const GRANULE: u64 = 1 << 20; + +/// One forwarded range, `start..end`, `end` exclusive. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Window { + pub start: u64, + pub end: u64, +} + +/// The window a Memory Base/Limit pair describes, or `None` where the bridge +/// forwards nothing. +/// +/// `pair` is the dword at [`MEMORY_BASE`] or [`PREFETCH_BASE`]: base in the low +/// half, limit in the high half. Only the top twelve bits of each half are the +/// address; the low four are the type field for a prefetchable window and are +/// reserved for a non-prefetchable one, and neither is part of the range. +pub fn window(pair: u32) -> Option { + // The twelve bits at 15:4 of each half *are* address bits 31:20, so each + // half moves left by sixteen and not by the four its own field is offset by. + let base = u64::from(pair & 0xFFF0) << 16; + let limit = u64::from((pair >> 16) & 0xFFF0) << 16; + // A base above its limit is the encoding for a bridge that forwards + // nothing, and it is what firmware writes into a window it did not need — + // read as a range it would be `0x00100000..0x0`, which wraps. + if base > limit { + return None; + } + // The limit names the last megabyte, not the first free one. + Some(Window { start: base, end: limit + GRANULE }) +} + +/// Whether a prefetchable window's registers name a 64-bit range, in which case +/// the upper dwords at 0x28 and 0x2C carry the rest of it. +/// +/// Answered rather than decoded, because a module that hands out only 32-bit +/// space needs to know that a window it read the low half of may reach far +/// above what it can see — and treating that as a 32-bit range would call +/// addresses free that the bridge forwards. +pub fn prefetch_is_64_bit(pair: u32) -> bool { + pair & 0xF == 1 +} + +/// Byte offsets of the two dwords that carry the rest of a 64-bit prefetchable +/// window. +pub const PREFETCH_BASE_UPPER: u64 = 0x28; +pub const PREFETCH_LIMIT_UPPER: u64 = 0x2C; + +/// The part of a prefetchable window that lies below 4 GiB, or `None` where +/// none of it does. +/// +/// **A survey of the low space may not count a window that is not in it.** A +/// 64-bit prefetchable window whose upper base is set begins above 4 GiB +/// entirely, and reading its low half as a range would call a megabyte of the +/// low space forwarded that no bridge forwards. Where the upper base is zero +/// the low half *is* the low part of the range, whatever the upper limit adds +/// above it. +pub fn prefetch_below_4g(pair: u32, base_upper: u32) -> Option { + if prefetch_is_64_bit(pair) && base_upper != 0 { + return None; + } + window(pair) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// The registers hold bits 31:20 and hardwire the rest, so a window is + /// whole megabytes and its limit names the last one. + #[test] + fn the_limit_names_the_last_megabyte_and_not_the_first_free_one() { + // Limit in the high half, base in the low one. Base 0xbc200000 and + // limit 0xbc2fffff: one megabyte forwarded. + assert_eq!(window(0xbc20_bc20), Some(Window { start: 0xbc20_0000, end: 0xbc30_0000 })); + // Base 0xbc200000, limit 0xbc3fffff: two. + assert_eq!(window(0xbc30_bc20), Some(Window { start: 0xbc20_0000, end: 0xbc40_0000 })); + // And the address really is bits 31:20 — a half read as if its field + // offset were the shift would land a thousandth of the way up. + assert_eq!(window(0x0000_0000), Some(Window { start: 0, end: 0x0010_0000 })); + } + + /// **A bridge that forwards nothing must not read as a range.** Firmware + /// writes a base above the limit for a window it did not need, and taken + /// literally that is a range which wraps — and a free-space search over a + /// wrapped range calls the whole of memory forwarded, or none of it. + #[test] + fn a_disabled_window_is_no_window() { + // The canonical disabled encoding: base 0x00100000, limit 0x00000000. + assert_eq!(window(0x0000_0010), None); + // And the widest form of the same thing. + assert_eq!(window(0x0000_fff0), None); + // A bridge forwarding *everything* is the opposite and not the same + // reading: base 0, limit 0xfff00000. + assert_eq!(window(0xfff0_0000), Some(Window { start: 0, end: 0xfff0_0000 + GRANULE })); + // Equal halves are one megabyte and not nothing. + assert_eq!(window(0x0010_0010), Some(Window { start: 0x0010_0000, end: 0x0020_0000 })); + } + + /// The low four bits of each half are the type field, never the address. + /// Reading them as address bits moves a window by up to a megabyte. + #[test] + fn the_type_field_is_not_part_of_the_address() { + let plain = window(0xbc30_bc20).expect("a forwarded window"); + let typed = window(0xbc3f_bc21).expect("the same window, prefetchable and 64-bit"); + assert_eq!(plain, typed); + assert!(prefetch_is_64_bit(0xbc3f_bc21)); + assert!(!prefetch_is_64_bit(0xbc30_bc20)); + } + + /// A 64-bit prefetchable window that starts above 4 GiB is not a low range, + /// and its low half is not one either. + #[test] + fn a_prefetchable_window_above_four_gigabytes_is_not_low_space() { + assert_eq!(prefetch_below_4g(0xbc31_bc21, 0x60), None); + assert_eq!( + prefetch_below_4g(0xbc31_bc21, 0), + Some(Window { start: 0xbc20_0000, end: 0xbc40_0000 }) + ); + // A 32-bit window's upper dwords are hardwired to zero and say nothing; + // a machine that answers otherwise must not lose the window over it. + assert_eq!( + prefetch_below_4g(0xbc30_bc20, 0x60), + Some(Window { start: 0xbc20_0000, end: 0xbc40_0000 }) + ); + assert_eq!(prefetch_below_4g(0x0000_0010, 0), None); + } +} diff --git a/toyos-pci/src/lib.rs b/toyos-pci/src/lib.rs index c8527e3b1e6..bef6d418101 100644 --- a/toyos-pci/src/lib.rs +++ b/toyos-pci/src/lib.rs @@ -23,6 +23,7 @@ #![forbid(unsafe_code)] pub mod bar; +pub mod bridge; pub mod caps; pub mod express; pub mod msi; diff --git a/toyos-pci/src/msi.rs b/toyos-pci/src/msi.rs index db05f0ea909..23a2c3dde80 100644 --- a/toyos-pci/src/msi.rs +++ b/toyos-pci/src/msi.rs @@ -79,8 +79,28 @@ impl Msi { pub fn enabled(message_control: u16) -> u16 { (message_control & !MULTI_MESSAGE_ENABLE) | ENABLE } + + /// Message Control with the function delivering nothing, and everything it + /// said about itself left alone. + /// + /// **The counterpart of a hand-over.** A function whose holder is gone has + /// to stop writing its message somewhere, and MSI-X's per-entry mask lives + /// in a table that does not exist here: this bit is what stands in for it, + /// because a capability's per-vector mask is optional and [`Msi::mask`] is + /// `None` on every function that does not implement one. + pub fn disabled(message_control: u16) -> u16 { + message_control & !ENABLE + } } +/// The Mask Bits value that masks the one vector [`Msi::enabled`] leaves a +/// function armed for, and the one that unmasks it. +/// +/// Bit *n* is vector *n* (PCIe §7.7.1.7), and Multiple Message Enable is zero +/// after `enabled`, so the armed vector is always index 0. +pub const MASKED: u32 = 1 << 0; +pub const UNMASKED: u32 = 0; + #[cfg(test)] mod tests { use super::*; @@ -135,4 +155,27 @@ mod tests { let ctrl = ADDRESS_64 | PER_VECTOR_MASK | (5 << 1); assert_eq!(Msi::enabled(ctrl), ctrl | ENABLE); } + + /// **Disabling is not the inverse of enabling, and must not be.** What a + /// hand-over back owes is that the function stops delivering; what it does + /// not owe is the Multiple Message Enable field an arming zeroed, and a + /// `disabled` that restored it would leave a function armed for as many + /// vectors as it can raise the moment anything set the enable bit again. + #[test] + fn disabling_clears_the_enable_bit_and_nothing_else() { + let ctrl = ADDRESS_64 | PER_VECTOR_MASK | (5 << 1); + assert_eq!(Msi::disabled(ctrl | ENABLE), ctrl); + assert_eq!(Msi::disabled(ctrl), ctrl); + // Round trip, on the field that moves: arming zeroes Multiple Message + // Enable and disarming leaves it zeroed. + assert_eq!(Msi::disabled(Msi::enabled(ctrl | MULTI_MESSAGE_ENABLE)), ctrl); + } + + /// The armed vector is index 0, so the mask that silences it is bit 0. + #[test] + fn the_mask_names_the_one_vector_that_was_armed() { + assert_eq!(MASKED, 1); + assert_eq!(UNMASKED, 0); + assert_eq!(Msi::enabled(MULTI_MESSAGE_ENABLE) & MULTI_MESSAGE_ENABLE, 0); + } } diff --git a/userland/.cargo/config.toml b/userland/.cargo/config.toml index 2ed8cca7f42..459d67cdc82 100644 --- a/userland/.cargo/config.toml +++ b/userland/.cargo/config.toml @@ -3,4 +3,12 @@ channel = "toyos" [build] target = "x86_64-unknown-toyos" -rustflags = ["-Dwarnings"] \ No newline at end of file +rustflags = ["-Dwarnings"] +# smoltcp's build script reads these; its defaults are one-device numbers. +# `DNS_MAX_SERVER_COUNT` is what the resolver holds, and its default of 1 makes +# `dns::Socket::update_servers` drop the rest of a lease silently — so netd's +# own lease record would name resolvers the stack does not have. Three is what +# `smoltcp::wire::DHCP_MAX_DNS_SERVER_COUNT` lets a lease carry, and +# `userland/netd/src/dhcp.rs` asserts the two at compile time. +[env] +SMOLTCP_DNS_MAX_SERVER_COUNT = "3" diff --git a/userland/netd/Cargo.toml b/userland/netd/Cargo.toml index b2004833e75..96e1482fc3d 100644 --- a/userland/netd/Cargo.toml +++ b/userland/netd/Cargo.toml @@ -17,5 +17,6 @@ features = [ "socket-tcp", "socket-udp", "socket-dns", + "socket-dhcpv4", "alloc", ] diff --git a/userland/netd/src/dhcp.rs b/userland/netd/src/dhcp.rs new file mode 100644 index 00000000000..be0012e9b91 --- /dev/null +++ b/userland/netd/src/dhcp.rs @@ -0,0 +1,195 @@ +//! This machine's address, taken from the network rather than written down. +//! +//! **There is no static configuration to fall back to.** A machine's address +//! belongs to the network it is plugged into, and both networks this program +//! has ever run on — QEMU's user-mode backend and the bench's router — serve +//! DHCP. +//! +//! What the lease decides is the whole of the interface: the address and its +//! prefix, the default route, and the resolvers the DNS socket queries. All +//! three are written together on every lease and cleared together when one is +//! lost, because a route left standing over an address that is gone sends +//! frames out with a source nothing will answer. +//! +//! **A machine that gets no lease says so and goes on serving.** Its clients +//! then get their connects refused, one refusal at a time, which is what they +//! are already written to survive. + +use std::time::{Duration, Instant}; + +use smoltcp::config::DNS_MAX_SERVER_COUNT; +use smoltcp::iface::Interface; +use smoltcp::socket::{dhcpv4, dns}; +use smoltcp::wire::{ + DhcpOption, IpAddress, IpCidr, Ipv4Address, Ipv4Cidr, DHCP_MAX_DNS_SERVER_COUNT, +}; + +/// **The resolver holds every server a lease can carry.** +/// `dns::Socket::update_servers` truncates to `DNS_MAX_SERVER_COUNT` without +/// saying so, and smoltcp's default for it is one — so a lease offering three +/// would leave this machine's own record naming two resolvers it does not have. +/// The count is raised in `userland/.cargo/config.toml`, and this is what makes +/// a build that lowers it again fail to compile. +const _: () = assert!(DNS_MAX_SERVER_COUNT >= DHCP_MAX_DNS_SERVER_COUNT); + +/// RFC 2132 §3.14. +const OPT_HOST_NAME: u8 = 12; + +/// The name this machine asks its network to record for it. One name, because +/// there is one machine. +const HOSTNAME: &[u8] = b"toyos-t14"; + +/// The options every DISCOVER and REQUEST carries. +static OUTGOING: [DhcpOption<'static>; 1] = + [DhcpOption { kind: OPT_HOST_NAME, data: HOSTNAME }]; + +/// How long this machine waits for its first lease before saying it has none. +/// +/// It bounds the *report*, never the client: the socket goes on retrying for +/// the life of the boot, and a lease that lands after this is applied like any +/// other. What the bound buys is a line in the log on a machine whose network +/// never answers, instead of a boot that is silent about the one thing wrong +/// with it. +const LEASE_BOUND: Duration = Duration::from_secs(20); + +/// The DHCP client socket this machine runs, asking for a lease under +/// [`HOSTNAME`]. +pub fn socket() -> dhcpv4::Socket<'static> { + let mut socket = dhcpv4::Socket::new(); + socket.set_outgoing_options(&OUTGOING); + socket +} + +/// What the client decided, owned. +/// +/// **Taken out of the socket before anything is applied**, because the resolver +/// this lease writes lives in the same `SocketSet` as the client: an event +/// still borrowing the client is an event nothing can be done about. +pub enum Change { + Leased { address: Ipv4Cidr, router: Option, server: Ipv4Address, dns: Vec }, + Lost, +} + +impl Change { + /// Whatever the client has to say this pass. + pub fn of(client: &mut dhcpv4::Socket) -> Option { + match client.poll()? { + dhcpv4::Event::Configured(config) => Some(Self::Leased { + address: config.address, + router: config.router, + server: config.server.address, + dns: config.dns_servers.to_vec(), + }), + dhcpv4::Event::Deconfigured => Some(Self::Lost), + } + } +} + +/// The lease's own state, and what the boot's log still owes about it. +pub struct Dhcp { + began: Instant, + /// Whether the interface currently holds a lease. + leased: bool, + /// Whether this boot has settled the question once — a lease landed, or the + /// bound passed with none. netd announces itself on the edge of this. + settled: bool, +} + +impl Dhcp { + pub fn new() -> Self { + Self { began: Instant::now(), leased: false, settled: false } + } + + /// Apply what the client decided, and answer whether this machine's address + /// question has just been settled — which is the moment netd has something + /// to serve with. + pub fn pass( + &mut self, + change: Option, + iface: &mut Interface, + resolver: &mut dns::Socket, + ) -> bool { + match change { + Some(Change::Leased { address, router, server, dns }) => { + self.write(Some((address, router)), &dns, iface, resolver); + // **One record carrying every field the lease decided.** A boot + // read off a stick or a stream has this line and nothing else + // to say what this machine's network was. + crate::say!( + "netd: DHCP: lease {}/{} from {server}, gateway {}, dns [{}], {} ms after \ + netd came up", + address.address(), + address.prefix_len(), + match router { + Some(router) => router.to_string(), + None => "none".to_string(), + }, + dns.iter().map(ToString::to_string).collect::>().join(" "), + self.began.elapsed().as_millis(), + ); + self.leased = true; + } + Some(Change::Lost) => { + // Only worth a line where there was something to lose: the + // client reports this on its way to a first lease too. + if self.leased { + crate::say!("netd: DHCP: the lease is gone; this machine has no address"); + } + self.write(None, &[], iface, resolver); + self.leased = false; + } + None => {} + } + if self.settled { + return false; + } + if self.leased { + self.settled = true; + return true; + } + if self.began.elapsed() >= LEASE_BOUND { + crate::say!( + "netd: DHCP: no lease as {} in {} s; this machine has no address and every \ + connect through it is refused", + String::from_utf8_lossy(HOSTNAME), + self.began.elapsed().as_secs(), + ); + self.settled = true; + return true; + } + false + } + + /// The address, the default route and the resolvers, written together; + /// `None` writes the absence of all three. + /// + /// **One writer for both**, so the path that drops a lease is the path that + /// takes one: a clearing function of its own would be reached only by a + /// network that took an address away, which nothing in this tree can + /// arrange. + fn write( + &self, + lease: Option<(Ipv4Cidr, Option)>, + dns: &[Ipv4Address], + iface: &mut Interface, + resolver: &mut dns::Socket, + ) { + iface.update_ip_addrs(|addrs| { + // Cleared before the push, so a list already holding an address + // cannot leave the old one standing beside the new. + addrs.clear(); + if let Some((address, _)) = lease { + addrs.push(IpCidr::Ipv4(address)).expect("an emptied address list takes one"); + } + }); + iface.routes_mut().remove_default_ipv4_route(); + if let Some(router) = lease.and_then(|(_, router)| router) { + iface + .routes_mut() + .add_default_ipv4_route(router) + .expect("an emptied route table takes one default route"); + } + let servers: Vec = dns.iter().map(|s| IpAddress::Ipv4(*s)).collect(); + resolver.update_servers(&servers); + } +} diff --git a/userland/netd/src/main.rs b/userland/netd/src/main.rs index d7115e6af84..20c286fafc2 100644 --- a/userland/netd/src/main.rs +++ b/userland/netd/src/main.rs @@ -30,6 +30,7 @@ macro_rules! say { } mod device; +mod dhcp; mod i219; mod virtio_net; @@ -59,9 +60,9 @@ use toyos::net::*; use smoltcp::iface::{Config, Interface, PollResult, SocketHandle, SocketSet}; use smoltcp::phy::{self, Device, DeviceCapabilities, Medium}; -use smoltcp::socket::{dns, tcp, udp}; +use smoltcp::socket::{dhcpv4, dns, tcp, udp}; use smoltcp::time::Instant as SmoltcpInstant; -use smoltcp::wire::{DnsQueryType, EthernetAddress, HardwareAddress, IpAddress, IpCidr, IpEndpoint}; +use smoltcp::wire::{DnsQueryType, EthernetAddress, HardwareAddress, IpAddress, IpEndpoint}; use std::net::Ipv4Addr; @@ -1313,30 +1314,19 @@ fn main() { let now = SmoltcpInstant::from_millis(0); let mut iface = Interface::new(config, &mut device, now); - iface.update_ip_addrs(|addrs| { - addrs.push(IpCidr::new(IpAddress::v4(10, 0, 2, 15), 24)).ok(); - }); - iface.routes_mut() - .add_default_ipv4_route(Ipv4Addr::new(10, 0, 2, 2)) - .ok(); - let mut socket_set = SocketSet::new(vec![]); - let dns_servers = &[IpAddress::v4(10, 0, 2, 3)]; - let dns_socket = dns::Socket::new(dns_servers, vec![]); + // Empty, because the lease names the resolvers and nothing else may: a + // server written down here would answer for one network on every other. + let dns_socket = dns::Socket::new(&[], vec![]); let dns_handle = socket_set.add(dns_socket); + let dhcp_handle = socket_set.add(dhcp::socket()); + let mut dhcp = dhcp::Dhcp::new(); let total_mem = total_memory(); let max_piped = max_piped_connections(total_mem); let mut daemon = NetDaemon::new(dns_handle, max_piped); - say!( - "netd: ready, at most {max_piped} piped connections \ - ({} MiB each of {} MiB total)", - PIPED_CONNECTION_BYTES / (1024 * 1024), - total_mem / (1024 * 1024), - ); - // Sized for the slot ceiling rather than for `max_piped`: the batch // between two `wait` calls is the two fixed registrations, one per live piped // connection and one per pending connection, and the ceiling is what that @@ -1359,6 +1349,23 @@ fn main() { let now = SmoltcpInstant::from_millis(epoch.elapsed().as_millis() as i64); while iface.poll(now, &mut device, &mut socket_set) != PollResult::None {} + // **After the poll and before anything is served.** The lease is what + // gives this machine an address, a route and its resolvers, so a client + // answered before it was applied would be answered on a machine that is + // on no network. + let change = dhcp::Change::of(socket_set.get_mut::(dhcp_handle)); + if dhcp.pass(change, &mut iface, socket_set.get_mut::(dns_handle)) { + // Every arm that waits for netd waits for this line, so it is said + // once this machine has an address to serve on — or has been told + // it will not get one. + say!( + "netd: ready, at most {max_piped} piped connections \ + ({} MiB each of {} MiB total)", + PIPED_CONNECTION_BYTES / (1024 * 1024), + total_mem / (1024 * 1024), + ); + } + daemon.bridge_piped(&mut socket_set); daemon.check_piped_listeners(&mut socket_set);