fix(os): normalize the kernel setup header to QEMU's layout, not zeros

docs/security/security-model.md

@@ -326,12 +326,20 @@ kernel header for CoCo VMs", first released in 10.2.0) stopped rewriting the
 header for confidential guests, so the same kernel would otherwise measure
 differently depending on which QEMU the host chose to run.
 
-dstack removes that dependency instead of modelling it. The image build zeroes
-the boot-loader-written fields in the kernel it ships, and dstack's OVMF zeroes
-them again before the kernel blob is measured and loaded. RTMR[1] is therefore
-the plain Authenticode hash of the `bzImage` listed in `sha256sum.txt`, and the
-verifier needs nothing from the host to predict it -- not a QEMU version, not a
-memory size.
+dstack removes that dependency instead of modelling it. The image build fills
+in the boot-loader-owned fields of the kernel it ships with the values QEMU
+<= 10.1 would write, and dstack's OVMF writes the same values again before the
+kernel blob is measured and loaded. RTMR[1] is therefore the plain Authenticode
+hash of the `bzImage` listed in `sha256sum.txt`, and the verifier needs nothing
+from the host to predict it -- not a QEMU version, not a memory size.
+
+Normalizing to QEMU's layout rather than to zeros keeps an earlier release able
+to verify these images: `dstack-mr` computes that layout for an image that does
+not declare the flag, so a KMS that predates the declaration can still verify a
+guest booted on a QEMU that no longer patches the header, and a root-key
+handover does not have to bypass image verification. Recomputing it needs the
+guest RAM size, which limits such a verifier to guests with exactly 2 GiB or at
+least 2816 MiB.
 
 Images built before this landed keep their original behavior: their firmware
 does not normalize, so their digest still covers QEMU's rewritten copy. Which
@@ -350,11 +358,14 @@ here there is no knob. It also removes a class of correct-but-rejected
 deployments, since the previous QEMU-patched digest varied with guest RAM and
 was only reproducible at specific memory sizes.
 
-The normalized field set comes from the boot protocol rather than from QEMU's
-behavior: every field `Documentation/arch/x86/boot.rst` types as `write` is one
-the boot loader fills in and the kernel supplies no value for, so zeroing it
-discards nothing the kernel provided. Fields typed `modify` carry real
-kernel-supplied values and are left measured.
+What is written is one fixed header -- the one QEMU <= 10.1 in fact wrote for
+these kernels -- and not a reimplementation of QEMU's loader. Every field
+written is one QEMU fills in as boot loader; all but one are typed `write` in
+`Documentation/arch/x86/boot.rst`, which the boot loader supplies and the
+kernel has no value for. The exception is `loadflags`, typed
+`modify (obligatory)`, of which only the `CAN_USE_HEAP` bit is set, a bit the
+protocol assigns to the boot loader. Fields that carry real kernel-supplied
+values are left as the kernel built them.
 
 ### TCB status is surfaced, not gated, during verification
 

dstack/dstack-mr/src/kernel.rs

@@ -173,7 +173,10 @@ fn patch_kernel(
         let mut initrd_max = if protocol >= 0x20c {
             let xlf =
                 u16::from_le_bytes(kd[0x236..0x238].try_into().context("impossible failure")?);
-            if (xlf & 0x40) != 0 {
+            // XLF_CAN_BE_LOADED_ABOVE_4G is bit 1. Bit 6, 0x40, is
+            // XLF_5LEVEL_ENABLED; a kernel built without 5-level paging sets
+            // one and not the other, and QEMU tests bit 1.
+            if (xlf & 0x02) != 0 {
                 u32::MAX
             } else {
                 0x37ffffff
@@ -240,8 +243,8 @@ pub(crate) fn patched_kernel_authenticode_sha384(
 /// Compute the first RTMR[1] event digest for an image whose OVMF normalizes
 /// the Linux setup header: the Authenticode SHA-384 of the kernel file itself.
 ///
-/// Both sides zero the boot-loader-written fields --
-/// `os/image/normalize-kernel-header.py` in the image build and
+/// Both sides fill in the boot-loader-owned fields with the values QEMU <= 10.1
+/// writes -- `os/image/normalize-kernel-header.py` in the image build and
 /// `0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch` in the
 /// firmware -- so what OVMF measures is the file, on every QEMU version and at
 /// every guest memory size.
@@ -308,7 +311,7 @@ mod tests {
         kernel[0x206..0x208].copy_from_slice(&0x020cu16.to_le_bytes());
         // XLF_CAN_BE_LOADED_ABOVE_4G, so QEMU derives the initrd address from
         // available low memory and the patched digest moves with guest RAM.
-        kernel[0x236..0x238].copy_from_slice(&0x0040u16.to_le_bytes());
+        kernel[0x236..0x238].copy_from_slice(&0x0002u16.to_le_bytes());
         kernel[0x224..0x226].copy_from_slice(&0x50a0u16.to_le_bytes());
         kernel
     }
@@ -340,13 +343,37 @@ mod tests {
         assert_ne!(patched_at(0x8000_0000), patched_at(0xA000_0000));
     }
 
+    /// QEMU derives the initrd ceiling from XLF_CAN_BE_LOADED_ABOVE_4G,
+    /// which is bit 1; bit 6 is XLF_5LEVEL_ENABLED and must not be read as it.
+    #[test]
+    fn only_xlf_bit_1_raises_the_initrd_ceiling() {
+        let mut kernel = vec![0u8; 0x1000];
+        kernel[0x206..0x208].copy_from_slice(&0x020cu16.to_le_bytes());
+
+        // The initrd the 0.6.0 image ships, so these are the addresses QEMU
+        // puts it at and the image build writes into the shipped kernel.
+        let initrd_size = 6_454_798;
+        let with_xlf = |xlf: u16| {
+            let mut k = kernel.clone();
+            k[0x236..0x238].copy_from_slice(&xlf.to_le_bytes());
+            initrd_addr(&patch_kernel(&k, initrd_size, 0x8000_0000, 0x28000).unwrap())
+        };
+
+        // 0x37ffffff is the ceiling for a kernel that cannot be loaded above
+        // 4G; the raised ceiling is the below-4G window minus the ACPI area.
+        assert_eq!(with_xlf(0x0000), 0x379D_8000);
+        assert_eq!(with_xlf(0x0040), 0x379D_8000, "the 5-level flag raised it");
+        assert_eq!(with_xlf(0x0002), 0x7F9B_0000, "the above-4G flag did not");
+        assert_eq!(with_xlf(0x0042), 0x7F9B_0000);
+    }
+
     #[test]
     fn tdx_kernel_patch_uses_precomputed_digest_at_2g_and_high_memory() {
         let mut kernel = vec![0u8; 0x1000];
         // Linux boot protocol >= 2.12 with XLF_CAN_BE_LOADED_ABOVE_4G makes
         // QEMU derive the initrd address from available low memory.
         kernel[0x206..0x208].copy_from_slice(&0x020cu16.to_le_bytes());
-        kernel[0x236..0x238].copy_from_slice(&0x0040u16.to_le_bytes());
+        kernel[0x236..0x238].copy_from_slice(&0x0002u16.to_le_bytes());
 
         let below_2g = patch_kernel(&kernel, 0x100000, 0x80000000 - 0x1000, 0x28000).unwrap();
         let at_2g = patch_kernel(&kernel, 0x100000, 0x80000000, 0x28000).unwrap();

os/image/README.md

@@ -71,36 +71,61 @@ host's QEMU version — and the host is the one that declares that version.
 
 The fix has two halves that must stay in sync:
 
-- this script zeroes those fields in the kernel we ship;
-- `0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch` zeroes them
-  again in OVMF, before the kernel blob is measured and loaded.
+- this script fills in those fields in the kernel we ship, with the values
+  QEMU <= 10.1 would write;
+- `0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch` writes the
+  same values in OVMF, before the kernel blob is measured and loaded.
 
 The result is that RTMR[1] is the plain Authenticode hash of `bzImage` as
 listed in `sha256sum.txt`, on every QEMU version and at every guest memory
 size.
 
+Normalizing to a header QEMU once wrote, rather than to zeros, is what keeps an
+earlier release able to verify these images: `dstack-mr` already computes that
+header (`patch_kernel`) for an image that does not declare the flag, so a 0.5.x
+KMS can still onboard a 0.6.0 root with image verification on. That
+recomputation takes the guest RAM size as an input, so it reproduces the
+normalized bytes only for guests of exactly 2 GiB or at least 2816 MiB. Below
+2 GiB, and between 2 GiB and 2816 MiB, QEMU placed the initrd somewhere else,
+so such a verifier computes a different RTMR[1]; the no-image-download path
+refuses that same range for the same reason. The shipped initrd's size is an
+input too, since it decides `ramdisk_image`: QEMU packs the initrd against a
+ceiling, which is 4G for a kernel that sets `XLF_CAN_BE_LOADED_ABOVE_4G`
+(bit 1 of `xloadflags` — bit 6, `0x40`, is `XLF_5LEVEL_ENABLED`, a different
+flag) and `0x37ffffff` for one that does not, then clamped to the below-4G
+window QEMU leaves free of ACPI tables.
+
+The kernel must be loaded high, which every kernel this build ships is. QEMU
+puts a low-loaded kernel's command line at `0x9a000 - cmdline_size`, a value
+that depends on the command line the host passes, so there is nothing to
+normalize to: the script refuses such a kernel and OVMF leaves its header as
+served.
+
 `assemble.sh` records this in `metadata.json` as `"kernel_header_normalized":
-true`, and re-runs the script with `--check` first so the build fails rather
-than shipping a kernel that disagrees with what the image declares. Images
-without the field are the ones built before this existed; `dstack-mr` measures
-those the old way, against QEMU's rewritten header.
-
-The field set comes from the boot protocol, not from QEMU's behavior: every
-field `Documentation/arch/x86/boot.rst` types as `write` is one the boot loader
-fills in and the kernel supplies no value for. Fields typed `modify` carry real
-kernel-supplied values — `code32_start` is the protected-mode entry point — and
-are deliberately left alone.
-
-In practice this rewrites **two bytes**: `heap_end_ptr` (0x224) is the only
-`write` field a built kernel leaves non-zero. That is safe for every boot path:
-the boot protocol types it `write (obligatory)`, `init_heap()` reads it only
-when the boot loader has set `CAN_USE_HEAP` (which the kernel builds clear and
-this script also clears), and on the EFI-stub path the real-mode setup code
-never runs at all. The PE headers sit at 0x40..0x170, so no setup-header field
-overlaps them and the EFI entry point is untouched.
+true`. What makes that declaration true is the OVMF half, which the same build
+applies -- `ovmf-build.sh` and the bitbake recipe both fail if the patch does
+not apply, so an image cannot ship the flag with firmware that ignores it.
+Images without the field are the ones built before this existed; `dstack-mr`
+measures those the old way, against QEMU's rewritten header.
+
+What gets written is one fixed header, not a reimplementation of QEMU's loader:
+every field written is one QEMU fills in as boot loader, and the script refuses
+any kernel outside the shape it reproduces. All but one of those fields are
+typed `write` in `Documentation/arch/x86/boot.rst`, meaning the boot loader
+supplies them and the kernel has no value there. The exception is `loadflags`,
+typed `modify (obligatory)`, of which only the `CAN_USE_HEAP` bit is set — a
+bit the protocol assigns to the boot loader. Fields that carry real
+kernel-supplied values, such as `code32_start`, the protected-mode entry point,
+are deliberately left alone, and so are the `write` fields QEMU never touches.
+
+That is safe for every boot path: these are the values QEMU itself wrote for
+every release before 10.2, and on the EFI-stub path the real-mode setup code
+never runs at all -- the stub takes the initrd through LoadFile2 and the
+command line through its load options. The PE headers sit at 0x40..0x170, so no
+setup-header field overlaps them and the EFI entry point is untouched.
 
 To check an image without modifying it:
 
 ```bash
-./normalize-kernel-header.py --check /path/to/bzImage
+./normalize-kernel-header.py --check /path/to/bzImage /path/to/initramfs.cpio.gz
 ```

os/image/assemble.sh

@@ -420,15 +420,16 @@ verbose cp "$KERNEL_IMAGE" "${OUTPUT_DIR}/bzImage"
 # result into RTMR[1]. QEMU >= 10.2 stopped doing that for confidential guests,
 # so leaving the header as built would make the same image measure differently
 # per QEMU version. Normalizing here, and again in OVMF before it measures,
-# makes RTMR[1] the plain Authenticode hash of this file. Runs before
+# makes RTMR[1] the plain Authenticode hash of this file. The initrd size is an
+# input, so this runs once both files are in place, and before
 # tdx-measurement-cbor and sha256sum.txt below, so both cover the normalized
 # kernel. The matching half is in OVMF: metadata.json declares
 # kernel_header_normalized below, and what makes that declaration true is
 # 0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch, which this
 # same build applies -- ovmf-build.sh and the bitbake recipe both fail if it
 # does not apply. See os/image/README.md.
 verbose "$(dirname "${BASH_SOURCE[0]}")/normalize-kernel-header.py" \
-    "${OUTPUT_DIR}/bzImage"
+    "${OUTPUT_DIR}/bzImage" "${OUTPUT_DIR}/initramfs.cpio.gz"
 verbose cp "$OVMF_FIRMWARE" "${OUTPUT_DIR}/ovmf.fd"
 
 # AMD SEV firmware (additive). Shipped alongside the TDX firmware so a SEV-SNP