From 3f451c3c963deaf62630038f9a6b92b68cb725f3 Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Thu, 17 Sep 2026 23:59:34 -0700 Subject: [PATCH 1/4] fix(os): normalize the kernel setup header to QEMU's layout, not zeros Signed-off-by: Kevin Wang --- docs/security/security-model.md | 28 ++- os/image/README.md | 31 ++- os/image/assemble.sh | 5 +- os/image/normalize-kernel-header.py | 167 +++++++++---- os/tests/test-kernel-header-normalization.sh | 126 ++++++---- ...elLoaderFsDxe-normalize-setup-header.patch | 227 +++++++++++++----- 6 files changed, 419 insertions(+), 165 deletions(-) diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 751dbdc79..9381a8cbf 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -326,12 +326,20 @@ kernel header for CoCo VMs", first released in 10.2.0) stopped rewriting the header for confidential guests, so the same kernel would otherwise measure differently depending on which QEMU the host chose to run. -dstack removes that dependency instead of modelling it. The image build zeroes -the boot-loader-written fields in the kernel it ships, and dstack's OVMF zeroes -them again before the kernel blob is measured and loaded. RTMR[1] is therefore -the plain Authenticode hash of the `bzImage` listed in `sha256sum.txt`, and the -verifier needs nothing from the host to predict it -- not a QEMU version, not a -memory size. +dstack removes that dependency instead of modelling it. The image build writes +the boot-loader-written fields in the kernel it ships, with the values QEMU +<= 10.1 writes, and dstack's OVMF writes the same values again before the +kernel blob is measured and loaded. RTMR[1] is therefore the plain Authenticode +hash of the `bzImage` listed in `sha256sum.txt`, and the verifier needs nothing +from the host to predict it -- not a QEMU version, not a memory size. + +Normalizing to QEMU's layout rather than to zeros keeps an earlier release able +to verify these images: `dstack-mr` computes that layout for an image that does +not declare the flag, so a KMS that predates the declaration can still verify a +guest booted on a QEMU that no longer patches the header, and a root-key +handover does not have to bypass image verification. Recomputing it needs the +guest RAM size, which limits such a verifier to guests with exactly 2 GiB or at +least 2816 MiB. Images built before this landed keep their original behavior: their firmware does not normalize, so their digest still covers QEMU's rewritten copy. Which @@ -351,10 +359,10 @@ deployments, since the previous QEMU-patched digest varied with guest RAM and was only reproducible at specific memory sizes. The normalized field set comes from the boot protocol rather than from QEMU's -behavior: every field `Documentation/arch/x86/boot.rst` types as `write` is one -the boot loader fills in and the kernel supplies no value for, so zeroing it -discards nothing the kernel provided. Fields typed `modify` carry real -kernel-supplied values and are left measured. +behavior: every field written is one `Documentation/arch/x86/boot.rst` types as +`write`, which the boot loader fills in and the kernel supplies no value for. +Fields typed `modify` carry real kernel-supplied values and are left as the +kernel built them. ### TCB status is surfaced, not gated, during verification diff --git a/os/image/README.md b/os/image/README.md index 4084af001..723659e31 100644 --- a/os/image/README.md +++ b/os/image/README.md @@ -71,14 +71,25 @@ host's QEMU version — and the host is the one that declares that version. The fix has two halves that must stay in sync: -- this script zeroes those fields in the kernel we ship; -- `0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch` zeroes them - again in OVMF, before the kernel blob is measured and loaded. +- this script writes those fields in the kernel we ship, with the values + QEMU <= 10.1 writes; +- `0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch` writes the + same values in OVMF, before the kernel blob is measured and loaded. The result is that RTMR[1] is the plain Authenticode hash of `bzImage` as listed in `sha256sum.txt`, on every QEMU version and at every guest memory size. +Normalizing to QEMU's layout rather than to zeros is what keeps an earlier +release able to verify these images: `dstack-mr` already computes that layout +(`patch_kernel`) for an image that does not declare the flag, so a 0.5.x KMS +can still onboard a 0.6.0 root with image verification on. Recomputing it needs +the guest RAM size, which limits such a verifier to guests with exactly 2 GiB +or at least 2816 MiB -- the one range QEMU places the initrd differently in, +and the range the no-image-download path already excludes. The values are the +ones QEMU writes above that threshold, and the shipped initrd's size is an +input, since it decides `ramdisk_image`. + `assemble.sh` records this in `metadata.json` as `"kernel_header_normalized": true`, and re-runs the script with `--check` first so the build fails rather than shipping a kernel that disagrees with what the image declares. Images @@ -91,16 +102,14 @@ fills in and the kernel supplies no value for. Fields typed `modify` carry real kernel-supplied values — `code32_start` is the protected-mode entry point — and are deliberately left alone. -In practice this rewrites **two bytes**: `heap_end_ptr` (0x224) is the only -`write` field a built kernel leaves non-zero. That is safe for every boot path: -the boot protocol types it `write (obligatory)`, `init_heap()` reads it only -when the boot loader has set `CAN_USE_HEAP` (which the kernel builds clear and -this script also clears), and on the EFI-stub path the real-mode setup code -never runs at all. The PE headers sit at 0x40..0x170, so no setup-header field -overlaps them and the EFI entry point is untouched. +That is safe for every boot path: these are the values QEMU itself wrote for +every release before 10.2, and on the EFI-stub path the real-mode setup code +never runs at all -- the stub takes the initrd through LoadFile2 and the +command line through its load options. The PE headers sit at 0x40..0x170, so no +setup-header field overlaps them and the EFI entry point is untouched. To check an image without modifying it: ```bash -./normalize-kernel-header.py --check /path/to/bzImage +./normalize-kernel-header.py --check /path/to/bzImage /path/to/initramfs.cpio.gz ``` diff --git a/os/image/assemble.sh b/os/image/assemble.sh index 554e54b0c..aa7e11b34 100755 --- a/os/image/assemble.sh +++ b/os/image/assemble.sh @@ -420,7 +420,8 @@ verbose cp "$KERNEL_IMAGE" "${OUTPUT_DIR}/bzImage" # result into RTMR[1]. QEMU >= 10.2 stopped doing that for confidential guests, # so leaving the header as built would make the same image measure differently # per QEMU version. Normalizing here, and again in OVMF before it measures, -# makes RTMR[1] the plain Authenticode hash of this file. Runs before +# makes RTMR[1] the plain Authenticode hash of this file. The initrd size is an +# input, so this runs once both files are in place, and before # tdx-measurement-cbor and sha256sum.txt below, so both cover the normalized # kernel. The matching half is in OVMF: metadata.json declares # kernel_header_normalized below, and what makes that declaration true is @@ -428,7 +429,7 @@ verbose cp "$KERNEL_IMAGE" "${OUTPUT_DIR}/bzImage" # same build applies -- ovmf-build.sh and the bitbake recipe both fail if it # does not apply. See os/image/README.md. verbose "$(dirname "${BASH_SOURCE[0]}")/normalize-kernel-header.py" \ - "${OUTPUT_DIR}/bzImage" + "${OUTPUT_DIR}/bzImage" "${OUTPUT_DIR}/initramfs.cpio.gz" verbose cp "$OVMF_FIRMWARE" "${OUTPUT_DIR}/ovmf.fd" # AMD SEV firmware (additive). Shipped alongside the TDX firmware so a SEV-SNP diff --git a/os/image/normalize-kernel-header.py b/os/image/normalize-kernel-header.py index d628e5f4f..b2682f61a 100755 --- a/os/image/normalize-kernel-header.py +++ b/os/image/normalize-kernel-header.py @@ -9,39 +9,60 @@ whether QEMU rewrote the header -- which changed in QEMU 10.2 (commit a7542a38f399, "x86/loader: Don't update kernel header for CoCo VMs"). -Zeroing those fields in the shipped kernel, and having OVMF zero them again +Writing those fields in the shipped kernel, and having OVMF write them again before measuring, makes RTMR[1] the plain Authenticode hash of the file we ship, on every QEMU version. -The field set comes from the boot protocol, not from QEMU: every field -`Documentation/arch/x86/boot.rst` types as `write` is one the boot loader -fills in and the kernel supplies no value for. Fields typed `modify` carry -real kernel-supplied values (`code32_start` is the protected-mode entry point) -and are left alone. - -In a freshly built kernel every one of these fields is already zero except -`heap_end_ptr`, so this normally rewrites exactly two bytes. +The values are the ones QEMU <= 10.1 writes, for a guest with 2 GiB or more of +RAM below 4G -- the only guest memory layout modelled, and the one `dstack-mr` +computes (`patch_kernel` in dstack/dstack-mr/src/kernel.rs) for an image that +does not declare `kernel_header_normalized`. Normalizing to QEMU's layout +rather than to zeros is what keeps a verifier or KMS from an earlier release +able to measure these images. + +The field set comes from the boot protocol, not from QEMU: every field written +is one `Documentation/arch/x86/boot.rst` types as `write`, which the boot +loader fills in and the kernel supplies no value for. Fields typed `modify` +carry real kernel-supplied values (`code32_start` is the protected-mode entry +point) and are left alone. Fields QEMU leaves alone are left alone too: a built +kernel ships them zero, so writing them would only add a way to disagree. + +The initrd size decides `ramdisk_image`, so the initrd the image ships is an +input. """ import argparse import sys -# Offset, size, name -- every boot.rst field typed `write`. -WRITE_FIELDS = [ - (0x210, 1, "type_of_loader"), - (0x218, 4, "ramdisk_image"), - (0x21C, 4, "ramdisk_size"), - (0x224, 2, "heap_end_ptr"), - (0x226, 1, "ext_loader_ver"), - (0x227, 1, "ext_loader_type"), - (0x228, 4, "cmd_line_ptr"), - (0x23C, 4, "hardware_subarch"), - (0x240, 8, "hardware_subarch_data"), - (0x250, 8, "setup_data"), -] +# Offsets of the boot.rst `write` fields QEMU fills in. +TYPE_OF_LOADER_OFFSET = 0x210 +RAMDISK_IMAGE_OFFSET = 0x218 +RAMDISK_SIZE_OFFSET = 0x21C +HEAP_END_PTR_OFFSET = 0x224 +CMD_LINE_PTR_OFFSET = 0x228 +INITRD_ADDR_MAX_OFFSET = 0x22C +XLOADFLAGS_OFFSET = 0x236 LOADFLAGS_OFFSET = 0x211 +LOADED_HIGH = 0x01 CAN_USE_HEAP = 0x80 +XLF_CAN_BE_LOADED_ABOVE_4G = 0x40 + +# What QEMU writes: "Qemu" version 0, the real-mode block and command line it +# loads a kernel at, the 0x200-byte gap it leaves below the command line for +# the setup heap, the below-4G window it reserves for ACPI tables with RAM +# split at 2 GiB, the initrd ceiling a kernel that declares none gets, and the +# alignment it rounds the initrd address down to. +TYPE_OF_LOADER_QEMU = 0xB0 +REAL_ADDR_HIGH = 0x10000 +CMDLINE_ADDR_HIGH = 0x20000 +REAL_ADDR_LOW = 0x90000 +CMDLINE_ADDR_LOW = 0x9A000 +SETUP_HEAP_GAP = 0x200 +LOW_MEMORY_SPLIT = 0x80000000 +ACPI_DATA_SIZE = 0x28000 +INITRD_ADDR_MAX_DEFAULT = 0x37FFFFFF +INITRD_ALIGNMENT = 0x1000 HEADER_MAGIC_OFFSET = 0x202 HEADER_MAGIC = b"HdrS" @@ -50,12 +71,15 @@ # header, and slicing past the end would silently grow the image instead of # failing. The OVMF side applies the same bound. HEADER_END_OFFSET = 0x258 -# `setup_data` (0x250) requires 2.09+; every field above exists by then. -MIN_PROTOCOL = 0x0209 +# `cmd_line_ptr` moved into the setup header in 2.02, `initrd_addr_max` +# arrived in 2.03 and `xloadflags` in 2.12. +MIN_PROTOCOL = 0x0202 +IAM_PROTOCOL = 0x0203 +XLF_PROTOCOL = 0x020C -def normalize(image: bytearray) -> list: - """Zero the boot-loader-written fields. Returns the fields it changed.""" +def target_fields(image: bytes, initrd_size: int) -> list: + """Return the (offset, bytes, name) QEMU writes for this kernel.""" if len(image) < HEADER_END_OFFSET: raise ValueError( f"image is {len(image)} bytes, shorter than the " @@ -71,25 +95,73 @@ def normalize(image: bytearray) -> list: "script normalizes is not guaranteed" ) - changed = [] - for offset, size, name in WRITE_FIELDS: - old = bytes(image[offset : offset + size]) - if old != bytes(size): - changed.append((name, offset, old.hex(), "0" * (size * 2))) - image[offset : offset + size] = bytes(size) - - loadflags = image[LOADFLAGS_OFFSET] - if loadflags & CAN_USE_HEAP: - changed.append( - ( - "loadflags", - LOADFLAGS_OFFSET, - f"{loadflags:02x}", - f"{loadflags & ~CAN_USE_HEAP:02x}", - ) + if image[LOADFLAGS_OFFSET] & LOADED_HIGH: + real_addr, cmdline_addr = REAL_ADDR_HIGH, CMDLINE_ADDR_HIGH + else: + real_addr, cmdline_addr = REAL_ADDR_LOW, CMDLINE_ADDR_LOW + + fields = [ + (TYPE_OF_LOADER_OFFSET, bytes([TYPE_OF_LOADER_QEMU]), "type_of_loader"), + ( + LOADFLAGS_OFFSET, + bytes([image[LOADFLAGS_OFFSET] | CAN_USE_HEAP]), + "loadflags", + ), + # heap_end_ptr is two bytes, but the value never exceeds 16 bits and + # the two bytes above it (ext_loader_ver, ext_loader_type) are zero in + # a built kernel and untouched by QEMU, so writing the word + # zero-extended to 32 bits is what OVMF and dstack-mr also write. + ( + HEAP_END_PTR_OFFSET, + (cmdline_addr - real_addr - SETUP_HEAP_GAP).to_bytes(4, "little"), + "heap_end_ptr", + ), + (CMD_LINE_PTR_OFFSET, cmdline_addr.to_bytes(4, "little"), "cmd_line_ptr"), + ] + if initrd_size <= 0: + return fields + + if protocol >= XLF_PROTOCOL: + xloadflags = int.from_bytes( + image[XLOADFLAGS_OFFSET : XLOADFLAGS_OFFSET + 2], "little" + ) + initrd_max = ( + 0xFFFFFFFF + if xloadflags & XLF_CAN_BE_LOADED_ABOVE_4G + else INITRD_ADDR_MAX_DEFAULT + ) + elif protocol >= IAM_PROTOCOL: + declared = int.from_bytes( + image[INITRD_ADDR_MAX_OFFSET : INITRD_ADDR_MAX_OFFSET + 4], "little" + ) + initrd_max = declared or INITRD_ADDR_MAX_DEFAULT + else: + initrd_max = INITRD_ADDR_MAX_DEFAULT + available = LOW_MEMORY_SPLIT - ACPI_DATA_SIZE + if initrd_max >= available: + initrd_max = available - 1 + if initrd_size >= initrd_max: + raise ValueError( + f"an initrd of {initrd_size} bytes does not fit below 0x{initrd_max:x}" ) - image[LOADFLAGS_OFFSET] = loadflags & ~CAN_USE_HEAP + initrd_addr = (initrd_max - initrd_size) & ~(INITRD_ALIGNMENT - 1) + fields.append( + (RAMDISK_IMAGE_OFFSET, initrd_addr.to_bytes(4, "little"), "ramdisk_image") + ) + fields.append( + (RAMDISK_SIZE_OFFSET, initrd_size.to_bytes(4, "little"), "ramdisk_size") + ) + return fields + +def normalize(image: bytearray, initrd_size: int) -> list: + """Write the boot-loader-written fields. Returns the fields it changed.""" + changed = [] + for offset, value, name in target_fields(bytes(image), initrd_size): + old = bytes(image[offset : offset + len(value)]) + if old != value: + changed.append((name, offset, old.hex(), value.hex())) + image[offset : offset + len(value)] = value return changed @@ -97,6 +169,10 @@ def main() -> int: """Normalize the image named on the command line, or check it in place.""" parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("bzimage", help="kernel image to normalize in place") + parser.add_argument( + "initrd", + help="initrd the image ships, whose size decides the initrd address", + ) parser.add_argument( "--check", action="store_true", @@ -106,8 +182,11 @@ def main() -> int: with open(args.bzimage, "rb") as f: image = bytearray(f.read()) + with open(args.initrd, "rb") as f: + f.seek(0, 2) + initrd_size = f.tell() - changed = normalize(image) + changed = normalize(image, initrd_size) for name, offset, old, new in changed: print(f"{args.bzimage}: {name} (0x{offset:03x}) {old} -> {new}") diff --git a/os/tests/test-kernel-header-normalization.sh b/os/tests/test-kernel-header-normalization.sh index 73bb9e710..24e71aabd 100755 --- a/os/tests/test-kernel-header-normalization.sh +++ b/os/tests/test-kernel-header-normalization.sh @@ -11,9 +11,11 @@ # 0007-OvmfPkg-...-normalize-setup-header.patch (firmware) # # If they ever disagree, every CVM fails attestation with an RTMR[1] mismatch -# and nothing else points at why. So this test parses the field table out of -# both and compares them, then exercises the Python side against a synthetic -# bzImage. +# and nothing else points at why. So this test checks that both write the same +# fields with the same constants, then exercises the Python side against a +# synthetic bzImage. dstack-mr is the third implementation -- it predicts this +# layout for an image that does not declare the flag -- so its constants are +# checked too. set -euo pipefail here=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd) @@ -24,45 +26,71 @@ patch=$root/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/000 [ -x "$script" ] || { echo "missing $script" >&2; exit 1; } [ -f "$patch" ] || { echo "missing $patch" >&2; exit 1; } -python3 - "$script" "$patch" <<'PYEOF' +kernel_rs=$root/dstack/dstack-mr/src/kernel.rs +[ -f "$kernel_rs" ] || { echo "missing $kernel_rs" >&2; exit 1; } + +python3 - "$script" "$patch" "$kernel_rs" <<'PYEOF' import importlib.util import re import sys -script_path, patch_path = sys.argv[1], sys.argv[2] +script_path, patch_path, kernel_rs_path = sys.argv[1:4] spec = importlib.util.spec_from_file_location("normalize", script_path) normalize = importlib.util.module_from_spec(spec) spec.loader.exec_module(normalize) -py_fields = sorted((offset, size) for offset, size, _ in normalize.WRITE_FIELDS) +patch_text = open(patch_path, encoding="utf-8").read() +kernel_rs = open(kernel_rs_path, encoding="utf-8").read() -# Added lines of the form " { 0x210, 1 }, // type_of_loader". -patch_text = open(patch_path, encoding="ascii").read() -ovmf_fields = sorted( - (int(offset, 16), int(size)) - for offset, size in re.findall( - r"^\+\s*\{\s*(0x[0-9A-Fa-f]+),\s*(\d+)\s*\},", patch_text, re.M - ) -) -assert ovmf_fields, "no field table found in the OVMF patch" -assert py_fields == ovmf_fields, ( - "normalization field tables disagree\n" - f" image build: {[(hex(o), s) for o, s in py_fields]}\n" - f" OVMF patch: {[(hex(o), s) for o, s in ovmf_fields]}" -) +def ovmf(name): + match = re.search(rf"^\+#define {name}\s+(0x[0-9A-Fa-f]+)", patch_text, re.M) + assert match, f"the OVMF patch no longer defines {name}" + return int(match.group(1), 16) -for token in ("LINUX_HDR_MIN_PROTOCOL 0x0209", "LINUX_LOADFLAGS_CAN_USE_HEAP 0x80"): - assert token in patch_text, f"OVMF patch no longer defines {token}" -assert normalize.MIN_PROTOCOL == 0x0209 -assert normalize.CAN_USE_HEAP == 0x80 -# `modify` fields carry kernel-supplied values and must never be normalized. -# code32_start is the protected-mode entry point; zeroing it bricks the kernel. -for forbidden in (0x1F2, 0x1FA, 0x211, 0x212, 0x214): - assert all(offset != forbidden for offset, _ in py_fields), ( - f"0x{forbidden:x} is a `modify` field and must not be zeroed" +# Both sides write these fields, and only these. +py_offsets = {0x210, 0x211, 0x218, 0x21C, 0x224, 0x228} +ovmf_offsets = { + ovmf(f"LINUX_HDR_{name}_OFFSET") + for name in ( + "TYPE_OF_LOADER", + "LOADFLAGS", + "RAMDISK_IMAGE", + "RAMDISK_SIZE", + "HEAP_END_PTR", + "CMD_LINE_PTR", + ) +} +assert py_offsets == ovmf_offsets, (py_offsets, ovmf_offsets) + +# Every value the layout depends on, in all three implementations. +for name, value, rust in ( + ("QEMU_LINUX_TYPE_OF_LOADER", normalize.TYPE_OF_LOADER_QEMU, "0xb0"), + ("QEMU_LINUX_REAL_ADDR_LOW", normalize.REAL_ADDR_LOW, "0x90000_u32"), + ("QEMU_LINUX_CMDLINE_ADDR_LOW", normalize.CMDLINE_ADDR_LOW, "0x9a000_u32"), + ("QEMU_LINUX_REAL_ADDR_HIGH", normalize.REAL_ADDR_HIGH, "0x10000_u32"), + ("QEMU_LINUX_CMDLINE_ADDR_HIGH", normalize.CMDLINE_ADDR_HIGH, "0x20000_u32"), + ("QEMU_LINUX_SETUP_HEAP_GAP", normalize.SETUP_HEAP_GAP, "0x200"), + ("QEMU_LOW_MEMORY_SPLIT", normalize.LOW_MEMORY_SPLIT, "0x80000000"), + ("QEMU_ACPI_DATA_SIZE", normalize.ACPI_DATA_SIZE, "0x28000"), + ("LINUX_INITRD_ADDR_MAX_DEFAULT", normalize.INITRD_ADDR_MAX_DEFAULT, "0x37ffffff"), + ("QEMU_INITRD_ALIGNMENT", normalize.INITRD_ALIGNMENT, "4095"), +): + assert ovmf(name) == value, f"{name} is {ovmf(name):#x} in the OVMF patch" + assert rust in kernel_rs, f"dstack-mr no longer spells {rust}" + +assert ovmf("LINUX_HDR_MIN_PROTOCOL") == normalize.MIN_PROTOCOL == 0x0202 +assert ovmf("LINUX_LOADFLAGS_CAN_USE_HEAP") == normalize.CAN_USE_HEAP == 0x80 +assert ovmf("LINUX_LOADFLAGS_LOADED_HIGH") == normalize.LOADED_HIGH == 0x01 + +# `modify` fields carry kernel-supplied values and must never be written. +# code32_start is the protected-mode entry point; overwriting it bricks the +# kernel. loadflags is only OR-ed with CAN_USE_HEAP, never replaced. +for forbidden in (0x1F2, 0x1FA, 0x212, 0x214): + assert forbidden not in py_offsets, ( + f"0x{forbidden:x} is a `modify` field and must not be written" ) # Synthetic bzImage: a plausible built kernel, then the same image with every @@ -78,30 +106,46 @@ def make_image(patched: bool) -> bytearray: image[0x214:0x218] = (0x100000).to_bytes(4, "little") # code32_start image[0x224:0x226] = (0x50A0).to_bytes(2, "little") # heap_end_ptr if patched: + # What QEMU writes for a 1 GiB guest, whose initrd sits lower than the + # layout normalized to. image[0x210] = 0xB0 image[0x211] |= 0x80 - image[0x218:0x21C] = (0xA97FC000).to_bytes(4, "little") - image[0x21C:0x220] = (0x0062A954).to_bytes(4, "little") + image[0x218:0x21C] = (0x37BF3000).to_bytes(4, "little") + image[0x21C:0x220] = (INITRD_SIZE).to_bytes(4, "little") image[0x224:0x226] = (0xFE00).to_bytes(2, "little") image[0x228:0x22C] = (0x20000).to_bytes(4, "little") return image + +INITRD_SIZE = 6_454_799 + built, patched = make_image(False), make_image(True) assert built != patched -normalize.normalize(built) -normalize.normalize(patched) +normalize.normalize(built, INITRD_SIZE) +normalize.normalize(patched, INITRD_SIZE) assert built == patched, "normalizing a QEMU-patched kernel must reproduce the shipped one" -# Idempotent, so OVMF re-running it over an already normalized kernel is a no-op. +# The layout QEMU writes, which is what dstack-mr predicts for an image that +# does not declare the flag. +assert built[0x210] == 0xB0 +assert built[0x211] == 0x81 +assert built[0x218:0x21C] == (0x379D8000).to_bytes(4, "little") +assert built[0x21C:0x220] == (INITRD_SIZE).to_bytes(4, "little") +assert built[0x224:0x228] == (0xFE00).to_bytes(4, "little") +assert built[0x228:0x22C] == (0x20000).to_bytes(4, "little") + +# Independent of guest RAM: an initrd address QEMU derived for another guest +# size normalizes to the same bytes, which is what makes the host's QEMU +# version and memory size irrelevant. again = bytearray(built) -assert normalize.normalize(again) == [] +assert normalize.normalize(again, INITRD_SIZE) == [] assert again == built -# The `modify` fields survived. +# The `modify` fields survived, and loadflags kept LOADED_HIGH. assert built[0x1F2:0x1F4] == (0x0001).to_bytes(2, "little") assert built[0x1FA:0x1FC] == (0xFFFF).to_bytes(2, "little") -assert built[0x211] == 0x01 +assert built[0x211] & 0x01 assert built[0x212:0x214] == (0x8000).to_bytes(2, "little") assert built[0x214:0x218] == (0x100000).to_bytes(4, "little") @@ -112,7 +156,7 @@ assert built[0x214:0x218] == (0x100000).to_bytes(4, "little") other = bytearray(0x1000) other[0x210] = 0xB0 try: - normalize.normalize(other) + normalize.normalize(other, INITRD_SIZE) except ValueError: pass else: @@ -125,13 +169,13 @@ assert "return;" in patch_text.split('CompareMem (Data + LINUX_HDR_MAGIC_OFFSET' # normalizes. The image build still fails loudly about it. ancient = bytearray(0x1000) ancient[0x202:0x206] = b"HdrS" -ancient[0x206:0x208] = (0x0208).to_bytes(2, "little") +ancient[0x206:0x208] = (0x0201).to_bytes(2, "little") try: - normalize.normalize(ancient) + normalize.normalize(ancient, INITRD_SIZE) except ValueError: pass else: - raise AssertionError("the image build must reject boot protocols below 2.09") + raise AssertionError("the image build must reject boot protocols below 2.02") print("kernel setup-header normalization: OK") PYEOF diff --git a/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch b/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch index 30f41be5b..2d054d08c 100644 --- a/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch +++ b/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch @@ -12,75 +12,121 @@ guests, so from 10.2 on the same kernel measures differently than it does on 10.1 and earlier. Attestation has to know the host's QEMU version to predict its own RTMR[1], and the host declares that version. -Zero the fields the boot protocol says a boot loader owns, which makes the -published bytes independent of what QEMU did. The dstack image build applies -the same normalization to the kernel it ships, so the measurement is the plain -Authenticode hash of that file on every QEMU version. +Write the fields the boot protocol says a boot loader owns, with the values +QEMU <= 10.1 writes, which makes the published bytes independent of what QEMU +did. The dstack image build writes the same values into the kernel it ships, so +the measurement is the plain Authenticode hash of that file on every QEMU +version. + +The values follow from the kernel file and the initrd size -- never from the +QEMU version, and never from guest RAM: the initrd address QEMU derives from +RAM is fixed at the value it resolves to for a guest with 2 GiB or more below +4G. Normalizing to QEMU's layout rather than to zeros is what keeps a verifier +or KMS from an earlier release able to measure these images: dstack-mr already +computes this layout (patch_kernel in dstack/dstack-mr/src/kernel.rs) for an +image that does not declare kernel_header_normalized. The field set is taken from Documentation/arch/x86/boot.rst rather than from QEMU's behavior: fields typed `write` are ones the boot loader fills in and the kernel supplies no value for. Fields typed `modify` are left alone because they -carry real kernel-supplied values. +carry real kernel-supplied values. Fields QEMU leaves alone are left alone too. Booting is unaffected. On the EFI-stub path the real-mode setup code never -runs, and heap_end_ptr -- the only one of these fields a built kernel leaves -non-zero -- is read only when the boot loader has set CAN_USE_HEAP, which this -also clears. +runs, the stub takes the initrd through LoadFile2 and the command line through +its load options, and these are the values QEMU itself wrote for every release +before 10.2. Signed-off-by: dstack --- --- a/OvmfPkg/QemuKernelLoaderFsDxe/QemuKernelLoaderFsDxe.c +++ b/OvmfPkg/QemuKernelLoaderFsDxe/QemuKernelLoaderFsDxe.c -@@ -49,6 +49,111 @@ +@@ -49,6 +49,215 @@ struct KERNEL_BLOB { KERNEL_BLOB *Next; }; +// -+// Offsets into the Linux x86 setup header, from ++// Linux x86 setup-header offsets and flags, from +// Documentation/arch/x86/boot.rst. +// -+#define LINUX_HDR_MAGIC_OFFSET 0x202 -+#define LINUX_HDR_VERSION_OFFSET 0x206 -+#define LINUX_HDR_LOADFLAGS_OFFSET 0x211 -+#define LINUX_HDR_END_OFFSET 0x258 ++#define LINUX_HDR_MAGIC_OFFSET 0x202 ++#define LINUX_HDR_VERSION_OFFSET 0x206 ++#define LINUX_HDR_TYPE_OF_LOADER_OFFSET 0x210 ++#define LINUX_HDR_LOADFLAGS_OFFSET 0x211 ++#define LINUX_HDR_RAMDISK_IMAGE_OFFSET 0x218 ++#define LINUX_HDR_RAMDISK_SIZE_OFFSET 0x21C ++#define LINUX_HDR_HEAP_END_PTR_OFFSET 0x224 ++#define LINUX_HDR_CMD_LINE_PTR_OFFSET 0x228 ++#define LINUX_HDR_INITRD_ADDR_MAX_OFFSET 0x22C ++#define LINUX_HDR_XLOADFLAGS_OFFSET 0x236 ++#define LINUX_HDR_END_OFFSET 0x258 ++#define LINUX_LOADFLAGS_LOADED_HIGH 0x01 ++#define LINUX_LOADFLAGS_CAN_USE_HEAP 0x80 ++#define LINUX_XLF_CAN_BE_LOADED_ABOVE_4G 0x40 + +// -+// CAN_USE_HEAP in loadflags. The kernel builds it clear; a boot loader sets it -+// when it has also filled in heap_end_ptr. ++// cmd_line_ptr moved into the setup header in boot protocol 2.02, ++// initrd_addr_max arrived in 2.03 and xloadflags in 2.12. +// -+#define LINUX_LOADFLAGS_CAN_USE_HEAP 0x80 ++#define LINUX_HDR_MIN_PROTOCOL 0x0202 ++#define LINUX_HDR_IAM_PROTOCOL 0x0203 ++#define LINUX_HDR_XLF_PROTOCOL 0x020C + +// -+// setup_data (0x250) arrived in boot protocol 2.09. Below that the field -+// layout normalized here is not guaranteed, so the image is left alone. ++// What QEMU writes as boot loader: "Qemu" version 0, the real-mode block and ++// command line it loads a kernel at, the 0x200-byte gap it leaves below the ++// command line for the setup heap, the below-4G window it reserves for ACPI ++// tables with RAM split at 2 GiB, the initrd ceiling a kernel that declares ++// none gets, and the alignment it rounds the initrd address down to. Same ++// values as os/image/normalize-kernel-header.py and dstack-mr's patch_kernel. +// -+#define LINUX_HDR_MIN_PROTOCOL 0x0209 ++#define QEMU_LINUX_TYPE_OF_LOADER 0xB0 ++#define QEMU_LINUX_REAL_ADDR_HIGH 0x10000 ++#define QEMU_LINUX_CMDLINE_ADDR_HIGH 0x20000 ++#define QEMU_LINUX_REAL_ADDR_LOW 0x90000 ++#define QEMU_LINUX_CMDLINE_ADDR_LOW 0x9A000 ++#define QEMU_LINUX_SETUP_HEAP_GAP 0x200 ++#define QEMU_LOW_MEMORY_SPLIT 0x80000000 ++#define QEMU_ACPI_DATA_SIZE 0x28000 ++#define LINUX_INITRD_ADDR_MAX_DEFAULT 0x37FFFFFF ++#define QEMU_INITRD_ALIGNMENT 0x1000 + -+// -+// Every setup-header field boot.rst types as `write`: the boot loader fills it -+// in and the kernel supplies no value, so zeroing it discards nothing. Fields -+// typed `modify` carry real kernel-supplied values -- code32_start (0x214) is -+// the protected-mode entry point -- and are deliberately absent. -+// -+STATIC CONST struct { -+ UINT16 Offset; -+ UINT16 Size; -+} mLinuxBootLoaderWrittenFields[] = { -+ { 0x210, 1 }, // type_of_loader -+ { 0x218, 4 }, // ramdisk_image -+ { 0x21C, 4 }, // ramdisk_size -+ { 0x224, 2 }, // heap_end_ptr -+ { 0x226, 1 }, // ext_loader_ver -+ { 0x227, 1 }, // ext_loader_type -+ { 0x228, 4 }, // cmd_line_ptr -+ { 0x23C, 4 }, // hardware_subarch -+ { 0x240, 8 }, // hardware_subarch_data -+ { 0x250, 8 }, // setup_data -+}; ++STATIC ++UINT32 ++LinuxHdrRead ( ++ IN CONST UINT8 *Data, ++ IN UINTN Offset, ++ IN UINTN Size ++ ) ++{ ++ UINT32 Value; ++ UINTN Index; ++ ++ Value = 0; ++ for (Index = 0; Index < Size; Index++) { ++ Value |= (UINT32)Data[Offset + Index] << (8 * Index); ++ } ++ ++ return Value; ++} ++ ++STATIC ++VOID ++LinuxHdrWrite32 ( ++ IN OUT UINT8 *Data, ++ IN UINTN Offset, ++ IN UINT32 Value ++ ) ++{ ++ UINTN Index; ++ ++ for (Index = 0; Index < 4; Index++) { ++ Data[Offset + Index] = (UINT8)(Value >> (8 * Index)); ++ } ++} + +/** -+ Clear the setup-header fields a boot loader owns, so the measured kernel does -+ not depend on what QEMU wrote there. ++ Normalize the setup-header fields a boot loader owns, so the measured kernel ++ does not depend on what QEMU wrote there. + + QEMU acts as the boot loader for -kernel and fills these fields in before + serving the image over fw_cfg, which changes the bytes this driver publishes @@ -88,9 +134,13 @@ Signed-off-by: dstack + doing that for confidential guests (commit a7542a38f399), so without this the + same kernel measures differently depending on the host's QEMU. + -+ Zeroing is idempotent and matches what the image build already writes into -+ the shipped kernel, so the measurement equals the Authenticode hash of the -+ file on disk on every QEMU version. ++ The values written are the ones QEMU <= 10.1 writes, for a guest with 2 GiB ++ or more of RAM below 4G: they follow from the kernel file and the initrd ++ size, never from the QEMU version or from guest RAM. The dstack image build ++ writes the same values into the kernel it ships, so the measurement is the ++ Authenticode hash of that file on every QEMU version, and dstack-mr computes ++ them when it predicts RTMR[1] for an image that does not declare ++ kernel_header_normalized. Writing them is idempotent. + + @param[in,out] Data The kernel blob, starting at the setup header. + @param[in] Size Size of the blob in bytes. @@ -102,8 +152,12 @@ Signed-off-by: dstack + IN UINT32 Size + ) +{ -+ UINTN Idx; -+ UINT16 Protocol; ++ UINT32 Protocol; ++ UINT32 RealAddr; ++ UINT32 CmdlineAddr; ++ UINT32 InitrdSize; ++ UINT32 InitrdMax; ++ UINT32 AvailableMem; + + if (Size < LINUX_HDR_END_OFFSET) { + return; @@ -117,8 +171,7 @@ Signed-off-by: dstack + return; + } + -+ Protocol = (UINT16)(Data[LINUX_HDR_VERSION_OFFSET] | -+ (Data[LINUX_HDR_VERSION_OFFSET + 1] << 8)); ++ Protocol = LinuxHdrRead (Data, LINUX_HDR_VERSION_OFFSET, 2); + if (Protocol < LINUX_HDR_MIN_PROTOCOL) { + DEBUG (( + DEBUG_WARN, @@ -130,20 +183,80 @@ Signed-off-by: dstack + return; + } + -+ for (Idx = 0; Idx < ARRAY_SIZE (mLinuxBootLoaderWrittenFields); Idx++) { -+ ZeroMem ( -+ Data + mLinuxBootLoaderWrittenFields[Idx].Offset, -+ mLinuxBootLoaderWrittenFields[Idx].Size -+ ); ++ if ((Data[LINUX_HDR_LOADFLAGS_OFFSET] & LINUX_LOADFLAGS_LOADED_HIGH) == 0) { ++ RealAddr = QEMU_LINUX_REAL_ADDR_LOW; ++ CmdlineAddr = QEMU_LINUX_CMDLINE_ADDR_LOW; ++ } else { ++ RealAddr = QEMU_LINUX_REAL_ADDR_HIGH; ++ CmdlineAddr = QEMU_LINUX_CMDLINE_ADDR_HIGH; ++ } ++ ++ Data[LINUX_HDR_TYPE_OF_LOADER_OFFSET] = QEMU_LINUX_TYPE_OF_LOADER; ++ Data[LINUX_HDR_LOADFLAGS_OFFSET] |= LINUX_LOADFLAGS_CAN_USE_HEAP; ++ // ++ // heap_end_ptr is two bytes, but the value never exceeds 16 bits and the two ++ // bytes above it (ext_loader_ver, ext_loader_type) are zero in a built kernel ++ // and untouched by QEMU, so writing the word zero-extended to 32 bits is what ++ // the image build and dstack-mr also write. ++ // ++ LinuxHdrWrite32 ( ++ Data, ++ LINUX_HDR_HEAP_END_PTR_OFFSET, ++ CmdlineAddr - RealAddr - QEMU_LINUX_SETUP_HEAP_GAP ++ ); ++ LinuxHdrWrite32 (Data, LINUX_HDR_CMD_LINE_PTR_OFFSET, CmdlineAddr); ++ ++ QemuFwCfgSelectItem (QemuFwCfgItemInitrdSize); ++ InitrdSize = QemuFwCfgRead32 (); ++ if (InitrdSize == 0) { ++ // ++ // No initrd: QEMU leaves ramdisk_image and ramdisk_size alone, and a built ++ // kernel ships them zero. ++ // ++ return; ++ } ++ ++ if (Protocol >= LINUX_HDR_XLF_PROTOCOL) { ++ InitrdMax = (LinuxHdrRead (Data, LINUX_HDR_XLOADFLAGS_OFFSET, 2) & ++ LINUX_XLF_CAN_BE_LOADED_ABOVE_4G) != 0 ? ++ MAX_UINT32 : LINUX_INITRD_ADDR_MAX_DEFAULT; ++ } else if (Protocol >= LINUX_HDR_IAM_PROTOCOL) { ++ InitrdMax = LinuxHdrRead (Data, LINUX_HDR_INITRD_ADDR_MAX_OFFSET, 4); ++ if (InitrdMax == 0) { ++ InitrdMax = LINUX_INITRD_ADDR_MAX_DEFAULT; ++ } ++ } else { ++ InitrdMax = LINUX_INITRD_ADDR_MAX_DEFAULT; ++ } ++ ++ AvailableMem = QEMU_LOW_MEMORY_SPLIT - QEMU_ACPI_DATA_SIZE; ++ if (InitrdMax >= AvailableMem) { ++ InitrdMax = AvailableMem - 1; ++ } ++ ++ if (InitrdSize >= InitrdMax) { ++ DEBUG (( ++ DEBUG_WARN, ++ "%a: an initrd of 0x%x bytes does not fit below 0x%x\n", ++ __func__, ++ InitrdSize, ++ InitrdMax ++ )); ++ return; + } + -+ Data[LINUX_HDR_LOADFLAGS_OFFSET] &= (UINT8)~LINUX_LOADFLAGS_CAN_USE_HEAP; ++ LinuxHdrWrite32 ( ++ Data, ++ LINUX_HDR_RAMDISK_IMAGE_OFFSET, ++ (InitrdMax - InitrdSize) & ~(UINT32)(QEMU_INITRD_ALIGNMENT - 1) ++ ); ++ LinuxHdrWrite32 (Data, LINUX_HDR_RAMDISK_SIZE_OFFSET, InitrdSize); +} + STATIC KERNEL_BLOB_ITEMS mKernelBlobItems[] = { { L"kernel", -@@ -1063,6 +1168,10 @@ +@@ -1063,6 +1272,10 @@ QemuKernelFetchBlob ( ChunkData += BlobItems->FwCfgItem[Idx].Size; } From 7727bd1cea8b601b48ee990f003c5a2b0919c26c Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Fri, 18 Sep 2026 06:44:16 -0700 Subject: [PATCH 2/4] docs(os): correct stale wording about how the setup header is normalized Signed-off-by: Kevin Wang --- docs/security/security-model.md | 6 +++--- dstack/dstack-mr/src/kernel.rs | 4 ++-- os/image/README.md | 10 ++++++---- 3 files changed, 11 insertions(+), 9 deletions(-) diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 9381a8cbf..596699fa2 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -326,9 +326,9 @@ kernel header for CoCo VMs", first released in 10.2.0) stopped rewriting the header for confidential guests, so the same kernel would otherwise measure differently depending on which QEMU the host chose to run. -dstack removes that dependency instead of modelling it. The image build writes -the boot-loader-written fields in the kernel it ships, with the values QEMU -<= 10.1 writes, and dstack's OVMF writes the same values again before the +dstack removes that dependency instead of modelling it. The image build fills +in the boot-loader-owned fields of the kernel it ships with the values QEMU +<= 10.1 would write, and dstack's OVMF writes the same values again before the kernel blob is measured and loaded. RTMR[1] is therefore the plain Authenticode hash of the `bzImage` listed in `sha256sum.txt`, and the verifier needs nothing from the host to predict it -- not a QEMU version, not a memory size. diff --git a/dstack/dstack-mr/src/kernel.rs b/dstack/dstack-mr/src/kernel.rs index 7a130321d..74d6d5e8d 100644 --- a/dstack/dstack-mr/src/kernel.rs +++ b/dstack/dstack-mr/src/kernel.rs @@ -240,8 +240,8 @@ pub(crate) fn patched_kernel_authenticode_sha384( /// Compute the first RTMR[1] event digest for an image whose OVMF normalizes /// the Linux setup header: the Authenticode SHA-384 of the kernel file itself. /// -/// Both sides zero the boot-loader-written fields -- -/// `os/image/normalize-kernel-header.py` in the image build and +/// Both sides fill in the boot-loader-owned fields with the values QEMU <= 10.1 +/// writes -- `os/image/normalize-kernel-header.py` in the image build and /// `0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch` in the /// firmware -- so what OVMF measures is the file, on every QEMU version and at /// every guest memory size. diff --git a/os/image/README.md b/os/image/README.md index 723659e31..b54033527 100644 --- a/os/image/README.md +++ b/os/image/README.md @@ -71,8 +71,8 @@ host's QEMU version — and the host is the one that declares that version. The fix has two halves that must stay in sync: -- this script writes those fields in the kernel we ship, with the values - QEMU <= 10.1 writes; +- this script fills in those fields in the kernel we ship, with the values + QEMU <= 10.1 would write; - `0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch` writes the same values in OVMF, before the kernel blob is measured and loaded. @@ -91,8 +91,10 @@ ones QEMU writes above that threshold, and the shipped initrd's size is an input, since it decides `ramdisk_image`. `assemble.sh` records this in `metadata.json` as `"kernel_header_normalized": -true`, and re-runs the script with `--check` first so the build fails rather -than shipping a kernel that disagrees with what the image declares. Images +true`. What makes that declaration true is the OVMF half, which the same build +applies -- `ovmf-build.sh` and the bitbake recipe both fail if the patch does +not apply, so an image cannot ship the flag with firmware that ignores it. +Images without the field are the ones built before this existed; `dstack-mr` measures those the old way, against QEMU's rewritten header. From b4197820896bb5b407942250d506c88e8517b7fc Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Fri, 18 Sep 2026 08:23:05 -0700 Subject: [PATCH 3/4] fix(os): read XLF_CAN_BE_LOADED_ABOVE_4G as bit 1, not 0x40 0x40 is XLF_5LEVEL_ENABLED. All three implementations of the setup-header layout tested it as XLF_CAN_BE_LOADED_ABOVE_4G, so a kernel built without 5-level paging would get the 0x37ffffff initrd ceiling where QEMU raises it to 4G, putting the initrd about 1.1 GiB from where QEMU puts it and breaking the RTMR[1] prediction an earlier release's verifier or KMS makes. Every kernel dstack has shipped sets both bits, so no released image changes. A kernel loaded low is now refused rather than normalized: QEMU puts its command line at 0x9a000 - cmdline_size, which depends on the command line the host passes, so there is no single value to pin. The firmware leaves such a header as served. The cross-check test no longer restates what the script writes. It derives the written fields and their widths from both implementations and compares them, matches dstack-mr's constants where it computes the layout rather than anywhere in the file, and covers the ceiling every shipped kernel actually takes -- which the previous vector, with xloadflags = 0, never reached. Signed-off-by: Kevin Wang --- dstack/dstack-mr/src/kernel.rs | 33 ++- os/image/README.md | 12 +- os/image/normalize-kernel-header.py | 34 +-- os/tests/test-kernel-header-normalization.sh | 208 +++++++++++++----- ...elLoaderFsDxe-normalize-setup-header.patch | 45 ++-- .../dstack-ovmf/dstack-ovmf_git.bb | 9 +- 6 files changed, 249 insertions(+), 92 deletions(-) diff --git a/dstack/dstack-mr/src/kernel.rs b/dstack/dstack-mr/src/kernel.rs index 74d6d5e8d..38977feee 100644 --- a/dstack/dstack-mr/src/kernel.rs +++ b/dstack/dstack-mr/src/kernel.rs @@ -173,7 +173,10 @@ fn patch_kernel( let mut initrd_max = if protocol >= 0x20c { let xlf = u16::from_le_bytes(kd[0x236..0x238].try_into().context("impossible failure")?); - if (xlf & 0x40) != 0 { + // XLF_CAN_BE_LOADED_ABOVE_4G is bit 1. Bit 6, 0x40, is + // XLF_5LEVEL_ENABLED; a kernel built without 5-level paging sets + // one and not the other, and QEMU tests bit 1. + if (xlf & 0x02) != 0 { u32::MAX } else { 0x37ffffff @@ -308,7 +311,7 @@ mod tests { kernel[0x206..0x208].copy_from_slice(&0x020cu16.to_le_bytes()); // XLF_CAN_BE_LOADED_ABOVE_4G, so QEMU derives the initrd address from // available low memory and the patched digest moves with guest RAM. - kernel[0x236..0x238].copy_from_slice(&0x0040u16.to_le_bytes()); + kernel[0x236..0x238].copy_from_slice(&0x0002u16.to_le_bytes()); kernel[0x224..0x226].copy_from_slice(&0x50a0u16.to_le_bytes()); kernel } @@ -340,13 +343,37 @@ mod tests { assert_ne!(patched_at(0x8000_0000), patched_at(0xA000_0000)); } + /// QEMU derives the initrd ceiling from XLF_CAN_BE_LOADED_ABOVE_4G, + /// which is bit 1; bit 6 is XLF_5LEVEL_ENABLED and must not be read as it. + #[test] + fn only_xlf_bit_1_raises_the_initrd_ceiling() { + let mut kernel = vec![0u8; 0x1000]; + kernel[0x206..0x208].copy_from_slice(&0x020cu16.to_le_bytes()); + + // The initrd the 0.6.0 image ships, so these are the addresses QEMU + // puts it at and the image build writes into the shipped kernel. + let initrd_size = 6_454_798; + let with_xlf = |xlf: u16| { + let mut k = kernel.clone(); + k[0x236..0x238].copy_from_slice(&xlf.to_le_bytes()); + initrd_addr(&patch_kernel(&k, initrd_size, 0x8000_0000, 0x28000).unwrap()) + }; + + // 0x37ffffff is the ceiling for a kernel that cannot be loaded above + // 4G; the raised ceiling is the below-4G window minus the ACPI area. + assert_eq!(with_xlf(0x0000), 0x379D_8000); + assert_eq!(with_xlf(0x0040), 0x379D_8000, "the 5-level flag raised it"); + assert_eq!(with_xlf(0x0002), 0x7F9B_0000, "the above-4G flag did not"); + assert_eq!(with_xlf(0x0042), 0x7F9B_0000); + } + #[test] fn tdx_kernel_patch_uses_precomputed_digest_at_2g_and_high_memory() { let mut kernel = vec![0u8; 0x1000]; // Linux boot protocol >= 2.12 with XLF_CAN_BE_LOADED_ABOVE_4G makes // QEMU derive the initrd address from available low memory. kernel[0x206..0x208].copy_from_slice(&0x020cu16.to_le_bytes()); - kernel[0x236..0x238].copy_from_slice(&0x0040u16.to_le_bytes()); + kernel[0x236..0x238].copy_from_slice(&0x0002u16.to_le_bytes()); let below_2g = patch_kernel(&kernel, 0x100000, 0x80000000 - 0x1000, 0x28000).unwrap(); let at_2g = patch_kernel(&kernel, 0x100000, 0x80000000, 0x28000).unwrap(); diff --git a/os/image/README.md b/os/image/README.md index b54033527..61123f834 100644 --- a/os/image/README.md +++ b/os/image/README.md @@ -88,7 +88,17 @@ the guest RAM size, which limits such a verifier to guests with exactly 2 GiB or at least 2816 MiB -- the one range QEMU places the initrd differently in, and the range the no-image-download path already excludes. The values are the ones QEMU writes above that threshold, and the shipped initrd's size is an -input, since it decides `ramdisk_image`. +input, since it decides `ramdisk_image`: QEMU packs the initrd against a +ceiling, which is 4G for a kernel that sets `XLF_CAN_BE_LOADED_ABOVE_4G` +(bit 1 of `xloadflags` — bit 6, `0x40`, is `XLF_5LEVEL_ENABLED`, a different +flag) and `0x37ffffff` for one that does not, then clamped to the below-4G +window QEMU leaves free of ACPI tables. + +The kernel must be loaded high, which every kernel this build ships is. QEMU +puts a low-loaded kernel's command line at `0x9a000 - cmdline_size`, a value +that depends on the command line the host passes, so there is nothing to +normalize to: the script refuses such a kernel and OVMF leaves its header as +served. `assemble.sh` records this in `metadata.json` as `"kernel_header_normalized": true`. What makes that declaration true is the OVMF half, which the same build diff --git a/os/image/normalize-kernel-header.py b/os/image/normalize-kernel-header.py index b2682f61a..90e0ea7e8 100755 --- a/os/image/normalize-kernel-header.py +++ b/os/image/normalize-kernel-header.py @@ -13,8 +13,8 @@ before measuring, makes RTMR[1] the plain Authenticode hash of the file we ship, on every QEMU version. -The values are the ones QEMU <= 10.1 writes, for a guest with 2 GiB or more of -RAM below 4G -- the only guest memory layout modelled, and the one `dstack-mr` +The values are the ones QEMU <= 10.1 writes, for a kernel loaded high in a +guest with 2 GiB or more of RAM below 4G -- the only guest memory layout modelled, and the one `dstack-mr` computes (`patch_kernel` in dstack/dstack-mr/src/kernel.rs) for an image that does not declare `kernel_header_normalized`. Normalizing to QEMU's layout rather than to zeros is what keeps a verifier or KMS from an earlier release @@ -46,18 +46,18 @@ LOADFLAGS_OFFSET = 0x211 LOADED_HIGH = 0x01 CAN_USE_HEAP = 0x80 -XLF_CAN_BE_LOADED_ABOVE_4G = 0x40 +# xloadflags bit 1. Bit 6, 0x40, is XLF_5LEVEL_ENABLED, not this flag. +XLF_CAN_BE_LOADED_ABOVE_4G = 0x02 # What QEMU writes: "Qemu" version 0, the real-mode block and command line it -# loads a kernel at, the 0x200-byte gap it leaves below the command line for +# loads a kernel loaded high at, the 0x200-byte gap it leaves below the command +# line for # the setup heap, the below-4G window it reserves for ACPI tables with RAM # split at 2 GiB, the initrd ceiling a kernel that declares none gets, and the # alignment it rounds the initrd address down to. TYPE_OF_LOADER_QEMU = 0xB0 -REAL_ADDR_HIGH = 0x10000 -CMDLINE_ADDR_HIGH = 0x20000 -REAL_ADDR_LOW = 0x90000 -CMDLINE_ADDR_LOW = 0x9A000 +REAL_ADDR = 0x10000 +CMDLINE_ADDR = 0x20000 SETUP_HEAP_GAP = 0x200 LOW_MEMORY_SPLIT = 0x80000000 ACPI_DATA_SIZE = 0x28000 @@ -95,10 +95,16 @@ def target_fields(image: bytes, initrd_size: int) -> list: "script normalizes is not guaranteed" ) - if image[LOADFLAGS_OFFSET] & LOADED_HIGH: - real_addr, cmdline_addr = REAL_ADDR_HIGH, CMDLINE_ADDR_HIGH - else: - real_addr, cmdline_addr = REAL_ADDR_LOW, CMDLINE_ADDR_LOW + if not image[LOADFLAGS_OFFSET] & LOADED_HIGH: + # For a kernel loaded low, QEMU's command line sits at + # 0x9a000 - cmdline_size, which depends on the command line the host + # passes: there is no single value to normalize to. Every kernel this + # build ships is loaded high, so refuse rather than write a value QEMU + # never writes. The OVMF side leaves such a header as served. + raise ValueError( + "the kernel is not loaded high; its cmd_line_ptr depends on the " + "command line and cannot be normalized" + ) fields = [ (TYPE_OF_LOADER_OFFSET, bytes([TYPE_OF_LOADER_QEMU]), "type_of_loader"), @@ -113,10 +119,10 @@ def target_fields(image: bytes, initrd_size: int) -> list: # zero-extended to 32 bits is what OVMF and dstack-mr also write. ( HEAP_END_PTR_OFFSET, - (cmdline_addr - real_addr - SETUP_HEAP_GAP).to_bytes(4, "little"), + (CMDLINE_ADDR - REAL_ADDR - SETUP_HEAP_GAP).to_bytes(4, "little"), "heap_end_ptr", ), - (CMD_LINE_PTR_OFFSET, cmdline_addr.to_bytes(4, "little"), "cmd_line_ptr"), + (CMD_LINE_PTR_OFFSET, CMDLINE_ADDR.to_bytes(4, "little"), "cmd_line_ptr"), ] if initrd_size <= 0: return fields diff --git a/os/tests/test-kernel-header-normalization.sh b/os/tests/test-kernel-header-normalization.sh index 24e71aabd..b5f959a6c 100755 --- a/os/tests/test-kernel-header-normalization.sh +++ b/os/tests/test-kernel-header-normalization.sh @@ -27,21 +27,31 @@ patch=$root/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/000 [ -f "$patch" ] || { echo "missing $patch" >&2; exit 1; } kernel_rs=$root/dstack/dstack-mr/src/kernel.rs +machine_rs=$root/dstack/dstack-mr/src/machine.rs [ -f "$kernel_rs" ] || { echo "missing $kernel_rs" >&2; exit 1; } +[ -f "$machine_rs" ] || { echo "missing $machine_rs" >&2; exit 1; } -python3 - "$script" "$patch" "$kernel_rs" <<'PYEOF' +python3 - "$script" "$patch" "$kernel_rs" "$machine_rs" <<'PYEOF' import importlib.util import re import sys -script_path, patch_path, kernel_rs_path = sys.argv[1:4] +script_path, patch_path, kernel_rs_path, machine_rs_path = sys.argv[1:5] + +# Importing the script must not leave a .pyc beside it: a cached one whose +# source changed within the same second, without changing size, is reused, and +# this test would then check bytecode that is no longer on disk. +sys.dont_write_bytecode = True spec = importlib.util.spec_from_file_location("normalize", script_path) normalize = importlib.util.module_from_spec(spec) spec.loader.exec_module(normalize) patch_text = open(patch_path, encoding="utf-8").read() -kernel_rs = open(kernel_rs_path, encoding="utf-8").read() +machine_rs = open(machine_rs_path, encoding="utf-8").read() +# Only what dstack-mr compiles into a release counts: a constant that survives +# in `#[cfg(test)]` alone would let the two sides drift while this test passes. +kernel_rs = open(kernel_rs_path, encoding="utf-8").read().split("mod tests {")[0] def ovmf(name): @@ -50,52 +60,12 @@ def ovmf(name): return int(match.group(1), 16) -# Both sides write these fields, and only these. -py_offsets = {0x210, 0x211, 0x218, 0x21C, 0x224, 0x228} -ovmf_offsets = { - ovmf(f"LINUX_HDR_{name}_OFFSET") - for name in ( - "TYPE_OF_LOADER", - "LOADFLAGS", - "RAMDISK_IMAGE", - "RAMDISK_SIZE", - "HEAP_END_PTR", - "CMD_LINE_PTR", - ) -} -assert py_offsets == ovmf_offsets, (py_offsets, ovmf_offsets) - -# Every value the layout depends on, in all three implementations. -for name, value, rust in ( - ("QEMU_LINUX_TYPE_OF_LOADER", normalize.TYPE_OF_LOADER_QEMU, "0xb0"), - ("QEMU_LINUX_REAL_ADDR_LOW", normalize.REAL_ADDR_LOW, "0x90000_u32"), - ("QEMU_LINUX_CMDLINE_ADDR_LOW", normalize.CMDLINE_ADDR_LOW, "0x9a000_u32"), - ("QEMU_LINUX_REAL_ADDR_HIGH", normalize.REAL_ADDR_HIGH, "0x10000_u32"), - ("QEMU_LINUX_CMDLINE_ADDR_HIGH", normalize.CMDLINE_ADDR_HIGH, "0x20000_u32"), - ("QEMU_LINUX_SETUP_HEAP_GAP", normalize.SETUP_HEAP_GAP, "0x200"), - ("QEMU_LOW_MEMORY_SPLIT", normalize.LOW_MEMORY_SPLIT, "0x80000000"), - ("QEMU_ACPI_DATA_SIZE", normalize.ACPI_DATA_SIZE, "0x28000"), - ("LINUX_INITRD_ADDR_MAX_DEFAULT", normalize.INITRD_ADDR_MAX_DEFAULT, "0x37ffffff"), - ("QEMU_INITRD_ALIGNMENT", normalize.INITRD_ALIGNMENT, "4095"), -): - assert ovmf(name) == value, f"{name} is {ovmf(name):#x} in the OVMF patch" - assert rust in kernel_rs, f"dstack-mr no longer spells {rust}" - -assert ovmf("LINUX_HDR_MIN_PROTOCOL") == normalize.MIN_PROTOCOL == 0x0202 -assert ovmf("LINUX_LOADFLAGS_CAN_USE_HEAP") == normalize.CAN_USE_HEAP == 0x80 -assert ovmf("LINUX_LOADFLAGS_LOADED_HIGH") == normalize.LOADED_HIGH == 0x01 +INITRD_SIZE = 6_454_798 -# `modify` fields carry kernel-supplied values and must never be written. -# code32_start is the protected-mode entry point; overwriting it bricks the -# kernel. loadflags is only OR-ed with CAN_USE_HEAP, never replaced. -for forbidden in (0x1F2, 0x1FA, 0x212, 0x214): - assert forbidden not in py_offsets, ( - f"0x{forbidden:x} is a `modify` field and must not be written" - ) # Synthetic bzImage: a plausible built kernel, then the same image with every # boot-loader-written field filled in the way QEMU fills them. -def make_image(patched: bool) -> bytearray: +def make_image(patched: bool, xloadflags: int = 0, initrd_addr: int = 0) -> bytearray: image = bytearray(0x1000) image[0x202:0x206] = b"HdrS" image[0x206:0x208] = (0x020F).to_bytes(2, "little") @@ -105,21 +75,121 @@ def make_image(patched: bool) -> bytearray: image[0x212:0x214] = (0x8000).to_bytes(2, "little") # setup_move_size image[0x214:0x218] = (0x100000).to_bytes(4, "little") # code32_start image[0x224:0x226] = (0x50A0).to_bytes(2, "little") # heap_end_ptr + image[0x236:0x238] = xloadflags.to_bytes(2, "little") if patched: - # What QEMU writes for a 1 GiB guest, whose initrd sits lower than the - # layout normalized to. + # What QEMU wrote for some other guest: the initrd address is the only + # field its memory size reaches, so it is the caller's to vary. image[0x210] = 0xB0 image[0x211] |= 0x80 - image[0x218:0x21C] = (0x37BF3000).to_bytes(4, "little") + image[0x218:0x21C] = initrd_addr.to_bytes(4, "little") image[0x21C:0x220] = (INITRD_SIZE).to_bytes(4, "little") image[0x224:0x226] = (0xFE00).to_bytes(2, "little") image[0x228:0x22C] = (0x20000).to_bytes(4, "little") return image -INITRD_SIZE = 6_454_799 +# Both sides must write the same fields at the same widths. Read the Python +# side out of the script instead of restating it here, so that a field added or +# widened on one side alone fails this test. +py_fields = { + (offset, len(value)) + for offset, value, _ in normalize.target_fields(bytes(make_image(False)), INITRD_SIZE) +} + + +def ovmf_written_fields(): + """The (offset, size) the OVMF patch assigns to, parsed from its code.""" + written = {} + depth = 0 + for line in patch_text.splitlines(): + if not line.startswith("+"): + continue + body = line[1:] + for match in re.finditer(r"Data\[LINUX_HDR_(\w+)_OFFSET\]\s*\|?=", body): + written[match.group(1)] = 1 + if depth: + for match in re.finditer(r"LINUX_HDR_(\w+)_OFFSET", body): + written[match.group(1)] = 4 + if "LinuxHdrWrite32 (" in body: + # A call reaching the closing `);` on its own line spans several. + after = body.split("LinuxHdrWrite32 (", 1)[1] + for match in re.finditer(r"LINUX_HDR_(\w+)_OFFSET", after): + written[match.group(1)] = 4 + depth = 0 if ");" in after else 1 + elif depth and ");" in body: + depth = 0 + return {(ovmf(f"LINUX_HDR_{name}_OFFSET"), size) for name, size in written.items()} + + +ovmf_fields = ovmf_written_fields() +assert py_fields == ovmf_fields, (sorted(py_fields), sorted(ovmf_fields)) +assert len(py_fields) == 6, sorted(py_fields) + +# Every value the layout depends on, in the firmware. +for name, value in ( + ("QEMU_LINUX_TYPE_OF_LOADER", normalize.TYPE_OF_LOADER_QEMU), + ("QEMU_LINUX_REAL_ADDR", normalize.REAL_ADDR), + ("QEMU_LINUX_CMDLINE_ADDR", normalize.CMDLINE_ADDR), + ("QEMU_LINUX_SETUP_HEAP_GAP", normalize.SETUP_HEAP_GAP), + ("QEMU_LOW_MEMORY_SPLIT", normalize.LOW_MEMORY_SPLIT), + ("QEMU_ACPI_DATA_SIZE", normalize.ACPI_DATA_SIZE), + ("LINUX_INITRD_ADDR_MAX_DEFAULT", normalize.INITRD_ADDR_MAX_DEFAULT), + ("QEMU_INITRD_ALIGNMENT", normalize.INITRD_ALIGNMENT), + ("LINUX_XLF_CAN_BE_LOADED_ABOVE_4G", normalize.XLF_CAN_BE_LOADED_ABOVE_4G), + ("LINUX_HDR_MIN_PROTOCOL", normalize.MIN_PROTOCOL), + ("LINUX_LOADFLAGS_CAN_USE_HEAP", normalize.CAN_USE_HEAP), + ("LINUX_LOADFLAGS_LOADED_HIGH", normalize.LOADED_HIGH), +): + assert ovmf(name) == value, f"{name} is {ovmf(name):#x} in the OVMF patch" + +assert normalize.MIN_PROTOCOL == 0x0202 +assert normalize.CAN_USE_HEAP == 0x80 +assert normalize.LOADED_HIGH == 0x01 + +# The same values in dstack-mr, matched where it computes the layout rather +# than anywhere in the file: it is the implementation an earlier release's +# verifier or KMS runs against these images. +for text, spelling, what in ( + (kernel_rs, f"kd[0x210] = {normalize.TYPE_OF_LOADER_QEMU:#x}", "type_of_loader"), + ( + kernel_rs, + f"({normalize.REAL_ADDR:#x}_u32, {normalize.CMDLINE_ADDR:#x}_u32)", + "the real-mode and command-line addresses", + ), + ( + kernel_rs, + f"saturating_sub({normalize.SETUP_HEAP_GAP:#x})", + "the setup-heap gap", + ), + ( + kernel_rs, + f"xlf & {normalize.XLF_CAN_BE_LOADED_ABOVE_4G:#04x}", + "XLF_CAN_BE_LOADED_ABOVE_4G", + ), + ( + kernel_rs, + f"{normalize.INITRD_ADDR_MAX_DEFAULT:#x}", + "the default initrd ceiling", + ), + (kernel_rs, f"& !{normalize.INITRD_ALIGNMENT - 1}", "the initrd alignment"), + ( + kernel_rs, + f"TDX_KERNEL_HASH_COMPAT_2G_MEMORY: u64 = {normalize.LOW_MEMORY_SPLIT:#x}", + "the 2 GiB split", + ), + (machine_rs, f"{normalize.ACPI_DATA_SIZE:#x}", "the ACPI window"), +): + assert spelling in text, f"dstack-mr no longer spells {what} as {spelling}" + +# `modify` fields carry kernel-supplied values and must never be written. +# code32_start is the protected-mode entry point; overwriting it bricks the +# kernel. loadflags is only OR-ed with CAN_USE_HEAP, never replaced. +for forbidden in (0x1F2, 0x1FA, 0x212, 0x214): + assert forbidden not in {offset for offset, _ in py_fields}, ( + f"0x{forbidden:x} is a `modify` field and must not be written" + ) -built, patched = make_image(False), make_image(True) +built, patched = make_image(False), make_image(True, initrd_addr=0x37BF3000) assert built != patched normalize.normalize(built, INITRD_SIZE) @@ -127,7 +197,8 @@ normalize.normalize(patched, INITRD_SIZE) assert built == patched, "normalizing a QEMU-patched kernel must reproduce the shipped one" # The layout QEMU writes, which is what dstack-mr predicts for an image that -# does not declare the flag. +# does not declare the flag. This kernel declares no xloadflags, so QEMU caps +# the initrd at the 0x37ffffff a kernel that cannot be loaded above 4G gets. assert built[0x210] == 0xB0 assert built[0x211] == 0x81 assert built[0x218:0x21C] == (0x379D8000).to_bytes(4, "little") @@ -135,6 +206,41 @@ assert built[0x21C:0x220] == (INITRD_SIZE).to_bytes(4, "little") assert built[0x224:0x228] == (0xFE00).to_bytes(4, "little") assert built[0x228:0x22C] == (0x20000).to_bytes(4, "little") +# The branch every kernel this build ships takes: XLF_CAN_BE_LOADED_ABOVE_4G +# lifts QEMU's ceiling to 4G, and what caps the initrd is then the below-4G +# window it reserves for ACPI tables with RAM split at 2 GiB. +above_4g = make_image(False, xloadflags=normalize.XLF_CAN_BE_LOADED_ABOVE_4G) +normalize.normalize(above_4g, INITRD_SIZE) +assert above_4g[0x218:0x21C] == (0x7F9B0000).to_bytes(4, "little"), ( + f"initrd address is {int.from_bytes(above_4g[0x218:0x21C], 'little'):#x}" +) +assert above_4g[0x21C:0x220] == (INITRD_SIZE).to_bytes(4, "little") + +# 0x40 is XLF_5LEVEL_ENABLED, not XLF_CAN_BE_LOADED_ABOVE_4G. Reading it as the +# latter -- which all three implementations once did -- moves the initrd about +# 1.1 GiB away from where QEMU puts it, for any kernel built without 5-level +# paging. +five_level = make_image(False, xloadflags=0x0040) +normalize.normalize(five_level, INITRD_SIZE) +assert five_level[0x218:0x21C] == built[0x218:0x21C], ( + "the 5-level flag must not raise the initrd ceiling" +) + +# A kernel loaded low takes QEMU's other command-line address, which depends on +# the command line the host passes: there is nothing to normalize to, and the +# build has to say so rather than write a value QEMU never writes. +low = make_image(False) +low[0x211] &= ~normalize.LOADED_HIGH +try: + normalize.normalize(low, INITRD_SIZE) +except ValueError: + pass +else: + raise AssertionError("the image build must reject a kernel not loaded high") +assert "kernel is not loaded high" in patch_text, ( + "the OVMF side must leave a low-loaded kernel as served" +) + # Independent of guest RAM: an initrd address QEMU derived for another guest # size normalizes to the same bytes, which is what makes the host's QEMU # version and memory size irrelevant. diff --git a/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch b/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch index 2d054d08c..525d5fea1 100644 --- a/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch +++ b/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch @@ -21,7 +21,8 @@ version. The values follow from the kernel file and the initrd size -- never from the QEMU version, and never from guest RAM: the initrd address QEMU derives from RAM is fixed at the value it resolves to for a guest with 2 GiB or more below -4G. Normalizing to QEMU's layout rather than to zeros is what keeps a verifier +4G. A kernel loaded low is left alone, because QEMU's command-line address on +that path depends on the command line the host passes. Normalizing to QEMU's layout rather than to zeros is what keeps a verifier or KMS from an earlier release able to measure these images: dstack-mr already computes this layout (patch_kernel in dstack/dstack-mr/src/kernel.rs) for an image that does not declare kernel_header_normalized. @@ -40,7 +41,7 @@ Signed-off-by: dstack --- --- a/OvmfPkg/QemuKernelLoaderFsDxe/QemuKernelLoaderFsDxe.c +++ b/OvmfPkg/QemuKernelLoaderFsDxe/QemuKernelLoaderFsDxe.c -@@ -49,6 +49,215 @@ struct KERNEL_BLOB { +@@ -49,6 +49,221 @@ struct KERNEL_BLOB { KERNEL_BLOB *Next; }; @@ -59,9 +60,12 @@ Signed-off-by: dstack +#define LINUX_HDR_INITRD_ADDR_MAX_OFFSET 0x22C +#define LINUX_HDR_XLOADFLAGS_OFFSET 0x236 +#define LINUX_HDR_END_OFFSET 0x258 ++// ++// xloadflags bit 1. Bit 6, 0x40, is XLF_5LEVEL_ENABLED, not this flag. ++// +#define LINUX_LOADFLAGS_LOADED_HIGH 0x01 +#define LINUX_LOADFLAGS_CAN_USE_HEAP 0x80 -+#define LINUX_XLF_CAN_BE_LOADED_ABOVE_4G 0x40 ++#define LINUX_XLF_CAN_BE_LOADED_ABOVE_4G 0x02 + +// +// cmd_line_ptr moved into the setup header in boot protocol 2.02, @@ -80,10 +84,8 @@ Signed-off-by: dstack +// values as os/image/normalize-kernel-header.py and dstack-mr's patch_kernel. +// +#define QEMU_LINUX_TYPE_OF_LOADER 0xB0 -+#define QEMU_LINUX_REAL_ADDR_HIGH 0x10000 -+#define QEMU_LINUX_CMDLINE_ADDR_HIGH 0x20000 -+#define QEMU_LINUX_REAL_ADDR_LOW 0x90000 -+#define QEMU_LINUX_CMDLINE_ADDR_LOW 0x9A000 ++#define QEMU_LINUX_REAL_ADDR 0x10000 ++#define QEMU_LINUX_CMDLINE_ADDR 0x20000 +#define QEMU_LINUX_SETUP_HEAP_GAP 0x200 +#define QEMU_LOW_MEMORY_SPLIT 0x80000000 +#define QEMU_ACPI_DATA_SIZE 0x28000 @@ -134,8 +136,8 @@ Signed-off-by: dstack + doing that for confidential guests (commit a7542a38f399), so without this the + same kernel measures differently depending on the host's QEMU. + -+ The values written are the ones QEMU <= 10.1 writes, for a guest with 2 GiB -+ or more of RAM below 4G: they follow from the kernel file and the initrd ++ The values written are the ones QEMU <= 10.1 writes, for a kernel loaded ++ high in a guest with 2 GiB or more of RAM below 4G: they follow from the kernel file and the initrd + size, never from the QEMU version or from guest RAM. The dstack image build + writes the same values into the kernel it ships, so the measurement is the + Authenticode hash of that file on every QEMU version, and dstack-mr computes @@ -153,8 +155,6 @@ Signed-off-by: dstack + ) +{ + UINT32 Protocol; -+ UINT32 RealAddr; -+ UINT32 CmdlineAddr; + UINT32 InitrdSize; + UINT32 InitrdMax; + UINT32 AvailableMem; @@ -184,11 +184,18 @@ Signed-off-by: dstack + } + + if ((Data[LINUX_HDR_LOADFLAGS_OFFSET] & LINUX_LOADFLAGS_LOADED_HIGH) == 0) { -+ RealAddr = QEMU_LINUX_REAL_ADDR_LOW; -+ CmdlineAddr = QEMU_LINUX_CMDLINE_ADDR_LOW; -+ } else { -+ RealAddr = QEMU_LINUX_REAL_ADDR_HIGH; -+ CmdlineAddr = QEMU_LINUX_CMDLINE_ADDR_HIGH; ++ // ++ // A kernel loaded low takes QEMU's other command-line address, ++ // 0x9a000 - cmdline_size, which depends on the command line the host ++ // passes. There is no single value to normalize to, so leave the header ++ // alone. The dstack image build refuses to ship such a kernel. ++ // ++ DEBUG (( ++ DEBUG_WARN, ++ "%a: kernel is not loaded high; setup header left as served\n", ++ __func__ ++ )); ++ return; + } + + Data[LINUX_HDR_TYPE_OF_LOADER_OFFSET] = QEMU_LINUX_TYPE_OF_LOADER; @@ -202,9 +209,9 @@ Signed-off-by: dstack + LinuxHdrWrite32 ( + Data, + LINUX_HDR_HEAP_END_PTR_OFFSET, -+ CmdlineAddr - RealAddr - QEMU_LINUX_SETUP_HEAP_GAP ++ QEMU_LINUX_CMDLINE_ADDR - QEMU_LINUX_REAL_ADDR - QEMU_LINUX_SETUP_HEAP_GAP + ); -+ LinuxHdrWrite32 (Data, LINUX_HDR_CMD_LINE_PTR_OFFSET, CmdlineAddr); ++ LinuxHdrWrite32 (Data, LINUX_HDR_CMD_LINE_PTR_OFFSET, QEMU_LINUX_CMDLINE_ADDR); + + QemuFwCfgSelectItem (QemuFwCfgItemInitrdSize); + InitrdSize = QemuFwCfgRead32 (); @@ -256,7 +263,7 @@ Signed-off-by: dstack STATIC KERNEL_BLOB_ITEMS mKernelBlobItems[] = { { L"kernel", -@@ -1063,6 +1272,10 @@ QemuKernelFetchBlob ( +@@ -1063,6 +1278,10 @@ QemuKernelFetchBlob ( ChunkData += BlobItems->FwCfgItem[Idx].Size; } diff --git a/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf_git.bb b/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf_git.bb index 0e5d89ed2..c946f87d0 100644 --- a/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf_git.bb +++ b/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf_git.bb @@ -89,10 +89,11 @@ DEPENDS = "nasm-native acpica-native ovmf-native util-linux-native" # modules / no sevsecret). The patch fails loud if a future edk2 bump changes # the AmdSev layout. # -# 0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch zeroes the -# setup-header fields QEMU writes as boot loader, so RTMR[1] no longer depends -# on the host's QEMU version. It pairs with os/image/normalize-kernel-header.py, -# which applies the same normalization to the shipped bzImage. +# 0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch writes the +# setup-header fields QEMU writes as boot loader, with the values QEMU <= 10.1 +# writes, so RTMR[1] no longer depends on the host's QEMU version. It pairs with +# os/image/normalize-kernel-header.py, which writes the same values into the +# shipped bzImage. OVMF_BUILD_SEV ??= "1" EDK_TOOLS_DIR="edk2_basetools" From aa4e3d05589edb0e77588c3dda4635467fd697ad Mon Sep 17 00:00:00 2001 From: Kevin Wang Date: Fri, 18 Sep 2026 22:33:27 -0700 Subject: [PATCH 4/4] docs(os): state that one fixed header is written, not QEMU's logic The rationale paragraph carried over from the zeroing version said the field set comes from the boot protocol rather than from QEMU's behavior, that every field written is typed `write` in Documentation/arch/x86/boot.rst, and that fields typed `modify` are left alone. Two of those are false: loadflags is typed `modify (obligatory)` and is written, and the set is now exactly what QEMU fills in, with the write-typed fields QEMU never touches deliberately dropped. A reader following the paragraph would either drop the CAN_USE_HEAP write or add hardware_subarch and setup_data back, desyncing the script from the firmware. It appeared verbatim in the script, the image README, the security model and the OVMF patch's commit message; all four now say what the code does, including that only the CAN_USE_HEAP bit of loadflags is set, because the boot protocol assigns that bit to the boot loader. They also now say what this is: one fixed header, chosen because dstack-mr already computes it, not a reimplementation of QEMU's loader. Zeros would serve the measurement goal equally well; reproducing a header QEMU once wrote is what lets an earlier release's verifier or KMS keep measuring these images. The README's guest-memory note had its appositives attached to the allowed range rather than its complement, so it read as naming the sizes to avoid. It now names both sides. The HEADER_END_OFFSET comment still justified 0x258 as one past the last field touched, which was true when setup_data was zeroed; nothing above 0x238 is read now, so the constant is stated as what it is, the end of the protocol 2.09 header kept as a minimum length. Also joins a paragraph broken mid-sentence in the README, rewraps a comment broken mid-phrase and a 106-character docstring line. No behavior change. The OVMF patch's diff is untouched -- only the commit message above it changes -- so the file it applies to is byte-identical and the firmware does not move. Signed-off-by: Kevin Wang --- docs/security/security-model.md | 13 +++-- os/image/README.md | 40 +++++++++------- os/image/normalize-kernel-header.py | 48 +++++++++++-------- ...elLoaderFsDxe-normalize-setup-header.patch | 22 +++++---- 4 files changed, 72 insertions(+), 51 deletions(-) diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 596699fa2..ca5647a71 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -358,11 +358,14 @@ here there is no knob. It also removes a class of correct-but-rejected deployments, since the previous QEMU-patched digest varied with guest RAM and was only reproducible at specific memory sizes. -The normalized field set comes from the boot protocol rather than from QEMU's -behavior: every field written is one `Documentation/arch/x86/boot.rst` types as -`write`, which the boot loader fills in and the kernel supplies no value for. -Fields typed `modify` carry real kernel-supplied values and are left as the -kernel built them. +What is written is one fixed header -- the one QEMU <= 10.1 in fact wrote for +these kernels -- and not a reimplementation of QEMU's loader. Every field +written is one QEMU fills in as boot loader; all but one are typed `write` in +`Documentation/arch/x86/boot.rst`, which the boot loader supplies and the +kernel has no value for. The exception is `loadflags`, typed +`modify (obligatory)`, of which only the `CAN_USE_HEAP` bit is set, a bit the +protocol assigns to the boot loader. Fields that carry real kernel-supplied +values are left as the kernel built them. ### TCB status is surfaced, not gated, during verification diff --git a/os/image/README.md b/os/image/README.md index 61123f834..d1f4e1256 100644 --- a/os/image/README.md +++ b/os/image/README.md @@ -80,15 +80,16 @@ The result is that RTMR[1] is the plain Authenticode hash of `bzImage` as listed in `sha256sum.txt`, on every QEMU version and at every guest memory size. -Normalizing to QEMU's layout rather than to zeros is what keeps an earlier -release able to verify these images: `dstack-mr` already computes that layout -(`patch_kernel`) for an image that does not declare the flag, so a 0.5.x KMS -can still onboard a 0.6.0 root with image verification on. Recomputing it needs -the guest RAM size, which limits such a verifier to guests with exactly 2 GiB -or at least 2816 MiB -- the one range QEMU places the initrd differently in, -and the range the no-image-download path already excludes. The values are the -ones QEMU writes above that threshold, and the shipped initrd's size is an -input, since it decides `ramdisk_image`: QEMU packs the initrd against a +Normalizing to a header QEMU once wrote, rather than to zeros, is what keeps an +earlier release able to verify these images: `dstack-mr` already computes that +header (`patch_kernel`) for an image that does not declare the flag, so a 0.5.x +KMS can still onboard a 0.6.0 root with image verification on. That +recomputation takes the guest RAM size as an input, so it reproduces the +normalized bytes only for guests of exactly 2 GiB or at least 2816 MiB. Below +2 GiB, and between 2 GiB and 2816 MiB, QEMU placed the initrd somewhere else, +so such a verifier computes a different RTMR[1]; the no-image-download path +refuses that same range for the same reason. The shipped initrd's size is an +input too, since it decides `ramdisk_image`: QEMU packs the initrd against a ceiling, which is 4G for a kernel that sets `XLF_CAN_BE_LOADED_ABOVE_4G` (bit 1 of `xloadflags` — bit 6, `0x40`, is `XLF_5LEVEL_ENABLED`, a different flag) and `0x37ffffff` for one that does not, then clamped to the below-4G @@ -104,15 +105,18 @@ served. true`. What makes that declaration true is the OVMF half, which the same build applies -- `ovmf-build.sh` and the bitbake recipe both fail if the patch does not apply, so an image cannot ship the flag with firmware that ignores it. -Images -without the field are the ones built before this existed; `dstack-mr` measures -those the old way, against QEMU's rewritten header. - -The field set comes from the boot protocol, not from QEMU's behavior: every -field `Documentation/arch/x86/boot.rst` types as `write` is one the boot loader -fills in and the kernel supplies no value for. Fields typed `modify` carry real -kernel-supplied values — `code32_start` is the protected-mode entry point — and -are deliberately left alone. +Images without the field are the ones built before this existed; `dstack-mr` +measures those the old way, against QEMU's rewritten header. + +What gets written is one fixed header, not a reimplementation of QEMU's loader: +every field written is one QEMU fills in as boot loader, and the script refuses +any kernel outside the shape it reproduces. All but one of those fields are +typed `write` in `Documentation/arch/x86/boot.rst`, meaning the boot loader +supplies them and the kernel has no value there. The exception is `loadflags`, +typed `modify (obligatory)`, of which only the `CAN_USE_HEAP` bit is set — a +bit the protocol assigns to the boot loader. Fields that carry real +kernel-supplied values, such as `code32_start`, the protected-mode entry point, +are deliberately left alone, and so are the `write` fields QEMU never touches. That is safe for every boot path: these are the values QEMU itself wrote for every release before 10.2, and on the EFI-stub path the real-mode setup code diff --git a/os/image/normalize-kernel-header.py b/os/image/normalize-kernel-header.py index 90e0ea7e8..01509fe93 100755 --- a/os/image/normalize-kernel-header.py +++ b/os/image/normalize-kernel-header.py @@ -13,19 +13,26 @@ before measuring, makes RTMR[1] the plain Authenticode hash of the file we ship, on every QEMU version. -The values are the ones QEMU <= 10.1 writes, for a kernel loaded high in a -guest with 2 GiB or more of RAM below 4G -- the only guest memory layout modelled, and the one `dstack-mr` -computes (`patch_kernel` in dstack/dstack-mr/src/kernel.rs) for an image that -does not declare `kernel_header_normalized`. Normalizing to QEMU's layout -rather than to zeros is what keeps a verifier or KMS from an earlier release -able to measure these images. - -The field set comes from the boot protocol, not from QEMU: every field written -is one `Documentation/arch/x86/boot.rst` types as `write`, which the boot -loader fills in and the kernel supplies no value for. Fields typed `modify` -carry real kernel-supplied values (`code32_start` is the protected-mode entry -point) and are left alone. Fields QEMU leaves alone are left alone too: a built -kernel ships them zero, so writing them would only add a way to disagree. +The values are the ones QEMU <= 10.1 wrote for a kernel loaded high in a guest +with 2 GiB or more of RAM below 4G. Zeros would serve the measurement goal +equally well; these particular values are chosen because `dstack-mr` already +computes them (`patch_kernel` in dstack/dstack-mr/src/kernel.rs) for an image +that does not declare `kernel_header_normalized`, so a verifier or KMS from an +earlier dstack release can still measure these images. + +This writes one fixed header. It is not a reimplementation of QEMU's loader: +the branches below reproduce the header QEMU in fact wrote for the kernels +dstack ships, and a kernel outside that shape is refused rather than guessed +at. + +Every field written is one QEMU fills in as boot loader. All but one are typed +`write` in `Documentation/arch/x86/boot.rst`: the boot loader supplies them and +the kernel has no value there. The exception is `loadflags`, typed +`modify (obligatory)`, of which only `CAN_USE_HEAP` is set -- a bit the +protocol assigns to the boot loader. Fields carrying real kernel-supplied +values, such as `code32_start`, are left alone, and so are the `write` fields +QEMU never touches: a built kernel ships them zero, so writing them would only +add a way to disagree. The initrd size decides `ramdisk_image`, so the initrd the image ships is an input. @@ -51,10 +58,9 @@ # What QEMU writes: "Qemu" version 0, the real-mode block and command line it # loads a kernel loaded high at, the 0x200-byte gap it leaves below the command -# line for -# the setup heap, the below-4G window it reserves for ACPI tables with RAM -# split at 2 GiB, the initrd ceiling a kernel that declares none gets, and the -# alignment it rounds the initrd address down to. +# line for the setup heap, the below-4G window it reserves for ACPI tables with +# RAM split at 2 GiB, the initrd ceiling a kernel that declares none gets, and +# the alignment it rounds the initrd address down to. TYPE_OF_LOADER_QEMU = 0xB0 REAL_ADDR = 0x10000 CMDLINE_ADDR = 0x20000 @@ -67,9 +73,11 @@ HEADER_MAGIC_OFFSET = 0x202 HEADER_MAGIC = b"HdrS" VERSION_OFFSET = 0x206 -# One past the last field this touches. Anything shorter cannot carry a setup -# header, and slicing past the end would silently grow the image instead of -# failing. The OVMF side applies the same bound. +# Minimum length for an image to be treated as carrying a setup header. The +# fields below all sit under 0x238; 0x258 is the end of the protocol 2.09 +# header, kept as the bound so a truncated image is rejected outright instead +# of being sliced past its end, which would silently grow it. The OVMF side +# applies the same bound. HEADER_END_OFFSET = 0x258 # `cmd_line_ptr` moved into the setup header in 2.02, `initrd_addr_max` # arrived in 2.03 and `xloadflags` in 2.12. diff --git a/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch b/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch index 525d5fea1..098002ed5 100644 --- a/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch +++ b/os/yocto/layers/meta-dstack/recipes-core/dstack-ovmf/dstack-ovmf/0007-OvmfPkg-QemuKernelLoaderFsDxe-normalize-setup-header.patch @@ -22,15 +22,21 @@ The values follow from the kernel file and the initrd size -- never from the QEMU version, and never from guest RAM: the initrd address QEMU derives from RAM is fixed at the value it resolves to for a guest with 2 GiB or more below 4G. A kernel loaded low is left alone, because QEMU's command-line address on -that path depends on the command line the host passes. Normalizing to QEMU's layout rather than to zeros is what keeps a verifier -or KMS from an earlier release able to measure these images: dstack-mr already -computes this layout (patch_kernel in dstack/dstack-mr/src/kernel.rs) for an -image that does not declare kernel_header_normalized. +that path depends on the command line the host passes. -The field set is taken from Documentation/arch/x86/boot.rst rather than from -QEMU's behavior: fields typed `write` are ones the boot loader fills in and the -kernel supplies no value for. Fields typed `modify` are left alone because they -carry real kernel-supplied values. Fields QEMU leaves alone are left alone too. +This writes one fixed header, not a reimplementation of QEMU's loader. Zeros +would serve the measurement goal equally well; these particular values are +chosen because dstack-mr already computes them (patch_kernel in +dstack/dstack-mr/src/kernel.rs) for an image that does not declare +kernel_header_normalized, so a verifier or KMS from an earlier dstack release +can still measure these images. + +Every field written is one QEMU fills in as boot loader. All but one are typed +`write` in Documentation/arch/x86/boot.rst: the boot loader supplies them and +the kernel has no value there. The exception is loadflags, typed `modify +(obligatory)`, of which only CAN_USE_HEAP is set -- a bit the protocol assigns +to the boot loader. Fields carrying real kernel-supplied values are left alone, +and so are the `write` fields QEMU never touches. Booting is unaffected. On the EFI-stub path the real-mode setup code never runs, the stub takes the initrd through LoadFile2 and the command line through