Skip to content

feat!: replace shared-memory init data with virtqueue buffers - #2030

Open
andreiltd wants to merge 2 commits into
hyperlight-dev:mainfrom
andreiltd:remove-init-data
Open

andreiltd wants to merge 2 commits into
hyperlight-dev:mainfrom
andreiltd:remove-init-data

Conversation

@andreiltd

Copy link
Copy Markdown
Member

Remove the peb backed init-data region and APIs. Use retained virtqueue ByteChunks for startup data. After removing the init section PEB no longer contains any data so it can be removed as well.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Guest entry ABI and persisted snapshot layout changes across x86_64 and aarch64 require validation of rebuilt guests and initialization, restore, fork, and OCI round trips on supported hypervisors before approval.

No open issues 🎉

What changed in this PR

This PR routes guest startup data through existing virtqueue buffers and removes the PEB-backed initialization region. This is a breaking change requiring rebuilt guests and regenerated snapshots.

Changes:

  • Remove initialization-data APIs and simplify host and guest startup arguments.
  • Update snapshots to ABI 10 and config schema v6, with legacy formats rejected.
  • Add retained-buffer snapshot tests and document startup-data migration.
File Description
src/​tests/​rust_guests/​simpleguest/​src/​main.rs Remove the legacy user-memory test function.
src/​tests/​rust_guests/​simpleguest/​src/​bin/​simpleguest-initial-data.rs Add a guest that retains startup buffers.
src/​tests/​rust_guests/​dummyguest/​src/​main.rs Validate the simplified entry arguments.
src/​hyperlight_host/​tests/​snapshot_goldens/​goldens_version.rs Set snapshot goldens to v10.0.
src/​hyperlight_host/​tests/​integration_test.rs Remove assertions for deleted region variants.
src/​hyperlight_host/​src/​sandbox/​uninitialized.rs Remove blob wrappers and accept GuestBinary directly.
src/​hyperlight_host/​src/​sandbox/​uninitialized_evolve.rs Initialize guests without a PEB address.
src/​hyperlight_host/​src/​sandbox/​snapshot/​tripwires.rs Pin the updated snapshot contract.
src/​hyperlight_host/​src/​sandbox/​snapshot/​mod.rs Build snapshots without PEB or init-data regions.
src/​hyperlight_host/​src/​sandbox/​snapshot/​file/​mod.rs Load schema v6 and reject older schemas.
src/​hyperlight_host/​src/​sandbox/​snapshot/​file/​media_types.rs Define config v6 and ABI 10.
src/​hyperlight_host/​src/​sandbox/​snapshot/​file/​config.rs Remove init-data fields from serialized configuration.
src/​hyperlight_host/​src/​sandbox/​snapshot/​file_tests.rs Test retained startup buffers and format compatibility.
src/​hyperlight_host/​src/​sandbox/​initialized.rs Update initialization and snapshot tests.
src/​hyperlight_host/​src/​sandbox/​builder.rs Remove the initialization-data builder option.
src/​hyperlight_host/​src/​mem/​virtq/​tests.rs Update the memory-layout fixture.
src/​hyperlight_host/​src/​mem/​mgr.rs Update snapshot and layout test constructors.
src/​hyperlight_host/​src/​mem/​memory_region.rs Remove PEB and init-data region definitions.
src/​hyperlight_host/​src/​mem/​layout.rs Simplify initial memory to guest code and update layout tests.
src/​hyperlight_host/​src/​mem/​exe.rs Update tests for Snapshot::from_binary.
src/​hyperlight_host/​src/​hypervisor/​mod.rs Update the guest initialization test.
src/​hyperlight_host/​src/​hypervisor/​hyperlight_vm/​x86_64.rs Adjust x86-64 startup registers.
src/​hyperlight_host/​src/​hypervisor/​hyperlight_vm/​aarch64.rs Adjust AArch64 startup registers.
src/​hyperlight_guest/​src/​lib.rs Remove the guest-handle module export.
src/​hyperlight_guest/​src/​guest_handle/​handle.rs Delete the PEB-backed guest handle.
src/​hyperlight_guest_bin/​src/​lib.rs Remove PEB initialization and simplify random seeding.
src/​hyperlight_guest_bin/​src/​host_comm.rs Remove the legacy user-memory reader.
src/​hyperlight_guest_bin/​src/​arch/​amd64/​init.rs Update entry arguments and stack-pivot register.
src/​hyperlight_guest_bin/​src/​arch/​aarch64/​mod.rs Update entry arguments and stack-pivot register.
src/​hyperlight_common/​src/​mem.rs Delete shared PEB structures.
src/​hyperlight_common/​src/​lib.rs Remove the deleted memory module export.
docs/​snapshot-versioning.md Document ABI 10 and config v6 compatibility.
docs/​snapshot-oci-format.md Reference config schema v6.
docs/​how-to-build-a-hyperlight-guest-binary.md Explain startup-buffer retention and pool sizing.
CHANGELOG.md Record breaking APIs and migration requirements.

Review effort: Balanced

@hyperlight-gh-bot

This comment has been minimized.

@andreiltd andreiltd added C-refactor For PRs that restructure or remove code without adding new functionality. S-needs-goldens Regenerate snapshot golden fixtures labels Nov 6, 2026
ludfjig
ludfjig previously approved these changes Nov 6, 2026

@ludfjig ludfjig left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, with 1 nit: build-rust-guests-non-pie runs cargo build without --bin, so it also builds this binary. Only simpleguest is copied to non_pie. Adding --bin simpleguest to that recipe avoids an extra build in each guest CI job.

@hyperlight-gh-bot

This comment has been minimized.

@hyperlight-gh-bot

This comment has been minimized.

@hyperlight-gh-bot

This comment has been minimized.

@hyperlight-gh-bot

This comment has been minimized.

@hyperlight-gh-bot

This comment has been minimized.

@hyperlight-gh-bot

This comment has been minimized.

@syntactically syntactically left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very glad to see this finally gone!

macro still happen automatically. Your `hyperlight_main` only needs to do
whatever extra setup the macros do not cover.

### Startup data

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this section is not very clear right now: it reads to me like it is mentioning 3 different approaches to init data (call host function in hyperlight_main returning vec, call guest function returning vec , call guest function with vec)) but doesn't really contrast them.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section has been rewritten.

@hyperlight-gh-bot

This comment has been minimized.

@andreiltd

Copy link
Copy Markdown
Member Author

Hey @syntactically, I moved the instruction cache fixes here: #2039

@hyperlight-gh-bot

This comment has been minimized.

jsturtevant
jsturtevant previously approved these changes Nov 11, 2026
@andreiltd

andreiltd commented Nov 12, 2026 •

Copy link
Copy Markdown
Member Author

Just to be clear, #2039 needs to land first. For some reason the changes in this PR triggered that bug.

Remove the peb backed init-data region and APIs. Use retained virtqueue
ByteChunks for startup data. After removing the init section PEB no
longer contains any data so it can be removed as well.

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>
Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>
@hyperlight-gh-bot

This comment has been minimized.

@hyperlight-gh-bot

Copy link
Copy Markdown

Benchmark Results

Measured commit: af2f732640d8
Baseline commit: d652690b1c7f

kvm / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 765.99 ns (➖ 1.00x slower)
vec_bytes 599.46 ns (➖ 1.02x slower)
380.84 µs (➖ 1.02x slower)

payload_allocation

slot_pool_segmented
262144 551.58 ns (➖ 1.06x faster)
65536 141.31 ns (➖ 1.00x slower)

sandboxes

create_initialized_and_drop
medium 13.37 ms (➖ 1.01x faster)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
7.72 ns (➖ 1.02x faster) 7.89 ns (➖ 1.03x slower) 7.68 ns (➖ 1.01x faster)

snapshot_files

load_snapshot_unverified
small 96.61 µs (➖ 1.00x faster)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 7.35 µs (➖ 1.04x slower) 6.99 µs (➖ 1.03x faster)
65536 1.93 µs (➖ 1.09x faster) 2.04 µs (➖ 1.05x slower)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 26.57 µs (➖ 1.19x faster)
65536 6.14 µs (➖ 1.06x faster) 6.38 µs (➖ 1.05x faster)
8192 1.01 µs (➖ 1.02x faster) 1.04 µs (➖ 1.01x slower)
kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 811.70 ns (➖ 1.02x faster)
vec_bytes 622.12 ns (➖ 1.00x slower)
742.90 µs (➖ 1.04x faster)

payload_allocation

slot_pool_segmented
262144 594.98 ns (➖ 1.00x slower)
65536 207.21 ns (➖ 1.00x slower)

sandboxes

create_initialized_and_drop
medium 14.97 ms (➖ 1.00x slower)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
8.25 ns (➖ 1.00x slower) 8.15 ns (➖ 1.00x faster) 8.22 ns (➖ 1.00x slower)

snapshot_files

load_snapshot_unverified
small 58.61 µs (➖ 1.00x slower)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 9.01 µs (➖ 1.00x faster) 9.02 µs (➖ 1.00x faster)
65536 2.56 µs (➖ 1.01x slower) 2.54 µs (➖ 1.01x faster)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 36.28 µs (➖ 1.01x slower)
65536 8.39 µs (➖ 1.02x faster) 8.44 µs (➖ 1.01x faster)
8192 875.65 ns (➖ 1.04x slower) 864.53 ns (➖ 1.01x faster)
mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 925.83 ns (➖ 1.03x faster)
vec_bytes 706.55 ns (➖ 1.03x faster)
322.82 µs (➖ 1.04x faster)

payload_allocation

slot_pool_segmented
262144 730.49 ns (➖ 1.00x faster)
65536 204.17 ns (➖ 1.03x slower)

sandboxes

create_initialized_and_drop
medium 61.84 ms (➖ 1.02x faster)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
10.10 ns (➖ 1.05x slower) 9.54 ns (➖ 1.00x faster) 9.60 ns (➖ 1.00x faster)

snapshot_files

load_snapshot_unverified
small 105.18 µs (➖ 1.01x faster)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 9.74 µs (➖ 1.06x faster) 10.94 µs (➖ 1.02x slower)
65536 2.24 µs (➖ 1.01x faster) 2.42 µs (➖ 1.02x faster)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 36.31 µs (➖ 1.07x faster)
65536 8.30 µs (➖ 1.27x faster) 8.31 µs (➖ 1.18x faster)
8192 1.24 µs (➖ 1.02x slower) 1.23 µs (➖ 1.05x faster)
mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 915.67 ns (➖ 1.02x faster)
vec_bytes 659.09 ns (➖ 1.00x slower)
686.90 µs (➖ 1.12x faster)

payload_allocation

slot_pool_segmented
262144 662.21 ns (➖ 1.00x slower)
65536 264.71 ns (➖ 1.01x slower)

sandboxes

create_initialized_and_drop
medium 81.48 ms (➖ 1.11x slower)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
8.69 ns (➖ 1.01x faster) 8.68 ns (➖ 1.05x faster) 8.69 ns (➖ 1.00x faster)

snapshot_files

load_snapshot_unverified
small 56.67 µs (➖ 1.05x slower)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 7.96 µs (➖ 1.01x slower) 7.94 µs (➖ 1.01x faster)
65536 2.27 µs (➖ 1.01x faster) 2.30 µs (➖ 1.02x slower)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 43.06 µs (➖ 1.07x slower)
65536 7.55 µs (➖ 1.01x slower) 7.51 µs (➖ 1.05x faster)
8192 905.91 ns (➖ 1.00x faster) 913.50 ns (➖ 1.01x faster)
hyperv-ws2025 / amd (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 1.27 µs (➖ 1.11x slower)
vec_bytes 747.43 ns (➖ 1.00x slower)
1.89 ms (➖ 1.01x faster)

payload_allocation

slot_pool_segmented
262144 794.67 ns (➖ 1.00x faster)
65536 231.41 ns (➖ 1.00x slower)

sandboxes

create_initialized_and_drop
medium 11.33 ms (➖ 1.10x faster)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
10.85 ns (➖ 1.03x slower) 10.60 ns (➖ 1.00x faster) 10.61 ns (➖ 1.01x faster)

snapshot_files

load_snapshot_unverified
small 704.66 µs (➖ 1.09x slower)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 8.63 µs (➖ 1.01x faster) 8.64 µs (➖ 1.00x faster)
65536 2.51 µs (➖ 1.00x slower) 2.55 µs (➖ 1.02x slower)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 38.26 µs (➖ 1.03x faster)
65536 9.06 µs (➖ 1.02x faster) 8.87 µs (➖ 1.02x faster)
8192 1.35 µs (➖ 1.01x faster) 1.35 µs (➖ 1.00x slower)
hyperv-ws2025 / intel (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 1.25 µs (➖ 1.03x slower)
vec_bytes 799.73 ns (➖ 1.01x slower)
3.03 ms (➖ 1.05x slower)

payload_allocation

slot_pool_segmented
262144 761.30 ns (➖ 1.01x faster)
65536 209.19 ns (➖ 1.02x faster)

sandboxes

create_initialized_and_drop
medium 13.38 ms (➖ 1.05x faster)

slot_pool

alloc_dealloc_128 alloc_dealloc_4096 alloc_dealloc_1500
9.75 ns (➖ 1.11x faster) 9.90 ns (➖ 1.19x faster) 9.62 ns (➖ 1.09x faster)

snapshot_files

load_snapshot_unverified
small 668.32 µs (➖ 1.00x slower)

virtq_readonly

slot_pool_segmented slot_pool_segmented_fragmented
262144 7.75 µs (➖ 1.01x slower) 7.47 µs (➖ 1.03x faster)
65536 2.31 µs (➖ 1.01x faster) 2.29 µs (➖ 1.02x faster)

virtq_readwrite

slot_pool_segmented slot_pool_segmented_fragmented
262144 45.08 µs (➖ 1.24x faster)
65536 8.41 µs (➖ 1.01x faster) 8.44 µs (➖ 1.02x faster)
8192 1.05 µs (➖ 1.14x faster) 1.13 µs (➖ 1.05x slower)

Reported by cargo ci bench-report --candidate run:43394283462 --config-file bench_report.toml.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

C-refactor For PRs that restructure or remove code without adding new functionality. S-needs-goldens Regenerate snapshot golden fixtures

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants