storage: KVM - enable RBD/Ceph volume encryption support#13556
Open
calvix wants to merge 10 commits into
Open
Conversation
added 10 commits
July 7, 2026 12:57
Flip StoragePoolType.RBD from EncryptionSupport.Unsupported to Hypervisor so the existing encryption control plane (allocator, endpoint selector, offerings) treats RBD pools as encryption-capable. The agent-side encrypted RBD create path is not implemented yet; it is delivered by two follow-up tracks (qemu-native engine='qemu' and ceph-native engine='librbd'). Until then, fail closed at the two RBD create chokepoints in LibvirtStorageAdaptor (createPhysicalDisk and createDiskFromTemplate) when a passphrase is present, so we never silently produce a plaintext volume that the control plane believes is encrypted. No change for existing unencrypted RBD volumes (guards only fire when a passphrase is set; supportsEncryption() only affects volumes that require encryption).
Implements encrypted RBD data and root disks using librbd's native LUKS2 encryption, decrypted at runtime by libvirt/qemu via <encryption engine='librbd'>. CloudStack manages the passphrase (existing model). - RbdEncryption: isolated helper wrapping `rbd encryption format luks2`, cephx via --id + keyfile (secret not on the command line), LUKS passphrase via KeyFile. Kept separate so the CLI can later be swapped for a JNA binding (rados-java has no rbd_encryption_format API). - LibvirtStorageAdaptor: create/clone the raw RBD image, then apply `rbd encryption format luks2`; mark the disk LUKS2 so encrypt_format propagates to the volume. Replaces the fail-closed guards. - QemuObject.EncryptFormat: add LUKS2. - LibvirtVMDef: render <encryption format='luks2' engine='librbd'>; the encrypt details now carry an optional engine. - attach (KVMStorageProcessor) and boot (LibvirtComputingResource): set engine='librbd' for RBD-backed encrypted volumes. NOTE: the CoW-clone-then-format path (encrypted root from an unencrypted template) needs live-cluster validation for the parent-grow / usable-size behaviour described in the Ceph image-encryption docs. Builds: api + plugins/hypervisors/kvm (JDK11).
hostSupportsVolumeEncryption() now advertises encryption capability if the host supports EITHER qemu-native LUKS (qemu-img LUKS + cryptsetup) OR librbd native encryption (rbd CLI with the encryption subcommand). Previously a Ceph-only host that lacked cryptsetup would not advertise encryption even though librbd can encrypt RBD volumes. Split into hostSupportsQemuNativeVolumeEncryption() and hostSupportsRbdVolumeEncryption(); kept HOST_VOLUME_ENCRYPTION as the single host-wide flag (documented limitation: not per-pool).
Encrypted RBD volumes are encrypted natively by librbd and must be resized with `rbd resize --encryption-passphrase-file` so librbd grows the encrypted payload and keeps the LUKS header consistent. The existing encrypted-resize path (resizeEncryptedQcowFile) uses qemu-img --object secret, which is for qemu-native LUKS and does not fit the librbd LUKS2 layout. - RbdEncryption.resize(): new `rbd resize` wrapper (cephx via --id + keyfile, passphrase via KeyFile, optional --allow-shrink). - LibvirtResizeVolumeCommandWrapper: detect encrypted RBD and route to the rbd resize path, bypassing the libvirt v.resize and qemu-img paths. Snapshot/revert, RBD<->RBD copy, and migration of encrypted RBD volumes need no code changes: they operate on the raw (LUKS-containing) image at the block level, and the destination passphrase secret is already created engine-agnostic in LibvirtPrepareForMigrationCommandWrapper. These still require live validation. Builds: plugins/hypervisors/kvm (JDK11).
For a running VM, an librbd-encrypted RBD volume must be resized in-band by qemu/librbd, not out-of-band by the rbd CLI. Gate the CLI rbd-resize path on !vmIsRunning so: - offline -> `rbd resize --encryption-passphrase-file` (librbd-aware), and - online -> existing NOTIFYONLY path -> virsh blockresize, where qemu's block_resize delegates to librbd to grow the encrypted payload and notify the guest in one step (no passphrase needed; qemu holds the secret). This avoids notify-less out-of-band growth and qemu/librbd size divergence while the image is open. Online behaviour still needs live validation that blockresize resizes the encrypted payload for engine='librbd' disks.
Root disks could not be encrypted: cloning a plaintext template and then `rbd encryption format`ing the clone leaves the inherited OS data unreadable (the LUKS header offsets it), so the guest could not mount root. Fix, in createDiskFromTemplateOnRBD, with two paths: - Option A (same-cluster cached RBD template): grow the template base to reserve LUKS2 header space, snapshot+protect it (cloudstack-base-snap-luks), clone from it, apply the LUKS2 header, resize the clone to the requested size. Inherited template data stays readable through the clone's encryption and the clone is a thin CoW image (only the header is written). - Option B (first-use / non-RBD template): create an empty image, apply a LUKS2 header, then import the template THROUGH the encryption layer via RbdEncryption.importTemplate (qemu-img convert -n into encrypt.key-secret). Correct but a full copy. Validated end-to-end on Ubuntu 26.04 / libvirt 12.0.0: both boot; A is thin (3.5 GiB provisioned, ~120 MiB used); LUKS2 verified at rest on Ceph.
Review pass over the librbd LUKS2 encryption feature to fix latent issues
and bring it in line with CloudStack conventions:
- RbdEncryption: reject empty/null passphrase with a clear error; round
rbd --size up to MiB so a non-aligned request never shrinks the volume
below what was asked for; create the temporary cephx conf/keyring 0600
explicitly instead of relying on the umask.
- LibvirtStorageAdaptor: close Rados/IoCTX/RbdImage in a finally block on
the encrypted-root paths (mirrors deleteVolume) so handles are not
leaked on exceptions; use parameterized log messages instead of string
concatenation; extract the encrypted-root Option A/B logic into
createEncryptedRootCoWClone / createEncryptedRootFullCopy.
- RbdEncryption: use an instance logger (matching the plugin convention)
and split argv construction into build{Format,Resize,Convert}Script so
the generated commands can be unit-tested.
Assert the rbd/qemu-img argv built for format, resize and convert-through-encryption (RBD and file sources), and that empty/null passphrases are rejected. Command construction is verified without a live Ceph cluster.
libvirt 10.0.0 has an object apply-order bug (fixed in 10.1.0) that breaks hot-plug of an encrypted rbd blockdev: on attach the disk is opened before its LUKS secret object is defined, so the attach fails with "No secret with id '...-format-encryption-secret0'". Booting a VM from an encrypted RBD disk is unaffected (the QEMU command line resolves all -object before -blockdev). Refuse the attach up front with a clear error (mirroring the existing openvswitch/io_uring libvirt-version gates) instead of letting libvirt fail opaquely. Only the RBD hot-plug path is gated; boot/root/detach are untouched.
|
Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
|
calvix
changed the title
calvix
changed the title
Member
|
@calvix great job ! I was planning to implement it later this year 😄 |
Member
|
@blueorangutan package |
|
@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #13556 +/- ##
============================================
- Coverage 20.67% 19.41% -1.26%
- Complexity 19156 19166 +10
============================================
Files 5805 6285 +480
Lines 523689 565108 +41419
Branches 61126 68957 +7831
============================================
+ Hits 108258 109739 +1481
- Misses 403720 443459 +39739
- Partials 11711 11910 +199
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18489 |
Member
|
@blueorangutan test |
|
@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
[SF] Trillian test result (tid-16501)
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hello, this PR adds a feature to support an encrypted RBD volume for CloudStack. As this is a necessary feature we need for infrastructure in our company and we currently run custom build with this and we would like to have it in the CloudStack upstream.
Description
This PR adds Ceph-native (librbd) LUKS2 encryption for RBD volumes on the KVM, covering both data disks and root disks. Encryption/decryption is done transparently by qemu+librbd via
<encryption format='luks2' engine='librbd'>. The guest sees a normal virtio disk.It reuses CloudStack's existing passphrase escrow , the same mechanism as the current qemu-native volume encryption. RBD's
StoragePoolTypeencryption support is flipped fromUnsupportedtoHypervisor, and the KVM agent gainsengine='librbd'path for the create, boot, attach, and resize flows.The
rbd/qemu-imgCLI dependency is used becauserados-java0.x does not expose therbd_encryption_formatAPI; therbdis a new dependency, and it means the user needs to installceph-commonpackage on the host.Operations covered
rbd encryption format luks2(standardLUKSheader)rbd clone- LUKS2 format - encryption-aware resizeqemu-img convert -ntemplate through the encryption layer<encryption format='luks2' engine='librbd'>rbd resize --size … --encryption-passphrase-file(encryption-aware; grows the payload, keeps the header consistent)virsh blockresize(librbd handles the layout)engine='librbd'; guest sees plaintextgetUsableBytesFromRawBytes) — the guest sees raw − header; the root path additionally reserves the header so the OS image always fits.Requirements/caveats
<encryption format='luks2' engine='librbd'>.ceph-commonpackage on the KVM Host (the rbd CLI)qemu_block.cthat emits the encryption secret after the rbd blockdev, so a hot-plug fails withNo secret with id 'xxxxxx'.This is fixed in version 10.1.0+. This bug does not affect booting from an encrypted volume. The agent refuses this attach on older libvirt with a corresponding error.
format='luks2' engine='librbd'), a separate stack from CloudStack's existing qemu-native LUKS1 encryption. librbd was chosen because it can encrypt a thin CoW clone of an unencrypted template with a per-volume key — qemu-native LUKS treats the whole image as one opaque LUKS container, so it would force a full, non-thin copy of the template for every encrypted VM (or share one key across all clones) — and because it's LUKS2 and keeps RBD's native snapshot/clone/resize working. A volume is decrypted by the same engine that encrypted it; the two are not mixed on a single volume.Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
How Has This Been Tested?
Environment: 9-node KVM + Ceph cluster - hosts on Ubuntu 26.04 (libvirt 12.0.0), Ceph 20.2 (dedicated ceph cluster +
ceph-commonon hosts), management server 4.22.1.0.Unit tests — new
RbdEncryptionTest(9 tests, all green) asserts the exactrbd/qemu-imgargv built forformat,resizeand convert-through-encryption (RBD and file sources), and that empty/null passphrases are rejected. Command construction is verified without a live Ceph cluster.End-to-end through the CloudStack API on the CS cluster:
mkfs.ext4+ write + remount + read-back - online resize - detach. Similar for offline resize.cryptsetup luksDump: LUKS2,aes-xts-plain64, 16 MiB data offset.test to ensure wrong values/behavior will not cause unexpected or hidden issues
format/resize/importTemplate→ guarded with a clear error.--sizerounds up so the volume never ends up below the request.-lukssnapshot.finally; temp cephx conf/keyring (0600) cleaned up on all paths.