Repair v19 EBS conversion, Hub TLS, and TKLBAM runtime - #107
Open
lirazsiri wants to merge 21 commits into
Open
Conversation
Keep root filesystem population credential-free so the release controller can own AWS calls while the converter handles only its attached block device. Load AWS SDK dependencies only on legacy API paths, support NVMe partition names, and clean nested mounts after failure. Install the reviewed digest-bound tklamq 0.12.1 candidate before signed-archive hubclient 0.3.0, remove the candidate file, and retain exact source and binary provenance without persistent unsigned APT state. Verified with the focused EC2 v19 sandbox suite.
Keep the existing host interpreter path when it is available. When an older build host lacks that runtime, execute the completed v19 root filesystem's relocatable PyPy against the root and load its TKLBAM libraries, avoiding incompatible host package installation. Add a focused LibreSBX fixture covering host selection, root-runtime output, filtered profile contents, absence of staging residue, and nonzero failure propagation.
lirazsiri
changed the title
Frozen TKLDev v18 workers cannot load the Trixie PyPy runtime directly because it requires a newer glibc. Keep the existing host-runtime path, but run the fallback through chroot so the generator uses the libraries from the completed v19 root. Stage only the generator and selected profile inputs inside a unique root-local directory, return the generated archive, and clean the stage after successful generation, generator failure, or output-copy failure. The focused runtime test covers selection, arguments, filtering, cleanup, and exact failure propagation.
The Trixie turnkey-pypy2 package installs libpypy-c.so beside its interpreter, outside the default loader search path. The v19-root fallback reached the chroot correctly but failed before running the profile generator because that library was not found. Add only the exact packaged directory to the fallback chroot environment. The host-runtime branch is unchanged, and the focused test now requires the loader path while retaining generation and cleanup failure coverage.
The raw chroot reached the packaged v19 PyPy but aborted without the normal build runtime mounts. TKLDev already provides fab-chroot for commands that need the completed root environment. Use its status-preserving no-argument script path with a root-local wrapper that only execs the staged generator. Pass the exact three required environment values, retain scoped cleanup, and cover the installed one-command boundary plus generator and output-copy failures in the focused test.
EC2 images enable sudoadmin for ordinary non-Hub launches, but the firstboot key hook still wrote the instance key to root. Neither root nor admin could authenticate to a fresh canonical TKLDev v19 builder even though sshd was healthy. Select admin when the established SUDOADMIN setting is true and preserve root for Hub-style launches where sudoadmin is disabled. Focused EC2 tests cover true, case-insensitive true, false, and missing configuration values.
Archive-mode rootfs synchronization can copy a private extraction directory mode onto the mounted filesystem root. That prevents non-root EC2 principals from traversing to their authorized_keys file even when the key hook selects the correct account. Set the populated filesystem root to the conventional 0755 mode before installing GRUB. The focused EC2 test reproduces a 0700 source container and verifies every chroot command observes the normalized mode.
Hub launches already reserve and publish a tklapp.com hostname, but EC2 guests keep serving their generated self-signed certificate until an administrator runs Confconsole manually. Request a Let’s Encrypt certificate after HubDNS publishes the instance address, retain an hourly retry only while DNS or ACME is unavailable, and leave Confconsole’s daily renewal path in charge after success. Point Webmin at the generated intermediate chain so its TLS endpoint validates alongside appliance HTTPS services. The automation is intentionally limited to TurnKey-managed tklapp.com names; custom domains keep the existing explicit flow.
The published HubDNS 1.4.0 package has broken command links and imports the obsolete Python curl-wrapper API, which prevents Hub first boot from assigning its reserved hostname. Verify and install the reviewed 1.4.0+fix1 package while applying the EC2 image layer, then record its artifact digest and source commits in the guest. Keep the candidate ahead of hubclient installation and assert the installed package version. Extend the focused EC2 checks to cover both the package provenance and the tklapp.com-only certificate automation.
The first-boot TLS helper could run before HubDNS assigned a hostname and remove its own hourly retry. It also combined OpenSSL hostname and expiration checks in a way that let the final expiration result mask a hostname mismatch, allowing the self-signed appliance certificate to be mistaken for an ACME certificate. Keep missing HubDNS names retryable and require the dehydrated full-chain certificate to pass separate hostname and lifetime checks before disabling retries. Explicit non-tklapp.com names still disable the automation. Verified with shell syntax validation and the focused ec2-v19 test suite.
Replace the earlier HubDNS candidate digest with the package containing the systemd lifecycle fix. The new package keeps the exact tklapp.com record published after its one-shot update, preventing fresh instances from falling back to the Hub wildcard endpoint after boot. Record the source commit in the installed provenance metadata. The focused EC2 contract tests and shell syntax checks pass with the updated candidate.
Pass the temporary session token alongside the access key and secret when the Python 3 EC2 client is created. Without it, scoped instance-role credentials fail authentication before an AMI conversion can create its build volume. Cover the complete temporary credential tuple in the focused EC2 tests. The full EC2 contract suite and Python compilation pass.
Add an explicit local-device conversion path that prepares a pre-attached block device while leaving volume creation, snapshots, image registration, and tags to the release controller. This keeps AWS credentials out of the environment that extracts and modifies appliance code. Retain the existing direct AWS path for compatibility. The focused EC2 tests cover command routing, and the complete EC2 contract suite and shell syntax checks pass.
Replace the Bookworm images fingerprint in the v19 build configuration with the Trixie primary fingerprint used for the release. This lets conversion and publication tooling authenticate signed v19 hash files against the intended release identity. Add a focused regression assertion that rejects the former fingerprint. The complete EC2 contract suite passes.
The Confconsole HTTP-01 hook returned as soon as it sent a challenge path to the local add-water process. A fast ACME request could arrive before that process registered the token, causing a public 404 and leaving automatic tklapp.com TLS on its hourly retry path. Probe the exact token through the local HTTP endpoint and compare its content before returning to dehydrated. This preserves the existing challenge server while closing the publication race observed on a Hub-launched v19 WordPress instance.
lirazsiri
changed the title
Start the HubDNS unit after Hub credentials have been provisioned and before the first automatic certificate request. On EC2 the unit runs earlier during system startup, before its credential file exists, so it is skipped and the TLS helper otherwise waits against the old wildcard DNS answer. The focused EC2 suite now verifies the ordering and passes all 15 tests. This keeps the first certificate request on the normal boot path instead of deferring it to the hourly retry.
Rotate across the instance resolver list while waiting for the Hub-managed hostname. Route 53 can publish the new record just after the first EC2 resolver lookup, leaving that resolver with the previous wildcard answer and delaying first-boot TLS until the retry hook. The focused EC2 suite passes all 15 tests. The updated helper was also installed on a reproduced Hub launch with a stale primary-resolver answer and obtained a hostname-valid Let’s Encrypt certificate in 11 seconds without an external resolver override.
Retry the add-water control connection until its socket is ready, then require three stable token responses through the instance public address before asking the certificate authority to validate. A fresh Hub launch reproduced a 404 even after a single localhost readiness response, while the stronger public-path check completed the next certificate request successfully. The focused EC2 suite passes all 15 tests, and the updated hook obtained and installed a hostname-valid certificate on the reproduced WordPress launch.
A newly launched EC2 instance may not be able to reach its own public address during the first ACME attempt even after HubDNS is published. The existing hourly retry leaves every TLS interface on the self-signed certificate for too long. Add a non-overlapping minutely retry that removes itself as soon as a current hostname-valid certificate is installed. Keep the first attempt bounded so this transient network condition does not delay first-boot completion. Verified by the 16 focused EC2 v19 tests and shell syntax checks.
AWS Marketplace prohibits SSH password authentication, root remote login, embedded passwords, and authorized keys. Keep sudoadmin active for Hub and standalone boots, replace generated user hashes with a locked non-secret marker, unexpire admin, and install an EC2-only sshd policy. Make EC2 conversion validate account state, embedded keys, sudo access, and the effective sshd configuration. Focused tests prevent reintroducing the Hub root-switch hook or weakening the policy.
The previous substring check interpreted appliance paths containing -h, including Faveo Helpdesk, as help requests and aborted EC2 conversion before package cleanup. Parse each argument for the exact -h and --help options so ordinary rootfs paths continue through conversion while the documented help flags retain their behavior. Verified by reproducing the old Faveo path failure, exercising the corrected path with a stub chroot helper, checking both help flags, and running bash syntax and diff checks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
The v19 ISO-to-AMI path uses Nitro block devices and deliberately gives the conversion worker no AWS credentials. The signed archive also needs reviewed compatibility candidates until their fixed packages are published.
A clean Hub launch showed that a new EC2 instance can temporarily fail to reach its own public address even after HubDNS publishes the correct record. Keeping that transient condition on the first-boot path delays initialization, while the prior hourly fallback leaves users on the self-signed certificate too long. The first attempt remains bounded, and a serialized minutely retry now completes automatic TLS as soon as public routing is ready.
Verification
93444ac099bdb1a0b1aecf9118117f8e00f21e22befce2c8e1c47dd371f1fd9eami-08ec58e636c6c80c0reachedavailableinus-east-1and is shared with the Hub accounti-0234fb2e63966a266from that exact AMI and reachedrunning,booted, approved, and finalized/wp-admin/successfullyAuthored by Liraz Siri liraz@liraz.org.