Repository navigation
Conversation
Proposes RFC `610.0000` (`Pachunk`), an in-place content-addressed installer and delta updater for the Flutter SDK.
|
I'm currently working on getting the prototype in a github repo to share. |
|
|
||
| --- | ||
|
|
||
| ### 3.5 The Universal Packfile, FlatBuffer Manifest, & `SHA-256` Verification |
There was a problem hiding this comment.
I am re-working my prototype and trying to remove the techdebt. I expect this section to change, but it does not change the RFC's goals and measures.
| 4. Host each release from a single static `<version>.pack` file on GCS/CDN for serverless delivery over HTTP/2 `Range` requests. | ||
| 5. Update in place (Zero-CAS) using the local files in `~/flutter` as the chunk cache instead of maintaining a duplicate object store on disk, and verify every reconstructed file against its authoritative `SHA-256` digest. | ||
|
|
||
| ### Critical User Journeys (CUJs) |
There was a problem hiding this comment.
What happens in terms of signing or validating hashes?
There was a problem hiding this comment.
For the packfiles? Nothing. We don't sign our zip or tar.xz files today.
The files that are signed (mac/ios binaries) remain signed and bit-for-bit equal.
What we should do: sha256 sum the packfile and toc on luci and upload to the same "release_*.json" files we use today. The tool can verify the toc hash quickly. We want to avoid downloading the entire 1.8GB file just to verify it and download it for the 50MB we need.
If I misunderstood the question, let me know.
There was a problem hiding this comment.
Today, the "Attestation bundles" linked from https://docs.flutter.dev/install/archive are .intoto.jsonl files whose payload contains the sha256 of the zip or tar.xz files. (This is for our SLSA compliance). We probably need to continue publishing the attestation bundle, but it could be for the packfiles instead of for the zip/tar.xz?
There was a problem hiding this comment.
Ah, I see. The Flutter Tool doesn't care about these in-toto files as far as I can tell (git grep for "in-toto" and "intoto" show nothing). I believe these intoto files are only for releases and not for commits on main.
The solution here would be produce these files for the packfiles as well. The packer could run at the same time as the tar.xz and .zip files are being generated, since it only relies on the zipfiles and flutter tool right now.
| ### 5.2 Open Questions for Reviewers | ||
|
|
||
| 1. How should `pachunk` integrate with `flutter_tools` and the official install/upgrade flow? | ||
| * Option A (Recommended): Make `pachunk` the primary installation method on `flutter.dev` so `flutter` "Just Works" out of the box. Because `flutter_tools` already uses `flutter precache` to download engine and tool artifacts today, have `flutter precache`, `flutter upgrade`, and `flutter downgrade` delegate to `pachunk` under the hood (migrating artifact download responsibility into `pachunk` over time) with an automatic fallback to legacy `.zip` archive downloads during transition. |
There was a problem hiding this comment.
For both of these options it's not clear to me from the descriptions:
- What does the user download from flutter.dev initially by clicking on a link.
- What are they missing at that point, and how do they get the rest.
Asking the user to copy-paste a command is not my favorite when others do it, but would be an option as well if it is technically necessary.
Also worth noting here that we currently document an "Install with VSCode" flow that would also need to be modified, I think? https://docs.flutter.dev/install/with-vs-code
There was a problem hiding this comment.
Good questions and worth arguing here - maybe I should leave off the recommended for now and just have it as an opinion.
- What does the user download from flutter.dev initially by clicking on a link.
I think a script or command much like https://brew.sh/ or https://github.com/jtmcdole/flutter_worktree#unattended-installation.
- What are they missing at that point, and how do they get the rest
If its the script, we could have it automatically install for the host/arch. If its just a binary, they would need to run it to download and install a version.
- We could consult
release_<host>.jsonfor latest stable / beta if they use--stableor--beta - We could offer code block copy-paste like brew for the different components.
VSCode: noted! I... assume devtools just downloads the sdk and didn't invent a new method of downloading Flutter? @kenzieschmoll
| * Recommended: Record the `sha256` of both `<version>.pack` and `manifest_<version>.fb.zst` in `releases_<platform>.json` and verify `manifest_<version>.fb.zst` before assembly. Is that sufficient, or do we need anything further for third-party mirrors (`FLUTTER_STORAGE_BASE_URL`)? | ||
| 5. Should we retire the `bin/internal/` bootstrap scripts (`.sh`, `.bat`, `.ps1`) and compile `flutter_tools` at install time? | ||
| * Today, every `flutter` command runs `bin/internal/` scripts (`shared.*`, `update_dart_sdk.*`, `update_engine_version.*`) to check locks, query Git, fetch the Dart SDK, and build snapshots. Because `pachunk` verifies the Dart SDK, engine files, and `flutter.version.json` during install or upgrade, these runtime scripts are obsolete. | ||
| * Recommended: Compile a standalone AOT native binary (`dart compile exe` or `aot-snapshot`) into `bin/cache/` during install and upgrade. Retire the `bin/internal/` scripts in `pachunk` installs, and remove them upstream once `pachunk` becomes the default installer. Keep `bin/flutter*` and `bin/dart*` (`sh`, `.bat`, `.ps1`) only as thin pointer scripts that `exec` the `bin/cache/` binaries directly. This lets users run `flutter` and `dart` without typing `.exe` across POSIX shells, CMD, PowerShell, and Git Bash. |
There was a problem hiding this comment.
The bootstrapping scripts currently download a Dart SDK as an early (first?) step. Where do they get the Dart SDK to run the rest of the code without the boostrapping scripts? (I think the answer here is "pachunk", but how do they get pachunk?)
Overall, let's say we take the recommended answers to all of these questions, I'm having trouble seeing exactly how the initial download/bootstrap/run flow works at a level of detail that gives me confidence that it will work. Can this doc dive into that a bit more?
There was a problem hiding this comment.
Yeah, I can go into that flow in the appendix if you like. "Expected workflows". But to quickly answer:
- Download pachunk
- pachunk downloads all the components (dart, engine artifacts, packages)
- pachunk compiles the flutter tool for your host-arch
- fin.
"flutter.bat" and "flutter" (bash) are only there as pointers to the compiled executables so the developer continues to see flutter run -d chorme.
I was chatting a little with @bkonyi about this and I think we'd like the simplicity of dropping some of these scripts.
There was a problem hiding this comment.
100% in favor of dropping gnarly shell scripts. Just want to understand how the design is intended to work =)
There was a problem hiding this comment.
Yeah, I think that's also "we can choose to do this in the future" incremental improvement and leave the scripts as-is today. There are some "please don't check git for tags or version" changes... "content aware hashing" continues to work for Contributors since they are working in git and need to solve for that.
- Remove "exact" numbers, e.g. 10,183 files, from all over the RFC - use relative numbers bigger: - address prototype changes (format of packfile, techdebt, pointer plane)
| - '"John McDole" <codefu@google.com>' | ||
| --- | ||
|
|
||
| # RFC 610.0000: Pachunk: Content-Addressed Packing & In-Place Delta Patching |
There was a problem hiding this comment.
The goals sound good and the overview of how we'll do it sounds good.
Skimming through it, how does this play with the .git directory? What will that look like after I checkout? How do I switch back to a git repo? I think some of these answers are scattered around but a clear declaration of that would be helpful.
Relying on git we get the advantage of git's testing. What are our plans for integration testing this to make sure it's correct in all situations. Running through typical CI workflows won't be enough since they won't duplicate situations developers will run into by default.
Some testing ideas from gemini:
• Messy local states: files the user edited or deleted, a half-finished update (killed process, Ctrl+C), legacy flutter precache mixed with pachunk, two flutter processes running
at once, and a stale .fast_cache.fb.
• Filesystem and OS cases: Windows file locks and Defender, symlinks and junctions, case-insensitive filesystems, permissions or read-only installs, a full disk, and long paths on Windows.
• Network cases: proxies or mirrors (FLUTTER_STORAGE_BASE_URL) that don't support Range or HTTP/2, dropped connections, and truncated responses.
• Version pairs: upgrade and downgrade tests across many pairs of backfilled releases, not just N→N+1. Also check that old clients reject newer PINF formats cleanly.
• Fuzz and property tests: round-trip zucchini split/merge and ZIP repacking on every binary in every release, plus fuzzing the manifest and pack parsers.
• Rollout data: telemetry on SHA-256 mismatches and how often fallbacks happen during the opt-in period, with a threshold that must be met before pachunk becomes the default.
There was a problem hiding this comment.
You're specifically wanting the view as a Contributor right (someone who has .git folders, the engine and dev folder?
There was a problem hiding this comment.
Yea, or an app developer that wants to try out a patch.
3 participants
Proposes RFC
610.0001pachunk(from "patch + chunk"), an in-place content-addressed installer and delta updater for the Flutter SDK.Flutter distributes SDK releases and engine artifacts through full-history
gitclones and monolithic archives (.zipand.tar.xz) on GCS. A fresh release archive unpacks at least 2.5 GB on Linux (1.5 GBarchive), 3.3 GB on Windows (1.9 GBarchive), and 4.1 GB on macOS ARM64 (2.2 GBarchive; 6.24 GB onceflutter precachepopulatesbin/cache/)...What if delta-upgrades (3.47.5 -> 3.47.6) were only 50MB and 20 seconds?
What if you could upgrade/downgrade from any version to any version?
What if Flutter had a dedicated installer?
Tracking issue: flutter/flutter#194032
Pre-launch Checklist
design doc.dart run bin/assign_rfc_number.dartlocally when instructed).