vfs: add --vfs-mount and --vfs-load startup flags - #65748
Conversation
|
Review requested:
|
nodejs-github-bot
added
lib / src
|
|
||
| ```console | ||
| $ node --experimental-vfs --vfs-mount=app.zip --vfs-load app.zip | ||
| $ node --experimental-vfs --vfs-mount=lib.zip --vfs-mount=app.zip --vfs-load=1 |
There was a problem hiding this comment.
The -vfs-load=n feels... brittle. I'd prefer the ability to label each mount and specify by name:
$node --experimental-vfs --vfs-mount=app:app.zip --vfs-mount=lib:lib.zip --vfs-load=appThere was a problem hiding this comment.
What happens when someone specifies --vfs-mount=app:app.zip and then the command-line is altered with --vfs-mount=app:something-else.zip? (Or even worse NODE_OPTIONS="--vfs-mount=app:malicious.zip")
I considered just always having --vfs-load load from the first --vfs-mount, but that seemed inflexible. So this was my compromise.
In short: I see your point and agree with the sentiment, but don't think labeling is the right solution. Happy to discuss, or be overruled, or whatever. This is a classic case of a strong opinion weakly held.
There was a problem hiding this comment.
The --vfs-load=N would have the same brittleness, wouldn't it? We could clarify that having multiple --vfs-mount with the same label is an error. Alternatively, --vfs-mount-load=... --vfs-mount=... where the vfs-mount-load= serves dual purpose could also work.
There was a problem hiding this comment.
The --vfs-load=N would have the same brittleness, wouldn't it?
No, because --vfs-load is prohibited in NODE_OPTIONS and the commandline args are taken before NODE_OPTIONS, so an index cannot be altered.
so NODE_OPTIONS="--vfs-mount=malicious.zip --vfs-load=1" node ... -vfs-load=0 --vfs-mount=myapp.zip would error.
We could just do --vfs-mount=data/ just do mounting and --vfs-load=myapp.zip. However that would inhibit workers as the mounts must be the same while NOT running from its root. So...
There was a problem hiding this comment.
Possibly we could do --vfs-load=myapp.zip and intersperse it with --vfs=mount=<x> and the pass the ---vfs-load to a worker as --vfs-mount maintaining the order.
I like that as that makes it absolutely clear what's what!
I'll push that changes when ready.
There was a problem hiding this comment.
I pushed that change. Do you like this better?
The zip support in node:zlib is released (v26.8.0) and ZipProvider merged today as nodejs/node#64915, so the old framing — "three things that are not in any release", pointing at #64339 and pipobscure/node#3 — was wrong in both directions: it undersold what has landed and misnamed what has not. What is actually outstanding is nodejs/node#65748, the --vfs-mount / --vfs-load flags and the vfs.registerProvider() that ships with them, plus nodejs/node#65680 for loading native addons out of a mount. Reading those two turned up semantics this repo's prose had wrong: --vfs-load runs the *first* mount (or --vfs-load=<index>), not the last; --vfs-mount takes no target, since node assigns a reserved mount point; argv[1] is the mounted source's real path rather than the mount point; and registered providers are now offered directories as well as files, which retires the constraint that made tools/observe.ts a runner rather than a preload. The deck's status slide said "This is in Node" with the flags marked as landed. It now says "landing", carries the two open PRs and the addon work as their own rows, and the speaker notes say plainly that the keystone is still a pull request. Deck republished to the artifact link in the README.
nodejs/node#65748 changed the flag on 2 Sep: --vfs-load used to select one of the --vfs-mount sources by 0-based index, which meant counting mounts out by hand and an optional-valued flag the options parser could only express as an alias onto a hidden --vfs-load-index. It now takes the source itself and mounts it, so `--vfs-load --vfs-mount X` is gone and `--vfs-load=X` does both. The launcher prefix was the breaking part: shell-base still exec'd node with the two-flag form, which a current build rejects outright with "--vfs-load requires an argument", so every archive signed with --launcher would have failed to run. It is now `--vfs-load="$0"`, and 77 bytes rather than 89. mountArgv() and the test helper follow, along with the command lines in the comments, the README, HISTORY, the deck and the release workflow's header. Two consequences worth recording rather than only fixing: the self-mounting shebang is one flag now, since the kernel-appended path becomes --vfs-load's value, and NODE_OPTIONS mounts sort before the command line's rather than after, because with no index to protect the order cannot change what runs. Verified against a node built from that branch (v27.0.0-pre): the whole suite passes, 136 of 136, and a signed launcher runs by name with dash-leading arguments reaching the program. While in there: tools/observe.ts explained itself with a constraint that no longer holds — registered providers are offered directories too — so it now gives the real reason it is a runner, which is that it wants the mount point.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #65748 +/- ##
==========================================
+ Coverage 89.95% 89.97% +0.01%
==========================================
Files 759 760 +1
Lines 258637 258899 +262
Branches 49015 49054 +39
==========================================
+ Hits 232665 232948 +283
+ Misses 17018 17007 -11
+ Partials 8954 8944 -10
🚀 New features to boost your workflow:
|
Mounting a virtual file system requires calling vfs.mount() from inside the program, so a program cannot itself be served from one: something already running has to mount the VFS first. Add two startup flags. --vfs-mount=<source> mounts a directory or an archive as a virtual file system, and may be repeated. --vfs-load=<source> mounts that source the same way and additionally runs the entry point and all subsequent require()/import resolution against it rather than against the real file system, so an application can be run straight out of a directory or a ZIP archive: node --experimental-vfs --vfs-load=my-app.zip Both options append to one list, so mounts happen in the order written: node --experimental-vfs --vfs-mount=a --vfs-load=b --vfs-mount=c mounts a, b and c in that order and runs b. Mounting the same source twice mounts it twice, at two mount points, and the entry point comes from the mount --vfs-load contributed rather than the earlier one. The provider backing a source is chosen from the source itself rather than from its name: a directory is served by RealFSProvider and a file whose bytes are a ZIP archive by ZipProvider, so an archive can carry any extension. vfs.registerProvider() registers a provider for formats there is no built-in for; selection is deferred until -r and --import preloads have run, so a preloaded module can register one before its source is claimed. The entry point comes from the mount the way `node <directory>` takes one, from package.json "main" or index.js. Nothing is consumed as an entry point argument, so every positional reaches the program unchanged from argv[2] on, and argv[1] reports the named source rather than the reserved mount point, which is an opaque implementation detail. --vfs-load may be given at most once, and is not permitted in NODE_OPTIONS: which entry point runs is the command line's decision, and NODE_OPTIONS is parsed first, so an environment variable could otherwise redirect any invocation. --vfs-mount is permitted there. Workers inherit the same mounts in the same order but not the loading: a worker mounts what the parent mounted and runs its own entry point. Signed-off-by: Philipp Dunkel <pip@pipobscure.com>
|
Rebased because of documentation conflict. |
|
As per our practice and documentation those should be |
github-actions
Bot
removed
the
request-ci
They are already behind --experimental-vfs to begin with, so does that still hold? Happy to oblige if so, just double checking. Precedence: It's --allow-vfs-fs and not --experimental-allow-vfs-fs |
4 participants
vfs: add --vfs-mount and --vfs-load startup flags
Mounting a virtual file system requires calling
vfs.mount()from inside theprogram, so a program cannot itself be served from one — something already
running has to mount the VFS first.
This adds two startup flags:
--vfs-mount=<source>mounts a directory or an archive as a virtual filesystem. May be repeated.
--vfs-load=<source>mounts<source>exactly as--vfs-mountdoes, andadditionally runs the entry point and all subsequent
require()/importresolution against that mount rather than the real file system.
Together they let an application run straight out of a directory or a ZIP
archive:
$ node --experimental-vfs --vfs-load=my-app.zipMount order
Both options append to one list, so mounts happen in the order written:
$ node --experimental-vfs --vfs-mount=a --vfs-load=b --vfs-mount=cmounts
a,bandcin that order and runsb. Mounting the same sourcetwice mounts it twice, at two separate mount points; the entry point comes from
the mount
--vfs-loadcontributed, not from an earlier--vfs-mountof thesame source.
Choosing a provider
The provider backing a source is chosen from the source itself rather than from
its file name: a directory is served by
RealFSProvider, and a file whose bytesare a ZIP archive by
ZipProvider, so an archive can carry any extension.vfs.registerProvider()registers a provider for formats there is no built-infor. Selection is deferred until
-rand--importpreloads have run, so apreloaded module can register one before its source is claimed.
Entry point and arguments
The entry point is taken from the mount the same way
node <directory>takesone: the mount's own
package.json"main", orindex.js. Because the entrypoint comes from the mount, no positional argument is consumed as one — every
positional reaches the program unchanged from
argv[2]onward.process.argv[1]reports the named source rather than the reserved mount point, which is an
opaque implementation detail.
Constraints
--vfs-loadmay be given at most once; a second is rejected at startup.--vfs-loadis not permitted inNODE_OPTIONS: which entry point runs is thecommand line's decision, and
NODE_OPTIONSis parsed first, so an environmentvariable could otherwise redirect any invocation.
--vfs-mountis permittedthere and its mounts precede the command line's.
worker mounts what the parent mounted and runs its own entry point.
--experimental-vfs.Notes for reviewers
--vfs-mountand--vfs-loadshare onevfs_mountslist so ordering ispreserved by construction; a bracketed internal boolean set via
Implies()records that a load was requested, and which entry it contributed is recovered
from
execArgv.in
EnvironmentOptions::CheckOptions(), which runs at the end of every parse —otherwise
NODE_OPTIONS=--vfs-mount=x node --experimental-vfsis rejected foran
--experimental-vfsthat has not been read yet.separately and is independent of this change.
For context what this gives us see bundling tools and slides