Repository navigation
gh-157917: Do not leak the \\?\ extended-length prefix into sys.prefix on Windows venvs - #158465
Conversation
….prefix on Windows venvs When a venv interpreter is invoked through a "\\?\" extended-length path, the venv launcher reports its own path via GetModuleFileNameW, which preserves that prefix. getpath.py consumed the value verbatim, so sys.prefix, sys.exec_prefix and sys.executable all kept the prefix. Windows does not resolve ".." or any other relative component in an extended-length path, so tools that join them onto a sysconfig path break. That is what makes "pip install" of a wheel with a "scripts=" entry fail with "OSError: [Errno 22] Invalid argument". Strip the prefix back off the launcher-provided path on Windows. A path that would still exceed MAX_PATH once unprefixed keeps the prefix, since the extended-length form is the only usable one for it.
8592297 to
82ad1e5
Compare
|
hi @FFY00 @vsajip, no urgency push here, just flagging since the change is small and all checks are green on it.. the fix stops the ?\ extended length prefix from leaking into sys.prefix on windows venvs, repro and the before/after paths are in the description.. a first look whenever you have a moment would help |
|
We have a customer waiting on this to be merged (an extension is misbehaving as a result). Do we have a status? |
|
Thanks @abhinav-phi for the PR, and @vsajip for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14. |
|
Thanks @abhinav-phi for the PR, and @vsajip for merging it 🌮🎉.. I'm working now to backport this PR to: 3.15. |
|
GH-162062 is a backport of this pull request to the 3.14 branch. |
|
GH-162063 is a backport of this pull request to the 3.15 branch. |
Fixes #157917.
Root cause
The reporter's analysis pointed at
Modules/getpath.py, and it is the right file, but the suggested fix (preferreal_executableover the configuredexecutable) does not work. I instrumentedgetpath.pyand captured the values it actually sees on 3.16.0a0 (main@00307b0e69) when the venv interpreter is invoked through a\\?\path:Two things follow:
real_executableis not a usable substitute. On Windows the venv'spython.exeis a launcher that execs the real interpreter, soGetModuleFileNameW(NULL)inModules/getpath.c:progname_to_dict()returns the base interpreter, not the venv one. Preferring it would make venv detection search the base installation forpyvenv.cfg, fail to find it, and fall out of venv mode entirely.The
\\?\prefix arrives through the launcher environment variable, not throughargv[0]orconfig['executable'].PC/venvlauncher.c:332does:GetModuleFileNameWhands back the extended-length form when that is how the process was invoked, so the launcher faithfully propagates\\?\in__PYVENV_LAUNCHER__.Modules/getpath.pythen consumed it verbatim:and the venv-detection block builds
venv_prefixfromexecutable_dir, sosys.prefixandsys.exec_prefixinherit the prefix. Becauseconfig['executable'] = executableat the end of the script,sys.executableis affected as well.The fix
Strip the extended-length prefix back off the launcher-provided path on Windows, right where that value is consumed. This mirrors what
ntpath.realpath()already does for the same prefix (Lib/ntpath.py, theprefix/unc_prefixhandling):\\?\C:\dirbecomesC:\dir, and\\?\UNC\server\sharebecomes\\server\share.Following the same spirit as
realpath(), a path that would still exceedMAX_PATHonce unprefixed keeps its prefix, because the extended-length form is the only usable one for it — so genuinely long venv paths keep working.Before / after
Repro from the issue, run against a local
PCbuild\build.bat -e -dbuild ofmainon Windows, before and after this patch:sys.prefix\\?\C:\Users\...\Temp\gh157917-x\venvC:\Users\...\Temp\gh157917-x\venvsys.exec_prefix\\?\C:\Users\...\Temp\gh157917-x\venvC:\Users\...\Temp\gh157917-x\venvsys.executable\\?\C:\Users\...\venv\Scripts\python.exeC:\Users\...\venv\Scripts\python.exesys.base_prefixC:\...\cpythonC:\...\cpythonThe downstream failure the issue reports, i.e. joining a RECORD-relative
../../Scripts/...path onto the venv'sLib\site-packages(what pip does when it rewrites ascripts=shebang):Test
New
test_venv.BasicTest.test_prefixes_extended_length_path(Windows-only) creates a venv, runs its interpreter through a\\?\path, and asserts that neithersys.prefixnorsys.exec_prefixkeeps the prefix. Verified that it fails without this patch:and passes with it.
Test suites run against the patched build:
Note for reviewers
I could not verify the reporter's "3.10 strips it" claim directly, as I have no 3.10 available here. From the code, 3.10's
PC/getpathp.cconsumed__PYVENV_LAUNCHER__the same way, so if 3.10 did print a clean prefix, the difference is likely in what itsGetModuleFileNameWcall returned rather than ingetpathitself. Either way, a cleansys.prefixis the expected behaviour, and this patch restores it.AI tools were used assistively; I reviewed and tested every change and take responsibility for it.