Skip to content

gh-157917: Do not leak the \\?\ extended-length prefix into sys.prefix on Windows venvs - #158465

Merged
vsajip merged 1 commit into
python:mainfrom
abhinav-phi:fix-157917-venv-extended-path-prefix
Nov 15, 2026
Merged

vsajip merged 1 commit into
python:mainfrom
abhinav-phi:fix-157917-venv-extended-path-prefix

Conversation

@abhinav-phi

@abhinav-phi abhinav-phi commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #157917.

Root cause

The reporter's analysis pointed at Modules/getpath.py, and it is the right file, but the suggested fix (prefer real_executable over the configured executable) does not work. I instrumented getpath.py and captured the values it actually sees on 3.16.0a0 (main @ 00307b0e69) when the venv interpreter is invoked through a \\?\ path:

os_name         = 'nt'
executable      = '\\?\C:\...\venv\Scripts\python.exe'
real_executable = 'C:\...\PCbuild\amd64\python_d.exe'   <-- the BASE install
executable_dir  = '\\?\C:\...\venv\Scripts'

Two things follow:

  1. real_executable is not a usable substitute. On Windows the venv's python.exe is a launcher that execs the real interpreter, so GetModuleFileNameW(NULL) in Modules/getpath.c:progname_to_dict() returns the base interpreter, not the venv one. Preferring it would make venv detection search the base installation for pyvenv.cfg, fail to find it, and fall out of venv mode entirely.

  2. The \\?\ prefix arrives through the launcher environment variable, not through argv[0] or config['executable']. PC/venvlauncher.c:332 does:

    DWORD len = GetModuleFileNameW(NULL, buffer, MAXLEN);
    ...
    SetEnvironmentVariableW(L"__PYVENV_LAUNCHER__", buffer);

    GetModuleFileNameW hands back the extended-length form when that is how the process was invoked, so the launcher faithfully propagates \\?\ in __PYVENV_LAUNCHER__. Modules/getpath.py then consumed it verbatim:

    executable = ENV_PYTHONEXECUTABLE or ENV___PYVENV_LAUNCHER__
    executable_dir = dirname(executable)

    and the venv-detection block builds venv_prefix from executable_dir, so sys.prefix and sys.exec_prefix inherit the prefix. Because config['executable'] = executable at the end of the script, sys.executable is affected as well.

The fix

Strip the extended-length prefix back off the launcher-provided path on Windows, right where that value is consumed. This mirrors what ntpath.realpath() already does for the same prefix (Lib/ntpath.py, the prefix / unc_prefix handling): \\?\C:\dir becomes C:\dir, and \\?\UNC\server\share becomes \\server\share.

Following the same spirit as realpath(), a path that would still exceed MAX_PATH once unprefixed keeps its prefix, because the extended-length form is the only usable one for it — so genuinely long venv paths keep working.

Before / after

Repro from the issue, run against a local PCbuild\build.bat -e -d build of main on Windows, before and after this patch:

before after
sys.prefix \\?\C:\Users\...\Temp\gh157917-x\venv C:\Users\...\Temp\gh157917-x\venv
sys.exec_prefix \\?\C:\Users\...\Temp\gh157917-x\venv C:\Users\...\Temp\gh157917-x\venv
sys.executable \\?\C:\Users\...\venv\Scripts\python.exe C:\Users\...\venv\Scripts\python.exe
sys.base_prefix C:\...\cpython C:\...\cpython

The downstream failure the issue reports, i.e. joining a RECORD-relative ../../Scripts/... path onto the venv's Lib\site-packages (what pip does when it rewrites a scripts= shebang):

before: open('\\?\C:\...\venv\Lib\site-packages\..\..\Scripts\bottle.py')
        -> OSError: [Errno 22] Invalid argument
after:  open('C:\...\venv\Lib\site-packages\..\..\Scripts\bottle.py')
        -> OK

Test

New test_venv.BasicTest.test_prefixes_extended_length_path (Windows-only) creates a venv, runs its interpreter through a \\?\ path, and asserts that neither sys.prefix nor sys.exec_prefix keeps the prefix. Verified that it fails without this patch:

FAIL: test_prefixes_extended_length_path [...] [prefix]
    self.assertFalse(out.strip().startswith('\\\\?\\'), out)
AssertionError: True is not false : \\?\C:\Users\abhin\AppData\Local\Temp\tmpz_hxibr8\venv
FAIL: test_prefixes_extended_length_path [...] [exec_prefix]
AssertionError: True is not false : \\?\C:\Users\abhin\AppData\Local\Temp\tmpz_hxibr8\venv

and passes with it.

Test suites run against the patched build:

python_d.exe -m test test_getpath test_embed test_sys test_venv test_sysconfig
== Tests result: SUCCESS ==
All 5 tests OK.
Total tests: run=315 skipped=38

Note for reviewers

I could not verify the reporter's "3.10 strips it" claim directly, as I have no 3.10 available here. From the code, 3.10's PC/getpathp.c consumed __PYVENV_LAUNCHER__ the same way, so if 3.10 did print a clean prefix, the difference is likely in what its GetModuleFileNameW call returned rather than in getpath itself. Either way, a clean sys.prefix is the expected behaviour, and this patch restores it.

AI tools were used assistively; I reviewed and tested every change and take responsibility for it.

@python-cla-bot

python-cla-bot Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

….prefix on Windows venvs

When a venv interpreter is invoked through a "\\?\" extended-length path, the
venv launcher reports its own path via GetModuleFileNameW, which preserves
that prefix. getpath.py consumed the value verbatim, so sys.prefix,
sys.exec_prefix and sys.executable all kept the prefix.

Windows does not resolve ".." or any other relative component in an
extended-length path, so tools that join them onto a sysconfig path break.
That is what makes "pip install" of a wheel with a "scripts=" entry fail with
"OSError: [Errno 22] Invalid argument".

Strip the prefix back off the launcher-provided path on Windows. A path that
would still exceed MAX_PATH once unprefixed keeps the prefix, since the
extended-length form is the only usable one for it.
@abhinav-phi
abhinav-phi force-pushed the fix-157917-venv-extended-path-prefix branch from 8592297 to 82ad1e5 Compare September 30, 2026 01:48
@abhinav-phi

Copy link
Copy Markdown
Contributor Author

hi @FFY00 @vsajip, no urgency push here, just flagging since the change is small and all checks are green on it.. the fix stops the ?\ extended length prefix from leaking into sys.prefix on windows venvs, repro and the before/after paths are in the description.. a first look whenever you have a moment would help

@nicholasgoss-dynatrace

Copy link
Copy Markdown

We have a customer waiting on this to be merged (an extension is misbehaving as a result). Do we have a status?

@vsajip
vsajip merged commit c4f4928 into python:main Nov 15, 2026
55 checks passed
@vsajip vsajip added needs backport to 3.14 bugs and security fixes needs backport to 3.15 bugs and security fixes labels Nov 15, 2026
@miss-islington-app

Copy link
Copy Markdown

Thanks @abhinav-phi for the PR, and @vsajip for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

@miss-islington-app

Copy link
Copy Markdown

Thanks @abhinav-phi for the PR, and @vsajip for merging it 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

@bedevere-app

bedevere-app Bot commented Nov 15, 2026

Copy link
Copy Markdown

GH-162062 is a backport of this pull request to the 3.14 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Nov 15, 2026
@bedevere-app

bedevere-app Bot commented Nov 15, 2026

Copy link
Copy Markdown

GH-162063 is a backport of this pull request to the 3.15 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.15 bugs and security fixes label Nov 15, 2026
vsajip pushed a commit that referenced this pull request Nov 15, 2026
…s.prefix on Windows venvs (GH-158465) (GH-162063)

(cherry picked from commit c4f4928)
vsajip pushed a commit that referenced this pull request Nov 15, 2026
…s.prefix on Windows venvs (GH-158465) (GH-162062)

(cherry picked from commit c4f4928)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants