[January 2027] Removing (now typed) setuptools._distutils

[Avasam]

For context: https://github.com/pypa/setuptools/commits/main/setuptools/_distutils is a direct vendor of https://github.com/pypa/distutils. Setuptools expands its own types from it (child classes) and does an import hack so that import distutils actually imports from setuptools._distutils.

That import hack is already covered by typeshed: We have a bunch of dummy distutils stub files that just star-import from setuptools._distutils, distributed as a stubs-only package in types-setuptools. (only works on Python 3.12, where distutils has been removed from stdlib)

Now that setuptools._distutils is as typed as typeshed is (and upstreamed synced to setuptools), this PR explores removing setuptools._distutils by using a partial setuptools stub.

This is in the same spirit of removing pkg_resources from typeshed's stubs once it was fully first-party typed by setuptools. (ref: #13369). Which are both following the idea of removing obsolete stubs.

Looks like from the context of our tests, mypy doesn't understand that the setuptools stub is partial and should reference the runtime setuptools._distutils. So I had to "teach" the mypy test about it by using [mypy-tests] section in METADATA.toml, and bring that feature to the stubtest and regression tests as well.

[Avasam]

The primer results for setuptools are either true positives (see pypa/setuptools#4861) or require updates in pypa/distutils (see pypa/distutils#357).

Other repos haven't been investigated yet.

[Avasam]

Are the mypy-primer failures because it's not yet downloading setuptools from types-setuptools ?

I'm not sure if it's an actual solution, but trying hauntsaninja/mypy_primer#254

[Avasam]

E stub_uploader.metadata.InvalidRequires: Expected dependency setuptools to be listed in cffi's requires_dist, the sdist's *.egg-info/requires.txt or pyproject.toml

Ah, stub_uploader tests doesn't support the dependency being found in [build-system].requires.

Suggested change

	dependencies = ["setuptools"]
	dependencies = ["types-setuptools"]

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

setuptools (https://github.com/pypa/setuptools)
- setuptools/monkey.py:76: error: Unused "type: ignore[misc, assignment]" comment  [unused-ignore]
- setuptools/monkey.py:82: error: Item Module of Module | Any has no attribute "Distribution"  [union-attr]
- setuptools/monkey.py:85: error: Unused "type: ignore[misc, assignment]" comment  [unused-ignore]
+ setuptools/monkey.py:85: error: Unused "type: ignore[assignment]" comment  [unused-ignore]
+ setuptools/logging.py:35: error: Incompatible types in assignment (expression has type Module, variable has type "Logger")  [assignment]
- setuptools/config/_apply_pyprojecttoml.py:336: error: Cannot access instance-only attribute "global_options" on class object  [misc]
+ setuptools/config/_apply_pyprojecttoml.py:272: error: List item 0 has incompatible type "list[setuptools._distutils.extension.Extension]"; expected "setuptools.extension.Extension"  [list-item]
+ setuptools/command/bdist_egg.py:139: error: Item "None" of "list[str | tuple[Any, ...]] | None" has no attribute "__iter__" (not iterable)  [union-attr]
- setuptools/build_meta.py:91: error: Unused "type: ignore" comment  [unused-ignore]
- setuptools/build_meta.py:95: error: Unused "type: ignore" comment  [unused-ignore]
+ setuptools/command/build_py.py:66: error: No overload variant of "copy_file" of "Command" matches argument types "str | PathLike[str]", "StrPathT", "bool", "bool", "str | None", "object"  [call-overload]
+ setuptools/command/build_py.py:66: note: Possible overload variants:
+ setuptools/command/build_py.py:66: note:     def [_StrPathT: str | PathLike[str]] copy_file(self, infile: str | PathLike[str], outfile: _StrPathT, preserve_mode: bool = ..., preserve_times: bool = ..., link: str | None = ..., level: int = ...) -> tuple[_StrPathT | str, bool]
+ setuptools/command/build_py.py:66: note:     def [_BytesPathT: bytes | PathLike[bytes]] copy_file(self, infile: bytes | PathLike[bytes], outfile: _BytesPathT, preserve_mode: bool = ..., preserve_times: bool = ..., link: str | None = ..., level: int = ...) -> tuple[_BytesPathT | bytes, bool]
+ setuptools/command/build_ext.py:226: error: Argument "force" to "new_compiler" has incompatible type "None"; expected "bool"  [arg-type]
- setuptools/command/build_clib.py:82: error: Value of type variable "_SourcesT" of "newer_pairwise_group" cannot be "list[Any]"  [type-var]

mypy (https://github.com/python/mypy)
+ mypyc/build.py:142: error: Module "distutils.core" has no attribute "Extension"  [attr-defined]
+ mypyc/build.py:147: error: Type alias target becomes "Extension | Any" due to an unfollowed import  [no-any-unimported]
+ mypyc/build.py:165: error: Module has no attribute "Extension"  [attr-defined]

comtypes (https://github.com/enthought/comtypes)
- comtypes/test/setup.py:3: error: Cannot find implementation or library stub for module named "distutils.core"  [import-not-found]
+ comtypes/test/setup.py:3: error: Module "distutils.core" has no attribute "setup"  [attr-defined]

aioredis (https://github.com/aio-libs/aioredis)
+ aioredis/connection.py:11: error: Module "distutils.version" has no attribute "StrictVersion"  [attr-defined]

pip (https://github.com/pypa/pip)
+ src/pip/_internal/locations/_distutils.py:22: error: Module "distutils.cmd" has no attribute "Command"  [attr-defined]
- src/pip/_internal/locations/_distutils.py:68: error: Incompatible types in assignment (expression has type "Command", variable has type "install")  [assignment]
+ src/pip/_internal/locations/_distutils.py:24: error: Module "distutils.command.install" has no attribute "install"  [attr-defined]
+ src/pip/_internal/locations/_distutils.py:25: error: Module "distutils.sysconfig" has no attribute "get_python_lib"  [attr-defined]
+ src/pip/_internal/locations/_distutils.py:49: error: Module "distutils.dist" has no attribute "Distribution"  [attr-defined]
+ src/pip/_internal/locations/__init__.py:131: error: Module "distutils.command.install" has no attribute "install"  [attr-defined]
+ src/pip/_internal/locations/__init__.py:132: error: Module "distutils.dist" has no attribute "Distribution"  [attr-defined]

dd-trace-py (https://github.com/DataDog/dd-trace-py)
- ddtrace/sourcecode/setuptools_auto.py:11: error: Unused "type: ignore" comment  [unused-ignore]

[srittau]

I don't pretend to understand what's going on here, but I trust that you know what you're doing (and I couldn't spot anything that looked out of order).

[Avasam]

@srittau I added more context to the PR description:

For context: https://github.com/pypa/setuptools/commits/main/setuptools/_distutils is a direct vendor of https://github.com/pypa/distutils. Setuptools expands its own types from it (child classes) and does an import hack so that import distutils actually imports from setuptools._distutils.

That import hack is already covered by typeshed: We have a bunch of dummy distutils stub files that just star-import from setuptools._distutils, distributed as a stubs-only package in types-setuptools. (only works on Python 3.12, where distutils has been removed from stdlib)

[...]

This is in the same spirit of removing pkg_resources from typeshed's stubs once it was fully first-party typed by setuptools. (ref: #13369). Which are both following the idea of removing obsolete stubs.

In other words, distutils is now first-party typed on Python 3.12+

I also can't find anything wrong with this in theory, but in practice those mypy_primer results do concern me (not sure if it's a mypy limitation, or if hauntsaninja/mypy_primer#254 would actually fix it.

Also, I just remembered, if the point of this is to reflect stubs obsoletion, I should be waiting 6 months from now as per our guidelines.