Skip to content

Handle overloaded method ambiguity caused by self-types - #22011

Open
ilevkivskyi wants to merge 7 commits into
python:masterfrom
ilevkivskyi:overload-self-amb
Open

Handle overloaded method ambiguity caused by self-types#22011
ilevkivskyi wants to merge 7 commits into
python:masterfrom
ilevkivskyi:overload-self-amb

Conversation

@ilevkivskyi

@ilevkivskyi ilevkivskyi commented Sep 20, 2026

Copy link
Copy Markdown
Member

Fixes #11347
Closes #17239

I think we should handle this consistently with overloaded functions, essentially foo.meth(x) should behave the same as Foo.meth(foo, x). Although this is a relatively niche situation, it seems to be important for some numerical libraries.

It looks like there are no simple way to implement this properly. The only way I see requires a new attribute on Overloaded and a lot of plumbing (form binging site to the call site). I tried to apply various optimizations to reduce possible performance impact.

cc @JukkaL @hauntsaninja

@github-actions

This comment has been minimized.

Sign in to view
@ilevkivskyi

ilevkivskyi commented Sep 20, 2026

Copy link
Copy Markdown
Member Author

The fallout in colour is likely because this uncovers some other bug. Normally inferring more Anys should never cause problems. I will take a look at this later.

What is more concerning, is all the test failure in pandas-stubs and scipy-stubs. @jorenham You mentioned in #18343 (comment) that #11347 negatively affects some numerical libraries, but from what I see, it may be not what you actually want. Looking at some test failures and corresponding definitions this PR behaves as expected. For example:

tests/spatial/test__rotation.pyi:115: error: Expression is of type "ndarray[Any, Any]", not "ndarray[tuple[Any, ...], dtype[float64]]"  [assert-type]

Relevant line

assert_type(_rot_nd.as_quat(), onp.ArrayND[np.float64])

relevant definition

    @overload
    def as_quat(
        self: Rotation[_JustAnyShape], /, canonical: bool = False, *, scalar_first: bool = False
    ) -> onp.ArrayND[np.float64]: ...
    @overload
    def as_quat(
        self: Rotation[tuple[()]], /, canonical: bool = False, *, scalar_first: bool = False
    ) -> onp.Array1D[np.float64]: ...
    @overload
    def as_quat(
        self: Rotation[tuple[int]], /, canonical: bool = False, *, scalar_first: bool = False
    ) -> onp.Array2D[np.float64]: ...
    @overload
    def as_quat(self, /, canonical: bool = False, *, scalar_first: bool = False) -> onp.ArrayND[np.float64]: ...

and

_rot_nd: Rotation

So Rotation is the same as Rotation[tuple[Any, ...]] (due to type variable default). Then since Rotation[tuple[Any, ...]] is a subtype of several self-types here (Rotation is covariant and tuple[Any, ...] is a subtype of tuple[int] etc). Therefore multiple overloads match. Therefore now (with this PR) we identify this as "ambiguous overload match caused by Any", and correctly use erased return type.

@jorenham Could you please comment on which overload(s) you expect to match here and why?

@jorenham

jorenham commented Sep 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

The first overload with Rotation[_JustAnyShape] is just a workaround for pyright, which eagerly selects the first overload in case of unknown shape-type. The last overload is the fallback overload, which intentionally has the same gradual-shape-typed result type as the first. This last overload returns onp.ArrayND[np.float64], which is assignable to all of the other overloads' return types. So according to the typing spec, using onp.ArrayND[np.float64] as return type for Rotation[tuple[Any, ...]] would be sound, and would match my intention.

You might also be interested in a recent similar discussion for Pyrefly: facebook/pyrefly#4910 (comment)

@ilevkivskyi

ilevkivskyi commented Sep 20, 2026

Copy link
Copy Markdown
Member Author

@jorenham Specific fallback type may be more precise, this is true. Mypy already infers ndarray[Any, Any] instead of plain Any, but better fallback logic may be (very) hard to specify in general.

Anyway, my question is different: you mentioned that the bug I am fixing here (where bug == mypy randomly selecting first overload in case if Any-ambiguity is caused by self-types) causes problems form numerical libraries. But from what I see, this PR only makes things worse (at least at first glance), so my question is do you actually want/need this?

@jorenham

jorenham commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

But from what I see, this PR only makes things worse (at least at first glance), so my question is do you actually want/need this?

Well, in the meantime I've worked around most of these cases where possible in NumPy's bundled stubs and in scipy-stubs (e.g. using the tuple[Never, Never, ...] trick in numpy/numpy#29218).

But it looks like this solution results in many Anys to be inferred, more than necessary (typing-spec-wise, see my previous comment). Maybe that's worth it, but that's hard to tell because the primer diff doesn't show the cases where my workarounds can now be removed, and at this point I'm not sure if those new Any inferences can be worked around yet.

The number of new primer errors for scipy-stubs isn't all too bad, so I can live with it. But the diff for colour looks a bit more worrying, and I can't speak for its maintainers. I'm kinda suprised that numpy isn't in the primer diff though, but I guess that's good sign? The new unused-ignore and red diff lines show that this is at least doing something right, so at the very least you're on the right track with this.

Anyway, don't consider scipy-stubs a blocker for this. I've seen those ndarray[Any, Any] before in other contexts in numpy and scipy-stubs, so improving those (i.e. maning them more precise and less Any-ish) is probably another topic altogether I'm guessing.

@ilevkivskyi

ilevkivskyi commented Sep 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

@jorenham
About color: after a brief look at the relevant code there I think I know what is going to on. Mypy has a special case in narrowing logic: it never narrows an annotated variable to a plain Any. So now that we infer more plain Anys, some cases that previously worked, will stop working. For the record I didn't like this special case (I still think mypy should follow SSA as close as possible), but caved to the peer pressure. cc @cdce8p and @JukkaL who said that not having this special case will cause too many false negatives.

Now back to the main topic

Anyway, don't consider scipy-stubs a blocker for this.

Note pandas-stubs also has a lot of new errors. In general it seems to me we may first need to find a way to find better fallback type in case of ambiguity. For example, one may check if one of the return types is both subtype and supertype of all other return types, like onp.ArrayND[np.float64] in the example we discussed. Such logic is however quadratic in number of matches, so we may need to put some arbitrary limit (which is not something new for overloads btw, several things related to overloads are quadratic and need artificial cut-offs).

This was referenced Sep 20, 2026
Open
Merged
@ilevkivskyi ilevkivskyi mentioned this pull request Sep 27, 2026
Merged
ilevkivskyi added a commit that referenced this pull request Sep 28, 2026
@ilevkivskyi
Limit binder special-casing to imprecise Any (#22050)
41ddcdb 
There is a bunch of special-casing for `Any` types in binder. The goal
of this special-casing is to limit the spread of `Any` types from legacy
code with missing/imprecise types. This special-casing is quite ad-hoc,
and thus should be limited only to imprecise kinds of `Any`.

Note this should help with #22011
@github-actions

This comment has been minimized.

Sign in to view
@ilevkivskyi

ilevkivskyi commented Sep 28, 2026

Copy link
Copy Markdown
Member Author

OK, mypy_primer looks broken, but I think I have a fix hauntsaninja/mypy_primer#266

@github-actions

This comment has been minimized.

Sign in to view
@ilevkivskyi

ilevkivskyi commented Sep 29, 2026

Copy link
Copy Markdown
Member Author

OK, primer now looks better. The pandas-stubs and scipy-stubs fallout will be hopefully reduced by #22060

ilevkivskyi added a commit that referenced this pull request Sep 30, 2026
@ilevkivskyi
Find precise fallback for ambiguous Any overloads (#22060)
b5b279f 
It is a common pattern (especially in numeric libraries) to have a
fallback overload that has a relatively precise return type (i.e. not
just `Any` or `list[Any]`). We should try to find and use that overload
if there is an ambiguity caused by an argument that contains `Any`.

This will likely cause some new errors, but this should limit the
fallout from #22011
@github-actions

This comment has been minimized.

Sign in to view
@github-actions

This comment has been minimized.

Sign in to view
@ilevkivskyi

ilevkivskyi commented Oct 1, 2026

Copy link
Copy Markdown
Member Author

@jorenham After I merged #22060 situation with scipy-stubs improved, but there are still a bunch of failures in tests/stats/test_multivariate.pyi Could you please check these? If mypy is doing something wrong there, could you please reduce it to a simple standalone repro?

@Dr-Irv @cmp0xff There are still a lot of new Anys in pandas-stubs. I am trying to understand expectations here. I am looking at one of the failures:

tests/indexes/test_mul.py:60: error: Expression is of type "Any", not "Index[Any]"  [assert-type]

the relevant lines are

def test_mul_numpy_array(left_i: pd.Index) -> None:
    """Test pd.Index[Any] (int) * numpy arrays"""
    b = np.array([True, False, True], np.bool_)
    ...
    check(assert_type(left_i * b, pd.Index), pd.Index)

I looked at the definition of Index.__mul__() at https://github.com/pandas-dev/pandas-stubs/blob/main/pandas-stubs/core/indexes/base.pyi#L861, mypy correctly matches argument types (Index[Any], ndarray[bool]) to three overloads: 0, 5, and 9 (simplified):

(self: Index[Never], other: <long union>) -> Index[Any]
(self: Index[_str], other: <long union>) -> Never
(self: Index[T_COMPLEX], other: np_ndarray_bool | Index[bool]) -> Index[T_COMPLEX]

Because of the Never return type, the only fallback return type due to Any argument ambiguity we can infer is Any. Why do you expect Index[Any] here? Do you expect some special-casing for Never return type in overloads?

@jorenham

jorenham commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

@jorenham After I merged #22060 situation with scipy-stubs improved, but there are still a bunch of failures in tests/stats/test_multivariate.pyi Could you please check these? If mypy is doing something wrong there, could you please reduce it to a simple standalone repro?

I'll try to look at this later today. But at a glance these all look like cases where the intended fallback overload returns a union that's not assignable to the other overloads, which doesn't really rhyme with the typing spec, so likely scipy-stubs' fault. If you're in a hurry getting this merged there's no need to consider this a blocker, as far as I'm concerned.

@ilevkivskyi

ilevkivskyi commented Oct 1, 2026

Copy link
Copy Markdown
Member Author

@jorenham Thanks!

If you're in a hurry getting this merged there's no need to consider this a blocker, as far as I'm concerned.

No hurry, I just want to make sure I am not missing something.

@cmp0xff

cmp0xff commented Oct 1, 2026

Copy link
Copy Markdown 
(self: Index[Never], other: <long union>) -> Index[Any]
(self: Index[_str], other: <long union>) -> Never
(self: Index[T_COMPLEX], other: np_ndarray_bool | Index[bool]) -> Index[T_COMPLEX]

We want have (Index[Any], other) giving Index[Any], whereas (Index[str], other) should be forbidden.

Currently the first match would give us the desired result (because Index[Never] matches Index[Any]).

Ideally we would like to simply write

(self: Index[T_COMPLEX], other: np_ndarray_bool | Index[bool]) -> Index[T_COMPLEX]
(self: Index[other stuff], other: something else) -> Index[desired type]

and without -> Never.

Upon Index[Any] for self, getting conflict types Index[type 1] and Index[type 2] would ideally give us Index[Any].

Upon Index[str] for self, there is no overload from us, so type checker would say "undefined".

However in many cases other defines a right operation for Index[str] (which fails at run time), and type checkers will use that. In such cases, we have no choice but defining a -> Never for Index[str] from our side.

@ilevkivskyi

ilevkivskyi commented Oct 1, 2026

Copy link
Copy Markdown
Member Author

@cmp0xff

Currently the first match would give us the desired result

I understand, but the current mypy behavior is inconsistent for self-types, if an overload match is caused by Any, mypy should consider all matching overloads, not just the first, and this is what this PR fixes. IIUC this self: Foo[Never] thing is a hack used by several libraries to work-around this mypy bug, after this PR is merged, you should be able to remove those overloads (unless it is needed for some other type checkers).

However in many cases other defines a right operation for Index[str] (which fails at run time), and type checkers will use that. In such cases, we have no choice but defining a -> Never for Index[str] from our side.

OK, I think I understand. You would prefer that mypy infers Never, rather than silently infers an incorrect type from right operation. I think we can simply special-case to ignore declared Never returns when deciding ambiguous overload fallback.

@jorenham

jorenham commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

It looks like there's a similar thing going on with scipy-stubs, which also relies on the "Never-trick". For example in https://github.com/scipy/scipy-stubs/blob/3d76a8e4d5dd2d602351dc0dcf5578ca4b948879/scipy-stubs/spatial/transform/_rotation.pyi#L196 it uses self: Rotation[_JustAnyShape], where _JustAnyShape is a Never-tuple.

I think we can simply special-case to ignore declared Never returns when deciding ambiguous overload fallback.

Until Pyright also fixes this (see microsoft/pyright#10232), these Never-self overloads are going to have to stay I'm afraid. So special-casing Never-ish self types (i.e. uninhabitable ones) here sounds to me like a good way to avoid divergent behavior. It shouldn't be much of a problem I think, because I think it's safe to say that those Never overloads specifically intended for gradual input.

@ilevkivskyi

ilevkivskyi commented Oct 1, 2026

Copy link
Copy Markdown
Member Author

@jorenham I was talking about special-casing a different Never: the Never return as in the pandas-stubs example above. The first overload with Foo[Never] argument shouldn't cause any problems with this PR, as soon as it has a reasonable return. (And it would be much trickier/uglier to special-case.)

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Diff from mypy_primer, showing the effect of this PR on open source code:

colour (https://github.com/colour-science/colour)
- colour/models/rgb/transfer_functions/log.py:240: error: Incompatible types in assignment (expression has type "ndarray[tuple[Any, ...], dtype[floating[_16Bit] | floating[_32Bit] | float64]]", variable has type "ndarray[tuple[Any, ...], dtype[float64]]")  [assignment]
+ colour/appearance/llab.py:411: error: Argument "a" to "CAM_Specification_LLAB" has incompatible type "floating[_16Bit] | floating[_32Bit] | float64"; expected "float | ndarray[tuple[Any, ...], dtype[floating[_16Bit] | floating[_32Bit] | float64]] | None"  [arg-type]
+ colour/appearance/llab.py:412: error: Argument "b" to "CAM_Specification_LLAB" has incompatible type "floating[_16Bit] | floating[_32Bit] | float64"; expected "float | ndarray[tuple[Any, ...], dtype[floating[_16Bit] | floating[_32Bit] | float64]] | None"  [arg-type]
+ colour/appearance/atd95.py:305: error: Argument "A_1" to "CAM_Specification_ATD95" has incompatible type "floating[_16Bit] | floating[_32Bit] | float64"; expected "float | ndarray[tuple[Any, ...], dtype[floating[_16Bit] | floating[_32Bit] | float64]] | None"  [arg-type]
+ colour/appearance/atd95.py:306: error: Argument "T_1" to "CAM_Specification_ATD95" has incompatible type "floating[_16Bit] | floating[_32Bit] | float64"; expected "float | ndarray[tuple[Any, ...], dtype[floating[_16Bit] | floating[_32Bit] | float64]] | None"  [arg-type]
+ colour/appearance/atd95.py:307: error: Argument "D_1" to "CAM_Specification_ATD95" has incompatible type "floating[_16Bit] | floating[_32Bit] | float64"; expected "float | ndarray[tuple[Any, ...], dtype[floating[_16Bit] | floating[_32Bit] | float64]] | None"  [arg-type]
+ colour/appearance/atd95.py:308: error: Argument "A_2" to "CAM_Specification_ATD95" has incompatible type "floating[_16Bit] | floating[_32Bit] | float64"; expected "float | ndarray[tuple[Any, ...], dtype[floating[_16Bit] | floating[_32Bit] | float64]] | None"  [arg-type]
+ colour/appearance/atd95.py:309: error: Argument "T_2" to "CAM_Specification_ATD95" has incompatible type "floating[_16Bit] | floating[_32Bit] | float64"; expected "float | ndarray[tuple[Any, ...], dtype[floating[_16Bit] | floating[_32Bit] | float64]] | None"  [arg-type]
+ colour/appearance/atd95.py:310: error: Argument "D_2" to "CAM_Specification_ATD95" has incompatible type "floating[_16Bit] | floating[_32Bit] | float64"; expected "float | ndarray[tuple[Any, ...], dtype[floating[_16Bit] | floating[_32Bit] | float64]] | None"  [arg-type]
- colour/models/hdr_ipt.py:145: error: Incompatible types in assignment (expression has type "ndarray[tuple[Any, ...], dtype[float64]]", variable has type "float")  [assignment]
- colour/models/hdr_ipt.py:147: error: Incompatible types in assignment (expression has type "ndarray[tuple[Any, ...], dtype[float64]]", variable has type "float")  [assignment]
- colour/models/hdr_ipt.py:149: error: Incompatible return value type (got "float", expected "ndarray[tuple[Any, ...], dtype[floating[_16Bit] | floating[_32Bit] | float64]]")  [return-value]
- colour/models/hdr_cie_lab.py:142: error: Incompatible types in assignment (expression has type "ndarray[tuple[Any, ...], dtype[float64]]", variable has type "float")  [assignment]
- colour/models/hdr_cie_lab.py:144: error: Incompatible types in assignment (expression has type "ndarray[tuple[Any, ...], dtype[float64]]", variable has type "float")  [assignment]
- colour/models/hdr_cie_lab.py:146: error: Incompatible return value type (got "float", expected "ndarray[tuple[Any, ...], dtype[floating[_16Bit] | floating[_32Bit] | float64]]")  [return-value]
- colour/colorimetry/illuminants.py:226: error: Incompatible types in assignment (expression has type "ndarray[tuple[Any, ...], dtype[floating[_16Bit] | floating[_32Bit] | float64]]", variable has type "ndarray[tuple[Any, ...], dtype[float64]]")  [assignment]
- colour/colorimetry/illuminants.py:227: error: Incompatible types in assignment (expression has type "ndarray[tuple[Any, ...], dtype[floating[_16Bit] | floating[_32Bit] | float64]]", variable has type "ndarray[tuple[Any, ...], dtype[float64]]")  [assignment]
+ colour/notation/munsell/centore2014.py:434: error: Incompatible types in assignment (expression has type "ndarray[tuple[Any, ...], dtype[floating[_16Bit] | floating[_32Bit] | float64]]", variable has type "floating[_16Bit] | floating[_32Bit] | float64")  [assignment]
+ colour/notation/munsell/centore2014.py:435: error: Incompatible types in assignment (expression has type "ndarray[tuple[Any, ...], dtype[floating[_16Bit] | floating[_32Bit] | float64]]", variable has type "floating[_16Bit] | floating[_32Bit] | float64")  [assignment]
- colour/notation/munsell/centore2014.py:714: error: Argument 1 to "hue_angle_to_hue" has incompatible type "ndarray[tuple[Any, ...], dtype[floating[Any]]]"; expected "float"  [arg-type]
+ colour/notation/munsell/centore2014.py:714: error: Argument 1 to "hue_angle_to_hue" has incompatible type "ndarray[tuple[Any, ...], dtype[float64]]"; expected "float"  [arg-type]
- colour/notation/munsell/centore2014.py:741: error: Argument 1 to "append" of "list" has incompatible type "ndarray[tuple[Any, ...], dtype[floating[Any]]]"; expected "float"  [arg-type]
+ colour/notation/munsell/centore2014.py:741: error: Argument 1 to "append" of "list" has incompatible type "ndarray[tuple[Any, ...], dtype[float64]]"; expected "float"  [arg-type]
- colour/notation/munsell/centore2014.py:742: error: Argument 1 to "append" of "list" has incompatible type "ndarray[tuple[Any, ...], dtype[floating[Any]]]"; expected "int"  [arg-type]
+ colour/notation/munsell/centore2014.py:742: error: Argument 1 to "append" of "list" has incompatible type "ndarray[Any, dtype[Any]]"; expected "int"  [arg-type]

zulip (https://github.com/zulip/zulip)
- zerver/lib/attachments.py:182: error: "Message" has no attribute "recipient_id"  [attr-defined]
- zerver/actions/user_groups.py:130: error: Argument 1 to "list" has incompatible type "QuerySet[UserProfile, dict[str, Any]]"; expected "Iterable[MemberGroupUserDict]"  [arg-type]
- zerver/actions/user_groups.py:135: error: Argument 1 to "list" has incompatible type "QuerySet[UserProfile, dict[str, Any]]"; expected "Iterable[MemberGroupUserDict]"  [arg-type]
- zerver/actions/user_groups.py:141: error: Argument 1 to "list" has incompatible type "QuerySet[UserProfile, dict[str, Any]]"; expected "Iterable[MemberGroupUserDict]"  [arg-type]
- zerver/actions/user_groups.py:144: error: Argument 1 to "list" has incompatible type "QuerySet[UserProfile, dict[str, Any]]"; expected "Iterable[MemberGroupUserDict]"  [arg-type]

pandas-stubs (https://github.com/pandas-dev/pandas-stubs)
+ tests/test_windowing.py:48: error: Returning Any from function declared to return "float"  [no-any-return]
+ tests/test_resampler.py:39: error: Returning Any from function declared to return "float"  [no-any-return]
+ tests/test_groupby.py:79: error: Returning Any from function declared to return "float"  [no-any-return]
+ tests/series/test_truediv.py:155: error: Expression is of type "Series[float]", not "Never"  [assert-type]
+ tests/series/test_truediv.py:195: error: Expression is of type "Series[float]", not "Never"  [assert-type]
+ tests/series/test_truediv.py:205: error: Expression is of type "Series[float]", not "Never"  [assert-type]
+ tests/series/test_truediv.py:241: error: Expression is of type "Series[float]", not "Never"  [assert-type]
+ tests/series/test_truediv.py:261: error: Expression is of type "Series[float]", not "Never"  [assert-type]
+ tests/series/test_truediv.py:272: error: Expression is of type "Series[float]", not "Never"  [assert-type]
+ tests/series/test_floordiv.py:64: error: Expression is of type "Series[Any]", not "Series[Timedelta]"  [assert-type]
+ tests/series/test_floordiv.py:135: error: Expression is of type "Series[int]", not "Never"  [assert-type]
+ tests/series/test_floordiv.py:171: error: Expression is of type "Series[int]", not "Never"  [assert-type]
+ tests/series/test_floordiv.py:180: error: Expression is of type "Series[Any]", not "Series[Timedelta]"  [assert-type]
+ tests/series/test_floordiv.py:203: error: Expression is of type "Series[int]", not "Never"  [assert-type]
+ tests/series/test_floordiv.py:211: error: Expression is of type "Series[Any]", not "Series[Timedelta]"  [assert-type]
+ tests/series/test_floordiv.py:221: error: Expression is of type "Series[int]", not "Never"  [assert-type]
+ tests/series/test_floordiv.py:230: error: Expression is of type "Series[Any]", not "Series[Timedelta]"  [assert-type]
+ tests/series/test_floordiv.py:277: error: Expression is of type "Series[Any]", not "Series[Timedelta]"  [assert-type]
+ tests/series/test_agg.py:21: error: Expression is of type "Any", not "float"  [assert-type]
+ tests/series/test_agg.py:26: error: Expression is of type "Any", not "float"  [assert-type]
+ tests/series/test_agg.py:31: error: Expression is of type "Any", not "float"  [assert-type]
+ tests/series/test_agg.py:36: error: Expression is of type "Any", not "float"  [assert-type]
+ tests/test_timefuncs.py:1824: error: Expression is of type "Series[Any]", not "Series[Timestamp]"  [assert-type]
+ tests/test_pandas.py:1702: error: Returning Any from function declared to return "float"  [no-any-return]
+ tests/test_pandas.py:2017: error: Returning Any from function declared to return "float"  [no-any-return]
+ tests/frame/test_groupby.py:640: error: Returning Any from function declared to return "float"  [no-any-return]
+ tests/frame/test_frame.py:1154: error: Expression is of type "Any", not "ndarray[tuple[int], dtype[Any]]"  [assert-type]
+ tests/series/test_series.py:682: error: Expression is of type "Any", not "floating[Any]"  [assert-type]
+ tests/series/test_series.py:2185: error: Expression is of type "ndarray[Any, Any]", not "ndarray[tuple[int], dtype[Any]]"  [assert-type]

scikit-learn (https://github.com/scikit-learn/scikit-learn)
- sklearn/linear_model/tests/test_ransac.py:26: error: Argument 1 to "__iadd__" of "ndarray" has incompatible type "ndarray[tuple[Any, ...], dtype[float64]]"; expected "_SupportsArray[dtype[numpy.bool[builtins.bool] | integer[Any]]] | _NestedSequence[_SupportsArray[dtype[numpy.bool[builtins.bool] | integer[Any]]]] | int | _NestedSequence[int]"  [arg-type]
- sklearn/linear_model/tests/test_ransac.py:28: error: Incompatible types in assignment (expression has type "ndarray[tuple[Any, ...], dtype[number[Any, float]]]", variable has type "ndarray[tuple[int], dtype[signedinteger[_32Bit | _64Bit]]]")  [assignment]
- sklearn/linear_model/tests/test_ransac.py:29: error: Incompatible types in assignment (expression has type "ndarray[tuple[Any, ...], dtype[number[Any, float]]]", variable has type "ndarray[tuple[Any, ...], dtype[floating[Any]]]")  [assignment]

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/util/hashing.py:437: error: Unused "type: ignore" comment  [unused-ignore]

xarray (https://github.com/pydata/xarray)
+ xarray/tests/test_typed_ops.py:28: error: Unused "type: ignore" comment  [unused-ignore]
+ xarray/tests/test_typed_ops.py:79: error: Unused "type: ignore" comment  [unused-ignore]

rclip (https://github.com/yurijmikhalevich/rclip)
+ rclip/utils/preprocess.py:44: error: Incompatible return value type (got "ndarray[tuple[Any, ...], dtype[float64]]", expected "ndarray[tuple[Any, ...], dtype[floating[_32Bit]]]")  [return-value]

static-frame (https://github.com/static-frame/static-frame)
+ static_frame/core/rank.py:166: error: Unused "type: ignore" comment  [unused-ignore]

scipy-stubs (https://github.com/scipy/scipy-stubs)
+ tests/spatial/test__rotation.pyi:151: error: Expression is of type "Any", not "float64 | ndarray[tuple[Any, ...], dtype[float64]]"  [assert-type]
+ tests/interpolate/test_interpolate.pyi:43: error: Expression is of type "ndarray[Any, Any]", not "ndarray[tuple[int], dtype[float64]]"  [assert-type]
+ tests/interpolate/test_cubic.pyi:75: error: Expression is of type "ndarray[Any, Any]", not "ndarray[tuple[int], dtype[float64]]"  [assert-type]
+ tests/stats/test_multivariate.pyi:539: error: Expression is of type "Any", not "float64 | ndarray[tuple[Any, ...], dtype[float64]]"  [assert-type]
+ tests/stats/test_multivariate.pyi:564: error: Expression is of type "Any", not "float64 | ndarray[tuple[Any, ...], dtype[float64]]"  [assert-type]
+ tests/stats/test_multivariate.pyi:685: error: Expression is of type "Any", not "float64 | ndarray[tuple[Any, ...], dtype[float64]]"  [assert-type]
+ tests/stats/test_multivariate.pyi:736: error: Expression is of type "ndarray[Any, Any]", not "ndarray[tuple[int, int, *tuple[Any, ...]], dtype[float64]]"  [assert-type]
+ tests/stats/test_multivariate.pyi:746: error: Expression is of type "ndarray[Any, Any]", not "ndarray[tuple[int, int, *tuple[Any, ...]], dtype[float64]]"  [assert-type]
+ tests/stats/test_multivariate.pyi:843: error: Expression is of type "tuple[Any, ...]", not "tuple[float64, float64] | tuple[ndarray[tuple[Any, ...], dtype[float64]], ndarray[tuple[Any, ...], dtype[float64]]]"  [assert-type]
+ tests/stats/test_multivariate.pyi:846: error: Expression is of type "tuple[Any, ...]", not "tuple[float64, float64] | tuple[ndarray[tuple[Any, ...], dtype[float64]], ndarray[tuple[Any, ...], dtype[float64]]]"  [assert-type]
+ tests/stats/test_multivariate.pyi:850: error: Expression is of type "tuple[Any, ...]", not "tuple[float64, float64] | tuple[ndarray[tuple[Any, ...], dtype[float64]], ndarray[tuple[Any, ...], dtype[float64]]]"  [assert-type]
+ tests/stats/test_multivariate.pyi:854: error: Expression is of type "tuple[Any, ...]", not "tuple[float64, float64] | tuple[ndarray[tuple[Any, ...], dtype[float64]], ndarray[tuple[Any, ...], dtype[float64]]]"  [assert-type]
+ tests/stats/test_multivariate.pyi:893: error: Expression is of type "Any", not "float64 | ndarray[tuple[Any, ...], dtype[float64]]"  [assert-type]
+ tests/stats/test_multivariate.pyi:907: error: Expression is of type "Any", not "float64 | ndarray[tuple[Any, ...], dtype[float64]]"  [assert-type]
+ tests/stats/test_multivariate.pyi:923: error: Expression is of type "tuple[Any, ...]", not "tuple[float64, float64] | tuple[ndarray[tuple[Any, ...], dtype[float64]], float64] | tuple[ndarray[tuple[Any, ...], dtype[float64]], ndarray[tuple[Any, ...], dtype[float64]]]"  [assert-type]

@ilevkivskyi

ilevkivskyi commented Oct 2, 2026

Copy link
Copy Markdown
Member Author

OK, there are still some failures in pandas-stubs, mostly in tests/series/test_truediv.py and in tests/series/test_floordiv.py. I looked at Series.__truediv__() and sorry, but it is going too far. I understand using Series[Never] as a simple workaround over mypy bug, but you should not rely on this. In that signature you have two overloads with Series[Never], which are not even first.

Anyway, I think there are just two choices I can give you: with special-casing Never returns (current commit), or without (previous commit). Which one do you prefer?

@cmp0xff

cmp0xff commented Oct 2, 2026

Copy link
Copy Markdown

Thanks. Here's the pandas-stubs side. We use self: Foo[Never] overloads as gradual-only entries:
uninhabited for concrete receivers, matched only when the receiver is Foo[Any]. Three cases.

Index[Any] * bool: an Index[Never] entry returning Index[Any]

class Index(Generic[T]):
    @overload
    def __mul__(self: Index[Never], other: int,  /) -> Index[Any]: ...
    @overload
    def __mul__(self: Index[str],   other: int,  /) -> Never: ...
    @overload
    def __mul__(self: Index[T],     other: bool, /) -> Index[T]: ...

i: Index[Any]
reveal_type(i * True)   # want: Index[Any]

Current commit → Index[Any] (that's the regression we reported)

Without the proposed special-case → Any.

Series[Any] / timedelta: a Series[Never] entry with a deliberate -> Never

class Series(Generic[T]):
    # @overload  # current code: extra Never overload
    # def __truediv__(self: Series[Never],     other: timedelta, /) -> Never: ...
    # @overload
    def __truediv__(self: Series[Timedelta], other: timedelta, /) -> Series[float]: ...

s: Series[Any]
reveal_type(s / timedelta())   # current code: Never

Current commit → Series[float]. We can adapt to this, I guess.

Series[Any].mean(): no -> Never anywhere; the Series[Never] entry returns float,

and it now competes with Series[Timestamp] -> Timestamp

class Series(Generic[T]):
    @overload
    def mean(self: Series[Never], /) -> float: ...
    @overload
    def mean(self: Series[Timestamp], /) -> Timestamp: ...

s: Series[Any]
reveal_type(s.mean())   # want: float

Current commit → Any (median/std/var/quantile/unique/to_numpy behave the same way).

So the special-case restores (1) but not (3). Behavioural change in (2) is acceptable (I guess).

Can we somehow still make (3) work?

@ilevkivskyi

ilevkivskyi commented Oct 2, 2026

Copy link
Copy Markdown
Member Author

@cmp0xff

Can we somehow still make (3) work?

Of course we can, but the real question is whether we should. I started working on this issues (I mean #11347) because:

  1. @jorenham mentioned it as something affecting NumPy (and possibly some related libraries). My understanding was that inferring Any (as mandated by typing spec) is desired.
  2. Current mypy behavior is inconsistent and an obvious bug.

To clarify on the second, mypy is built on some fundamental principles, one of those is what is sometimes called as "gradual guarantee": replacing a precise type with Any should never cause new errors. Of course this is an ideal, and there may be some deviations from this rule. But those are either mypy bugs, or have very good reasons and a well defined scope. What you are asking here is adding an obvious violation of this rule. For example:

def get_mean(s: Series[Timestamp]) -> Timestamp:
    return s.mean()

This code type-checks, but replacing Series[Timestamp] with Series[Any] will cause an error.

So no, I don't think we should make (3) return float.

@jorenham

jorenham commented Oct 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor
  1. inferring Any (as mandated by typing spec)

This was recently changed in the typing spec, so that it now (also) allows types that are assignable to all overloads in case of gradual input: python/typing#2350

So for example ndarray[tuple[Any, ...], np.dtype[Any]] would now also be valid for Any input in case all overloads return -> ndarray

@cmp0xff

cmp0xff commented Oct 2, 2026

Copy link
Copy Markdown

At this point I'd like to invite @Dr-Irv to join the discussion, who will be available in a few days.

@ilevkivskyi

ilevkivskyi commented Oct 2, 2026

Copy link
Copy Markdown
Member Author

@jorenham

This was recently changed in the typing spec, so that it now (also) allows types that are assignable to all overloads in case of gradual input

Sure, I oversimplified, but:

  1. We are already inferring more precise types (when we can) after Find precise fallback for ambiguous Any overloads #22060
  2. This doesn't apply to the specific example in question, i.e. example (3) from pandas-stubs, they specifically ask for float, which is IMO just wrong.

@cmp0xff OK, I can wait couple more days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Overload ambiguity is ignored in self-annotated methods

3 participants

@ilevkivskyi @jorenham @cmp0xff