Skip to content

Fully type _operator module - #16014

Merged
srittau merged 29 commits into
python:mainfrom
jonathandung:patch-5
Nov 15, 2026
Merged

srittau merged 29 commits into
python:mainfrom
jonathandung:patch-5

Conversation

@jonathandung

@jonathandung jonathandung commented Jul 15, 2026 •

Copy link
Copy Markdown
Contributor

Closes #15611.

@github-actions

This comment has been minimized.

@JelleZijlstra

Copy link
Copy Markdown
Member

This is a lot of changes and the mypy-primer output is not clearly positive. These changes may be correct in principle but it's not clear they all help users. Could you make smaller PRs and if mypy-primer shows any changes, write a comment showing that the change is positive?

@OutSquareCapital

OutSquareCapital commented Aug 15, 2026 •

Copy link
Copy Markdown

I struggle to see why they couldn't help users.

As of today the use of operator module is encouraged by populars linters such as Ruff when it can replace lambdas, but some basic function are not typed, defeating all their purpose on strict type checking

@jonathandung

Copy link
Copy Markdown
Contributor Author

write a comment showing that the change is positive?

Sorry, but I'm not sure what you mean. Besides, some of my changes are interdependent. I think I can still split them up, it would just be a Herculean task.

@github-actions

This comment has been minimized.

@JelleZijlstra

Copy link
Copy Markdown
Member

mypy-primer shows that this change will introduce new type errors into a number of projects. Is that a good thing?

@jonathandung

jonathandung commented Aug 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Certainly not, but as I said in the previous iteration of this PR, I am unable to understand most of those errors and have little idea how to fix them. It would be much appreciated if I could get some guidance on this.

@github-actions

This comment has been minimized.

@hunterhogan

Copy link
Copy Markdown
Contributor

I use operator all the time. I accidentally incited the de-typing of the operator module with a narrow PR. I was happy to see @jonathandung undertake the huge task of re-typing the module.

We are choosing to not merge because @jonathandung "only" knows how to write good stubs and doesn't know how to navigate the illogical politics of the Python ecosystem. Full stop. There are no other reasons for not merging.

There are many effects of not merging, but most of those effects are invisible to the Pythonati, so they are ignored.

If we know the merge will affect pandas and scipy and we don't want to ruffle their feathers, then use the automated system that identifies the "victims" of this merge to automatically open Issues in the repos of the victims warning them of the impending destruction of their fragile project due to the marge of patch-5 by @jonathandung.

@github-actions

This comment has been minimized.

@srittau srittau left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this change is worth it overall, since it finally adds types to this module. Most primer hits are situations where multiple operators are used, for example like this:

dict = {"+": add, "-": sub}

# or

if is_bool:
    op = operator.xor
else:
    op = operator.sub

This will need explicit annotations. Not great, but also not terrible.

One genuine problem below, though.

Comment thread stdlib/_operator.pyi Outdated
@overload
def iand(a: _T_contra, b: _SupportsRAnd[_T_contra, _T_co], /) -> _T_co: ...

def iconcat(a: MutableSequence[_T], b: Sequence[_T], /) -> MutableSequence[_T]: ... # currently impossible to type more precisely

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The kornia primer hit shows a problem with this. I think we should try the following:

_R_co = TypeVar("_R_co", contravariant=True)

class _SupportsIConcat(Protocol[_T_contra, _R_co]):
    def __iadd__(self, value: _T_contra, /) -> _R_co: ...
    def __getitem__(self, key: Any) -> Any: ...  # method needs only to exists

# Currently impossible to type more precisely.
# Ideally this would be typed as (a: _IConcatList[_T], b: _T) -> _IConcatList[_T].
def iconcat(a: _SupportsIConcat[_T, _R], b: _T, /) -> _R: ... 

(Move the protocol and type var up.)

Comment thread stdlib/_operator.pyi Resolved
@github-actions

This comment has been minimized.

Co-authored-by: Sebastian Rittau <sebastian.rittau@zfutura.de>
@github-actions

Copy link
Copy Markdown
Contributor

Diff from mypy_primer, showing the effect of this PR on open source code:

colour (https://github.com/colour-science/colour)
+ colour/io/luts/lut.py:642: error: Cannot call function of unknown type  [operator]
+ colour/continuous/signal.py:1203: error: Cannot call function of unknown type  [operator]
+ colour/continuous/signal.py:1207: error: Cannot call function of unknown type  [operator]
+ colour/continuous/multi_signals.py:1531: error: Cannot call function of unknown type  [operator]
+ colour/continuous/multi_signals.py:1535: error: Cannot call function of unknown type  [operator]

core (https://github.com/home-assistant/core)
+ homeassistant/components/python_script/helpers.py:127: error: Cannot call function of unknown type  [operator]

pandas (https://github.com/pandas-dev/pandas)
- pandas/core/roperator.py:56: error: Call to untyped function "and_" in typed context  [no-untyped-call]
- pandas/core/roperator.py:60: error: Call to untyped function "or_" in typed context  [no-untyped-call]
- pandas/core/roperator.py:64: error: Call to untyped function "xor" in typed context  [no-untyped-call]
+ pandas/core/computation/ops.py:346: error: Need type annotation for "_binary_ops_dict" (hint: "_binary_ops_dict: dict[<type>, <type>] = ...")  [var-annotated]
+ pandas/core/algorithms.py:1617: error: Incompatible types in assignment (expression has type overloaded function, variable has type overloaded function)  [assignment]
+ pandas/core/indexes/datetimes.py:2515: error: Incompatible types in assignment (expression has type overloaded function, variable has type overloaded function)  [assignment]

egglog-python (https://github.com/egraphs-good/egglog-python)
+ python/tests/param_eq/evaluation.py:15: error: Dict entry 4 has incompatible type "type[Pow]": overloaded function; expected "type[operator]": "Callable[[float, float], float]"  [dict-item]

ibis (https://github.com/ibis-project/ibis)
+ ibis/util.py:268: error: Incompatible types in assignment (expression has type "overloaded function | overloaded function", variable has type overloaded function)  [assignment]
+ ibis/expr/api.py:746: error: Argument 1 to "reduce" has incompatible type overloaded function; expected "Callable[[BooleanValue | bool, BooleanValue | bool], BooleanValue | bool]"  [arg-type]
+ ibis/expr/api.py:766: error: Argument 1 to "reduce" has incompatible type overloaded function; expected "Callable[[BooleanValue | bool, BooleanValue | bool], BooleanValue | bool]"  [arg-type]

@srittau
srittau merged commit 4d46730 into python:main Nov 15, 2026
88 checks passed
@jonathandung
jonathandung deleted the patch-5 branch November 15, 2026 00:34
@hunterhogan

Copy link
Copy Markdown
Contributor

Thank you to everyone.

@srittau, I hope I have told you before: I aspire to have your temperament. We are all blessed every time you gift us with your hard work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standard library _operator module is poorly typed

6 participants