Skip to content

Commit e920e61

Browse files
address Jelle and Carl's feedback
1 parent d5d2571 commit e920e61

1 file changed

Lines changed: 83 additions & 45 deletions

File tree

‎peps/pep-0835.rst‎

Lines changed: 83 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -37,8 +37,8 @@ However, its verbosity remains a barrier to adoption:
3737
code unreadable, contributing to the PEP's withdrawal [5]_.
3838

3939
Of the 10,000 most downloaded PyPI projects, **43.4%** transitively depend on
40-
``Annotated``. Migrating the FastAPI codebase to the new shorthand removed
41-
2,524 LOCs.
40+
a package utilizing ``typing.Annotated``. Migrating the FastAPI codebase to the
41+
new shorthand removed 2,524 LOCs.
4242

4343
A container-like syntax opposes Python's evolution. Python is actively replacing
4444
generic wrappers like ``typing.Union`` with operators like ``|`` (and the
@@ -52,7 +52,7 @@ postfix operator, it:
5252
- **Reduces tokens** (no ``typing.Annotated`` import).
5353
- **Improves locality** (the base type remains front-and-center).
5454
- **Extends the arithmetic of types** (type metadata acts as an operation on an
55-
existing type expression rather than a generic container).
55+
existing annotation expression rather than a generic container).
5656

5757
This resolves the conflict between concise syntax and strict typing::
5858

@@ -114,12 +114,12 @@ apply:
114114
that evaluates to a valid type, as specified in the `Typing Specification
115115
<https://typing.python.org/en/latest/spec/annotations.html#type-and-annotation-expressions>`__
116116
(:pep:`484`).
117-
- **Type metadata**: e.g., ``int @ <this>``. Data attached to a type expression
118-
via ``Annotated`` (:pep:`593`, :pep:`746`).
117+
- **Type metadata**: e.g., ``int @ <this>``. Data attached to an annotation
118+
expression via ``Annotated`` (:pep:`593`, :pep:`746`).
119119
- **Non-typing annotation**: e.g., `@no_type_check
120120
<https://docs.python.org/3/library/typing.html#typing.no_type_check>`__
121121
wrapping ``x: <this>``. Annotations that are not intended to be evaluated as
122-
type expressions (e.g.: ignored by type checkers).
122+
annotation expressions (e.g.: ignored by type checkers).
123123
- **Symbol decorator**: Applying metadata directly to a variable or field
124124
declaration, rather than to its underlying type. This distinction is
125125
well-established in languages like Java:
@@ -161,57 +161,71 @@ parentheses. For example, a US zip code might be a 5-digit integer *or* a
161161

162162
zip_code: int @ Ge(10000) @ Le(99999) | str @ Len(5)
163163

164-
To attach metadata to the entire union, use parentheses::
164+
To attach metadata to the entire union, or when metadata itself is a union, use
165+
parentheses::
165166

166167
# Attaches only to 'str'
167168
int | str @ Metadata # equivalent to: int | Annotated[str, Metadata]
168169

169170
# Attaches to the entire union
170171
(int | str) @ Metadata # equivalent to: Annotated[int | str, Metadata]
171172

173+
# Metadata is a union
174+
int @ (str | float) # equivalent to: Annotated[int, str | float]
172175

173176

174177
Flattening Multiple Metadata
175178
----------------------------
176179

177-
Chained metadata flattens the resulting ``Annotated`` object. ``T @ m1 @ m2``
180+
Because ``@`` is left-associative, ``T @ m1 @ m2`` parses as ``(T @ m1) @ m2``.
181+
Chained metadata flattens the resulting ``Annotated`` object: ``T @ m1 @ m2``
178182
evaluates to ``Annotated[T, m1, m2]``, never
179183
``Annotated[Annotated[T, m1], m2]``. This mirrors ``typing.Annotated``'s
180184
existing runtime behavior.
181185

182186
This same logic applies when the left-hand operand is an existing ``Annotated``
183-
type::
187+
type, whereas parenthesizing the right-hand side evaluates ``m1 @ m2`` first as
188+
a single metadata element::
184189

185-
Annotated[int, m1] @ m2 # AnnotatedType(int, m1, m2) (flattened)
190+
Annotated[int, m1] @ m2 # Annotated[int, m1, m2] (flattened, two metadata elements)
191+
T @ (m1 @ m2) # Annotated[T, m1 @ m2] (single metadata element)
186192

187193
Runtime Behavior
188194
----------------
189195

190-
``@`` produces a ``types.AnnotatedType`` (a new built-in C type). The existing
191-
``typing.Annotated`` unifies with this type. ``typing.Annotated[X, Y]`` and ``X
196+
``@`` produces a ``typing.Annotated`` instance (now implemented as a built-in C
197+
type, similar to ``typing.TypeAliasType``). ``typing.Annotated[X, Y]`` and ``X
192198
@ Y`` return the exact same object:
193199

194200
.. code-block:: pycon
195201
196202
>>> type(int @ Field()) is type(Annotated[int, Field()])
197203
True
198-
>>> typing.Annotated is types.AnnotatedType
204+
>>> isinstance(int @ Field(), typing.Annotated)
199205
True
200206
201-
An ``AnnotatedType`` object exposes the following attributes:
207+
An ``Annotated`` object exposes the following attributes:
202208

203209
- ``__origin__``: The base type (e.g., ``int``).
204210
- ``__metadata__``: A tuple of metadata items.
205211
- ``__args__``: The tuple ``(origin, *metadata)``, for compatibility with
206212
``typing.get_args()``.
207-
- ``__parameters__``: A tuple of unique free type parameters of the type.
213+
- ``__parameters__``: A tuple of unique free type parameters in ``__origin__``.
208214

209-
The ``repr()`` of an ``AnnotatedType`` uses the shorthand syntax:
215+
The ``repr()`` of an ``Annotated`` object uses the shorthand syntax. Because
216+
metadata may be an arbitrary object whose ``repr()`` contains lower- or
217+
equal-precedence operators (such as ``str | float``, ``str @ 'a'``, or custom
218+
user metadata), non-string metadata items are parenthesized on the right-hand
219+
side (see `Open Issues`_):
210220

211221
.. code-block:: pycon
212222
223+
>>> int @ "positive"
224+
int @ 'positive'
225+
>>> int @ (str | float)
226+
int @ (str | float)
213227
>>> int @ Field(gt=0)
214-
int @ Field(gt=0)
228+
int @ (Field(gt=0))
215229
216230
Handling of ``None``
217231
--------------------
@@ -229,32 +243,36 @@ multiplication:
229243
return a @ b # oops, forgot to check for None
230244
231245
If ``NoneType.__matmul__`` existed, this would silently return an
232-
``AnnotatedType`` instead of raising a ``TypeError``.
246+
``Annotated`` type instead of raising a ``TypeError``.
233247

234248
Because ``NoneType`` lacks ``__matmul__``, expressions like ``None @ Metadata``
235249
will raise a ``TypeError`` at runtime.
236250

237251
To support this safely at runtime, library authors are encouraged to implement
238252
``__rmatmul__`` on their metadata objects. This provides a robust fallback for
239-
types that do not invoke ``__matmul__`` on the left operand (such as
240-
``NoneType`` or ``ForwardRef``).
253+
types that do not natively invoke ``__matmul__`` on the left operand (such as
254+
``NoneType``). (Note: while ``typing.ForwardRef`` implements ``__matmul__``,
255+
``annotationlib`` does not invoke it; it evaluates strings using proxy types
256+
and then replaces them with ``ForwardRef`` instances in a second pass).
241257

242258
Supported Left-Hand Operands
243259
-----------------------------
244260

245-
The ``@`` operator adds ``nb_matrix_multiply`` to ``type`` and to all typing
246-
constructs that support the ``|`` union operator (``types.GenericAlias``,
247-
``types.UnionType``, ``types.AnnotatedType``, ``typing.TypeVar``,
248-
``typing.ParamSpec``, ``typing.TypeVarTuple``, ``typing.TypeAliasType``,
249-
``typing.ForwardRef``, and ``sentinel`` objects).
261+
The ``@`` operator adds ``__matmul__`` to ``type`` and to typing constructs
262+
that can be wrapped by ``Annotated`` (``types.GenericAlias``,
263+
``types.UnionType``, ``typing.Annotated``, ``typing.TypeVar``,
264+
``typing.ParamSpec``, ``typing.ParamSpecArgs``, ``typing.ParamSpecKwargs``,
265+
``typing.TypeVarTuple``, ``typing.TypeAliasType``, ``typing.ForwardRef``,
266+
``typing.NewType``, ``typing._SpecialForm``, ``typing._BaseGenericAlias``,
267+
``dataclasses.InitVar``, and ``sentinel`` objects).
250268

251269
Applying ``@`` to a class evaluates as type metadata; applying it to an instance
252270
performs arithmetic.
253271

254-
For example, ``int @ Field()`` produces an ``AnnotatedType``, while ``42 @
272+
For example, ``int @ Field()`` produces a ``typing.Annotated`` type, while ``42 @
255273
something`` raises a ``TypeError`` (or delegates to ``__rmatmul__``).
256274

257-
Likewise, ``ndarray @ Field()`` produces an ``AnnotatedType``, even though
275+
Likewise, ``ndarray @ Field()`` produces a ``typing.Annotated`` type, even though
258276
``ndarray`` instances define ``__matmul__`` for matrix multiplication.
259277

260278
Custom metaclasses can overload ``__matmul__`` provided ``@`` is avoided in type
@@ -278,7 +296,7 @@ modification.
278296
Parentheses and Verbosity
279297
-------------------------
280298

281-
Writing ``address: (str | None) @Field(...)`` adds verbosity due to the lack
299+
Writing ``address: (str | None) @ Field(...)`` adds verbosity due to the lack
282300
of native symbol decorators. Frameworks co-opt ``Annotated`` to configure
283301
fields, even though the developer's intent is to configure the ``address``
284302
field, not the ``str | None`` type. While the ``@`` shorthand cannot fully
@@ -325,8 +343,9 @@ Analyzing the top 10,000 PyPI projects revealed 0 metaclass overloads for
325343

326344

327345
The pure-Python ``typing._AnnotatedAlias`` class is replaced with a native C
328-
implementation (``types.AnnotatedType``). ``typing.Annotated`` becomes a
329-
reference to this C type rather than a special form with a custom metaclass.
346+
implementation of ``typing.Annotated``. ``typing.Annotated`` becomes a built-in
347+
C type (similar to ``typing.TypeAliasType``) rather than a special form with a
348+
custom metaclass.
330349

331350
To ensure a smooth transition, this legacy class is retained as a deprecated
332351
compatibility shim. Code using ``isinstance(x, typing._AnnotatedAlias)`` will
@@ -336,9 +355,9 @@ removal in Python 3.21 (see `Open Issues`_).
336355
Code that should be updated:
337356

338357
- ``type(ann).__name__ == '_AnnotatedAlias'`` → use ``isinstance(ann,
339-
types.AnnotatedType)`` or ``typing.get_origin(ann) is Annotated``
358+
typing.Annotated)`` or ``typing.get_origin(ann) is Annotated``
340359
- ``typing._AnnotatedAlias(origin, metadata)`` → use ``Annotated[origin,
341-
*metadata]`` or ``origin @m1 @m2``
360+
*metadata]`` or ``origin @ m1 @ m2``
342361

343362
**Backporting via typing_extensions:** Like ``X | Y``, the ``@`` shorthand
344363
requires changes to the metatype (``type.__matmul__``), which cannot be patched
@@ -465,8 +484,7 @@ Prototype implementations are available for the following tools:
465484
<https://github.com/till-varoquaux/ruff/compare/main...till-varoquaux:ruff:feature/at-type-annot>`_
466485

467486
``builtins.type`` in Typeshed will be updated to include
468-
``def __matmul__(self, other: Any) -> types.AnnotatedType: ...`` to support type
469-
checkers.
487+
``def __matmul__(self, other: Any) -> Self: ...`` to support type checkers.
470488

471489
Rejected Ideas
472490
==============
@@ -497,10 +515,11 @@ We explicitly reject relying *solely* on metadata objects implementing
497515
...
498516

499517
This approach is rejected for two reasons. First, relying on arbitrary
500-
right-hand objects to implement ``__rmatmul__`` breaks the type expression
501-
model, complicating the assignment of fixed static meanings to operators. This
502-
is inconsistent with how ``|`` works and contradicts the type expression grammar
503-
defined in the specification. Second, :pep:`593` explicitly permits any valid
518+
right-hand objects to implement ``__rmatmul__`` breaks the annotation
519+
expression model, complicating the assignment of fixed static meanings to
520+
operators. This is inconsistent with how ``|`` works and contradicts the
521+
annotation expression grammar defined in the specification. Second, :pep:`593`
522+
explicitly permits any valid
504523
Python object as metadata (e.g., strings, dicts). Implementing ``__matmul__``
505524
directly on the ``type`` metaclass avoids opt-in base classes and provides
506525
consistent behavior.
@@ -510,14 +529,14 @@ Parameter/Field Decorators
510529

511530
True parameter/field decorators were rejected for now. Modifying the core parser
512531
for symbol-level decorators opens a complex design space; this PEP scopes the
513-
``@`` operator to type expressions.
532+
``@`` operator to annotation expressions.
514533

515534
Avoiding @ Due to Matrix Multiplication
516535
---------------------------------------
517536

518537
We rejected the argument that ``@`` should be avoided because of its association
519-
with matrix multiplication. Within a type expression, Python already reuses
520-
standard operators (like ``|`` for unions and ``[]`` for generics) with
538+
with matrix multiplication. Within an annotation expression, Python already
539+
reuses standard operators (like ``|`` for unions and ``[]`` for generics) with
521540
typing-specific semantics (see `Why the @ Operator?`_).
522541

523542
Structural Evaluation Format (``Format.TYPE``)
@@ -531,6 +550,15 @@ This was rejected. Prototype integrations (``beartype``, ``pydantic``,
531550
``fastapi``, ``sqlalchemy``, and ``hypothesis``) showed the existing ecosystem
532551
handles the ``@`` operator using ``annotationlib``'s current tools.
533552

553+
However, we acknowledge a tradeoff: evaluating undefined annotations with
554+
``Format.FORWARDREF`` now produces an opaque ``ForwardRef("undefined @
555+
'metadata'")``. This makes it harder to extract metadata than with
556+
``Annotated`` (where ``get_args()`` yields ``(ForwardRef("undefined"),
557+
'metadata')``). Despite this regression, we found no breakage in our prototype
558+
migrations, indicating that runtime tools generally depend on successful type
559+
resolution or structural evaluation rather than raw forward reference strings.
560+
This is further mitigated by adding ``__rmatmul__`` on annotation classes.
561+
534562
No-Space Formatting (``@annot``)
535563
--------------------------------
536564

@@ -567,8 +595,8 @@ This proposal aligns with the metadata model. Type-directed libraries typically
567595
inspect static definitions during class creation rather than relying on
568596
standalone descriptors.
569597

570-
``@Field(...) id: int`` would evaluate identically to ``id: int
571-
@Field(...)``. This allows existing frameworks to inspect field configurations
598+
``@Field(...) id: int`` would evaluate identically to ``id: int @
599+
Field(...)``. This allows existing frameworks to inspect field configurations
572600
and continue working unmodified. Under this model, value-space decorators modify
573601
objects at runtime, while type-space decorators attach metadata to a type.
574602

@@ -597,15 +625,25 @@ future extensions like annotation targets.
597625
Open Issues
598626
===========
599627

628+
**Parenthesization in ``repr()``:** Because metadata can be any arbitrary object
629+
with a custom ``__repr__``, formatting ``Annotated`` as ``T @ M`` without
630+
parentheses can produce misleading output when ``repr(M)`` contains lower- or
631+
equal-precedence operators (e.g., ``Annotated[int, str | float]`` or
632+
``int @ (str @ 'a')``). The reference implementation currently parenthesizes all
633+
non-string metadata on the right-hand side (e.g., ``int @ (str | float)``,
634+
``int @ (Field(gt=0))``) while leaving string literals unparenthesized
635+
(``int @ 'meta'``). Should ``repr()`` wrap all non-string metadata, or use
636+
another heuristic to omit parentheses when safe?
637+
600638
**Deprecation Timeline:** As a private class, ``typing._AnnotatedAlias`` could
601639
bypass the standard 5-year deprecation policy (:pep:`387`). Should we
602640
fast-track its removal?
603641

604642
Acknowledgements
605643
================
606644

607-
Thanks to Hugo van Kemenade, Jelle Zijlstra, and Eric Traut for their feedback,
608-
guidance, and assistance in refining this proposal.
645+
Thanks to Hugo van Kemenade, Jelle Zijlstra, Carl Meyer, and Eric Traut for their
646+
feedback, guidance, and assistance in refining this proposal.
609647

610648
References
611649
==========

0 commit comments

Comments
 (0)