Ban negative extras (extra != markers)

[konstin]

Negative extras (extra !=) are currently technically allowed in the grammar, but they behave strangely: extra != means that package could transitively remove a package from selection through activating an extra. This turns incremental resolution into a global problem: We can't exit a resolution with a conflict when we've exhausted all versions in range including backtracking, we also have to exhaustively search for any package version that may activate the extra which deactivates the extra != requirement which is part of the conflict. This would be fatal to resolver complexity and performance, preventing most of the short-circuiting that makes resolving tractable.

Much weirder, with negative extras, adding a new dependency can make a previously impossible resolution solvable:

[project]
name = "foo"
version = "0.1.0"
dependencies = [
  "numpy >2",
  "numpy <2; extra != 'old-science'",
]

Only by adding a package with Requires-Dist: foo[old-science], this resolves.

Using PEP 621, users only get positive extras: project.optional-dependencies.foo gets translated into a conjunction of extra == "foo" with each requirement's marker.

While this may look like a breaking change, it only clarifies the current state of support in tools. From pip not supporting it and uv not supporting it, we know it is not currently used and safe to remove. This solves the confusion for users who want to use negative extras for conflicting packages.

---

📚 Documentation preview 📚: https://python-packaging-user-guide--2085.org.readthedocs.build/en/2085/

[konstin]

DPO thread: https://discuss.python.org/t/ban-negative-extras/108022

[ncoghlan]

Do we also need to amend the "Set of Strings" definition? If extra != "name" doesn't make sense, do "name" not in extras and "name" not in dependency_groups actually make sense?

[ncoghlan]

The required change before approval is the history entry.

The proposed wording tweak is genuinely just a suggestion, and the "Set of Strings" question could be resolved in a follow-up PR if we're not sure the exact same concern applies.

[ncoghlan]

History entry is needed for the change

[ncoghlan]

Perhaps explicitly call out "... other comparison operations on extra (including !=) are not`" so it's clear the omission from the earlier sentence isn't accidental?

[dimbleby]

@radoering

[radoering]

@dimbleby Thanks for the notification.

While this may look like a breaking change, it only clarifies the current state of support in tools.

If "tools" is only pip and uv, you are probably right. However, I think it is kind of breaking for Poetry, see e.g. https://python-poetry.org/docs/dependency-specification/#exclusive-extras

I mean, Poetry can still document that this is not allowed anymore by the standard and should only be used in Poetry-only projects.

[notatallshaw]

If "tools" is only pip and uv, you are probably right. However, I think it is kind of breaking for Poetry, see e.g. https://python-poetry.org/docs/dependency-specification/#exclusive-extras

Hi @radoering, I checked Poetry's dependency selection behavior and found it is almost identical to uv's: it evaluates negative extras (extra != ...) as always false in dependency metadata. There is already an open issue for this: python-poetry/poetry#10120.

So I would consider that Poetry also doesn't support negative extras in dependency metadata.

Where Poetry does honor a negative extra is when it installs the root project itself (poetry install keeps a ; extra != 'cuda' dependency, poetry install --extras cuda drops it). It does not carry them any further than that: poetry-core drops negative-extra dependencies at build time, so a Poetry-built wheel never publishes them. I'm not sure the root-project behavior is a concern for interoperability standards, since we don't have any standards governing how a tool resolves its own project, but perhaps we need to be clear on the language here?

So banning extra != in dependency metadata wouldn't break anything Poetry publishes.

Independently, I'd strongly recommend Poetry document the limitation that the pattern only works for the top-level project: poetry-core drops the dependency from built wheels, and Poetry evaluates the marker as false when it appears in a dependency's metadata.

FYI here's an MRE for my findings: https://gist.github.com/notatallshaw/b0ec3db17c7647e3e9e8db7d9b6288a3

I mean, Poetry can still document that this is not allowed anymore by the standard and should only be used in Poetry-only projects.

My findings indicate that this is effectively the case already, just undocumented.

[radoering]

@notatallshaw Thanks for the analysis.

I checked why poetry-core ignores dependencies with negative extras. Actually, that does not seem to be intentionally but is just a side effect of an unrelated bugfix. We are probably lucky now that python-poetry/poetry#10120 has not been fixed yet. Same for ignoring negative extras from wheels.

I mean, Poetry can still document that this is not allowed anymore by the standard and should only be used in Poetry-only projects.

My findings indicate that this is effectively the case already, just undocumented.

You are right. Due to the bugs, the impact of banning negative extras is less severe than I thought. It is probably sufficient to update Poetry's documentation and pay attention not to fix the bugs in a way that ignores the ban.

[notatallshaw]

Actually, that does not seem to be intentionally but is just a side effect of an unrelated bugfix. We are probably lucky now that python-poetry/poetry#10120 has not been fixed yet. Same

Ah I see, but even if it was fixed and the marker evaluation honored extra != ... I suspect it would still lead to resolver inconsistencies that I mention here: https://discuss.python.org/t/ban-negative-extras-for-extras-marker/108022/4 (I have my own experimental pubgrub style resolver that I am working on so I am trying to pin this down).

[ncoghlan]

From https://discuss.python.org/t/ban-negative-extras-for-extras-marker/108022/18, it sounds like the intended use case for != support in poetry is for the tool-specific "conflicting extras" case. As described in the post, this is the same reason that not in support actually makes sense in lock files for both extras and dependency groups.

The guidance we give for attempts to use the "Set of Strings" fields in public dependency declarations is:

Publishing tools SHOULD emit an error if projects attempt to reference the extras or dependency_groups fields in their published dependency declaration metadata, and index servers SHOULD NOT accept uploads referencing these fields.

The actual comparison operations themselves are left defined as normal rather than giving tools permission to treat them as False without evaluation.

This isn't the effect we get from undefining extra != "name" entirely. Undefining the comparison instead drops us into this clause (which is weaker than we want for index servers, and stronger than we want for locking and installation tools):

Other comparison operations on extra are not defined and publishing tools SHOULD emit an error, index servers MAY disallow uploads containing such environment markers, while locking and installation tools SHOULD evaluate them as False.

Accordingly, I think the end result will be clearer if we add an explicit clause for negative extra markers along the lines of:

Excluding dependencies with extra != "name" is only considered valid in already resolved dependency trees, so publishing tools SHOULD emit an error for such comparisons, index servers SHOULD disallow uploads containing them, and locking and installation tools MAY evaluate them as False.

(Having that explicit clause there should also help remind us that even if we someday allow "name" in extras on PyPI, we should still disallow "name" not in extras)