Free-threading: GC untracks immortal objects, violating the always-tracked dict invariant asserted by dict_merge_api (debug abort at shutdown)

[jonasvq]

Bug report

Summary

On the free-threaded build, the cyclic GC permanently untracks any immortal object it
encounters during collection. Free-threaded dicts, however, are expected to stay GC-tracked:
dict_merge_api() asserts _PyObject_GC_IS_TRACKED(a) after every C-API dict merge. As a
result, once a GC-tracked dict is made immortal and one collection runs, the next C-API
PyDict_Update()/PyDict_Merge() on it aborts a debug build. If the dict is the builtins
module dict, the abort is deterministic at interpreter shutdown (the builtins-restore merge in
finalization).

The two sides each enforce a reasonable invariant; they cannot both hold for an immortal
tracked dict. This is the same fragility class as gh-142975 (frozen objects vs. the FT GC's
refcount validation), one layer over: immortal objects vs. the FT GC's untracking policy.

Reproducer

There is no pure-Python way to create a tracked immortal dict (_Py_SetImmortal() and
PyUnstable_SetImmortal() untrack first), so the reproducer uses a 15-line extension around
the internal _Py_SetImmortalUntracked():

/* _immortal.c */
#include <Python.h>
extern void _Py_SetImmortalUntracked(PyObject *op);

static PyObject *
immortalize_tracked(PyObject *self, PyObject *obj)
{
    _Py_SetImmortalUntracked(obj);   /* immortal, still GC-tracked */
    Py_RETURN_NONE;
}
static PyMethodDef methods[] = {
    {"immortalize_tracked", immortalize_tracked, METH_O, NULL}, {NULL}};
static struct PyModuleDef mod = {PyModuleDef_HEAD_INIT, "_immortal", NULL, -1, methods};
PyMODINIT_FUNC PyInit__immortal(void) { return PyModule_Create(&mod); }

# repro.py  (run with PYTHON_GIL=0; the helper extension predates Py_mod_gil)
import _immortal, gc, builtins

b = vars(builtins)
_immortal.immortalize_tracked(b)     # immortal AND still tracked
print("tracked before collect:", gc.is_tracked(b))   # True
gc.collect()                          # FT GC untracks immortal objects
print("tracked after collect:", gc.is_tracked(b))    # False
# interpreter shutdown -> builtins restore does PyDict_Update() -> abort

Observed output:

tracked before collect: True
tracked after collect: False
python: Objects/dictobject.c:4374: dict_merge_api: Assertion `_PyObject_GC_IS_TRACKED(a)' failed.
Aborted (core dumped)

Any tracked immortal dict shows the state corruption (gc.is_tracked() flips to False after
one collection); the abort fires on the first C-API merge into such a dict — the
Python-level dict.update() method does not abort because dict_update_common() calls
dict_merge() directly, bypassing dict_merge_api() and its assert, and nothing on that path
re-tracks the dict either.

The three code points

1. The GC untracks immortals — Python/gc_free_threading.c, update_refs() (line ~982 at
   ec5f154):

   // Exclude immortal objects from garbage collection
   if (_Py_IsImmortal(op)) {
       op->ob_tid = 0;
       _PyObject_GC_UNTRACK(op);
       gc_clear_unreachable(op);
       return true;
   }

2. Dicts must stay tracked — Objects/dictobject.c:4374:

   int res = dict_merge(a, b, override, dupkey);
   assert(_PyObject_GC_IS_TRACKED(a));

3. Shutdown makes it deterministic — interpreter finalization restores the builtins
   dict via a C-API merge, so an immortal builtins dict aborts every clean exit.

Why the invariants conflict (and a failed naive fix)

I tried the obvious one-line fix — skip immortal objects in update_refs() without
untracking them. That trades this abort for a different one: the tracked immortal objects then
flow into later GC phases and trip validate_gc_objects (gc_free_threading.c:1119,
"unmerged objects should have ob_tid != 0" class assertions), and under thread churn I also
saw Objects/listobject.c:82: ensure_shared_on_resize asserts. So there are (at least) three
mutually inconsistent expectations:

• the FT GC wants immortal objects out of its heaps entirely (untracked, skipped by every
  phase — untracking is how it guarantees "never scan them again");
• free-threaded dict code wants dict instances always tracked (dict_merge_api asserts it);
• immortalization of a tracked container produces exactly the state neither side can handle.

Resolving this likely needs a decision about which invariant gives way:
(a) the GC skips immortals without untracking and every later phase learns to skip them too;
(b) the GC exempts types that rely on always-tracked (or re-tracks on demand);
(c) the dict assert is relaxed to tracked || immortal — only safe if the tracked bit is not
load-bearing for the shared-dict thread-safety paths, which needs an assessment by the
free-threading owners.

Reachability / impact

Today a tracked immortal dict can only be produced via internal APIs
(_Py_SetImmortalUntracked() is PyAPI_FUNC and extensions do reach for the immortalization
internals). Anything that immortalizes live object graphs — per-interpreter object sharing,
pre-fork copy-on-write immortalization experiments, or a future public immortalization API —
will produce them in bulk, so the conflict is worth resolving before more machinery is built
on top. On release builds the assert is compiled out and the visible symptom is only the
silently flipped tracked bit; whether an untracked dict skips any FT thread-safety path that
keys off the tracked bit is part of the assessment needed here.

Environment

• CPython main @ ec5f154 (3.16.0a0), ./configure --disable-gil --with-pydebug
• Linux x86_64 (WSL2 kernel 6.18, Ubuntu 24.04), gcc 13.3.0
• Not bisected; the update_refs() untrack and the dict_merge_api assert both predate this
  report, so 3.14t is presumably affected as well (untested).

[picnixz]

I'm not sure this is a legitimate usage though. Is it possible to end up with such dictionaries in general? if it's not possible to do it using pure python code I would say the extension module is then buggy (?)

cc @colesbury

[jonasvq]

Fair question — pure Python can't produce this state. But the helper extension isn't
load-bearing: the same abort is reachable via PyUnstable_SetImmortal() alone, and on the
default debug build too, not just free-threading:

PyObject *d = PyDict_New();
PyDict_SetItemString(d, "k", Py_None);
int ok = PyUnstable_SetImmortal(d);   /* returns 1 (success), untracks d */
...
PyDict_Update(d, other);              /* aborts any --with-pydebug build */

Verified on an unpatched default build (./configure --with-pydebug, main @ ec5f154):

GIL enabled: True
PyUnstable_SetImmortal returned: 1
tracked before: True  tracked after: False
python: Objects/dictobject.c:4374: dict_merge_api: Assertion `_PyObject_GC_IS_TRACKED(a)' failed.

(The dict is created inside C so it is uniquely referenced at the call —
PyUnstable_SetImmortal refuses objects passed in from Python because of the extra
references from the call machinery. Python-level dict.update() bypasses
dict_merge_api(), so only C-API merges hit the assert; nothing ever re-tracks the dict.
The FT GC untrack in update_refs() is a second, independent way to reach the same state.)

So the question becomes: is immortalizing a GC container supported?

• If yes: the untrack paths need reconciling with the dict invariant (re-track in the C-API
  merge, or relax the assert to tracked || immortal — only safe if the tracked bit isn't
  load-bearing for the FT shared-dict paths).
• If no: PyUnstable_SetImmortal() should refuse GC containers the way it already refuses
  strings, and the docs should say so; today it returns success.

Happy to test either direction against the reproducers here.

[picnixz]

Docs say that GC-tracked objects are supported:

Immortal objects do not participate in reference counting and will never be garbage collected. If the object is GC-tracked, it is untracked.

So, I would say it should be expected. What about non-dicts? like lists, or even custom GC objects? if GC-tracked custom objects cannot be made immortal for that reason, I think there isn't lots of applications.

@ngoldbaum WDYT?

[jonasvq]

Checked exactly that: dict is the only affected container.
Objects/dictobject.c:4374 is the sole assert(_PyObject_GC_IS_TRACKED(...)) in all of
Objects/ — a post-dict_merge sanity check dating from when dict untracking was removed
("dicts are always tracked"). Immortalization is now the only remaining untrack path, so the
assumption is stale.

Lists, sets, and custom GC objects have no such tracked-assumption. Verified on the same
unpatched --with-pydebug build (main @ ec5f154): an immortalized list and set — untracked,
PyUnstable_SetImmortal returned 1 — survive 100 C-API PyList_Append calls (through a
resize), PySet_Add/PySet_Discard, Python-level append/sort/add, and gc.collect(),
with a clean interpreter exit:

list: ok=1 len=101 tracked=False
set:  ok=1 len=1 tracked=False
after python ops + gc.collect: list len=102 set len=2
exit=0

(The remaining _PyObject_GC_IS_TRACKED asserts in Python/gc*.c are GC-internal and only
touch objects the GC itself still tracks, so untracked immortals never reach them.)

So the fix is dict-local: either drop the assert in dict_merge_api(), relax it to
tracked || immortal, or re-track there. On the free-threaded build the same choice has to
hold against update_refs() untracking immortal dicts during collection — re-tracking is the
option that keeps the tracked bit meaningful for the shared-dict paths.

[picnixz]

Yeah but the problem is: what if someone writes a custom GC-tracked object and then hits a similar assertion? is it a wrong design on their side or not? I don't think extension modules writers should expect their types to be immortalized.

I don't have an answer on how to fix that which is why I want more feedback from core devs.