Skip to content

Improve performance of dict merges by avoiding incref/decref pairs #158863

Description

@eendebakpt

Feature or enhancement

The per-item loop in dict_dict_merge() takes an extra reference to key and value before dispatching on override. On the override == 1 path insertdict() already receives its own references through Py_NewRef() and
nothing uses key or value afterwards, so the outer pair is wasted refcount operations per item (atomic ones in the free-threaded build). The pair is still needed on the other path, where _PyDict_Contains_KnownHash() can run __eq__ and *dupkey hands the reference to the caller.

override == 1 covers dict.update(d), d1 \| d2, {**a, **b}, any merge into an empty dict, and dict(d) / d.copy() when the key-table clone fast path does not apply (sparse, split-table or subclass source).

Benchmarks

case main PR change
ctl: a + b 41.1 ns 41.0 ns not significant
ctl: dct_100[key] 48.3 ns 48.6 ns not significant
ctl: dict(dct_100) (clone path) 369 ns 371 ns not significant
dct.update(dct_5) 150 ns 146 ns not significant
dct.update(dct_100) 1.63 us 1.56 us 1.05x faster
dct.update(dct_1000) 17.1 us 16.5 us 1.04x faster
dct.update(dct_100) (int keys) 1.18 us 1.09 us 1.08x faster
dct_100 | other_100 2.84 us 2.74 us 1.04x faster
{**dct_5, **other_5} 230 ns 224 ns 1.03x faster
sparse_1000.copy() (500 deleted) 10.4 us 9.93 us 1.05x faster
dict(vars(obj)) (4 attributes) 170 ns 168 ns not significant

Free-threaded build shows a slightly larger gain.

Benchmark script
"""dict_dict_merge() cases. ctl rows do not run the per-item merge loop:
dense copies take the key-table clone fast path."""
import os
import pyperf

runner = pyperf.Runner()
def S(n): return "{'key%%d' %% i: i for i in range(%d)}" % n

CASES = [
    ("ctl: a + b", "a + b", "a = 1000; b = 2000"),
    ("ctl: dct_100[key]", "d[k]", "d = " + S(100) + "; k = 'key50'"),
    ("ctl: dict(dct_5)  (clone path)", "dict(d)", "d = dict.fromkeys('abcde', 1)"),
    ("ctl: dict(dct_100)  (clone path)", "dict(d)", "d = " + S(100)),

    ("dct.update(dct_5)", "c.update(b)", "b = dict.fromkeys('abcde', 1); c = dict(b)"),
    ("dct.update(dct_100)", "c.update(b)", "b = " + S(100) + "; c = dict(b)"),
    ("dct.update(dct_1000)", "c.update(b)", "b = " + S(1000) + "; c = dict(b)"),
    ("dct.update(dct_100)  (int keys)", "c.update(b)", "b = {i: i for i in range(1000, 1100)}; c = dict(b)"),
    ("dct_100 | other_100", "a | b", "a = " + S(100) + "; b = {'other%d' % i: i for i in range(100)}"),
    ("{**dct_5, **other_5}", "{**a, **b}", "a = dict.fromkeys('abcde', 1); b = dict.fromkeys('fghij', 1)"),
    ("sparse_1000.copy()  (500 deleted)", "d.copy()",
     "d = " + S(1000) + "\nfor i in range(0, 1000, 2): del d['key%d' % i]"),
    ("dict(vars(obj))  (4 attributes)", "dict(v)",
     "class C:\n    def __init__(self): self.a = 1; self.b = 2; self.c = 3; self.d = 4\nv = vars(C())"),
]
for name, stmt, setup in CASES:
    runner.timeit(name, stmt, setup=setup)

Generated with help from Claude Code

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs

Activity

  1. added
    performancePerformance or resource usage
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    type-featureA feature request or enhancement
    on Oct 5, 2026
  2. added a commit that references this issue on Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    interpreter-core(Objects, Python, Grammar, and Parser dirs)performancePerformance or resource usagetype-featureA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions