Skip to content

Stable code identifier pickling - #40711

Draft
kristynsmith wants to merge 18 commits into
apache:masterfrom
kristynsmith:stable-code-identifier-pickling
Draft

kristynsmith wants to merge 18 commits into
apache:masterfrom
kristynsmith:stable-code-identifier-pickling

Conversation

@kristynsmith

Copy link
Copy Markdown
Contributor

Fixes several edge cases in code_object_pickler when generating and resolving stable code object identifiers:

  • Addressed the case where multiple nested def functions have the same name (e.g. defined across if/else branches) by applying the same signature and bytecode-hash disambiguation in _search_locals (formerly _search_lambda) to any duplicate code object name in co_consts, and updating _OBJECT_WITH_ARGS_PATTERN and _OBJECT_WITH_HASH_PATTERN to match named functions in addition to <lambda>.
  • Addressed the case where one of multiple lambdas/nested functions takes no arguments by updating _OBJECT_WITH_ARGS_PATTERN to match empty argument tuples ().
  • Addressed the case where two lambdas or nested functions have identical co_code bytecode instructions (e.g. referencing different closure variables or constants) by including co_consts, co_names, co_varnames, co_freevars, and co_cellvars in _create_bytecode_hash.
  • Addressed the case where a parent package re-exports a symbol with the same name as a submodule (e.g. from pkg.submod import submod in pkg/__init__.py) by resolving the longest matching module prefix in get_code_from_identifier before traversing attributes.
  • Addressed the case where an outer and inner function share the same decorator wrapper by returning None (falling back to default pickling) instead of raising ValueError('Qual name parts too long') when a branch does not match the full __qualname__.

Replace config assignment with dataclasses.replace for stability.
Refactor code object handling and improve normalization for hashing.
Added new test cases for lambda functions and decorators in code_object_pickler.
Refactor code object search logic to handle local code objects and update lambda handling. Replace lambda-specific functions with more general object handling functions.
Added tests for handling same-named nested functions with different and same arguments in code object pickler.
@github-actions github-actions Bot added the size/m M: 50-249 lines changed label Oct 27, 2026
@codecov

codecov Bot commented Oct 27, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.44068% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.91%. Comparing base (a8d6afd) to head (1afac1e).
⚠️ Report is 112 commits behind head on master.

Files with missing lines Patch % Lines
...python/apache_beam/internal/code_object_pickler.py 86.44% 8 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master   #40711      +/-   ##
==========================================
- Coverage   80.04%   79.91%   -0.14%     
==========================================
  Files         490      493       +3     
  Lines       82916    83500     +584     
==========================================
+ Hits        66373    66726     +353     
- Misses      16543    16774     +231     
Flag Coverage Δ
python 79.91% <86.44%> (-0.14%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python size/m M: 50-249 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant