Skip to content

Use effective identity for DryRunEvaluator node_key - #262

Merged
timkpaine merged 1 commit into
mainfrom
fix/dry-run-effective-node-key
Aug 24, 2026
Merged

Use effective identity for DryRunEvaluator node_key#262
timkpaine merged 1 commit into
mainfrom
fix/dry-run-effective-node-key

Conversation

@ptomecek

@ptomecek ptomecek commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

Summary

DryRunEvaluator builds a plan of the nodes that would run and emits a node_key identifying each one. That key was computed with the plain structural cache_key(...), while the actual runtime — the in-process MemoryCacheEvaluator and the dependency-graph dedup — identifies the same nodes with cache_key(..., effective=True).

So the dry run could label a node differently than the real run caches and dedupes it. For models that use a narrower "effective" identity, two nodes that the real run treats as distinct could share a node_key in the plan (or vice-versa) — making the dry-run report an unreliable preview of what actually happens.

Fix

Compute the dry-run node_key with cache_key(..., effective=True), the same identity the memory cache and graph dedup already use. The dry-run plan now matches runtime behavior node-for-node.

This is a no-op for ordinary models (which use the structural key), so existing plans are unchanged; it only affects models that opt into effective identity.

Test

Adds test_node_key_uses_effective_identity: a model that ignores an unused context field. Two contexts that differ only in that ignored field previously produced different node_keys in the plan; now they collapse to the same key, matching how the runtime caches/dedupes them — and the emitted key equals cache_key(..., effective=True).

Validation

ccflow/tests/evaluators/ and ccflow/tests/test_effective_key_characterization.py pass (75 tests), including all pre-existing dry-run tests.

The dry-run report node_key was derived from the raw structural
cache_key(...) (effective=False), so it ignored the opt-in
_CallableModel._evaluation_identity_payload() effective-identity hook.
The in-process MemoryCacheEvaluator and the dependency-graph dedup
already key on cache_key(..., effective=True), so a model that opts
into a collision-safe effective identity was reported under the
structural key in dry-run plans, inconsistently with how it is actually
cached and deduplicated.

Pass effective=True when building the dry-run node_key. For models that
opt out (the base hook returns None) this is a byte-for-byte no-op and
preserves the existing structural key; opt-in models now get their
effective identity in the plan, consistent with memory/graph dedup.

Add a regression test using a generated @Flow.model that ignores an
unused ambient context field: the emitted node_key must equal
cache_key(..., effective=True) and must merge two contexts that differ
only in an ignored field, even though their structural keys differ.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Pascal Tomecek <pascal.tomecek@cubistsystematic.com>
@github-actions

github-actions Bot commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Test Results

    1 files  ±0      1 suites  ±0   3m 17s ⏱️ +17s
1 329 tests +1  1 327 ✅ +1  2 💤 ±0  0 ❌ ±0 
1 335 runs  +1  1 333 ✅ +1  2 💤 ±0  0 ❌ ±0 

Results for commit b80b76a. ± Comparison against base commit 6a788a6.

♻️ This comment has been updated with latest results.

@codecov

codecov Bot commented Aug 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.23810% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 93.48%. Comparing base (6a788a6) to head (b80b76a).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
ccflow/tests/evaluators/test_reporting.py 95.00% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #262   +/-   ##
=======================================
  Coverage   93.48%   93.48%           
=======================================
  Files         176      176           
  Lines       20327    20347   +20     
  Branches     1350     1350           
=======================================
+ Hits        19002    19021   +19     
- Misses       1052     1053    +1     
  Partials      273      273           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ptomecek
ptomecek marked this pull request as ready for review August 24, 2026 12:15
@timkpaine
timkpaine merged commit 60e7412 into main Aug 24, 2026
20 checks passed
@timkpaine
timkpaine deleted the fix/dry-run-effective-node-key branch August 24, 2026 12:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants