Skip to content

Port SplittableParDoProcessFnTest off DoFnTester - #40865

Open
Rishavbhattarai wants to merge 1 commit into
apache:masterfrom
Rishavbhattarai:port-processfn-test-off-dofntester
Open

Rishavbhattarai wants to merge 1 commit into
apache:masterfrom
Rishavbhattarai:port-processfn-test-off-dofntester

Conversation

@Rishavbhattarai

Copy link
Copy Markdown

Fixes #18691.

Ports SplittableParDoProcessFnTest off the deprecated DoFnTester.

These tests drive ProcessFn by hand: they advance processing time, fire its timers and check the watermark hold between calls. TestPipeline can't do that, so I kept the ProcessFnTester helper and changed what's inside it:

  • ProcessFn now runs through DoFnRunners.simpleRunner, the same runner SplittableProcessElementsEvaluatorFactory uses in the DirectRunner. StepContext hands it the test's existing InMemoryTimerInternals and TestInMemoryStateInternals.
  • close() calls finishBundle() and then ProcessFn.tearDown(), the same order DoFnTester.close() used.
  • Outputs from the splittable invoker go into a plain list. takeOutputElements() and peekOutputElementsInWindow() read from that list, so none of the test bodies change.
  • Removed DoFnTesterWindowedValueReceiver and the DoFnTester import.

Testing: ./gradlew :runners:core-java:test --tests SplittableParDoProcessFnTest passes (15/15), and spotlessCheck is clean.

Run ProcessFn through DoFnRunners.simpleRunner, the same runner the
DirectRunner uses, instead of the deprecated DoFnTester. The
ProcessFnTester helper keeps its API, so the test bodies don't change.

Fixes apache#18691
@github-actions github-actions Bot added runners size/m M: 50-249 lines changed labels Oct 28, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR cleanly migrates SplittableParDoProcessFnTest off the deprecated DoFnTester by updating the ProcessFnTester test helper to execute ProcessFn directly via DoFnRunners.simpleRunner. The implementation preserves bundle lifecycle management, state/timer handling, and output verification without needing to alter any of the existing test cases.

🔍 General Feedback

  • The setup of DoFnRunners.simpleRunner and custom WindowedValueMultiReceiver accurately mirrors how ProcessFn is invoked in runners like DirectRunner.
  • Bundle lifecycle transitions (setup, startBundle, finishBundle, tearDown) are properly preserved in ProcessFnTester.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @shunping added as fallback since no labels match configuration
Backup reviewer: kennknowles (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Port SplittableParDoProcessFnTest off DoFnTester

1 participant