Skip to content

[DirectRunner] Reuse ParDo metadata between bundles - #40862

Draft
bvolpato wants to merge 1 commit into
apache:masterfrom
bvolpato:bvolpato/cache-direct-pardo-metadata
Draft

bvolpato wants to merge 1 commit into
apache:masterfrom
bvolpato:bvolpato/cache-direct-pardo-metadata

Conversation

@bvolpato

Copy link
Copy Markdown
Contributor

DirectRunner extracts ParDo schema information and side-input mappings for every bundle. Those helpers translate the full ParDo payload each time. Profiling Nexmark's streaming queries showed repeated translation and serialization on the worker threads.

Cache the serialized metadata per applied transform. Deserialize a fresh copy for each evaluator because schema conversion functions have mutable state, and clear the cache during cleanup.

In matched warm local runs on a busy shared host, Nexmark's streaming Query 7 took 174 seconds before and 26 seconds after, excluding Gradle setup. The input, assertions, JDK and runner settings were unchanged.

The full Nexmark suite and focused DirectRunner tests covering schemas, side inputs, state and lifecycle passed locally. Formatting and compilation checks passed as well.

@github-actions github-actions Bot added runners direct size/s S: 10-49 lines changed labels Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

direct runners size/s S: 10-49 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant