Skip to content

[FLINK-41470][table] Deduplicate implementation insert input - #29967

Open
VasShabu wants to merge 9 commits into
apache:masterfrom
VasShabu:deduplicateImplementationInsertInput
Open

VasShabu wants to merge 9 commits into
apache:masterfrom
VasShabu:deduplicateImplementationInsertInput

Conversation

@VasShabu

Copy link
Copy Markdown
Contributor

What is the purpose of the change

Adds the built-in DEDUPLICATE process table function from FLIP-622 §4.1 for insert-only input. It keeps one row per PARTITION BY key, chosen by arrival order or by sequence_by, and emits an updating changelog whose upsert key is the partition key.

Updating input and the docs page follow in separate PRs.

Brief change log

  • DEDUPLICATE in BuiltInFunctionDefinitions with DeduplicateTypeStrategy: validates PARTITION BY, keep, sequence_by, state_ttl and reset_ttl_on_duplicate at planning time, and infers the output type and changelog mode (+I, +U, and -U only when a consumer requires it).
  • Deduplicate is specialized per call, and each specialization declares only the state it needs:
    • keep => 'first' without sequence_by: a BOOLEAN per key.
    • Otherwise: the kept row, without the key columns.
  • Table API PartitionedTable#deduplicate(...); PyFlink Table.partition_by(...) and PartitionedTable.deduplicate(...).

Verifying this change

This change added tests and can be verified as follows:

  • DeduplicateTypeStrategyTest: every validation error, the output type and the changelog mode.
  • DeduplicateTest: the state each specialization declares.
  • DeduplicateTableFunctionTest: plans for the upsert key, a retracting consumer, dropped time attributes and the Table API.
  • DeduplicateSemanticTests: arrival order, a composite non-adjacent and NULL key, sequence_by ordering (ties, NULL, NaN, multiple columns), and the missing PARTITION BY error.
  • DeduplicateRestoreTest: kept state for 'last', 'first' and sequence_by survives a restore.
  • DeduplicateHarnessTest: state_ttl expiry and reset_ttl_on_duplicate.
  • PyFlink: test_explain_deduplicate and PartitionedTable API completeness.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): yes (PartitionedTable#deduplicate, PyFlink PartitionedTable)
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): yes, only for queries that call DEDUPLICATE
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? JavaDocs; the SQL docs page follows in a separate PR

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Claude Opus 5.5)

@flinkbot

flinkbot commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@gustavodemorais gustavodemorais left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Vas, it's a good start. Looking forward to getting this ready. In general, some general pointers:

  • Let the framework do the work: Some checks and code paths duplicate what the PTF framework already does, or work around it. For example, a missing PARTITION BY is already rejected for set-semantic tables, and declaring a trait we don't support yet (SUPPORT_UPDATES) leads to an internal exception instead of a validation error. Let's keep custom code only for what is specific to deduplication, and share helpers that DEDUPLICATE_WINDOW will need as well.

  • Tests: Let's aim for full coverage with as few, real-world tests as possible. Avoid tests of internal details that other tests already cover, keep one test per behavior or state layout, and only use harness tests where nothing else can test it (processing-time TTL).

  • Structure and readability: The specialization dispatch should read like a table: which arguments lead to which implementation. Please also follow the conventions of the recent PTFs (e.g. ToChangelogFunction/FromChangelogFunction): naming, one class per file, and clear names that match what each variant does.

}

@Override
public UserDefinedFunction specialize(final SpecializedContext context) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we make this an explicit, small dispatch (which arguments lead to which implementation) and move each variant to its own file? Also, Sequenced covers keep => 'last' without sequence_by too, so a name that describes what it keeps might be clearer.

final CallContext callContext = context.getCallContext();
this.outputIndices =
ChangelogTypeStrategyUtils.computeOutputIndices(tableSemantics(context));
this.isResetTtlOnDuplicate =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With no TTL configured (state_ttl omitted and table.exec.state.ttl = 0), this is still true, so duplicates write state for nothing. Could we derive it from the effective TTL? Worth checking the rest of the hot path for similar per-record costs.

@VasShabu
VasShabu force-pushed the deduplicateImplementationInsertInput branch 2 times, most recently from 60ccdcd to 31ce87c Compare November 16, 2026 16:51
},
"inputChangelogModes" : [ [ "INSERT" ] ],
"outputChangelogMode" : [ "INSERT", "UPDATE_AFTER" ],
"inputUpsertKeys" : [ [ ] ]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we avoid empty collections being serialized?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a framework issue, this json file represents the plan that flink had, TO/FROM _CHANGELOG has some similar issue, I will write a ticket for this and have a look at this after the deduplicate PTFs are in.

},
"inputChangelogModes" : [ [ "INSERT" ] ],
"outputChangelogMode" : [ "INSERT", "UPDATE_AFTER" ],
"inputUpsertKeys" : [ [ ] ]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same message as above

@gustavodemorais gustavodemorais left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Vas, nice progress! I really like the new specialize() dispatch :)

A few more general pointers:

  • Rebase: please rebase onto master. DEDUPLICATE_KEEP_FIRST was removed there (FLINK-41422), which is why the PR shows conflicts.

  • Less code for the same behavior: DeduplicateTypeStrategy and DeduplicateTypeStrategyUtils together are about 430 lines for what they do, much of it long error messages and utils that only deduplicate uses. Checks that every PTF with a state_ttl needs (literal, not negative) should be shared by all PTFs, not live in a deduplicate-specific util. The same goes for rejecting on_time: every PTF without event time has this problem, so a general solution in the framework would be better than a custom check per function.

  • Hot path and readability: eval() should be cheap and easy to read top to bottom. Today it does several projections, serializations and copies per record. Please look at it with that lens and keep only what the result needs.

  • Tests: let's only keep tests that cover deduplicate-specific behavior. The framework already tests the PTF TTL plumbing (PROCESS_STATE_TTL in ProcessTableFunctionTestPrograms), and TTL expiry itself is state backend behavior. So the harness test only adds value for reset_ttl_on_duplicate, which is still an open point in FLIP-622.

super.close();
}

public void eval(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a lot of work per record: copying the fields into a GenericRowData (boxing), serializing it, running a second serializer on the kept row (a no-op, since the row from state is already binary), copying again before writing to state, and emitting through a JoinedRowData with an empty row. Could we project once, serialize once, and compare directly with the row from state? That would also make the function much easier to read.

@gustavodemorais gustavodemorais left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll do a next pass tomorrow. One thing we're missing is documentation @vasshab. We could also do it as follow up ticket but we usually do it as we go so it matches the tests and cases that we are covering

@gustavodemorais gustavodemorais left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Vas, great to see the docs! In general, let's keep them short and write them for someone who just wants to use the function: what it does, its arguments, and a few examples. Internal details (changelog flags, state layout, argument order rules) can mostly go.

@snuyanzin snuyanzin changed the title [FLINK-41470][table]Deduplicate implementation insert input [FLINK-41470][table] Deduplicate implementation insert input Nov 19, 2026
@gustavodemorais

Copy link
Copy Markdown
Contributor

Thanks for all the changes, @VasShabu. I did a longer pass on this one now. The PR is almost there.

…rop redundant parts

Address review feedback on the DEDUPLICATE PTF:

- Make Deduplicate a small dispatcher and move its variants into
  DeduplicateFirstArrivalFunction and DeduplicateKeptRowFunction, with
  the shared state and TTL setup in DeduplicateFunctionBase and
  DeduplicateUtils.
- Stop declaring SUPPORT_UPDATES so the framework rejects updating input
  at validation time, and remove the unreachable check in specialize().
- Drop the PARTITION BY check, which the framework already enforces for
  set-semantic tables, and fix the leftover text in the on_time error.
- Remove the PyFlink partition_by/PartitionedTable additions.
- Remove DeduplicateTest and the arrival-order restore program, which
  duplicate the semantic and remaining restore tests.
…cut per-record work

Compare and emit through views of the input row instead of projecting and serializing a payload row per record, and rename the variants to DeduplicateEmitOnceFunction and DeduplicateCurrentRowFunction. Reduce the TTL harness test to reset_ttl_on_duplicate and regenerate the sequence-by restore test.
…he framework for on_time

Drop the custom on_time check, since the planner already rejects on_time for PTFs that produce updates, and keep each validation message to the bad input and its fix.
…im the parameter docs

Show the final rows in the examples, keep the change flags in a hint box, shorten the state_ttl row, and use INTERVAL '0' SECOND.
…_ttl hint

Flink SQL cannot parse INTERVAL '0' without a unit, so the hint now names the unit and the test pins the full message.
Add a generic valueState helper to BuiltInProcessTableFunction and name the emit-once state 'emitted', strip time attributes from the input type with a single call, serialize the candidate row only when it can replace the current one, and move the replace decision into replacesCurrent. Regenerate the first-arrival restore test for the renamed state.
@VasShabu
VasShabu force-pushed the deduplicateImplementationInsertInput branch from 88a0704 to 11431f4 Compare November 20, 2026 02:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants