Repository navigation
Conversation
gustavodemorais
left a comment
There was a problem hiding this comment.
Thanks Vas, it's a good start. Looking forward to getting this ready. In general, some general pointers:
-
Let the framework do the work: Some checks and code paths duplicate what the PTF framework already does, or work around it. For example, a missing PARTITION BY is already rejected for set-semantic tables, and declaring a trait we don't support yet (SUPPORT_UPDATES) leads to an internal exception instead of a validation error. Let's keep custom code only for what is specific to deduplication, and share helpers that DEDUPLICATE_WINDOW will need as well.
-
Tests: Let's aim for full coverage with as few, real-world tests as possible. Avoid tests of internal details that other tests already cover, keep one test per behavior or state layout, and only use harness tests where nothing else can test it (processing-time TTL).
-
Structure and readability: The specialization dispatch should read like a table: which arguments lead to which implementation. Please also follow the conventions of the recent PTFs (e.g. ToChangelogFunction/FromChangelogFunction): naming, one class per file, and clear names that match what each variant does.
| } | ||
|
|
||
| @Override | ||
| public UserDefinedFunction specialize(final SpecializedContext context) { |
There was a problem hiding this comment.
Could we make this an explicit, small dispatch (which arguments lead to which implementation) and move each variant to its own file? Also, Sequenced covers keep => 'last' without sequence_by too, so a name that describes what it keeps might be clearer.
| final CallContext callContext = context.getCallContext(); | ||
| this.outputIndices = | ||
| ChangelogTypeStrategyUtils.computeOutputIndices(tableSemantics(context)); | ||
| this.isResetTtlOnDuplicate = |
There was a problem hiding this comment.
With no TTL configured (state_ttl omitted and table.exec.state.ttl = 0), this is still true, so duplicates write state for nothing. Could we derive it from the effective TTL? Worth checking the rest of the hot path for similar per-record costs.
60ccdcd to
31ce87c
Compare
| }, | ||
| "inputChangelogModes" : [ [ "INSERT" ] ], | ||
| "outputChangelogMode" : [ "INSERT", "UPDATE_AFTER" ], | ||
| "inputUpsertKeys" : [ [ ] ] |
There was a problem hiding this comment.
can we avoid empty collections being serialized?
There was a problem hiding this comment.
I think this is a framework issue, this json file represents the plan that flink had, TO/FROM _CHANGELOG has some similar issue, I will write a ticket for this and have a look at this after the deduplicate PTFs are in.
| }, | ||
| "inputChangelogModes" : [ [ "INSERT" ] ], | ||
| "outputChangelogMode" : [ "INSERT", "UPDATE_AFTER" ], | ||
| "inputUpsertKeys" : [ [ ] ] |
There was a problem hiding this comment.
same message as above
There was a problem hiding this comment.
Thanks Vas, nice progress! I really like the new specialize() dispatch :)
A few more general pointers:
-
Rebase: please rebase onto master. DEDUPLICATE_KEEP_FIRST was removed there (FLINK-41422), which is why the PR shows conflicts.
-
Less code for the same behavior: DeduplicateTypeStrategy and DeduplicateTypeStrategyUtils together are about 430 lines for what they do, much of it long error messages and utils that only deduplicate uses. Checks that every PTF with a
state_ttlneeds (literal, not negative) should be shared by all PTFs, not live in a deduplicate-specific util. The same goes for rejectingon_time: every PTF without event time has this problem, so a general solution in the framework would be better than a custom check per function. -
Hot path and readability:
eval()should be cheap and easy to read top to bottom. Today it does several projections, serializations and copies per record. Please look at it with that lens and keep only what the result needs. -
Tests: let's only keep tests that cover deduplicate-specific behavior. The framework already tests the PTF TTL plumbing (
PROCESS_STATE_TTLinProcessTableFunctionTestPrograms), and TTL expiry itself is state backend behavior. So the harness test only adds value forreset_ttl_on_duplicate, which is still an open point in FLIP-622.
| super.close(); | ||
| } | ||
|
|
||
| public void eval( |
There was a problem hiding this comment.
This is a lot of work per record: copying the fields into a GenericRowData (boxing), serializing it, running a second serializer on the kept row (a no-op, since the row from state is already binary), copying again before writing to state, and emitting through a JoinedRowData with an empty row. Could we project once, serialize once, and compare directly with the row from state? That would also make the function much easier to read.
31ce87c to
b7669cc
Compare
b7669cc to
978bf2e
Compare
gustavodemorais
left a comment
There was a problem hiding this comment.
I'll do a next pass tomorrow. One thing we're missing is documentation @vasshab. We could also do it as follow up ticket but we usually do it as we go so it matches the tests and cases that we are covering
gustavodemorais
left a comment
There was a problem hiding this comment.
Thanks Vas, great to see the docs! In general, let's keep them short and write them for someone who just wants to use the function: what it does, its arguments, and a few examples. Internal details (changelog flags, state layout, argument order rules) can mostly go.
|
Thanks for all the changes, @VasShabu. I did a longer pass on this one now. The PR is almost there. |
…rop redundant parts Address review feedback on the DEDUPLICATE PTF: - Make Deduplicate a small dispatcher and move its variants into DeduplicateFirstArrivalFunction and DeduplicateKeptRowFunction, with the shared state and TTL setup in DeduplicateFunctionBase and DeduplicateUtils. - Stop declaring SUPPORT_UPDATES so the framework rejects updating input at validation time, and remove the unreachable check in specialize(). - Drop the PARTITION BY check, which the framework already enforces for set-semantic tables, and fix the leftover text in the on_time error. - Remove the PyFlink partition_by/PartitionedTable additions. - Remove DeduplicateTest and the arrival-order restore program, which duplicate the semantic and remaining restore tests.
…cut per-record work Compare and emit through views of the input row instead of projecting and serializing a payload row per record, and rename the variants to DeduplicateEmitOnceFunction and DeduplicateCurrentRowFunction. Reduce the TTL harness test to reset_ttl_on_duplicate and regenerate the sequence-by restore test.
…he framework for on_time Drop the custom on_time check, since the planner already rejects on_time for PTFs that produce updates, and keep each validation message to the bad input and its fix.
…im the parameter docs Show the final rows in the examples, keep the change flags in a hint box, shorten the state_ttl row, and use INTERVAL '0' SECOND.
…_ttl hint Flink SQL cannot parse INTERVAL '0' without a unit, so the hint now names the unit and the test pins the full message.
Add a generic valueState helper to BuiltInProcessTableFunction and name the emit-once state 'emitted', strip time attributes from the input type with a single call, serialize the candidate row only when it can replace the current one, and move the replace decision into replacesCurrent. Regenerate the first-arrival restore test for the renamed state.
88a0704 to
11431f4
Compare
What is the purpose of the change
Adds the built-in
DEDUPLICATEprocess table function from FLIP-622 §4.1 for insert-only input. It keeps one row perPARTITION BYkey, chosen by arrival order or bysequence_by, and emits an updating changelog whose upsert key is the partition key.Updating input and the docs page follow in separate PRs.
Brief change log
DEDUPLICATEinBuiltInFunctionDefinitionswithDeduplicateTypeStrategy: validatesPARTITION BY,keep,sequence_by,state_ttlandreset_ttl_on_duplicateat planning time, and infers the output type and changelog mode (+I,+U, and-Uonly when a consumer requires it).Deduplicateis specialized per call, and each specialization declares only the state it needs:keep => 'first'withoutsequence_by: a BOOLEAN per key.PartitionedTable#deduplicate(...); PyFlinkTable.partition_by(...)andPartitionedTable.deduplicate(...).Verifying this change
This change added tests and can be verified as follows:
DeduplicateTypeStrategyTest: every validation error, the output type and the changelog mode.DeduplicateTest: the state each specialization declares.DeduplicateTableFunctionTest: plans for the upsert key, a retracting consumer, dropped time attributes and the Table API.DeduplicateSemanticTests: arrival order, a composite non-adjacent and NULL key,sequence_byordering (ties, NULL, NaN, multiple columns), and the missingPARTITION BYerror.DeduplicateRestoreTest: kept state for'last','first'andsequence_bysurvives a restore.DeduplicateHarnessTest:state_ttlexpiry andreset_ttl_on_duplicate.test_explain_deduplicateandPartitionedTableAPI completeness.Does this pull request potentially affect one of the following parts:
@Public(Evolving): yes (PartitionedTable#deduplicate, PyFlinkPartitionedTable)DEDUPLICATEDocumentation
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Claude Opus 5.5)