Repository navigation
[FLINK-41549][table] Add built-in TO_APPEND PTF - #30016
gustavodemorais wants to merge 4 commits into
Conversation
Generated-by: Claude Code 2.1.304 (Opus 5.5)
|
|
||
| | Function | Description | | ||
| |---------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | `TO_APPEND(input => TABLE t)` | Converts an updating table into an append-only table: new rows and updates are appended, deletes are dropped. See Changelog Conversion for the full list of arguments, semantics, and usage. | |
There was a problem hiding this comment.
seems the zh version is not insync with the english version
There was a problem hiding this comment.
The zh docs don't have the Changelog Conversion page yet, so the zh table has no links. FLINK-39615 (translation of changelog.md) covers it.
There was a problem hiding this comment.
I'll probably do a callout in the mailing list for a couple of missing chinese translations that I'm aware of and then we can cover this
|
|
||
| ```sql | ||
| SELECT * FROM TO_APPEND(input => TABLE source_table) | ||
| ``` |
There was a problem hiding this comment.
we should add a warning that this operation is deliberately lossy. people should know what they are doing.
There was a problem hiding this comment.
Added a warning after the syntax: deletes are dropped, so a consumer cannot tell that a row was deleted. It points to an upsert sink or TO_CHANGELOG to keep deletes.
| + "WITH order_changes AS (" | ||
| + "SELECT * FROM TO_APPEND(input => TABLE orders)) " | ||
| + "SELECT window_start, window_end, COUNT(*) AS changes " | ||
| + "FROM TUMBLE(TABLE order_changes, DESCRIPTOR(ts), INTERVAL '1' HOUR) " |
There was a problem hiding this comment.
why does this work? is this actually a bug? the input time attribute should be materialized.
There was a problem hiding this comment.
As discussed, I added an internal PRESERVE_TIME_ATTRIBUTES trait. By default PTFs now drop all time attributes from their output (only the on_time rowtime stays) and materialize proctime before the PTF. TO_APPEND and ML_PREDICT declare the trait because they emit input rows unchanged. on_time together with the trait is rejected.
| super(BuiltInFunctionDefinitions.TO_APPEND, context); | ||
| } | ||
|
|
||
| public void eval(final Context ctx, final RowData input) { |
There was a problem hiding this comment.
| public void eval(final Context ctx, final RowData input) { | |
| public void eval(final RowData input) { |
…erved Generated-by: Claude Code 2.1.304 (Opus 5.5)
55b7d59 to
a3f0322
Compare
|
Thanks for the review, @twalthr. Feedback addressed, PTAL 🙂 |
…unction Generated-by: Claude Code 2.1.304 (Opus 5.5)
a3f0322 to
97af3fd
Compare
| } | ||
|
|
||
| @Test | ||
| void testOnTimeNotSupported() { |
There was a problem hiding this comment.
can we remove this check? we support on_time, we should avoid error where we can. if somebody likes a unified on_time call for all PTFs, we should not disallow it.
There was a problem hiding this comment.
I like that, makes sense. done
There was a problem hiding this comment.
on_time has the priority now and is always supported
| newScan.getInputs().stream() | ||
| .map(this::materializeProcTime) | ||
| .collect(Collectors.toList()); | ||
| return newScan.copy(newScan.getTraitSet(), newInputs); |
There was a problem hiding this comment.
I'm not sure if this fully fixes the problem. Usually you also have to update the consumer and all its references. In this case the RexTableArg (TABLE()) needs an update, otherwise there is a type mismatch.
There was a problem hiding this comment.
Good point. The RexTableArgCall type is now updated from the materialized input, keeping its field names
| private boolean preservesTimeAttributes(List<StaticArgument> staticArgs) { | ||
| return functionKind != FunctionKind.PROCESS_TABLE | ||
| || staticArgs.stream() | ||
| .anyMatch(arg -> arg.is(StaticArgumentTrait.PRESERVE_TIME_ATTRIBUTES)); |
There was a problem hiding this comment.
allMatch on table attributes
| * Defines that the function emits the rows of this table argument unchanged and without | ||
| * reordering them across watermarks, so the time attributes of the copied columns stay valid. |
There was a problem hiding this comment.
| * Defines that the function emits the rows of this table argument unchanged and without | |
| * reordering them across watermarks, so the time attributes of the copied columns stay valid. | |
| * Specifies that the function emits rows of this table argument in accordance with the watermark strategy, so that the time attributes in the output columns remain valid. |
| // this whole topic is kind of vendor specific already | ||
| fields.addAll(derivePassThroughFields(callContext, resolvedArgs)); | ||
| fields.addAll(deriveFunctionOutputFields(functionDataType)); | ||
| if (!preservesTimeAttributes(resolvedArgs)) { |
There was a problem hiding this comment.
let's move this to deriveFunctionOutputFields, pass through columns should be fine as they are highly specialized
There was a problem hiding this comment.
Done. Pass-through columns keep their time attributes now.
Generated-by: Claude Code 2.1.304 (Opus 5.5)
|
Thanks for the quick reviews, @twalthr. Feedback has been addressed. Take a look 🙂 |
| newScan.getColumnMappings()); | ||
| } | ||
|
|
||
| private RexNode withInputType(RexTableArgCall tableArg, List<RelNode> inputs) { |
There was a problem hiding this comment.
| private RexNode withInputType(RexTableArgCall tableArg, List<RelNode> inputs) { | |
| private RexNode adjustTableArg(RexTableArgCall tableArg, List<RelNode> inputs) { |
| return tableArg.clone( | ||
| rexBuilder | ||
| .getTypeFactory() | ||
| .createStructType( |
There was a problem hiding this comment.
Double check if the StructKind is preserved
| disableSystemArgs | ||
| ? List.of() | ||
| : deriveRowtimeField(callContext, resolvedArgs); | ||
| // The on_time rowtime replaces any forwarded time attribute |
There was a problem hiding this comment.
in theory we could allow both, there is no limitation of multiple time attributes per row. would maybe simplify things?
What is the purpose of the change
Implements FLIP-621: a built-in TO_APPEND process table function that converts an updating table into an append-only table. Every new row and every update is appended as a new row, deletes are dropped, and the output keeps the input schema.
Brief change log
Verifying this change
This change added tests and can be verified as follows:
Does this pull request potentially affect one of the following parts:
@Public(Evolving): yes (Table#toAppend)Documentation
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code 2.1.304 (Opus 5.5)