Skip to content

[FLINK-41549][table] Add built-in TO_APPEND PTF - #30016

Draft
gustavodemorais wants to merge 4 commits into
apache:masterfrom
confluentinc:FLINK-41549
Draft

gustavodemorais wants to merge 4 commits into
apache:masterfrom
confluentinc:FLINK-41549

Conversation

@gustavodemorais

Copy link
Copy Markdown
Contributor

What is the purpose of the change

Implements FLIP-621: a built-in TO_APPEND process table function that converts an updating table into an append-only table. Every new row and every update is appended as a new row, deletes are dropped, and the output keeps the input schema.

Brief change log

  • Add the TO_APPEND built-in PTF (row semantics, stateless) and Table#toAppend() / PyFlink to_append()
  • Document TO_APPEND as the first function on the changelog conversion page

Verifying this change

This change added tests and can be verified as follows:

  • ToAppendSemanticTests: upsert, retract, tumbling window and deduplication inputs, plus the Table API
  • ToAppendTest: plans for upsert sources with partial and full deletes (no ChangelogNormalize), a retract source and an input without upsert key

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): yes (Table#toAppend)
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? docs / JavaDocs

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code 2.1.304 (Opus 5.5)

Generated-by: Claude Code 2.1.304 (Opus 5.5)
@flinkbot

flinkbot commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@twalthr twalthr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


| Function | Description |
|---------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `TO_APPEND(input => TABLE t)` | Converts an updating table into an append-only table: new rows and updates are appended, deletes are dropped. See Changelog Conversion for the full list of arguments, semantics, and usage. |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems the zh version is not insync with the english version

@gustavodemorais gustavodemorais Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The zh docs don't have the Changelog Conversion page yet, so the zh table has no links. FLINK-39615 (translation of changelog.md) covers it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll probably do a callout in the mailing list for a couple of missing chinese translations that I'm aware of and then we can cover this


```sql
SELECT * FROM TO_APPEND(input => TABLE source_table)
```

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should add a warning that this operation is deliberately lossy. people should know what they are doing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a warning after the syntax: deletes are dropped, so a consumer cannot tell that a row was deleted. It points to an upsert sink or TO_CHANGELOG to keep deletes.

+ "WITH order_changes AS ("
+ "SELECT * FROM TO_APPEND(input => TABLE orders)) "
+ "SELECT window_start, window_end, COUNT(*) AS changes "
+ "FROM TUMBLE(TABLE order_changes, DESCRIPTOR(ts), INTERVAL '1' HOUR) "

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why does this work? is this actually a bug? the input time attribute should be materialized.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, I added an internal PRESERVE_TIME_ATTRIBUTES trait. By default PTFs now drop all time attributes from their output (only the on_time rowtime stays) and materialize proctime before the PTF. TO_APPEND and ML_PREDICT declare the trait because they emit input rows unchanged. on_time together with the trait is rejected.

super(BuiltInFunctionDefinitions.TO_APPEND, context);
}

public void eval(final Context ctx, final RowData input) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
public void eval(final Context ctx, final RowData input) {
public void eval(final RowData input) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

…erved

Generated-by: Claude Code 2.1.304 (Opus 5.5)
@gustavodemorais

Copy link
Copy Markdown
Contributor Author

Thanks for the review, @twalthr. Feedback addressed, PTAL 🙂

…unction

Generated-by: Claude Code 2.1.304 (Opus 5.5)
}

@Test
void testOnTimeNotSupported() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we remove this check? we support on_time, we should avoid error where we can. if somebody likes a unified on_time call for all PTFs, we should not disallow it.

@gustavodemorais gustavodemorais Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like that, makes sense. done

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on_time has the priority now and is always supported

newScan.getInputs().stream()
.map(this::materializeProcTime)
.collect(Collectors.toList());
return newScan.copy(newScan.getTraitSet(), newInputs);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if this fully fixes the problem. Usually you also have to update the consumer and all its references. In this case the RexTableArg (TABLE()) needs an update, otherwise there is a type mismatch.

@gustavodemorais gustavodemorais Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. The RexTableArgCall type is now updated from the materialized input, keeping its field names

private boolean preservesTimeAttributes(List<StaticArgument> staticArgs) {
return functionKind != FunctionKind.PROCESS_TABLE
|| staticArgs.stream()
.anyMatch(arg -> arg.is(StaticArgumentTrait.PRESERVE_TIME_ATTRIBUTES));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

allMatch on table attributes

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment on lines +51 to +52
* Defines that the function emits the rows of this table argument unchanged and without
* reordering them across watermarks, so the time attributes of the copied columns stay valid.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Defines that the function emits the rows of this table argument unchanged and without
* reordering them across watermarks, so the time attributes of the copied columns stay valid.
* Specifies that the function emits rows of this table argument in accordance with the watermark strategy, so that the time attributes in the output columns remain valid.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

// this whole topic is kind of vendor specific already
fields.addAll(derivePassThroughFields(callContext, resolvedArgs));
fields.addAll(deriveFunctionOutputFields(functionDataType));
if (!preservesTimeAttributes(resolvedArgs)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's move this to deriveFunctionOutputFields, pass through columns should be fine as they are highly specialized

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Pass-through columns keep their time attributes now.

Generated-by: Claude Code 2.1.304 (Opus 5.5)
@gustavodemorais

Copy link
Copy Markdown
Contributor Author

Thanks for the quick reviews, @twalthr. Feedback has been addressed. Take a look 🙂

newScan.getColumnMappings());
}

private RexNode withInputType(RexTableArgCall tableArg, List<RelNode> inputs) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
private RexNode withInputType(RexTableArgCall tableArg, List<RelNode> inputs) {
private RexNode adjustTableArg(RexTableArgCall tableArg, List<RelNode> inputs) {

return tableArg.clone(
rexBuilder
.getTypeFactory()
.createStructType(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Double check if the StructKind is preserved

disableSystemArgs
? List.of()
: deriveRowtimeField(callContext, resolvedArgs);
// The on_time rowtime replaces any forwarded time attribute

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in theory we could allow both, there is no limitation of multiple time attributes per row. would maybe simplify things?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants