Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ The built-in process table functions (PTFs) take a table as input and return a t
|---------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `FROM_CHANGELOG(input => TABLE t [, ...])` | Converts an append-only table with an explicit operation column into a dynamic table. See Changelog Conversion for the full list of arguments, semantics, and usage. |
| `TO_CHANGELOG(input => TABLE t [, ...])` | Converts a dynamic table into an append-only table with an explicit operation column. See Changelog Conversion for the full list of arguments, semantics, and usage. |
| `DEDUPLICATE(input => TABLE t PARTITION BY k [, ...])` | Keeps the first or the last row per key of an append-only table. See [Deduplication]({{< ref "docs/sql/reference/queries/deduplication" >}}#deduplicate) for the full list of arguments, semantics, and usage. |
| `SNAPSHOT(input => TABLE t [, ...])` | Returns the current state of a dynamic table `t`. `SNAPSHOT` can only be used in a `LATERAL` context and not as a stand-alone table function. See [LATERAL SNAPSHOT join]({{< ref "docs/sql/reference/queries/joins" >}}#lateral-snapshot-join) for the full list of arguments, the join semantics, and usage. |
| `ML_PREDICT(TABLE t, MODEL m, DESCRIPTOR(...) [, ...])` | Applies the model `m` to the columns of table `t` that the `DESCRIPTOR` names, and returns the columns of `t` plus the prediction columns of the model. See [Model Inference]({{< ref "docs/sql/reference/queries/model-inference" >}}#ml_predict-函数) for the full list of arguments, semantics, and usage. |

Expand Down
139 changes: 139 additions & 0 deletions docs/content.zh/docs/sql/reference/queries/deduplication.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,4 +69,143 @@ FROM (
WHERE row_num = 1
```

## DEDUPLICATE

{{< label Streaming >}}

The `DEDUPLICATE` process table function (PTF) keeps the first or the last row per key of an append-only table, as long as the state of the key is retained. Rows are ordered by arrival or by `sequence_by` columns.

### Syntax

```sql
SELECT * FROM DEDUPLICATE(
input => TABLE source_table PARTITION BY key_col[, ...],
[keep => 'first' | 'last',]
[sequence_by => DESCRIPTOR(col[, ...]),]
[state_ttl => INTERVAL '1' HOUR,]
[reset_ttl_on_duplicate => true | false,]
[uid => 'my-id']
)
```

### Parameters

| Parameter | Required | Description |
|:----------|:---------|:------------|
| `input` | Yes | The append-only input table. `PARTITION BY` is required and its columns are the key of the output.|
| `keep` | No | `'first'` (default) keeps the earliest arrival or the smallest `sequence_by` value, `'last'` keeps the latest arrival or the greatest. Must be a literal. Arrival order is not deterministic across reruns, use `sequence_by` for a repeatable result. |
| `sequence_by` | No | One or more columns that order the records of a key instead of their arrival. The first column decides, later columns break ties, and `NULL` orders below every other value. On a tie, the later arrival wins for `'last'` and the earlier one for `'first'`. Columns must be numeric, string, binary, boolean or date-time, and not part of `PARTITION BY`. |
| `state_ttl` | No | How long the state of an idle key is retained, as an `INTERVAL` literal. Defaults to `table.exec.state.ttl`, and `INTERVAL '0' SECOND` disables expiry. After the state of a key expires, its next record is emitted as `+I` again. |
| `reset_ttl_on_duplicate` | No | Whether a record that is dropped refreshes the TTL of its key. Defaults to `true`. Has no effect without a TTL. |


### Output

The output columns are the `PARTITION BY` columns, in the order listed, followed by the remaining input columns. Time attributes are emitted as regular timestamps.

The result has one row per key. When a later row replaces the kept one, the row of that key is updated, so the result must be written to a table that accepts updates, for example one with a primary key on the `PARTITION BY` columns. Currently this holds for every `keep` mode, including `keep => 'first'` without `sequence_by`, which never replaces a row.

{{< hint info >}}
The first record of a key is emitted as `+I`. A record that replaces the kept one is emitted as `+U`, preceded by `-U` if the consumer needs retractions. A record that equals the kept one, or does not replace it, emits nothing. See [Updates and Changelogs]({{< ref "docs/dev/table/functions/ptfs" >}}#updates-and-changelogs).
{{< /hint >}}

### Examples

#### Keep the first or the last record

Rows of `user_events` arrive in this order:

| `user_events` |
|:----------------|
| `(u1, login)` |
| `(u2, login)` |
| `(u1, click)` |

`keep` defaults to `'first'`, which keeps the first row of each user:

```sql
SELECT * FROM DEDUPLICATE(
input => TABLE user_events PARTITION BY user_id)
```

| Result |
|:----------------|
| `(u1, login)` |
| `(u2, login)` |

With `keep => 'last'`, a later row replaces the kept one:

```sql
SELECT * FROM DEDUPLICATE(
input => TABLE user_events PARTITION BY user_id,
keep => 'last')
```

| Result |
|:----------------|
| `(u1, click)` |
| `(u2, login)` |

#### Order by a column

`sequence_by` ranks the rows of a key by a column instead of by arrival, so a late, older row does not replace a newer one:

| `customer_changes` |
|:----------------------------|
| `(c1, Berlin, 100)` |
| `(c1, Munich, 120)` |
| `(c1, Hamburg, 110)` |

```sql
SELECT * FROM DEDUPLICATE(
input => TABLE customer_changes PARTITION BY customer_id,
keep => 'last',
sequence_by => DESCRIPTOR(lsn),
state_ttl => INTERVAL '1' DAY)
```

| Result |
|:----------------------|
| `(c1, Munich, 120)` |

`(c1, Hamburg, 110)` is dropped because its `lsn` is smaller than that of the kept row.

#### Composite key

The columns of `PARTITION BY` come first in the output, in the order listed:

| `sessions` (`user_id`, `page`, `device`) |
|:-----------------------------------------|
| `(u1, a, web)` |
| `(u1, b, mobile)` |
| `(u1, c, web)` |

```sql
SELECT * FROM DEDUPLICATE(
input => TABLE sessions PARTITION BY (device, user_id),
keep => 'last')
```

| Result (`device`, `user_id`, `page`) |
|:-------------------------------------|
| `(web, u1, c)` |
| `(mobile, u1, b)` |

#### Table API

```java
// Keep the first record of each payment
Table result = payments
.partitionBy($("payment_id"))
.deduplicate();

// Keep the record with the greatest lsn per customer
Table result = customerChanges
.partitionBy($("customer_id"))
.deduplicate(
lit("last").asArgument("keep"),
descriptor("lsn").asArgument("sequence_by"),
lit(Duration.ofDays(1)).asArgument("state_ttl"));
```

{{< top >}}
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ The built-in process table functions (PTFs) take a table as input and return a t
|---------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `FROM_CHANGELOG(input => TABLE t [, ...])` | Converts an append-only table with an explicit operation column into a dynamic table. See [Changelog Conversion]({{< ref "docs/sql/reference/queries/changelog" >}}#from_changelog) for the full list of arguments, semantics, and usage. |
| `TO_CHANGELOG(input => TABLE t [, ...])` | Converts a dynamic table into an append-only table with an explicit operation column. See [Changelog Conversion]({{< ref "docs/sql/reference/queries/changelog" >}}#to_changelog) for the full list of arguments, semantics, and usage. |
| `DEDUPLICATE(input => TABLE t PARTITION BY k [, ...])` | Keeps the first or the last row per key of an append-only table. See [Deduplication]({{< ref "docs/sql/reference/queries/deduplication" >}}#deduplicate) for the full list of arguments, semantics, and usage. |
Comment thread
VasShabu marked this conversation as resolved.
Resolved
| `SNAPSHOT(input => TABLE t [, ...])` | Returns the current state of a dynamic table `t`. `SNAPSHOT` can only be used in a `LATERAL` context and not as a stand-alone table function. See [LATERAL SNAPSHOT join]({{< ref "docs/sql/reference/queries/joins" >}}#lateral-snapshot-join) for the full list of arguments, the join semantics, and usage. |
| `ML_PREDICT(TABLE t, MODEL m, DESCRIPTOR(...) [, ...])` | Applies the model `m` to the columns of table `t` that the `DESCRIPTOR` names, and returns the columns of `t` plus the prediction columns of the model. See [Model Inference]({{< ref "docs/sql/reference/queries/model-inference" >}}#ml_predict-function) for the full list of arguments, semantics, and usage. |

Expand Down
139 changes: 139 additions & 0 deletions docs/content/docs/sql/reference/queries/deduplication.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,4 +69,143 @@ FROM (
WHERE row_num = 1
```

## DEDUPLICATE

{{< label Streaming >}}

The `DEDUPLICATE` process table function (PTF) keeps the first or the last row per key of an append-only table, as long as the state of the key is retained. Rows are ordered by arrival or by `sequence_by` columns.

### Syntax

```sql
SELECT * FROM DEDUPLICATE(
input => TABLE source_table PARTITION BY key_col[, ...],
[keep => 'first' | 'last',]
[sequence_by => DESCRIPTOR(col[, ...]),]
[state_ttl => INTERVAL '1' HOUR,]
[reset_ttl_on_duplicate => true | false,]
[uid => 'my-id']
)
```

### Parameters

| Parameter | Required | Description |
|:----------|:---------|:------------|
| `input` | Yes | The append-only input table. `PARTITION BY` is required and its columns are the key of the output.|
| `keep` | No | `'first'` (default) keeps the earliest arrival or the smallest `sequence_by` value, `'last'` keeps the latest arrival or the greatest. Must be a literal. Arrival order is not deterministic across reruns, use `sequence_by` for a repeatable result. |
| `sequence_by` | No | One or more columns that order the records of a key instead of their arrival. The first column decides, later columns break ties, and `NULL` orders below every other value. On a tie, the later arrival wins for `'last'` and the earlier one for `'first'`. Columns must be numeric, string, binary, boolean or date-time, and not part of `PARTITION BY`. |
| `state_ttl` | No | How long the state of an idle key is retained, as an `INTERVAL` literal. Defaults to `table.exec.state.ttl`, and `INTERVAL '0' SECOND` disables expiry. After the state of a key expires, its next record is emitted as `+I` again. |
| `reset_ttl_on_duplicate` | No | Whether a record that is dropped refreshes the TTL of its key. Defaults to `true`. Has no effect without a TTL. |


### Output

The output columns are the `PARTITION BY` columns, in the order listed, followed by the remaining input columns. Time attributes are emitted as regular timestamps.

The result has one row per key. When a later row replaces the kept one, the row of that key is updated, so the result must be written to a table that accepts updates, for example one with a primary key on the `PARTITION BY` columns. Currently this holds for every `keep` mode, including `keep => 'first'` without `sequence_by`, which never replaces a row.

{{< hint info >}}
The first record of a key is emitted as `+I`. A record that replaces the kept one is emitted as `+U`, preceded by `-U` if the consumer needs retractions. A record that equals the kept one, or does not replace it, emits nothing. See [Updates and Changelogs]({{< ref "docs/dev/table/functions/ptfs" >}}#updates-and-changelogs).
{{< /hint >}}

### Examples

#### Keep the first or the last record

Rows of `user_events` arrive in this order:

| `user_events` |
|:----------------|
| `(u1, login)` |
| `(u2, login)` |
| `(u1, click)` |

`keep` defaults to `'first'`, which keeps the first row of each user:

```sql
SELECT * FROM DEDUPLICATE(
input => TABLE user_events PARTITION BY user_id)
```

| Result |
|:----------------|
| `(u1, login)` |
| `(u2, login)` |

With `keep => 'last'`, a later row replaces the kept one:

```sql
SELECT * FROM DEDUPLICATE(
input => TABLE user_events PARTITION BY user_id,
keep => 'last')
```

| Result |
|:----------------|
| `(u1, click)` |
| `(u2, login)` |

#### Order by a column

`sequence_by` ranks the rows of a key by a column instead of by arrival, so a late, older row does not replace a newer one:

| `customer_changes` |
|:----------------------------|
| `(c1, Berlin, 100)` |
| `(c1, Munich, 120)` |
| `(c1, Hamburg, 110)` |

```sql
SELECT * FROM DEDUPLICATE(
input => TABLE customer_changes PARTITION BY customer_id,
keep => 'last',
sequence_by => DESCRIPTOR(lsn),
state_ttl => INTERVAL '1' DAY)
```

| Result |
|:----------------------|
| `(c1, Munich, 120)` |

`(c1, Hamburg, 110)` is dropped because its `lsn` is smaller than that of the kept row.

#### Composite key

The columns of `PARTITION BY` come first in the output, in the order listed:

| `sessions` (`user_id`, `page`, `device`) |
|:-----------------------------------------|
| `(u1, a, web)` |
| `(u1, b, mobile)` |
| `(u1, c, web)` |

```sql
SELECT * FROM DEDUPLICATE(
input => TABLE sessions PARTITION BY (device, user_id),
keep => 'last')
```

| Result (`device`, `user_id`, `page`) |
|:-------------------------------------|
| `(web, u1, c)` |
| `(mobile, u1, b)` |

#### Table API

```java
// Keep the first record of each payment
Table result = payments
.partitionBy($("payment_id"))
.deduplicate();

// Keep the record with the greatest lsn per customer
Table result = customerChanges
.partitionBy($("customer_id"))
.deduplicate(
lit("last").asArgument("keep"),
descriptor("lsn").asArgument("sequence_by"),
lit(Duration.ofDays(1)).asArgument("state_ttl"));
```

{{< top >}}
Original file line number Diff line number Diff line change
Expand Up @@ -290,4 +290,35 @@ public interface PartitionedTable {
* @see #orderBy(Expression...)
*/
Table fromChangelog(Expression... arguments);

/**
* Keeps one row per partition key using the built-in {@code DEDUPLICATE} process table
* function. The result is an updating table with the partition columns as its upsert key.
*
* <p>{@code keep => 'first'} (default) keeps the earliest arrival or the smallest {@code
* sequence_by} value, {@code keep => 'last'} keeps the latest arrival or the greatest value.
*
* <p>Examples:
*
* <pre>{@code
* // Keep the first delivery of each payment
* Table result = payments
* .partitionBy($("payment_id"))
* .deduplicate();
*
* // Keep the version with the greatest 'lsn' per customer, with bounded state retention
* Table result = customerChanges
* .partitionBy($("customer_id"))
* .deduplicate(
* lit("last").asArgument("keep"),
* descriptor("lsn").asArgument("sequence_by"),
* lit(Duration.ofDays(1)).asArgument("state_ttl"));
* }</pre>
*
* @param arguments optional named arguments for {@code keep}, {@code sequence_by}, {@code
* state_ttl}, {@code reset_ttl_on_duplicate}, and {@code uid}
* @return an updating {@link Table} with output schema {@code [partition_keys,
* non_partition_input_columns]}
*/
Table deduplicate(Expression... arguments);
}
Original file line number Diff line number Diff line change
Expand Up @@ -962,6 +962,11 @@ public Table fromChangelog(final Expression... arguments) {
BuiltInFunctionDefinitions.FROM_CHANGELOG.getName(), (Object[]) arguments);
}

@Override
public Table deduplicate(final Expression... arguments) {
return process(BuiltInFunctionDefinitions.DEDUPLICATE.getName(), (Object[]) arguments);
}

private QueryOperation createQueryOperation() {
if (orderKeys.isEmpty()) {
return table.operationTreeBuilder.partition(partitionKeys, table.operationTree);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
import org.apache.flink.table.types.inference.TraitCondition;
import org.apache.flink.table.types.inference.TypeStrategies;
import org.apache.flink.table.types.inference.strategies.ArrayOfStringArgumentTypeStrategy;
import org.apache.flink.table.types.inference.strategies.DeduplicateTypeStrategy;
import org.apache.flink.table.types.inference.strategies.FromChangelogTypeStrategy;
import org.apache.flink.table.types.inference.strategies.SpecificInputTypeStrategies;
import org.apache.flink.table.types.inference.strategies.SpecificTypeStrategies;
Expand Down Expand Up @@ -939,6 +940,30 @@ ANY, and(logical(LogicalTypeRoot.BOOLEAN), LITERAL)
"org.apache.flink.table.runtime.functions.ptf.FromChangelogFunction")
.build();

public static final BuiltInFunctionDefinition DEDUPLICATE =
BuiltInFunctionDefinition.newBuilder()
.name("DEDUPLICATE")
.kind(PROCESS_TABLE)
.staticArguments(
StaticArgument.table(
"input",
Row.class,
false,
EnumSet.of(
StaticArgumentTrait.TABLE,
StaticArgumentTrait.SET_SEMANTIC_TABLE)),
StaticArgument.scalar("keep", DataTypes.STRING(), true),
StaticArgument.scalar("sequence_by", DataTypes.DESCRIPTOR(), true),
StaticArgument.scalar(
"state_ttl", DataTypes.INTERVAL(DataTypes.SECOND()), true),
StaticArgument.scalar(
"reset_ttl_on_duplicate", DataTypes.BOOLEAN(), true))
.changelogModeStrategy(DeduplicateTypeStrategy.CHANGELOG_MODE_STRATEGY)
.inputTypeStrategy(DeduplicateTypeStrategy.INPUT_TYPE_STRATEGY)
.outputTypeStrategy(DeduplicateTypeStrategy.OUTPUT_TYPE_STRATEGY)
.runtimeClass("org.apache.flink.table.runtime.functions.ptf.Deduplicate")
.build();

/**
* Built-in proxy function for the LATERAL SNAPSHOT temporal join.
*
Expand Down
Loading