Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
164 changes: 164 additions & 0 deletions docs/content.zh/docs/libs/state_processor_api.md
Original file line number Diff line number Diff line change
Expand Up @@ -771,3 +771,167 @@ then it can be overridden with the two mentioned config parameters on a per-colu

SHORTCUT: When a complex java class is defined in a column with STRING SQL type then the class instance
`toString` method result will be the column value. This can be useful when a quick explanatory query is required.

## State Catalog

The `state` catalog is a read-only catalog that finds savepoints and checkpoints in one or more directories and exposes their state as tables.
Table schemas are inferred from the snapshot, so you don't need `CREATE TABLE` statements or the job's POJO or Avro classes on the classpath.

The catalog is part of `flink-state-processor-api`. For the SQL Client, copy the jar from `opt/` to `lib/`.

@RocMarshal RocMarshal Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The catalog is part of `flink-state-processor-api`. For the SQL Client, copy the jar from `opt/` to `lib/`.
The catalog is part of `flink-state-processor-api`. For the SQL Client, copy the jar from `${FLINK_HOME}/opt/` to `${FLINK_HOME}/lib/`.


```sql
SET 'execution.runtime-mode' = 'batch';
LOAD MODULE state; -- needed for the metadata view

CREATE CATALOG state WITH (
'type' = 'state',
'directory.my-app' = 's3://my-bucket/flink/my-app',
'directory.staging' = '/savepoints/staging'
);
```

### Catalog Options

| Option | Required | Default | Type | Description |
|-------------------------|----------|---------|---------|-------------|
| type | required | (none) | String | Must be `'state'`. |
| directory.&lt;label&gt; | required | (none) | String | A directory to search for savepoints and checkpoints, on any file system Flink supports. `<label>` becomes the first part of the database names. Set at least one. A directory can't be configured twice or be inside another configured directory. |
| listing-parallelism | optional | 10 | Integer | Maximum number of concurrent directory listings while searching for snapshots. Increase it for remote file systems such as S3. |
| db-name.include-ts | optional | true | Boolean | Whether database names include the snapshot creation time, for example `my-app/2026-07-22T10:30:45Z/savepoint-acce1c-1a2b3c4d5e6f` instead of `my-app/savepoint-acce1c-1a2b3c4d5e6f`. |

The directories are searched again every time databases are listed, for example on `SHOW DATABASES`, so new snapshots show up without recreating the catalog.

### Catalog Layout

Each directory that contains a `_metadata` file becomes a database, and each operator state becomes a table:

- `state` (catalog)
- `my-app/2026-07-22T10:30:45Z/checkpoints/a1b2c3d4e5f6/chk-42` (database: a checkpoint)
- `metadata` (view: one row per operator)
- `uid_counter_keyed` (all keyed state of the operator with UID `counter`)
- `uid_counter_events_keyed_flat` (keyed `ListState` `events`, one row per element)
- `uid_session-window_windowed` (window state of the operator with UID `session-window`)
- `uid_source_offsets_list` (operator `ListState` `offsets`)
- ...
- `my-app/2026-07-23T08:00:00Z/savepoints/savepoint-acce1c-1a2b3c4d5e6f` (database: a savepoint)
- ...
- `staging/2026-07-21T12:15:00Z/savepoint-bd31a2-6f5e4d3c2b1a` (database from the second directory)
- ...

Database names are `<label>/<creation time>/<path relative to the directory>`. The creation time is the modification time of the `_metadata` file in UTC.
Names contain characters such as `/`, `:` and `-`, so quote them with backticks:

```sql
USE CATALOG state;
SHOW DATABASES;

USE `my-app/2026-07-22T10:30:45Z/checkpoints/a1b2c3d4e5f6/chk-42`;
SHOW TABLES;

SELECT * FROM metadata;
SELECT * FROM `uid_counter_keyed` WHERE state_key = 42;
```

### State Type to Table Mapping

`<op>` is `uid_<operator uid>`, or `id_<operator uid hash>` for operators without a UID. `<state>` is the name in the state descriptor.

| State type | Table | Columns | Primary key |
|------------|-------|---------|-------------|
| Keyed state | `<op>_keyed` | `state_key`, then one column per state, for example `my_val_state` (`ValueState<T>` as `T`), `my_list_state` (`ListState<T>` as `ARRAY<T>`), `my_map_state` (`MapState<K, V>` as `MAP<K, V>`) | `state_key` |
| Keyed `ListState`, flattened | `<op>_<state>_keyed_flat` | `state_key`, `list_index`, `list_value` | `state_key`, `list_index` |
| Keyed `MapState`, flattened | `<op>_<state>_keyed_flat` | `state_key`, `map_key`, `map_value` | `state_key`, `map_key` |
| Window state | `<op>_windowed` | `state_key`, `state_window`, then one column per state, for example `my_val_state` (`ValueState<T>` as `T`), `my_list_state` (`ListState<T>` as `ARRAY<T>`), `my_map_state` (`MapState<K, V>` as `MAP<K, V>`) | `state_key`, `state_window` |
| Window `ListState`, flattened | `<op>_<state>_windowed_flat` | `state_key`, `state_window`, `list_index`, `list_value` | `state_key`, `state_window`, `list_index` |
| Window `MapState`, flattened | `<op>_<state>_windowed_flat` | `state_key`, `state_window`, `map_key`, `map_value` | `state_key`, `state_window`, `map_key` |
| Operator `ListState` | `<op>_<state>_list` | one row per element | none |
| Operator union `ListState` | `<op>_<state>_union` | one row per element | none |
| `BroadcastState` | `<op>_<state>_broadcast` | `map_key`, `map_value` | `map_key` |
| Snapshot metadata | `metadata` (view) | same as [`savepoint_metadata`](#metadata) | none |

### Examples

**Keyed state**

```java
stream.keyBy(e -> e.id)
.process(new MyKeyedFunction()) // ValueState<Long> "count", ListState<Long> "events", MapState<Long, Long> "attributes"
.uid("counter");
```

```sql
-- One row per key, ListState and MapState as ARRAY and MAP columns
SELECT state_key, `count`, events, attributes FROM `uid_counter_keyed`;

-- One row per list element of key 3
SELECT state_key, list_index, list_value FROM `uid_counter_events_keyed_flat` WHERE state_key = 3;

-- One row per map entry
SELECT state_key, map_key, map_value FROM `uid_counter_attributes_keyed_flat`;
```

**Window state**

```java
stream.keyBy(t -> t.f0)
.window(EventTimeSessionWindows.withGap(Duration.ofSeconds(5)))
.reduce((a, b) -> Tuple2.of(a.f0, a.f1 + b.f1))
.uid("session-window");

stream.keyBy(t -> t.f0)
.window(TumblingEventTimeWindows.of(Duration.ofSeconds(5)))
.process(new MyProcessWindowFunction())
.uid("tumbling-window");
```

```sql
-- The reduced value of each open window
SELECT state_key, state_window, `window-contents` FROM `uid_session-window_windowed`;

-- The buffered elements of each window, one row per element
SELECT * FROM `uid_tumbling-window_window-contents_windowed_flat` ORDER BY list_index;
```

**Operator and broadcast state**

```java
public class MyStatefulMap extends RichMapFunction<Integer, Integer> implements CheckpointedFunction {
@Override
public void initializeState(FunctionInitializationContext context) throws Exception {
OperatorStateStore store = context.getOperatorStateStore();
listState = store.getListState(new ListStateDescriptor<>("list-values", Integer.class));
unionState = store.getUnionListState(new ListStateDescriptor<>("union-values", MyPojo.class)); // fields: name, score
broadcastState = store.getBroadcastState(new MapStateDescriptor<>("broadcast-values", Integer.class, String.class));
}
...
}

stream.map(new MyStatefulMap()).uid("operator-state-writer");
```

```sql
SELECT * FROM `uid_operator-state-writer_list-values_list`;
-- +-------------+
-- | list-values |
-- +-------------+
-- | 10 |
-- | 20 |
-- | 30 |
-- +-------------+

SELECT * FROM `uid_operator-state-writer_union-values_union`;
-- +------+-------+
-- | name | score |
-- +------+-------+
-- | a | 100 |
-- | b | 200 |
-- +------+-------+

SELECT * FROM `uid_operator-state-writer_broadcast-values_broadcast` ORDER BY map_key;
-- +---------+-----------+
-- | map_key | map_value |
-- +---------+-----------+
-- | 1 | one |
-- | 2 | two |
-- +---------+-----------+
```
5 changes: 5 additions & 0 deletions docs/content.zh/docs/sql/catalogs.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,11 @@ Flink 的 [Hive 文档]({{< ref "docs/connectors/table/hive/overview" >}}) 提

<span class="label label-danger">警告</span> Hive Metastore 以小写形式存储所有元数据对象名称。而 `GenericInMemoryCatalog` 区分大小写。

### StateCatalog

`StateCatalog` 是一个只读 Catalog,它将 savepoint 和 checkpoint 中的状态以表的形式暴露出来,每个快照对应一个数据库。
参考 [State Catalog 文档]({{< ref "docs/libs/state_processor_api" >}}#state-catalog) 获取关于配置 State Catalog 的详细信息。

### 用户自定义 Catalog

Catalog 是可扩展的,用户可以通过实现 `Catalog` 接口来开发自定义 Catalog。
Expand Down
164 changes: 164 additions & 0 deletions docs/content/docs/libs/state_processor_api.md
Original file line number Diff line number Diff line change
Expand Up @@ -782,3 +782,167 @@ using the mapping below.

SHORTCUT: When a complex java class is defined in a column with STRING SQL type then the class instance
`toString` method result will be the column value. This can be useful when a quick explanatory query is required.

## State Catalog

The `state` catalog is a read-only catalog that finds savepoints and checkpoints in one or more directories and exposes their state as tables.
Table schemas are inferred from the snapshot, so you don't need `CREATE TABLE` statements or the job's POJO or Avro classes on the classpath.

The catalog is part of `flink-state-processor-api`. For the SQL Client, copy the jar from `opt/` to `lib/`.

```sql
SET 'execution.runtime-mode' = 'batch';
LOAD MODULE state; -- needed for the metadata view

CREATE CATALOG state WITH (
'type' = 'state',
'directory.my-app' = 's3://my-bucket/flink/my-app',
'directory.staging' = '/savepoints/staging'
);
```

### Catalog Options

| Option | Required | Default | Type | Description |
|-------------------------|----------|---------|---------|-------------|
| type | required | (none) | String | Must be `'state'`. |
| directory.&lt;label&gt; | required | (none) | String | A directory to search for savepoints and checkpoints, on any file system Flink supports. `<label>` becomes the first part of the database names. Set at least one. A directory can't be configured twice or be inside another configured directory. |
| listing-parallelism | optional | 10 | Integer | Maximum number of concurrent directory listings while searching for snapshots. Increase it for remote file systems such as S3. |
| db-name.include-ts | optional | true | Boolean | Whether database names include the snapshot creation time, for example `my-app/2026-07-22T10:30:45Z/savepoint-acce1c-1a2b3c4d5e6f` instead of `my-app/savepoint-acce1c-1a2b3c4d5e6f`. |

The directories are searched again every time databases are listed, for example on `SHOW DATABASES`, so new snapshots show up without recreating the catalog.

### Catalog Layout

Each directory that contains a `_metadata` file becomes a database, and each operator state becomes a table:

- `state` (catalog)
- `my-app/2026-07-22T10:30:45Z/checkpoints/a1b2c3d4e5f6/chk-42` (database: a checkpoint)
- `metadata` (view: one row per operator)
- `uid_counter_keyed` (all keyed state of the operator with UID `counter`)
- `uid_counter_events_keyed_flat` (keyed `ListState` `events`, one row per element)
- `uid_session-window_windowed` (window state of the operator with UID `session-window`)
- `uid_source_offsets_list` (operator `ListState` `offsets`)
- ...
- `my-app/2026-07-23T08:00:00Z/savepoints/savepoint-acce1c-1a2b3c4d5e6f` (database: a savepoint)
- ...
- `staging/2026-07-21T12:15:00Z/savepoint-bd31a2-6f5e4d3c2b1a` (database from the second directory)
- ...

Database names are `<label>/<creation time>/<path relative to the directory>`. The creation time is the modification time of the `_metadata` file in UTC.
Names contain characters such as `/`, `:` and `-`, so quote them with backticks:

```sql
USE CATALOG state;
SHOW DATABASES;

USE `my-app/2026-07-22T10:30:45Z/checkpoints/a1b2c3d4e5f6/chk-42`;
SHOW TABLES;

SELECT * FROM metadata;
SELECT * FROM `uid_counter_keyed` WHERE state_key = 42;
```

### State Type to Table Mapping

`<op>` is `uid_<operator uid>`, or `id_<operator uid hash>` for operators without a UID. `<state>` is the name in the state descriptor.

| State type | Table | Columns | Primary key |
|------------|-------|---------|-------------|
| Keyed state | `<op>_keyed` | `state_key`, then one column per state, for example `my_val_state` (`ValueState<T>` as `T`), `my_list_state` (`ListState<T>` as `ARRAY<T>`), `my_map_state` (`MapState<K, V>` as `MAP<K, V>`) | `state_key` |
| Keyed `ListState`, flattened | `<op>_<state>_keyed_flat` | `state_key`, `list_index`, `list_value` | `state_key`, `list_index` |
| Keyed `MapState`, flattened | `<op>_<state>_keyed_flat` | `state_key`, `map_key`, `map_value` | `state_key`, `map_key` |
| Window state | `<op>_windowed` | `state_key`, `state_window`, then one column per state, for example `my_val_state` (`ValueState<T>` as `T`), `my_list_state` (`ListState<T>` as `ARRAY<T>`), `my_map_state` (`MapState<K, V>` as `MAP<K, V>`) | `state_key`, `state_window` |
| Window `ListState`, flattened | `<op>_<state>_windowed_flat` | `state_key`, `state_window`, `list_index`, `list_value` | `state_key`, `state_window`, `list_index` |
| Window `MapState`, flattened | `<op>_<state>_windowed_flat` | `state_key`, `state_window`, `map_key`, `map_value` | `state_key`, `state_window`, `map_key` |
| Operator `ListState` | `<op>_<state>_list` | one row per element | none |
| Operator union `ListState` | `<op>_<state>_union` | one row per element | none |
| `BroadcastState` | `<op>_<state>_broadcast` | `map_key`, `map_value` | `map_key` |
| Snapshot metadata | `metadata` (view) | same as [`savepoint_metadata`](#metadata) | none |

### Examples

**Keyed state**

```java
stream.keyBy(e -> e.id)
.process(new MyKeyedFunction()) // ValueState<Long> "count", ListState<Long> "events", MapState<Long, Long> "attributes"
.uid("counter");
```

```sql
-- One row per key, ListState and MapState as ARRAY and MAP columns
SELECT state_key, `count`, events, attributes FROM `uid_counter_keyed`;

-- One row per list element of key 3
SELECT state_key, list_index, list_value FROM `uid_counter_events_keyed_flat` WHERE state_key = 3;

-- One row per map entry
SELECT state_key, map_key, map_value FROM `uid_counter_attributes_keyed_flat`;
```

**Window state**

```java
stream.keyBy(t -> t.f0)
.window(EventTimeSessionWindows.withGap(Duration.ofSeconds(5)))
.reduce((a, b) -> Tuple2.of(a.f0, a.f1 + b.f1))
.uid("session-window");

stream.keyBy(t -> t.f0)
.window(TumblingEventTimeWindows.of(Duration.ofSeconds(5)))
.process(new MyProcessWindowFunction())
.uid("tumbling-window");
```

```sql
-- The reduced value of each open window
SELECT state_key, state_window, `window-contents` FROM `uid_session-window_windowed`;

-- The buffered elements of each window, one row per element
SELECT * FROM `uid_tumbling-window_window-contents_windowed_flat` ORDER BY list_index;
```

**Operator and broadcast state**

```java
public class MyStatefulMap extends RichMapFunction<Integer, Integer> implements CheckpointedFunction {
@Override
public void initializeState(FunctionInitializationContext context) throws Exception {
OperatorStateStore store = context.getOperatorStateStore();
listState = store.getListState(new ListStateDescriptor<>("list-values", Integer.class));
unionState = store.getUnionListState(new ListStateDescriptor<>("union-values", MyPojo.class)); // fields: name, score
broadcastState = store.getBroadcastState(new MapStateDescriptor<>("broadcast-values", Integer.class, String.class));
}
...
}

stream.map(new MyStatefulMap()).uid("operator-state-writer");
```

```sql
SELECT * FROM `uid_operator-state-writer_list-values_list`;
-- +-------------+
-- | list-values |
-- +-------------+
-- | 10 |
-- | 20 |
-- | 30 |
-- +-------------+

SELECT * FROM `uid_operator-state-writer_union-values_union`;
-- +------+-------+
-- | name | score |
-- +------+-------+
-- | a | 100 |
-- | b | 200 |
-- +------+-------+

SELECT * FROM `uid_operator-state-writer_broadcast-values_broadcast` ORDER BY map_key;
-- +---------+-----------+
-- | map_key | map_value |
-- +---------+-----------+
-- | 1 | one |
-- | 2 | two |
-- +---------+-----------+
```
5 changes: 5 additions & 0 deletions docs/content/docs/sql/catalogs.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,11 @@ Flink's [Hive documentation]({{< ref "docs/connectors/table/hive/overview" >}})
{{< hint warning >}} The Hive Metastore stores all meta-object names in lower case. This is unlike `GenericInMemoryCatalog` which is case-sensitive
{{< /hint >}}

### StateCatalog

The `StateCatalog` is a read-only catalog that exposes the state in savepoints and checkpoints as tables, one database per snapshot.
See the [State Catalog documentation]({{< ref "docs/libs/state_processor_api" >}}#state-catalog) for more details on setting up the catalog.

### User-Defined Catalog

Catalogs are pluggable and users can develop custom catalogs by implementing the `Catalog` interface.
Expand Down