Skip to content

cdc: add mode support for debezium #23912

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wk989898 wants to merge 1 commit into
base: master
Choose a base branch
from
+75 −2
Open

cdc: add mode support for debezium #23912

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions ticdc/ticdc-changefeed-config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -353,6 +353,27 @@ The following configuration parameters control the sending behavior of bootstrap
- Controls whether to output the value before the row data changes. The default value is true. When it is disabled, the `UPDATE` event does not output the "before" field.
- Default value: `true`

##### `decimal-handling-mode`

- Controls how TiCDC encodes `DECIMAL` and `NUMERIC` values in Debezium JSON. `double` encodes values as float64 JSON numbers, which might lose precision. `string` encodes values as strings to preserve the full precision.
- Default value: `double`
- Value options: `double`, `string`
- This parameter only takes effect with `protocol=debezium`. The `debezium-decimal-handling-mode` parameter in `sink-uri` takes precedence over this setting. For details and examples, see [Numeric and binary handling](/ticdc/ticdc-debezium.md#numeric-and-binary-handling).

##### `bigint-unsigned-handling-mode`

- Controls how TiCDC encodes `BIGINT UNSIGNED` values in Debezium JSON. `long` encodes values as signed 64-bit JSON numbers; values greater than `9223372036854775807` wrap to negative numbers. `string` encodes values as strings to preserve the full unsigned value.
- Default value: `long`
- Value options: `long`, `string`
- This parameter only takes effect with `protocol=debezium`. The `debezium-bigint-unsigned-handling-mode` parameter in `sink-uri` takes precedence over this setting. For details and examples, see [Numeric and binary handling](/ticdc/ticdc-debezium.md#numeric-and-binary-handling).

##### `binary-handling-mode`

- Controls how TiCDC encodes binary column values in Debezium JSON. `base64`, `base64-url-safe`, and `hex` encode values as strings in the corresponding format. `bytes` uses a `bytes` schema type, but the JSON value remains a Base64-encoded string. This parameter does not affect `BIT` columns.
- Default value: `base64`
- Value options: `bytes`, `base64`, `base64-url-safe`, `hex`
- This parameter only takes effect with `protocol=debezium`. The `debezium-binary-handling-mode` parameter in `sink-uri` takes precedence over this setting. For details and examples, see [Numeric and binary handling](/ticdc/ticdc-debezium.md#numeric-and-binary-handling).

### consistent

Specifies the replication consistency configurations for a changefeed when using the redo log. For more information, see [Eventually consistent replication in disaster scenarios](/ticdc/ticdc-sink-to-mysql.md#eventually-consistent-replication-in-disaster-scenarios).
Expand Down
53 changes: 51 additions & 2 deletions ticdc/ticdc-debezium.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,55 @@ The Debezium output format contains the schema information of the current row so

In addition, the original Debezium format does not include important fields such as the unique transaction identifier of the `CommitTS` in TiDB. To ensure data integrity, TiCDC adds two fields, `CommitTs` and `ClusterID`, to the Debezium format to identify the relevant information of TiDB data changes.

### Numeric and binary handling

For Debezium JSON (`protocol=debezium`), you can configure how TiCDC encodes decimal values, unsigned BIGINT values, and binary data. Set the following options in the [`[sink.debezium]`](/ticdc/ticdc-changefeed-config.md#sinkdebezium) section of the changefeed configuration file or use the corresponding `sink-uri` parameters. If you set an option in both places, the `sink-uri` value takes precedence. These options do not apply to `protocol=debezium-avro`.

@coderabbitai coderabbitai Bot Sep 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

#!/bin/bash
set -euo pipefail

rg -n -S \
  'debezium-avro|protocol\s*[:=].*(avro|debezium)|Protocol.*(Avro|Debezium)' \
  --glob '*.go' --glob '*.md' .

Repository: pingcap/docs

Length of output: 7178

🏁 Script executed:

set -euo pipefail
printf '%s\n' '--- ticdc/ticdc-changefeed-config.md ---'
sed -n '188,205p' ticdc/ticdc-changefeed-config.md
printf '%s\n' '--- ticdc/ticdc-sink-to-kafka.md ---'
sed -n '68,84p' ticdc/ticdc-sink-to-kafka.md
printf '%s\n' '--- ticdc/ticdc-debezium.md ---'
sed -n '28,40p' ticdc/ticdc-debezium.md

Repository: pingcap/docs

Length of output: 6134

Use the supported Avro protocol identifier. The supported value is protocol=avro, not protocol=debezium-avro.

Suggested change
For Debezium JSON (`protocol=debezium`), you can configure how TiCDC encodes decimal values, unsigned BIGINT values, and binary data. Set the following options in the [`[sink.debezium]`](/ticdc/ticdc-changefeed-config.md#sinkdebezium) section of the changefeed configuration file or use the corresponding `sink-uri` parameters. If you set an option in both places, the `sink-uri` value takes precedence. These options do not apply to `protocol=debezium-avro`.
For Debezium JSON (`protocol=debezium`), you can configure how TiCDC encodes decimal values, unsigned BIGINT values, and binary data. Set the following options in the [`[sink.debezium]`](/ticdc/ticdc-changefeed-config.md#sinkdebezium) section of the changefeed configuration file or use the corresponding `sink-uri` parameters. If you set an option in both places, the `sink-uri` value takes precedence. These options do not apply to `protocol=avro`.

Source: MCP tools


| Configuration option | Sink URI parameter | Default value | Value options |
| :--- | :--- | :--- | :--- |
| `decimal-handling-mode` | `debezium-decimal-handling-mode` | `double` | `double`, `string` |
| `bigint-unsigned-handling-mode` | `debezium-bigint-unsigned-handling-mode` | `long` | `long`, `string` |
| `binary-handling-mode` | `debezium-binary-handling-mode` | `base64` | `bytes`, `base64`, `base64-url-safe`, `hex` |

The default values preserve the existing encoding behavior. To preserve the full precision of `DECIMAL` and `BIGINT UNSIGNED` values, set the corresponding handling modes to `string`:

- `decimal-handling-mode = "double"` encodes `DECIMAL` and `NUMERIC` values as float64 JSON numbers with schema type `double`. High-precision values might lose precision. With `string`, TiCDC encodes the values as decimal strings and uses schema type `string`.
- `bigint-unsigned-handling-mode = "long"` encodes `BIGINT UNSIGNED` values as signed 64-bit JSON numbers with schema type `int64`. Values greater than `9223372036854775807` wrap to negative numbers. For example, `18446744073709551615` is encoded as `-1`. With `string`, TiCDC preserves the full unsigned value as a string, such as `"18446744073709551615"`, and uses schema type `string`. Signed `BIGINT` columns are unaffected by this option.

The `binary-handling-mode` option controls the encoding of string-like columns with the BINARY flag, such as `BINARY`, `VARBINARY`, and `BLOB`. For example, TiCDC encodes the binary value `X'FBFF'` as follows:

| Mode | JSON value | Schema type |
| :--- | :--- | :--- |
| `base64` | `"+/8="` | `string` |
| `bytes` | `"+/8="` | `bytes` |
| `base64-url-safe` | `"-_8="` | `string` |
| `hex` | `"fbff"` | `string` |

In `bytes` mode, the JSON value is still a Base64-encoded string; only the schema type changes. This option does not change the encoding of `BIT` columns or columns without the BINARY flag.

These handling modes apply to column values in message keys, `payload.before`, and `payload.after`, and to column defaults in the schema. Setting `debezium-disable-schema=true` omits the schema but does not change the configured payload encoding.

For example, save the following configuration as `changefeed.toml` to preserve numeric precision and encode binary data as hexadecimal strings:

```toml
[sink.debezium]
decimal-handling-mode = "string"
bigint-unsigned-handling-mode = "string"
binary-handling-mode = "hex"
```

Then create the changefeed using this file:

```shell
cdc cli changefeed create --server=http://127.0.0.1:8300 --changefeed-id="kafka-debezium" --sink-uri="kafka://127.0.0.1:9092/topic-name?protocol=debezium" --config=changefeed.toml
```

Alternatively, set the options directly in `sink-uri`:

```shell
cdc cli changefeed create --server=http://127.0.0.1:8300 --changefeed-id="kafka-debezium" --sink-uri="kafka://127.0.0.1:9092/topic-name?protocol=debezium&debezium-decimal-handling-mode=string&debezium-bigint-unsigned-handling-mode=string&debezium-binary-handling-mode=hex"
```

## Message format definition

This section describes the message formats of DDL events, DML events and WATERMARK events.
Expand Down Expand Up @@ -785,9 +834,9 @@ The data format mapping in the TiCDC Debezium message basically follows the [Deb

- Currently, TiDB does not support spatial data types, including GEOMETRY, LINESTRING, POLYGON, MULTIPOINT, MULTILINESTRING, MULTIPOLYGON, and GEOMETRYCOLLECTION.

- For string-like data types, including Varchar, String, VarString, TinyBlob, MediumBlob, BLOB, and LongBlob, when the column has the BINARY flag, TiCDC encodes it as a String type after encoding it in Base64; when the column does not have the BINARY flag, TiCDC encodes it directly as a String type. The native Debezium Connector encodes it in different ways according to `binary.handling.mode`.
- For string-like data types, including Varchar, String, VarString, TinyBlob, MediumBlob, BLOB, and LongBlob, when the column has the BINARY flag, TiCDC encodes it according to `binary-handling-mode`, which defaults to Base64 encoding with schema type `string`; when the column does not have the BINARY flag, TiCDC encodes it directly as a String type. The native Debezium Connector uses `binary.handling.mode`. For the supported TiCDC modes, see [Numeric and binary handling](#numeric-and-binary-handling).

- For the Decimal data type, including DECIMAL and NUMERIC, TiCDC uses the float64 type to represent it. The native Debezium Connector encodes it in float32 or float64 according to the different precision of the data type.
- For the Decimal data type, including DECIMAL and NUMERIC, TiCDC uses the float64 type by default. Set `decimal-handling-mode` to `string` to preserve the full precision. For `BIGINT UNSIGNED`, TiCDC uses signed 64-bit integers by default. Set `bigint-unsigned-handling-mode` to `string` to preserve the full unsigned value. For details, see [Numeric and binary handling](#numeric-and-binary-handling).

- TiCDC converts REAL to DOUBLE, and converts BOOLEAN to TINYINT(1) when the length is one.

Expand Down
3 changes: 3 additions & 0 deletions ticdc/ticdc-sink-to-kafka.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,9 @@ The following are descriptions of sink URI parameters and values that can be con
| `write-timeout` | The timeout in sending a request to the downstream Kafka. The default value is `10s`. |
| `avro-decimal-handling-mode` | Only effective with the `avro` protocol. Determines how Avro handles the DECIMAL field. The value can be `string` or `precise`, indicating either mapping the DECIMAL field to a string or a precise floating number. |
| `avro-bigint-unsigned-handling-mode` | Only effective with the `avro` protocol. Determines how Avro handles the BIGINT UNSIGNED field. The value can be `string` or `long`, indicating either mapping the BIGINT UNSIGNED field to a 64-bit signed number or a string. |
| `debezium-decimal-handling-mode` | Only effective with `protocol=debezium` (JSON). Controls how TiCDC encodes `DECIMAL` and `NUMERIC` values. Value options are `double` (default) and `string`. Use `string` to preserve the full precision. This parameter takes precedence over `sink.debezium.decimal-handling-mode`. See [Numeric and binary handling](/ticdc/ticdc-debezium.md#numeric-and-binary-handling). |
| `debezium-bigint-unsigned-handling-mode` | Only effective with `protocol=debezium` (JSON). Controls how TiCDC encodes `BIGINT UNSIGNED` values. Value options are `long` (default) and `string`. Use `string` to preserve the full unsigned value. This parameter takes precedence over `sink.debezium.bigint-unsigned-handling-mode`. See [Numeric and binary handling](/ticdc/ticdc-debezium.md#numeric-and-binary-handling). |
| `debezium-binary-handling-mode` | Only effective with `protocol=debezium` (JSON). Controls how TiCDC encodes binary column values. Value options are `bytes`, `base64` (default), `base64-url-safe`, and `hex`. In `bytes` mode, the JSON value is still Base64-encoded. This parameter takes precedence over `sink.debezium.binary-handling-mode`. See [Numeric and binary handling](/ticdc/ticdc-debezium.md#numeric-and-binary-handling). |

### Best practices

Expand Down
Loading