sink: add Debezium numeric and binary handling modes (#6263)

[ti-chi-bot]

This is an automated cherry-pick of #6263

What problem does this PR solve?

Issue Number: close #6253

What is changed and how it works?

• Add lossless string encoding for Debezium JSON DECIMAL and BIGINT UNSIGNED, and configurable binary handling modes.
• Keep existing defaults and update schemas, encoding/decoding, and CLI/API configuration accordingly.
• Add unit tests and Kafka integration coverage for Debezium JSON and Debezium Avro.

Check List

Tests

• Unit test

• Integration test
  The debezium01/02/03 integration tests compare TiCDC's Debezium JSON output with native Debezium, checking both message schemas and payloads under
  matching configurations:

  • debezium01: binary.handling.mode=bytes
  • debezium02: binary.handling.mode=base64-url-safe
  • debezium03: binary.handling.mode=hex

  All three use decimal.handling.mode=string and bigint.unsigned.handling.mode=long, covering binary encoding, string-encoded decimals, and unsigned BIGINT compatibility. All three tests have passed.

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

  Add lossless string encoding options for Debezium JSON decimal and unsigned bigint columns, and support configurable binary encoding.

Summary by CodeRabbit

• New Features
  • Added Debezium encoding options for decimal, unsigned BIGINT, and binary values.
  • Decimal and unsigned BIGINT values can be emitted as numbers or strings.
  • Binary values support Base64, URL-safe Base64, hexadecimal, and raw byte formats.
  • Settings can be configured through API, TOML files, or URI parameters.
  • Added validation for supported values and protocol-specific usage.
• Bug Fixes
  • Improved handling of unsigned integer defaults and binary decoding across supported modes.

[ti-chi-bot]

@wk989898 This PR has conflicts, I have hold it.
Please resolve them or ask others to resolve them, then comment /unhold to remove the hold label.

[ti-chi-bot]

@ti-chi-bot: ## If you want to know how to resolve it, please read the guide in TiDB Dev Guide.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 082b6209-4119-428e-ba46-4ba57dd926a5

📥 Commits

Reviewing files that changed from the base of the PR and between dd42e7f and 31156c3.

📒 Files selected for processing (1)

• tests/integration_tests/_utils/kafka_dump

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The change adds Debezium decimal, unsigned BIGINT, and binary handling modes. It propagates these modes through API and codec configuration, applies them during JSON encoding and decoding, and adds unit, integration, and Kafka utility coverage.

Changes

Debezium handling modes

Layer / File(s)	Summary
API and configuration contracts api/v2/model.go, api/v2/model_test.go, pkg/config/sink.go	API and internal sink configurations expose the three handling modes. Conversion tests cover populated values, explicit false, nil values, and round trips.
Mode resolution and validation pkg/sink/codec/common/config.go, pkg/sink/codec/common/config_test.go	TOML and URI configuration supports the modes, with defaults, URI precedence, and protocol-specific validation.
Debezium encoding and defaults pkg/sink/codec/debezium/codec.go, pkg/sink/codec/debezium/helper.go, pkg/sink/codec/debezium/codec_test.go	The encoder applies string, numeric, base64, URL-safe base64, hex, and bytes representations to schemas, defaults, and field values. Unsigned BIGINT defaults use unsigned parsing.
Binary decoding and codec coverage pkg/sink/codec/debezium/decoder.go, pkg/sink/codec/debezium/binary_test.go, pkg/sink/codec/debezium/debezium_test.go	The decoder supports configured binary modes and retains base64 for Debezium Avro. Tests cover binary payloads, schemas, numeric strings, defaults, and round trips.
Integration scenarios and Kafka utilities tests/integration_tests/debezium_basic/*, tests/integration_tests/debezium_avro/*, tests/utils/kafka_dump/*, Makefile	Integration tests exercise handling modes for Debezium and Debezium Avro. The Kafka utility consumes row events, optionally decodes Avro messages through Schema Registry, and stops after the configured table event count.

Priority: ➖ Normal

Estimated code review effort: 5 (Critical) | ~90 minutes

Change: Feature · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant SinkConfig
  participant ConfigApply
  participant dbzCodec
  participant Kafka
  participant DebeziumDecoder
  SinkConfig->>ConfigApply: TOML and URI handling modes
  ConfigApply->>ConfigApply: Merge and validate mode values
  ConfigApply->>dbzCodec: Resolved Debezium configuration
  dbzCodec->>Kafka: Encode schemas, defaults, and row values
  Kafka->>DebeziumDecoder: Deliver Debezium payload
  DebeziumDecoder->>DebeziumDecoder: Decode configured binary representation

Loading

Suggested reviewers: ginkgoch, wk989898

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Out of Scope Changes check	⚠️ Warning	Issue #6253 covers BIGINT and DECIMAL precision. The PR also adds binary-handling modes and their production API, configuration, encoder, decoder, and binary-only tests. It adds a general Kafka du…	Remove the unrelated binary-handling production, API, decoder, and binary-only test changes, and remove the unrelated Kafka dump utility and build changes, unless a linked issue adds those requirements. Keep the BIGINT and DECIMAL preci…
Docstring Coverage	⚠️ Warning	Docstring coverage is 9.09% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 33 functions across 19 files. (1 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: adding Debezium numeric and binary handling modes to the sink.
Description check	✅ Passed	The description follows the required template. It includes the issue number, problem statement, implementation summary, unit and integration test coverage, compatibility and documentation questions, a…
Linked Issues check	✅ Passed	Issue #6253 requires precision preservation for Debezium BIGINT and DECIMAL values. The PR adds configurable string handling for Debezium JSON decimals and unsigned BIGINT values. It updates sch…

Full details: Out of Scope Changes check

Explanation

Issue #6253 covers BIGINT and DECIMAL precision. The PR also adds binary-handling modes and their production API, configuration, encoder, decoder, and binary-only tests. It adds a general Kafka dump utility, build targets, and utility tests. These changes do not implement or test the linked issue's numeric precision requirement.

Resolution

Remove the unrelated binary-handling production, API, decoder, and binary-only test changes, and remove the unrelated Kafka dump utility and build changes, unless a linked issue adds those requirements. Keep the BIGINT and DECIMAL precision changes and their supporting tests.

Full details: Docstring Coverage

Explanation

Docstring coverage is 9.09% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 33 functions across 19 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{A rabbit reads each line,
The patch grows clear beneath the moon,
Small changes hop in place,
Tests guard the garden path,
Reviews bloom before the dawn.}

_{Comment @coderabbitai help to get the list of available commands.}

[ti-chi-bot]

Cherry-pick conflicts appear resolved; removing the do-not-merge/hold label.

[ti-chi-bot]

@coderabbitai[bot]: adding LGTM is restricted to approvers and reviewers in OWNERS files.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: coderabbitai[bot], ginkgoch, wk989898

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [wk989898]
• pkg/config/OWNERS [ginkgoch]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment