schemastore: fix replication when DDL makes a table eligible (#6254)

[ti-chi-bot]

This is an automated cherry-pick of #6254

What problem does this PR solve?

Issue Number: close #6243

What is changed and how it works?

• Detect tables becoming eligible after DDL and notify the table trigger to create dispatchers, preventing subsequent CREATE TABLE LIKE from blocking.
• Add DDL whitelist integration tests covering replication, filtering, and eligibility transitions.

According to the doc, there may be a risk of data inconsistency:

When the upstream table has no valid index and force-replicate=true is not configured, the table will not be replicated. However, subsequent DDL statements (including CREATE INDEX, ADD INDEX, and ADD PRIMARY KEY) that create a valid index on this table will be replicated, which might cause inconsistency between downstream and upstream table schemas and lead to subsequent data replication failure.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

1. Create a cluster

tiup playground --db 1 --pd 1 --kv 1 --tiflash 0 --ticdc 1 --ticdc.binpath xxx --ticdc.config ./ticdc-newarch.toml
#Create a changefeed
tiup cdc:v8.5.8 cli changefeed create -c test --sink-uri 'blackhole://'

2. Create an ineligible table

CREATE TABLE test.a (pk BIGINT NOT NULL);
#Add a primary key to this table
ALTER TABLE test.a ADD PRIMARY KEY (pk);
#Create another table that depends on it
CREATE TABLE test.b LIKE test.a;

3. Changefeed lag is normal

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

  Fix an issue where CREATE TABLE LIKE could stall replication after adding a primary or unique key to a previously ineligible table.

Summary by CodeRabbit

• New Features

  • DDL replication now tracks both acquisition and loss of a table’s effective replication key.
  • Replication-key transitions are handled across default and forced replication modes, including partitioned tables and multiple schemas.
  • DDL events preserve replication-key state across storage reloads and remain compatible with previously persisted data.
  • Tables losing their final key are correctly removed from dispatching and downstream table scheduling.

• Bug Fixes

  • Improved historical schema lookup and handling of garbage-collected snapshots.
  • Preserved recovery behavior for replication-key loss during unfinished DDL operations.

• Tests

  • Expanded coverage for whitelist filtering, transitions, recovery, and dispatcher behavior.

[ti-chi-bot]

@wk989898 This PR has conflicts, I have hold it.
Please resolve them or ask others to resolve them, then comment /unhold to remove the hold label.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign asddongmen for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

@ti-chi-bot: ## If you want to know how to resolve it, please read the guide in TiDB Dev Guide.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[coderabbitai]

Warning

Review limit reached

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 02d1cbc0-9fad-459b-b8ae-d84a54c3f358

📥 Commits

Reviewing files that changed from the base of the PR and between af3bf8c and 026b08b.

📒 Files selected for processing (16)

• logservice/schemastore/eligibility_test.go
• logservice/schemastore/persist_storage.go
• logservice/schemastore/persist_storage_ddl_handlers.go
• logservice/schemastore/persist_storage_test.go
• logservice/schemastore/table_info_lookup_test.go
• logservice/schemastore/types.go
• logservice/schemastore/types_gen.go
• pkg/applier/redo_test.go
• pkg/filter/filter.go
• pkg/filter/filter_test.go
• pkg/sink/mysql/mysql_writer_ddl_ts_test.go
• pkg/sink/mysql/mysql_writer_for_ddl_ts.go
• tests/integration_tests/ddl_whitelist/conf/tidb_config.toml
• tests/integration_tests/ddl_whitelist/run.sh
• tests/integration_tests/ddl_whitelist/test.py
• tests/integration_tests/run_heavy_it_in_ci.sh

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}