Skip to content

api,cli: support creating paused changefeeds - #6265

Open
sdojjy wants to merge 5 commits into
pingcap:masterfrom
sdojjy:feat/create-paused-changefeed
Open

sdojjy wants to merge 5 commits into
pingcap:masterfrom
sdojjy:feat/create-paused-changefeed

Conversation

@sdojjy

@sdojjy sdojjy commented Sep 15, 2026

Copy link
Copy Markdown
Member

What problem does this PR solve?

Creating a changefeed immediately schedules replication. Callers need to perform the normal validation and reserve the start position while leaving the new task paused until an explicit resume.

Issue Number: close #6266

What is changed and how it works?

  • Add the optional top-level pause boolean to the v2 create request. Omission and false retain automatic startup; true persists stopped and registers the task in the stopped collection without scheduling a maintainer.
  • Support top-level pause = true in CLI TOML via the internal replica configuration and forward it to the create request. It is a create-only option, not an additional runtime state or nested API replica setting.
  • Keep the existing table/sink/configuration/TSO checks, initial checkpoint, temporary GC protection and coordinator GC handoff. Persist an internal pending-bootstrap marker so the first resume and coordinator restart retain fresh-changefeed semantics, including stale MySQL DDL recovery record cleanup. Clear it only after the current maintainer reports successful bootstrap, using an incarnation/epoch check and a metadata revision comparison. Failed acknowledgements are retried on subsequent heartbeats; later resumes keep normal recovery semantics.
  • Add creation/resume/GC unit coverage and a MySQL integration case covering API and CLI creation, default compatibility, rejected requests, no replication while paused, restart, GC handoff on classic, and data consistency after resume with newer stale ddl_ts_v1 records for a reused name. Register the case in MySQL light CI G04 and document usage.

Check List

Tests

Passed locally:

  • Full ./coordinator, ./coordinator/changefeed, and ./coordinator/operator package tests with -p 1 -ldflags='-s -w' -count=1, including first resume/restart, stale owner, failed acknowledgement retry, and metadata compare-and-swap coverage.

  • MySQL TestGetTableRecoveryInfo_StartTsGreaterThanDDLTs and TestGetTableRecoveryInfo_RemoveDDLTs; local targeted unconvert lint.

  • go test -p 1 -ldflags='-s -w' ./api/v2 ./cmd/cdc/cli ./pkg/config -count=1

  • go test -p 1 -ldflags='-s -w' ./coordinator -run '^Test(CreateChangefeedDoesNotUpdateGCSafepoint|UpdateGCSafepointCallsGCManagerUpdate)$' -count=1

  • Repository-pinned Go/import/shell formatters on changed files, shell syntax checks, diff line-width and log-style checks, and git diff --check.

Not completed locally:

  • The same Coordinator tests with -tags=nextgen could not compile because the disk ran out of space.
  • make integration_test_mysql CASE=changefeed_create_paused stopped at check_third_party_binary: local database test binaries are missing, and the Docker daemon is unavailable. The new integration test is committed and registered in CI but has not been run end-to-end locally.
  • Full binary build and full repository/race suites were not run given disk capacity. An initial broad -run TestCreateChangefeed also hit an existing test's missing global MessageCenter initialization; the focused GC creation tests above initialize their own services and pass.

Will it cause performance regression or break compatibility?

The default creation path is unchanged. No new lifecycle state, storage migration, or scheduler abstraction is introduced; existing metadata without the optional bootstrap marker retains its current behavior. Pause affects creation only, and existing paused-task GC TTL limits still apply.

Do you need to update user documentation, design documentation or monitoring documentation?

Added docs/create-paused-changefeed.md with API and CLI TOML examples and lifecycle/GC semantics.

Release note

Support creating paused changefeeds through the v2 API pause option and CLI TOML configuration while preserving validation and GC protection.

Summary by CodeRabbit

  • New Features

    • Changefeeds can now be created in a paused state through the REST API or CLI using the pause option.
    • Paused changefeeds remain stopped across restarts and can be resumed later while preserving fresh-start behavior.
    • Omitting pause or setting it to false retains automatic startup.
    • Existing validation and GC protections continue to apply.
  • Documentation

    • Added guidance for configuring and using paused changefeed creation.
  • Tests

    • Added coverage for API, CLI, restart, validation, resumption, and replication scenarios.

@ti-chi-bot ti-chi-bot Bot added do-not-merge/needs-linked-issue release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 15, 2026
@coderabbitai

coderabbitai Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 5d41d081-883d-4ca6-93a3-6579138d2220

📥 Commits

Reviewing files that changed from the base of the PR and between 2aecc11 and 366ee05.

📒 Files selected for processing (16)
  • api/v2/changefeed.go
  • api/v2/model.go
  • cmd/cdc/cli/cli_changefeed_create_test.go
  • coordinator/changefeed/changefeed.go
  • coordinator/changefeed/changefeed_db.go
  • coordinator/changefeed/changefeed_db_backend.go
  • coordinator/changefeed/etcd_backend.go
  • coordinator/changefeed/etcd_backend_test.go
  • coordinator/changefeed/mock/changefeed_db_backend.go
  • coordinator/controller.go
  • coordinator/coordinator.go
  • coordinator/create_changefeed_gc_test.go
  • docs/create-paused-changefeed.md
  • pkg/config/changefeed.go
  • pkg/config/replica_config.go
  • tests/integration_tests/changefeed_create_paused/run.sh

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

Paused changefeed creation now accepts an optional API or CLI pause setting. Paused feeds persist in the stopped state with bootstrap metadata, retain their start checkpoint, and complete initialization after a successful resume. Tests and documentation cover restart, GC, validation, and replication behavior.

Changes

Paused Changefeed Creation

Layer / File(s) Summary
Creation contract and state selection
api/v2/model.go, api/v2/changefeed.go, pkg/config/replica_config.go, cmd/cdc/cli/*, coordinator/controller.go
The API and CLI preserve omitted versus explicit pause values. pause=true creates a stopped changefeed with a pending bootstrap marker.
Bootstrap metadata persistence
pkg/config/changefeed.go, coordinator/changefeed/*, coordinator/controller.go, coordinator/coordinator.go
The coordinator tracks new changefeeds atomically, validates maintainer identity and epoch, and clears bootstrap metadata through a serialized, compare-and-swap persistence operation.
Coordinator, GC, and lifecycle validation
coordinator/create_changefeed_gc_test.go, coordinator/changefeed/etcd_backend_test.go, coordinator/changefeed/mock/*
Tests cover GC state, restart reconstruction, stale owners, failed acknowledgements, CAS outcomes, and successful bootstrap completion.
Documentation and integration validation
docs/create-paused-changefeed.md, tests/integration_tests/changefeed_create_paused/*, tests/integration_tests/run_light_it_in_ci.sh
Documentation and integration tests cover API and CLI creation, automatic startup, validation errors, GC handoff, restart persistence, resume, and downstream replication.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant CDCAPI
  participant Coordinator
  participant Etcd
  participant Maintainer
  participant Downstream
  Client->>CDCAPI: Create changefeed with pause=true
  CDCAPI->>Coordinator: Save stopped changefeed with bootstrap pending
  Coordinator->>Etcd: Persist changefeed metadata
  Client->>CDCAPI: Resume changefeed
  Maintainer->>Coordinator: Report completed bootstrap
  Coordinator->>Etcd: Clear bootstrap pending with FinishInit
  Coordinator->>Downstream: Start replication
Loading

Suggested reviewers: wk989898

Merge Risk: ⚪ Minimal · up to 366ee

Paused changefeeds retain their validated checkpoint and fresh-start behavior until successfully resumed, while normal creation remains unchanged. No actionable merge-blocking risk remains.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 37.50% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 16 functions across 17 files. (1 skipped:… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed PR #6266 requirements are met. POST /api/v2/changefeeds accepts optional pause; true sets the existing stopped state and BootstrapPending, while omitted or false keeps StateNormal in `ap…
Out of Scope Changes check ✅ Passed The changes stay within #6266. API and CLI fields implement paused creation. Coordinator metadata and locking preserve first-resume semantics. Unit tests, MySQL integration coverage, documentation, an…
Title check ✅ Passed The title clearly and concisely describes the main change: adding API and CLI support for creating paused changefeeds.
Description check ✅ Passed The description follows the required template. It includes the issue number, problem statement, implementation details, test coverage and limitations, compatibility assessment, documentation updates, …
Full details: Docstring Coverage

Explanation

Docstring coverage is 37.50% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 16 functions across 17 files. (1 skipped: 1 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

A rabbit taps pause by moonlit light
The feed sleeps safely through the night
Its checkpoint waits, precise and still
A resume starts it by command and will
Bootstrap clears when proof is done
Then rows hop onward, one by one

Comment @coderabbitai help to get the list of available commands.

@asddongmen

Copy link
Copy Markdown
Collaborator

I think we need to preserve the new-changefeed semantics for a changefeed created with pause = true.

The changefeed is constructed with isNew = true, but its first normal resume goes through MoveToSchedulingQueue, which sets isNew to overwriteCheckpointTs and therefore turns it into false. If the coordinator restarts before the first resume, bootstrap also reconstructs the stopped changefeed as not new.

This matters for MySQL sinks because IsNewChangefeed is eventually used as removeDDLTs. With the flag set to false, the sink reuses existing ddl_ts_v1 recovery records instead of clearing them. If a changefeed ID is reused and then created paused with a historical start_ts earlier than stale ddl_ts, the effective start position can be moved forward and some events may be skipped.

Could we keep a durable marker for a paused changefeed that has never started, preserve it across coordinator restart, and clear it only after the first maintainer bootstrap succeeds? It would also be good to assert IsNewChangefeed on the first resume and cover a stale ddl_ts case.

@ti-chi-bot ti-chi-bot Bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Sep 16, 2026
@ti-chi-bot ti-chi-bot Bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 16, 2026
@ti-chi-bot ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Sep 16, 2026
@ti-chi-bot

ti-chi-bot Bot commented Sep 16, 2026

Copy link
Copy Markdown

[LGTM Timeline notifier]

Timeline:

  • 2026-09-16 02:33:58.273803385 +0000 UTC m=+172484.211460979: ☑️ agreed by wk989898.
  • 2026-09-16 03:36:05.856153115 +0000 UTC m=+176211.793810710: ☑️ agreed by asddongmen.

@ti-chi-bot

ti-chi-bot Bot commented Sep 17, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: asddongmen, nongfushanquan, wk989898

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added the approved label Sep 17, 2026
@sdojjy

sdojjy commented Sep 17, 2026

Copy link
Copy Markdown
Member Author

/retest-required

@sdojjy

sdojjy commented Sep 17, 2026

Copy link
Copy Markdown
Member Author

/test pull-cdc-pulsar-integration-heavy

@ti-chi-bot

ti-chi-bot Bot commented Sep 17, 2026

Copy link
Copy Markdown

@sdojjy: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-cdc-mysql-integration-light 366ee05 link unknown /test pull-cdc-mysql-integration-light
pull-cdc-kafka-integration-heavy 366ee05 link unknown /test pull-cdc-kafka-integration-heavy
pull-cdc-mysql-integration-light-next-gen 366ee05 link unknown /test pull-cdc-mysql-integration-light-next-gen
pull-cdc-mysql-integration-light-next-gen-legacy-safepoint 366ee05 link unknown /test pull-cdc-mysql-integration-light-next-gen-legacy-safepoint

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved lgtm release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support creating changefeeds in a paused state

4 participants