Conversation
…ance for CDC-created tables Discover routed maintenance targets at submission and allow CDC to create missing Iceberg tables after the job starts. Build the native TableMaintenance topology and adapt initialization and catalog ownership before submission. Support data-file rewriting, snapshot expiration, and orphan cleanup using the configured FileIO, with Flink 1.20 and 2.2 compatibility. Validation: 134 tests passed per Flink version, including MiniCluster maintenance and savepoint recovery; formatting, Checkstyle, and package passed. Local implementation checkpoint; assign a JIRA issue before opening a PR. Generated-by: Codex GPT-6
…lization and simplify setup Preserve the original FileIO implementation, properties, and effective Hadoop configuration across metadata split JSON serialization. Retain prefix listing and bulk deletion capabilities for native orphan cleanup. Share task registration across maintenance builders and normalize discovered targets in MaintenanceOptions. Clarify the discovery contract and cover the Planner-to-Reader path and duplicate targets in regression tests. Validation: 135 related tests passed on both Flink 1.20 / Java 11 and Flink 2.2 / Java 17, including MiniCluster maintenance and savepoint recovery. Package, Spotless, and Checkstyle passed. Local development commit; assign a JIRA issue before opening a PR. Generated-by: Codex GPT-6
… and use native FileIO Refresh cached tables before native maintenance planners copy metadata, so planning observes new commits and avoids reading expired manifest lists. Use Iceberg's native FileIO serialization and remove the Hadoop configuration wrapper and its dedicated tests. Consolidate the English and Chinese maintenance documentation and describe the executor configuration requirement. Validation: the new regression fails with NotFoundException before the fix. All 72 selected maintenance tests pass on both Flink 1.20 and Flink 2.2, including MiniCluster execution and savepoint recovery. Package, Spotless, and Checkstyle checks passed. Local development commit; assign a JIRA issue before opening a PR. Generated-by: Codex GPT-6
github-actions
Bot
added
docs
…tion handling Add PostgreSQL, Oracle, and SQL Server discovery tests using JDBC metadata fixtures and the native dialect filtering paths. Cover table identifiers, empty results, PostgreSQL options, and database filtering. Distinguish runtime failures from SQLException handling at each query stage, and verify connection cleanup and exception causes. Extract package-private dialect creation methods to inject the test connections. Validation: all 23 selected discovery tests pass on Flink 1.20 / Java 11 and Flink 2.2 / Java 17. Package, Spotless, and Checkstyle checks passed. Generated-by: Codex GPT-6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of this pull request?
Add optional Iceberg table maintenance to CDC pipelines, supporting data file compaction, snapshot expiration, and orphan file cleanup.
Maintenance supports both existing target tables and tables created by CDC after startup. It is disabled by default and configured through
sink.maintenance.*.Brief change log
TableMaintenanceAPI into the Iceberg pipeline sink.Verifying this change
This change added tests and can be verified as follows:
MaintenanceOptionsTest,TargetTableDiscoveryTest, andMaintenanceGraphAdapterTestfor configuration validation, routed target discovery, and topology adaptation.MySqlTableDiscoveryTest,PostgresTableDiscoveryTest,OracleTableDiscoveryTest, andSqlServerTableDiscoveryTest, covering capture filtering, table identifier conversion, source-specific options, JDBC error handling, and connection cleanup.MaintenanceRegressionTest,DeferredTableLoaderTest, andRefreshingTableLoaderTestcovering resource cleanup, deferred table loading, and metadata freshness.TableMaintenanceExecutionTestcovering CDC writes, all three maintenance operations, and savepoint recovery before and after target creation.Documentation
Was generative AI tooling used to co-author this PR?
Generated-by: Codex GPT-6