Plain merge tree partition exports by arthurpassos · Pull Request #2290 · Altinity/ClickHouse

arthurpassos · 2026-08-28T17:30:25Z

List of changes:

Implements partition export for plain merge tree using Json task descriptors stored on disk, the structure is simplified because it is much simpler than the replicated case
Introduces the system.partition_exports table that keeps track of both plain and replicated cases. The old system.replicated_partition_exports is aliased to this new one for backwards compatibility.
Escapes each portion of the partition export key - this is not backwards compatible.
Stop shipping empty parquet files on plain object storage for exports with zero rows
Throws on a few errors zk that were left as todo
Refactor the test suite so that it is parametrized for both engines instead of duplicating the tests

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Export partition on plain merge tree. Should close #1908

User facing & functional changes:

Implements partition export for plain merge tree
Stop shipping empty parquet files on zero rows exports
New system.partition_exports table that covers both the previous replicated_partition_exports and the new one for plain merge tree

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

Regression jobs to run:

github-actions · 2026-08-28T17:31:49Z

Workflow [PR], commit [2662683]

arthurpassos · 2026-08-29T17:14:51Z

@codex review

chatgpt-codex-connector · 2026-08-29T17:15:05Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-12T14:11:12.740611Z	`2662683`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

arthurpassos · 2026-08-29T17:20:28Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 240da8173c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-29T17:37:02Z

+        catch (...)
+        {
+            replicated_error = std::current_exception();
+            tryLogCurrentException(getLogger("InterpreterKillQueryQuery"),
+                "KILL EXPORT PARTITION: could not read system.replicated_partition_exports (the WHERE may "
+                "reference columns that only exist for plain MergeTree); ignoring ReplicatedMergeTree tables");


Propagate system-table read failures during KILL

If either system-table query fails for an operational reason such as a memory limit, timeout, or internal exception, this broad catch treats it like an engine-specific unknown column and continues with the other table. The command can therefore report successful results while silently leaving matching exports from the failed table running. Only suppress the specific predicate-resolution error that proves the filter belongs exclusively to the other table; propagate all other failures.

AGENTS.md reference: AGENTS.md:L153-L153

Useful? React with 👍 / 👎.

…ya-26.6/export-partition-plain-merge-tree

arthurpassos · 2026-09-04T11:53:25Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c8d038d350

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…n-plain-merge-tree

…n-plain-merge-tree Keep the skip-policy commit-file short-circuit from antalya-26.6 together with lazy first-sink creation, and port the new skip-policy tests into the unified object-storage suite. Co-authored-by: Cursor <cursoragent@cursor.com>

…n-plain-merge-tree

arthurpassos · 2026-09-11T11:50:18Z

+
+### Pending mutations {#plain-merge-tree-pending-mutations}
+
+The pending-mutation gate is more conservative than on a `Replicated*MergeTree`. A plain `MergeTree` does not scope its mutation snapshot by partition, so a mutation restricted with `IN PARTITION` still marks the parts of every other partition as having pending mutations, and exporting an unaffected partition is refused with `PENDING_MUTATIONS_NOT_ALLOWED`. The gate fails closed - it never exports data that a pending mutation would have changed - so the effect is that you may have to wait for an unrelated mutation to finish, or set `export_merge_tree_part_throw_on_pending_mutations` to `false`.


I should check ways around this later

arthurpassos · 2026-09-11T11:52:25Z

 ### Commit info columns

-These columns surface paths produced by the destination storage during commit, so it is possible to inspect what was written without consulting the destination directly:
+These columns surface paths produced by the destination storage during commit, so it is possible to inspect what was written without consulting the destination directly. They are populated for `Replicated*MergeTree` sources only; a plain `MergeTree` does not persist the commit paths, so they stay empty there even after a successful commit.


This shall be implemented for plain merge tree as well

Is this going to be covered in another PR?

I plan to cover it in this pr unless it is too complex (I doubt it is).

For now I am reviewing my own code (AI code :D)

DimensionWieldr · 2026-09-11T17:11:57Z

Iceberg export partition regression suite is fully passing on no/ice/glue catalogs with plain merge tree tables.

I will wait for dev review and the CI run to do a final CI triage to confirm no PR caused failures.

mkmkme

Overall looks good and worth an approval. I left some comments, I'll let you decide what to do about them

mkmkme · 2026-09-12T09:33:23Z

+The partition export tasks of both engines can be observed through `system.partition_exports`.

-The export task can be killed by issuing the kill command: `KILL EXPORT PARTITION <where predicate for system.replicated_partition_exports>`.
+`system.replicated_partition_exports` is kept as an alias of `system.partition_exports` for backwards compatibility. It returns exactly the same rows, including exports of plain `MergeTree` tables. Filter on `source_table` (or join against `system.tables`) if you need only one engine.


That's a bit counter-intuitive IMHO. But I don't have a good and simple solution for that. I can suggest one of two things:

Make replicated_partition_exports only return exports from RMT (but it will likely require quite a bit of work for not exactly a great result)

Mark system.replicated_partition_exports explicitly as deprecated (but I am not sure we have such a mechanism for system tables)

mkmkme · 2026-09-12T10:04:44Z

 ### Exception columns

- `last_exception_per_replica` is an `Array(Tuple(replica String, message String, part String, time DateTime, count UInt64))`. Each tuple is the most recent exception observed by a single replica plus a best-effort within-replica `count`. Replicas that have never reported an exception are omitted.
+- `last_exception_per_replica` is an `Array(Tuple(replica String, message String, part String, time DateTime, count UInt64))`. Each tuple is the most recent exception observed by a single replica plus a best-effort within-replica `count`. Replicas that have never reported an exception are omitted. A plain `MergeTree` export runs on a single node, so it contributes at most one tuple and its `replica` is empty.


Haven't found (yet) in the documentation, but wanted to ask: does it make sense for EXPORT PARTITION on a plain MergeTree to be synchronous? And instead of finding the exception in the system table, it would raise the exception as a response?

mkmkme · 2026-09-12T10:34:45Z

+        if (!iceberg_metadata)
+            throw Exception(ErrorCodes::BAD_ARGUMENTS, "Destination storage {} is a data lake but not an iceberg table", dest_storage->getName());
+
+        if (!context->getSettingsRef()[Setting::allow_insert_into_iceberg])


Nitpick: I would probably move this check before checking for iceberg_metadata being nullptr, but it's not a biggie

mkmkme · 2026-09-12T11:12:15Z

+        auto & entry = it->second;
+
+        if (entry.getDescriptor().status != MergeTreePartitionExportTask::Status::PENDING)
+            return CancellationCode::CancelCannotBeSent;


I'd say this also deserves a LOG_INFO

mkmkme · 2026-09-12T11:49:33Z

+        std::lock_guard lock(mutex);
+        auto it = findByTransactionId(transaction_id);
+        if (it == tasks.end())
+            return;


Maybe a LOG_DEBUG here? Why can this happen?

mkmkme · 2026-09-12T12:39:33Z

    expected_status="COMPLETED",
    timeout=60,
    poll_interval=0.5,
+    system_table="partition_exports",


Is this argument (here and below) is actually used in a way that replicated_partition_exports is passed explicitly?

mkmkme · 2026-09-12T13:54:19Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2662683255

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-09-12T14:11:10Z

+        return ExportPartitionUtils::compositeKey(
+            manifest.partition_id, manifest.destination_database, manifest.destination_table);


Preserve or migrate the previous ZooKeeper task keys

After upgrading with an existing replicated export, its ZooKeeper child still uses the old <partition>_<database>.<table> key, while this method now derives the escaped dot-separated key. Although poll reads the old child, the inserted entry is indexed by this new derived key and removeStaleEntries immediately removes it because that key is absent from the ZooKeeper child list; the scheduler therefore cannot resume its remaining parts, and a new export also misses the old path during duplicate detection. Migrate existing nodes or retain the actual ZooKeeper child key in the entry.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-09-12T14:11:10Z

+        entry.part_references = std::move(part_references);
+        entry.setDescriptor(std::move(descriptor));
+        persist(composite_key, entry.getDescriptor().toJsonString());
+        tasks.insert_or_assign(composite_key, std::move(entry));


Block force replacement while the prior commit is running

When a forced re-export arrives while the existing entry has committing == true, this replacement proceeds even though KILL explicitly refuses that state. killExportPart cannot cancel the destination commit, so the old commit can publish successfully and then find its registry entry replaced, while the new transaction subsequently exports and commits the same partition again; for an Iceberg destination this can produce duplicate rows under two transaction IDs. Reject or defer the replacement until the existing commit lease is released.

Useful? React with 👍 / 👎.

arthurpassos added 5 commits August 28, 2026 14:02

1st vibe coded version

69f071a

some vibe coded fixes

3190603

persist-then-apply and uuidv4 instead of snowflakeid

d94ea53

idk

b2af06c

possibly fix build

f2ee03f

arthurpassos mentioned this pull request Aug 28, 2026

Plain merge tree partition export #2032

Closed

28 tasks

svb-alt added antalya antalya-26.6 labels Aug 28, 2026

chmod

240da81

arthurpassos marked this pull request as ready for review August 29, 2026 17:20

chatgpt-codex-connector Bot reviewed Aug 29, 2026

View reviewed changes

arthurpassos added 9 commits August 31, 2026 14:44

fix kill operation

9ff4a4a

fix backoff on plain

6c86d2f

make code a bit more readable, ai

a622f44

release parts

23d90fa

fix endless commit due to part ref lost

fe7e61a

well, I don't like it, but ok

7c959fb

fix dispatch err handling

f06a7dd

fix

7482d6b

Merge remote-tracking branch 'origin/antalya-26.6' into feature/antal…

c8d038d

…ya-26.6/export-partition-plain-merge-tree

chatgpt-codex-connector Bot reviewed Sep 4, 2026

View reviewed changes

arthurpassos added 2 commits September 4, 2026 09:15

throw exceptions to the user

875cc47

throw on unknown values

9496d85

arthurpassos changed the title ~~Feature/antalya 26.6/export partition plain merge tree~~ Plain merge tree partition exports Sep 4, 2026

simpĺify docs

3af519a

arthurpassos added 6 commits September 4, 2026 10:19

throw on exporting unreadable entries

31b04b1

Merge branch 'antalya-26.6' into feature/antalya-26.6/export-partitio…

01b5d38

…n-plain-merge-tree

unify tables

a0badb4

fix old bug

2eab452

add no-cas

cd502a6

rmv unnecessary tests

3fc5e5f

arthurpassos added the port-antalya PRs to be ported to all new Antalya releases label Sep 8, 2026

arthurpassos and others added 7 commits September 8, 2026 13:51

fix long standing issue with key

cae460a

unify replicated and plain export tests

d4a3859

Merge branch 'antalya-26.6' into feature/antalya-26.6/export-partitio…

b50cee3

…n-plain-merge-tree

fix few bugs

b89bfa1

improve testing suite

00b1540

Merge branch 'antalya-26.6' into feature/antalya-26.6/export-partitio…

487b674

…n-plain-merge-tree

arthurpassos commented Sep 11, 2026

View reviewed changes

arthurpassos added 4 commits September 11, 2026 08:57

some documentation fixes

0433384

fix some more docs

b9f4406

improve naming

ff54cf5

more comments docs

7a8b440

arthurpassos added 4 commits September 11, 2026 14:34

rename method

968163b

documentation

8ef5a76

ai fixes

c70028c

persist commit info plain mt

2662683

mkmkme self-requested a review September 12, 2026 07:36

mkmkme approved these changes Sep 12, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Sep 12, 2026

View reviewed changes


		### Pending mutations {#plain-merge-tree-pending-mutations}

		The pending-mutation gate is more conservative than on a `Replicated*MergeTree`. A plain `MergeTree` does not scope its mutation snapshot by partition, so a mutation restricted with `IN PARTITION` still marks the parts of every other partition as having pending mutations, and exporting an unaffected partition is refused with `PENDING_MUTATIONS_NOT_ALLOWED`. The gate fails closed - it never exports data that a pending mutation would have changed - so the effect is that you may have to wait for an unrelated mutation to finish, or set `export_merge_tree_part_throw_on_pending_mutations` to `false`.

		return ExportPartitionUtils::compositeKey(
		manifest.partition_id, manifest.destination_database, manifest.destination_table);

Uh oh!

Conversation

arthurpassos commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI/CD Options

Exclude tests:

Regression jobs to run:

Uh oh!

github-actions Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arthurpassos commented Aug 29, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

arthurpassos commented Aug 29, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot Aug 29, 2026

Choose a reason for hiding this comment

Uh oh!

arthurpassos commented Sep 4, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DimensionWieldr commented Sep 11, 2026

Uh oh!

mkmkme left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mkmkme commented Sep 12, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Sep 12, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Sep 12, 2026

Choose a reason for hiding this comment

Uh oh!

arthurpassos commented Aug 28, 2026 •

edited

Loading

github-actions Bot commented Aug 28, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Aug 29, 2026 •

edited

Loading