Skip to content

Antalya 26.8: Antalya-26.6: iceberg v3 multi-arg transforms ,added logic to read source_ids and not throw an error - #2643

Merged
zvonand merged 2 commits into
antalya-26.8from
feature/antalya-26.8/pr-2318
Oct 23, 2026
Merged

Antalya 26.8: Antalya-26.6: iceberg v3 multi-arg transforms ,added logic to read source_ids and not throw an error#2643
zvonand merged 2 commits into
antalya-26.8from
feature/antalya-26.8/pr-2318

Conversation

@zvonand

@zvonand zvonand commented Oct 22, 2026

Copy link
Copy Markdown
Member

Changelog category (leave one):

  • New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Implement iceberg v3 feature when the transformations are created in multiple columns, source-ids data is stored in the manifest.
This PR will read iceberg tables created with source-ids and not throw an error (#2318 by @subkanthi).

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Performance tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All with Aarch64
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

Cherry-picked from #2318.

#2316
implementation similar to apache/iceberg-python#3630

@zvonand
Cherry-pick of #2318 with unresolved conflict markers (resolution in …
b9798e9 
…next commit)

---
Original cherry-pick message follows:

Merge pull request #2318 from Altinity/iceberg-v3-multi-arg-transforms

Antalya-26.6: iceberg v3 multi-arg transforms ,added logic to read source_ids and not throw an error
# Conflicts:
#	src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp
#	src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFile.h
#	src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFileIterator.cpp
#	src/Storages/ObjectStorage/DataLakes/Iceberg/ManifestFilesPruning.cpp
@zvonand
Resolve conflicts in cherry-pick of #2318
2943b5b 
Combined the source PR's `source_ids` / `partition_key_spec_positions`
changes with `antalya-26.8` additions:
- kept `PartitionSpecsEntry::tuple_index` and set it to the spec position
  (the spec vector now has one entry per spec field);
- kept `partition_columns_seen` deduplication alongside
  `partition_key_spec_positions`;
- applied `wrapIntegerPartitionValuesAsDateTime64` to the projected
  partition values;
- token swap `getClickhouseTableSchemaById` -> `getClickHouseTableSchemaById`;
- switched base-branch readers of the removed `PartitionSpecsEntry::source_id`
  (`getIdentityPartitionColumnValues`, `ManifestFilesPruner::canBePruned`,
  `mapTargetFieldsToSpecPositions`) to `source_ids[0]`, skipping
  multi-argument fields where a single column is matched.

Adapted: PartitionSpecsEntry::source_id consumers added on antalya-26.8 switched to source_ids[0]
@zvonand zvonand added releasy Created/managed by RelEasy antalya-26.8 Session label (releasy session config) forwardport This is a frontport of code that existed in previous Antalya versions ai-resolved Port conflict auto-resolved by Claude labels Oct 22, 2026
@github-actions

github-actions Bot commented Oct 22, 2026

Copy link
Copy Markdown

Workflow [PR], commit [2943b5b]

@zvonand zvonand mentioned this pull request Oct 22, 2026
Open
35 tasks done
@zvonand

zvonand commented Oct 23, 2026

Copy link
Copy Markdown
Member Author

@blau-ai

@blau-ai

blau-ai commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

CI triage for #2643

Verdict: 4 failing checks, all the same failure — 0 PR-caused. The Swarms regression suite is pre-existing / infra-flaky on antalya-26.8.

The only red checks are the Swarms regression suite, reported twice per arch (GitHub job + S3 report):

Check Status
Regression release swarms / RegressionTestsRelease / Swarms / swarms fail
Regression aarch64 swarms / RegressionTestsAarch64 / Swarms / swarms fail

Everything else that actually ran is green: 113 pass (incl. all AST-fuzzer variants), 4 fail (the Swarms pair above), 210 skipping, 0 pending.

Why this isn't caused by your PR

The same 3 leaf scenarios fail on both arches, and all are cluster-resilience tests with zero overlap with your diff:

  • /swarms/feature/cluster discovery/check cluster discovery with wrong cluster name
  • /swarms/feature/node failure/cpu overload
  • /swarms/feature/node failure/cpu overload all swarm nodes

Decisive log lines from the reports (x86_64 and aarch64 identical):

Code: 279. DB::Exception: ... Cannot connect to any replica for query execution. (ALL_CONNECTION_TRIES_FAILED)
Code: 209. DB::Exception: ... Timeout exceeded while reading from socket (receive timeout 300000 ms) ... (SOCKET_TIMEOUT)

These are s3Cluster / node-failure tests that deliberately CPU-overload swarm nodes and then assert query behavior — they fail with connection-timeout / resource-starvation errors, the classic signature of an under-resourced CI runner, not a code regression.

  1. No code-path overlap. This PR touches only Iceberg (src/Storages/ObjectStorage/DataLakes/Iceberg/*, ExportPartitionUtils.cpp, iceberg gtests + one stateless .sh). None of that is reachable from cluster discovery, s3Cluster, or node-failure CPU-overload paths.

  2. The same failure appears on unrelated sibling PRs on the same base. The identical 3 Swarms scenarios fail on:

    Three different, unrelated changes all red on the same suite ⇒ it's the suite/infra, not any one PR.

Suggested next step

Nothing to fix in this PR for these. Safe to re-run the Swarms regression (it's flaky under CPU-overload load); if it stays red it's a suite/infra issue for the regression owners, not a merge blocker for #2643. Worth pinging whoever owns the swarms regression if it's persistently red across the branch.

Heads-up (not a failure)

The functional suites — Stateless / Integration / Stress / Build — show as skipping (not executed for commit 2943b5b); only the regression + AST-fuzzer workflow ran. So this triage has a clean signal from the regression/fuzzer jobs, but no stateless/integration coverage was produced for your Iceberg changes on this commit. You may want to trigger a full CI run so the new 05301_iceberg_optimize_multi_arg_spec_no_orphans test and the iceberg gtests actually execute before merge.

Reports: x86_64 swarms · aarch64 swarms

@zvonand
zvonand merged commit 2f37e86 into antalya-26.8 Oct 23, 2026
323 of 327 checks passed
@zvonand zvonand added verified Approved for release port-antalya PRs to be ported to all new Antalya releases labels Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

ai-resolved Port conflict auto-resolved by Claude antalya-26.8 Session label (releasy session config) forwardport This is a frontport of code that existed in previous Antalya versions port-antalya PRs to be ported to all new Antalya releases releasy Created/managed by RelEasy verified Approved for release

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zvonand @blau-ai @svb-alt