From 207f5bf4f9655f2d5eb1dd8c1c09103deb0f3eda Mon Sep 17 00:00:00 2001 From: 924060929 Date: Tue, 20 Oct 2026 11:22:39 +0800 Subject: [PATCH 1/5] [improvement](fe) Version Iceberg partition views and SQL cache metadata MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Issue Number: None Problem Summary: Ordinary Iceberg scans and SQL result-cache validation materialize and copy the full Doris partition view even when native file pruning or a pinned metadata version is sufficient. Cache immutable engine partition views and their pruning index by connector metadata generation, defer conversion for native planning, and obtain SQL cache versions through a source-neutral version provider. Record distinct current, historical and branch/tag references when a statement reads several versions of one table, and revalidate them through the existing snapshot resolver. Reuse statement-retained table handles and combine relation-name discovery through the connector SPI to avoid repeated remote existence and listing requests. Preserve MV/partition metadata consumers, refresh invalidation and frozen-reader schema consistency. Review repairs: retain heterogeneous display/freshness partition maps without an invalid sorted index, carry entry-specific scalable byte estimates through catalog cache definitions, and align partition-metadata regression settings and the Hive fixture with native planning. Rebase includes upstream source-nullability and the existing Paimon CHAR regression repair; connector API is 20.0. Final rebased validation: 210 scoped unit tests, an overlapping 21-test weighted-cache/runtime pass, and build.sh --fe -j4 with Checkstyle passed. Historical performance and live-SQL validation below predate these review repairs. Review follow-up: keep Iceberg tableExists authoritative for DDL and use that live preflight before DROP TABLE. Cached/pinned handles remain available to read planning. Added externally dropped-table cases for IF EXISTS and strict DROP, plus cached/pinned read isolation. Keep the partition-rule Env accessor independent of the scheduler import removal so the FE PRs compile together. 96 scoped tests and build.sh --fe with Checkstyle passed. Review follow-up: use live tableExists for both CREATE TABLE existence probes, including IF NOT EXISTS race recovery. Added stale cached-handle tests for both CREATE modes and failed-create propagation. All 90 DDL routing tests, full FE build and Checkstyle passed. Iceberg ordinary scans use native file pruning by default. The session variable enable_iceberg_internal_partition_pruning enables Doris internal partition pruning when needed. SQL result caching supports independently versioned Iceberg snapshot and branch/tag references without enumerating partitions. - Test: Unit Test - run-fe-ut.sh: 325 targeted metadata, MVCC, partition view, SQL cache and dependent test cases passed - run-fe-ut.sh: all 85 connector SPI tests passed separately - build.sh --fe -j4 passed, including Checkstyle and connector architecture gates - Behavior changed: Yes, defer ordinary Iceberg partition conversion by default and support version-aware external SQL caching - Does this need documentation: Yes, updated fe/fe-connector/README.md Measured independently against unoptimized master `e0bf4874070`, with identical Linux AArch64 Release BE, data/footer prewarming, and session settings. Numbers are complete SQL medians, in milliseconds. The baseline column shows two separate baseline runs. | Scenario | Baseline runs | This PR | |---|---:|---:| | 120,000 partitions, warm LIMIT 10 | 389.38–463.29 | 9.69 | | 120,000 partitions, warm EXPLAIN | 188.19–189.07 | 3.45 | | 120,000 partitions, warm LIMIT 10 with internal pruning enabled | 333.18–391.80 | 9.76 | | 1,000 manifests / 10,000 files, cold LIMIT 10 | 925.44–933.77 | 26.50 | The large-partition protocol enables SQL Cache validation but varies an always-true predicate to avoid result-cache hits. Ordinary LIMIT cases have 40 measured samples; EXPLAIN has 20. Baseline EXPLAIN retention is bounded by using equivalent SELECT warmup, identically for all variants. Function-heavy partition predicates remain approximately 5.5 seconds in this fixture and are not claimed as a benefit. - Live SQL Cache checks: 170 assertions for current reads, snapshots, timestamps, branch/tag movement, multiple versions of one table, schema evolution/rename, snapshot expiry and ref removal/recreation. - The final combined tree passed a clean FE build, 997 scoped unit tests without skips, and the Hive streaming / Iceberg scheduled-split / table-handle-reuse regression suites. --- fe/fe-connector/README.md | 36 ++ .../connector/iceberg/IcebergCatalogOps.java | 17 + .../iceberg/IcebergConnectorMetadata.java | 134 +++++- .../iceberg/IcebergPartitionUtils.java | 7 +- .../iceberg/IcebergStatementScope.java | 5 + .../connector/iceberg/IcebergTableCache.java | 5 + .../CatalogBackedIcebergCatalogOpsTest.java | 23 + .../IcebergConnectorMetadataMvccTest.java | 80 ++++ ...nnectorMetadataPartitionViewCacheTest.java | 61 +++ .../iceberg/IcebergTableHandleReuseTest.java | 311 +++++++++++++ .../connector/iceberg/TestStatementScope.java | 6 + .../connector/spi/ConnectorMetadata.java | 35 ++ .../spi/ConnectorPartitionListingOps.java | 12 + .../spi/ConnectorStatementScope.java | 5 + .../spi/ConnectorStatementScopes.java | 14 +- .../spi/ConnectorTableMetadataOps.java | 5 + .../spi/mvcc/ConnectorSqlCacheVersion.java | 44 ++ .../spi/ConnectorPluginSurfaceTest.java | 3 + ...ConnectorRelationMetadataDefaultsTest.java | 59 +++ .../spi/ConnectorStatementScopesTest.java | 23 + .../resources/connector-metadata-methods.txt | 5 + .../resources/connector-plugin-surface.txt | 10 + .../doris/blockrule/SqlBlockRuleMgr.java | 16 + .../common/cache/NereidsSqlCacheManager.java | 35 +- .../ConnectorStatementScopeImpl.java | 6 + .../datasource/ExternalMetaCacheMgr.java | 21 +- .../doris/datasource/ExternalTable.java | 7 +- .../metacache/ExternalCatalogMetaCache.java | 13 +- .../metacache/MetaCacheEntryDef.java | 29 ++ .../datasource/mvcc/MvccReadSelector.java | 116 +++++ .../mvcc/PluginDrivenMvccExternalTable.java | 209 ++++++--- .../mvcc/PluginDrivenMvccSnapshot.java | 75 +++- .../mvcc/PluginDrivenPartitionView.java | 309 +++++++++++++ .../plugin/PluginDrivenExternalCatalog.java | 38 +- .../datasource/scan/PluginDrivenScanNode.java | 23 +- .../doris/mtmv/MTMVRelationManager.java | 9 +- .../apache/doris/nereids/SqlCacheContext.java | 59 ++- .../doris/nereids/StatementContext.java | 42 +- .../nereids/rules/analysis/BindRelation.java | 20 +- .../rules/rewrite/PruneFileScanPartition.java | 11 +- .../rewrite/QueryPartitionCollector.java | 14 +- .../trees/plans/logical/LogicalFileScan.java | 22 +- .../org/apache/doris/qe/SessionVariable.java | 7 + .../apache/doris/qe/cache/CacheAnalyzer.java | 33 +- .../qe/cache/SqlCacheVersionProvider.java | 32 ++ .../alter/ChangelogConfigPublicationTest.java | 4 +- .../doris/blockrule/SqlBlockRuleMgrTest.java | 27 ++ ...tedPartitionsCacheManagerExternalTest.java | 33 +- ...NereidsSqlCacheManagerPluginTableTest.java | 86 +++- .../connector/ConnectorSessionImplTest.java | 19 + .../ConnectorStatementScopeTest.java | 16 + .../ExternalCatalogMetaCacheTest.java | 43 +- .../datasource/mvcc/MvccReadSelectorTest.java | 76 ++++ .../PluginDrivenMvccExternalTableTest.java | 223 ++++++++++ .../mvcc/PluginDrivenPartitionViewTest.java | 407 ++++++++++++++++++ ...inDrivenExternalCatalogDdlRoutingTest.java | 87 +++- ...nDrivenExternalCatalogViewListingTest.java | 1 + ...PluginDrivenScanNodeHiveStreamingTest.java | 2 + ...luginDrivenScanNodePartitionCountTest.java | 22 + .../doris/mtmv/MTMVRelationManagerTest.java | 33 ++ .../SqlCacheContextPluginTableTest.java | 78 ++++ .../rewrite/QueryPartitionCollectorTest.java | 140 ++++++ .../cache/PluginTableCacheAnalyzerTest.java | 53 +++ .../test_iceberg_table_handle_reuse.out | 44 ++ .../test_iceberg_table_handle_reuse.groovy | 84 ++++ ...est_iceberg_write_binary_partitions.groovy | 2 + .../test_iceberg_write_uuid_inputs.groovy | 2 + 67 files changed, 3288 insertions(+), 240 deletions(-) create mode 100644 fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableHandleReuseTest.java create mode 100644 fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/mvcc/ConnectorSqlCacheVersion.java create mode 100644 fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorRelationMetadataDefaultsTest.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/MvccReadSelector.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenPartitionView.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/qe/cache/SqlCacheVersionProvider.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/mvcc/MvccReadSelectorTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/mvcc/PluginDrivenPartitionViewTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/QueryPartitionCollectorTest.java create mode 100644 regression-test/data/external_table_p0/iceberg/test_iceberg_table_handle_reuse.out create mode 100644 regression-test/suites/external_table_p0/iceberg/test_iceberg_table_handle_reuse.groovy diff --git a/fe/fe-connector/README.md b/fe/fe-connector/README.md index b455e1c0bfcda9..89d1a0fb422070 100644 --- a/fe/fe-connector/README.md +++ b/fe/fe-connector/README.md @@ -311,6 +311,42 @@ String, convert inside `of()`, log a warning) explicitly. `fe-connector-adbc` is the reference implementation of both classes. +## Snapshot Metadata and Partition Views + +The engine must pin a table read before binding columns or building scan ranges. +Derived metadata caches must describe that same read, including the selected +snapshot/ref, schema and partition-spec generation. Never obtain a newer table +independently to build one part of an already pinned read. + +Connectors can supply `getPartitionViewVersion` as a cheap generation key for +an immutable partition view and its engine pruning index. Derive this key from +the pinned metadata; do not enumerate or hash every partition to calculate it. +A metadata refresh invalidates the corresponding engine cache, while in-flight +readers retain their existing view. Query-specific conversions must not mutate +objects shared by the connector cache. + +A connector declaring `supportsNativePartitionPlanning` can defer the engine +partition view until a consumer actually requires internal partition semantics. +For Iceberg, `enable_iceberg_internal_partition_pruning` defaults to `false`: +ordinary scans use native manifest/file pruning. Setting it to `true` enables +Doris partition pruning. MV planning and partition metadata rules still request +the full view; deferred metadata must never be treated as an empty table. + +`getSqlCacheVersion` supplies a lightweight result-cache version for a pinned +read without loading its partitions. The version must distinguish table +identity and all metadata that can change the query result or bound schema. +The engine records each table reference separately, so reading the same table +through current, historical and branch/tag references does not collapse their +cache dependencies. Cache validation resolves those original references again: +a moved or removed ref, changed schema, expired snapshot or recreated table +must not reuse an incompatible cached answer. + +Within a statement, reuse the frozen table and its resource lease through +`ConnectorStatementScope`. Its read-only lookup must not load metadata on a +miss. Relation existence and name listing can reuse this metadata through the +SPI without source-specific checks in fe-core. This does not extend cache TTLs +or authorize cross-user reuse of metadata. + ## Testing and Verification - **Unit tests** live in each module; build/test recipes are in `AGENTS.md`. diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java index 8d63b2b01588cc..a07db5a2c18d23 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java @@ -55,6 +55,7 @@ import java.io.IOException; import java.util.ArrayList; import java.util.Collections; +import java.util.LinkedHashSet; import java.util.List; import java.util.Locale; import java.util.Map; @@ -97,6 +98,13 @@ public interface IcebergCatalogOps { /** Lists the view names in {@code dbName}; empty when the catalog is not a (view-enabled) ViewCatalog. */ List listViewNames(String dbName); + /** Lists relations without requiring callers to subtract and then re-add view names. */ + default List listRelationNames(String dbName) { + LinkedHashSet names = new LinkedHashSet<>(listTableNames(dbName)); + names.addAll(listViewNames(dbName)); + return new ArrayList<>(names); + } + /** Returns {@code true} iff {@code dbName.tableName} exists. */ boolean tableExists(String dbName, String tableName); @@ -400,6 +408,15 @@ public List listViewNames(String dbName) { .collect(Collectors.toList()), Collections.emptyList()); } + @Override + public List listRelationNames(String dbName) { + LinkedHashSet names = catalog.listTables(toNamespace(dbName)).stream() + .map(TableIdentifier::name) + .collect(Collectors.toCollection(LinkedHashSet::new)); + names.addAll(listViewNames(dbName)); + return new ArrayList<>(names); + } + @Override public boolean tableExists(String dbName, String tableName) { return catalog.tableExists(toTableIdentifier(dbName, tableName)); diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java index 9eed1556c26c99..b0d49647b5dfa8 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java @@ -25,6 +25,7 @@ import org.apache.doris.connector.spi.ConnectorMetadata; import org.apache.doris.connector.spi.ConnectorPartitionInfo; import org.apache.doris.connector.spi.ConnectorSession; +import org.apache.doris.connector.spi.ConnectorStatementScope; import org.apache.doris.connector.spi.ConnectorStorageContext; import org.apache.doris.connector.spi.ConnectorTableSchema; import org.apache.doris.connector.spi.ConnectorTableStatistics; @@ -45,6 +46,7 @@ import org.apache.doris.connector.spi.handle.WriteOperation; import org.apache.doris.connector.spi.mvcc.ConnectorMvccPartitionView; import org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot; +import org.apache.doris.connector.spi.mvcc.ConnectorSqlCacheVersion; import org.apache.doris.connector.spi.mvcc.ConnectorTimeTravelSpec; import org.apache.doris.connector.spi.pushdown.ConnectorAnd; import org.apache.doris.connector.spi.pushdown.ConnectorComparison; @@ -79,6 +81,7 @@ import org.apache.iceberg.SnapshotSummary; import org.apache.iceberg.SortOrder; import org.apache.iceberg.Table; +import org.apache.iceberg.TableMetadata; import org.apache.iceberg.TableOperations; import org.apache.iceberg.TableProperties; import org.apache.iceberg.TableScan; @@ -342,6 +345,17 @@ public List listTableNames(ConnectorSession session, String dbName) { } } + @Override + public List listRelationNames(ConnectorSession session, String dbName) { + try { + return executeAuthenticated(() -> catalogOps.listRelationNames(dbName)); + } catch (RuntimeException e) { + throw e; + } catch (Exception e) { + throw new RuntimeException("Failed to list relation names, error message is: " + e.getMessage(), e); + } + } + @Override public List listViewNames(ConnectorSession session, String dbName) { // Mirror legacy IcebergMetadataOps.listViewNames: wrap in the auth context; a RuntimeException @@ -455,8 +469,37 @@ private String loadTableComment(ConnectorSession session, String dbName, String @Override public Optional getTableHandle( ConnectorSession session, String dbName, String tableName) { + IcebergTableHandle handle = new IcebergTableHandle(dbName, tableName); + if (IcebergStatementScope.hasReadTable(session, dbName, tableName) + || (tableCache != null && tableCache.contains(TableIdentifier.of(dbName, tableName)))) { + return Optional.of(handle); + } + // Offline callers cannot retain a cache lease. Pure existence checks use tableExists instead. + if (session == null || session.getStatementScope() == ConnectorStatementScope.NONE) { + return tableExists(session, dbName, tableName) ? Optional.of(handle) : Optional.empty(); + } + try { + // Loading proves existence and pins the same metadata used by schema/snapshot/scan planning. + // Borrow through the existing generation-aware cache so concurrent cold readers share one load. + executeAuthenticated(() -> resolveTableForRead(session, handle)); + return Optional.of(handle); + } catch (Exception e) { + if (ExceptionUtils.getThrowableList(e).stream().anyMatch(NoSuchTableException.class::isInstance)) { + return Optional.empty(); + } + throw IcebergExceptionUtils.wrapTableLoadFailure(handle, e, + "Failed to load table, error message is:"); + } + } + + @Override + public boolean tableExists( + ConnectorSession session, String dbName, String tableName) { + // Existence checks are used by DDL and must observe out-of-band drops even while a + // read-table cache entry or an in-flight reader still owns the previous metadata. + // Query planning reuses pinned/cached metadata through getTableHandle instead. // Mirror legacy IcebergMetadataOps.tableExist: wrap the remote existence check in the auth context - // (the handle build below is pure — no remote call). + // without loading and pinning a table merely to answer a DDL existence check. boolean exists; try { exists = executeAuthenticated(() -> catalogOps.tableExists(dbName, tableName)); @@ -464,17 +507,14 @@ public Optional getTableHandle( // Preserve Optional.empty even when an auth/catalog layer wraps NoSuchTableException. if (ExceptionUtils.getThrowableList(e).stream() .anyMatch(NoSuchTableException.class::isInstance)) { - return Optional.empty(); + return false; } // Existence checks in several catalogs load metadata internally, so normalize them at the shared // handle boundary just like explicit table loads. throw IcebergExceptionUtils.wrapTableLoadFailure(new IcebergTableHandle(dbName, tableName), e, "Failed to check table exist, error message is:"); } - if (!exists) { - return Optional.empty(); - } - return Optional.of(new IcebergTableHandle(dbName, tableName)); + return exists; } @Override @@ -2423,6 +2463,32 @@ public boolean supportsCastPredicatePushdown(ConnectorSession session) { return true; } + @Override + public Optional getSqlCacheVersion( + ConnectorSession session, ConnectorTableHandle handle) { + IcebergTableHandle iceHandle = (IcebergTableHandle) handle; + try { + return executeAuthenticated(() -> { + Table table = resolveTableForRead(session, iceHandle); + String version = metadataVersion(table, iceHandle).orElseThrow( + () -> new UnsupportedOperationException("Iceberg SQL cache requires published table metadata")); + long updateMillis = iceHandle.getSnapshotId() < 0 ? 0 + : table.snapshot(iceHandle.getSnapshotId()).timestampMillis(); + return Optional.of(new ConnectorSqlCacheVersion(version, updateMillis)); + }); + } catch (Exception e) { + throw IcebergExceptionUtils.wrapTableLoadFailure(iceHandle, e, + "Failed to get iceberg SQL cache version, error message is:"); + } + } + + @Override + public boolean supportsNativePartitionPlanning(ConnectorSession session, ConnectorTableHandle handle) { + return !((IcebergTableHandle) handle).isSystemTable() + && (session == null || !Boolean.parseBoolean(session.getSessionProperties() + .getOrDefault("enable_iceberg_internal_partition_pruning", "false"))); + } + // ========== B-2: partition enumeration (MTMV RANGE view + SHOW PARTITIONS) ========== /** @@ -2527,6 +2593,39 @@ public List listPartitionNames(ConnectorSession session, ConnectorTableH } } + private static Optional metadataVersion(Table table, IcebergTableHandle handle) { + // Custom catalog tables and unpublished metadata cannot supply a stable metadata generation. + if (!(table instanceof BaseTable)) { + return Optional.empty(); + } + TableMetadata metadata = ((BaseTable) table).operations().current(); + String location = metadata.metadataFileLocation(); + if (location == null) { + return Optional.empty(); + } + return Optional.of(metadata.uuid() + ":" + handle.getSnapshotId() + ":" + + handle.getSchemaId() + ":" + location); + } + + @Override + public Optional getPartitionViewVersion(ConnectorSession session, ConnectorTableHandle handle) { + IcebergTableHandle iceHandle = (IcebergTableHandle) handle; + if (!iceHandle.hasSnapshotPin() || session == null + || session.getStatementScope() == ConnectorStatementScope.NONE) { + // Without a statement-frozen table, metadata could change between version lookup and listing. + return Optional.empty(); + } + try { + return executeAuthenticated(() -> { + Table table = resolveTableForRead(session, iceHandle); + return metadataVersion(table, iceHandle); + }); + } catch (Exception e) { + throw IcebergExceptionUtils.wrapTableLoadFailure(iceHandle, e, + "Failed to resolve iceberg partition view version, error message is:"); + } + } + /** * The physical iceberg partitions of {@code handle} with per-partition value maps, so the generic * {@code PluginDrivenExternalTable.getNameToPartitionItems} can populate {@code selectedPartitionNum} @@ -2595,7 +2694,7 @@ private List listPartitionsUncached( } private List listPartitions(Table table, IcebergTableHandle iceHandle) { - return IcebergPartitionUtils.listPartitions(table, + return IcebergPartitionUtils.listPartitions(table, iceHandle.getSnapshotId(), TableIdentifier.of(iceHandle.getDbName(), iceHandle.getTableName()), partitionCache); } @@ -2607,7 +2706,8 @@ private List listPartitions(Table table, IcebergTableHan * snapshot id (or {@code -1} for an empty table — iceberg DOES support MVCC, so it still pins, mirroring * paimon), and the LATEST schema id (NOT {@code currentSnapshot().schemaId()} — a schema-only change without * a new snapshot advances the schema while the snapshot's id lags; legacy reads {@code table.schema() - * .schemaId()}). T08 serves this through the per-catalog {@link IcebergLatestSnapshotCache}: within the TTL + * .schemaId()}). An already frozen statement read takes precedence over the shared pin. Otherwise T08 + * serves this through the per-catalog {@link IcebergLatestSnapshotCache}: within the TTL * a HIT returns the cached pin without re-loading the table (saves the load I/O and keeps the snapshot * STABLE across queries — the legacy with-cache catalog). The cached value carries BOTH ids atomically so a * schema-only ALTER between two queries cannot skew snapshotId vs schemaId. A disabled cache @@ -2623,11 +2723,21 @@ public Optional beginQuerySnapshot( // per-user loadTable authorization) reads live per-user every call, so authorization runs every time (no // stale-authz window); mirrors resolveTableForRead's tableCache null-fallback. A disabled cache (the table // entry turned off) is a non-null cache that reads live internally. - IcebergLatestSnapshotCache.CachedSnapshot pin = latestSnapshotCache != null - ? latestSnapshotCache.getOrLoad(id, () -> loadLatestSnapshotPin(session, iceHandle)) - : loadLatestSnapshotPin(session, iceHandle); + IcebergLatestSnapshotCache.CachedSnapshot pin; + if (IcebergStatementScope.hasReadTable(session, iceHandle.getDbName(), iceHandle.getTableName())) { + // Handle/schema resolution may have frozen an older generation before another query's REFRESH. + // A newer shared pin must not point outside that frozen metadata, nor may this reader publish its + // old pin over the refreshed cache entry. + pin = latestSnapshotPin(loadTable(session, iceHandle)); + } else { + pin = latestSnapshotCache != null + ? latestSnapshotCache.getOrLoad(id, () -> loadLatestSnapshotPin(session, iceHandle)) + : loadLatestSnapshotPin(session, iceHandle); + } + // SQL result-cache identity and column binding must describe the same read generation. The engine + // must not combine this snapshot/schema pin with an independently expiring name-keyed schema cache. ConnectorMvccSnapshot.Builder snapshot = ConnectorMvccSnapshot.builder() - .snapshotId(pin.snapshotId).schemaId(pin.schemaId); + .snapshotId(pin.snapshotId).schemaId(pin.schemaId).retainSchema(true); if (pin.snapshotId < 0) { snapshot.property(EMPTY_PARTITION_STYLE_PROPERTY, pin.emptyPartitionStyle.name()); } diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java index 7c6d929d6031c6..9e34b54279206f 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java @@ -876,10 +876,15 @@ static List listPartitions(Table table) { /** Cache-aware overload (PERF-02): see {@link #buildMvccPartitionView(Table, long, TableIdentifier, * IcebergPartitionCache)}. Keyed by the table's CURRENT snapshot id. */ static List listPartitions(Table table, TableIdentifier id, IcebergPartitionCache cache) { + return listPartitions(table, -1L, id, cache); + } + + static List listPartitions(Table table, long pinnedSnapshotId, + TableIdentifier id, IcebergPartitionCache cache) { if (table.spec().isUnpartitioned()) { return Collections.emptyList(); } - Snapshot current = table.currentSnapshot(); + Snapshot current = pinnedSnapshotId >= 0 ? table.snapshot(pinnedSnapshotId) : table.currentSnapshot(); if (current == null) { return Collections.emptyList(); } diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergStatementScope.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergStatementScope.java index 06919691b1b4b9..6bb44ce3612ed2 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergStatementScope.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergStatementScope.java @@ -74,6 +74,11 @@ final class IcebergStatementScope { private IcebergStatementScope() {} + /** A frozen read or its live lease proves existence even if REFRESH retires the catalog cache. */ + static boolean hasReadTable(ConnectorSession session, String dbName, String tableName) { + return ConnectorStatementScopes.getIfPresent(session, TABLE_NAMESPACE, dbName, tableName) != null; + } + /** * Loads and freezes the iceberg {@link Table} for {@code db.tbl} once per statement. The frozen operations * prevent another statement's write refresh from changing the metadata generation after slots are bound. diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableCache.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableCache.java index 5526a52587dcb8..b727259d491fcd 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableCache.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableCache.java @@ -113,6 +113,11 @@ boolean isEnabled() { return entry.isEnabled(); } + /** Only a retained, unexpired table proves existence; never loads or caches a boolean answer. */ + boolean contains(TableIdentifier identifier) { + return entry.getIfPresent(identifier) != null; + } + /** * Returns the cached table for {@code identifier} if present and unexpired, else runs {@code loader} (the * live remote {@code loadTable}), caches and returns it. When caching is disabled ({@link #isEnabled()} is diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/CatalogBackedIcebergCatalogOpsTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/CatalogBackedIcebergCatalogOpsTest.java index 5b8f1bfd5e73d2..4c70aa1c32ed9f 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/CatalogBackedIcebergCatalogOpsTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/CatalogBackedIcebergCatalogOpsTest.java @@ -226,6 +226,29 @@ public void listTableNamesDoesNotFilterWhenRestViewDisabled() { "listViews must not be called when view filtering is disabled"); } + @Test + public void relationListingLoadsEachRemoteListOnceAndDeduplicatesViews() { + FakeIcebergViewCatalog catalog = new FakeIcebergViewCatalog(); + catalog.tablesByNs.put(Namespace.of("db1"), Arrays.asList("t1", "v1", "t2")); + catalog.viewsByNs.put(Namespace.of("db1"), Arrays.asList("v1", "v2")); + IcebergCatalogOps ops = ops(catalog, true, false, true, Optional.empty()); + Assertions.assertEquals(Arrays.asList("t1", "v1", "t2", "v2"), ops.listRelationNames("db1")); + Assertions.assertEquals(1, Collections.frequency(catalog.log, "listTables:db1")); + Assertions.assertEquals(1, Collections.frequency(catalog.log, "listViews:db1")); + catalog.viewsByNs.put(Namespace.of("db1"), Collections.singletonList("v3")); + Assertions.assertEquals(Arrays.asList("t1", "v1", "t2", "v3"), ops.listRelationNames("db1"), + "the combined listing must not introduce a long-lived view-name cache"); + } + + @Test + public void relationListingDoesNotAccessDisabledRestViews() { + FakeIcebergViewCatalog catalog = new FakeIcebergViewCatalog(); + catalog.tablesByNs.put(Namespace.of("db1"), Arrays.asList("t1", "v1")); + IcebergCatalogOps ops = ops(catalog, true, false, false, Optional.empty()); + Assertions.assertEquals(Arrays.asList("t1", "v1"), ops.listRelationNames("db1")); + Assertions.assertFalse(catalog.log.contains("listViews:db1")); + } + // --------------------------------------------------------------------- // listViewNames / viewExists — the inverse of listTableNames' view subtraction (B0 view SPI) // --------------------------------------------------------------------- diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataMvccTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataMvccTest.java index 7aa7bef38e2fd0..de4536c64bb862 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataMvccTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataMvccTest.java @@ -18,6 +18,7 @@ package org.apache.doris.connector.iceberg; import org.apache.doris.connector.spi.ConnectorColumn; +import org.apache.doris.connector.spi.ConnectorSession; import org.apache.doris.connector.spi.ConnectorTableSchema; import org.apache.doris.connector.spi.DorisConnectorException; import org.apache.doris.connector.spi.handle.ConnectorTableHandle; @@ -42,6 +43,7 @@ import java.util.Collections; import java.util.List; +import java.util.Map; import java.util.Optional; import java.util.stream.Collectors; @@ -108,6 +110,82 @@ private static IcebergConnectorMetadata metadataFor(Table table, RecordingIceber return new IcebergConnectorMetadata(ops, IcebergCatalogProperties.of(Collections.emptyMap()), new RecordingConnectorContext()); } + @Test + public void nativePartitionPlanningDoesNotOptInSystemTables() { + Fixture f = fixture(); + IcebergConnectorMetadata md = metadataFor(f.table, new RecordingIcebergCatalogOps()); + Assertions.assertTrue(md.supportsNativePartitionPlanning(null, handle())); + Assertions.assertFalse(md.supportsNativePartitionPlanning(null, + IcebergTableHandle.forSystemTable("db1", "t1", "snapshots", -1, null, -1))); + } + + @Test + public void nativePartitionPlanningFollowsInternalPartitionPruningSwitch() { + Fixture f = fixture(); + IcebergConnectorMetadata md = metadataFor(f.table, new RecordingIcebergCatalogOps()); + String key = "enable_iceberg_internal_partition_pruning"; + Assertions.assertTrue(md.supportsNativePartitionPlanning(session(Map.of(), Map.of()), handle())); + Assertions.assertTrue(md.supportsNativePartitionPlanning(session(Map.of(key, "false"), Map.of()), handle())); + Assertions.assertTrue(md.supportsNativePartitionPlanning(session(Map.of(), Map.of(key, "true")), handle()), + "catalog properties must not override the session switch"); + for (String value : List.of("true", "TRUE")) { + ConnectorSession session = session(Map.of(key, value), Map.of()); + Assertions.assertFalse(md.supportsNativePartitionPlanning(session, handle())); + Assertions.assertFalse(md.supportsNativePartitionPlanning(session, + IcebergTableHandle.forSystemTable("db1", "t1", "snapshots", -1, null, -1))); + } + } + + private static ConnectorSession session(Map properties, Map catalogProperties) { + return new ConnectorSession() { + @Override + public String getQueryId() { + return "q"; + } + + @Override + public String getUser() { + return "u"; + } + + @Override + public String getTimeZone() { + return "UTC"; + } + + @Override + public String getLocale() { + return "en_US"; + } + + @Override + public long getCatalogId() { + return 0; + } + + @Override + public String getCatalogName() { + return "c"; + } + + @Override + public T getProperty(String name, Class type) { + String value = properties.getOrDefault(name, catalogProperties.get(name)); + return value == null ? null : type.cast(value); + } + + @Override + public Map getCatalogProperties() { + return catalogProperties; + } + + @Override + public Map getSessionProperties() { + return properties; + } + }; + } + private static ConnectorTableHandle handle() { return new IcebergTableHandle("db1", "t1"); } @@ -129,6 +207,8 @@ public void beginQuerySnapshotPinsCurrentSnapshotAndLatestSchema() { // MUTATION: pinning currentSnapshot().schemaId() instead of table.schema().schemaId() would still be // schemaIdS2 here (same after the latest snapshot), so the load-bearing assertion is "current snapshot". Assertions.assertTrue(snap.isPresent()); + Assertions.assertTrue(snap.get().isSchemaRetained(), + "SQL cache identity and column binding must use the same pinned schema"); Assertions.assertEquals(f.s2, snap.get().getSnapshotId()); Assertions.assertEquals(f.schemaIdS2, snap.get().getSchemaId()); // The remote load goes through the seam (auth-wrapped). diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataPartitionViewCacheTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataPartitionViewCacheTest.java index 204266c1536c66..d20daa98f70052 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataPartitionViewCacheTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataPartitionViewCacheTest.java @@ -25,6 +25,7 @@ import org.apache.doris.connector.spi.mvcc.ConnectorMvccPartition; import org.apache.doris.connector.spi.mvcc.ConnectorMvccPartitionView; import org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot; +import org.apache.doris.connector.spi.mvcc.ConnectorSqlCacheVersion; import org.apache.doris.connector.spi.pushdown.ConnectorExpression; import org.apache.iceberg.DataFiles; @@ -123,11 +124,71 @@ private static ConnectorMetadataCache> listCache() return new ConnectorMetadataCache<>("iceberg", "partition_view", Collections.emptyMap()); } + @Test + public void sqlCacheVersionPinsMetadataAndDetectsSchemaChangesAndRollback() { + TwoSnap f = twoSnapshotTable(); + RecordingIcebergCatalogOps ops = new RecordingIcebergCatalogOps(); + ops.table = f.table; + IcebergConnectorMetadata md = metadataWithMvccCache(ops, mvccCache()); + IcebergTableHandle first = handle().withSnapshot(f.s1, null, f.schemaId); + IcebergTableHandle second = handle().withSnapshot(f.s2, null, f.schemaId); + IcebergStatementScopeTest.ScopeSession session = new IcebergStatementScopeTest.ScopeSession( + 1L, "cache-query", new TestStatementScope()); + ConnectorSqlCacheVersion v1 = md.getSqlCacheVersion(session, first).orElseThrow(); + ConnectorSqlCacheVersion v2 = md.getSqlCacheVersion(session, second).orElseThrow(); + Assertions.assertNotEquals(v1.getVersion(), v2.getVersion()); + Assertions.assertEquals(f.table.snapshot(f.s1).timestampMillis(), v1.getUpdateTimeMillis()); + Assertions.assertEquals(v2.getVersion(), md.getSqlCacheVersion(session, + handle().withSnapshot(f.s2, "branch", f.schemaId)).orElseThrow().getVersion()); + f.table.updateSchema().addColumn("extra", Types.StringType.get()).commit(); + Assertions.assertEquals(v2.getVersion(), md.getSqlCacheVersion(session, second).orElseThrow().getVersion(), + "an existing statement must not observe a later metadata-only commit"); + IcebergStatementScopeTest.ScopeSession next = new IcebergStatementScopeTest.ScopeSession( + 1L, "next-cache-query", new TestStatementScope()); + IcebergTableHandle evolved = handle().withSnapshot(f.s2, null, f.table.schema().schemaId()); + Assertions.assertNotEquals(v2.getVersion(), md.getSqlCacheVersion(next, evolved).orElseThrow().getVersion()); + // Snapshot ids are opaque; a rollback changes the identity even when its commit time is older. + f.table.manageSnapshots().rollbackTo(f.s1).commit(); + IcebergStatementScopeTest.ScopeSession rollback = new IcebergStatementScopeTest.ScopeSession( + 1L, "rollback-query", new TestStatementScope()); + Assertions.assertNotEquals(v2.getVersion(), md.getSqlCacheVersion(rollback, + handle().withSnapshot(f.s1, null, f.table.schema().schemaId())).orElseThrow().getVersion()); + Assertions.assertTrue(ops.log.stream().noneMatch(entry -> entry.startsWith("listPartitions")), ops.log.toString()); + } + private static List mvccNames(Optional view) { return view.get().getPartitions().stream() .map(ConnectorMvccPartition::getName).collect(Collectors.toList()); } + @Test + public void versionTokenAndListUseThePinnedSnapshot() { + TwoSnap f = twoSnapshotTable(); + RecordingIcebergCatalogOps ops = new RecordingIcebergCatalogOps(); + ops.table = f.table; + IcebergConnectorMetadata md = metadataWithListCache(ops, listCache()); + IcebergTableHandle first = handle().withSnapshot(f.s1, null, f.schemaId); + IcebergTableHandle second = handle().withSnapshot(f.s2, null, f.schemaId); + IcebergStatementScopeTest.ScopeSession session = new IcebergStatementScopeTest.ScopeSession( + 1L, "version-query", new TestStatementScope()); + String firstVersion = md.getPartitionViewVersion(session, first).orElseThrow(); + Assertions.assertEquals(firstVersion, md.getPartitionViewVersion(session, + handle().withSnapshot(f.s1, "branch-alias", f.schemaId)).orElseThrow()); + Assertions.assertNotEquals(firstVersion, md.getPartitionViewVersion(session, second).orElseThrow()); + Assertions.assertFalse(md.getPartitionViewVersion(session, handle()).isPresent()); + Assertions.assertEquals(1, md.listPartitions(session, first, Optional.empty()).size()); + Assertions.assertEquals(2, md.listPartitions(session, second, Optional.empty()).size()); + // Metadata-only evolution must change the token even without a new data snapshot. + f.table.updateSpec().addField("id").commit(); + Assertions.assertEquals(firstVersion, md.getPartitionViewVersion(session, first).orElseThrow(), + "an existing statement must retain its metadata generation"); + IcebergStatementScopeTest.ScopeSession next = new IcebergStatementScopeTest.ScopeSession( + 1L, "next-query", new TestStatementScope()); + Assertions.assertNotEquals(firstVersion, md.getPartitionViewVersion(next, first).orElseThrow()); + Assertions.assertFalse(md.getPartitionViewVersion(null, first).isPresent()); + Assertions.assertEquals(f.s2, f.table.currentSnapshot().snapshotId()); + } + @Test public void derivedViewWeightsIncludeTailAtEveryPosition() { ConnectorTableKey key = new ConnectorTableKey("db", "table", 1L, 1L); diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableHandleReuseTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableHandleReuseTest.java new file mode 100644 index 00000000000000..93111136cd3de7 --- /dev/null +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableHandleReuseTest.java @@ -0,0 +1,311 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.iceberg; + +import org.apache.doris.connector.spi.ConnectorSession; +import org.apache.doris.connector.spi.ConnectorStatementScope; +import org.apache.doris.connector.spi.handle.ConnectorTableHandle; +import org.apache.doris.connector.spi.mvcc.ConnectorTimeTravelSpec; + +import org.apache.iceberg.DataFiles; +import org.apache.iceberg.PartitionSpec; +import org.apache.iceberg.Schema; +import org.apache.iceberg.Table; +import org.apache.iceberg.catalog.Namespace; +import org.apache.iceberg.catalog.TableIdentifier; +import org.apache.iceberg.inmemory.InMemoryCatalog; +import org.apache.iceberg.types.Types; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.concurrent.atomic.AtomicInteger; + +class IcebergTableHandleReuseTest { + private static final TableIdentifier ID = TableIdentifier.of("db1", "t1"); + private static final Schema SCHEMA = new Schema(Types.NestedField.required(1, "id", Types.IntegerType.get())); + + private static ConnectorSession session(String query, ConnectorStatementScope scope) { + return new IcebergStatementScopeTest.ScopeSession(7, query, scope); + } + + private static RecordingIcebergCatalogOps ops() { + RecordingIcebergCatalogOps ops = new RecordingIcebergCatalogOps(); + ops.table = new FakeIcebergTable("t1", SCHEMA, PartitionSpec.unpartitioned(), + "s3://bucket/db1/t1", Collections.emptyMap()); + ops.tableExists = true; + return ops; + } + + private static IcebergConnectorMetadata metadata(RecordingIcebergCatalogOps ops, IcebergTableCache cache) { + return new IcebergConnectorMetadata(ops, IcebergCatalogProperties.of(Collections.emptyMap()), + new RecordingConnectorContext(), new IcebergLatestSnapshotCache(0L, 1), cache); + } + + private static long existenceCalls(RecordingIcebergCatalogOps ops) { + return ops.log.stream().filter(s -> s.startsWith("tableExists:")).count(); + } + + @Test + void loadedStatementTableProvesExistenceWithoutSharedCache() { + RecordingIcebergCatalogOps ops = ops(); + IcebergConnectorMetadata metadata = metadata(ops, null); + TestStatementScope scope = new TestStatementScope(); + ConnectorSession session = session("q1", scope); + ConnectorTableHandle handle = metadata.getTableHandle(session, "db1", "t1").orElseThrow(); + metadata.getTableSchema(session, handle); + ops.tableExistsFailure = new IllegalStateException("must not contact remote after pinning"); + ConnectorTableHandle again = metadata.getTableHandle(session, "db1", "t1").orElseThrow(); + Assertions.assertNotSame(handle, again, "scan handles remain independent of snapshot/pushdown refinements"); + Assertions.assertEquals(0, existenceCalls(ops)); + ops.loadTableFailure = new IllegalStateException("new statement must load independently"); + Assertions.assertThrows(RuntimeException.class, + () -> metadata.getTableHandle(session("q2", scope), "db1", "t1")); + Assertions.assertThrows(RuntimeException.class, + () -> metadata.getTableHandle(new IcebergStatementScopeTest.ScopeSession(8, "q1", scope), + "db1", "t1")); + Assertions.assertThrows(RuntimeException.class, + () -> metadata.getTableHandle(session, "db1", "other")); + scope.closeAll(); + } + + @Test + void cacheHitAvoidsRemoteAcrossStatementsAndInvalidationKeepsOnlyExistingReader() { + AtomicInteger closes = new AtomicInteger(); + IcebergTableCache cache = new IcebergTableCache(3600, 10, table -> closes::incrementAndGet); + RecordingIcebergCatalogOps ops = ops(); + IcebergConnectorMetadata metadata = metadata(ops, cache); + TestStatementScope first = new TestStatementScope(); + ConnectorSession reader = session("q1", first); + metadata.getTableSchema(reader, new IcebergTableHandle("db1", "t1")); + Assertions.assertTrue(cache.contains(ID)); + Assertions.assertTrue(metadata.getTableHandle(session("q2", new TestStatementScope()), + "db1", "t1").isPresent()); + Assertions.assertEquals(0, existenceCalls(ops)); + cache.invalidate(ID); + ops.tableExists = false; + ops.throwNoSuchTableOnLoadTable = true; + Assertions.assertFalse(cache.contains(ID)); + Assertions.assertTrue(metadata.getTableHandle(reader, "db1", "t1").isPresent()); + Assertions.assertTrue(metadata.getTableHandle(session("q3", new TestStatementScope()), + "db1", "t1").isEmpty()); + Assertions.assertEquals(0, closes.get(), "REFRESH cannot close an active reader's metadata/FileIO"); + first.closeAll(); + Assertions.assertEquals(1, closes.get()); + Assertions.assertTrue(metadata.getTableHandle(reader, "db1", "t1").isEmpty()); + cache.close(); + } + + @Test + void existenceChecksDoNotLoadTablesOrRetainPositiveNegativeOrFailedChecks() { + RecordingIcebergCatalogOps ops = ops(); + IcebergTableCache cache = new IcebergTableCache(3600, 10); + IcebergConnectorMetadata metadata = metadata(ops, cache); + ConnectorSession session = session("ddl", new TestStatementScope()); + Assertions.assertTrue(metadata.tableExists(session, "db1", "t1")); + ops.tableExists = false; + ops.throwNoSuchTableOnLoadTable = true; + Assertions.assertFalse(metadata.tableExists(session, "db1", "t1")); + ops.tableExists = true; // e.g. CTAS resolves the same target after creation. + Assertions.assertTrue(metadata.tableExists(session, "db1", "t1")); + ops.tableExistsFailure = new IllegalStateException("permission denied"); + Assertions.assertThrows(RuntimeException.class, + () -> metadata.tableExists(session, "db1", "t1")); + ops.tableExistsFailure = null; + Assertions.assertTrue(metadata.tableExists(session, "db1", "t1")); + Assertions.assertEquals(5, existenceCalls(ops)); + Assertions.assertFalse(ops.log.stream().anyMatch(s -> s.startsWith("loadTable:"))); + Assertions.assertEquals(0, cache.size()); + cache.close(); + } + + @Test + void ddlExistenceRemainsLiveWithCachedAndPinnedReadMetadata() { + RecordingIcebergCatalogOps ops = ops(); + IcebergTableCache cache = new IcebergTableCache(3600, 10); + IcebergConnectorMetadata metadata = metadata(ops, cache); + TestStatementScope scope = new TestStatementScope(); + ConnectorSession reader = session("reader", scope); + metadata.getTableSchema(reader, new IcebergTableHandle("db1", "t1")); + Assertions.assertTrue(cache.contains(ID)); + ops.tableExists = false; + ops.throwNoSuchTableOnLoadTable = true; + Assertions.assertFalse(metadata.tableExists(reader, "db1", "t1")); + Assertions.assertFalse(metadata.tableExists(session("ddl", new TestStatementScope()), "db1", "t1")); + Assertions.assertFalse(metadata.tableExists(null, "db1", "t1")); + Assertions.assertTrue(metadata.getTableHandle(reader, "db1", "t1").isPresent(), + "the existing reader keeps its frozen metadata independently of live DDL existence"); + Assertions.assertEquals(3, existenceCalls(ops)); + scope.closeAll(); + cache.close(); + } + + @Test + void disabledCacheSharesOnlyTheCurrentStatementAndNoContextNeverRetainsProof() { + RecordingIcebergCatalogOps ops = ops(); + IcebergTableCache cache = new IcebergTableCache(0, 10); + IcebergConnectorMetadata metadata = metadata(ops, cache); + TestStatementScope scope = new TestStatementScope(); + ConnectorSession session = session("q1", scope); + metadata.getTableSchema(session, new IcebergTableHandle("db1", "t1")); + Assertions.assertFalse(cache.contains(ID)); + ops.tableExists = false; + ops.throwNoSuchTableOnLoadTable = true; + Assertions.assertTrue(metadata.getTableHandle(session, "db1", "t1").isPresent()); + Assertions.assertTrue(metadata.getTableHandle(session("q2", new TestStatementScope()), + "db1", "t1").isEmpty()); + Assertions.assertTrue(metadata.getTableHandle(null, "db1", "t1").isEmpty()); + Assertions.assertTrue(metadata.getTableHandle(session("offline", ConnectorStatementScope.NONE), + "db1", "t1").isEmpty()); + scope.closeAll(); + cache.close(); + } + + @Test + void refreshRetainsOldSchemaAndRefsWhileNewStatementSeesSchemaAndBranchMovement() { + InMemoryCatalog catalog = new InMemoryCatalog(); + catalog.initialize("handles", Collections.emptyMap()); + catalog.createNamespace(Namespace.of("db1")); + Table table = catalog.createTable(ID, SCHEMA); + append(table, "first.parquet"); + long firstSnapshot = table.currentSnapshot().snapshotId(); + table.manageSnapshots().createBranch("audit", firstSnapshot).createTag("old", firstSnapshot).commit(); + RecordingIcebergCatalogOps ops = ops(); + ops.table = table; + IcebergTableCache cache = new IcebergTableCache(3600, 10); + IcebergConnectorMetadata metadata = metadata(ops, cache); + TestStatementScope first = new TestStatementScope(); + ConnectorSession old = session("q1", first); + ConnectorTableHandle oldHandle = metadata.getTableHandle(old, "db1", "t1").orElseThrow(); + // The cold handle must pin metadata before schema binding, not merely after getTableSchema. + table.updateSchema().updateColumn("id", Types.LongType.get()).commit(); + append(table, "second.parquet"); + long secondSnapshot = table.currentSnapshot().snapshotId(); + table.manageSnapshots().replaceBranch("audit", secondSnapshot).commit(); + cache.invalidate(ID); + Assertions.assertTrue(metadata.getTableHandle(old, "db1", "t1").isPresent()); + Assertions.assertEquals("INT", metadata.getTableSchema(old, oldHandle).getColumns().get(0) + .getType().getTypeName()); + Assertions.assertEquals(firstSnapshot, metadata.resolveTimeTravel(old, oldHandle, + ConnectorTimeTravelSpec.branch("audit")).orElseThrow().getSnapshotId()); + TestStatementScope next = new TestStatementScope(); + ConnectorSession fresh = session("q2", next); + ConnectorTableHandle freshHandle = metadata.getTableHandle(fresh, "db1", "t1").orElseThrow(); + Assertions.assertEquals("BIGINT", metadata.getTableSchema(fresh, freshHandle).getColumns().get(0) + .getType().getTypeName()); + Assertions.assertEquals(secondSnapshot, metadata.resolveTimeTravel(fresh, freshHandle, + ConnectorTimeTravelSpec.branch("audit")).orElseThrow().getSnapshotId()); + Assertions.assertEquals(firstSnapshot, metadata.resolveTimeTravel(fresh, + metadata.getTableHandle(fresh, "db1", "t1").orElseThrow(), + ConnectorTimeTravelSpec.tag("old")).orElseThrow().getSnapshotId()); + Assertions.assertEquals(firstSnapshot, metadata.resolveTimeTravel(fresh, freshHandle, + ConnectorTimeTravelSpec.snapshotId(Long.toString(firstSnapshot))).orElseThrow().getSnapshotId()); + Assertions.assertEquals(0, existenceCalls(ops), "the pinned load proves existence without a HEAD request"); + first.closeAll(); + next.closeAll(); + cache.close(); + } + + @Test + void coldHandleLoadsOnceWithoutExistenceCheckAndMissingTableCanBeCreatedLater() { + RecordingIcebergCatalogOps ops = ops(); + IcebergTableCache cache = new IcebergTableCache(3600, 10); + IcebergConnectorMetadata metadata = metadata(ops, cache); + TestStatementScope scope = new TestStatementScope(); + ConnectorSession session = session("cold", scope); + ops.throwNoSuchTableOnLoadTable = true; + Assertions.assertTrue(metadata.getTableHandle(session, "db1", "t1").isEmpty()); + ops.throwNoSuchTableOnLoadTable = false; + ConnectorTableHandle handle = metadata.getTableHandle(session, "db1", "t1").orElseThrow(); + metadata.getTableSchema(session, handle); + metadata.getTableHandle(session, "db1", "t1").orElseThrow(); + Assertions.assertEquals(0, existenceCalls(ops)); + Assertions.assertEquals(2, ops.log.stream().filter(call -> call.startsWith("loadTable:")).count(), + "one failed load and one successful load; no negative entry or schema reload"); + scope.closeAll(); + cache.close(); + } + + @Test + void coldHandleDoesNotHidePermissionFailureOrCacheItAsMissing() { + RecordingIcebergCatalogOps ops = ops(); + IcebergTableCache cache = new IcebergTableCache(3600, 10); + IcebergConnectorMetadata metadata = metadata(ops, cache); + TestStatementScope scope = new TestStatementScope(); + ConnectorSession session = session("denied", scope); + ops.loadTableFailure = new org.apache.iceberg.exceptions.ForbiddenException("denied"); + Assertions.assertThrows(RuntimeException.class, () -> metadata.getTableHandle(session, "db1", "t1")); + ops.loadTableFailure = null; + Assertions.assertTrue(metadata.getTableHandle(session, "db1", "t1").isPresent()); + Assertions.assertEquals(0, existenceCalls(ops)); + scope.closeAll(); + cache.close(); + } + + @Test + void refreshedSharedSnapshotPinCannotOverrideAnAlreadyFrozenColdHandle() { + InMemoryCatalog catalog = new InMemoryCatalog(); + catalog.initialize("cold-pin-race", Collections.emptyMap()); + catalog.createNamespace(Namespace.of("db1")); + Table table = catalog.createTable(ID, SCHEMA); + append(table, "first.parquet"); + long oldSnapshot = table.currentSnapshot().snapshotId(); + RecordingIcebergCatalogOps ops = ops(); + ops.table = table; + IcebergTableCache cache = new IcebergTableCache(3600, 10); + IcebergLatestSnapshotCache snapshots = new IcebergLatestSnapshotCache(3600, 10); + IcebergConnectorMetadata metadata = new IcebergConnectorMetadata(ops, + IcebergCatalogProperties.of(Collections.emptyMap()), new RecordingConnectorContext(), snapshots, cache); + TestStatementScope oldScope = new TestStatementScope(); + TestStatementScope freshScope = new TestStatementScope(); + ConnectorSession old = session("old", oldScope); + ConnectorSession fresh = session("fresh", freshScope); + try { + ConnectorTableHandle oldHandle = metadata.getTableHandle(old, "db1", "t1").orElseThrow(); + table.updateSchema().updateColumn("id", Types.LongType.get()).commit(); + append(table, "second.parquet"); + long newSnapshot = table.currentSnapshot().snapshotId(); + cache.invalidate(ID); + snapshots.invalidate(ID); + ConnectorTableHandle freshHandle = metadata.getTableHandle(fresh, "db1", "t1").orElseThrow(); + Assertions.assertEquals(newSnapshot, + metadata.beginQuerySnapshot(fresh, freshHandle).orElseThrow().getSnapshotId()); + snapshots.getOrLoad(ID, () -> new IcebergLatestSnapshotCache.CachedSnapshot( + newSnapshot, table.schema().schemaId())); + org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot oldPin = + metadata.beginQuerySnapshot(old, oldHandle).orElseThrow(); + Assertions.assertEquals(oldSnapshot, oldPin.getSnapshotId(), + "another query's refreshed pin cannot refer to a snapshot missing from the frozen old table"); + Assertions.assertEquals("INT", metadata.getTableSchema(old, oldHandle, oldPin) + .getColumns().get(0).getType().getTypeName()); + Assertions.assertEquals(newSnapshot, snapshots.getOrLoad(ID, + () -> { + throw new AssertionError("the refreshed shared pin must remain cached"); + }).snapshotId); + } finally { + oldScope.closeAll(); + freshScope.closeAll(); + cache.close(); + } + } + + private static void append(Table table, String path) { + table.newAppend().appendFile(DataFiles.builder(table.spec()).withPath(path).withFormat("PARQUET") + .withFileSizeInBytes(10).withRecordCount(1).build()).commit(); + } +} diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/TestStatementScope.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/TestStatementScope.java index 599b702e253ebe..0bf50731e5abed 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/TestStatementScope.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/TestStatementScope.java @@ -43,6 +43,12 @@ public T computeIfAbsent(String key, Supplier loader) { return (T) cache.computeIfAbsent(key, k -> loader.get()); } + @Override + @SuppressWarnings("unchecked") + public T getIfPresent(String key) { + return (T) cache.get(key); + } + @Override public T computeIfAbsentForPlanning(String key, Supplier loader) { T value = computeIfAbsent(key, loader); diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorMetadata.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorMetadata.java index b22e13194d49ed..f58326302f9276 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorMetadata.java @@ -20,12 +20,15 @@ import org.apache.doris.connector.spi.handle.ConnectorTableHandle; import org.apache.doris.connector.spi.mvcc.ConnectorMvccPartitionView; import org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot; +import org.apache.doris.connector.spi.mvcc.ConnectorSqlCacheVersion; import org.apache.doris.connector.spi.mvcc.ConnectorTableFreshness; import org.apache.doris.connector.spi.mvcc.ConnectorTimeTravelSpec; import org.apache.doris.connector.spi.pushdown.ConnectorExpression; import java.io.Closeable; import java.io.IOException; +import java.util.ArrayList; +import java.util.LinkedHashSet; import java.util.List; import java.util.Map; import java.util.Optional; @@ -57,6 +60,17 @@ public interface ConnectorMetadata extends ConnectorIdentifierOps, Closeable { + /** + * Lists tables and views for the engine's relation-name cache. The default combines the two domains; + * connectors with a shared remote listing may override it to avoid fetching either listing twice. + * The engine calls this only for connectors declaring SUPPORTS_VIEW. + */ + default List listRelationNames(ConnectorSession session, String dbName) { + LinkedHashSet names = new LinkedHashSet<>(listTableNames(session, dbName)); + names.addAll(listViewNames(session, dbName)); + return new ArrayList<>(names); + } + // ──────────────────── MVCC Snapshots ──────────────────── /** @@ -71,6 +85,27 @@ default Optional beginQuerySnapshot( return Optional.empty(); } + /** + * A lightweight SQL result-cache version for the supplied snapshot-pinned handle. + * Empty keeps the engine's legacy freshness path. An implementation that cannot identify a generation + * reliably must throw to disable caching for that query, rather than return an unsafe constant. + * Implementations must use the statement's frozen metadata and must not enumerate files or partitions. + */ + default Optional getSqlCacheVersion( + ConnectorSession session, ConnectorTableHandle handle) { + return Optional.empty(); + } + + /** + * Whether ordinary scans can plan files from predicates without enumerating engine partitions. + * Opting in permits the engine to defer the pinned partition view until a partition-aware consumer + * requests it. The connector must honor the pinned handle independently of an engine partition selection. + * Materialized-view planning and partition-based query limits may still request the complete view. + */ + default boolean supportsNativePartitionPlanning(ConnectorSession session, ConnectorTableHandle handle) { + return false; + } + /** Whether relation-level options need the statement's latest snapshot as their version fence. */ default boolean usesStatementSnapshotForOptions( ConnectorSession session, ConnectorTableHandle handle, Map options) { diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorPartitionListingOps.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorPartitionListingOps.java index 97aa2e05bd654c..b0864953a626f5 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorPartitionListingOps.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorPartitionListingOps.java @@ -37,6 +37,18 @@ */ public interface ConnectorPartitionListingOps { + /** + * Optional immutable generation of the unfiltered LIST and MVCC partition views for this handle. + * Equal tokens within a table must imply identical partition conversion inputs, including freshness + * and partition rules. A snapshot-applied handle must enumerate that snapshot, not a live latest. + * Resolving the token must not enumerate partitions. The token and subsequent listings must use the + * same statement-frozen metadata. Return empty when the connector cannot provide this guarantee. + * The engine additionally keys its converted view by column types and session timezone. + */ + default Optional getPartitionViewVersion(ConnectorSession session, ConnectorTableHandle handle) { + return Optional.empty(); + } + /** * Lists all partition display names (e.g., {@code "year=2024/month=01"}). * diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorStatementScope.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorStatementScope.java index 5f767a9c7e8fbd..2809da1cb1c9f1 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorStatementScope.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorStatementScope.java @@ -44,6 +44,11 @@ public interface ConnectorStatementScope { */ T computeIfAbsent(String key, Supplier loader); + /** Returns an already loaded statement value without loading or retaining anything on a miss. */ + default T getIfPresent(String key) { + return null; + } + /** * {@link #computeIfAbsent} for a value only planning reads, such as a memo that lets the identical scans of * one plan share their planned ranges. The engine drops such values once planning ends ({@link #endPlanning}), diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorStatementScopes.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorStatementScopes.java index 99d39f2e0e57d6..521cb5e369d3ef 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorStatementScopes.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorStatementScopes.java @@ -69,8 +69,16 @@ public static T resolveInStatement(ConnectorSession session, String keyNames if (session == null) { return loader.get(); } - String key = keyNamespace + ":" + session.getCatalogId() + ":" + db + ":" + table - + ":" + session.getQueryId(); - return session.getStatementScope().computeIfAbsent(key, loader); + return session.getStatementScope().computeIfAbsent(key(session, keyNamespace, db, table), loader); + } + + /** Looks up the same coordinate as resolveInStatement without starting a load. */ + public static T getIfPresent(ConnectorSession session, String keyNamespace, String db, String table) { + return session == null ? null + : session.getStatementScope().getIfPresent(key(session, keyNamespace, db, table)); + } + + private static String key(ConnectorSession session, String keyNamespace, String db, String table) { + return keyNamespace + ":" + session.getCatalogId() + ":" + db + ":" + table + ":" + session.getQueryId(); } } diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorTableMetadataOps.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorTableMetadataOps.java index 4d29727e1b84d9..26cbb583bd8e8a 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorTableMetadataOps.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorTableMetadataOps.java @@ -70,6 +70,11 @@ default Optional getTableHandle( return Optional.empty(); } + /** Checks existence without requiring the caller to retain a planning handle or read metadata. */ + default boolean tableExists(ConnectorSession session, String dbName, String tableName) { + return getTableHandle(session, dbName, tableName).isPresent(); + } + /** * Lists the system-table names supported for the given base table * (e.g. ["snapshots", "schemas", "options", "audit_log", "binlog"]). diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/mvcc/ConnectorSqlCacheVersion.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/mvcc/ConnectorSqlCacheVersion.java new file mode 100644 index 00000000000000..8d27173534eb60 --- /dev/null +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/mvcc/ConnectorSqlCacheVersion.java @@ -0,0 +1,44 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.spi.mvcc; + +import java.util.Objects; + +/** + * Immutable whole-table SQL result-cache identity and its wall-clock update time. + * The identity is compared for equality, never ordered. It must cover data, schema and table recreation, + * and describe the same pinned generation as the scan. Reading it must not enumerate files or partitions. + * The timestamp is only for the cache's quiet-window eligibility check, never for identity comparisons. + */ +public final class ConnectorSqlCacheVersion { + private final String version; + private final long updateTimeMillis; + + public ConnectorSqlCacheVersion(String version, long updateTimeMillis) { + this.version = Objects.requireNonNull(version, "version"); + this.updateTimeMillis = updateTimeMillis; + } + + public String getVersion() { + return version; + } + + public long getUpdateTimeMillis() { + return updateTimeMillis; + } +} diff --git a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java index cab3d00cb4b4be..e108648739ad9e 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java +++ b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java @@ -99,11 +99,14 @@ public void connectorApiMajorTracksTheRecordedSurfaceChange() throws IOException ConnectorProvider.class, ConnectorContext.class, ConnectorSession.class, + ConnectorStatementScope.class, + ConnectorStatementScopes.class, Connector.class, ConnectorColumnHandle.class, ConnectorColumn.class, ConnectorTableSchema.class, org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot.class, + org.apache.doris.connector.spi.mvcc.ConnectorSqlCacheVersion.class, org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot.Builder.class, ConnectorScanPlanProvider.class, org.apache.doris.connector.spi.scan.ConnectorScanRange.class, diff --git a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorRelationMetadataDefaultsTest.java b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorRelationMetadataDefaultsTest.java new file mode 100644 index 00000000000000..2997f0dae89b98 --- /dev/null +++ b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorRelationMetadataDefaultsTest.java @@ -0,0 +1,59 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.spi; + +import org.apache.doris.connector.spi.handle.ConnectorTableHandle; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.List; +import java.util.Optional; + +class ConnectorRelationMetadataDefaultsTest { + @Test + void relationNamesMergeBothDomainsWithoutDuplicates() { + ConnectorMetadata metadata = new ConnectorMetadata() { + @Override + public List listTableNames(ConnectorSession session, String dbName) { + return Arrays.asList("table", "view"); + } + + @Override + public List listViewNames(ConnectorSession session, String dbName) { + return Arrays.asList("view", "other_view"); + } + }; + Assertions.assertEquals(Arrays.asList("table", "view", "other_view"), + metadata.listRelationNames(null, "db")); + } + + @Test + void existenceDefaultPreservesHandleBasedConnectors() { + ConnectorMetadata metadata = new ConnectorMetadata() { + @Override + public Optional getTableHandle( + ConnectorSession session, String dbName, String tableName) { + return "exists".equals(tableName) ? Optional.of(new ConnectorTableHandle() {}) : Optional.empty(); + } + }; + Assertions.assertTrue(metadata.tableExists(null, "db", "exists")); + Assertions.assertFalse(metadata.tableExists(null, "db", "missing")); + } +} diff --git a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorStatementScopesTest.java b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorStatementScopesTest.java index e23eaa6fa83e18..1eda0ec9d97f07 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorStatementScopesTest.java +++ b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorStatementScopesTest.java @@ -35,6 +35,23 @@ */ public class ConnectorStatementScopesTest { + @Test + public void lookupUsesTheSameCoordinateWithoutLoadingOnMiss() { + RecordingMemoScope scope = new RecordingMemoScope(); + TestSession session = new TestSession(7L, "q1", scope); + Assertions.assertNull(ConnectorStatementScopes.getIfPresent(session, "x.table", "db", "t")); + Assertions.assertTrue(scope.cache.isEmpty()); + Object table = ConnectorStatementScopes.resolveInStatement(session, "x.table", "db", "t", Object::new); + Assertions.assertSame(table, ConnectorStatementScopes.getIfPresent(session, "x.table", "db", "t")); + Assertions.assertNull(ConnectorStatementScopes.getIfPresent( + new TestSession(8L, "q1", scope), "x.table", "db", "t")); + Assertions.assertNull(ConnectorStatementScopes.getIfPresent( + new TestSession(7L, "q2", scope), "x.table", "db", "t")); + Assertions.assertNull(ConnectorStatementScopes.getIfPresent(session, "x.table", "other", "t")); + Assertions.assertNull(ConnectorStatementScopes.getIfPresent(session, "other.table", "db", "t")); + Assertions.assertNull(ConnectorStatementScopes.getIfPresent(null, "x.table", "db", "t")); + } + @Test public void sameCoordinateResolvesOncePerStatement() { // Same (namespace, catalog, db, table, queryId) within one scope -> loader runs once, one shared instance. @@ -149,6 +166,12 @@ private static final class RecordingMemoScope implements ConnectorStatementScope private final ConcurrentHashMap cache = new ConcurrentHashMap<>(); private String lastKey; + @Override + @SuppressWarnings("unchecked") + public T getIfPresent(String key) { + return (T) cache.get(key); + } + @Override @SuppressWarnings("unchecked") public T computeIfAbsent(String key, Supplier loader) { diff --git a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-metadata-methods.txt b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-metadata-methods.txt index 8c48399c75f970..955bb070bde2bc 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-metadata-methods.txt +++ b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-metadata-methods.txt @@ -36,7 +36,9 @@ getColumnStatistics(org.apache.doris.connector.spi.ConnectorSession,org.apache.d getDatabase(org.apache.doris.connector.spi.ConnectorSession,java.lang.String) getMvccPartitionView(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle) getPartitionFreshnessMillis(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.lang.String) +getPartitionViewVersion(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle) getPartitionsFreshnessMillis(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.List) +getSqlCacheVersion(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle) getSyntheticScanPredicates(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot) getSysTableHandle(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.lang.String) getTableComment(org.apache.doris.connector.spi.ConnectorSession,java.lang.String,java.lang.String) @@ -52,6 +54,7 @@ listDatabaseNames(org.apache.doris.connector.spi.ConnectorSession) listFileSizes(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle) listPartitionNames(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle) listPartitions(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.Optional) +listRelationNames(org.apache.doris.connector.spi.ConnectorSession,java.lang.String) listSupportedSysTables(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle) listTableNames(org.apache.doris.connector.spi.ConnectorSession,java.lang.String) listViewNames(org.apache.doris.connector.spi.ConnectorSession,java.lang.String) @@ -69,6 +72,8 @@ resolveTimeTravel(org.apache.doris.connector.spi.ConnectorSession,org.apache.dor setTableProperties(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.Map) supportsCastPredicatePushdown(org.apache.doris.connector.spi.ConnectorSession) supportsColumnHandleSnapshotPin(org.apache.doris.connector.spi.ConnectorSession) +supportsNativePartitionPlanning(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle) +tableExists(org.apache.doris.connector.spi.ConnectorSession,java.lang.String,java.lang.String) truncateTable(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.List) usesStatementSnapshotForOptions(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.Map) validateRowLevelDmlMode(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,org.apache.doris.connector.spi.handle.WriteOperation) diff --git a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt index 81918e6daa1431..556404bb594a59 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt +++ b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt @@ -105,6 +105,14 @@ org.apache.doris.connector.spi.ConnectorSession#getTimeZone():java.lang.String org.apache.doris.connector.spi.ConnectorSession#getUser():java.lang.String org.apache.doris.connector.spi.ConnectorSession#isExternalScanTaskReuseEnabled():boolean org.apache.doris.connector.spi.ConnectorSession#setCurrentTransaction(org.apache.doris.connector.spi.handle.ConnectorTransaction):void +org.apache.doris.connector.spi.ConnectorStatementScope#closeAll():void +org.apache.doris.connector.spi.ConnectorStatementScope#computeIfAbsent(java.lang.String,java.util.function.Supplier):java.lang.Object +org.apache.doris.connector.spi.ConnectorStatementScope#computeIfAbsentForPlanning(java.lang.String,java.util.function.Supplier):java.lang.Object +org.apache.doris.connector.spi.ConnectorStatementScope#endPlanning():void +org.apache.doris.connector.spi.ConnectorStatementScope#getIfPresent(java.lang.String):java.lang.Object +org.apache.doris.connector.spi.ConnectorStatementScope#getOrCreateMetadata(java.lang.String,java.util.function.Supplier):org.apache.doris.connector.spi.ConnectorMetadata +org.apache.doris.connector.spi.ConnectorStatementScopes#getIfPresent(org.apache.doris.connector.spi.ConnectorSession,java.lang.String,java.lang.String,java.lang.String):java.lang.Object +org.apache.doris.connector.spi.ConnectorStatementScopes#resolveInStatement(org.apache.doris.connector.spi.ConnectorSession,java.lang.String,java.lang.String,java.lang.String,java.util.function.Supplier):java.lang.Object org.apache.doris.connector.spi.ConnectorTableSchema#equals(java.lang.Object):boolean org.apache.doris.connector.spi.ConnectorTableSchema#getColumns():java.util.List org.apache.doris.connector.spi.ConnectorTableSchema#getProperties():java.util.Map @@ -149,6 +157,8 @@ org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot$Builder#property(java. org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot$Builder#retainSchema(boolean):org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot$Builder org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot$Builder#schemaId(long):org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot$Builder org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot$Builder#snapshotId(long):org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot$Builder +org.apache.doris.connector.spi.mvcc.ConnectorSqlCacheVersion#getUpdateTimeMillis():long +org.apache.doris.connector.spi.mvcc.ConnectorSqlCacheVersion#getVersion():java.lang.String org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#adjustFileCompressType(org.apache.doris.thrift.TFileCompressType):org.apache.doris.thrift.TFileCompressType org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#appendExplainInfo(java.lang.StringBuilder,java.lang.String,java.util.Map):void org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#applyColumnDefaultsOnRead():boolean diff --git a/fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java b/fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java index f68307a3d2ae80..3790912a51a9a2 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java +++ b/fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java @@ -267,6 +267,15 @@ private void matchSql(SqlBlockRule rule, String originSql, String sqlHash) throw } } + /** + * Partition-aware rules require the existing engine partition view. Conservatively include enabled + * user-bound rules too, so rebinding an existing rule during planning cannot bypass its checks. + */ + public boolean hasPartitionMetadataRules() { + return nameToSqlBlockRuleMap.values().stream().anyMatch(rule -> rule.getEnable() + && (rule.getPartitionNum() > 0 || Boolean.TRUE.equals(rule.getRequirePartitionFilter()))); + } + /** * Check number whether legal by user. **/ @@ -312,6 +321,13 @@ void checkLimitations(SqlBlockRule rule, Long partitionNum, Long tabletNum, Long if (!rule.getEnable()) { return; } + if (partitionNum < 0 && (rule.getPartitionNum() > 0 + || Boolean.TRUE.equals(rule.getRequirePartitionFilter()))) { + // A partition-aware rule was created after the scan chose native planning. Do not treat an + // unknown partition count as zero; a retry plans with the newly installed rule's exact view. + throw new AnalysisException("Partition metadata rule changed during planning; retry the query: " + + rule.getName()); + } if (Boolean.TRUE.equals(rule.getRequirePartitionFilter()) && isPartitionedTable && !hasPartitionPredicate) { MetricRepo.COUNTER_HIT_SQL_BLOCK_RULE.increase(1L); throw new AnalysisException("sql hits sql block rule: " + rule.getName() + ", missing partition filter"); diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/cache/NereidsSqlCacheManager.java b/fe/fe-core/src/main/java/org/apache/doris/common/cache/NereidsSqlCacheManager.java index 4b3e9c34109e2d..5c9fb5fb33e995 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/common/cache/NereidsSqlCacheManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/common/cache/NereidsSqlCacheManager.java @@ -33,8 +33,11 @@ import org.apache.doris.common.Pair; import org.apache.doris.common.Status; import org.apache.doris.common.util.DebugUtil; +import org.apache.doris.connector.spi.mvcc.ConnectorSqlCacheVersion; import org.apache.doris.datasource.CatalogIf; import org.apache.doris.datasource.CatalogMgr; +import org.apache.doris.datasource.mvcc.MvccReadSelector; +import org.apache.doris.datasource.mvcc.MvccSnapshot; import org.apache.doris.mtmv.MTMVRelatedTableIf; import org.apache.doris.nereids.CascadesContext; import org.apache.doris.nereids.SqlCacheContext; @@ -42,6 +45,7 @@ import org.apache.doris.nereids.SqlCacheContext.FullColumnName; import org.apache.doris.nereids.SqlCacheContext.FullTableName; import org.apache.doris.nereids.SqlCacheContext.ScanTable; +import org.apache.doris.nereids.SqlCacheContext.TableReference; import org.apache.doris.nereids.SqlCacheContext.TableVersion; import org.apache.doris.nereids.StatementContext; import org.apache.doris.nereids.analyzer.UnboundVariable; @@ -67,6 +71,7 @@ import org.apache.doris.qe.StmtExecutor; import org.apache.doris.qe.cache.CacheAnalyzer; import org.apache.doris.qe.cache.SqlCache; +import org.apache.doris.qe.cache.SqlCacheVersionProvider; import org.apache.doris.rpc.RpcException; import com.github.benmanes.caffeine.cache.Cache; @@ -128,8 +133,8 @@ public void invalidateAboutTable(TableIf tableIf) { for (Entry kv : sqlCaches.asMap().entrySet()) { String key = kv.getKey(); SqlCacheContext context = kv.getValue(); - for (Entry nameToVersion : context.getUsedTables().entrySet()) { - FullTableName tableName = nameToVersion.getKey(); + for (Entry nameToVersion : context.getUsedTables().entrySet()) { + FullTableName tableName = nameToVersion.getKey().tableName; TableVersion tableVersion = nameToVersion.getValue(); if (tableVersion.id == tableIf.getId()) { invalidateKeys.add(key); @@ -343,7 +348,7 @@ private Optional tryParseSql( if (privilegeChanged(connectContext, env, sqlCacheContext)) { return invalidateCache(key); } - IsChanged tableIsChanged = tablesOrDataChanged(env, sqlCacheContext); + IsChanged tableIsChanged = tablesOrDataChanged(env, sqlCacheContext, connectContext.getStatementContext()); if (tableIsChanged.changed) { if (tableIsChanged.invalidCache) { return invalidateCache(key); @@ -429,7 +434,7 @@ private Optional tryParseSql( } } - private IsChanged tablesOrDataChanged(Env env, SqlCacheContext sqlCacheContext) { + private IsChanged tablesOrDataChanged(Env env, SqlCacheContext sqlCacheContext, StatementContext statement) { if (sqlCacheContext.hasUnsupportedTables()) { return IsChanged.CHANGED_AND_INVALIDATE_CACHE; } @@ -437,13 +442,13 @@ private IsChanged tablesOrDataChanged(Env env, SqlCacheContext sqlCacheContext) // the query maybe scan empty partition of the table, we should check these table version too, // but the table not exists in sqlCacheContext.getScanTables(), so we need check here. // check table type and version - for (Entry scanTable : sqlCacheContext.getUsedTables().entrySet()) { + for (Entry scanTable : sqlCacheContext.getUsedTables().entrySet()) { TableVersion tableVersion = scanTable.getValue(); if (tableVersion.type != TableType.OLAP && tableVersion.type != TableType.MATERIALIZED_VIEW && tableVersion.type != TableType.PLUGIN_EXTERNAL_TABLE) { return IsChanged.CHANGED_AND_INVALIDATE_CACHE; } - TableIf tableIf = findTableIf(env, scanTable.getKey()); + TableIf tableIf = findTableIf(env, scanTable.getKey().tableName); if (tableIf == null) { return IsChanged.CHANGED_AND_INVALIDATE_CACHE; } else if (tableIf.isTemporary()) { @@ -473,6 +478,21 @@ private IsChanged tablesOrDataChanged(Env env, SqlCacheContext sqlCacheContext) return IsChanged.CHANGED_AND_INVALIDATE_CACHE; } } + } else if (tableVersion.sqlCacheVersion != null) { + MvccReadSelector selector = scanTable.getKey().selector; + Optional pin = Optional.empty(); + if (!selector.isLatest()) { + // Reuse the analyzer's resolver: snapshot existence, timestamp semantics, ref kind, + // branch/tag movement and schema selection all remain owned by the existing MVCC path. + statement.loadSnapshots(tableIf, selector.getTableSnapshot(), selector.getScanParams()); + pin = statement.getSnapshot(tableIf, selector.getTableSnapshot(), selector.getScanParams()); + } + Optional current = ((SqlCacheVersionProvider) tableIf) + .getSqlCacheVersion(pin); + if (!current.isPresent() || !tableVersion.sqlCacheVersion.getVersion() + .equals(current.get().getVersion())) { + return IsChanged.CHANGED_AND_INVALIDATE_CACHE; + } } else if (tableIf instanceof MTMVRelatedTableIf) { // External MVCC table (flipped hive/iceberg/paimon/hudi): compare the stored data-version // token against the live one. OlapTable/MTMV are matched by the OlapTable arm above. @@ -578,7 +598,8 @@ private boolean dataMaskPoliciesChanged( */ private boolean tryLockTables(ConnectContext connectContext, Env env, SqlCacheContext sqlCacheContext) { StatementContext currentStatementContext = connectContext.getStatementContext(); - for (FullTableName fullTableName : sqlCacheContext.getUsedTables().keySet()) { + for (TableReference reference : sqlCacheContext.getUsedTables().keySet()) { + FullTableName fullTableName = reference.tableName; TableIf tableIf = findTableIf(env, fullTableName); if (tableIf == null) { return false; diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorStatementScopeImpl.java b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorStatementScopeImpl.java index 07de8217d5d15c..e097e08d55c6bc 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorStatementScopeImpl.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorStatementScopeImpl.java @@ -62,6 +62,12 @@ public T computeIfAbsent(String key, Supplier loader) { return (T) cache.computeIfAbsent(key, k -> loader.get()); } + @Override + @SuppressWarnings("unchecked") + public T getIfPresent(String key) { + return (T) cache.get(key); + } + @Override public T computeIfAbsentForPlanning(String key, Supplier loader) { T value = computeIfAbsent(key, loader); diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaCacheMgr.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaCacheMgr.java index f6f4045e570285..de53810f544536 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaCacheMgr.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaCacheMgr.java @@ -32,6 +32,7 @@ import org.apache.doris.datasource.metacache.MetaCacheEntryDef; import org.apache.doris.datasource.metacache.MetaCacheEntryInvalidation; import org.apache.doris.datasource.metacache.MetaCacheEntryStats; +import org.apache.doris.datasource.mvcc.PluginDrivenPartitionView; import org.apache.doris.fs.FileSystemCache; import com.github.benmanes.caffeine.cache.stats.CacheStats; @@ -50,6 +51,7 @@ import java.util.OptionalLong; import java.util.concurrent.ExecutorService; import java.util.function.Consumer; +import java.util.function.Function; import javax.annotation.Nullable; /** @@ -479,6 +481,15 @@ public Optional getSchemaCacheValue(ExternalTable table, Schem } } + /** The caller resolves connector metadata first; this cache only reuses its FE projection. */ + public PluginDrivenPartitionView getPartitionView(ExternalTable table, PluginDrivenPartitionView.Key key, + Function loader) { + long catalogId = table.getCatalog().getId(); + prepareCatalogByEngine(catalogId, ENGINE_DEFAULT); + return engine(ENGINE_DEFAULT).entry(catalogId, "partition_view", + PluginDrivenPartitionView.Key.class, PluginDrivenPartitionView.class).get(key, loader); + } + public FileSystemCache getFsCache() { return fsCache; } @@ -523,10 +534,11 @@ void replaceEngineCachesForTest(List caches) { *

Registered entries: *

    *
  • {@code schema}: schema-only cache keyed by {@link SchemaCacheKey}
    • + *
  • {@code partition_view}: FE partition items and their sorted pruning index
    • *
* - *

This class keeps compatibility for generic external engines and routes only schema - * loading/invalidation. No engine-specific metadata (partitions/files/snapshots) is cached. + *

Connector metadata is still loaded by the connector; only source-independent FE projections + * are stored here, with the same catalog/database/table invalidation as schemas. */ private static class DefaultExternalMetaCache extends ExternalCatalogMetaCache { DefaultExternalMetaCache(String engine, ExecutorService refreshExecutor) { @@ -540,6 +552,11 @@ private static class DefaultExternalMetaCache extends ExternalCatalogMetaCache { MetaCacheEntryInvalidation.forTableIdentity( key -> key.getNameMapping().getLocalDbName(), key -> key.getNameMapping().getLocalTblName()))); + registerEntry(MetaCacheEntryDef.contextualOnly( + "partition_view", PluginDrivenPartitionView.Key.class, PluginDrivenPartitionView.class, + defaultEntryCacheSpec(), MetaCacheEntryInvalidation.forTableIdentity( + PluginDrivenPartitionView.Key::getDatabase, PluginDrivenPartitionView.Key::getTable)) + .withSizeEstimator(PluginDrivenPartitionView::estimateSize)); } @Override diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java index 4c874fe372179d..1a107879cfff53 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java @@ -540,9 +540,10 @@ public boolean supportInternalPartitionPruned(Optional snapshot) { /** * Cross-query cache of the pre-built {@link SortedPartitionRanges} for binary-search partition - * pruning. Tables that implement {@link SupportBinarySearchFilteringPartitions} (external MVCC: - * iceberg/paimon) route through the shared {@link NereidsSortedPartitionsCacheManager}, keyed by the - * pinned connector snapshot; others return empty (the caller falls back to building ranges fresh). + * pruning. The default implementation delegates tables implementing + * {@link SupportBinarySearchFilteringPartitions} to {@link NereidsSortedPartitionsCacheManager}. + * Plugin MVCC tables override this to return the index owned by their pinned partition view; + * other tables return empty (the caller falls back to building ranges fresh). * * @param scan the catalog relation * @return sorted partition ranges, or empty if not supported diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/ExternalCatalogMetaCache.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/ExternalCatalogMetaCache.java index 4a639d8b20259f..e90837d2083678 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/ExternalCatalogMetaCache.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/ExternalCatalogMetaCache.java @@ -22,8 +22,6 @@ import org.apache.doris.connector.cache.CacheSpec; import org.apache.doris.connector.cache.MetaCache; import org.apache.doris.connector.cache.MetaCacheDefinition; -import org.apache.doris.connector.cache.MetaCacheSizeEstimator; -import org.apache.doris.connector.cache.MetaCacheSizeEstimators; import org.apache.doris.datasource.CacheException; import org.apache.doris.datasource.CatalogIf; import org.apache.doris.datasource.ExternalCatalog; @@ -31,7 +29,6 @@ import org.apache.doris.datasource.NameMapping; import org.apache.doris.datasource.SchemaCacheValue; -import com.google.common.annotations.VisibleForTesting; import com.google.common.collect.Maps; import java.time.Duration; @@ -292,7 +289,7 @@ private MetaCache newMetaCacheEntry( Function loader = wrapSchemaValidator(entryDef.getLoader(), entryDef.getValueType()); MetaCacheDefinition.Builder builder = MetaCacheDefinition.builder( entryDef.getName(), cacheSpec, entryDef.getInvalidation()::scope) - .sizeEstimator(sizeEstimator(entryDef.getValueType())); + .sizeEstimator(entryDef.getSizeEstimator()); if (loader != null) { builder.loader(loader); } @@ -303,14 +300,6 @@ private MetaCache newMetaCacheEntry( return group.owner().create(builder.build()); } - /** A whole-value walk of a wide table's schema would exhaust the visit budget, so it goes column by column. */ - @VisibleForTesting - static MetaCacheSizeEstimator sizeEstimator(Class valueType) { - return SchemaCacheValue.class.isAssignableFrom(valueType) - ? MetaCacheSizeEstimators.reflectivePerElement(value -> ((SchemaCacheValue) value).getColumnLists()) - : MetaCacheSizeEstimators.reflective(); - } - private Function wrapSchemaValidator(Function loader, Class valueType) { if (loader == null) { return null; diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/MetaCacheEntryDef.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/MetaCacheEntryDef.java index 6a081e28871a4c..3d2525f6242eea 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/MetaCacheEntryDef.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/MetaCacheEntryDef.java @@ -18,6 +18,9 @@ package org.apache.doris.datasource.metacache; import org.apache.doris.connector.cache.CacheSpec; +import org.apache.doris.connector.cache.MetaCacheSizeEstimator; +import org.apache.doris.connector.cache.MetaCacheSizeEstimators; +import org.apache.doris.datasource.SchemaCacheValue; import java.util.Objects; import java.util.function.Function; @@ -100,6 +103,7 @@ public final class MetaCacheEntryDef { * Keeping it on the definition makes each entry's default policy explicit. */ private final CacheSpec defaultCacheSpec; + private final MetaCacheSizeEstimator sizeEstimator; private final boolean autoRefresh; private final boolean contextualOnly; private final MetaCacheEntryInvalidation invalidation; @@ -107,6 +111,21 @@ public final class MetaCacheEntryDef { private MetaCacheEntryDef(String name, Class keyType, Class valueType, @Nullable Function loader, CacheSpec defaultCacheSpec, boolean autoRefresh, boolean contextualOnly, MetaCacheEntryInvalidation invalidation) { + this(name, keyType, valueType, loader, defaultCacheSpec, autoRefresh, contextualOnly, + invalidation, defaultSizeEstimator(valueType)); + } + + /** A whole-value walk of a wide schema exceeds the visit budget, so account for its columns individually. */ + static MetaCacheSizeEstimator defaultSizeEstimator(Class valueType) { + return SchemaCacheValue.class.isAssignableFrom(valueType) + ? MetaCacheSizeEstimators.reflectivePerElement(value -> ((SchemaCacheValue) value).getColumnLists()) + : MetaCacheSizeEstimators.reflective(); + } + + private MetaCacheEntryDef(String name, Class keyType, Class valueType, + @Nullable Function loader, CacheSpec defaultCacheSpec, boolean autoRefresh, boolean contextualOnly, + MetaCacheEntryInvalidation invalidation, MetaCacheSizeEstimator sizeEstimator) { + this.sizeEstimator = Objects.requireNonNull(sizeEstimator, "size estimator is required"); this.name = Objects.requireNonNull(name, "entry name is required"); this.keyType = Objects.requireNonNull(keyType, "entry key type is required"); this.valueType = Objects.requireNonNull(valueType, "entry value type is required"); @@ -184,6 +203,16 @@ public static MetaCacheEntryDef contextualOnly( invalidation); } + /** Preserve entry-specific accounting when creating the catalog cache runtime. */ + public MetaCacheEntryDef withSizeEstimator(MetaCacheSizeEstimator estimator) { + return new MetaCacheEntryDef<>(name, keyType, valueType, loader, defaultCacheSpec, autoRefresh, + contextualOnly, invalidation, estimator); + } + + public MetaCacheSizeEstimator getSizeEstimator() { + return sizeEstimator; + } + /** * @return logical entry name. */ diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/MvccReadSelector.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/MvccReadSelector.java new file mode 100644 index 00000000000000..02d356a5fe6dca --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/MvccReadSelector.java @@ -0,0 +1,116 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource.mvcc; + +import org.apache.doris.analysis.TableScanParams; +import org.apache.doris.analysis.TableSnapshot; + +import java.util.Map; +import java.util.Optional; +import java.util.TreeMap; + +/** + * Immutable read selector shared by statement snapshot lookup and SQL result-cache dependencies. + * Stores the original selector, never a prior execution's resolved dynamic parameters. Every resolution + * receives fresh copies, so rebinding or PREPARE/EXECUTE cannot mutate a cached dependency. + */ +public final class MvccReadSelector { + public static final MvccReadSelector LATEST = new MvccReadSelector(Optional.empty(), Optional.empty()); + + private final Optional tableSnapshot; + private final Optional scanParams; + private final String key; + + private MvccReadSelector(Optional tableSnapshot, Optional scanParams) { + this.tableSnapshot = tableSnapshot.map(TableSnapshot::new); + this.scanParams = scanParams.map(MvccReadSelector::copyScanParams); + this.key = versionKeyOf(this.tableSnapshot, this.scanParams); + } + + public static MvccReadSelector of(Optional tableSnapshot, Optional scanParams) { + return tableSnapshot.isEmpty() && scanParams.isEmpty() + ? LATEST : new MvccReadSelector(tableSnapshot, scanParams); + } + + public Optional getTableSnapshot() { + return tableSnapshot.map(TableSnapshot::new); + } + + public Optional getScanParams() { + return scanParams.map(MvccReadSelector::copyScanParams); + } + + public boolean isLatest() { + return key.isEmpty(); + } + + public String getKey() { + return key; + } + + private static TableScanParams copyScanParams(TableScanParams params) { + return new TableScanParams(params.getParamType(), params.getMapParams(), params.getListParams()); + } + + @Override + public boolean equals(Object other) { + return other instanceof MvccReadSelector && key.equals(((MvccReadSelector) other).key); + } + + @Override + public int hashCode() { + return key.hashCode(); + } + + private static String versionKeyOf(Optional tableSnapshot, Optional scanParams) { + // Concatenate both selectors (rather than returning on the first) so the key stays injective even + // if a reference ever carried BOTH a snapshot and scan params. Today the two are mutually exclusive + // (e.g. IcebergUtils.getQuerySpecSnapshot rejects FOR-TIME together with @branch/@tag), so in every + // reachable case exactly one part is non-empty and the key is "v:...", "p:...", or "". + StringBuilder key = new StringBuilder(); + if (tableSnapshot.isPresent()) { + TableSnapshot ts = tableSnapshot.get(); + key.append('v'); + appendVersionKeyPart(key, ts.getType()); + appendVersionKeyPart(key, ts.getValue()); + } + if (scanParams.isPresent()) { + TableScanParams sp = scanParams.get(); + key.append('p'); + appendVersionKeyPart(key, sp.getParamType()); + Map sortedParams = new TreeMap<>(sp.getMapParams()); + key.append('m').append(sortedParams.size()).append(':'); + for (Map.Entry entry : sortedParams.entrySet()) { + // Length prefixes preserve entry boundaries even when user values contain Map.toString() + // delimiters; sorting separately keeps semantically identical maps order-independent. + appendVersionKeyPart(key, entry.getKey()); + appendVersionKeyPart(key, entry.getValue()); + } + key.append('l').append(sp.getListParams().size()).append(':'); + for (String value : sp.getListParams()) { + appendVersionKeyPart(key, value); + } + } + return key.toString(); + } + + private static void appendVersionKeyPart(StringBuilder key, Object value) { + String text = String.valueOf(value); + key.append(text.length()).append(':').append(text); + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java index ddcfc805f81ac0..487461630fecee 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java @@ -21,6 +21,7 @@ import org.apache.doris.analysis.TableScanParams; import org.apache.doris.analysis.TableSnapshot; import org.apache.doris.catalog.Column; +import org.apache.doris.catalog.Env; import org.apache.doris.catalog.ListPartitionItem; import org.apache.doris.catalog.MTMV; import org.apache.doris.catalog.PartitionItem; @@ -39,6 +40,7 @@ import org.apache.doris.connector.spi.mvcc.ConnectorMvccPartition; import org.apache.doris.connector.spi.mvcc.ConnectorMvccPartitionView; import org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot; +import org.apache.doris.connector.spi.mvcc.ConnectorSqlCacheVersion; import org.apache.doris.connector.spi.mvcc.ConnectorTableFreshness; import org.apache.doris.connector.spi.mvcc.ConnectorTimeTravelSpec; import org.apache.doris.datasource.ExternalCatalog; @@ -56,8 +58,11 @@ import org.apache.doris.mtmv.MTMVSnapshotIdSnapshot; import org.apache.doris.mtmv.MTMVSnapshotIf; import org.apache.doris.mtmv.MTMVTimestampSnapshot; +import org.apache.doris.nereids.rules.expression.rules.SortedPartitionRanges; import org.apache.doris.nereids.trees.plans.algebra.CatalogRelation; import org.apache.doris.nereids.trees.plans.logical.LogicalFileScan; +import org.apache.doris.qe.ConnectContext; +import org.apache.doris.qe.cache.SqlCacheVersionProvider; import com.google.common.base.Preconditions; import com.google.common.collect.Lists; @@ -75,6 +80,8 @@ import java.util.Optional; import java.util.OptionalLong; import java.util.Set; +import java.util.function.Function; +import java.util.function.Supplier; import java.util.regex.Pattern; import java.util.stream.Collectors; @@ -93,7 +100,8 @@ * degrades MTMV to conservative over-refresh (the partition never matches its prior snapshot).

*/ public class PluginDrivenMvccExternalTable extends PluginDrivenExternalTable - implements MTMVRelatedTableIf, MTMVBaseTableIf, MvccTable, SupportBinarySearchFilteringPartitions { + implements MTMVRelatedTableIf, MTMVBaseTableIf, MvccTable, SupportBinarySearchFilteringPartitions, + SqlCacheVersionProvider { private static final Logger LOG = LogManager.getLogger(PluginDrivenMvccExternalTable.class); @@ -183,12 +191,38 @@ private PluginDrivenMvccSnapshot materializeLatest( // Eager pins bypass the schema cache loader, so preserve its mapped-column validation. pinnedSchema.validateSchema(); } - // This pin is not in StatementContext yet; ambient schema lookup can see a later generation. + if (canDeferPartitionView(metadata, session, pinnedHandle)) { + PluginDrivenSchemaCacheValue schema = pinnedSchema != null ? pinnedSchema + : (PluginDrivenSchemaCacheValue) getLatestSchemaCacheValue().orElseThrow( + () -> new IllegalStateException("Missing schema for pinned table " + getName())); + return new PluginDrivenMvccSnapshot(connectorSnapshot, schema, + () -> materializePartitionView(metadata, session, pinnedHandle, connectorSnapshot, schema, + schema.getPartitionColumns())); + } List partitionColumns = pinnedSchema == null ? getPartitionColumns() : pinnedSchema.getPartitionColumns(); - // Partition counts feed COUNT(*) pushdown and SQL block rules, so even the initial latest - // materialization must enumerate the same pinned generation that the data scan reads. + return materializePartitionView(metadata, session, pinnedHandle, connectorSnapshot, + pinnedSchema, partitionColumns); + } + + private boolean canDeferPartitionView(ConnectorMetadata metadata, ConnectorSession session, + ConnectorTableHandle handle) { + ConnectContext ctx = ConnectContext.get(); + // Candidates are collected before binding/pinning. Keep the existing MV path intact in this first + // step, including its partition pruning and compensation. Non-query MTMV callers stay eager too. + return ctx != null && ctx.getStatementContext() != null && !ctx.getState().isInternal() + && metadata.supportsNativePartitionPlanning(session, handle) + && ctx.getStatementContext().getCandidateMTMVs().isEmpty() + && !ctx.getEnv().getSqlBlockRuleMgr().hasPartitionMetadataRules(); + } + + private PluginDrivenMvccSnapshot materializePartitionView(ConnectorMetadata metadata, ConnectorSession session, + ConnectorTableHandle pinnedHandle, ConnectorMvccSnapshot connectorSnapshot, + PluginDrivenSchemaCacheValue pinnedSchema, List partitionColumns) { + // Partition-aware consumers must enumerate the same pinned generation that the data scan reads, + // whether this view is requested during binding or loaded later. ConnectorTableHandle partitionHandle = pinnedHandle; + Optional partitionViewVersion = metadata.getPartitionViewVersion(session, pinnedHandle); Optional viewOpt = metadata.getMvccPartitionView(session, pinnedHandle); if (viewOpt.isPresent()) { @@ -196,7 +230,8 @@ private PluginDrivenMvccSnapshot materializeLatest( if (connectorSnapshot.getSnapshotId() < 0) { // A negative query-begin pin is the connector's resolved-empty generation. Falling through to // a live LIST read would mix partitions committed after the data scan's empty boundary. - return buildFromRangeView(connectorSnapshot, view, pinnedSchema, partitionColumns); + return buildFromRangeView( + connectorSnapshot, view, pinnedSchema, partitionColumns, partitionViewVersion); } // A non-RANGE (UNPARTITIONED) view is the connector's "not RANGE / not MTMV-range-eligible" verdict // (iceberg's range view only covers single time-transform specs). If the table nonetheless declares @@ -209,22 +244,16 @@ private PluginDrivenMvccSnapshot materializeLatest( // unaffected. Freshness matches the legacy LIST path (timestamps / 0), harmless here because the // UNPARTITIONED verdict keeps this table out of the snapshot-id MTMV path. if (view.getStyle() != ConnectorMvccPartitionView.Style.RANGE && !partitionColumns.isEmpty()) { - Map listItems = Maps.newHashMap(); - Map listLastModifiedMillis = Maps.newHashMap(); - listPartitions(metadata, session, partitionHandle, partitionColumns, - listItems, listLastModifiedMillis); - return new PluginDrivenMvccSnapshot(connectorSnapshot, listItems, listLastModifiedMillis, - pinnedSchema, PartitionType.UNPARTITIONED, false, 0L); + PluginDrivenPartitionView viewItems = listPartitions(metadata, session, partitionHandle, + partitionColumns); + return new PluginDrivenMvccSnapshot(connectorSnapshot, viewItems, + pinnedSchema, PartitionType.UNPARTITIONED, false, 0L, 0L); } - return buildFromRangeView(connectorSnapshot, view, pinnedSchema, partitionColumns); + return buildFromRangeView(connectorSnapshot, view, pinnedSchema, partitionColumns, partitionViewVersion); } - Map nameToPartitionItem = Maps.newHashMap(); - Map nameToLastModifiedMillis = Maps.newHashMap(); - listPartitions(metadata, session, partitionHandle, partitionColumns, - nameToPartitionItem, nameToLastModifiedMillis); - return new PluginDrivenMvccSnapshot(connectorSnapshot, nameToPartitionItem, - nameToLastModifiedMillis, pinnedSchema); + PluginDrivenPartitionView viewItems = listPartitions(metadata, session, partitionHandle, partitionColumns); + return new PluginDrivenMvccSnapshot(connectorSnapshot, viewItems, pinnedSchema); } /** @@ -237,30 +266,49 @@ private PluginDrivenMvccSnapshot materializeLatest( * per-partition log-and-skip). */ private PluginDrivenMvccSnapshot buildFromRangeView(ConnectorMvccSnapshot connectorSnapshot, - ConnectorMvccPartitionView view, PluginDrivenSchemaCacheValue pinnedSchema, List partitionColumns) { + ConnectorMvccPartitionView view, PluginDrivenSchemaCacheValue pinnedSchema, List partitionColumns, + Optional version) { PartitionType partitionType = view.getStyle() == ConnectorMvccPartitionView.Style.RANGE ? PartitionType.RANGE : PartitionType.UNPARTITIONED; boolean snapshotIdFreshness = view.getFreshness() == ConnectorMvccPartitionView.Freshness.SNAPSHOT_ID; + List types = partitionColumns.stream().map(Column::getType).collect(Collectors.toList()); + List partitions = view.getStyle() == ConnectorMvccPartitionView.Style.RANGE + ? view.getPartitions() : Collections.emptyList(); + PluginDrivenPartitionView.Key key = version + .map(v -> PluginDrivenPartitionView.Key.versioned(getId(), getDbName(), getName(), v, types, true)) + .orElseGet(() -> PluginDrivenPartitionView.Key.range( + getId(), getDbName(), getName(), partitions, types)); + PluginDrivenPartitionView converted = cachedPartitionView(key, k -> { + List frozenTypes = k.frozenTypes(); + List columns = Lists.newArrayListWithCapacity(partitionColumns.size()); + for (int i = 0; i < partitionColumns.size(); i++) { + columns.add(new Column(partitionColumns.get(i).getName(), frozenTypes.get(i))); + } + return buildRangePartitionView(version.isPresent() ? partitions : k.rangePartitions(), columns); + }); + return new PluginDrivenMvccSnapshot(connectorSnapshot, converted, + pinnedSchema, partitionType, snapshotIdFreshness, view.getNewestUpdateMonotonicMarker(), + view.getNewestUpdateWallClockMillis()); + } + + private PluginDrivenPartitionView buildRangePartitionView(List partitions, + List partitionColumns) { Map nameToPartitionItem = Maps.newHashMap(); Map nameToFreshnessValue = Maps.newHashMap(); - if (view.getStyle() == ConnectorMvccPartitionView.Style.RANGE) { - for (ConnectorMvccPartition partition : view.getPartitions()) { - try { - nameToPartitionItem.put(partition.getName(), - toRangePartitionItem(partition, partitionColumns)); - } catch (AnalysisException e) { - // Fail loud: a range partition that cannot build is a real metadata error, not a - // skippable row (legacy loadPartitionInfo lets getPartitionRange throw). - throw new RuntimeException("Failed to build range partition item for " - + partition.getName() + ", partitionColumns: " + partitionColumns, e); - } - nameToFreshnessValue.put(partition.getName(), partition.getFreshnessValue()); + for (ConnectorMvccPartition partition : partitions) { + try { + nameToPartitionItem.put(partition.getName(), + toRangePartitionItem(partition, partitionColumns)); + } catch (AnalysisException e) { + // Fail loud: a range partition that cannot build is a real metadata error, not a + // skippable row (legacy loadPartitionInfo lets getPartitionRange throw). + throw new RuntimeException("Failed to build range partition item for " + + partition.getName() + ", partitionColumns: " + partitionColumns, e); } + nameToFreshnessValue.put(partition.getName(), partition.getFreshnessValue()); } - return new PluginDrivenMvccSnapshot(connectorSnapshot, nameToPartitionItem, nameToFreshnessValue, - pinnedSchema, partitionType, snapshotIdFreshness, view.getNewestUpdateMonotonicMarker(), - view.getNewestUpdateWallClockMillis()); + return new PluginDrivenPartitionView(nameToPartitionItem, nameToFreshnessValue); } /** @@ -296,11 +344,33 @@ private static List toPartitionValues(List bound) { * Both an opted-in latest pin and explicit time travel must use their retained schema here. * Unparseable partitions remain logged and skipped so the snapshot can fall back to scan-all. */ - private void listPartitions(ConnectorMetadata metadata, ConnectorSession session, - ConnectorTableHandle handle, List partitionColumns, - Map nameToPartitionItem, Map nameToLastModifiedMillis) { + private PluginDrivenPartitionView listPartitions(ConnectorMetadata metadata, ConnectorSession session, + ConnectorTableHandle handle, List partitionColumns) { List types = partitionColumns.stream().map(Column::getType).collect(Collectors.toList()); - List parts = metadata.listPartitions(session, handle, Optional.empty()); + Optional version = metadata.getPartitionViewVersion(session, handle); + if (version.isPresent()) { + PluginDrivenPartitionView.Key key = PluginDrivenPartitionView.Key.versioned( + getId(), getDbName(), getName(), version.get(), types, false); + return cachedPartitionView(key, k -> buildListPartitionView( + metadata.listPartitions(session, handle, Optional.empty()), k.frozenTypes())); + } + return partitionView(metadata.listPartitions(session, handle, Optional.empty()), types); + } + + PluginDrivenPartitionView partitionView(List parts, List types) { + PluginDrivenPartitionView.Key key = PluginDrivenPartitionView.Key.list( + getId(), getDbName(), getName(), parts, types); + return cachedPartitionView(key, k -> buildListPartitionView(k.listPartitions(), k.frozenTypes())); + } + + PluginDrivenPartitionView cachedPartitionView(PluginDrivenPartitionView.Key key, + Function loader) { + return Env.getCurrentEnv().getExtMetaCacheMgr().getPartitionView(this, key, loader); + } + + private PluginDrivenPartitionView buildListPartitionView(List parts, List types) { + Map nameToPartitionItem = Maps.newHashMapWithExpectedSize(parts.size()); + Map nameToLastModifiedMillis = Maps.newHashMapWithExpectedSize(parts.size()); for (ConnectorPartitionInfo part : parts) { String partitionName = part.getPartitionName(); nameToLastModifiedMillis.put(partitionName, part.getLastModifiedMillis()); @@ -327,7 +397,7 @@ private void listPartitions(ConnectorMetadata metadata, ConnectorSession session part.getOrderedPartitionValues(), part.getPartitionValueNullFlags())); } catch (Exception e) { LOG.warn("toListPartitionItem failed, partitionColumns: {}, partitionName: {}", - partitionColumns, partitionName, e); + types, partitionName, e); } } // One line for the listing, next to the per-partition ones above. @@ -344,8 +414,9 @@ private void listPartitions(ConnectorMetadata metadata, ConnectorSession session + "table is reported UNPARTITIONED for this read: {} listed, 0 built, " + "typed by {} partition column(s) {}. The connector's partition columns " + "and the values it lists do not agree.", - getDbName(), getName(), parts.size(), types.size(), partitionColumns); + getDbName(), getName(), parts.size(), types.size(), types); } + return new PluginDrivenPartitionView(nameToPartitionItem, nameToLastModifiedMillis); } private ConnectorMvccSnapshot emptySnapshot() { @@ -492,12 +563,8 @@ private MvccSnapshot loadSnapshotInternal( // pin (connectorSnapshot.getProperties()); the scan node's applySnapshot threads them onto the handle. // Partitions are listed on the BASE handle at latest — the full latest set, identical to the // normal-read materializeLatest path — NOT a snapshot-pinned handle. - Map nameToPartitionItem = Maps.newHashMap(); - Map nameToLastModifiedMillis = Maps.newHashMap(); - listPartitions(metadata, session, handle, getPartitionColumns(), - nameToPartitionItem, nameToLastModifiedMillis); - return new PluginDrivenMvccSnapshot(connectorSnapshot, nameToPartitionItem, - nameToLastModifiedMillis, null); + PluginDrivenPartitionView viewItems = listPartitions(metadata, session, handle, getPartitionColumns()); + return new PluginDrivenMvccSnapshot(connectorSnapshot, viewItems, null); } // Schema-at-snapshot: thread the pin onto the handle FIRST (applySnapshot), so a BRANCH pin @@ -525,13 +592,11 @@ private MvccSnapshot loadSnapshotInternal( // read the pin when listing (listsPartitionsAtSnapshot) gets the real set instead, listed on the // snapshot-applied handle, and pruning and partition=N/M both become truthful. if (metadata.listsPartitionsAtSnapshot(session, pinnedHandle)) { - Map pinnedPartitionItems = Maps.newHashMap(); - Map pinnedLastModifiedMillis = Maps.newHashMap(); // Typed by the PINNED schema, which is the one this snapshot publishes: see listPartitions. - listPartitions(metadata, session, pinnedHandle, pinnedSchema.getPartitionColumns(), - pinnedPartitionItems, pinnedLastModifiedMillis); - return new PluginDrivenMvccSnapshot(connectorSnapshot, pinnedPartitionItems, - pinnedLastModifiedMillis, pinnedSchema); + Supplier loader = () -> new PluginDrivenMvccSnapshot(connectorSnapshot, + listPartitions(metadata, session, pinnedHandle, pinnedSchema.getPartitionColumns()), pinnedSchema); + return canDeferPartitionView(metadata, session, pinnedHandle) + ? new PluginDrivenMvccSnapshot(connectorSnapshot, pinnedSchema, loader) : loader.get(); } return new PluginDrivenMvccSnapshot(connectorSnapshot, Collections.emptyMap(), Collections.emptyMap(), pinnedSchema); @@ -719,6 +784,15 @@ static boolean schemaCacheDisabled(Connector connector) { // ──────────────────── partition view (snapshot-aware) ──────────────────── + @Override + public LogicalFileScan.SelectedPartitions initSelectedPartitions(Optional snapshot) { + PluginDrivenMvccSnapshot pin = getOrMaterialize(snapshot); + if (pin.hasDeferredPartitions()) { + return LogicalFileScan.SelectedPartitions.NATIVE; + } + return super.initSelectedPartitions(Optional.of(pin)); + } + @Override public boolean supportInternalPartitionPruned(Optional snapshot) { // Non-range connector views enumerate raw transforms and historical specs for counts. @@ -752,6 +826,9 @@ public PartitionType getPartitionType(Optional snapshot) { @Override public List getPartitionColumns(Optional snapshot) { PluginDrivenMvccSnapshot pin = getOrMaterialize(snapshot); + if (pin.hasDeferredPartitions()) { + return super.getPartitionColumns(Optional.of(pin)); + } if (pin.getPartitionType() != null) { // Range-view path: do NOT empty the columns here (parity master getIcebergPartitionColumns, which // always returns the spec columns); UNPARTITIONED is enforced via getPartitionType above. @@ -806,11 +883,15 @@ public long getPartitionMetaLoadTimeMillis(CatalogRelation scan) { return 0L; } - // getSortedPartitionRanges is now a pure pass-through to ExternalTable's cache-manager delegation - // (5c17b748880's snapshotId==-1 short-circuit was removed: getPartitionMetaVersion above derives a - // content-comparable version from the frozen partition name set for ALL engines, so Cache B is - // correct uniformly -- snapshot-less hive and real-snapshot iceberg/paimon alike) -- no override - // needed here. + @Override + public Optional> getSortedPartitionRanges(CatalogRelation scan) { + ConnectContext context = ConnectContext.get(); + if (context != null && !context.getSessionVariable().enableBinarySearchFilteringPartitions) { + return Optional.empty(); + } + // The index belongs to the query's frozen view, not a separately refreshed table-wide cache. + return getOrMaterialize(pinnedSnapshot(scan)).getPartitionView().getSortedRanges(); + } private Optional pinnedSnapshot(CatalogRelation scan) { if (scan instanceof LogicalFileScan) { @@ -981,6 +1062,20 @@ private FreshnessProbe(ConnectorSession session, ConnectorMetadata metadata, } } + @Override + public Optional getSqlCacheVersion(Optional snapshot) { + Optional probe = resolveFreshnessProbe(); + if (!probe.isPresent()) { + return Optional.empty(); + } + FreshnessProbe p = probe.get(); + ConnectorMvccSnapshot pin = snapshot + .map(value -> ((PluginDrivenMvccSnapshot) value).getConnectorSnapshot()) + .orElseGet(() -> p.metadata.beginQuerySnapshot(p.session, p.handle).orElseGet(this::emptySnapshot)); + ConnectorTableHandle pinnedHandle = p.metadata.applySnapshot(p.session, p.handle, pin); + return p.metadata.getSqlCacheVersion(p.session, pinnedHandle); + } + @Override public long getNewestUpdateVersionOrTime() { // Dictionary-update path: always probe LATEST (bypass any context pin), mirroring legacy diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccSnapshot.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccSnapshot.java index 62922816519d3e..9b0871c86358d6 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccSnapshot.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccSnapshot.java @@ -22,18 +22,20 @@ import org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot; import org.apache.doris.datasource.SchemaCacheValue; +import com.google.common.base.Suppliers; + import java.util.Collections; -import java.util.HashMap; import java.util.Map; +import java.util.function.Supplier; /** * Generic MVCC snapshot for plugin-driven (connector SPI) tables. * *

Pins a single point-in-time view of an MVCC-capable connector table for the whole duration of * a query: the scalar {@link ConnectorMvccSnapshot} (the snapshot-id pin used for reads) plus the - * materialized partition view listed at that moment. Holding the materialized view here means every - * MTMV/MvccTable accessor that receives this snapshot reads the SAME partition set with NO extra - * connector round-trip (single-pin invariant).

+ * partition view for that same generation. Native file planning can defer partition enumeration; the + * first partition-aware accessor loads it once using the already pinned handle and schema. All later + * accessors share that immutable view, without resolving latest again (single-pin invariant).

* *

Source-agnostic: it carries already-rendered partition names/items and per-partition staleness * timestamps, so no data-source-specific logic lives in fe-core. Parity with the legacy @@ -54,9 +56,10 @@ public class PluginDrivenMvccSnapshot implements MvccSnapshot { private final ConnectorMvccSnapshot connectorSnapshot; - private final Map nameToPartitionItem; - private final Map nameToLastModifiedMillis; + private final PluginDrivenPartitionView partitionView; private final SchemaCacheValue pinnedSchema; + // Memoized per pin, never a new latest read. Released after loading along with its captured loader. + private final Supplier deferredPartitions; // Range-view path (connector-supplied); null/false/0 on the legacy path so its behavior is byte-unchanged. private final PartitionType partitionType; // null => legacy LIST/UNPARTITIONED computed from isPartitionInvalid private final boolean snapshotIdFreshness; // true => getPartitionSnapshot wraps a snapshot id, else a timestamp @@ -129,13 +132,23 @@ public PluginDrivenMvccSnapshot(ConnectorMvccSnapshot connectorSnapshot, boolean snapshotIdFreshness, long newestUpdateMonotonicMarker, long newestUpdateWallClockMillis) { + this(connectorSnapshot, new PluginDrivenPartitionView( + nameToPartitionItem == null ? Collections.emptyMap() : nameToPartitionItem, + nameToFreshnessValue == null ? Collections.emptyMap() : nameToFreshnessValue), pinnedSchema, + partitionType, snapshotIdFreshness, newestUpdateMonotonicMarker, newestUpdateWallClockMillis); + } + + PluginDrivenMvccSnapshot(ConnectorMvccSnapshot connectorSnapshot, PluginDrivenPartitionView partitionView, + SchemaCacheValue pinnedSchema) { + this(connectorSnapshot, partitionView, pinnedSchema, null, false, 0L, 0L); + } + + PluginDrivenMvccSnapshot(ConnectorMvccSnapshot connectorSnapshot, PluginDrivenPartitionView partitionView, + SchemaCacheValue pinnedSchema, PartitionType partitionType, boolean snapshotIdFreshness, + long newestUpdateMonotonicMarker, long newestUpdateWallClockMillis) { this.connectorSnapshot = connectorSnapshot; - this.nameToPartitionItem = nameToPartitionItem == null - ? Collections.emptyMap() - : Collections.unmodifiableMap(new HashMap<>(nameToPartitionItem)); - this.nameToLastModifiedMillis = nameToFreshnessValue == null - ? Collections.emptyMap() - : Collections.unmodifiableMap(new HashMap<>(nameToFreshnessValue)); + this.partitionView = partitionView; + this.deferredPartitions = null; this.pinnedSchema = pinnedSchema; this.partitionType = partitionType; this.snapshotIdFreshness = snapshotIdFreshness; @@ -143,6 +156,30 @@ public PluginDrivenMvccSnapshot(ConnectorMvccSnapshot connectorSnapshot, this.newestUpdateWallClockMillis = newestUpdateWallClockMillis; } + PluginDrivenMvccSnapshot(ConnectorMvccSnapshot connectorSnapshot, SchemaCacheValue pinnedSchema, + Supplier partitionLoader) { + this.connectorSnapshot = connectorSnapshot; + this.pinnedSchema = pinnedSchema; + this.partitionView = null; + this.partitionType = null; + this.snapshotIdFreshness = false; + this.newestUpdateMonotonicMarker = 0L; + this.newestUpdateWallClockMillis = 0L; + this.deferredPartitions = Suppliers.memoize(partitionLoader::get); + } + + public boolean hasDeferredPartitions() { + return deferredPartitions != null; + } + + private PluginDrivenMvccSnapshot partitionSnapshot() { + return hasDeferredPartitions() ? deferredPartitions.get() : this; + } + + PluginDrivenPartitionView getPartitionView() { + return partitionSnapshot().partitionView; + } + public ConnectorMvccSnapshot getConnectorSnapshot() { return connectorSnapshot; } @@ -162,11 +199,11 @@ public long getSchemaId() { } public Map getNameToPartitionItem() { - return nameToPartitionItem; + return getPartitionView().getItems(); } public Map getNameToLastModifiedMillis() { - return nameToLastModifiedMillis; + return getPartitionView().getFreshness(); } /** @@ -174,7 +211,7 @@ public Map getNameToLastModifiedMillis() { * on the legacy path (then the caller computes LIST/UNPARTITIONED from {@link #isPartitionInvalid()}). */ public PartitionType getPartitionType() { - return partitionType; + return partitionSnapshot().partitionType; } /** @@ -183,7 +220,7 @@ public PartitionType getPartitionType() { * {@code MTMVTimestampSnapshot} in {@code getPartitionSnapshot}. */ public boolean isSnapshotIdFreshness() { - return snapshotIdFreshness; + return partitionSnapshot().snapshotIdFreshness; } /** @@ -192,7 +229,7 @@ public boolean isSnapshotIdFreshness() { * the dictionary auto-refresh probe compares. Only meaningful when {@link #getPartitionType()} is non-null. */ public long getNewestUpdateMonotonicMarker() { - return newestUpdateMonotonicMarker; + return partitionSnapshot().newestUpdateMonotonicMarker; } /** @@ -201,7 +238,7 @@ public long getNewestUpdateMonotonicMarker() { * the range-view path. */ public long getNewestUpdateWallClockMillis() { - return newestUpdateWallClockMillis; + return partitionSnapshot().newestUpdateWallClockMillis; } /** @@ -213,6 +250,6 @@ public long getNewestUpdateWallClockMillis() { * flagged invalid. Parity with legacy {@code PaimonPartitionInfo.isPartitionInvalid}. */ public boolean isPartitionInvalid() { - return nameToLastModifiedMillis.size() != nameToPartitionItem.size(); + return getPartitionView().getFreshness().size() != getPartitionView().getItems().size(); } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenPartitionView.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenPartitionView.java new file mode 100644 index 00000000000000..f94b9805fb310e --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenPartitionView.java @@ -0,0 +1,309 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource.mvcc; + +import org.apache.doris.catalog.ListPartitionItem; +import org.apache.doris.catalog.PartitionItem; +import org.apache.doris.catalog.PartitionKey; +import org.apache.doris.catalog.PrimitiveType; +import org.apache.doris.catalog.RangePartitionItem; +import org.apache.doris.catalog.ScalarType; +import org.apache.doris.catalog.Type; +import org.apache.doris.connector.cache.JvmSizeUtils; +import org.apache.doris.connector.cache.MetaCacheSizeEstimate; +import org.apache.doris.connector.cache.ReflectiveObjectSizeEstimator; +import org.apache.doris.connector.spi.ConnectorPartitionInfo; +import org.apache.doris.connector.spi.mvcc.ConnectorMvccPartition; +import org.apache.doris.nereids.rules.expression.rules.SortedPartitionRanges; +import org.apache.doris.qe.ConnectContext; +import org.apache.doris.thrift.TScalarType; +import org.apache.doris.thrift.TTypeDesc; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; + +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Optional; + +/** One published partition generation: maps and pruning index share the same read-only partition items. */ +public final class PluginDrivenPartitionView { + private final ImmutableMap items; + private final ImmutableMap freshness; + private final SortedPartitionRanges sortedRanges; + + PluginDrivenPartitionView(Map items, Map freshness) { + this.items = ImmutableMap.copyOf(items); + this.freshness = ImmutableMap.copyOf(freshness); + // Build before cache admission, like HivePartitionValues: the cache accounts for the index + // together with its maps, and readers never observe a partially initialized generation. + this.sortedRanges = hasComparableKeys(this.items) ? SortedPartitionRanges.build(this.items) : null; + } + + // Iceberg display partitions can come from different specs. They remain useful for counts and + // display, but different tuple shapes/types do not form one ordered pruning index. + private static boolean hasComparableKeys(Map items) { + List types = null; + for (PartitionItem item : items.values()) { + if (item instanceof ListPartitionItem) { + for (PartitionKey key : ((ListPartitionItem) item).getItems()) { + if (types == null) { + types = key.getTypes(); + } else if (!types.equals(key.getTypes())) { + return false; + } + } + } + } + return true; + } + + /** Admission-only traversal, bounded per partition rather than per complete table. */ + public static MetaCacheSizeEstimate estimateSize(Key key, PluginDrivenPartitionView view) { + long bytes = JvmSizeUtils.instanceSize(Key.class); + bytes = JvmSizeUtils.saturatedAdd(bytes, ReflectiveObjectSizeEstimator.estimateComplete( + java.util.Arrays.asList(key.database, key.table, key.types, key.timeZone, key.version))); + bytes = JvmSizeUtils.saturatedAdd(bytes, JvmSizeUtils.objectArraySize(key.partitions.size())); + for (Object partition : key.partitions) { + bytes = JvmSizeUtils.saturatedAdd(bytes, ReflectiveObjectSizeEstimator.estimateComplete(partition)); + } + bytes = JvmSizeUtils.saturatedAdd(bytes, JvmSizeUtils.instanceSize(PluginDrivenPartitionView.class)); + // ImmutableMap tables, entries, and freshness values; shared names/items are counted once below. + bytes = JvmSizeUtils.saturatedAdd(bytes, JvmSizeUtils.saturatedMultiply( + view.items.size() + (long) view.freshness.size(), 64L)); + for (Map.Entry entry : view.items.entrySet()) { + bytes = JvmSizeUtils.saturatedAdd(bytes, JvmSizeUtils.stringSize(entry.getKey())); + bytes = JvmSizeUtils.saturatedAdd(bytes, estimatePartitionItem(entry.getValue())); + } + for (String name : view.freshness.keySet()) { + bytes = JvmSizeUtils.saturatedAdd(bytes, JvmSizeUtils.stringSize(name)); + } + if (view.sortedRanges != null) { + bytes = JvmSizeUtils.saturatedAdd(bytes, + JvmSizeUtils.instanceSize(SortedPartitionRanges.class)); + bytes = JvmSizeUtils.saturatedAdd(bytes, + JvmSizeUtils.objectArraySize(view.sortedRanges.sortedPartitions.size())); + bytes = JvmSizeUtils.saturatedAdd(bytes, + JvmSizeUtils.objectArraySize(view.sortedRanges.defaultPartitions.size())); + // Index ranges keep converted bounds/literals in addition to the shared PartitionItem. + for (SortedPartitionRanges.PartitionItemAndRange range : view.sortedRanges.sortedPartitions) { + bytes = JvmSizeUtils.saturatedAdd(bytes, estimatePartitionItem(range.partitionItem)); + bytes = JvmSizeUtils.saturatedAdd(bytes, 128L); + } + bytes = JvmSizeUtils.saturatedAdd(bytes, + JvmSizeUtils.saturatedMultiply(view.sortedRanges.defaultPartitions.size(), 32L)); + } + return MetaCacheSizeEstimate.complete(bytes); + } + + private static long estimatePartitionItem(PartitionItem item) { + long bytes = JvmSizeUtils.instanceSize(item.getClass()); + if (item instanceof ListPartitionItem) { + List keys = ((ListPartitionItem) item).getItems(); + bytes = JvmSizeUtils.saturatedAdd(bytes, JvmSizeUtils.arrayListSize(keys.size())); + for (PartitionKey key : keys) { + bytes = JvmSizeUtils.saturatedAdd(bytes, estimatePartitionKey(key)); + } + } else { + com.google.common.collect.Range range = ((RangePartitionItem) item).getItems(); + // Range and its two cut objects. + bytes = JvmSizeUtils.saturatedAdd(bytes, 96L); + if (range.hasLowerBound()) { + bytes = JvmSizeUtils.saturatedAdd(bytes, estimatePartitionKey(range.lowerEndpoint())); + } + if (range.hasUpperBound()) { + bytes = JvmSizeUtils.saturatedAdd(bytes, estimatePartitionKey(range.upperEndpoint())); + } + } + return bytes; + } + + private static long estimatePartitionKey(PartitionKey key) { + long bytes = JvmSizeUtils.instanceSize(PartitionKey.class); + // Value, primitive-type and original-value lists. Overcount shared scalar types/strings instead + // of traversing Expr's memoized supplier (which points back at the expression). + bytes = JvmSizeUtils.saturatedAdd(bytes, + JvmSizeUtils.saturatedMultiply(JvmSizeUtils.arrayListSize(key.getKeys().size()), 3L)); + for (org.apache.doris.analysis.LiteralExpr value : key.getKeys()) { + bytes = JvmSizeUtils.saturatedAdd(bytes, JvmSizeUtils.instanceSize(value.getClass())); + bytes = JvmSizeUtils.saturatedAdd(bytes, + ReflectiveObjectSizeEstimator.estimateComplete(value.getType())); + // Empty child list, memoizing supplier and its bound supplier, and numeric/date payload. + bytes = JvmSizeUtils.saturatedAdd(bytes, 128L); + bytes = JvmSizeUtils.saturatedAdd(bytes, JvmSizeUtils.stringSize(value.getStringValue())); + } + return bytes; + } + + ImmutableMap getItems() { + return items; + } + + ImmutableMap getFreshness() { + return freshness; + } + + Optional> getSortedRanges() { + return Optional.ofNullable(sortedRanges); + } + + /** + * Uses a connector-certified immutable generation when available, without retaining or visiting + * partition records. Otherwise matches the actual connector listing rather than an independently + * refreshed snapshot token. + * LIST records' conversion fields are immutable; compare only those fields, not unrelated + * property maps or statistics. This also reuses equal records from uncached connectors. + * RANGE bounds are defensively copied because their SPI wrappers can + * retain mutable caller lists. Schema types and session timezone also affect literal conversion. + */ + public static final class Key { + private final long tableId; + private final String database; + private final String table; + private final List partitions; + private final boolean range; + private final List types; + private final String timeZone; + private final int hash; + private final String version; + + private Key(long tableId, String database, String table, List partitions, + List types, boolean range) { + this(tableId, database, table, partitions, types, range, null); + } + + private Key(long tableId, String database, String table, List partitions, + List types, boolean range, String version) { + this.version = version; + this.tableId = tableId; + this.database = database; + this.table = table; + this.partitions = ImmutableList.copyOf(partitions); + this.range = range; + this.types = types.stream().map(Type::toThrift).collect(ImmutableList.toImmutableList()); + ConnectContext context = ConnectContext.get(); + // Same fallback as TimestampTzLiteral when parsing an unqualified value. + this.timeZone = context == null ? "UTC" : context.getSessionVariable().getTimeZone(); + int recordsHash = 1; + for (Object partition : this.partitions) { + recordsHash = 31 * recordsHash + (range ? partition.hashCode() + : listRecordHash((ConnectorPartitionInfo) partition)); + } + this.hash = Objects.hash(tableId, database, table, range, this.types, timeZone, version, recordsHash); + } + + static Key versioned(long tableId, String database, String table, + String version, List types, boolean range) { + return new Key(tableId, database, table, ImmutableList.of(), types, range, + Objects.requireNonNull(version, "version")); + } + + static Key list(long tableId, String database, String table, + List partitions, List types) { + return new Key(tableId, database, table, partitions, types, false); + } + + static Key range(long tableId, String database, String table, + List partitions, List types) { + List frozen = partitions.stream().map(partition -> new ConnectorMvccPartition( + partition.getName(), ImmutableList.copyOf(partition.getLowerBound()), + ImmutableList.copyOf(partition.getUpperBound()), partition.getFreshnessValue())) + .collect(ImmutableList.toImmutableList()); + return new Key(tableId, database, table, frozen, types, true); + } + + public String getDatabase() { + return database; + } + + public String getTable() { + return table; + } + + @SuppressWarnings("unchecked") + List listPartitions() { + return (List) partitions; + } + + @SuppressWarnings("unchecked") + List rangePartitions() { + return (List) partitions; + } + + List frozenTypes() { + return types.stream().map(type -> { + // Type.fromThrift does not retain TIMESTAMPTZ scale or VARBINARY length. + // Preserve all scalar parameters when freezing the conversion's schema. + if (type.getTypes().get(0).isSetScalarType()) { + TScalarType scalar = type.getTypes().get(0).getScalarType(); + return ScalarType.createType(PrimitiveType.fromThrift(scalar.getType()), + scalar.getLen(), scalar.getPrecision(), scalar.getScale()); + } + return Type.fromThrift(type); + }).collect(ImmutableList.toImmutableList()); + } + + private static int listRecordHash(ConnectorPartitionInfo partition) { + int result = partition.getPartitionName().hashCode(); + result = 31 * result + partition.getOrderedPartitionValues().hashCode(); + result = 31 * result + partition.getPartitionValueNullFlags().hashCode(); + result = 31 * result + partition.getPartitionValueTypes().hashCode(); + return 31 * result + Long.hashCode(partition.getLastModifiedMillis()); + } + + private static boolean sameListRecord(ConnectorPartitionInfo left, ConnectorPartitionInfo right) { + return left == right || (left.getLastModifiedMillis() == right.getLastModifiedMillis() + && left.getPartitionName().equals(right.getPartitionName()) + && left.getOrderedPartitionValues().equals(right.getOrderedPartitionValues()) + && left.getPartitionValueNullFlags().equals(right.getPartitionValueNullFlags()) + && left.getPartitionValueTypes().equals(right.getPartitionValueTypes())); + } + + @Override + public int hashCode() { + return hash; + } + + @Override + public boolean equals(Object other) { + if (this == other) { + return true; + } + if (!(other instanceof Key)) { + return false; + } + Key that = (Key) other; + if (hash != that.hash || tableId != that.tableId || range != that.range + || !Objects.equals(version, that.version) + || !Objects.equals(database, that.database) || !Objects.equals(table, that.table) + || !types.equals(that.types) || !timeZone.equals(that.timeZone) + || partitions.size() != that.partitions.size()) { + return false; + } + for (int i = 0; i < partitions.size(); i++) { + if (range ? !partitions.get(i).equals(that.partitions.get(i)) + : !sameListRecord((ConnectorPartitionInfo) partitions.get(i), + (ConnectorPartitionInfo) that.partitions.get(i))) { + return false; + } + } + return true; + } + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java index 60c2a5422e4334..147709566fca2b 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java @@ -88,7 +88,6 @@ import org.apache.logging.log4j.Logger; import java.io.IOException; -import java.util.ArrayList; import java.util.Arrays; import java.util.HashMap; import java.util.List; @@ -494,23 +493,10 @@ protected List listTableNamesFromRemote(SessionContext ctx, String dbNam ConnectorSession session = buildCrossStatementSession(); try { ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); - List tableNames = metadata.listTableNames(session, dbName); - // Deliberately the raw field, NOT hasConnectorCapability(): this already runs inside an initialized - // catalog, so re-entering makeSureInitialized() here would be pointless work on a listing path. - if (!connector.getCapabilities().contains(ConnectorCapability.SUPPORTS_VIEW)) { - return tableNames; + if (connector.getCapabilities().contains(ConnectorCapability.SUPPORTS_VIEW)) { + return metadata.listRelationNames(session, dbName); } - // Mirror legacy IcebergExternalCatalog.listTableNamesFromRemote: for a view-exposing connector - // (iceberg) SHOW TABLES includes both tables AND views, because the connector's listTableNames - // subtracts the view names. Re-merge the connector's view names here (the two sets are disjoint - // by construction, so a plain addAll cannot introduce duplicates). - List viewNames = metadata.listViewNames(session, dbName); - if (viewNames.isEmpty()) { - return tableNames; - } - List merged = new ArrayList<>(tableNames); - merged.addAll(viewNames); - return merged; + return metadata.listTableNames(session, dbName); } finally { session.getStatementScope().closeAll(); } @@ -520,7 +506,7 @@ protected List listTableNamesFromRemote(SessionContext ctx, String dbNam public boolean tableExist(SessionContext ctx, String dbName, String tblName) { ConnectorSession session = buildConnectorSession(); return PluginDrivenMetadata.get(session, connector) - .getTableHandle(session, dbName, tblName).isPresent(); + .tableExists(session, dbName, tblName); } @Override @@ -664,8 +650,8 @@ public boolean createTable(CreateTableInfo createTableInfo) throws UserException ConnectorSession session = buildConnectorSession(); ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); // Both views matter: name folding can expose a local conflict that the remote lookup does not see. - boolean remoteExists = metadata.getTableHandle(session, db.getRemoteName(), - createTableInfo.getTableName()).isPresent(); + boolean remoteExists = metadata.tableExists(session, db.getRemoteName(), + createTableInfo.getTableName()); boolean localExists = db.getTableNullable(localTableName) != null; if (remoteExists || localExists) { if (createTableInfo.isIfNotExists()) { @@ -690,8 +676,8 @@ public boolean createTable(CreateTableInfo createTableInfo) throws UserException // The probe and create are not atomic. A successful re-probe turns a lost IF NOT EXISTS race into // the same CTAS-short-circuit result as an initially existing table. if (createTableInfo.isIfNotExists() - && metadata.getTableHandle(session, db.getRemoteName(), - createTableInfo.getTableName()).isPresent()) { + && metadata.tableExists(session, db.getRemoteName(), + createTableInfo.getTableName())) { LOG.info("create table[{}.{}.{}] lost the race to a concurrent creator; " + "treating as an IF NOT EXISTS no-op", getName(), createTableInfo.getDbName(), createTableInfo.getTableName()); @@ -857,8 +843,12 @@ public void dropTable(String dbName, String tableName, boolean isView, boolean i LOG.info("finished to drop view {}.{}.{}", getName(), dbName, tableName); return; } - Optional handle = metadata.getTableHandle( - session, dorisTable.getRemoteDbName(), dorisTable.getRemoteName()); + // A query handle may refer to cached read metadata after an out-of-band drop. DDL must + // first check current existence so IF EXISTS still performs local cleanup in that case. + Optional handle = metadata.tableExists( + session, dorisTable.getRemoteDbName(), dorisTable.getRemoteName()) + ? metadata.getTableHandle(session, dorisTable.getRemoteDbName(), dorisTable.getRemoteName()) + : Optional.empty(); // The table is present in the FE cache but may have been dropped out-of-band on the remote // side; preserve the existing IF EXISTS handling for that case. if (!handle.isPresent()) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java index e569392b6b5432..1808f6d0f4464f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java @@ -395,6 +395,13 @@ private static final class ResolvedScanProvider { */ public void setSelectedPartitions(SelectedPartitions selectedPartitions) { this.selectedPartitions = selectedPartitions; + // Streaming starts before getSplits(); publish both exact and unknown counts at this common + // injection boundary, including the eager fallback used by partition-aware SQL block rules. + long[] partitionCounts = displayPartitionCounts(selectedPartitions); + if (partitionCounts != null) { + this.selectedPartitionNum = partitionCounts[0]; + this.totalPartitionNum = partitionCounts[1]; + } } /** @@ -489,6 +496,9 @@ static long[] displayPartitionCounts(SelectedPartitions selectedPartitions) { if (selectedPartitions == null || selectedPartitions == SelectedPartitions.NOT_PRUNED) { return null; } + if (selectedPartitions.isNativePlanning()) { + return new long[] {-1, -1}; + } return new long[] { selectedPartitions.selectedPartitions.size(), selectedPartitions.totalPartitionNum}; } @@ -687,8 +697,12 @@ public String getNodeExplainString(String prefix, TExplainLevel detailLevel) { // parent's body wholesale (custom TABLE/QUERY/PREDICATES format), so it must re-emit the // line itself; the counts are populated from the Nereids pruning result in // getSplits()/startSplit() (see setSelectedPartitions). - output.append(prefix).append("partition=").append(selectedPartitionNum) - .append("/").append(totalPartitionNum).append("\n"); + if (totalPartitionNum < 0) { + output.append(prefix).append("partition=unknown (native planning)\n"); + } else { + output.append(prefix).append("partition=").append(selectedPartitionNum) + .append("/").append(totalPartitionNum).append("\n"); + } // FIX-E / FIX-R3-RESIDUAL (explain gap): the VERBOSE per-backend block (the backends: list, // per-file "path start/length" lines, and dataFileNum/deleteFileNum/deleteSplitNum) lives in // the parent FileScanNode but this override does not call super, so re-emit it under the SAME @@ -1991,6 +2005,11 @@ private boolean computeBatchMode() { long estimate = onPluginClassLoader(scanProvider, () -> scanProvider.streamingSplitEstimate( connectorSession, currentHandle, buildRemainingFilter(), countPushdown)); if (estimate >= 0) { + // Streaming does not know the final scanned-partition count before dispatch. Partition + // rules need that count now; use synchronous planning, which publishes the connector count. + if (org.apache.doris.catalog.Env.getCurrentEnv().getSqlBlockRuleMgr().hasPartitionMetadataRules()) { + return false; + } streamingSplitEstimate = estimate; streamingBatch = true; return true; diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java index 38239a084b68b2..1814b7e4c44617 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java @@ -156,8 +156,7 @@ private void markIvmBaselineRebuild(BaseTableInfo baseTableInfo, boolean allPart public Set getAvailableMTMVs(Set candidateMTMVs, ConnectContext ctx, boolean forceConsistent, BiPredicate predicate) { Set res = Sets.newLinkedHashSet(); - Map, Set> queryUsedPartitions = PartitionCompensator.getQueryUsedPartitions( - ctx.getStatementContext(), new BitSet()); + Map, Set> queryUsedPartitions = null; for (MTMV mtmv : candidateMTMVs) { if (predicate.test(ctx, mtmv)) { continue; @@ -165,6 +164,12 @@ public Set getAvailableMTMVs(Set candidateMTMVs, ConnectContext ctx, if (!mtmv.isUseForRewrite()) { continue; } + // Collect potentially large partition sets only when a candidate needs validation, once + // for all such candidates. Empty or filtered candidate sets need no partition union. + if (queryUsedPartitions == null) { + queryUsedPartitions = PartitionCompensator.getQueryUsedPartitions( + ctx.getStatementContext(), new BitSet()); + } if (isMVPartitionValid(mtmv, ctx, forceConsistent, queryUsedPartitions)) { res.add(mtmv); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/SqlCacheContext.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/SqlCacheContext.java index df7a2b101224f9..7675aafdf5e0a0 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/SqlCacheContext.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/SqlCacheContext.java @@ -27,7 +27,10 @@ import org.apache.doris.catalog.TableIf; import org.apache.doris.catalog.TableIf.TableType; import org.apache.doris.common.Pair; +import org.apache.doris.connector.spi.mvcc.ConnectorSqlCacheVersion; import org.apache.doris.datasource.CatalogIf; +import org.apache.doris.datasource.mvcc.MvccReadSelector; +import org.apache.doris.datasource.mvcc.MvccSnapshot; import org.apache.doris.mtmv.MTMVRelatedTableIf; import org.apache.doris.mysql.FieldInfo; import org.apache.doris.mysql.privilege.Auth; @@ -40,6 +43,7 @@ import org.apache.doris.qe.ResultSet; import org.apache.doris.qe.SessionVariable; import org.apache.doris.qe.cache.CacheProxy; +import org.apache.doris.qe.cache.SqlCacheVersionProvider; import org.apache.doris.thrift.TUniqueId; import com.google.common.collect.ImmutableList; @@ -74,7 +78,7 @@ public class SqlCacheContext { private volatile long latestPartitionVersion = -1; private volatile long sumOfPartitionNum = -1; // value: version of table - private final Map usedTables = Maps.newLinkedHashMap(); + private final Map usedTables = Maps.newLinkedHashMap(); // value: ddl sql private final Map usedViews = Maps.newLinkedHashMap(); // value: usedColumns @@ -172,6 +176,17 @@ sessionVariable. readAffectQueryResultVariables((k, v) -> { /** addUsedTable */ public synchronized void addUsedTable(TableIf tableIf) { + addUsedTable(tableIf, Optional.empty()); + } + + /** Record the exact MVCC generation consumed by this relation. */ + public synchronized void addUsedTable(TableIf tableIf, Optional snapshot) { + addUsedTable(tableIf, snapshot, MvccReadSelector.LATEST); + } + + /** Record one table reference independently of other versions of the same table. */ + public synchronized void addUsedTable(TableIf tableIf, Optional snapshot, + MvccReadSelector selector) { if (tableIf == null) { return; } @@ -190,10 +205,19 @@ public synchronized void addUsedTable(TableIf tableIf) { } long version = 0; + ConnectorSqlCacheVersion sqlCacheVersion = null; try { + if (tableIf instanceof SqlCacheVersionProvider) { + sqlCacheVersion = ((SqlCacheVersionProvider) tableIf).getSqlCacheVersion(snapshot).orElse(null); + } + if (!selector.isLatest() && sqlCacheVersion == null) { + // Legacy freshness is table-wide, so it cannot validate a historical/branch dependency. + setHasUnsupportedTables(true); + return; + } if (tableIf instanceof OlapTable) { version = ((OlapTable) tableIf).getVisibleVersion(); - } else if (tableIf instanceof MTMVRelatedTableIf) { + } else if (sqlCacheVersion == null && tableIf instanceof MTMVRelatedTableIf) { // Connector-agnostic data-version token (hive: max transient_lastDdlTime; iceberg/paimon: // monotonic snapshot version). OlapTable is handled above; MTMV is an OlapTable too, so this // arm only catches external MVCC tables (flipped hive/iceberg/paimon/hudi). @@ -212,11 +236,12 @@ public synchronized void addUsedTable(TableIf tableIf) { } usedTables.put( - new FullTableName(database.getCatalog().getName(), database.getFullName(), tableIf.getName()), + new TableReference(new FullTableName(database.getCatalog().getName(), + database.getFullName(), tableIf.getName()), selector), new TableVersion( tableIf.getId(), version, - tableIf.getType() + tableIf.getType(), sqlCacheVersion ) ); } @@ -362,7 +387,7 @@ public void setCacheProxy(CacheProxy cacheProxy) { this.cacheProxy = cacheProxy; } - public Map getUsedTables() { + public Map getUsedTables() { return Collections.unmodifiableMap(usedTables); } @@ -446,15 +471,18 @@ public PUniqueId getOrComputeCacheKeyMd5(SessionVariable sessionVariable) { public synchronized PUniqueId doComputeCacheKeyMd5( Set usedVariables, SessionVariable sessionVariable) { StringBuilder cacheKey = new StringBuilder(NereidsParser.removeCommentAndTrimBlank(originSql.trim())); - for (Entry entry : usedTables.entrySet()) { + for (Entry entry : usedTables.entrySet()) { TableVersion tableVersion = entry.getValue(); - FullTableName tableName = entry.getKey(); + TableReference reference = entry.getKey(); + String selectorKey = reference.selector.getKey(); cacheKey.append("|") - .append(tableName) + .append(reference.tableName) + .append("#").append(selectorKey.length()).append(":").append(selectorKey) .append("=") .append(tableVersion.id) .append("@") - .append(tableVersion.version); + .append(tableVersion.sqlCacheVersion == null ? Long.toString(tableVersion.version) + : "sql-cache:" + tableVersion.sqlCacheVersion.getVersion()); } for (Entry entry : usedViews.entrySet()) { cacheKey.append("|") @@ -590,6 +618,14 @@ public void addScanPartition(Long partitionId) { } } + /** A table dependency identified by the original read selector, not by an alias or resolved snapshot. */ + @lombok.Data + @lombok.AllArgsConstructor + public static class TableReference { + public final FullTableName tableName; + public final MvccReadSelector selector; + } + /** TableVersion */ @lombok.Data @lombok.AllArgsConstructor @@ -597,6 +633,11 @@ public static class TableVersion { public final long id; public final long version; public final TableType type; + public final ConnectorSqlCacheVersion sqlCacheVersion; + + public TableVersion(long id, long version, TableType type) { + this(id, version, type, null); + } } /** CacheKeyType */ diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java index cd71830ed64f79..80bd559d339b6a 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java @@ -35,6 +35,7 @@ import org.apache.doris.connector.spi.ConnectorStatementScope; import org.apache.doris.connector.spi.handle.ConnectorTransaction; import org.apache.doris.datasource.ExternalTable; +import org.apache.doris.datasource.mvcc.MvccReadSelector; import org.apache.doris.datasource.mvcc.MvccSnapshot; import org.apache.doris.datasource.mvcc.MvccTable; import org.apache.doris.datasource.mvcc.MvccTableInfo; @@ -83,6 +84,7 @@ import com.google.common.base.Supplier; import com.google.common.base.Suppliers; import com.google.common.base.Throwables; +import com.google.common.collect.ArrayListMultimap; import com.google.common.collect.HashMultimap; import com.google.common.collect.ImmutableList; import com.google.common.collect.Maps; @@ -337,9 +339,10 @@ public enum TableFrom { // if greater than 0 means the duration has used private long materializedViewRewriteDuration = 0L; - // Record used table and it's used partitions + // Records are consumed by iteration/union, not membership lookup. List values avoid hashing every + // partition name inside Pair when recording a scan; repeated records do not change the union. private final Multimap, Pair>> tableUsedPartitionNameMap = - HashMultimap.create(); + ArrayListMultimap.create(); // Record query common table id to relation id mapping, this is used for mv rewrite private final Multimap commonTableIdToRelationIdToMap = HashMultimap.create(); @@ -1456,40 +1459,7 @@ public Optional getSnapshot(TableIf tableIf, Optional tableSnapshot, Optional scanParams) { - // Concatenate both selectors (rather than returning on the first) so the key stays injective even - // if a reference ever carried BOTH a snapshot and scan params. Today the two are mutually exclusive - // (e.g. IcebergUtils.getQuerySpecSnapshot rejects FOR-TIME together with @branch/@tag), so in every - // reachable case exactly one part is non-empty and the key is "v:...", "p:...", or "". - StringBuilder key = new StringBuilder(); - if (tableSnapshot != null && tableSnapshot.isPresent()) { - TableSnapshot ts = tableSnapshot.get(); - key.append('v'); - appendVersionKeyPart(key, ts.getType()); - appendVersionKeyPart(key, ts.getValue()); - } - if (scanParams != null && scanParams.isPresent()) { - TableScanParams sp = scanParams.get(); - key.append('p'); - appendVersionKeyPart(key, sp.getParamType()); - Map sortedParams = new TreeMap<>(sp.getMapParams()); - key.append('m').append(sortedParams.size()).append(':'); - for (Map.Entry entry : sortedParams.entrySet()) { - // Length prefixes preserve entry boundaries even when user values contain Map.toString() - // delimiters; sorting separately keeps semantically identical maps order-independent. - appendVersionKeyPart(key, entry.getKey()); - appendVersionKeyPart(key, entry.getValue()); - } - key.append('l').append(sp.getListParams().size()).append(':'); - for (String value : sp.getListParams()) { - appendVersionKeyPart(key, value); - } - } - return key.toString(); - } - - private static void appendVersionKeyPart(StringBuilder key, Object value) { - String text = String.valueOf(value); - key.append(text.length()).append(':').append(text); + return MvccReadSelector.of(tableSnapshot, scanParams).getKey(); } /** diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java index 87c4d800486ba3..b73b1c098fa204 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java @@ -48,6 +48,7 @@ import org.apache.doris.datasource.ExternalTable; import org.apache.doris.datasource.ExternalView; import org.apache.doris.datasource.doris.RemoteDorisExternalTable; +import org.apache.doris.datasource.mvcc.MvccReadSelector; import org.apache.doris.datasource.plugin.PluginDrivenExternalTable; import org.apache.doris.datasource.plugin.PluginDrivenSysExternalTable; import org.apache.doris.datasource.systable.SysTableResolver; @@ -938,11 +939,20 @@ private LogicalPlan getLogicalPlan(TableIf table, UnboundRelation unboundRelatio sqlCacheContext.addUsedTable(table); } else if (table instanceof ExternalTable && table instanceof MTMVRelatedTableIf && cascadesContext.getConnectContext().getSessionVariable().enableHiveSqlCache) { - // Any external lakehouse plugin table that exposes a stable data-version token - // (MTMVRelatedTableIf#getNewestUpdateVersionOrTime) is cacheable; addUsedTable - // records the token and fails safe if it is unavailable. Gated by the (default - // false) enable_hive_sql_cache switch so behavior is unchanged unless opted in. - sqlCacheContext.addUsedTable(table); + // Record a connector's lightweight SQL-cache version when available, otherwise + // retain the legacy freshness path. The external cache opt-in applies to both. + // Incremental/options scans need their own cache identity contract; point-in-time + // references use the same selector and snapshot as ordinary analysis and scanning. + TableScanParams params = unboundRelation.getScanParams(); + if (params != null && !params.isBranch() && !params.isTag()) { + sqlCacheContext.setHasUnsupportedTables(true); + } else { + MvccReadSelector selector = MvccReadSelector.of(unboundRelation.getTableSnapshot(), + Optional.ofNullable(params)); + sqlCacheContext.addUsedTable(table, cascadesContext.getStatementContext() + .getSnapshot(table, selector.getTableSnapshot(), selector.getScanParams()), + selector); + } } else { sqlCacheContext.setHasUnsupportedTables(true); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneFileScanPartition.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneFileScanPartition.java index 5589026206a849..4ee20db8c1cc43 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneFileScanPartition.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneFileScanPartition.java @@ -61,7 +61,8 @@ public class PruneFileScanPartition extends OneRewriteRuleFactory { @Override public Rule build() { - return logicalFilter(logicalFileScan()).whenNot(p -> p.child().getSelectedPartitions().isPruned) + return logicalFilter(logicalFileScan()).whenNot(p -> p.child().getSelectedPartitions().isPruned + || p.child().getSelectedPartitions().isNativePlanning()) .thenApply(ctx -> { LogicalFilter filter = ctx.root; LogicalFileScan scan = filter.child(); @@ -134,12 +135,8 @@ private SelectedPartitions pruneExternalPartitions(ExternalTable externalTable, for (String name : prunedPartitions) { PartitionItem item = nameToPartitionItem.get(name); - // Within THIS query, nameToPartitionItem and sortedPartitionRanges are built from the same - // frozen map. On a cross-query cache HIT, sortedPartitionRanges instead reuses ranges built by - // an earlier query keyed by the identical (snapshotId, schemaId) version token -- content is - // identical only via the MVCC determinism premise (same version token => same partition set), - // not because the two maps are literally the same object. A missing item here means that - // premise was violated, so fail loud rather than silently returning a partial scan. + // The index and selected map must describe the same pinned partition generation. + // Plugin MVCC views publish them together and share the very same PartitionItem objects. Preconditions.checkState(item != null, "pruned partition %s is missing in the selected partitions snapshot", name); selectedPartitionItems.put(name, item); diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/QueryPartitionCollector.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/QueryPartitionCollector.java index e0c833f95b14b1..e9d69f96163b7d 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/QueryPartitionCollector.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/QueryPartitionCollector.java @@ -61,8 +61,8 @@ public Void visitLogicalCatalogRelation(LogicalCatalogRelation catalogRelation, // Collect table used partition mapping Multimap, Pair>> tableUsedPartitionNameMap = statementContext .getTableUsedPartitionNameMap(); - Set tablePartitions = new HashSet<>(); if (catalogRelation instanceof LogicalOlapScan) { + Set tablePartitions = new HashSet<>(); // Handle olap table LogicalOlapScan logicalOlapScan = (LogicalOlapScan) catalogRelation; for (Long partitionId : logicalOlapScan.getSelectedPartitionIds()) { @@ -72,12 +72,18 @@ public Void visitLogicalCatalogRelation(LogicalCatalogRelation catalogRelation, Pair.of(catalogRelation.getRelationId(), tablePartitions)); } else if (catalogRelation instanceof LogicalFileScan && catalogRelation.getTable() != null - && ((ExternalTable) catalogRelation.getTable()).supportInternalPartitionPruned()) { + && (((LogicalFileScan) catalogRelation).getSelectedPartitions().isNativePlanning() + || ((ExternalTable) catalogRelation.getTable()).supportInternalPartitionPruned())) { LogicalFileScan logicalFileScan = (LogicalFileScan) catalogRelation; SelectedPartitions selectedPartitions = logicalFileScan.getSelectedPartitions(); - tablePartitions.addAll(selectedPartitions.selectedPartitions.keySet()); + if (selectedPartitions.isNativePlanning()) { + tableUsedPartitionNameMap.put(table.getFullQualifiers(), PartitionCompensator.ALL_PARTITIONS); + return null; + } + // SelectedPartitions owns an immutable map. Keep its key view rather than copying every + // partition name; compensation builds its own mutable union when it is actually needed. tableUsedPartitionNameMap.put(table.getFullQualifiers(), - Pair.of(catalogRelation.getRelationId(), tablePartitions)); + Pair.of(catalogRelation.getRelationId(), selectedPartitions.selectedPartitions.keySet())); } else { // not support get partition scene, we consider query all partitions from table tableUsedPartitionNameMap.put(table.getFullQualifiers(), PartitionCompensator.ALL_PARTITIONS); diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalFileScan.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalFileScan.java index 09b5e549b5a473..2dc541f5a45d78 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalFileScan.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalFileScan.java @@ -305,6 +305,10 @@ public static class SelectedPartitions { // Or used to indicate that the partition pruning is not processed. public static SelectedPartitions NOT_PRUNED = new SelectedPartitions(0, ImmutableMap.of(), false, false, Optional.empty()); + /** Connector-owned file planning; an unknown universe, never an empty pruned selection. */ + public static final SelectedPartitions NATIVE = new SelectedPartitions( + -1, ImmutableMap.of(), false, false, Optional.empty(), true); + /** * total partition number */ @@ -334,6 +338,8 @@ public static class SelectedPartitions { /** * Constructor for SelectedPartitions. */ + private final boolean nativePlanning; + public SelectedPartitions(long totalPartitionNum, Map selectedPartitions, boolean isPruned) { this(totalPartitionNum, selectedPartitions, isPruned, false, Optional.empty()); @@ -353,6 +359,13 @@ public SelectedPartitions(long totalPartitionNum, Map sel public SelectedPartitions(long totalPartitionNum, Map selectedPartitions, boolean isPruned, boolean hasPartitionPredicate, Optional> sortedPartitionRanges) { + this(totalPartitionNum, selectedPartitions, isPruned, hasPartitionPredicate, sortedPartitionRanges, false); + } + + private SelectedPartitions(long totalPartitionNum, Map selectedPartitions, + boolean isPruned, boolean hasPartitionPredicate, + Optional> sortedPartitionRanges, boolean nativePlanning) { + this.nativePlanning = nativePlanning; this.totalPartitionNum = totalPartitionNum; this.selectedPartitions = ImmutableMap.copyOf(Objects.requireNonNull(selectedPartitions, "selectedPartitions is null")); @@ -361,6 +374,10 @@ public SelectedPartitions(long totalPartitionNum, Map sel this.sortedPartitionRanges = sortedPartitionRanges; } + public boolean isNativePlanning() { + return nativePlanning; + } + @Override public boolean equals(Object o) { if (this == o) { @@ -370,7 +387,7 @@ public boolean equals(Object o) { return false; } SelectedPartitions that = (SelectedPartitions) o; - return isPruned == that.isPruned + return nativePlanning == that.nativePlanning && isPruned == that.isPruned && hasPartitionPredicate == that.hasPartitionPredicate && Objects.equals( selectedPartitions.keySet(), that.selectedPartitions.keySet()) @@ -380,7 +397,8 @@ public boolean equals(Object o) { @Override public int hashCode() { - return Objects.hash(selectedPartitions, isPruned, hasPartitionPredicate, sortedPartitionRanges.isPresent()); + return Objects.hash(selectedPartitions, isPruned, hasPartitionPredicate, + sortedPartitionRanges.isPresent(), nativePlanning); } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java index 6c65b9ddaf40c0..f16f9639e13573 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java @@ -589,6 +589,8 @@ public String toString() { public static final String ICEBERG_WRITE_TARGET_FILE_SIZE_BYTES = "iceberg_write_target_file_size_bytes"; public static final String ENABLE_ICEBERG_MERGE_PARTITIONING = "enable_iceberg_merge_partitioning"; public static final String ENABLE_PAIMON_WRITE_CHAR_PADDING = "enable_paimon_write_char_padding"; + public static final String ENABLE_ICEBERG_INTERNAL_PARTITION_PRUNING = + "enable_iceberg_internal_partition_pruning"; public static final String NUM_PARTITIONS_IN_BATCH_MODE = "num_partitions_in_batch_mode"; @@ -3734,6 +3736,11 @@ public void checkAnnIndexCandidateRowsPercentThreshold(String value) { options = {ICEBERG_PARTITION_OVERWRITE_MODE_STATIC, ICEBERG_PARTITION_OVERWRITE_MODE_DYNAMIC}) public String icebergPartitionOverwriteMode = ICEBERG_PARTITION_OVERWRITE_MODE_STATIC; + @VarAttrDef.VarAttr(name = ENABLE_ICEBERG_INTERNAL_PARTITION_PRUNING, + description = "Enable Doris FE partition pruning for Iceberg scans. When disabled, ordinary scans " + + "defer the Doris partition view and rely on Iceberg native file pruning. Materialized view " + + "planning and partition metadata rules still require the full partition view.") + public boolean enableIcebergInternalPartitionPruning = false; // If this fe is in fuzzy mode, then will use initFuzzyModeVariables to generate some variables, // not the default value set in the code. diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/cache/CacheAnalyzer.java b/fe/fe-core/src/main/java/org/apache/doris/qe/cache/CacheAnalyzer.java index 1b5827ad878f6a..d73315facc9cee 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/qe/cache/CacheAnalyzer.java +++ b/fe/fe-core/src/main/java/org/apache/doris/qe/cache/CacheAnalyzer.java @@ -33,13 +33,17 @@ import org.apache.doris.common.Status; import org.apache.doris.common.UserException; import org.apache.doris.common.util.DebugUtil; +import org.apache.doris.connector.spi.mvcc.ConnectorSqlCacheVersion; import org.apache.doris.datasource.CatalogIf; +import org.apache.doris.datasource.mvcc.MvccReadSelector; +import org.apache.doris.datasource.scan.FileQueryScanNode; import org.apache.doris.metric.MetricRepo; import org.apache.doris.mtmv.MTMVRelatedTableIf; import org.apache.doris.nereids.NereidsPlanner; import org.apache.doris.nereids.SqlCacheContext; import org.apache.doris.nereids.SqlCacheContext.FullTableName; import org.apache.doris.nereids.SqlCacheContext.ScanTable; +import org.apache.doris.nereids.SqlCacheContext.TableReference; import org.apache.doris.nereids.glue.LogicalPlanAdapter; import org.apache.doris.planner.OlapScanNode; import org.apache.doris.planner.Planner; @@ -234,6 +238,10 @@ private CacheMode innerCheckCacheModeForNereids(long now) { return CacheMode.NoNeed; } + if (Optional.ofNullable(context.getStatementContext()).flatMap(statement -> statement.getSqlCacheContext()) + .map(cacheContext -> !cacheContext.supportSqlCache()).orElse(false)) { + return CacheMode.NoNeed; + } //Check the last version time of the table List tblTimeList = buildCacheTableList(); if (CollectionUtils.isEmpty(tblTimeList)) { @@ -514,6 +522,26 @@ private CacheTable buildCacheTableForExternalScanNode(ScanNode node) { CacheTable cacheTable = new CacheTable(); TableIf tableIf = node.getTupleDesc().getTable(); cacheTable.table = tableIf; + DatabaseIf database = tableIf.getDatabase(); + CatalogIf catalog = database.getCatalog(); + FullTableName name = new FullTableName(catalog.getName(), database.getFullName(), tableIf.getName()); + MvccReadSelector selector = node instanceof FileQueryScanNode + ? MvccReadSelector.of(Optional.ofNullable(((FileQueryScanNode) node).getQueryTableSnapshot()), + Optional.ofNullable(((FileQueryScanNode) node).getScanParams())) + : MvccReadSelector.LATEST; + Optional version = Optional.ofNullable(context.getStatementContext()) + .flatMap(statement -> statement.getSqlCacheContext()) + .map(cacheContext -> cacheContext.getUsedTables().get(new TableReference(name, selector))) + .map(tableVersion -> tableVersion.sqlCacheVersion); + if (version.isPresent()) { + // The full identity is already in the SQL MD5 key. PCache needs only a stable timestamp and + // one logical whole-table unit; neither physical partition counts nor another latest probe. + cacheTable.partitionNum = 1; + cacheTable.latestPartitionTime = version.get().getUpdateTimeMillis(); + cacheTable.latestPartitionUpdateMillis = version.get().getUpdateTimeMillis(); + scanTables.add(Pair.of(new ScanTable(name), tableIf)); + return cacheTable; + } cacheTable.partitionNum = node.getSelectedPartitionNum(); // Connector-agnostic data-version token (hive: max transient_lastDdlTime; iceberg/paimon: monotonic // snapshot version). Gated to MTMVRelatedTableIf tables by isExternalCacheableScanNode above. This is @@ -524,10 +552,7 @@ private CacheTable buildCacheTableForExternalScanNode(ScanNode node) { // millis here; hive/paimon already return millis). Kept separate from the token above so the gate never // subtracts a non-wall-clock value. cacheTable.latestPartitionUpdateMillis = ((MTMVRelatedTableIf) tableIf).getNewestUpdateTimeMillisForCache(); - DatabaseIf database = tableIf.getDatabase(); - CatalogIf catalog = database.getCatalog(); - ScanTable scanTable = new ScanTable(new FullTableName( - catalog.getName(), database.getFullName(), tableIf.getName())); + ScanTable scanTable = new ScanTable(name); scanTables.add(Pair.of(scanTable, tableIf)); return cacheTable; } diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/cache/SqlCacheVersionProvider.java b/fe/fe-core/src/main/java/org/apache/doris/qe/cache/SqlCacheVersionProvider.java new file mode 100644 index 00000000000000..0591ab239b719c --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/qe/cache/SqlCacheVersionProvider.java @@ -0,0 +1,32 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.qe.cache; + +import org.apache.doris.connector.spi.mvcc.ConnectorSqlCacheVersion; +import org.apache.doris.datasource.mvcc.MvccSnapshot; + +import java.util.Optional; + +/** + * Supplies a whole-table SQL result-cache version independently of MV/dictionary freshness semantics. + * A supplied snapshot is the query's read generation; empty probes the current generation for cache lookup. + * Empty results preserve the caller's legacy freshness path. Failures disable caching for the query. + */ +public interface SqlCacheVersionProvider { + Optional getSqlCacheVersion(Optional snapshot); +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/alter/ChangelogConfigPublicationTest.java b/fe/fe-core/src/test/java/org/apache/doris/alter/ChangelogConfigPublicationTest.java index 5c364b5fcf0ec2..7d4a0c9d3139e0 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/alter/ChangelogConfigPublicationTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/alter/ChangelogConfigPublicationTest.java @@ -39,6 +39,7 @@ import org.apache.doris.common.cache.NereidsSqlCacheManager; import org.apache.doris.common.util.PropertyAnalyzer; import org.apache.doris.datasource.InternalCatalog; +import org.apache.doris.datasource.mvcc.MvccReadSelector; import org.apache.doris.nereids.SqlCacheContext; import org.apache.doris.nereids.trees.plans.commands.info.ModifyTablePropertiesOp; import org.apache.doris.persist.EditLog; @@ -199,7 +200,8 @@ public void failedPublicationCanBeReplayedAndRetriedByANewMaster() throws Except SqlCacheContext cachedRead = Mockito.mock(SqlCacheContext.class); SqlCacheContext.TableVersion version = new SqlCacheContext.TableVersion(table.getId(), 1, Table.TableType.OLAP); Mockito.when(cachedRead.getUsedTables()).thenReturn(ImmutableMap.of( - new SqlCacheContext.FullTableName("internal", "db", "tbl"), version)); + new SqlCacheContext.TableReference(new SqlCacheContext.FullTableName("internal", "db", "tbl"), + MvccReadSelector.LATEST), version)); cache.getSqlCaches().put("cached-incr-read", cachedRead); Mockito.when(replayEnv.getSqlCacheManager()).thenReturn(cache); InternalCatalog catalog = Mockito.mock(InternalCatalog.class); diff --git a/fe/fe-core/src/test/java/org/apache/doris/blockrule/SqlBlockRuleMgrTest.java b/fe/fe-core/src/test/java/org/apache/doris/blockrule/SqlBlockRuleMgrTest.java index cb66b2a52f33ce..cc083c62c40eb3 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/blockrule/SqlBlockRuleMgrTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/blockrule/SqlBlockRuleMgrTest.java @@ -30,6 +30,33 @@ import java.util.concurrent.ConcurrentHashMap; public class SqlBlockRuleMgrTest { + @Test + public void testPartitionMetadataIsRequiredOnlyByEnabledPartitionRules() { + SqlBlockRuleMgr mgr = new SqlBlockRuleMgr(); + SqlBlockRule rule = new SqlBlockRule("r", "NULL", "NULL", 0L, 0L, 10L, + false, false, true); + mgr.getNameToSqlBlockRuleMap().put("r", rule); + Assertions.assertFalse(mgr.hasPartitionMetadataRules()); + rule.setPartitionNum(2L); + Assertions.assertTrue(mgr.hasPartitionMetadataRules()); + rule.setEnable(false); + Assertions.assertFalse(mgr.hasPartitionMetadataRules()); + rule.setPartitionNum(0L); + rule.setRequirePartitionFilter(true); + rule.setEnable(true); + Assertions.assertTrue(mgr.hasPartitionMetadataRules()); + } + + @Test + public void testNewPartitionRuleCannotTreatAnUnknownUniverseAsZero() { + SqlBlockRuleMgr mgr = new SqlBlockRuleMgr(); + SqlBlockRule rule = new SqlBlockRule("r", "NULL", "NULL", 1L, 0L, 0L, + false, true, true); + AnalysisException error = Assertions.assertThrows(AnalysisException.class, + () -> mgr.checkLimitations(rule, -1L, 1L, 10L, true, false)); + Assertions.assertTrue(error.getMessage().contains("retry the query")); + } + @BeforeAll public static void setUp() { MetricRepo.init(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/common/cache/NereidsSortedPartitionsCacheManagerExternalTest.java b/fe/fe-core/src/test/java/org/apache/doris/common/cache/NereidsSortedPartitionsCacheManagerExternalTest.java index 467b96dd2fb011..23180262aa5533 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/common/cache/NereidsSortedPartitionsCacheManagerExternalTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/common/cache/NereidsSortedPartitionsCacheManagerExternalTest.java @@ -296,22 +296,10 @@ public void testPluginDrivenMvccExternalTableRealOriginPartitionsAndVersion() th // ──────────────────── Cache B version is the frozen partition NAME-SET for ALL engines ───── // - // getPartitionMetaVersion always returns the frozen partition NAME SET - // (getOriginPartitions(scan).keySet(), copied) -- the SAME map the ranges are built from, so - // version == exact content and Cache B rebuilds precisely when the partition set changes, never on a - // stale set. This is uniform across hive (snapshotId == -1 sentinel), paimon and iceberg: the old - // "@" O(1) token was removed because it is unsafe wherever a connector's - // listPartitions content is not a pure function of the snapshot id (iceberg non-RANGE: identity / - // bucket / truncate / multi-field partitioning), whose nameToPartitionItem (Cache A) and snapshot-id - // token (a DIFFERENT cache) expire independently. PluginDrivenMvccExternalTable no longer overrides - // getSortedPartitionRanges (the -1 short-circuit added in 5c17b748880 was removed): every pin goes - // through the inherited ExternalTable#getSortedPartitionRanges -> the cache manager. + // External MVCC tables return the index owned by their pinned partition view. These cases + // verify that changes in partition content remain visible even at a constant snapshot id. - /** - * Builds a live Env whose {@code getSortedPartitionsCacheManager()} returns {@code rangesCacheMgr}, - * mockStatic-scoped so {@code table.getSortedPartitionRanges(scan)} (now purely inherited from - * {@link ExternalTable}, no override) can resolve {@code Env.getCurrentEnv()} on the real dispatch path. - */ + /** Supplies the legacy manager for tests that also cover generic ExternalTable delegation. */ private static MockedStatic mockEnvWithRangesCacheManager(NereidsSortedPartitionsCacheManager rangesCacheMgr) { Env env = Mockito.mock(Env.class); Mockito.when(env.getSortedPartitionsCacheManager()).thenReturn(rangesCacheMgr); @@ -381,8 +369,7 @@ public void testGetSortedPartitionRangesPresentForSnapshotLessNonEmptyPartitions Optional> ranges = table.getSortedPartitionRanges(scan); Assertions.assertTrue(ranges.isPresent(), - "snapshotId == -1 (hive sentinel) with a non-empty partition set must now use Cache B, " - + "keyed by the partition NAME-SET version token"); + "snapshot-less tables also publish an index with their non-empty partition view"); } finally { ConnectContext.remove(); } @@ -402,8 +389,7 @@ public void testGetSortedPartitionRangesRebuildsWhenSnapshotLessNameSetChanges() Assertions.assertEquals(2, first.sortedPartitions.size()); // A second query ("ALTER ADD PARTITION" between queries): the pin's partition NAME SET grew. - // The manager's Objects.equals compares the two HashSet version tokens by CONTENT, so it must - // detect this change even though snapshotId is still the constant -1 on both pins. + // The new pin owns its new index even though snapshotId is still -1 on both pins. Map partsAtQuery2 = Maps.newHashMap(); partsAtQuery2.put("id=1", listItem(1)); partsAtQuery2.put("id=2", listItem(2)); @@ -435,16 +421,17 @@ public void testGetSortedPartitionRangesRebuildsWhenRealSnapshotNameSetChanges() Map partsAtQuery1 = Maps.newHashMap(); partsAtQuery1.put("id=1", listItem(1)); partsAtQuery1.put("id=2", listItem(2)); - LogicalFileScan scan1 = pinLatestAndBuildScan(table, realSnapshotPin(partsAtQuery1)); + PluginDrivenMvccSnapshot firstPin = realSnapshotPin(partsAtQuery1); + LogicalFileScan scan1 = pinLatestAndBuildScan(table, firstPin); SortedPartitionRanges first = table.getSortedPartitionRanges(scan1).orElse(null); Assertions.assertNotNull(first, "ranges built and cached at the first (2-partition) name set"); Assertions.assertEquals(2, first.sortedPartitions.size()); - // Same query again: IDENTICAL name set at the SAME real snapshot id => cache HIT (same instance). - LogicalFileScan scanHit = pinLatestAndBuildScan(table, realSnapshotPin(partsAtQuery1)); + // A reused partition view owns the same index; no second table-wide sorted-cache lookup. + LogicalFileScan scanHit = pinLatestAndBuildScan(table, firstPin); SortedPartitionRanges hit = table.getSortedPartitionRanges(scanHit).orElse(null); Assertions.assertSame(first, hit, - "unchanged name set at a real snapshotId=42 => cache HIT returns the SAME instance"); + "the same pinned view must return the same index"); // A later query whose Cache A refreshed to include a new partition while the connector snapshot // id is STILL 42: the name set grew, so Cache B must rebuild even though "42@7" is unchanged. diff --git a/fe/fe-core/src/test/java/org/apache/doris/common/cache/NereidsSqlCacheManagerPluginTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/common/cache/NereidsSqlCacheManagerPluginTableTest.java index 44cf75f2f11d75..baa61b6c91614b 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/common/cache/NereidsSqlCacheManagerPluginTableTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/common/cache/NereidsSqlCacheManagerPluginTableTest.java @@ -17,15 +17,20 @@ package org.apache.doris.common.cache; +import org.apache.doris.analysis.TableSnapshot; import org.apache.doris.analysis.UserIdentity; import org.apache.doris.catalog.DatabaseIf; import org.apache.doris.catalog.Env; import org.apache.doris.catalog.TableIf.TableType; import org.apache.doris.common.jmockit.Deencapsulation; +import org.apache.doris.connector.spi.mvcc.ConnectorSqlCacheVersion; import org.apache.doris.datasource.CatalogIf; import org.apache.doris.datasource.CatalogMgr; +import org.apache.doris.datasource.mvcc.MvccReadSelector; +import org.apache.doris.datasource.mvcc.MvccSnapshot; import org.apache.doris.datasource.mvcc.PluginDrivenMvccExternalTable; import org.apache.doris.nereids.SqlCacheContext; +import org.apache.doris.nereids.StatementContext; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -49,6 +54,81 @@ public class NereidsSqlCacheManagerPluginTableTest { private static final String TBL = "t"; private static final long TABLE_ID = 42L; + @Test + public void nativeVersionInvalidatesByIdentityWithoutOrderingOrPartitionFreshness() { + PluginDrivenMvccExternalTable stored = mockTable(123L); + Mockito.when(stored.getSqlCacheVersion(Optional.empty())) + .thenReturn(Optional.of(new ConnectorSqlCacheVersion("uuid:9:schema1:metadata1", 1000L))); + SqlCacheContext context = new SqlCacheContext(UserIdentity.ROOT); + context.addUsedTable(stored); + PluginDrivenMvccExternalTable live = mockTable(123L); + Env env = mockEnvResolvingTo(live); + for (String token : new String[] {"uuid:9:schema1:metadata1", "uuid:2:schema1:metadata2", + "uuid:9:schema2:metadata3", "new-uuid:9:schema1:metadata1"}) { + Mockito.when(live.getSqlCacheVersion(Optional.empty())) + .thenReturn(Optional.of(new ConnectorSqlCacheVersion(token, 1000L))); + Object verdict = Deencapsulation.invoke(new NereidsSqlCacheManager(), "tablesOrDataChanged", + env, context, new StatementContext()); + Assertions.assertEquals(!token.equals("uuid:9:schema1:metadata1"), isChangedField(verdict, "changed")); + } + Mockito.when(live.getSqlCacheVersion(Optional.empty())).thenReturn(Optional.empty()); + Object verdict = Deencapsulation.invoke(new NereidsSqlCacheManager(), "tablesOrDataChanged", + env, context, new StatementContext()); + Assertions.assertTrue(isChangedField(verdict, "invalidCache")); + Mockito.verify(stored, Mockito.never()).getNewestUpdateVersionOrTime(); + Mockito.verify(live, Mockito.never()).getNewestUpdateVersionOrTime(); + } + + @Test + public void revalidationResolvesEachOriginalReferenceAndInvalidatesOnlyWhenItsVersionChanges() { + PluginDrivenMvccExternalTable table = mockTable(999L); + MvccSnapshot mainPin = Mockito.mock(MvccSnapshot.class); + MvccSnapshot devPin = Mockito.mock(MvccSnapshot.class); + ConnectorSqlCacheVersion main = new ConnectorSqlCacheVersion("uuid:100:schema3:metadata", 1000L); + ConnectorSqlCacheVersion dev = new ConnectorSqlCacheVersion("uuid:80:schema3:metadata", 800L); + Mockito.when(table.getSqlCacheVersion(Optional.of(mainPin))).thenReturn(Optional.of(main)); + Mockito.when(table.getSqlCacheVersion(Optional.of(devPin))).thenReturn(Optional.of(dev)); + Mockito.when(table.getSqlCacheVersion(Optional.empty())).thenReturn(Optional.of(main)); + MvccReadSelector selector = MvccReadSelector.of(Optional.of(TableSnapshot.versionOf("dev")), Optional.empty()); + SqlCacheContext context = new SqlCacheContext(UserIdentity.ROOT); + context.addUsedTable(table, Optional.of(mainPin)); + context.addUsedTable(table, Optional.of(devPin), selector); + StatementContext statement = Mockito.mock(StatementContext.class); + Mockito.when(statement.getSnapshot(Mockito.eq(table), Mockito.any(), Mockito.any())) + .thenAnswer(call -> { + Optional requested = call.getArgument(1); + Assertions.assertEquals("dev", requested.orElseThrow().getValue()); + return Optional.of(devPin); + }); + Env env = mockEnvResolvingTo(table); + Object same = Deencapsulation.invoke(new NereidsSqlCacheManager(), "tablesOrDataChanged", env, context, statement); + Assertions.assertFalse(isChangedField(same, "changed")); + Mockito.verify(statement).loadSnapshots(Mockito.eq(table), Mockito.any(), Mockito.any()); + Mockito.when(table.getSqlCacheVersion(Optional.of(devPin))) + .thenReturn(Optional.of(new ConnectorSqlCacheVersion("uuid:90:schema3:metadata", 800L))); + Object moved = Deencapsulation.invoke(new NereidsSqlCacheManager(), "tablesOrDataChanged", env, context, statement); + Assertions.assertTrue(isChangedField(moved, "invalidCache")); + Mockito.verify(table, Mockito.never()).getNewestUpdateVersionOrTime(); + } + + @Test + public void expiredSnapshotCannotPassVersionValidation() { + PluginDrivenMvccExternalTable table = mockTable(999L); + MvccSnapshot pin = Mockito.mock(MvccSnapshot.class); + Mockito.when(table.getSqlCacheVersion(Optional.of(pin))) + .thenReturn(Optional.of(new ConnectorSqlCacheVersion("uuid:80:schema3:metadata", 800L))); + SqlCacheContext context = new SqlCacheContext(UserIdentity.ROOT); + context.addUsedTable(table, Optional.of(pin), + MvccReadSelector.of(Optional.of(TableSnapshot.versionOf("80")), Optional.empty())); + StatementContext statement = Mockito.mock(StatementContext.class); + Mockito.doThrow(new IllegalStateException("can't find snapshot by id: 80")) + .when(statement).loadSnapshots(Mockito.eq(table), Mockito.any(), Mockito.any()); + Env env = mockEnvResolvingTo(table); + Assertions.assertThrows(IllegalStateException.class, + () -> Deencapsulation.invoke(new NereidsSqlCacheManager(), "tablesOrDataChanged", env, context, statement)); + Mockito.verify(table, Mockito.never()).getSqlCacheVersion(Optional.empty()); + } + private PluginDrivenMvccExternalTable mockTable(long liveToken) { PluginDrivenMvccExternalTable table = Mockito.mock(PluginDrivenMvccExternalTable.class); DatabaseIf db = Mockito.mock(DatabaseIf.class); @@ -93,7 +173,8 @@ public void testNotChangedWhenTokenUnchanged() { Env env = mockEnvResolvingTo(mockTable(token)); // live token unchanged Object verdict = Deencapsulation.invoke(new NereidsSqlCacheManager(), - "tablesOrDataChanged", env, context); + "tablesOrDataChanged", + env, context, new StatementContext()); Assertions.assertFalse(isChangedField(verdict, "changed"), "an unchanged token must keep the cache (NOT_CHANGED)"); @@ -108,7 +189,8 @@ public void testInvalidateWhenTokenChanged() { Env env = mockEnvResolvingTo(mockTable(storedToken + 1000L)); // data mutated: newer token Object verdict = Deencapsulation.invoke(new NereidsSqlCacheManager(), - "tablesOrDataChanged", env, context); + "tablesOrDataChanged", + env, context, new StatementContext()); Assertions.assertTrue(isChangedField(verdict, "changed"), "a changed token must invalidate the cache"); diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorSessionImplTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorSessionImplTest.java index d59e5edfc2115d..0df3af0bacf59a 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorSessionImplTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorSessionImplTest.java @@ -24,6 +24,7 @@ import org.apache.doris.connector.spi.handle.ConnectorTransaction; import org.apache.doris.nereids.StatementContext; import org.apache.doris.qe.ConnectContext; +import org.apache.doris.qe.SessionVariable; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -141,6 +142,24 @@ public void testBeExecVersionFallsBackToTheEngineConfigOutsideAPlannedStatement( } } + @Test + public void testIcebergInternalPartitionPruningFollowsTheSessionVariable() { + ConnectContext ctx = new ConnectContext(); + ctx.setThreadLocalInfo(); + try { + ConnectorSession original = ConnectorSessionBuilder.from(ctx).build(); + String key = SessionVariable.ENABLE_ICEBERG_INTERNAL_PARTITION_PRUNING; + Assertions.assertEquals("false", original.getSessionProperties().get(key)); + ctx.getSessionVariable().enableIcebergInternalPartitionPruning = true; + Assertions.assertEquals("true", ConnectorSessionBuilder.from(ctx).build() + .getSessionProperties().get(key)); + Assertions.assertEquals("false", original.getSessionProperties().get(key), + "changing the session must not change a captured connector session"); + } finally { + ConnectContext.remove(); + } + } + @Test public void testSessionPropertyOverridesCatalogProperty() { Map catalogProps = new HashMap<>(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorStatementScopeTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorStatementScopeTest.java index cb7777dd108db6..ada9a96ef20a53 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorStatementScopeTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorStatementScopeTest.java @@ -55,6 +55,22 @@ */ public class ConnectorStatementScopeTest { + @Test + public void lookupDoesNotLoadAndRespectsPlanningAndStatementCleanup() { + ConnectorStatementScopeImpl scope = new ConnectorStatementScopeImpl(); + Assertions.assertNull(scope.getIfPresent("missing")); + Object table = scope.computeIfAbsent("table", Object::new); + Object planning = scope.computeIfAbsentForPlanning("planning", Object::new); + Assertions.assertSame(table, scope.getIfPresent("table")); + Assertions.assertSame(planning, scope.getIfPresent("planning")); + scope.endPlanning(); + Assertions.assertNull(scope.getIfPresent("planning")); + Assertions.assertSame(table, scope.getIfPresent("table")); + scope.closeAll(); + Assertions.assertNull(scope.getIfPresent("table")); + Assertions.assertNull(ConnectorStatementScope.NONE.getIfPresent("table")); + } + @Test public void noneNeverMemoizes() { // NONE = the off-context default (offline / no live statement): the loader runs every call, so a diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/metacache/ExternalCatalogMetaCacheTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/metacache/ExternalCatalogMetaCacheTest.java index 7524288f2cb593..174872b738d057 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/metacache/ExternalCatalogMetaCacheTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/metacache/ExternalCatalogMetaCacheTest.java @@ -45,9 +45,48 @@ import java.util.concurrent.Executors; import java.util.concurrent.atomic.AtomicInteger; import java.util.function.Supplier; +import java.util.stream.Collectors; +import java.util.stream.IntStream; public class ExternalCatalogMetaCacheTest { + @Test + public void contextualEntriesPreserveTheirSizeEstimatorAcrossCatalogRuntimes() { + ExecutorService refreshExecutor = Executors.newSingleThreadExecutor(); + AtomicInteger estimates = new AtomicInteger(); + ExternalCatalogMetaCache cache = new ExternalCatalogMetaCache("weighted_test", refreshExecutor) { + { + registerEntry(MetaCacheEntryDef.contextualOnly("large", String.class, LargeValue.class, + CacheSpec.ofWeight(true, CacheSpec.CACHE_NO_TTL, 10, 2_000_000)) + .withSizeEstimator((key, value) -> { + estimates.incrementAndGet(); + return MetaCacheSizeEstimate.complete(800_000); + })); + } + }; + try { + LargeValue value = new LargeValue(); + Assertions.assertEquals(20_000, value.values.size()); + for (long catalogId : new long[] {1L, 2L}) { + cache.initCatalog(catalogId, Map.of("meta.cache.weighted_test.large.max-weight", "2000000")); + MetaCache entry = cache.entry(catalogId, "large", String.class, LargeValue.class); + Assertions.assertSame(value, entry.get("key", key -> value)); + Assertions.assertSame(value, entry.get("key", key -> { + throw new AssertionError("contextual entry must use its own complete size estimate"); + })); + cache.invalidateCatalog(catalogId); + } + Assertions.assertEquals(2, estimates.get()); + } finally { + refreshExecutor.shutdownNow(); + } + } + + private static final class LargeValue { + private final List values = IntStream.range(0, 20_000) + .mapToObj(Integer::toString).collect(Collectors.toList()); + } + @Test public void testEntryRequiresExplicitInit() { ExecutorService refreshExecutor = Executors.newSingleThreadExecutor(); @@ -191,7 +230,7 @@ public void testWideSchemaIsCachedUnderAWeightLimit() { @Test public void testSchemaEstimateIsCompleteLinearAndCountsReusedDisplayColumnsOnce() { MetaCacheSizeEstimator estimator = - ExternalCatalogMetaCache.sizeEstimator(SchemaCacheValue.class); + MetaCacheEntryDef.defaultSizeEstimator(SchemaCacheValue.class); SchemaCacheKey key = new SchemaCacheKey(NameMapping.createForTest(3L, "db1", "wide")); MetaCacheSizeEstimate half = estimator.estimate(key, new SchemaCacheValue(wideColumns(2_000, 500))); List columns = wideColumns(4_000, 1_000); @@ -210,7 +249,7 @@ public void testSchemaEstimateIsCompleteLinearAndCountsReusedDisplayColumnsOnce( // A plugin table's display columns reuse its column objects; they must not be weighed twice. MUTATION: // estimating every element of every column list -> about twice the weight -> red. MetaCacheSizeEstimator pluginEstimator = - ExternalCatalogMetaCache.sizeEstimator(PluginDrivenSchemaCacheValue.class); + MetaCacheEntryDef.defaultSizeEstimator(PluginDrivenSchemaCacheValue.class); long sameList = pluginEstimator.estimate(key, new PluginDrivenSchemaCacheValue(columns, Collections.emptyList(), Collections.emptyList(), null, null, null, columns)).getBytes(); long copiedList = pluginEstimator.estimate(key, new PluginDrivenSchemaCacheValue(columns, diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/mvcc/MvccReadSelectorTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/mvcc/MvccReadSelectorTest.java new file mode 100644 index 00000000000000..efd64f4b7c4acb --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/mvcc/MvccReadSelectorTest.java @@ -0,0 +1,76 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource.mvcc; + +import org.apache.doris.analysis.TableScanParams; +import org.apache.doris.analysis.TableSnapshot; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Optional; + +public class MvccReadSelectorTest { + @Test + public void historicalSelectorIsUnaffectedByParserOrGetterMutations() { + TableSnapshot original = TableSnapshot.versionOf("100"); + MvccReadSelector selector = MvccReadSelector.of(Optional.of(original), Optional.empty()); + original.setValue("200"); + selector.getTableSnapshot().orElseThrow().setValue("300"); + Assertions.assertEquals("100", selector.getTableSnapshot().orElseThrow().getValue()); + Assertions.assertEquals(MvccReadSelector.of(Optional.of(TableSnapshot.versionOf("100")), Optional.empty()), + selector); + Assertions.assertNotEquals(MvccReadSelector.of(Optional.of(TableSnapshot.timeOf("100")), Optional.empty()), + selector); + } + + @Test + public void dynamicParametersAreResolvedAgainForEachUse() { + TableScanParams original = new TableScanParams("options", Map.of("branch", "dev"), Collections.emptyList()); + original.reuseResolvedMapParams(Map.of("snapshot-id", "100")); + MvccReadSelector selector = MvccReadSelector.of(Optional.empty(), Optional.of(original)); + TableScanParams first = selector.getScanParams().orElseThrow(); + Assertions.assertTrue(first.getResolvedMapParams().isEmpty()); + first.reuseResolvedMapParams(Map.of("snapshot-id", "200")); + Assertions.assertTrue(selector.getScanParams().orElseThrow().getResolvedMapParams().isEmpty()); + Assertions.assertEquals(Map.of("branch", "dev"), selector.getScanParams().orElseThrow().getMapParams()); + } + + @Test + public void canonicalKeysPreserveMapEntryBoundariesAndIgnoreMapOrder() { + Map ordered = new LinkedHashMap<>(); + ordered.put("x", "1"); + ordered.put("y", "2"); + Map reversed = new LinkedHashMap<>(); + reversed.put("y", "2"); + reversed.put("x", "1"); + MvccReadSelector a = MvccReadSelector.of(Optional.empty(), Optional.of( + new TableScanParams("options", ordered, Collections.emptyList()))); + MvccReadSelector b = MvccReadSelector.of(Optional.empty(), Optional.of( + new TableScanParams("options", reversed, Collections.emptyList()))); + MvccReadSelector collision = MvccReadSelector.of(Optional.empty(), Optional.of( + new TableScanParams("options", Map.of("x", "1, y=2"), Collections.emptyList()))); + Assertions.assertEquals(a, b); + Assertions.assertEquals(a.hashCode(), b.hashCode()); + Assertions.assertNotEquals(a, collision); + Assertions.assertSame(MvccReadSelector.LATEST, MvccReadSelector.of(Optional.empty(), Optional.empty())); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTableTest.java index f232aa9f0cca4d..062af920cf97ff 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTableTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTableTest.java @@ -19,8 +19,12 @@ import org.apache.doris.analysis.TableScanParams; import org.apache.doris.analysis.TableSnapshot; +import org.apache.doris.blockrule.SqlBlockRule; +import org.apache.doris.blockrule.SqlBlockRuleMgr; import org.apache.doris.catalog.Column; +import org.apache.doris.catalog.Env; import org.apache.doris.catalog.ListPartitionItem; +import org.apache.doris.catalog.MTMV; import org.apache.doris.catalog.PartitionItem; import org.apache.doris.catalog.PartitionKey; import org.apache.doris.catalog.PartitionType; @@ -40,11 +44,13 @@ import org.apache.doris.connector.spi.mvcc.ConnectorMvccPartition; import org.apache.doris.connector.spi.mvcc.ConnectorMvccPartitionView; import org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot; +import org.apache.doris.connector.spi.mvcc.ConnectorSqlCacheVersion; import org.apache.doris.connector.spi.mvcc.ConnectorTableFreshness; import org.apache.doris.connector.spi.mvcc.ConnectorTimeTravelSpec; import org.apache.doris.connector.spi.scan.ConnectorPartitionValues; import org.apache.doris.datasource.ExternalCatalog; import org.apache.doris.datasource.ExternalDatabase; +import org.apache.doris.datasource.InternalCatalog; import org.apache.doris.datasource.SchemaCacheValue; import org.apache.doris.datasource.SessionContext; import org.apache.doris.datasource.plugin.PluginDrivenExternalCatalog; @@ -85,6 +91,11 @@ import java.util.Optional; import java.util.OptionalLong; import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.function.Function; import java.util.stream.Collectors; /** @@ -159,6 +170,167 @@ public void sourceColumnPruningUsesTheRetainedPartitionView() { Assertions.assertTrue(range.table.supportInternalPartitionPruned(rangePin)); } + @Test + public void sqlCacheVersionUsesTheSuppliedScalarPinWithoutMaterializingPartitions() { + Fixture f = Fixture.partitioned(); + ConnectorMvccSnapshot scalar = ConnectorMvccSnapshot.builder().snapshotId(7L).build(); + PluginDrivenMvccSnapshot pin = new PluginDrivenMvccSnapshot(scalar, + Collections.emptyMap(), Collections.emptyMap()); + ConnectorSqlCacheVersion version = new ConnectorSqlCacheVersion("uuid:7:schema1:metadata1", 1000L); + Mockito.when(f.metadata.getSqlCacheVersion(f.session, f.pinnedHandle)).thenReturn(Optional.of(version)); + Assertions.assertSame(version, f.table.getSqlCacheVersion(Optional.of(pin)).orElseThrow()); + Mockito.verify(f.metadata).applySnapshot(f.session, f.handle, scalar); + Mockito.verify(f.metadata, Mockito.never()).beginQuerySnapshot(Mockito.any(), Mockito.any()); + Mockito.verify(f.metadata, Mockito.never()).getMvccPartitionView(Mockito.any(), Mockito.any()); + Mockito.verify(f.metadata, Mockito.never()).listPartitions(Mockito.any(), Mockito.any(), Mockito.any()); + } + + @Test + public void sqlCacheLookupResolvesLatestScalarVersionWithoutPartitionView() { + Fixture f = Fixture.partitioned(); + ConnectorSqlCacheVersion version = new ConnectorSqlCacheVersion("uuid:latest:schema1:metadata1", 1000L); + Mockito.when(f.metadata.getSqlCacheVersion(f.session, f.pinnedHandle)).thenReturn(Optional.of(version)); + Assertions.assertSame(version, f.table.getSqlCacheVersion(Optional.empty()).orElseThrow()); + Mockito.verify(f.metadata).beginQuerySnapshot(f.session, f.handle); + Mockito.verify(f.metadata, Mockito.never()).getMvccPartitionView(Mockito.any(), Mockito.any()); + Mockito.verify(f.metadata, Mockito.never()).listPartitions(Mockito.any(), Mockito.any(), Mockito.any()); + } + + private static ConnectContext nativeContext(Fixture f, SqlBlockRuleMgr rules) { + ConnectContext ctx = new ConnectContext(); + Env env = Mockito.mock(Env.class); + Mockito.when(env.getSqlBlockRuleMgr()).thenReturn(rules); + InternalCatalog internal = Mockito.mock(InternalCatalog.class); + Mockito.when(internal.getName()).thenReturn(InternalCatalog.INTERNAL_CATALOG_NAME); + Mockito.when(env.getInternalCatalog()).thenReturn(internal); + ctx.setEnv(env); + ctx.setStatementContext(new StatementContext(ctx, null)); + ctx.setThreadLocalInfo(); + Mockito.when(f.metadata.supportsNativePartitionPlanning(f.session, f.pinnedHandle)).thenReturn(true); + return ctx; + } + + @Test + public void ordinaryNativePlanningDoesNotEnumeratePartitions() { + Fixture f = Fixture.partitioned(); + nativeContext(f, new SqlBlockRuleMgr()); + PluginDrivenMvccSnapshot pin = (PluginDrivenMvccSnapshot) + f.table.loadSnapshot(Optional.empty(), Optional.empty()); + Assertions.assertTrue(pin.hasDeferredPartitions()); + Assertions.assertEquals(PINNED_SNAPSHOT_ID, pin.getConnectorSnapshot().getSnapshotId()); + Assertions.assertSame(f.latestCacheValue, pin.getPinnedSchema()); + Assertions.assertEquals(1, f.table.getPartitionColumns(Optional.of(pin)).size()); + Assertions.assertSame(LogicalFileScan.SelectedPartitions.NATIVE, + f.table.initSelectedPartitions(Optional.of(pin))); + Mockito.verify(f.metadata, Mockito.never()).getMvccPartitionView(Mockito.any(), Mockito.any()); + Mockito.verify(f.metadata, Mockito.never()).listPartitions(Mockito.any(), Mockito.any(), Mockito.any()); + } + + @Test + public void nativeSelectionKeepsTheFilterAndSkipsInternalPruning() { + Fixture f = Fixture.partitioned(); + ConnectContext ctx = nativeContext(f, new SqlBlockRuleMgr()); + MvccSnapshot pin = f.table.loadSnapshot(Optional.empty(), Optional.empty()); + ctx.getStatementContext().setSnapshot(new MvccTableInfo(f.table), pin); + SlotReference slot = new SlotReference("dt", DateV2Type.INSTANCE); + LogicalFileScan scan = new LogicalFileScan(new RelationId(11), f.table, + Collections.singletonList("db"), Collections.emptyList(), Optional.empty(), + Optional.empty(), Optional.empty(), Optional.of(Collections.singletonList(slot))); + LogicalFilter filter = new LogicalFilter<>( + Collections.singleton(new EqualTo(slot, new DateV2Literal(2024, 1, 1))), scan); + Assertions.assertSame(LogicalFileScan.SelectedPartitions.NATIVE, scan.getSelectedPartitions()); + // The rewrite framework checks this pattern before invoking transform. + Assertions.assertFalse(new PruneFileScanPartition().build().getPattern().matchPlanTree(filter)); + Assertions.assertEquals(Collections.singleton(new EqualTo(slot, new DateV2Literal(2024, 1, 1))), + filter.getConjuncts()); + Mockito.verify(f.metadata, Mockito.never()).listPartitions(Mockito.any(), Mockito.any(), Mockito.any()); + } + + @Test + public void deferredViewLoadsOnceOnThePinnedHandleAcrossThreads() throws Exception { + Fixture f = Fixture.partitioned(); + nativeContext(f, new SqlBlockRuleMgr()); + PluginDrivenMvccSnapshot pin = (PluginDrivenMvccSnapshot) + f.table.loadSnapshot(Optional.empty(), Optional.empty()); + // A later latest generation must not change what the deferred loader enumerates. + Mockito.when(f.metadata.beginQuerySnapshot(f.session, f.handle)).thenReturn(Optional.of( + ConnectorMvccSnapshot.builder().snapshotId(PINNED_SNAPSHOT_ID + 1).build())); + Mockito.when(f.metadata.listPartitions(Mockito.eq(f.session), Mockito.eq(f.handle), Mockito.any())) + .thenReturn(Collections.singletonList(cpi("dt=2025-01-01", 1L))); + ExecutorService pool = Executors.newFixedThreadPool(4); + try { + List>> results = new ArrayList<>(); + for (int i = 0; i < 8; i++) { + results.add(pool.submit(pin::getNameToPartitionItem)); + } + Map first = results.get(0).get(); + Assertions.assertEquals(2, first.size()); + for (Future> result : results) { + Assertions.assertSame(first, result.get()); + } + Assertions.assertEquals(PINNED_SNAPSHOT_ID, pin.getConnectorSnapshot().getSnapshotId()); + Mockito.verify(f.metadata, Mockito.times(1)).listPartitions( + Mockito.eq(f.session), Mockito.eq(f.pinnedHandle), Mockito.any()); + Mockito.verify(f.metadata, Mockito.never()).listPartitions( + Mockito.eq(f.session), Mockito.eq(f.handle), Mockito.any()); + } finally { + pool.shutdownNow(); + } + } + + @Test + public void deferredViewFailureDoesNotPublishPartialData() { + Fixture f = Fixture.partitioned(); + nativeContext(f, new SqlBlockRuleMgr()); + PluginDrivenMvccSnapshot pin = (PluginDrivenMvccSnapshot) + f.table.loadSnapshot(Optional.empty(), Optional.empty()); + Mockito.when(f.metadata.listPartitions(Mockito.eq(f.session), Mockito.eq(f.pinnedHandle), Mockito.any())) + .thenThrow(new IllegalStateException("manifest unavailable")) + .thenReturn(Collections.singletonList(cpi("dt=2024-01-01", TS_2024_01_01))); + Assertions.assertThrows(IllegalStateException.class, pin::getNameToPartitionItem); + Assertions.assertEquals(1, pin.getNameToPartitionItem().size()); + } + + @Test + public void candidateMvKeepsTheExistingPartitionViewAndPruning() { + Fixture f = Fixture.partitioned(); + ConnectContext ctx = nativeContext(f, new SqlBlockRuleMgr()); + ctx.getStatementContext().getCandidateMTMVs().add(Mockito.mock(MTMV.class)); + PluginDrivenMvccSnapshot pin = (PluginDrivenMvccSnapshot) + f.table.loadSnapshot(Optional.empty(), Optional.empty()); + Assertions.assertFalse(pin.hasDeferredPartitions()); + Assertions.assertEquals(2, pin.getNameToPartitionItem().size()); + Assertions.assertFalse(f.table.initSelectedPartitions(Optional.of(pin)).isNativePlanning()); + } + + @Test + public void partitionRuleKeepsExactPartitionMetadata() { + Fixture f = Fixture.partitioned(); + SqlBlockRuleMgr rules = new SqlBlockRuleMgr(); + rules.getNameToSqlBlockRuleMap().put("part_limit", new SqlBlockRule( + "part_limit", "NULL", "NULL", 1L, 0L, 0L, false, true, true)); + nativeContext(f, rules); + PluginDrivenMvccSnapshot pin = (PluginDrivenMvccSnapshot) + f.table.loadSnapshot(Optional.empty(), Optional.empty()); + Assertions.assertFalse(pin.hasDeferredPartitions()); + Assertions.assertEquals(2, f.table.initSelectedPartitions(Optional.of(pin)).totalPartitionNum); + } + + @Test + public void explicitTimeTravelDefersOnlyItsOwnPinnedListing() { + Fixture f = Fixture.timeTravel(); + nativeContext(f, new SqlBlockRuleMgr()); + Mockito.when(f.metadata.listsPartitionsAtSnapshot(f.session, f.pinnedHandle)).thenReturn(true); + PluginDrivenMvccSnapshot pin = (PluginDrivenMvccSnapshot) f.table.loadSnapshot( + Optional.of(TableSnapshot.versionOf("7")), Optional.empty()); + Assertions.assertTrue(pin.hasDeferredPartitions()); + Assertions.assertEquals(Fixture.TT_SCHEMA_ID, pin.getSchemaId()); + Assertions.assertEquals(Type.STRING, f.table.getPartitionColumns(Optional.of(pin)).get(0).getType()); + Mockito.verify(f.metadata, Mockito.never()).listPartitions(Mockito.any(), Mockito.any(), Mockito.any()); + Assertions.assertEquals(2, pin.getNameToPartitionItem().size()); + Mockito.verify(f.metadata).listPartitions(Mockito.eq(f.session), Mockito.eq(f.pinnedHandle), Mockito.any()); + } + @AfterEach public void cleanup() { ConnectContext.remove(); @@ -680,6 +852,27 @@ public void testLatestRangeUsesPinnedPartitionTypeAndArity() { Assertions.assertEquals("10", item.getItems().lowerEndpoint().getKeys().get(0).getStringValue()); } + @Test + public void testPartitionViewReuseStillChecksConnectorAndPinsSnapshot() { + Fixture f = pinnedPartitionFixture(); + List parts = Collections.singletonList(cpi("key=10", TS_2024_01_01)); + Mockito.when(f.metadata.listPartitions(Mockito.eq(f.session), Mockito.eq(f.pinnedHandle), Mockito.any())) + .thenReturn(parts); + PluginDrivenMvccSnapshot first = (PluginDrivenMvccSnapshot) + f.table.loadSnapshot(Optional.empty(), Optional.empty()); + PluginDrivenMvccSnapshot second = (PluginDrivenMvccSnapshot) + f.table.loadSnapshot(Optional.empty(), Optional.empty()); + Assertions.assertNotSame(first, second); + Assertions.assertSame(first.getNameToPartitionItem(), second.getNameToPartitionItem()); + Mockito.verify(f.metadata, Mockito.times(2)).beginQuerySnapshot(f.session, f.handle); + Mockito.verify(f.metadata, Mockito.times(2)) + .listPartitions(Mockito.eq(f.session), Mockito.eq(f.pinnedHandle), Mockito.any()); + Mockito.when(f.metadata.listPartitions(Mockito.eq(f.session), Mockito.eq(f.pinnedHandle), Mockito.any())) + .thenThrow(new IllegalStateException("listing denied")); + Assertions.assertThrows(IllegalStateException.class, + () -> f.table.loadSnapshot(Optional.empty(), Optional.empty())); + } + @Test public void testLatestListUsesPinnedPartitionTypeAndArity() { checkLatestPinnedList(false); @@ -1597,6 +1790,27 @@ public void testAbsentRangeViewKeepsLegacyListPath() throws AnalysisException { Mockito.verify(f.metadata).listPartitions(Mockito.any(), Mockito.any(), Mockito.any()); } + @Test + public void testVersionedPartitionViewSkipsListingOnHitAndKeepsOldPin() throws AnalysisException { + Fixture f = Fixture.partitioned(); + Mockito.when(f.metadata.getPartitionViewVersion(f.session, f.pinnedHandle)) + .thenReturn(Optional.of("snapshot-1/schema-1/spec-1")); + Optional first = Optional.of(f.table.loadSnapshot(Optional.empty(), Optional.empty())); + Optional again = Optional.of(f.table.loadSnapshot(Optional.empty(), Optional.empty())); + Assertions.assertSame(f.table.getNameToPartitionItems(first), f.table.getNameToPartitionItems(again)); + Mockito.verify(f.metadata, Mockito.times(1)).listPartitions(Mockito.any(), Mockito.any(), Mockito.any()); + + Mockito.when(f.metadata.getPartitionViewVersion(f.session, f.pinnedHandle)) + .thenReturn(Optional.of("snapshot-2/schema-1/spec-1")); + Mockito.when(f.metadata.listPartitions(Mockito.eq(f.session), Mockito.eq(f.pinnedHandle), Mockito.any())) + .thenReturn(Collections.singletonList(cpi("dt=2024-02-01", 20L))); + Optional next = Optional.of(f.table.loadSnapshot(Optional.empty(), Optional.empty())); + Assertions.assertEquals(Collections.singleton("dt=2024-02-01"), + f.table.getNameToPartitionItems(next).keySet()); + Assertions.assertFalse(f.table.getNameToPartitionItems(first).containsKey("dt=2024-02-01")); + Mockito.verify(f.metadata, Mockito.times(2)).listPartitions(Mockito.any(), Mockito.any(), Mockito.any()); + } + // ==================== fixtures / helpers ==================== private static ConnectorPartitionInfo cpi(String name, long lastModifiedMillis) { @@ -1834,6 +2048,15 @@ private static Fixture build(List partitions, boolean ti PluginDrivenMvccExternalTable table = new PluginDrivenMvccExternalTable(1L, "tbl", "REMOTE_TBL", catalog, db) { + private final Map views = + new ConcurrentHashMap<>(); + + @Override + PluginDrivenPartitionView cachedPartitionView(PluginDrivenPartitionView.Key key, + Function loader) { + return views.computeIfAbsent(key, loader); + } + @Override protected synchronized void makeSureInitialized() { // no-op: skip Env-backed catalog/db init diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/mvcc/PluginDrivenPartitionViewTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/mvcc/PluginDrivenPartitionViewTest.java new file mode 100644 index 00000000000000..28582ec048ff0e --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/mvcc/PluginDrivenPartitionViewTest.java @@ -0,0 +1,407 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource.mvcc; + +import org.apache.doris.analysis.IntLiteral; +import org.apache.doris.catalog.Env; +import org.apache.doris.catalog.ListPartitionItem; +import org.apache.doris.catalog.PartitionKey; +import org.apache.doris.catalog.PrimitiveType; +import org.apache.doris.catalog.RangePartitionItem; +import org.apache.doris.catalog.ScalarType; +import org.apache.doris.catalog.Type; +import org.apache.doris.common.cache.NereidsSortedPartitionsCacheManager; +import org.apache.doris.connector.cache.CacheSpec; +import org.apache.doris.connector.cache.CatalogMetaCache; +import org.apache.doris.connector.cache.MetaCache; +import org.apache.doris.connector.cache.MetaCacheDefinition; +import org.apache.doris.connector.cache.ScopePath; +import org.apache.doris.connector.spi.ConnectorPartitionInfo; +import org.apache.doris.connector.spi.ConnectorType; +import org.apache.doris.connector.spi.mvcc.ConnectorMvccPartition; +import org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot; +import org.apache.doris.datasource.CatalogMgr; +import org.apache.doris.datasource.ExternalCatalog; +import org.apache.doris.datasource.ExternalDatabase; +import org.apache.doris.datasource.ExternalMetaCacheMgr; +import org.apache.doris.nereids.trees.plans.logical.LogicalFileScan.SelectedPartitions; +import org.apache.doris.qe.ConnectContext; + +import com.google.common.collect.Range; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.MockedStatic; +import org.mockito.Mockito; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.function.Function; + +class PluginDrivenPartitionViewTest { + private final CatalogMetaCache owner = CatalogMetaCache.unmanaged(); + private final MetaCache cache = owner.create( + MetaCacheDefinition.builder( + "partition_view", CacheSpec.of(true, 600, 100), key -> ScopePath.table("db", "tbl")).build()); + + private PluginDrivenMvccExternalTable table() { + return new PluginDrivenMvccExternalTable() { + @Override + PluginDrivenPartitionView cachedPartitionView(PluginDrivenPartitionView.Key key, + Function loader) { + return cache.get(key, loader); + } + }; + } + + @AfterEach + void close() { + owner.close(); + } + + private ConnectorPartitionInfo partition(String value, long modified, boolean isNull, + List types) { + return new ConnectorPartitionInfo("p=" + value, Collections.singletonMap("p", value), + Collections.emptyMap(), -1, -1, modified, -1, + Collections.singletonList(value), Collections.singletonList(isNull), types); + } + + private ConnectorPartitionInfo partition(String value, long modified) { + return partition(value, modified, false, Collections.emptyList()); + } + + @Test + void mixedSpecDisplayPartitionsDoNotBuildAnOrderedIndex() { + ConnectorPartitionInfo one = new ConnectorPartitionInfo("a=1", Collections.emptyMap(), + Collections.emptyMap(), -1, -1, 1, -1, List.of("1"), List.of(false), + List.of(ConnectorType.of("INT"))); + ConnectorPartitionInfo two = new ConnectorPartitionInfo("a=1/b=1", Collections.emptyMap(), + Collections.emptyMap(), -1, -1, 1, -1, List.of("1", "1"), List.of(false, false), + List.of(ConnectorType.of("INT"), ConnectorType.of("INT"))); + PluginDrivenPartitionView view = table().partitionView(List.of(one, two), List.of(Type.INT, Type.INT)); + Assertions.assertEquals(2, view.getItems().size()); + Assertions.assertEquals(2, view.getFreshness().size()); + Assertions.assertTrue(view.getSortedRanges().isEmpty()); + Assertions.assertTrue(table().partitionView(List.of(one), List.of(Type.INT)).getSortedRanges().isPresent()); + } + + @Test + void rangeViewAccountingDoesNotTraverseExpressionSuppliers() { + PartitionKey lower = new PartitionKey(); + lower.pushColumn(new IntLiteral(1), PrimitiveType.BIGINT); + PartitionKey upper = new PartitionKey(); + upper.pushColumn(new IntLiteral(2), PrimitiveType.BIGINT); + PluginDrivenPartitionView view = new PluginDrivenPartitionView( + Map.of("p", new RangePartitionItem(Range.closedOpen(lower, upper))), + Map.of("p", 1L)); + Assertions.assertTrue(view.getSortedRanges().isPresent()); + Assertions.assertTrue(PluginDrivenPartitionView.estimateSize( + PluginDrivenPartitionView.Key.versioned(1, "db", "tbl", "v1", List.of(Type.BIGINT), true), + view).isComplete()); + } + + @Test + void weightedCacheAdmitsViewsBeyondTheReflectiveTraversalBudget() { + List parts = new ArrayList<>(); + for (int i = 0; i < 12000; i++) { + parts.add(partition(Integer.toString(i), i)); + } + PluginDrivenPartitionView view = table().partitionView(parts, List.of(Type.INT)); + PluginDrivenPartitionView.Key key = PluginDrivenPartitionView.Key.list(1, "db", "tbl", parts, + List.of(Type.INT)); + org.apache.doris.connector.cache.MetaCacheSizeEstimate estimate = + PluginDrivenPartitionView.estimateSize(key, view); + Assertions.assertTrue(estimate.isComplete()); + Assertions.assertTrue(estimate.getBytes() > 12000 * 64L); + MetaCache weighted = owner.create( + MetaCacheDefinition.builder( + "weighted_partition_view", CacheSpec.ofWeight(true, 600, 100, estimate.getBytes() * 2), + k -> ScopePath.table("db", "tbl")) + .sizeEstimator(PluginDrivenPartitionView::estimateSize).build()); + Assertions.assertSame(view, weighted.get(key, k -> view)); + Assertions.assertSame(view, weighted.get(key, k -> { + throw new AssertionError("large partition view must remain cached"); + })); + owner.invalidateTable("db", "tbl"); + Assertions.assertNull(weighted.getIfPresent(key)); + } + + @Test + void versionedKeysSeparateGenerationsAndConversionParameters() { + PluginDrivenPartitionView.Key first = PluginDrivenPartitionView.Key.versioned( + 1, "db", "tbl", "snapshot-1", List.of(Type.INT), false); + Assertions.assertEquals(first, PluginDrivenPartitionView.Key.versioned( + 1, "db", "tbl", "snapshot-1", List.of(Type.INT), false)); + Assertions.assertNotEquals(first, PluginDrivenPartitionView.Key.versioned( + 1, "db", "tbl", "snapshot-2", List.of(Type.INT), false)); + Assertions.assertNotEquals(first, PluginDrivenPartitionView.Key.versioned( + 1, "db", "tbl", "snapshot-1", List.of(Type.BIGINT), false)); + Assertions.assertNotEquals(first, PluginDrivenPartitionView.Key.versioned( + 1, "db", "tbl", "snapshot-1", List.of(Type.INT), true)); + Assertions.assertNotEquals(first, PluginDrivenPartitionView.Key.versioned( + 2, "db", "tbl", "snapshot-1", List.of(Type.INT), false)); + PluginDrivenPartitionView a = cache.get(first, + key -> new PluginDrivenPartitionView(Collections.emptyMap(), Collections.emptyMap())); + Assertions.assertSame(a, cache.get(PluginDrivenPartitionView.Key.versioned( + 1, "db", "tbl", "snapshot-1", List.of(Type.INT), false), key -> { + throw new AssertionError("same version must not load partitions"); + })); + owner.invalidateTable("db", "tbl"); + Assertions.assertNotSame(a, cache.get(first, + key -> new PluginDrivenPartitionView(Collections.emptyMap(), Collections.emptyMap()))); + } + + @Test + void reuseRecordsButNotChangedListing() { + PluginDrivenMvccExternalTable table = table(); + List parts = new ArrayList<>(); + parts.add(partition("1", 10)); + PluginDrivenPartitionView first = table.partitionView(parts, List.of(Type.INT)); + Assertions.assertSame(first, table.partitionView(new ArrayList<>(parts), List.of(Type.INT))); + parts.add(partition("2", 20)); + PluginDrivenPartitionView added = table.partitionView(parts, List.of(Type.INT)); + Assertions.assertEquals(1, first.getItems().size()); + Assertions.assertEquals(2, added.getItems().size()); + parts.remove(0); + PluginDrivenPartitionView removed = table.partitionView(parts, List.of(Type.INT)); + Assertions.assertEquals(Collections.singleton("p=2"), removed.getItems().keySet()); + // An uncached connector can return new objects for unchanged conversion inputs. + parts.set(0, partition("2", 20)); + Assertions.assertSame(removed, table.partitionView(parts, List.of(Type.INT))); + } + + @Test + void schemaFreshnessNullAndTransformChangesMiss() { + PluginDrivenMvccExternalTable table = table(); + List parts = List.of(partition("1", 10)); + PluginDrivenPartitionView integer = table.partitionView(parts, List.of(Type.INT)); + PluginDrivenPartitionView string = table.partitionView(parts, List.of(Type.STRING)); + Assertions.assertNotSame(integer, string); + Assertions.assertEquals(Type.STRING, ((ListPartitionItem) string.getItems().get("p=1")) + .getItems().get(0).getKeys().get(0).getType()); + PluginDrivenPartitionView modified = table.partitionView(List.of(partition("1", 30)), List.of(Type.INT)); + Assertions.assertEquals(30L, modified.getFreshness().get("p=1")); + Assertions.assertEquals(10L, integer.getFreshness().get("p=1")); + PluginDrivenPartitionView nullValue = table.partitionView( + List.of(partition("1", 30, true, Collections.emptyList())), List.of(Type.INT)); + Assertions.assertTrue(((ListPartitionItem) nullValue.getItems().get("p=1")) + .getItems().get(0).getKeys().get(0).isNullLiteral()); + PluginDrivenPartitionView transformed = table.partitionView( + List.of(partition("1", 30, false, List.of(ConnectorType.of("INT")))), List.of(Type.STRING)); + Assertions.assertEquals(Type.INT, ((ListPartitionItem) transformed.getItems().get("p=1")) + .getItems().get(0).getKeys().get(0).getType()); + } + + @Test + void timestampWithoutZoneDoesNotReuseAnotherSessionsInterpretation() { + ConnectContext previous = ConnectContext.get(); + ConnectContext context = new ConnectContext(); + context.setThreadLocalInfo(); + try { + PluginDrivenMvccExternalTable table = table(); + String value = "2024-01-01 08:00:00"; + List parts = List.of(partition(value, 10)); + List types = List.of(ScalarType.createTimeStampTzType(6)); + Assertions.assertEquals(types, PluginDrivenPartitionView.Key.list( + 1, "db", "tbl", parts, types).frozenTypes()); + context.getSessionVariable().setTimeZone("UTC"); + PluginDrivenPartitionView utc = table.partitionView(parts, types); + context.getSessionVariable().setTimeZone("Asia/Shanghai"); + PluginDrivenPartitionView shanghai = table.partitionView(parts, types); + Assertions.assertNotSame(utc, shanghai); + Assertions.assertNotEquals(utc.getItems().get("p=" + value), shanghai.getItems().get("p=" + value)); + Assertions.assertSame(shanghai, table.partitionView(parts, types)); + } finally { + ConnectContext.remove(); + if (previous != null) { + previous.setThreadLocalInfo(); + } + } + } + + @Test + void mutableSchemaTypeCannotChangeCachedSignature() { + PluginDrivenMvccExternalTable table = table(); + List parts = List.of(partition("a", 10)); + ScalarType type = ScalarType.createVarcharType(10); + PluginDrivenPartitionView first = table.partitionView(parts, List.of(type)); + List binary = List.of(ScalarType.createVarbinaryType(23)); + Assertions.assertEquals(binary, PluginDrivenPartitionView.Key.list( + 1, "db", "tbl", parts, binary).frozenTypes()); + type.setLength(20); + Assertions.assertNotSame(first, table.partitionView(parts, List.of(type))); + } + + @Test + void refreshAndHistoricalPinsRemainIndependent() { + PluginDrivenMvccExternalTable table = table(); + List oldParts = List.of(partition("1", 10)); + PluginDrivenPartitionView old = table.partitionView(oldParts, List.of(Type.INT)); + PluginDrivenMvccSnapshot oldPin = new PluginDrivenMvccSnapshot( + ConnectorMvccSnapshot.builder().snapshotId(1).build(), old, null); + owner.invalidateTable("db", "tbl"); + Assertions.assertNotSame(old, table.partitionView(oldParts, List.of(Type.INT))); + PluginDrivenPartitionView latest = table.partitionView(List.of(partition("2", 20)), List.of(Type.INT)); + PluginDrivenMvccSnapshot latestPin = new PluginDrivenMvccSnapshot( + ConnectorMvccSnapshot.builder().snapshotId(2).build(), latest, null); + Assertions.assertEquals(Collections.singleton("p=1"), oldPin.getNameToPartitionItem().keySet()); + Assertions.assertEquals(Collections.singleton("p=2"), latestPin.getNameToPartitionItem().keySet()); + Assertions.assertEquals(old.getItems(), table.partitionView(oldParts, List.of(Type.INT)).getItems()); + Assertions.assertSame(old.getSortedRanges().get(), oldPin.getPartitionView().getSortedRanges().get()); + Assertions.assertSame(old.getItems().get("p=1"), old.getSortedRanges().get().sortedPartitions.get(0).partitionItem); + Assertions.assertSame(old.getItems(), oldPin.getNameToPartitionItem()); + Assertions.assertSame(old.getFreshness(), oldPin.getNameToLastModifiedMillis()); + SelectedPartitions selected = new SelectedPartitions(1, oldPin.getNameToPartitionItem(), false); + Assertions.assertSame(old.getItems(), selected.selectedPartitions); + Assertions.assertThrows(UnsupportedOperationException.class, () -> oldPin.getNameToPartitionItem().clear()); + } + + @Test + void concurrentGenerationsNeverReturnAnotherQueriesPartitions() { + PluginDrivenMvccExternalTable table = table(); + List> futures = new ArrayList<>(); + for (int i = 0; i < 8; i++) { + String value = Integer.toString(i); + futures.add(CompletableFuture.runAsync(() -> { + List parts = List.of(partition(value, 10)); + for (int j = 0; j < 100; j++) { + PluginDrivenPartitionView view = table.partitionView(parts, List.of(Type.INT)); + Assertions.assertEquals(Collections.singleton("p=" + value), view.getItems().keySet()); + if (j % 10 == 0) { + owner.invalidateTable("db", "tbl"); + } + } + })); + } + futures.forEach(CompletableFuture::join); + } + + @Test + void refreshDuringLoadCannotPublishOldView() throws Exception { + PluginDrivenMvccExternalTable table = table(); + List parts = List.of(partition("1", 10)); + PluginDrivenPartitionView old = table.partitionView(parts, List.of(Type.INT)); + PluginDrivenPartitionView.Key key = PluginDrivenPartitionView.Key.list(0, null, null, parts, List.of(Type.INT)); + owner.invalidateTable("db", "tbl"); + CountDownLatch entered = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + CompletableFuture loading = CompletableFuture.supplyAsync(() -> cache.get(key, k -> { + entered.countDown(); + try { + Assertions.assertTrue(release.await(10, TimeUnit.SECONDS)); + } catch (InterruptedException e) { + throw new RuntimeException(e); + } + return old; + })); + try { + Assertions.assertTrue(entered.await(10, TimeUnit.SECONDS)); + owner.invalidateTable("db", "tbl"); + } finally { + release.countDown(); + } + Assertions.assertSame(old, loading.get(10, TimeUnit.SECONDS)); + Assertions.assertNotSame(old, cache.get(key, k -> new PluginDrivenPartitionView( + old.getItems(), old.getFreshness()))); + Assertions.assertEquals("p=1", old.getSortedRanges().get().sortedPartitions.get(0).id); + } + + @Test + void disabledCacheStillPublishesConsistentMapAndIndex() { + MetaCache disabled = owner.create( + MetaCacheDefinition.builder( + "disabled", CacheSpec.of(false, 600, 100), key -> ScopePath.table("db", "tbl")).build()); + PluginDrivenMvccExternalTable table = new PluginDrivenMvccExternalTable() { + @Override + PluginDrivenPartitionView cachedPartitionView(PluginDrivenPartitionView.Key key, + Function loader) { + return disabled.get(key, loader); + } + }; + List parts = List.of(partition("1", 10)); + PluginDrivenPartitionView first = table.partitionView(parts, List.of(Type.INT)); + PluginDrivenPartitionView second = table.partitionView(parts, List.of(Type.INT)); + Assertions.assertNotSame(first, second); + Assertions.assertSame(second.getItems().get("p=1"), second.getSortedRanges().get().sortedPartitions.get(0).partitionItem); + } + + @Test + void rangeKeysFreezeBoundsAndIncludeSchemaAndTableIdentity() { + List lower = new ArrayList<>(List.of("1")); + ConnectorMvccPartition partition = new ConnectorMvccPartition("p", lower, List.of("2"), 10); + PluginDrivenPartitionView.Key first = PluginDrivenPartitionView.Key.range( + 1, "db", "tbl", List.of(partition), List.of(Type.INT)); + Assertions.assertEquals(first, PluginDrivenPartitionView.Key.range( + 1, "db", "tbl", List.of(partition), List.of(Type.INT))); + lower.set(0, "0"); + Assertions.assertNotEquals(first, PluginDrivenPartitionView.Key.range( + 1, "db", "tbl", List.of(partition), List.of(Type.INT))); + Assertions.assertEquals(List.of("1"), first.rangePartitions().get(0).getLowerBound()); + Assertions.assertNotEquals(first, PluginDrivenPartitionView.Key.range( + 2, "db", "tbl", first.rangePartitions(), List.of(Type.INT))); + Assertions.assertNotEquals(first, PluginDrivenPartitionView.Key.range( + 1, "db", "tbl", first.rangePartitions(), List.of(Type.STRING))); + } + + @Test + void realManagerInvalidatesTheMapAndIndexTogetherAtEveryScope() { + long catalogId = 987654321; + Env env = Mockito.mock(Env.class); + CatalogMgr catalogs = Mockito.mock(CatalogMgr.class); + ExternalCatalog catalog = Mockito.mock(ExternalCatalog.class); + ExternalDatabase database = Mockito.mock(ExternalDatabase.class); + Mockito.when(catalog.getId()).thenReturn(catalogId); + Mockito.when(catalog.getName()).thenReturn("partition_view_test"); + Mockito.when(catalog.getProperties()).thenReturn(Collections.emptyMap()); + Mockito.when(database.getFullName()).thenReturn("db"); + Mockito.doReturn(catalog).when(catalogs).getCatalog(catalogId); + Mockito.when(env.getCatalogMgr()).thenReturn(catalogs); + Mockito.when(env.getSortedPartitionsCacheManager()).thenReturn(new NereidsSortedPartitionsCacheManager()); + ExternalMetaCacheMgr manager = new ExternalMetaCacheMgr(true); + Mockito.when(env.getExtMetaCacheMgr()).thenReturn(manager); + try (MockedStatic envStatic = Mockito.mockStatic(Env.class)) { + envStatic.when(Env::getCurrentEnv).thenReturn(env); + PluginDrivenMvccExternalTable table = new PluginDrivenMvccExternalTable( + 1L, "tbl", "tbl", catalog, database); + List parts = List.of(partition("1", 10)); + PluginDrivenPartitionView first = table.partitionView(parts, List.of(Type.INT)); + Assertions.assertSame(first, table.partitionView(parts, List.of(Type.INT))); + manager.invalidateTable(catalogId, 1L, "db", 1L, "tbl"); + PluginDrivenPartitionView second = table.partitionView(parts, List.of(Type.INT)); + Assertions.assertNotSame(first, second); + Assertions.assertNotSame(first.getSortedRanges().get(), second.getSortedRanges().get()); + manager.invalidateDb(catalogId, 1L, "db"); + PluginDrivenPartitionView third = table.partitionView(parts, List.of(Type.INT)); + Assertions.assertNotSame(second, third); + manager.invalidateCatalog(catalogId); + Assertions.assertNotSame(third, table.partitionView(parts, List.of(Type.INT))); + Assertions.assertEquals("p=1", first.getSortedRanges().get().sortedPartitions.get(0).id); + manager.removeCatalog(catalogId); + } finally { + manager.commonRefreshExecutor().shutdownNow(); + manager.getFileListingExecutor().shutdownNow(); + manager.getScheduleExecutor().shutdownNow(); + } + } + +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogDdlRoutingTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogDdlRoutingTest.java index b57172cb1d9c11..2f5942b68a8fe2 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogDdlRoutingTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogDdlRoutingTest.java @@ -68,6 +68,8 @@ import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; import org.mockito.ArgumentCaptor; import org.mockito.InOrder; import org.mockito.MockedStatic; @@ -103,6 +105,8 @@ public void setUp() { connector = Mockito.mock(Connector.class); metadata = Mockito.mock(ConnectorMetadata.class); session = Mockito.mock(ConnectorSession.class); + Mockito.when(metadata.tableExists(Mockito.any(), Mockito.anyString(), Mockito.anyString())) + .thenCallRealMethod(); Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(metadata); Mockito.when(metadata.fromRemoteDatabaseName(Mockito.eq(session), Mockito.anyString())) @@ -379,7 +383,7 @@ public void testDropTableKeepsExactCleanupAfterDatabaseCacheEviction() throws Ex catalog.dropTable("DB1", "T1", false, false, false, false, false, false); - Mockito.verify(metadata).getTableHandle(session, "DB1", "TBL1"); + Mockito.verify(metadata, Mockito.times(2)).getTableHandle(session, "DB1", "TBL1"); Mockito.verify(metadata).dropTable(session, handle); ArgumentCaptor dropInfo = ArgumentCaptor.forClass(org.apache.doris.persist.DropInfo.class); @@ -464,6 +468,35 @@ public void testDropTemporaryTableIfExistsIsNoopAndKeepsPermanentExternalTable() Mockito.verify(connector, Mockito.never()).invalidateTable(Mockito.any(), Mockito.any()); } + @ParameterizedTest + @ValueSource(booleans = {true, false}) + public void testDropTableIgnoresStaleReadHandleAfterRemoteDrop(boolean ifExists) throws Exception { + ExternalDatabase db = mockExternalDatabase(); + ExternalTable table = Mockito.mock(ExternalTable.class); + Mockito.when(table.getName()).thenReturn("t1"); + Mockito.when(table.getId()).thenReturn(TABLE_ID); + Mockito.when(table.getRemoteDbName()).thenReturn("DB1"); + Mockito.when(table.getRemoteName()).thenReturn("TBL1"); + Mockito.doReturn(table).when(db).getTableNullable("t1"); + catalog.dbNullableResult = db; + Mockito.when(metadata.getTableHandle(session, "DB1", "TBL1")) + .thenReturn(Optional.of(Mockito.mock(ConnectorTableHandle.class))); + Mockito.doReturn(false).when(metadata).tableExists(session, "DB1", "TBL1"); + + if (ifExists) { + catalog.dropTable("db1", "t1", false, false, false, true, false, false); + Mockito.verify(connector).invalidateTable("DB1", "TBL1"); + Mockito.verify(mockEditLog).logDropTable(Mockito.any()); + } else { + Assertions.assertThrows(DdlException.class, + () -> catalog.dropTable("db1", "t1", false, false, false, false, false, false)); + Mockito.verify(connector, Mockito.never()).invalidateTable(Mockito.any(), Mockito.any()); + } + + Mockito.verify(metadata, Mockito.never()).getTableHandle(session, "DB1", "TBL1"); + Mockito.verify(metadata, Mockito.never()).dropTable(Mockito.any(), Mockito.any()); + } + @Test public void testDropTableHandleAbsentAfterLocalResolveCleansLocalStateWithIfExists() throws Exception { ExternalDatabase db = mockExternalDatabase(); @@ -836,6 +869,58 @@ public void testCreateTableIfNotExistsEvictsPreexistingRowCount() throws Excepti } } + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void testCreateTableAfterExternalDropIgnoresCachedReadHandle(boolean ifNotExists) throws Exception { + ExternalDatabase db = mockExternalDatabase(); + Mockito.when(db.getRemoteName()).thenReturn("DB1"); + catalog.dbNullableResult = db; + // The engine name cache has expired, but the connector still holds the previous read handle. + ConnectorTableHandle stale = Mockito.mock(ConnectorTableHandle.class); + Mockito.when(metadata.getTableHandle(session, "DB1", "t1")).thenReturn(Optional.of(stale)); + Mockito.doReturn(false).when(metadata).tableExists(session, "DB1", "t1"); + CreateTableInfo info = Mockito.mock(CreateTableInfo.class); + Mockito.when(info.getDbName()).thenReturn("db1"); + Mockito.when(info.getTableName()).thenReturn("t1"); + Mockito.when(info.isIfNotExists()).thenReturn(ifNotExists); + try (MockedStatic converter = + Mockito.mockStatic(CreateTableInfoToConnectorRequestConverter.class)) { + ConnectorCreateTableRequest request = Mockito.mock(ConnectorCreateTableRequest.class); + converter.when(() -> CreateTableInfoToConnectorRequestConverter.convert(info, "DB1")) + .thenReturn(request); + Assertions.assertFalse(catalog.createTable(info)); + Mockito.verify(metadata).createTable(session, request); + Mockito.verify(metadata, Mockito.never()).getTableHandle(session, "DB1", "t1"); + Mockito.verify(mockEditLog).logCreateTable(Mockito.any()); + } + } + + @Test + void testCreateFailureCannotUseCachedHandleAsConcurrentCreateProof() throws Exception { + ExternalDatabase db = mockExternalDatabase(); + Mockito.when(db.getRemoteName()).thenReturn("DB1"); + catalog.dbNullableResult = db; + ConnectorTableHandle stale = Mockito.mock(ConnectorTableHandle.class); + Mockito.when(metadata.getTableHandle(session, "DB1", "t1")).thenReturn(Optional.of(stale)); + Mockito.doReturn(false).when(metadata).tableExists(session, "DB1", "t1"); + CreateTableInfo info = Mockito.mock(CreateTableInfo.class); + Mockito.when(info.getDbName()).thenReturn("db1"); + Mockito.when(info.getTableName()).thenReturn("t1"); + Mockito.when(info.isIfNotExists()).thenReturn(true); + Mockito.doThrow(new DorisConnectorException("create rejected")) + .when(metadata).createTable(Mockito.any(), Mockito.any()); + try (MockedStatic converter = + Mockito.mockStatic(CreateTableInfoToConnectorRequestConverter.class)) { + converter.when(() -> CreateTableInfoToConnectorRequestConverter.convert(info, "DB1")) + .thenReturn(Mockito.mock(ConnectorCreateTableRequest.class)); + DdlException error = Assertions.assertThrows(DdlException.class, () -> catalog.createTable(info)); + Assertions.assertTrue(error.getMessage().contains("create rejected")); + Mockito.verify(metadata, Mockito.times(2)).tableExists(session, "DB1", "t1"); + Mockito.verify(metadata, Mockito.never()).getTableHandle(session, "DB1", "t1"); + Mockito.verify(mockEditLog, Mockito.never()).logCreateTable(Mockito.any()); + } + } + @Test public void testCreateTableIfNotExistsLosingTheCreateRaceReturnsTrueWithoutEditLog() throws Exception { // The existence probe and the remote create are not atomic: here the probe says ABSENT and the diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogViewListingTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogViewListingTest.java index e09173109f1fee..7248841d1479fa 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogViewListingTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogViewListingTest.java @@ -51,6 +51,7 @@ public class PluginDrivenExternalCatalogViewListingTest { public void setUp() { connector = Mockito.mock(Connector.class); metadata = Mockito.mock(ConnectorMetadata.class); + Mockito.when(metadata.listRelationNames(Mockito.any(), Mockito.anyString())).thenCallRealMethod(); ConnectorSession session = Mockito.mock(ConnectorSession.class); Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(metadata); catalog = new TestableCatalog(connector, session); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeHiveStreamingTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeHiveStreamingTest.java index 79c603ee3baff7..0be56ce2953520 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeHiveStreamingTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeHiveStreamingTest.java @@ -19,6 +19,7 @@ import org.apache.doris.analysis.SlotDescriptor; import org.apache.doris.analysis.TupleDescriptor; +import org.apache.doris.blockrule.SqlBlockRuleMgr; import org.apache.doris.catalog.Env; import org.apache.doris.catalog.PartitionItem; import org.apache.doris.catalog.TableIf; @@ -287,6 +288,7 @@ public void close() { } void start() { + Mockito.when(Env.getCurrentEnv().getSqlBlockRuleMgr()).thenReturn(Mockito.mock(SqlBlockRuleMgr.class)); Assertions.assertTrue(node.isBatchMode()); node.startSplit(1); } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodePartitionCountTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodePartitionCountTest.java index 8dea308d07bc47..cbd22f9afa96a5 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodePartitionCountTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodePartitionCountTest.java @@ -49,6 +49,28 @@ */ public class PluginDrivenScanNodePartitionCountTest { + @Test + void selectionPublishesCountsBeforeStreamingStarts() { + PluginDrivenScanNode scan = Mockito.mock(PluginDrivenScanNode.class, Mockito.CALLS_REAL_METHODS); + scan.setSelectedPartitions(SelectedPartitions.NATIVE); + Assertions.assertEquals(-1, scan.getSelectedPartitionNum()); + scan.setSelectedPartitions(new SelectedPartitions(3, + items(1), false)); + Assertions.assertEquals(1, scan.getSelectedPartitionNum()); + } + + @Test + public void testNativePlanningIsUnknownRatherThanEmpty() { + SelectedPartitions selection = SelectedPartitions.NATIVE; + Assertions.assertFalse(selection.isPruned); + Assertions.assertNotEquals(SelectedPartitions.NOT_PRUNED, selection); + Assertions.assertArrayEquals(new long[] {-1, -1}, + PluginDrivenScanNode.displayPartitionCounts(selection)); + Assertions.assertNull(PluginDrivenScanNode.resolveRequiredPartitions(selection, false)); + Assertions.assertNull(PluginDrivenScanNode.resolveRequiredPartitions(selection, true)); + Assertions.assertFalse(PluginDrivenScanNode.partitionsPrunedToEmpty(selection, true)); + } + private static Map items(int count) { Map items = new LinkedHashMap<>(); for (int i = 0; i < count; i++) { diff --git a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVRelationManagerTest.java b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVRelationManagerTest.java index 5109d399fb162d..e01ff7c24ce9ad 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVRelationManagerTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVRelationManagerTest.java @@ -20,7 +20,9 @@ import org.apache.doris.catalog.MTMV; import org.apache.doris.catalog.info.TableNameInfo; import org.apache.doris.common.AnalysisException; +import org.apache.doris.nereids.StatementContext; import org.apache.doris.persist.EditLog.EditLogItem; +import org.apache.doris.qe.ConnectContext; import com.google.common.collect.Sets; import org.apache.commons.collections4.CollectionUtils; @@ -59,6 +61,37 @@ public void setUp() throws NoSuchMethodException, SecurityException, AnalysisExc Mockito.when(t4.getTableName()).thenReturn("t4"); } + @Test + public void noEligibleCandidateDoesNotCollectQueryPartitions() { + MTMVRelationManager manager = new MTMVRelationManager(); + ConnectContext context = Mockito.mock(ConnectContext.class); + Assertions.assertTrue(manager.getAvailableMTMVs(Set.of(), context, false, (ctx, mv) -> false).isEmpty()); + MTMV disabled = Mockito.mock(MTMV.class); + MTMV excluded = Mockito.mock(MTMV.class); + Assertions.assertTrue(manager.getAvailableMTMVs(Set.of(disabled, excluded), context, false, + (ctx, mv) -> mv == excluded).isEmpty()); + Mockito.verifyNoInteractions(context); + } + + @Test + public void eligibleCandidatesShareOneQueryPartitionCollection() { + MTMVRelationManager manager = Mockito.spy(new MTMVRelationManager()); + ConnectContext context = Mockito.mock(ConnectContext.class); + StatementContext statement = new StatementContext(); + Mockito.when(context.getStatementContext()).thenReturn(statement); + MTMV first = Mockito.mock(MTMV.class); + MTMV second = Mockito.mock(MTMV.class); + Mockito.when(first.isUseForRewrite()).thenReturn(true); + Mockito.when(second.isUseForRewrite()).thenReturn(true); + Mockito.doReturn(true).when(manager).isMVPartitionValid( + Mockito.any(), Mockito.eq(context), Mockito.eq(false), Mockito.anyMap()); + Assertions.assertEquals(Set.of(first, second), manager.getAvailableMTMVs( + Set.of(first, second), context, false, (ctx, mv) -> false)); + Mockito.verify(context, Mockito.times(1)).getStatementContext(); + Mockito.verify(manager, Mockito.times(2)).isMVPartitionValid( + Mockito.any(), Mockito.eq(context), Mockito.eq(false), Mockito.anyMap()); + } + @Test public void testGetMtmvsByBaseTableOneLevelAndFromView() { // mock mv2==>mv1,t3; mv1==>t4 diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/SqlCacheContextPluginTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/SqlCacheContextPluginTableTest.java index 3026bad1ed8947..62ffd12781c725 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/SqlCacheContextPluginTableTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/SqlCacheContextPluginTableTest.java @@ -17,11 +17,17 @@ package org.apache.doris.nereids; +import org.apache.doris.analysis.TableSnapshot; import org.apache.doris.analysis.UserIdentity; import org.apache.doris.catalog.DatabaseIf; import org.apache.doris.catalog.TableIf.TableType; +import org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot; +import org.apache.doris.connector.spi.mvcc.ConnectorSqlCacheVersion; import org.apache.doris.datasource.CatalogIf; +import org.apache.doris.datasource.mvcc.MvccReadSelector; +import org.apache.doris.datasource.mvcc.MvccSnapshot; import org.apache.doris.datasource.mvcc.PluginDrivenMvccExternalTable; +import org.apache.doris.datasource.mvcc.PluginDrivenMvccSnapshot; import org.apache.doris.nereids.SqlCacheContext.TableVersion; import com.google.common.collect.Maps; @@ -29,6 +35,9 @@ import org.junit.jupiter.api.Test; import org.mockito.Mockito; +import java.util.Collections; +import java.util.Optional; + /** * Unit tests for the connector-agnostic invalidation token the SQL-result-cache migration wired into * {@link SqlCacheContext#addUsedTable}. A flipped lakehouse table is a @@ -39,6 +48,75 @@ */ public class SqlCacheContextPluginTableTest { + @Test + public void nativeVersionUsesTheReadPinAndChangesTheCacheKey() { + PluginDrivenMvccExternalTable table = mockTable(42L, 999L); + MvccSnapshot pin = new PluginDrivenMvccSnapshot(ConnectorMvccSnapshot.builder().snapshotId(7L).build(), + Collections.emptyMap(), Collections.emptyMap()); + ConnectorSqlCacheVersion first = new ConnectorSqlCacheVersion("table:snapshot7:schema1:metadata1", 1000L); + Mockito.when(table.getSqlCacheVersion(Optional.of(pin))).thenReturn(Optional.of(first)); + SqlCacheContext context = new SqlCacheContext(UserIdentity.ROOT); + context.setOriginSql("select count(*) from t"); + context.addUsedTable(table, Optional.of(pin)); + Assertions.assertSame(first, context.getUsedTables().values().iterator().next().sqlCacheVersion); + Mockito.verify(table, Mockito.never()).getNewestUpdateVersionOrTime(); + Mockito.verify(table, Mockito.never()).getNewestUpdateTimeMillisForCache(); + // Metadata/schema changes can keep both the snapshot id and the wall-clock update time unchanged. + ConnectorSqlCacheVersion second = new ConnectorSqlCacheVersion("table:snapshot7:schema2:metadata2", 1000L); + Mockito.when(table.getSqlCacheVersion(Optional.of(pin))).thenReturn(Optional.of(second)); + SqlCacheContext changed = new SqlCacheContext(UserIdentity.ROOT); + changed.setOriginSql("select count(*) from t"); + changed.addUsedTable(table, Optional.of(pin)); + Assertions.assertNotEquals(context.doComputeCacheKeyMd5(Collections.emptySet(), null), + changed.doComputeCacheKeyMd5(Collections.emptySet(), null)); + } + + @Test + public void nativeVersionFailureDisablesCachingWithoutLegacyEnumeration() { + PluginDrivenMvccExternalTable table = mockTable(42L, 999L); + Mockito.when(table.getSqlCacheVersion(Optional.empty())) + .thenThrow(new UnsupportedOperationException("no reliable metadata generation")); + SqlCacheContext context = new SqlCacheContext(UserIdentity.ROOT); + context.addUsedTable(table); + Assertions.assertFalse(context.supportSqlCache()); + Mockito.verify(table, Mockito.never()).getNewestUpdateVersionOrTime(); + } + + @Test + public void sameTableReferencesKeepSeparatePinsAndAllAffectTheCacheKey() { + PluginDrivenMvccExternalTable table = mockTable(42L, 999L); + MvccSnapshot main = new PluginDrivenMvccSnapshot(ConnectorMvccSnapshot.builder().snapshotId(100L).build(), + Collections.emptyMap(), Collections.emptyMap()); + MvccSnapshot dev = new PluginDrivenMvccSnapshot(ConnectorMvccSnapshot.builder().snapshotId(80L).build(), + Collections.emptyMap(), Collections.emptyMap()); + Mockito.when(table.getSqlCacheVersion(Optional.of(main))) + .thenReturn(Optional.of(new ConnectorSqlCacheVersion("uuid:100:schema3:metadata", 1000L))); + Mockito.when(table.getSqlCacheVersion(Optional.of(dev))) + .thenReturn(Optional.of(new ConnectorSqlCacheVersion("uuid:80:schema3:metadata", 800L))); + MvccReadSelector selector = MvccReadSelector.of(Optional.of(TableSnapshot.versionOf("dev")), Optional.empty()); + SqlCacheContext context = new SqlCacheContext(UserIdentity.ROOT); + context.setOriginSql("select * from t a join t for version as of 'dev' b on a.id=b.id"); + context.addUsedTable(table, Optional.of(main)); + context.addUsedTable(table, Optional.of(dev), selector); + Assertions.assertEquals(2, context.getUsedTables().size()); + org.apache.doris.proto.Types.PUniqueId before = context.doComputeCacheKeyMd5(Collections.emptySet(), null); + Mockito.when(table.getSqlCacheVersion(Optional.of(dev))) + .thenReturn(Optional.of(new ConnectorSqlCacheVersion("uuid:90:schema3:metadata", 800L))); + context.addUsedTable(table, Optional.of(dev), selector); + Assertions.assertEquals(2, context.getUsedTables().size()); + Assertions.assertNotEquals(before, context.doComputeCacheKeyMd5(Collections.emptySet(), null)); + Mockito.verify(table, Mockito.never()).getNewestUpdateVersionOrTime(); + } + + @Test + public void historicalReferencesWithoutVersionAwareConnectorRemainUncacheable() { + SqlCacheContext context = new SqlCacheContext(UserIdentity.ROOT); + context.addUsedTable(mockTable(42L, 999L), Optional.empty(), + MvccReadSelector.of(Optional.of(TableSnapshot.versionOf("100")), Optional.empty())); + Assertions.assertFalse(context.supportSqlCache()); + Assertions.assertTrue(context.getUsedTables().isEmpty()); + } + private PluginDrivenMvccExternalTable mockTable(long id, long token) { PluginDrivenMvccExternalTable table = Mockito.mock(PluginDrivenMvccExternalTable.class); DatabaseIf db = Mockito.mock(DatabaseIf.class); diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/QueryPartitionCollectorTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/QueryPartitionCollectorTest.java new file mode 100644 index 00000000000000..710ce766190466 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/QueryPartitionCollectorTest.java @@ -0,0 +1,140 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids.rules.rewrite; + +import org.apache.doris.catalog.DatabaseIf; +import org.apache.doris.catalog.PartitionItem; +import org.apache.doris.common.Pair; +import org.apache.doris.datasource.ExternalTable; +import org.apache.doris.nereids.CascadesContext; +import org.apache.doris.nereids.StatementContext; +import org.apache.doris.nereids.rules.exploration.mv.PartitionCompensator; +import org.apache.doris.nereids.trees.plans.RelationId; +import org.apache.doris.nereids.trees.plans.logical.LogicalFileScan; +import org.apache.doris.nereids.trees.plans.logical.LogicalFileScan.SelectedPartitions; + +import com.google.common.collect.ImmutableMap; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +import java.util.AbstractSet; +import java.util.BitSet; +import java.util.Collections; +import java.util.Iterator; +import java.util.List; +import java.util.Map; +import java.util.Set; + +class QueryPartitionCollectorTest { + private static final List QUALIFIERS = List.of("catalog", "db", "table"); + + private final StatementContext statement = new StatementContext(); + private final CascadesContext context = Mockito.mock(CascadesContext.class); + private final ExternalTable table = Mockito.mock(ExternalTable.class); + private final QueryPartitionCollector collector = new QueryPartitionCollector(); + + private LogicalFileScan scan(int id, String... names) { + Mockito.when(context.getStatementContext()).thenReturn(statement); + Mockito.when(table.getDatabase()).thenReturn(Mockito.mock(DatabaseIf.class)); + Mockito.when(table.getFullQualifiers()).thenReturn(QUALIFIERS); + Mockito.when(table.supportInternalPartitionPruned()).thenReturn(true); + ImmutableMap.Builder items = ImmutableMap.builder(); + for (String name : names) { + items.put(name, Mockito.mock(PartitionItem.class)); + } + LogicalFileScan scan = Mockito.mock(LogicalFileScan.class); + Mockito.when(scan.getTable()).thenReturn(table); + Mockito.when(scan.getRelationId()).thenReturn(new RelationId(id)); + Mockito.when(scan.getSelectedPartitions()).thenReturn(new SelectedPartitions(10, items.build(), true)); + return scan; + } + + @Test + void nativePlanningRecordsUnknownAsAllPartitionsWithoutLoadingTheView() { + LogicalFileScan scan = scan(1); + Mockito.when(scan.getSelectedPartitions()).thenReturn(SelectedPartitions.NATIVE); + Mockito.clearInvocations(table); + collector.visitLogicalCatalogRelation(scan, context); + Assertions.assertTrue(PartitionCompensator.getQueryUsedPartitions(statement, new BitSet()) + .containsKey(QUALIFIERS)); + Assertions.assertSame(PartitionCompensator.ALL_PARTITIONS, + statement.getTableUsedPartitionNameMap().get(QUALIFIERS).iterator().next()); + Mockito.verify(table, Mockito.never()).supportInternalPartitionPruned(); + } + + @Test + void keepsTheSelectedSnapshotKeyView() { + LogicalFileScan first = scan(1, "p1", "p2"); + collector.visitLogicalCatalogRelation(first, context); + Set recorded = statement.getTableUsedPartitionNameMap().get(QUALIFIERS).iterator().next().value(); + Assertions.assertSame(first.getSelectedPartitions().selectedPartitions.keySet(), recorded); + Assertions.assertThrows(UnsupportedOperationException.class, () -> recorded.remove("p1")); + collector.visitLogicalCatalogRelation(scan(2, "p3"), context); + Assertions.assertEquals(Set.of("p1", "p2"), recorded); + } + + @Test + void selfJoinAndRepeatedRecordsPreserveUnionAndRelationFiltering() { + LogicalFileScan first = scan(1, "p1", "p2"); + collector.visitLogicalCatalogRelation(first, context); + collector.visitLogicalCatalogRelation(scan(2, "p2", "p3"), context); + collector.visitLogicalCatalogRelation(first, context); + Map, Set> union = PartitionCompensator.getQueryUsedPartitions(statement, new BitSet()); + Assertions.assertEquals(Set.of("p1", "p2", "p3"), union.get(QUALIFIERS)); + BitSet relation = new BitSet(); + relation.set(1); + Assertions.assertEquals(Set.of("p1", "p2"), + PartitionCompensator.getQueryUsedPartitions(statement, relation).get(QUALIFIERS)); + union.get(QUALIFIERS).clear(); + Assertions.assertEquals(Set.of("p1", "p2"), first.getSelectedPartitions().selectedPartitions.keySet()); + } + + @Test + void recordingDoesNotVisitOrHashPartitionNames() { + Set names = new AbstractSet<>() { + @Override + public Iterator iterator() { + throw new AssertionError("recording must not iterate partition names"); + } + + @Override + public int size() { + return 120000; + } + + @Override + public int hashCode() { + throw new AssertionError("recording must not hash partition names"); + } + }; + statement.getTableUsedPartitionNameMap().put(QUALIFIERS, Pair.of(new RelationId(1), names)); + Assertions.assertSame(names, + statement.getTableUsedPartitionNameMap().get(QUALIFIERS).iterator().next().value()); + } + + @Test + void emptySelectionAndAllPartitionsKeepTheirDistinctMeaning() { + collector.visitLogicalCatalogRelation(scan(1), context); + Assertions.assertEquals(Collections.emptySet(), + PartitionCompensator.getQueryUsedPartitions(statement, new BitSet()).get(QUALIFIERS)); + statement.getTableUsedPartitionNameMap().put(QUALIFIERS, PartitionCompensator.ALL_PARTITIONS); + statement.getTableUsedPartitionNameMap().put(QUALIFIERS, PartitionCompensator.ALL_PARTITIONS); + Assertions.assertNull(PartitionCompensator.getQueryUsedPartitions(statement, new BitSet()).get(QUALIFIERS)); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/qe/cache/PluginTableCacheAnalyzerTest.java b/fe/fe-core/src/test/java/org/apache/doris/qe/cache/PluginTableCacheAnalyzerTest.java index f1392d7a9cd36d..e52339b2a2b395 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/qe/cache/PluginTableCacheAnalyzerTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/qe/cache/PluginTableCacheAnalyzerTest.java @@ -17,15 +17,22 @@ package org.apache.doris.qe.cache; +import org.apache.doris.analysis.TableSnapshot; import org.apache.doris.analysis.TupleDescriptor; import org.apache.doris.analysis.TupleId; +import org.apache.doris.analysis.UserIdentity; import org.apache.doris.catalog.DatabaseIf; import org.apache.doris.catalog.FunctionGenTable; import org.apache.doris.catalog.TableIf; import org.apache.doris.common.jmockit.Deencapsulation; +import org.apache.doris.connector.spi.mvcc.ConnectorSqlCacheVersion; import org.apache.doris.datasource.CatalogIf; +import org.apache.doris.datasource.mvcc.MvccReadSelector; +import org.apache.doris.datasource.mvcc.MvccSnapshot; import org.apache.doris.datasource.mvcc.PluginDrivenMvccExternalTable; import org.apache.doris.datasource.scan.PluginDrivenScanNode; +import org.apache.doris.nereids.SqlCacheContext; +import org.apache.doris.nereids.StatementContext; import org.apache.doris.qe.ConnectContext; import org.apache.doris.qe.SessionVariable; @@ -35,6 +42,7 @@ import org.mockito.Mockito; import java.util.Collections; +import java.util.Optional; /** * Unit tests for the connector-agnostic scan-node recognition the SQL-result-cache migration added to @@ -60,6 +68,51 @@ public void setUp() { analyzer = new CacheAnalyzer(ctx, null, Collections.emptyList()); } + @Test + public void nativeCacheRequestUsesCapturedVersionWithoutReprobingLatestOrCountingPartitions() { + ConnectContext ctx = Mockito.mock(ConnectContext.class); + Mockito.when(ctx.getSessionVariable()).thenReturn(new SessionVariable()); + StatementContext statement = Mockito.mock(StatementContext.class); + SqlCacheContext cacheContext = new SqlCacheContext(UserIdentity.ROOT); + Mockito.when(ctx.getStatementContext()).thenReturn(statement); + Mockito.when(statement.getSqlCacheContext()).thenReturn(Optional.of(cacheContext)); + CacheAnalyzer nativeAnalyzer = new CacheAnalyzer(ctx, null, Collections.emptyList()); + PluginDrivenMvccExternalTable table = Mockito.mock(PluginDrivenMvccExternalTable.class); + DatabaseIf db = Mockito.mock(DatabaseIf.class); + CatalogIf catalog = Mockito.mock(CatalogIf.class); + Mockito.when(catalog.isInternalCatalog()).thenReturn(false); + Mockito.when(catalog.getProperties()).thenReturn(Collections.emptyMap()); + Mockito.when(catalog.getName()).thenReturn("ice"); + Mockito.when(db.getCatalog()).thenReturn(catalog); + Mockito.when(db.getFullName()).thenReturn("db"); + Mockito.when(table.getDatabase()).thenReturn(db); + Mockito.when(table.getName()).thenReturn("t"); + Mockito.when(table.getSqlCacheVersion(Optional.empty())) + .thenReturn(Optional.of(new ConnectorSqlCacheVersion("uuid:7:schema1:metadata1", 1000L))); + cacheContext.addUsedTable(table); + MvccSnapshot branchPin = Mockito.mock(MvccSnapshot.class); + Mockito.when(table.getSqlCacheVersion(Optional.of(branchPin))) + .thenReturn(Optional.of(new ConnectorSqlCacheVersion("uuid:8:schema1:metadata1", 2000L))); + cacheContext.addUsedTable(table, Optional.of(branchPin), MvccReadSelector.of( + Optional.of(TableSnapshot.versionOf("dev")), Optional.empty())); + Mockito.clearInvocations(table); + PluginDrivenScanNode node = mockPluginScanNode(table, -1L); + CacheAnalyzer.CacheTable result = Deencapsulation.invoke(nativeAnalyzer, "buildCacheTableForExternalScanNode", node); + Assertions.assertEquals(1L, result.partitionNum); + Assertions.assertEquals(1000L, result.latestPartitionTime); + Assertions.assertEquals(1000L, result.latestPartitionUpdateMillis); + PluginDrivenScanNode branchNode = mockPluginScanNode(table, -1L); + Mockito.when(branchNode.getQueryTableSnapshot()).thenReturn(TableSnapshot.versionOf("dev")); + CacheAnalyzer.CacheTable branchResult = + Deencapsulation.invoke(nativeAnalyzer, "buildCacheTableForExternalScanNode", branchNode); + Assertions.assertEquals(2000L, branchResult.latestPartitionTime); + Assertions.assertEquals(2000L, branchResult.latestPartitionUpdateMillis); + Mockito.verify(table, Mockito.never()).getSqlCacheVersion(Mockito.any()); + Mockito.verify(table, Mockito.never()).getNewestUpdateVersionOrTime(); + Mockito.verify(table, Mockito.never()).getNewestUpdateTimeMillisForCache(); + Mockito.verify(node, Mockito.never()).getSelectedPartitionNum(); + } + private PluginDrivenScanNode mockPluginScanNode(TableIf table, long selectedPartitionNum) { PluginDrivenScanNode node = Mockito.mock(PluginDrivenScanNode.class); TupleDescriptor desc = new TupleDescriptor(new TupleId(1)); diff --git a/regression-test/data/external_table_p0/iceberg/test_iceberg_table_handle_reuse.out b/regression-test/data/external_table_p0/iceberg/test_iceberg_table_handle_reuse.out new file mode 100644 index 00000000000000..f0ca427a054340 --- /dev/null +++ b/regression-test/data/external_table_p0/iceberg/test_iceberg_table_handle_reuse.out @@ -0,0 +1,44 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !cold -- +1 +2 + +-- !warm -- +1 +2 + +-- !multiple_versions -- +branch 2 3 +main 3 3000000003 +tag 2 3 + +-- !moved_branch -- +1 +2 +3000000000 + +-- !original_tag -- +1 +2 + +-- !table_cold -- +3 3000000003 + +-- !table_warm -- +3 3000000003 + +-- !database_cold -- +3 3000000003 + +-- !database_warm -- +3 3000000003 + +-- !catalog_cold -- +3 3000000003 + +-- !catalog_warm -- +3 3000000003 + +-- !recreated -- +42 + diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_table_handle_reuse.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_table_handle_reuse.groovy new file mode 100644 index 00000000000000..efaec28ac02edd --- /dev/null +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_table_handle_reuse.groovy @@ -0,0 +1,84 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +suite("test_iceberg_table_handle_reuse", "p0,external,iceberg") { + if (!"true".equalsIgnoreCase(context.config.otherConfigs.get("enableIcebergTest"))) { + return + } + String catalog = context.config.otherConfigs.get("icebergSplitCatalog") + if (!catalog) { + catalog = "iceberg_table_handle_reuse" + String host = context.config.otherConfigs.get("externalEnvIp") + String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port") + String minioPort = context.config.otherConfigs.get("iceberg_minio_port") + sql "drop catalog if exists ${catalog}" + sql """create catalog ${catalog} properties ( + 'type'='iceberg', 'iceberg.catalog.type'='rest', 'uri'='http://${host}:${restPort}', + 's3.access_key'='admin', 's3.secret_key'='password', + 's3.endpoint'='http://${host}:${minioPort}', 's3.region'='us-east-1', + 'meta.cache.iceberg.manifest.enable'='true')""" + } + String sqlCache = sql("show variables like 'enable_sql_cache'")[0][1] + try { + sql "set enable_sql_cache=false" + sql "create database if not exists ${catalog}.iceberg_table_handle_reuse" + sql "use ${catalog}.iceberg_table_handle_reuse" + sql "drop table if exists handle_versions" + test { + sql "select * from handle_versions" + exception "does not exist" + } + sql "create table handle_versions (id int) engine=iceberg" + sql "create table if not exists handle_versions (id int) engine=iceberg" + sql "insert into handle_versions values (1), (2)" + sql "alter table handle_versions create branch audit" + sql "alter table handle_versions create tag before_change" + order_qt_cold "select id from handle_versions" + order_qt_warm "select id from handle_versions" + sql "alter table handle_versions modify column id bigint" + sql "insert into handle_versions values (3000000000)" + // Independent handles within one statement must retain their own snapshot/ref. + order_qt_multiple_versions """select 'main', count(*), sum(id) from handle_versions + union all select 'branch', count(*), sum(id) from handle_versions@branch(audit) + union all select 'tag', count(*), sum(id) from handle_versions@tag(before_change)""" + sql "alter table handle_versions create or replace branch audit" + order_qt_moved_branch "select id from handle_versions@branch(audit)" + order_qt_original_tag "select id from handle_versions@tag(before_change)" + for (String level : ["table", "database", "catalog"]) { + if (level == "table") { + sql "refresh table handle_versions" + } else if (level == "database") { + sql "refresh database ${catalog}.iceberg_table_handle_reuse" + } else { + sql "refresh catalog ${catalog}" + } + quickTest("${level}_cold", "select count(*), sum(id) from handle_versions", true) + quickTest("${level}_warm", "select count(*), sum(id) from handle_versions", true) + } + // DROP removes positive proof and a previous miss must not mask recreation. + sql "drop table handle_versions" + test { + sql "select * from handle_versions" + exception "does not exist" + } + sql "create table handle_versions (id bigint) engine=iceberg" + sql "insert into handle_versions values (42)" + order_qt_recreated "select id from handle_versions" + } finally { + sql "set enable_sql_cache=${sqlCache}" + } +} diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_binary_partitions.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_binary_partitions.groovy index 88e9f47102a84b..7a0223e748330e 100644 --- a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_binary_partitions.groovy +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_binary_partitions.groovy @@ -16,6 +16,8 @@ // under the License. suite("test_iceberg_write_binary_partitions", "p0,external,iceberg,external_docker,external_docker_iceberg") { + // This suite explicitly checks the converted partition metadata. + sql "SET enable_iceberg_internal_partition_pruning=true" if (!context.config.otherConfigs.get("enableIcebergTest")?.toString()?.equalsIgnoreCase("true")) { return } diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_uuid_inputs.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_uuid_inputs.groovy index c0e25e529ebad2..4a3223b7ba4c2f 100644 --- a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_uuid_inputs.groovy +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_uuid_inputs.groovy @@ -18,6 +18,8 @@ import groovy.json.JsonOutput suite("test_iceberg_write_uuid_inputs", "p0,external,iceberg,external_docker,external_docker_iceberg") { + // This suite explicitly checks the converted partition metadata. + sql "SET enable_iceberg_internal_partition_pruning=true" if (!context.config.otherConfigs.get("enableIcebergTest")?.toString()?.equalsIgnoreCase("true")) { return } From 607d4809acc37a9b7a73803eceb899b84fc104b9 Mon Sep 17 00:00:00 2001 From: 924060929 Date: Fri, 23 Oct 2026 16:58:25 +0800 Subject: [PATCH 2/5] [fix](fe) Advance connector API version after Iceberg rebase Issue Number: None Problem Summary: The Iceberg PR adds public connector SPI after master already advanced the plugin API to 20.0. Rebasing without a further version increment would misrepresent compatibility. Advance the plugin API and surface contract to 21.0. None - Test: Unit Test: ConnectorPluginSurfaceTest, IcebergTableHandleReuseTest, MvccReadSelectorTest, and PluginDrivenPartitionViewTest passed (28 tests). - Behavior changed: No (version declaration reflects the public SPI change). - Does this need documentation: No --- .../apache/doris/connector/spi/ConnectorPluginSurfaceTest.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java index e108648739ad9e..a2027678bd4e8e 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java +++ b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java @@ -89,7 +89,7 @@ public void connectorApiMajorTracksTheRecordedSurfaceChange() throws IOException version.load(in); } // Major 20 adds deleteUsingMetadata to the connector write surface; major 21 adds - // typed filesystem credential descriptions to that surface. + // typed filesystem credential descriptions and statement-scoped metadata. Assertions.assertEquals("21.0", version.getProperty("api.version")); } From 6db4660f3b757adfde5259a955105874b563057a Mon Sep 17 00:00:00 2001 From: 924060929 Date: Mon, 26 Oct 2026 11:24:32 +0800 Subject: [PATCH 3/5] [fix](fe) Resolve Iceberg DDL handles without scan planning ### What problem does this PR solve? Issue Number: None Problem Summary: A cold REST Iceberg table with server-side scan planning can pass the live existence check for DROP TABLE, then fail while the query read-handle path rejects local scan planning. Add a DDL handle path that returns an identifier-only Iceberg handle after a live existence check. Keep the query read path unchanged and route DROP and ALTER through the DDL path. ### Release note REST Iceberg DROP TABLE and ALTER TABLE no longer require local scan-planning support. ### Check List (For Author) - Test: Unit Test: IcebergTableHandleReuseTest (10 tests) and PluginDrivenExternalCatalogDdlRoutingTest (90 tests) via run-fe-ut.sh; BUILD SUCCESS. - Behavior changed: Yes, REST Iceberg DDL uses an identifier-only handle after checking table existence. - Does this need documentation: No --- .../iceberg/IcebergConnectorMetadata.java | 7 +++++ .../iceberg/IcebergTableHandleReuseTest.java | 28 +++++++++++++++++++ .../spi/ConnectorTableMetadataOps.java | 10 +++++++ .../plugin/PluginDrivenExternalCatalog.java | 4 +-- ...inDrivenExternalCatalogDdlRoutingTest.java | 2 ++ 5 files changed, 49 insertions(+), 2 deletions(-) diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java index b0d49647b5dfa8..1b483f09a443c5 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java @@ -492,6 +492,13 @@ public Optional getTableHandle( } } + @Override + public Optional getTableHandleForDdl( + ConnectorSession session, String dbName, String tableName) { + return tableExists(session, dbName, tableName) + ? Optional.of(new IcebergTableHandle(dbName, tableName)) : Optional.empty(); + } + @Override public boolean tableExists( ConnectorSession session, String dbName, String tableName) { diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableHandleReuseTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableHandleReuseTest.java index 93111136cd3de7..1000582d2bbf8c 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableHandleReuseTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableHandleReuseTest.java @@ -25,6 +25,7 @@ import org.apache.iceberg.DataFiles; import org.apache.iceberg.PartitionSpec; import org.apache.iceberg.Schema; +import org.apache.iceberg.SupportsDistributedScanPlanning; import org.apache.iceberg.Table; import org.apache.iceberg.catalog.Namespace; import org.apache.iceberg.catalog.TableIdentifier; @@ -33,6 +34,7 @@ import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import java.lang.reflect.Proxy; import java.util.Collections; import java.util.concurrent.atomic.AtomicInteger; @@ -61,6 +63,32 @@ private static long existenceCalls(RecordingIcebergCatalogOps ops) { return ops.log.stream().filter(s -> s.startsWith("tableExists:")).count(); } + @Test + void ddlHandleDoesNotRequireLocalScanPlanning() { + RecordingIcebergCatalogOps ops = ops(); + Table serverPlanned = (Table) Proxy.newProxyInstance(Table.class.getClassLoader(), + new Class[] {Table.class, SupportsDistributedScanPlanning.class}, + (proxy, method, args) -> { + if (method.getName().equals("allowDistributedPlanning")) { + return false; + } + throw new AssertionError("DDL must not read scan metadata: " + method.getName()); + }); + ops.table = serverPlanned; + IcebergConnectorMetadata metadata = metadata(ops, null); + ConnectorSession session = session("ddl", new TestStatementScope()); + + RuntimeException scanFailure = Assertions.assertThrows(RuntimeException.class, + () -> metadata.getTableHandle(session, "db1", "t1")); + Assertions.assertTrue(scanFailure.getMessage().contains("server-side scan planning")); + Assertions.assertTrue(metadata.tableExists(session, "db1", "t1")); + int loads = (int) ops.log.stream().filter(s -> s.startsWith("loadTable:")).count(); + ConnectorTableHandle handle = metadata.getTableHandleForDdl(session, "db1", "t1").orElseThrow(); + Assertions.assertEquals(loads, ops.log.stream().filter(s -> s.startsWith("loadTable:")).count()); + metadata.dropTable(session, handle); + Assertions.assertTrue(ops.log.stream().anyMatch(s -> s.startsWith("dropTable:"))); + } + @Test void loadedStatementTableProvesExistenceWithoutSharedCache() { RecordingIcebergCatalogOps ops = ops(); diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorTableMetadataOps.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorTableMetadataOps.java index 26cbb583bd8e8a..23007fad9dec3c 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorTableMetadataOps.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorTableMetadataOps.java @@ -70,6 +70,16 @@ default Optional getTableHandle( return Optional.empty(); } + /** + * Resolves a handle for DDL after the caller has checked live table existence. Connectors whose read handles + * require scan-capable metadata may return an identifier-only handle here. Other connectors retain their + * existing handle resolution behavior. + */ + default Optional getTableHandleForDdl( + ConnectorSession session, String dbName, String tableName) { + return getTableHandle(session, dbName, tableName); + } + /** Checks existence without requiring the caller to retain a planning handle or read metadata. */ default boolean tableExists(ConnectorSession session, String dbName, String tableName) { return getTableHandle(session, dbName, tableName).isPresent(); diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java index 147709566fca2b..78039d974896e5 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java @@ -847,7 +847,7 @@ public void dropTable(String dbName, String tableName, boolean isView, boolean i // first check current existence so IF EXISTS still performs local cleanup in that case. Optional handle = metadata.tableExists( session, dorisTable.getRemoteDbName(), dorisTable.getRemoteName()) - ? metadata.getTableHandle(session, dorisTable.getRemoteDbName(), dorisTable.getRemoteName()) + ? metadata.getTableHandleForDdl(session, dorisTable.getRemoteDbName(), dorisTable.getRemoteName()) : Optional.empty(); // The table is present in the FE cache but may have been dropped out-of-band on the remote // side; preserve the existing IF EXISTS handling for that case. @@ -1414,7 +1414,7 @@ private ExternalTable checkExternalTable(TableIf dorisTable) { */ private ConnectorTableHandle resolveAlterHandle(ExternalTable externalTable, ConnectorSession session, ConnectorMetadata metadata) throws DdlException { - Optional handle = metadata.getTableHandle( + Optional handle = metadata.getTableHandleForDdl( session, externalTable.getRemoteDbName(), externalTable.getRemoteName()); if (!handle.isPresent()) { throw new DdlException("Failed to get table: '" + externalTable.getName() diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogDdlRoutingTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogDdlRoutingTest.java index 2f5942b68a8fe2..dae69278b34ec6 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogDdlRoutingTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogDdlRoutingTest.java @@ -107,6 +107,8 @@ public void setUp() { session = Mockito.mock(ConnectorSession.class); Mockito.when(metadata.tableExists(Mockito.any(), Mockito.anyString(), Mockito.anyString())) .thenCallRealMethod(); + Mockito.when(metadata.getTableHandleForDdl(Mockito.any(), Mockito.anyString(), Mockito.anyString())) + .thenCallRealMethod(); Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(metadata); Mockito.when(metadata.fromRemoteDatabaseName(Mockito.eq(session), Mockito.anyString())) From 068d3c723090bc1171a6d82f7c7a4610f15abaf7 Mon Sep 17 00:00:00 2001 From: 924060929 Date: Mon, 26 Oct 2026 12:00:57 +0800 Subject: [PATCH 4/5] [fix](fe) Complete DDL handle routing and avoid unused partition indexes ### What problem does this PR solve? Issue Number: None Problem Summary: The new metadata-only DDL handle method was absent from the recorded SPI surface and plugin API version. Metadata-only EXECUTE actions still acquired scan-capable read handles, so a cold REST Iceberg table configured for server-side scan planning rejected rollback before execution. Partition views also sorted all partition ranges while binary-search filtering was disabled. Record the DDL method and advance the connector API version, use the DDL handle for single-call procedures while retaining a scan handle for distributed rewrites, and key partition views by the index setting so disabled queries skip index construction without poisoning enabled queries. ### Release note Metadata-only Iceberg EXECUTE actions support server-planned REST tables. External partition views skip sorted-index construction when binary-search partition filtering is disabled. ### Check List (For Author) - Test: Unit Test: ConnectorMetadataSurfaceTest (4), ConnectorPluginSurfaceTest (2), IcebergTableHandleReuseTest (10), PluginDrivenPartitionViewTest (15), and ConnectorExecuteActionTest (16) via run-fe-ut.sh; strengthened ConnectorExecuteActionTest (16) rerun. Full FE build via build.sh --fe -j4, including Checkstyle, passed after temporarily preserving existing output plugin configs. - Behavior changed: Yes, metadata-only EXECUTE uses the DDL handle path and disabled binary-search partition filtering does not build the sorted index. - Does this need documentation: No --- .../spi/ConnectorPluginSurfaceTest.java | 3 ++- .../resources/connector-metadata-methods.txt | 1 + fe/fe-connector/pom.xml | 2 +- .../mvcc/PluginDrivenMvccExternalTable.java | 18 ++++++++----- .../mvcc/PluginDrivenPartitionView.java | 19 ++++++++++++-- .../execute/ConnectorExecuteAction.java | 6 ++++- .../mvcc/PluginDrivenPartitionViewTest.java | 26 +++++++++++++++++++ .../execute/ConnectorExecuteActionTest.java | 12 +++++++-- 8 files changed, 73 insertions(+), 14 deletions(-) diff --git a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java index a2027678bd4e8e..eb5d693d6f57f4 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java +++ b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java @@ -90,7 +90,8 @@ public void connectorApiMajorTracksTheRecordedSurfaceChange() throws IOException } // Major 20 adds deleteUsingMetadata to the connector write surface; major 21 adds // typed filesystem credential descriptions and statement-scoped metadata. - Assertions.assertEquals("21.0", version.getProperty("api.version")); + // Major 22 adds the metadata-only DDL handle path. + Assertions.assertEquals("22.0", version.getProperty("api.version")); } /** Root entry points plus provider/handle types returned to connector plugins. */ diff --git a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-metadata-methods.txt b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-metadata-methods.txt index 955bb070bde2bc..3d65343de22afc 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-metadata-methods.txt +++ b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-metadata-methods.txt @@ -44,6 +44,7 @@ getSysTableHandle(org.apache.doris.connector.spi.ConnectorSession,org.apache.dor getTableComment(org.apache.doris.connector.spi.ConnectorSession,java.lang.String,java.lang.String) getTableFreshness(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle) getTableHandle(org.apache.doris.connector.spi.ConnectorSession,java.lang.String,java.lang.String) +getTableHandleForDdl(org.apache.doris.connector.spi.ConnectorSession,java.lang.String,java.lang.String) getTableSchema(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle) getTableSchema(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,org.apache.doris.connector.spi.mvcc.ConnectorMvccSnapshot) getTableStatistics(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle) diff --git a/fe/fe-connector/pom.xml b/fe/fe-connector/pom.xml index 1d91dc0c412092..55ea25afcf097f 100644 --- a/fe/fe-connector/pom.xml +++ b/fe/fe-connector/pom.xml @@ -55,7 +55,7 @@ under the License. of the latter two means bumping this property as well (and fe-extension-spi means bumping all five families). --> - 21.0 + 22.0 diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java index 487461630fecee..6db3cc933c72ff 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java @@ -285,7 +285,8 @@ private PluginDrivenMvccSnapshot buildFromRangeView(ConnectorMvccSnapshot connec for (int i = 0; i < partitionColumns.size(); i++) { columns.add(new Column(partitionColumns.get(i).getName(), frozenTypes.get(i))); } - return buildRangePartitionView(version.isPresent() ? partitions : k.rangePartitions(), columns); + return buildRangePartitionView(version.isPresent() ? partitions : k.rangePartitions(), + columns, k.buildSortedRanges()); }); return new PluginDrivenMvccSnapshot(connectorSnapshot, converted, pinnedSchema, partitionType, snapshotIdFreshness, view.getNewestUpdateMonotonicMarker(), @@ -293,7 +294,7 @@ private PluginDrivenMvccSnapshot buildFromRangeView(ConnectorMvccSnapshot connec } private PluginDrivenPartitionView buildRangePartitionView(List partitions, - List partitionColumns) { + List partitionColumns, boolean buildSortedRanges) { Map nameToPartitionItem = Maps.newHashMap(); Map nameToFreshnessValue = Maps.newHashMap(); for (ConnectorMvccPartition partition : partitions) { @@ -308,7 +309,7 @@ private PluginDrivenPartitionView buildRangePartitionView(List buildListPartitionView( - metadata.listPartitions(session, handle, Optional.empty()), k.frozenTypes())); + metadata.listPartitions(session, handle, Optional.empty()), k.frozenTypes(), + k.buildSortedRanges())); } return partitionView(metadata.listPartitions(session, handle, Optional.empty()), types); } @@ -360,7 +362,8 @@ private PluginDrivenPartitionView listPartitions(ConnectorMetadata metadata, Con PluginDrivenPartitionView partitionView(List parts, List types) { PluginDrivenPartitionView.Key key = PluginDrivenPartitionView.Key.list( getId(), getDbName(), getName(), parts, types); - return cachedPartitionView(key, k -> buildListPartitionView(k.listPartitions(), k.frozenTypes())); + return cachedPartitionView(key, k -> buildListPartitionView( + k.listPartitions(), k.frozenTypes(), k.buildSortedRanges())); } PluginDrivenPartitionView cachedPartitionView(PluginDrivenPartitionView.Key key, @@ -368,7 +371,8 @@ PluginDrivenPartitionView cachedPartitionView(PluginDrivenPartitionView.Key key, return Env.getCurrentEnv().getExtMetaCacheMgr().getPartitionView(this, key, loader); } - private PluginDrivenPartitionView buildListPartitionView(List parts, List types) { + private PluginDrivenPartitionView buildListPartitionView(List parts, List types, + boolean buildSortedRanges) { Map nameToPartitionItem = Maps.newHashMapWithExpectedSize(parts.size()); Map nameToLastModifiedMillis = Maps.newHashMapWithExpectedSize(parts.size()); for (ConnectorPartitionInfo part : parts) { @@ -416,7 +420,7 @@ private PluginDrivenPartitionView buildListPartitionView(List sortedRanges; PluginDrivenPartitionView(Map items, Map freshness) { + this(items, freshness, true); + } + + PluginDrivenPartitionView(Map items, Map freshness, + boolean buildSortedRanges) { this.items = ImmutableMap.copyOf(items); this.freshness = ImmutableMap.copyOf(freshness); // Build before cache admission, like HivePartitionValues: the cache accounts for the index // together with its maps, and readers never observe a partially initialized generation. - this.sortedRanges = hasComparableKeys(this.items) ? SortedPartitionRanges.build(this.items) : null; + this.sortedRanges = buildSortedRanges && hasComparableKeys(this.items) + ? SortedPartitionRanges.build(this.items) : null; } // Iceberg display partitions can come from different specs. They remain useful for counts and @@ -180,6 +186,7 @@ public static final class Key { private final boolean range; private final List types; private final String timeZone; + private final boolean buildSortedRanges; private final int hash; private final String version; @@ -200,12 +207,15 @@ private Key(long tableId, String database, String table, List partitions, ConnectContext context = ConnectContext.get(); // Same fallback as TimestampTzLiteral when parsing an unqualified value. this.timeZone = context == null ? "UTC" : context.getSessionVariable().getTimeZone(); + this.buildSortedRanges = context == null + || context.getSessionVariable().enableBinarySearchFilteringPartitions; int recordsHash = 1; for (Object partition : this.partitions) { recordsHash = 31 * recordsHash + (range ? partition.hashCode() : listRecordHash((ConnectorPartitionInfo) partition)); } - this.hash = Objects.hash(tableId, database, table, range, this.types, timeZone, version, recordsHash); + this.hash = Objects.hash(tableId, database, table, range, this.types, timeZone, + buildSortedRanges, version, recordsHash); } static Key versioned(long tableId, String database, String table, @@ -236,6 +246,10 @@ public String getTable() { return table; } + boolean buildSortedRanges() { + return buildSortedRanges; + } + @SuppressWarnings("unchecked") List listPartitions() { return (List) partitions; @@ -290,6 +304,7 @@ public boolean equals(Object other) { } Key that = (Key) other; if (hash != that.hash || tableId != that.tableId || range != that.range + || buildSortedRanges != that.buildSortedRanges || !Objects.equals(version, that.version) || !Objects.equals(database, that.database) || !Objects.equals(table, that.table) || !types.equals(that.types) || !timeZone.equals(that.timeZone) diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteAction.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteAction.java index 89e0ea8654e17e..eff54ad15ebc50 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteAction.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteAction.java @@ -128,7 +128,7 @@ public ResultSet execute(TableIf ignored) throws UserException { ConnectorSession session = catalog.buildConnectorSession(); ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); ConnectorTableHandle tableHandle = metadata - .getTableHandle(session, table.getRemoteDbName(), table.getRemoteName()) + .getTableHandleForDdl(session, table.getRemoteDbName(), table.getRemoteName()) .orElseThrow(() -> new AnalysisException("Table not found: " + table.getRemoteDbName() + "." + table.getRemoteName() + " in catalog " + catalog.getName())); @@ -156,6 +156,10 @@ public ResultSet execute(TableIf ignored) throws UserException { // here (engine half, no iceberg types); the converter is fail-loud, so an unrepresentable WHERE throws // rather than silently widening the rewrite scope. if (mode == ProcedureExecutionMode.DISTRIBUTED) { + // Distributed rewriting plans file scans; acquire the scan-capable read handle only for this mode. + tableHandle = metadata.getTableHandle(session, table.getRemoteDbName(), table.getRemoteName()) + .orElseThrow(() -> new AnalysisException("Table not found: " + table.getRemoteDbName() + + "." + table.getRemoteName() + " in catalog " + catalog.getName())); ConnectorPredicate loweredWhere = whereCondition.isPresent() ? UnboundExpressionToConnectorPredicateConverter.convert(whereCondition.get(), table) : null; diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/mvcc/PluginDrivenPartitionViewTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/mvcc/PluginDrivenPartitionViewTest.java index 28582ec048ff0e..ba3f97113044dc 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/mvcc/PluginDrivenPartitionViewTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/mvcc/PluginDrivenPartitionViewTest.java @@ -105,6 +105,32 @@ void mixedSpecDisplayPartitionsDoNotBuildAnOrderedIndex() { Assertions.assertTrue(table().partitionView(List.of(one), List.of(Type.INT)).getSortedRanges().isPresent()); } + @Test + void disabledBinarySearchDoesNotBuildOrReuseAnOrderedIndex() { + ConnectContext previous = ConnectContext.get(); + ConnectContext context = new ConnectContext(); + context.setThreadLocalInfo(); + try { + PluginDrivenMvccExternalTable table = table(); + List parts = List.of(partition("1", 10)); + context.getSessionVariable().enableBinarySearchFilteringPartitions = false; + PluginDrivenPartitionView withoutIndex = table.partitionView(parts, List.of(Type.INT)); + Assertions.assertTrue(withoutIndex.getSortedRanges().isEmpty()); + Assertions.assertSame(withoutIndex, table.partitionView(parts, List.of(Type.INT))); + + context.getSessionVariable().enableBinarySearchFilteringPartitions = true; + PluginDrivenPartitionView withIndex = table.partitionView(parts, List.of(Type.INT)); + Assertions.assertNotSame(withoutIndex, withIndex); + Assertions.assertTrue(withIndex.getSortedRanges().isPresent()); + Assertions.assertSame(withIndex, table.partitionView(parts, List.of(Type.INT))); + } finally { + ConnectContext.remove(); + if (previous != null) { + previous.setThreadLocalInfo(); + } + } + } + @Test void rangeViewAccountingDoesNotTraverseExpressionSuppliers() { PartitionKey lower = new PartitionKey(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteActionTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteActionTest.java index 52b4cc2e1a8da4..c61130a0d36b5c 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteActionTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteActionTest.java @@ -146,6 +146,8 @@ public void executeThreadsSessionAndHandleIntoConnectorAndWrapsResult() throws E Mockito.verify(f.procedureOps).execute(Mockito.eq(f.session), Mockito.eq(f.handle), Mockito.eq("rollback_to_snapshot"), Mockito.eq(f.props), Mockito.isNull(), Mockito.eq(Collections.emptyList())); + Mockito.verify(f.metadata, Mockito.never()).getTableHandle( + Mockito.any(), Mockito.anyString(), Mockito.anyString()); // The ConnectorProcedureResult was converted into a CommonResultSet (columns via ConnectorColumnConverter). List colNames = Arrays.asList(rs.getMetaData().getColumns().get(0).getName(), @@ -232,6 +234,9 @@ public void executeDistributedActionLowersWhereAndThreadsItToPlanRewrite() throw // no groups so the driver takes the early return without opening a transaction, and stub the connector's // result rendering (the result SHAPE is the connector's, so the mock must supply it). Fixture f = new Fixture(); + ConnectorTableHandle scanHandle = Mockito.mock(ConnectorTableHandle.class); + Mockito.when(f.metadata.getTableHandle(f.session, REMOTE_DB, REMOTE_TBL)) + .thenReturn(Optional.of(scanHandle)); Mockito.when(f.procedureOps.getExecutionMode("rewrite_data_files")) .thenReturn(ProcedureExecutionMode.DISTRIBUTED); Mockito.when(f.procedureOps.planRewrite(Mockito.any(), Mockito.any(), Mockito.anyString(), @@ -244,9 +249,10 @@ public void executeDistributedActionLowersWhereAndThreadsItToPlanRewrite() throw ConnectorExecuteAction action = new ConnectorExecuteAction("rewrite_data_files", f.props, Optional.empty(), Optional.of(where), f.table); ResultSet rs = action.execute(f.table); + Mockito.verify(f.metadata).getTableHandle(f.session, REMOTE_DB, REMOTE_TBL); ArgumentCaptor captor = ArgumentCaptor.forClass(ConnectorPredicate.class); - Mockito.verify(f.procedureOps).planRewrite(Mockito.any(), Mockito.any(), + Mockito.verify(f.procedureOps).planRewrite(Mockito.eq(f.session), Mockito.eq(scanHandle), Mockito.eq("rewrite_data_files"), Mockito.anyMap(), captor.capture(), Mockito.anyList()); ConnectorPredicate lowered = captor.getValue(); Assertions.assertNotNull(lowered, @@ -338,7 +344,7 @@ public void executeSelectsProcedureOpsByResolvedHandleNotNoArg() { @Test public void executeThrowsWhenTableHandleMissing() { Fixture f = new Fixture(); - Mockito.when(f.metadata.getTableHandle(Mockito.any(), Mockito.anyString(), Mockito.anyString())) + Mockito.when(f.metadata.getTableHandleForDdl(Mockito.any(), Mockito.anyString(), Mockito.anyString())) .thenReturn(Optional.empty()); ConnectorExecuteAction action = new ConnectorExecuteAction("rollback_to_snapshot", @@ -486,6 +492,8 @@ private static final class Fixture { Mockito.when(connector.getProcedureOps(Mockito.any())).thenReturn(procedureOps); Mockito.when(metadata.getTableHandle(Mockito.any(ConnectorSession.class), Mockito.anyString(), Mockito.anyString())).thenReturn(Optional.of(handle)); + Mockito.when(metadata.getTableHandleForDdl(Mockito.any(ConnectorSession.class), + Mockito.anyString(), Mockito.anyString())).thenReturn(Optional.of(handle)); TestPluginCatalog catalog = new TestPluginCatalog(connector, session); this.db = Mockito.mock(ExternalDatabase.class); From 4cedfb0c5b8518c5ab2412fd3b7fb3b5ea5e57fb Mon Sep 17 00:00:00 2001 From: 924060929 Date: Mon, 26 Oct 2026 12:54:59 +0800 Subject: [PATCH 5/5] [fix](fe) Keep Iceberg snapshot pins aligned with cached table generations ### What problem does this PR solve? Issue Number: None Problem Summary: A cold read of an Iceberg table retained a raw table without filling the separate latest-snapshot cache. After capacity eviction and an external append, a cold read could use the new table generation while the next warm read reused the old snapshot pin, hiding newly visible rows. For catalogs with a raw table cache, derive the pin from that same cached table generation. Keep the independent latest-snapshot cache only for catalogs without a raw table cache. Add a regression covering cold and warm reads around capacity eviction and an external append. ### Release note Fix stale Iceberg snapshot selection after table-cache eviction. ### Check List (For Author) - Test: Unit Test: IcebergTableHandleReuseTest (11) via run-fe-ut.sh; confirmed the new case fails on the previous implementation and passes after the fix. FE Maven package and Checkstyle passed via build.sh --fe; existing output plugin config caused the packaging step to exit 1. - Behavior changed: Yes, a warm read pins the snapshot from its retained raw table generation. - Does this need documentation: No --- .../iceberg/IcebergConnectorMetadata.java | 18 +++--- .../iceberg/IcebergTableHandleReuseTest.java | 58 +++++++++++++++++++ 2 files changed, 69 insertions(+), 7 deletions(-) diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java index 1b483f09a443c5..b8b415f6f88033 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java @@ -2713,13 +2713,13 @@ private List listPartitions(Table table, IcebergTableHan * snapshot id (or {@code -1} for an empty table — iceberg DOES support MVCC, so it still pins, mirroring * paimon), and the LATEST schema id (NOT {@code currentSnapshot().schemaId()} — a schema-only change without * a new snapshot advances the schema while the snapshot's id lags; legacy reads {@code table.schema() - * .schemaId()}). An already frozen statement read takes precedence over the shared pin. Otherwise T08 - * serves this through the per-catalog {@link IcebergLatestSnapshotCache}: within the TTL - * a HIT returns the cached pin without re-loading the table (saves the load I/O and keeps the snapshot - * STABLE across queries — the legacy with-cache catalog). The cached value carries BOTH ids atomically so a - * schema-only ALTER between two queries cannot skew snapshotId vs schemaId. A disabled cache - * ({@code meta.cache.iceberg.table.enable=false}, {@code ttl-second <= 0} or {@code capacity=0}) reads live - * every call (the legacy no-cache catalog). + * .schemaId()}). An already frozen statement read takes precedence. With a raw table cache, the pin comes + * from that same table generation: its eviction and the separate latest-snapshot cache can occur at + * different times, so consulting both could pair a current table with an older snapshot. Catalogs without + * a raw table cache use the per-catalog {@link IcebergLatestSnapshotCache}, which keeps a stable pin within + * its TTL. The pin carries BOTH ids atomically so a schema-only ALTER cannot skew snapshotId vs schemaId. + * A disabled cache ({@code meta.cache.iceberg.table.enable=false}, {@code ttl-second <= 0} or + * {@code capacity=0}) reads live every call (the legacy no-cache catalog). */ @Override public Optional beginQuerySnapshot( @@ -2736,6 +2736,10 @@ public Optional beginQuerySnapshot( // A newer shared pin must not point outside that frozen metadata, nor may this reader publish its // old pin over the refreshed cache entry. pin = latestSnapshotPin(loadTable(session, iceHandle)); + } else if (tableCache != null) { + // The table cache owns this generation. Its capacity eviction must evict the latest pin as well; + // the independently sized latestSnapshotCache cannot establish that relationship. + pin = loadLatestSnapshotPin(session, iceHandle); } else { pin = latestSnapshotCache != null ? latestSnapshotCache.getOrLoad(id, () -> loadLatestSnapshotPin(session, iceHandle)) diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableHandleReuseTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableHandleReuseTest.java index 1000582d2bbf8c..8a91fce6a5e75e 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableHandleReuseTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableHandleReuseTest.java @@ -332,6 +332,64 @@ void refreshedSharedSnapshotPinCannotOverrideAnAlreadyFrozenColdHandle() { } } + @Test + void warmSnapshotPinFollowsRawTableAfterCapacityEviction() { + InMemoryCatalog catalog = new InMemoryCatalog(); + catalog.initialize("eviction-pin", Collections.emptyMap()); + catalog.createNamespace(Namespace.of("db1")); + Table table = catalog.createTable(ID, SCHEMA); + append(table, "first.parquet"); + long firstSnapshot = table.currentSnapshot().snapshotId(); + TableIdentifier otherId = TableIdentifier.of("db1", "t2"); + Table other = catalog.createTable(otherId, SCHEMA); + RecordingIcebergCatalogOps ops = ops(); + ops.table = table; + IcebergTableCache cache = new IcebergTableCache(3600, 1); + IcebergLatestSnapshotCache snapshots = new IcebergLatestSnapshotCache(3600, 1); + IcebergConnectorMetadata metadata = new IcebergConnectorMetadata(ops, + IcebergCatalogProperties.of(Collections.emptyMap()), new RecordingConnectorContext(), snapshots, cache); + try { + TestStatementScope coldScope = new TestStatementScope(); + ConnectorSession cold = session("cold-a", coldScope); + ConnectorTableHandle coldHandle = metadata.getTableHandle(cold, "db1", "t1").orElseThrow(); + Assertions.assertEquals(firstSnapshot, + metadata.beginQuerySnapshot(cold, coldHandle).orElseThrow().getSnapshotId()); + coldScope.closeAll(); + + TestStatementScope warmScope = new TestStatementScope(); + ConnectorSession warm = session("warm-a", warmScope); + ConnectorTableHandle warmHandle = metadata.getTableHandle(warm, "db1", "t1").orElseThrow(); + Assertions.assertEquals(firstSnapshot, + metadata.beginQuerySnapshot(warm, warmHandle).orElseThrow().getSnapshotId()); + warmScope.closeAll(); + + ops.table = other; + TestStatementScope otherScope = new TestStatementScope(); + metadata.getTableHandle(session("cold-b", otherScope), "db1", "t2").orElseThrow(); + otherScope.closeAll(); + Assertions.assertFalse(cache.contains(ID)); + + append(table, "second.parquet"); + long secondSnapshot = table.currentSnapshot().snapshotId(); + ops.table = catalog.loadTable(ID); + TestStatementScope refreshedScope = new TestStatementScope(); + ConnectorSession refreshed = session("cold-a-again", refreshedScope); + ConnectorTableHandle refreshedHandle = metadata.getTableHandle(refreshed, "db1", "t1").orElseThrow(); + Assertions.assertEquals(secondSnapshot, + metadata.beginQuerySnapshot(refreshed, refreshedHandle).orElseThrow().getSnapshotId()); + refreshedScope.closeAll(); + + TestStatementScope repeatScope = new TestStatementScope(); + ConnectorSession repeat = session("warm-a-again", repeatScope); + ConnectorTableHandle repeatHandle = metadata.getTableHandle(repeat, "db1", "t1").orElseThrow(); + Assertions.assertEquals(secondSnapshot, + metadata.beginQuerySnapshot(repeat, repeatHandle).orElseThrow().getSnapshotId()); + repeatScope.closeAll(); + } finally { + cache.close(); + } + } + private static void append(Table table, String path) { table.newAppend().appendFile(DataFiles.builder(table.spec()).withPath(path).withFormat("PARQUET") .withFileSizeInBytes(10).withRecordCount(1).build()).commit();