[core][rest][spark] Add explicit Format Table partition locations

[sundapeng]

Purpose

As a follow-up to #8750 and #8751, this adds Spark's standard explicit-location
form for registering existing data in a catalog-managed Format Table:

ALTER TABLE t ADD PARTITION (dt = '2026-09-02')
LOCATION 's3://bucket/existing/dt=2026-09-02';

For ADD PARTITION ... LOCATION, Spark canonicalizes and validates the absolute
URI before registration. It rejects roots, table ancestors, and the partition's
derived default directory, registers the location as partition metadata without
creating or probing the supplied directory, and verifies that the REST server
stored and echoed the same canonical value. Partitions added without LOCATION
keep their current derived-path behavior.

When explicit locations may exist, reads and mutation preflights validate the
complete catalog registry as one path-ownership set. Duplicate, ancestor,
descendant, table-root, and derived-path conflicts fail closed, as does a mismatch
between the authoritative explicit-location count and the listed registry.
Locations outside the table root use a FileIO resolved from the client's
CatalogContext; locations under the table root continue using the table's
FileIO.

Data in an explicitly located partition is read-only to Paimon. Write commits,
overwrites, truncates, and ANALYZE TABLE operations whose scope includes such a
partition fail before publishing or deleting files or mutating catalog metadata.
DROP PARTITION only unregisters it and never probes or deletes the explicit URI.
MSCK REPAIR TABLE leaves the registration intact and does not inspect or collect
statistics from that location.

The REST contract uses a dedicated partitions/with-locations resource and the
format-table-partition-location-v1 capability token. A location-aware POST is
refused unless the server advertises the capability. Spark REST requests advertise
the client capability; Flink requests intentionally do not. The existing partition
resource and location-free payloads remain compatible.

This change is limited to catalog-managed Format Tables; it does not add
multi-location storage to native Paimon tables or change the table-level
SET LOCATION rejection from #9435.

Tests

• 514 tests in 14 focused paimon-api and paimon-core suites covering JSON and
  REST compatibility, catalog delegation and caching, partition management, scans,
  reads, commits, statistics, FileIO routing, registry consistency, and path ownership.
• Spark 3: 96 focused Scala tests, 19 FormatTablePartitionRepairTest tests, and
  2 REST capability-header tests.
• Spark 4: 83 focused Scala tests and 2 REST capability-header tests.
• Flink REST compatibility integration tests: 2.
• Checkstyle, Spotless, Maven Enforcer, and RAT.
• JDK 8 clean package.
• REST OpenAPI validation: 57 Catalog operations and 6 Management operations.
• Docusaurus production build.

[JingsongLi]

Requirement fit: SUPPORTED
Implementation: FINDINGS

[JingsongLi]

[P1] Validate the prospective registry atomically before registering locations.

This path validates each explicit location only against the table root and the default directory of that same spec, then immediately mutates the catalog. A batch can therefore assign the same or nested LOCATION values to different specs; MSCK can similarly add a default spec whose directory is already owned by another explicit spec. Registration succeeds, but scans, writes, and DROP later reject the full registry as overlapping, so normal SQL cannot even unregister the bad entry. Please make the catalog/REST mutation atomically validate the complete post-update ownership set for both ordinary create and /with-locations; a client-side listing alone would still race concurrent ADDs.

[JingsongLi]

[P1] Canonicalize the filesystem identity before comparing ownership.

The ownership key uses the raw lower-cased scheme and authority, although different URI spellings can resolve to the same filesystem. For example, Hadoop can resolve hdfs://nn and hdfs://nn:8020 to the same NameNode, but this code treats them as separate roots. A table at hdfs://nn:8020/warehouse/t can therefore accept an explicit location such as hdfs://nn/warehouse/t, bypassing the table-root check and potentially reading the whole table as one partition or writing through a default sibling into explicitly owned data. Please derive the key from the resolved/canonical FileSystem URI, including default ports and configured aliases, and use that identity consistently for overlap checks and FileIO routing.

[JingsongLi]

[P1] Do not reject URI forms used by supported filesystems.

These checks reject hdfs:///path because it has no authority, even though Paimon documents that form and HadoopFileIO can resolve it through the catalog Hadoop configuration. They also reject the standard ABFS form abfs://filesystem@account.dfs.core.windows.net/path, because java.net.URI parses filesystem as user info even though it is the Azure filesystem/container name rather than a credential. As a result, explicit partition locations are unusable for common HDFS deployments and standard ADLS Gen2 addresses. Please make validation scheme/FileIO-aware: qualify authorityless HDFS through CatalogContext, recognize the structural ABFS authority, and continue rejecting actual embedded credentials.

[JingsongLi]

[P1] Preserve the documented unknown-count fallback.

The OpenAPI description explicitly allows explicitPartitionLocationCount to be omitted, and Catalog#getExplicitPartitionLocationCount defines an empty result as a signal to load and validate the full registry. Requiring the count whenever the server advertises the location capability contradicts both contracts: a location-capable provider that cannot supply this optional optimization will make every getTable and table-detail conversion fail before any fallback can run. Please accept null as OptionalLong.empty() and use the existing full-registry path, or advertise count support through a separate capability that makes the field mandatory.

[JingsongLi]

[P2] Validate the full registry before scoped ANALYZE.

This query is already prefix-pruned, so checking location only on the returned partitions does not validate global path ownership or the authoritative explicit-location count. If selected default partition A has a directory that is also owned by an unselected explicit partition B, this guard passes and FormatTablePartitionStatsCollector reads that directory as A, probing explicitly owned data and replacing A statistics with B data. That state is reachable through the current ADD/MSCK registration gap or an inconsistent provider. Please run the same count-aware full-registry preflight used by scans and commits, then apply the ANALYZE prefix and reject selected explicit partitions.