[HWORKS-2807] Native Partitioning, Liquid Clustering, Z-Ordering - #617
Open
jimdowling wants to merge 6 commits into
Open
[HWORKS-2807] Native Partitioning, Liquid Clustering, Z-Ordering#617jimdowling wants to merge 6 commits into
jimdowling wants to merge 6 commits into
Conversation
… hidden partitioning, Delta liquid clustering, z-ordering https://hopsworks.atlassian.net/browse/HWORKS-2807 Reinterpret partitioned_by as a list of native partition transform expressions instead of the materialized time-grain columns shipped under HWORKS-2802, with each layout axis its own parameter: partitioned_by (Iceberg partition spec; Hudi grains), clustered_by (Delta liquid clustering), bucket_index (Hudi), zorder_by, and sort_order (Iceberg). The docs gain a "Partitioning and Clustering" user guide covering the transform grammar with partition-field aliases (Iceberg-only, unique field names), the explicit identity form for grain-named columns, and per-format availability, hidden partitioning on Iceberg with a day + bucket example, the persistent Iceberg sort_order (mutually exclusive with zorder_by), liquid clustering on Delta with the Spark-only writer warning and the auto-widened delta.dataSkippingNumIndexedCols note, a catalog-capability matrix for the Iceberg operations, the rewrite_all=False optimize default with the explicit initial full z-order and the Iceberg where filter, the Hudi grain columns and bucket index (engine "simple" only; hoodie.bucket.index.* options and a partition-level bucket index settable directly through write_options), zorder_by with an hour + z-order example, the per-format optimize() behavior including the Delta full/zorder combination rules, layout evolution (update_partition_spec with actual-spec persistence, the partition_key restriction, and the no-arg metadata re-sync; update/disable_clustering), layout introspection (the format-gated get_* methods and describe_layout, with the user-catalog caveat), the migration note for the removed bare-grain form including the explicit write rejection, and the stream/online restrictions. The create guide's table-format list gains ICEBERG and its partition-key section links to the new guide. Merge after the hopsworks-api branch: the API cross-references (FeatureGroup.optimize) resolve against hopsworks-api@main in the docs build. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMpkURDybXS7BnP4C6eMsW
…io into HWORKS-2807-partition-transforms # Conflicts: # docs/user_guides/fs/feature_group/create.md
…data https://hopsworks.atlassian.net/browse/HWORKS-2807 Adds a "Layout for point-in-time training data" subsection to the partitioning guide: day(event_ts) pruning engages through the PIT join when a lookback (or event-time filter) bounds the history, bucket(N, pk) plus a key sort order serve the key-equality join, appends preserve the clustering that upserts discard, and partition sizing guidance (day x bucket combinations become files, so match the granularity to volume). Backed by an in-cluster verification of the generated PIT plans showing the bound pushed into the Iceberg BatchScan for both mechanisms. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UMpkURDybXS7BnP4C6eMsW
…eoil https://hopsworks.atlassian.net/browse/HWORKS-2807 Two Python blocks in the partitioning guide ended in blank lines that snakeoil strips, and CI enforces its output with git diff --exit-code, so the committed text has to be the normalised form. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The changes are limited to documentation and navigation wiring, with only a minor Markdown style issue noted.
Pull request overview
Adds documentation for the revamped Feature Group layout APIs (native partition transforms, Delta liquid clustering, Hudi bucket index, z-ordering, and Iceberg sort order), and wires the new guide into the Feature Group user-guide navigation.
Changes:
- Adds a new “Partitioning and Clustering” user guide for Feature Groups covering format-specific layout mechanisms and APIs.
- Updates the Feature Group “Create” guide to point readers to the new dedicated partitioning/clustering guide (and removes the older time-grain
partitioned_bysection). - Updates
mkdocs.ymlnav to include the new page.
File summaries
| File | Description |
|---|---|
| mkdocs.yml | Adds the new Feature Group “Partitioning and Clustering” page to the docs navigation. |
| docs/user_guides/fs/feature_group/partitioning.md | New comprehensive guide documenting partition transforms, clustering, z-ordering, optimize/evolution/introspection APIs, and restrictions. |
| docs/user_guides/fs/feature_group/create.md | Replaces the removed legacy partitioned_by section with a pointer to the new guide. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+235
to
+236
| The Iceberg feature is fully supported on the default path-based (`HadoopTables`) layout. Some operations are not yet available when the table is backed by an external catalog: | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JIRA: https://hopsworks.atlassian.net/browse/HWORKS-2807
Redesigns
partitioned_byfrom materialized time-grain columns (HWORKS-2802) into native partition transform expressions, and splits each layout axis into its own creation parameter so no format accepts and erases a mechanism it cannot express:partitioned_by: native partition spec from transforms (identity,bucket,truncate,year/month/day/hour,weekHudi-only,voidIceberg-only), optionally aliased. Iceberg hidden partitioning; Hudi identity + grains; rejected on Delta.clustered_by: Delta liquid clustering (Spark-only writes; statistics widened when a clustering column sits past the indexed range).bucket_index: Hudi bucket index (engine="simple").zorder_by: z-order columns (Iceberg optimize rewrite; Hudi inline clustering).sort_order: persistent Iceberg write sort order (mutually exclusive withzorder_by).Also adds layout evolution (
update_partition_spec/update_clustering/disable_clustering), actual-table introspection (get_partition_spec(s)/get_sort_order/get_clustering_columns/describe_layout), and richeroptimize()(strategy, columns, rewrite_all, target_file_size_mb, where, full) returning rewrite metrics. Every layout column reference is canonicalized to the sanitized feature name. Breaking the bare-grain form is accepted: it is rejected with a migration hint, and writes to pre-transform feature groups fail explicitly.Coordinated change across hopsworks-api, hopsworks-ee, loadtest, and logicalclocks.github.io. Cluster-verified on jim-partitioned-by: stage A REST matrix 46/46 and stage B in-cluster Spark job 38/38.
In this repository (logicalclocks.github.io): a new "Partitioning and Clustering" user guide, plus
create.mdandmkdocs.ymlupdates.Status
The code landed on 2026-07-17: hopsworks-api#1060, hopsworks-ee#3172 and loadtest#957 are merged, so the
FeatureGroup.optimizeand layout API cross-references now resolve againsthopsworks-api@main.Brought up to date with
mainon 2026-09-05. The strict build (hopsworks-docs check, against the currenthopsworks-apimain) passes,markdownlintpasses, and the two code blocks snakeoil normalised are committed in that form so itsgit diff --exit-codegate passes.🤖 Generated with Claude Code