docs: add spec-style description of the consolidated metadata format by d-v-b · Pull Request #4283 · zarr-developers/zarr-python

d-v-b · 2026-08-25T08:11:33Z

Summary

This is a claude-authored contribution that adds specification-style documentation for how zarr-python implements consolidated metadata. I'm pretty busy these days with a newborn baby so I can't give this careful review. I am opening this as a draft and leaving it to other folks to push it forward.

cc @normanrz

🤖 AI text below 🤖

Add a new user-guide page that describes exactly what zarr-python reads and writes for consolidated metadata in Zarr formats 2 and 3, so that other implementations can interoperate. Quotes and attributes the schema text from zarr-specs#309 (Tom Augspurger, CC-BY-4.0), documents the key ordering, the empty child-group marker, the v2 .zmetadata layout and its deviation from zarr-python 2.x, and the reader/writer procedures.

Also correct the 3.1.1 sort-order note on the existing page, which said "lexicographic" while the implementation uses NFKC-casefolded ordering.

Assisted-by: ClaudeCode:claude-fable-5

Author attestation

I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct.

TODO

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/user-guide/*.md
Changes documented as a new file in changes/
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

Add a new user-guide page that describes exactly what zarr-python reads and writes for consolidated metadata in Zarr formats 2 and 3, so that other implementations can interoperate. Quotes and attributes the schema text from zarr-specs#309 (Tom Augspurger, CC-BY-4.0), documents the key ordering, the empty child-group marker, the v2 `.zmetadata` layout and its deviation from zarr-python 2.x, and the reader/writer procedures. Also correct the 3.1.1 sort-order note on the existing page, which said "lexicographic" while the implementation uses NFKC-casefolded ordering. Assisted-by: ClaudeCode:claude-fable-5

d-v-b · 2026-08-25T08:12:25Z

@TomAugspurger you should probably also have a look, since this is an LLM summarizing your work

TomAugspurger · 2026-08-25T19:39:42Z

I'm pretty busy these days with a newborn baby so I can't give this careful review.

Congrats! I'll take a look when I get a chance.

Do you have any more info on the motivation for this document. I gather it's primarily for other implementations looking to interoperate with what we write, and am trying to balance this approach vs. telling them to read the spec + source code :)

normanrz · 2026-08-25T19:45:01Z

I asked for a spec on Zulip and Davis was kind enough to create one. The motivation is that we picked up the consolidated metadata in the newly-formed Zarr Format Working Group and I wanted to get a better understanding of the current zarr-python implementation. Having a spec here would be great, but even better would be if we could eventually bring this document into the Zarr 3 spec.

d-v-b · 2026-08-26T09:43:06Z

Do you have any more info on the motivation for this document. I gather it's primarily for other implementations looking to interoperate with what we write, and am trying to balance this approach vs. telling them to read the spec + source code :)

IMO we don't really have a place for a consolidated metadata spec in the zarr-specs repo until we unblock the blockage that prevented your original PR from getting merged in some form or another. For folks curious about the normative structure of consolidated metadata, but also zarr-python's particular implementation choices, I think some kind of document in our docs is a good play. Long term our goal should be to replace an actual spec with a link to a spec defined elsewhere.

normanrz · 2026-08-26T12:56:11Z

IMO we don't really have a place for a consolidated metadata spec in the zarr-specs repo until we unblock the blockage that prevented your original PR from getting merged in some form or another.

Right. With @joshmoore, I am working on a proposal for the ZFWG governance and I could imagine using consolidated metadata as testcase for the governance process.

TomAugspurger

Thanks for the context. With that in mind, I'll view this as a somewhat temporary document. Though just merging the zarr-specs PR feels infinitely better.

And for those interested in the implementation, rereading the discussion in #2113 might also be fruitful.

TomAugspurger · 2026-08-30T19:59:07Z

 Consolidated metadata essentially stores all the metadata for a hierarchy in the
 metadata of the root Group.

+This page describes how to use consolidated metadata from Python. For a precise


I have some hesitancy about linking to this document from user-facing docs. I think the spec (or the PR for the spec) should be sufficient for users, and if it isn't then the docs should be improved there.

TomAugspurger · 2026-08-30T20:01:08Z

+    of strings joined by `"/"`. For keys with the same depth, the tie is broken by
+    comparing the paths after Unicode NFKC normalization and case-folding. This
+    behavior ensures deterministic metadata output for a given group.


This seems worse for causal readers.

Let's at least keep "lexicographic" (since it isn't incorrect, right?) And if we want to be more specific we can, but let's link to the Python docs on normalization.

TomAugspurger · 2026-08-30T20:02:38Z

+
+## Concepts
+
+A hierarchy is **consolidated at** a group (the *consolidating group*). The


"consolidating group" isn't a term in zarr-developers/zarr-specs#309. If we want to make it one, let's propose it there.

TomAugspurger · 2026-08-30T20:05:03Z

+    (`GroupMetadata.consolidated_metadata.metadata`). It is never written to a
+    store and is mentioned here only because it leaks into the on-disk form in
+    one place (the [empty child marker](#child-groups-carry-an-empty-marker)).
+


I don't think "leaks into" is accurate here, or it conveys the wrong impression. That's deliberate, saying that there aren't any children.

TomAugspurger · 2026-08-30T20:05:32Z

+
+## Paths
+
+A **path** is the name of a node relative to the consolidating group:


IIRC paths are already defined in the spec. Link to that.

TomAugspurger · 2026-08-30T20:07:36Z

+at `B` produces `C`, `y`; consolidating at `C` produces no paths (an empty
+mapping, which is still written).
+
+!!! note "Difference from the zarr-specs#309 text"


Ha, I had a review pending from April 2025 pointing out this issue. Submitted that: zarr-developers/zarr-specs#309 (comment) and we can fix it in the spec.

TomAugspurger · 2026-08-30T20:09:36Z

+Each value is a complete node metadata document, i.e. exactly what would be
+found in that node's own `zarr.json`, with the following rules:
+
+* The document MUST contain `zarr_format`. Readers discriminate on this first;


I'm guessing this isn't intended / well-tested.

TomAugspurger · 2026-08-30T20:11:24Z

+This is the one place the nested in-memory form shows through. The marker
+does **not** mean the child group has no children: the child's descendants are
+still listed in the flat mapping at the consolidating group. It exists so


This is wrong (or I'm misunderstanding something). But AFAIK the presence of metadata: {} definitely does indicate a group with no children.

Co-authored-by: Tom Augspurger <tom.augspurger88@gmail.com>

codecov · 2026-08-30T20:32:59Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.21%. Comparing base (5b5f3a3) to head (c1bae56).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4283   +/-   ##
=======================================
  Coverage   94.21%   94.21%           
=======================================
  Files          92       92           
  Lines       12861    12861           
=======================================
  Hits        12117    12117           
  Misses        744      744

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions Bot added the needs release notes Automatically applied to PRs which haven't added release notes label Aug 25, 2026

Merge branch 'main' into claude/docs-consolidated-metadata-spec-6f3c45

95831b2

Merge branch 'main' into claude/docs-consolidated-metadata-spec-6f3c45

c7255d9

TomAugspurger reviewed Aug 30, 2026

View reviewed changes

d-v-b and others added 2 commits August 30, 2026 22:22

Update docs/user-guide/consolidated_metadata_format.md

2fee4a8

Co-authored-by: Tom Augspurger <tom.augspurger88@gmail.com>

Update docs/user-guide/consolidated_metadata_format.md

9c17bda

Co-authored-by: Tom Augspurger <tom.augspurger88@gmail.com>

Merge branch 'main' into claude/docs-consolidated-metadata-spec-6f3c45

c1bae56


		## Concepts

		A hierarchy is consolidated at a group (the consolidating group). The


		## Paths

		A path is the name of a node relative to the consolidating group:

Uh oh!

Uh oh!

Conversation

d-v-b commented Aug 25, 2026

Summary

Author attestation

TODO

Uh oh!

d-v-b commented Aug 25, 2026

Uh oh!

TomAugspurger commented Aug 25, 2026

Uh oh!

normanrz commented Aug 25, 2026

Uh oh!

d-v-b commented Aug 26, 2026

Uh oh!

normanrz commented Aug 26, 2026

Uh oh!

TomAugspurger left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Aug 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Aug 30, 2026 •

edited

Loading