Skip to content

feat(api): add URL-pipeline-native create_v3_array, create_v3_group, create_v2_array and create_v2_group - #8

Open
d-v-b wants to merge 7 commits into
jhamman:feature/url-pipeline-corefrom
d-v-b:feature/create-v3
Open

d-v-b wants to merge 7 commits into
jhamman:feature/url-pipeline-corefrom
d-v-b:feature/create-v3

Conversation

@d-v-b

@d-v-b d-v-b commented Oct 6, 2026

Copy link
Copy Markdown

🤖 AI text below 🤖

Added zarr.create_v3_array, zarr.create_v3_group, zarr.create_v2_array and zarr.create_v2_group: creation functions that are native to URL pipelines and to one Zarr format each. Their keyword arguments are that format's metadata fields (for V3 shape, data_type, chunk_grid, codecs, chunk_key_encoding, fill_value, attributes, dimension_names, storage_transformers; for V2 shape, dtype, chunks, fill_value, order, dimension_separator, compressor, filters, attributes), with no translation layer between formats. Fields left at the new zarr.AUTO sentinel are computed from the shape, the data type and the configuration, while None keeps the meaning the metadata gives it (for example no compressor). A string location is always a URL pipeline such as "file:/data/example.zarr|zarr3:group/array", so its root must carry a scheme and a literal | in a local path is spelled %7C; a URLPipeline, a Store or a StorePath is accepted as well. Creation fails if a node already exists at the location unless overwrite=True.

This is the creation-side counterpart of the read-only zarr.open_url introduced in #7, and it stacks on that PR: until #7 merges, this diff includes its commits as well, and only the three commits listed in the notes below are new here. The same change is open against the feature/url-pipeline-object branch alone as d-v-b#394. The agent prepared it at d-v-b's direction, with the design settled in discussion beforehand.

Design

  • One function per format, metadata fields as keyword arguments. create_array abstracts over Zarr formats, and the price is parameters like compressors, serializer, filters and shards that mean different things per format or exist in only one. These functions build ArrayV3Metadata / ArrayV2Metadata (or GroupMetadata) directly from their arguments and write it; there is nothing to translate.
  • A single AUTO sentinel for every derived default. Every optional metadata argument whose default is computed (chunk grid from shape and dtype, codecs and compressor from the config, fill value from the dtype, V2 order from array.order) defaults to zarr.AUTO, a typing_extensions.Sentinel used in type position. None is never overloaded: on V2 it still means a null fill value, no compressor, or no filters. No "auto" strings appear in the new signatures. fill_value="auto" being a valid string fill value is the concrete reason an object sentinel is used.
  • A string is always a pipeline. The location is str | URLPipeline | Store | StorePath. Strings are read as URL pipelines whose root must carry a scheme, so | is unambiguous and %7C is the escape. To make that escape work, file: and memory: pipeline roots now percent-decode their bodies, as any URL does; adapter segments still receive their bodies verbatim. A format segment that contradicts the function (zarr2: for create_v3_array) raises. Store and StorePath remain accepted because a URL cannot name an arbitrary Python store object.
  • Creation never opens. overwrite=False (the default) raises ContainsArrayError / ContainsGroupError on an existing node; overwrite=True deletes it first. There is no mode string.

Also in this PR

  • mypy is bumped from 2.3.1 to 2.4.0, the first release that accepts a PEP 661 sentinel in type position (pyright has supported it since 1.1.414).
  • The _Unset enum in PipelineContext.resolve_preceding becomes a Sentinel for the same reason, and the creation functions drop a scheme check that URLPipeline.from_url now performs itself.

Not in this PR

  • Migrating create_array and friends to AUTO, and aliasing DEFAULT_FILL_VALUE to it. Both are additive follow-ups once this shape has settled.
  • A single create_at_url was considered and rejected: creation always knows whether it is making an array or a group, and one function could only learn that from its arguments, which is the hidden dispatch open_url was reshaped to avoid.
Stack and review notes

🤖 Generated with Claude Code

d-v-b and others added 7 commits October 6, 2026 10:40
…never from plain strings

Strings passed as a StoreLike keep their pre-existing meaning: a local
path containing `|` still names a directory, whatever adapters are
installed. A pipeline is opened either through the new `zarr.open_url`,
or by passing a `zarr.storage.URLPipeline` (built with `from_url`) to
any function that accepts a StoreLike.

- `URLPipeline` holds every parsed segment; a trailing `zarr2:`/`zarr3:`
  format segment is read from it (`zarr_format`, `path`, `store_segments`)
  instead of through pseudo-adapters, so `AdapterResolution.zarr_format`
  is gone.
- `StorePath` is back to its state on main: no `zarr_format` attribute,
  and mode "a" on a read-only store raises again. The open-or-create
  downgrade applies only to pipelines that resolve read-only.
- The format selected by a pipeline is merged with the caller's
  `zarr_format` by `resolve_zarr_format(store, zarr_format)` at each
  call site, before the store is resolved. `create_array` and
  `Group.from_store` default to 3 again; `Array.open` keeps its new
  `zarr_format` parameter.
- `is_url_pipeline` and the `pathlib.Path` escape hatch are removed;
  `resolve_pipeline` accepts a string or a `URLPipeline`, and a single
  URL is a trivial pipeline.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`zarr.open_url(url, *, mode="r" | "r+", storage_options=None)` no longer
mirrors `zarr.open`: it never creates or overwrites, takes no `path`
(the node is addressed by the URL, via the format segment's body), no
`zarr_format` (taken from the pipeline or inferred), and no `**kwargs`.
Creating a node at a pipeline-addressed location is done by passing a
`URLPipeline` to `create_array` / `create_group`.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…solver

With strings never routed as pipelines, four pieces of the resolver only
served the old design:

- The schemeless-root extension (`data.zip|zip:`). The root of a pipeline
  now carries a URL scheme, as the specification requires; a bare path is
  rejected with a message pointing at `file:` URLs and `%7C`. This also
  removes the one place where `?` and `#` were treated as filename
  characters.
- The negative lookahead that kept fsspec's `zip::...` chained URLs out of
  root-scheme detection; it protected `zarr.open` strings that no longer
  reach the resolver.
- The mode "a" downgrade for read-only pipelines in `make_store_path`;
  `open_url` defaults to "r", and a read-only pipeline now raises on "a"
  exactly like a plain read-only store.
- The "non-dict `storage_options` reserved for per-segment configuration"
  reservation in docstrings and the user guide.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… root scheme detection

`URLPipeline.resolve(mode=..., storage_options=...)` replaces the
module-level `resolve_pipeline`, and `parse_pipeline` leaves the public
`zarr.storage` surface: the object is the one entry point for parsing
and resolving a pipeline.

Root scheme detection no longer consults `parse_store_url` for
single-letter candidates. A letter followed by `:` is a URL scheme on
every platform, so parsing is deterministic across operating systems; a
Windows drive path is spelled `file:/C:/...` in a pipeline, as the root
must carry a scheme anyway.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ve creation functions

Their keyword arguments are the Zarr V3 metadata fields (`shape`,
`data_type`, `chunk_grid`, `codecs`, `chunk_key_encoding`, `fill_value`,
`attributes`, `dimension_names`, `storage_transformers`) with no
translation layer between formats. Fields left at the new `zarr.AUTO`
sentinel (a `typing_extensions.Sentinel`, used in type position) are
computed from the shape, the data type and the configuration; `None`
keeps whatever meaning the metadata gives it.

The location is `str | URLPipeline | Store | StorePath`. A string is
always a URL pipeline whose root must carry a scheme, so `|` is
unambiguous and a literal `|` in a local path is spelled `%7C`. To make
that escape work, `file:` and `memory:` pipeline roots now percent-decode
their bodies, as any URL does; adapter segments are still passed
verbatim. A `zarr2:` segment raises. Creation fails on an existing node
unless `overwrite=True`.

mypy is bumped from 2.3.1 to 2.4.0, the first release that accepts a
PEP 661 sentinel in type position.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Zarr format 2 counterparts of `create_v3_array` / `create_v3_group`.
Their keyword arguments are the V2 metadata fields (`shape`, `dtype`,
`chunks`, `fill_value`, `order`, `dimension_separator`, `compressor`,
`filters`, `attributes`). `AUTO` computes a field from the data type,
the shape and the configuration (`array.order`, the default compressor,
the data type's object codec as default filter), while `None` keeps the
meaning V2 gives it: a `null` fill value, no compressor, no filters.
Rectilinear chunk specifications are rejected. A `zarr3:` segment in the
location raises.

The location resolver shared with the V3 functions now takes the format
it must agree with.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s; drop a redundant scheme check

`PipelineContext.resolve_preceding` marked "not overridden" with a
single-member enum, a workaround from when pyright and mypy could not
type a PEP 661 sentinel in type position. Both can now (pyright 1.1.414,
mypy 2.4), so it is a `typing_extensions.Sentinel` like `AUTO`.

`_resolve_creation_location` no longer re-checks that a string's root
carries a scheme: `URLPipeline.from_url` rejects a schemeless root
itself since the base branch dropped the bare-path extension.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant