Repository navigation
feat(api): add URL-pipeline-native create_v3_array, create_v3_group, create_v2_array and create_v2_group - #8
Open
d-v-b wants to merge 7 commits into
Conversation
…never from plain strings Strings passed as a StoreLike keep their pre-existing meaning: a local path containing `|` still names a directory, whatever adapters are installed. A pipeline is opened either through the new `zarr.open_url`, or by passing a `zarr.storage.URLPipeline` (built with `from_url`) to any function that accepts a StoreLike. - `URLPipeline` holds every parsed segment; a trailing `zarr2:`/`zarr3:` format segment is read from it (`zarr_format`, `path`, `store_segments`) instead of through pseudo-adapters, so `AdapterResolution.zarr_format` is gone. - `StorePath` is back to its state on main: no `zarr_format` attribute, and mode "a" on a read-only store raises again. The open-or-create downgrade applies only to pipelines that resolve read-only. - The format selected by a pipeline is merged with the caller's `zarr_format` by `resolve_zarr_format(store, zarr_format)` at each call site, before the store is resolved. `create_array` and `Group.from_store` default to 3 again; `Array.open` keeps its new `zarr_format` parameter. - `is_url_pipeline` and the `pathlib.Path` escape hatch are removed; `resolve_pipeline` accepts a string or a `URLPipeline`, and a single URL is a trivial pipeline. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`zarr.open_url(url, *, mode="r" | "r+", storage_options=None)` no longer mirrors `zarr.open`: it never creates or overwrites, takes no `path` (the node is addressed by the URL, via the format segment's body), no `zarr_format` (taken from the pipeline or inferred), and no `**kwargs`. Creating a node at a pipeline-addressed location is done by passing a `URLPipeline` to `create_array` / `create_group`. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…solver With strings never routed as pipelines, four pieces of the resolver only served the old design: - The schemeless-root extension (`data.zip|zip:`). The root of a pipeline now carries a URL scheme, as the specification requires; a bare path is rejected with a message pointing at `file:` URLs and `%7C`. This also removes the one place where `?` and `#` were treated as filename characters. - The negative lookahead that kept fsspec's `zip::...` chained URLs out of root-scheme detection; it protected `zarr.open` strings that no longer reach the resolver. - The mode "a" downgrade for read-only pipelines in `make_store_path`; `open_url` defaults to "r", and a read-only pipeline now raises on "a" exactly like a plain read-only store. - The "non-dict `storage_options` reserved for per-segment configuration" reservation in docstrings and the user guide. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… root scheme detection `URLPipeline.resolve(mode=..., storage_options=...)` replaces the module-level `resolve_pipeline`, and `parse_pipeline` leaves the public `zarr.storage` surface: the object is the one entry point for parsing and resolving a pipeline. Root scheme detection no longer consults `parse_store_url` for single-letter candidates. A letter followed by `:` is a URL scheme on every platform, so parsing is deterministic across operating systems; a Windows drive path is spelled `file:/C:/...` in a pipeline, as the root must carry a scheme anyway. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ve creation functions Their keyword arguments are the Zarr V3 metadata fields (`shape`, `data_type`, `chunk_grid`, `codecs`, `chunk_key_encoding`, `fill_value`, `attributes`, `dimension_names`, `storage_transformers`) with no translation layer between formats. Fields left at the new `zarr.AUTO` sentinel (a `typing_extensions.Sentinel`, used in type position) are computed from the shape, the data type and the configuration; `None` keeps whatever meaning the metadata gives it. The location is `str | URLPipeline | Store | StorePath`. A string is always a URL pipeline whose root must carry a scheme, so `|` is unambiguous and a literal `|` in a local path is spelled `%7C`. To make that escape work, `file:` and `memory:` pipeline roots now percent-decode their bodies, as any URL does; adapter segments are still passed verbatim. A `zarr2:` segment raises. Creation fails on an existing node unless `overwrite=True`. mypy is bumped from 2.3.1 to 2.4.0, the first release that accepts a PEP 661 sentinel in type position. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Zarr format 2 counterparts of `create_v3_array` / `create_v3_group`. Their keyword arguments are the V2 metadata fields (`shape`, `dtype`, `chunks`, `fill_value`, `order`, `dimension_separator`, `compressor`, `filters`, `attributes`). `AUTO` computes a field from the data type, the shape and the configuration (`array.order`, the default compressor, the data type's object codec as default filter), while `None` keeps the meaning V2 gives it: a `null` fill value, no compressor, no filters. Rectilinear chunk specifications are rejected. A `zarr3:` segment in the location raises. The location resolver shared with the V3 functions now takes the format it must agree with. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s; drop a redundant scheme check `PipelineContext.resolve_preceding` marked "not overridden" with a single-member enum, a workaround from when pyright and mypy could not type a PEP 661 sentinel in type position. Both can now (pyright 1.1.414, mypy 2.4), so it is a `typing_extensions.Sentinel` like `AUTO`. `_resolve_creation_location` no longer re-checks that a string's root carries a scheme: `URLPipeline.from_url` rejects a schemeless root itself since the base branch dropped the bare-path extension. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
4 of 7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Added
zarr.create_v3_array,zarr.create_v3_group,zarr.create_v2_arrayandzarr.create_v2_group: creation functions that are native to URL pipelines and to one Zarr format each. Their keyword arguments are that format's metadata fields (for V3shape,data_type,chunk_grid,codecs,chunk_key_encoding,fill_value,attributes,dimension_names,storage_transformers; for V2shape,dtype,chunks,fill_value,order,dimension_separator,compressor,filters,attributes), with no translation layer between formats. Fields left at the newzarr.AUTOsentinel are computed from the shape, the data type and the configuration, whileNonekeeps the meaning the metadata gives it (for example no compressor). A string location is always a URL pipeline such as"file:/data/example.zarr|zarr3:group/array", so its root must carry a scheme and a literal|in a local path is spelled%7C; aURLPipeline, aStoreor aStorePathis accepted as well. Creation fails if a node already exists at the location unlessoverwrite=True.This is the creation-side counterpart of the read-only
zarr.open_urlintroduced in #7, and it stacks on that PR: until #7 merges, this diff includes its commits as well, and only the three commits listed in the notes below are new here. The same change is open against thefeature/url-pipeline-objectbranch alone as d-v-b#394. The agent prepared it at d-v-b's direction, with the design settled in discussion beforehand.Design
create_arrayabstracts over Zarr formats, and the price is parameters likecompressors,serializer,filtersandshardsthat mean different things per format or exist in only one. These functions buildArrayV3Metadata/ArrayV2Metadata(orGroupMetadata) directly from their arguments and write it; there is nothing to translate.AUTOsentinel for every derived default. Every optional metadata argument whose default is computed (chunk grid from shape and dtype, codecs and compressor from the config, fill value from the dtype, V2orderfromarray.order) defaults tozarr.AUTO, atyping_extensions.Sentinelused in type position.Noneis never overloaded: on V2 it still means anullfill value, no compressor, or no filters. No"auto"strings appear in the new signatures.fill_value="auto"being a valid string fill value is the concrete reason an object sentinel is used.str | URLPipeline | Store | StorePath. Strings are read as URL pipelines whose root must carry a scheme, so|is unambiguous and%7Cis the escape. To make that escape work,file:andmemory:pipeline roots now percent-decode their bodies, as any URL does; adapter segments still receive their bodies verbatim. A format segment that contradicts the function (zarr2:forcreate_v3_array) raises.StoreandStorePathremain accepted because a URL cannot name an arbitrary Python store object.overwrite=False(the default) raisesContainsArrayError/ContainsGroupErroron an existing node;overwrite=Truedeletes it first. There is no mode string.Also in this PR
_Unsetenum inPipelineContext.resolve_precedingbecomes aSentinelfor the same reason, and the creation functions drop a scheme check thatURLPipeline.from_urlnow performs itself.Not in this PR
create_arrayand friends toAUTO, and aliasingDEFAULT_FILL_VALUEto it. Both are additive follow-ups once this shape has settled.create_at_urlwas considered and rejected: creation always knows whether it is making an array or a group, and one function could only learn that from its arguments, which is the hidden dispatchopen_urlwas reshaped to avoid.Stack and review notes
feature/url-pipeline-core. The branch isd-v-b:feature/create-v3, which stacks ond-v-b:feature/url-pipeline-object(refactor(url-pipeline): resolve pipelines from a URLPipeline object, never from plain strings #7); once refactor(url-pipeline): resolve pipelines from a URLPipeline object, never from plain strings #7 merges, the diff here shrinks to the creation work alone.AUTO, percent-decoding, mypy bump), d7c7b61 (V2 pair), 3294266 (Sentinelswap, redundant check removed).🤖 Generated with Claude Code