feat(api): add URL-pipeline-native create_v3_array, create_v3_group, create_v2_array and create_v2_group

changes/+create-v3.feature.md

@@ -0,0 +1,12 @@
+Added `zarr.create_v3_array`, `zarr.create_v3_group`, `zarr.create_v2_array` and
+`zarr.create_v2_group`: creation functions that are native to URL pipelines and to one Zarr
+format each. Their keyword arguments are that format's metadata fields (for V3 `shape`,
+`data_type`, `chunk_grid`, `codecs`, `chunk_key_encoding`, `fill_value`, `attributes`,
+`dimension_names`, `storage_transformers`; for V2 `shape`, `dtype`, `chunks`, `fill_value`,
+`order`, `dimension_separator`, `compressor`, `filters`, `attributes`), with no translation
+layer between formats. Fields left at the new `zarr.AUTO` sentinel are computed from the
+shape, the data type and the configuration, while `None` keeps the meaning the metadata
+gives it (for example no compressor). A string location is always a URL pipeline such as
+`"file:/data/example.zarr|zarr3:group/array"`, so its root must carry a scheme and a literal
+`|` in a local path is spelled `%7C`; a `URLPipeline`, a `Store` or a `StorePath` is accepted
+as well. Creation fails if a node already exists at the location unless `overwrite=True`.

changes/4192.feature.md

@@ -1,46 +1,34 @@
 Added core support for URL pipelines (https://github.com/jbms/url-pipeline):
 `|`-chained URLs that address zarr data through nested storage layers, e.g.
-`s3://bucket/data.zip|zip:|zarr3:`. This includes the parser
-(`zarr.storage.parse_pipeline` / `zarr.storage.resolve_pipeline`), the
-single-method `zarr.abc.url_pipeline.URLPipelineAdapter` interface, and the
-`zarr.url_adapters` entry-point group through which third-party packages
-(e.g. Icechunk) register adapters for their own schemes. Adapters for a scheme
-are loaded lazily and individually. Builtin adapters (`zip:`,
-`zarr2:`/`zarr3:`) will follow separately.
+`s3://bucket/data.zip|zip:|zarr3:`. A parsed `zarr.storage.URLPipeline` (built with
+`URLPipeline.from_url`) can be passed to any function that accepts a `StoreLike`, and
+the new `zarr.open_url(url, mode="r" | "r+")` opens the existing node a pipeline string
+addresses without ever creating or overwriting anything. Adapters for the
+sub-URLs after the first are provided by third-party packages (e.g. Icechunk) through
+the `zarr.url_adapters` entry-point group and the single-method
+`zarr.abc.url_pipeline.URLPipelineAdapter` interface; they are loaded lazily and
+individually. A builtin `zip:` adapter will follow separately.
 
 Behavior notes:
 
-- The `|` character is now reserved as the pipeline delimiter in every string
-  store specification, and no percent-escape is decoded; pass a `pathlib.Path`
-  to address a local file whose name contains `|`. URLs without a `|` (and
-  without a registered root adapter scheme) are handled exactly as before —
-  registered adapters cannot intercept zarr's native `file:`/`memory:`
-  routing, and fsspec chained URLs (`zip::s3://...`) keep flowing to fsspec.
-- Inside a pipeline, `memory:` and `file:` roots follow the URL pipeline
-  spec's spelling rules (`memory:` ≡ `memory:/` ≡ `memory://`; `file:` must
-  carry an absolute path with at most a `localhost` authority). Percent-escapes
-  are *not* decoded in `file:` roots, matching `LocalStore`; this is a
-  documented divergence from RFC 8089. When fsspec is installed, a plain
-  `memory://name` URL is still routed to fsspec's in-memory filesystem, whereas
-  a pipeline's `memory:` root is zarr's `ManagedMemoryStore`; the two do not
-  share data.
+- Plain strings are never interpreted as pipelines: `zarr.open("data|x.zarr")` still
+  addresses a local directory of that name, whatever adapters are installed. Only
+  `zarr.open_url` and `URLPipeline.from_url` read a string as a pipeline.
+- A trailing `zarr2:`/`zarr3:` segment selects the zarr format of the open or create
+  call, and its body is the path of the node to open. A `zarr_format` argument that
+  conflicts with it raises `ValueError`; functions whose `zarr_format` defaults to 3
+  (`zarr.create_array`, `Group.from_store`, `Group.open`, `Array.open`) need an explicit
+  `zarr_format=None` to defer to the pipeline. `Array.open` gains a `zarr_format`
+  parameter mirroring `Group.open`.
+- Inside a pipeline, `memory:` and `file:` roots follow the URL pipeline spec's spelling
+  rules (`memory:` ≡ `memory:/` ≡ `memory://`; `file:` must carry an absolute path with
+  at most a `localhost` authority). Percent-escapes in these roots are decoded, as in
+  any URL. A pipeline's `memory:` root is zarr's `ManagedMemoryStore`, which does
+  not share data with the fsspec in-memory filesystem a plain `memory://name` string
+  resolves to.
 - Wrapper adapters can open a local *file* root only in mode `"r"` (via
-  `PipelineContext.resolve_preceding(mode="r")`); other modes create a
-  directory at the root and fail. Writing through a container file is not yet
-  supported.
-- A `zarr.url_adapters` entry point whose scheme is already registered (e.g. by
-  a builtin adapter) is ignored with a `ZarrUserWarning`, as are duplicate
-  entry-point names; an entry point that fails to import raises
-  `URLPipelineError` naming it, and stays discoverable for a retry.
-- `zarr.create_array` and `Group.from_store` now default `zarr_format` to
-  `None`, resolving to the configured default (3) as before unless a URL
-  pipeline selects a format; `Array.open` gains a `zarr_format` parameter
-  mirroring `Group.open`. `Group.open` and `Array.open` keep their default of
-  3, so pass `zarr_format=None` to defer to a pipeline's format. A format
-  selected by a pipeline that conflicts with an explicit `zarr_format` raises
-  `ValueError` in every `open*`/`create*`/`save*` entry point.
-- Mode `"a"` (open-or-create, the `zarr.open` default) on a *read-only* store
-  now serves the "open" half instead of raising upfront, for all stores;
-  unambiguous write modes (`"w"`, `"w-"`, `"r+"`) still raise.
-- For root-adapter URLs (e.g. `gh://org/repo`), `storage_options` are handed
-  to the adapter and are not validated as used by `make_store`.
+  `PipelineContext.resolve_preceding(mode="r")`); writing through a container file is
+  not yet supported.
+- A `zarr.url_adapters` entry point whose scheme is already registered is ignored with a
+  `ZarrUserWarning`, as are duplicate entry-point names; an entry point that fails to
+  import raises `URLPipelineError` naming it, and stays discoverable for a retry.

docs/api/zarr/functions/create_v2_array.md

@@ -0,0 +1,5 @@
+---
+title: zarr.create_v2_array
+---
+
+::: zarr.create_v2_array

docs/api/zarr/functions/create_v2_group.md

@@ -0,0 +1,5 @@
+---
+title: zarr.create_v2_group
+---
+
+::: zarr.create_v2_group

docs/api/zarr/functions/create_v3_array.md

@@ -0,0 +1,5 @@
+---
+title: zarr.create_v3_array
+---
+
+::: zarr.create_v3_array

docs/api/zarr/functions/create_v3_group.md

@@ -0,0 +1,5 @@
+---
+title: zarr.create_v3_group
+---
+
+::: zarr.create_v3_group

docs/api/zarr/functions/open_url.md

@@ -0,0 +1,5 @@
+---
+title: zarr.open_url
+---
+
+::: zarr.open_url

docs/user-guide/extending.md

@@ -112,7 +112,7 @@ class MyAdapter(URLPipelineAdapter):
     ) -> AdapterResolution:
         # A *wrapper* adapter opens the resource to its left and wraps it. Join the
         # preceding residual path with this segment's own path, and keep every other
-        # field (e.g. zarr_format) via dataclasses.replace.
+        # field via dataclasses.replace.
         preceding = await context.resolve_preceding()
         store = WrapperStore(preceding.store)
         path = "/".join(part.strip("/") for part in (preceding.path, segment.body) if part)
@@ -126,10 +126,11 @@ inner adapter sees the outer one's residual path `a` and the pipeline resolves t
 import asyncio
 
 from zarr.registry import register_url_adapter
-from zarr.storage import resolve_pipeline
+from zarr.storage import URLPipeline
 
 register_url_adapter("mypackage.myscheme", MyAdapter)
-resolution = asyncio.run(resolve_pipeline("memory://x|mypackage.myscheme:a|mypackage.myscheme:b"))
+pipeline = URLPipeline.from_url("memory://x|mypackage.myscheme:a|mypackage.myscheme:b")
+resolution = asyncio.run(pipeline.resolve())
 assert resolution.path == "a/b"
 ```
 
@@ -144,7 +145,8 @@ scheme as the entry-point name. Nonstandard schemes should be vendor-prefixed
 
 Adapters are loaded lazily, only when a pipeline naming their scheme is resolved. An
 adapter may also be used as a *root* scheme (e.g. `mypackage.myscheme://org/repo`), in
-which case `context.preceding` is empty. The returned store must honor
+which case `context.preceding` is empty; such a URL reaches the adapter only through
+`zarr.open_url` or a `URLPipeline`, never as a plain string store specification. The returned store must honor
 `context.read_only`, and `open_pipeline_segment` runs on zarr's internal event loop, so
 it must not block or call zarr's synchronous API. See
 [`zarr.abc.url_pipeline`][] for the full contract.

docs/user-guide/storage.md

@@ -100,8 +100,9 @@ print(group)
    group = zarr.open_group(UPath('s3://noaa-nwm-retro-v2-zarr-pds', anon=True), mode='r')
    ```
 
-- a [URL pipeline](#user-guide-url-pipelines) string containing `|`, such as
-  `s3://bucket/data.zip|zip:`, which is resolved through registered adapters.
+- a [`URLPipeline`][zarr.storage.URLPipeline], the parsed form of a
+  [URL pipeline](#user-guide-url-pipelines) such as `s3://bucket/data.zip|zip:`, which
+  is resolved through registered adapters. Strings are never interpreted as pipelines.
 
 - a [`Store`][zarr.abc.store.Store] or [`StorePath`][zarr.storage.StorePath] -
   see explicit store creation below.
@@ -114,28 +115,66 @@ sub-URL locates a resource with a conventional URL; each subsequent sub-URL name
 *adapter* that reinterprets everything to its left (e.g.
 `s3://bucket/data.zip|zip:|zarr3:`). Adapters are provided by packages through the
 `zarr.url_adapters` entry-point group — see
-[`zarr.abc.url_pipeline`][zarr.abc.url_pipeline] for the adapter interface. Builtin
-adapters (`zip:`, `zarr2:`/`zarr3:`) are under development and will expand this
-section. URLs without a `|` (and without a registered root scheme) are handled
-exactly as before.
-
-`storage_options` passed to `zarr.open` apply to the *root* sub-URL (e.g. fsspec
-options for `s3://...`); adapters may consume adapter-specific, namespaced keys.
-Non-dict forms of `storage_options` are reserved for future per-segment
-configuration.
-
-The `|` character is reserved as the pipeline delimiter in every string store
-specification, and no percent-escape is decoded: to address a local file whose
-*name* contains `|` (or `#`), pass a `pathlib.Path` instead of a string.
-Registered adapters cannot intercept zarr's native `file:` and `memory:` root
-schemes, and fsspec's chained-URL syntax (`zip::s3://...`) keeps flowing to
-fsspec.
+[`zarr.abc.url_pipeline`][zarr.abc.url_pipeline] for the adapter interface. A builtin
+`zip:` adapter is under development and will expand this section.
+
+A parsed [`URLPipeline`][zarr.storage.URLPipeline] can be passed to any function that
+accepts a `StoreLike`, and [`zarr.open_url`][] opens the *existing* node a pipeline
+string addresses, read-only unless `mode="r+"` is given:
+
+```python exec="true" session="storage" source="above" result="ansi"
+from zarr.storage import URLPipeline
+
+pipeline = URLPipeline.from_url("memory://pipeline-demo|zarr2:")
+group = zarr.open_group(pipeline, mode="w")
+print(group.metadata.zarr_format)
+# the same thing, as a one-liner
+print(zarr.open_url("memory://pipeline-demo|zarr2:").metadata.zarr_format)
+```
+
+Only these two entry points read a string as a pipeline. Plain string store
+specifications keep their existing meaning everywhere else, so a local path that
+happens to contain `|` still names a directory, whatever adapters are installed.
+Unlike `zarr.open`, `zarr.open_url` never creates or overwrites. To create a node at a
+pipeline-addressed location, use [`zarr.create_v3_array`][] / [`zarr.create_v3_group`][]
+(or their Zarr format 2 counterparts [`zarr.create_v2_array`][] / [`zarr.create_v2_group`][]),
+which read a string location as a pipeline and take that format's metadata fields as
+keyword arguments, or pass the `URLPipeline` to `zarr.create_array` / `zarr.create_group`:
+
+```python exec="true" session="storage" source="above" result="ansi"
+arr = zarr.create_v3_array(
+    "memory://pipeline-demo|zarr3:temperature",
+    shape=(365, 180, 360),
+    data_type="float32",
+    chunk_grid=(30, 180, 360),
+    dimension_names=("time", "lat", "lon"),
+)
+print(arr)
+print(zarr.open_url("memory://pipeline-demo|zarr3:temperature").metadata.dimension_names)
+```
+
+In these functions a string is *always* a pipeline, so its root must carry a URL scheme and
+a literal `|` in a local path is spelled `%7C`, as in `file:/data/a%7Cb.zarr`.
+
+A trailing `zarr2:` or `zarr3:` segment is a *format segment*. It is not an adapter:
+zarr-python reads it from the pipeline itself and uses it as the `zarr_format` of the
+open or create call, and its body (`zarr3:path/to/node`) is the path of the node to
+open. A `zarr_format` argument that conflicts with the format segment raises
+`ValueError`; functions whose `zarr_format` defaults to 3 (such as `zarr.create_array`
+and `Group.open`) need an explicit `zarr_format=None` to defer to the pipeline.
+
+`storage_options` apply to the *root* sub-URL (e.g. fsspec options for `s3://...`);
+adapters may consume adapter-specific, namespaced keys. Percent-escapes are decoded in
+zarr's native `file:` and `memory:` roots, as in any URL; adapter segments receive their
+bodies verbatim. Registered adapters cannot intercept zarr's native `file:` and `memory:`
+root schemes.
 
 Inside a pipeline, a `memory:` root is zarr's managed in-memory store (`memory:`,
 `memory:/` and `memory://` are equivalent, and `memory:name` selects a named store).
-When fsspec is installed, a plain `memory://name` URL *without* a `|` is still routed to
-fsspec's in-memory filesystem, which is a different store. A `file:` root must carry an
-absolute path; percent-escapes are not decoded, matching the [local store](#local-store).
+When fsspec is installed, a plain `memory://name` string store specification is still
+routed to fsspec's in-memory filesystem, which is a different store. A `file:` root
+must carry an absolute path. The root always carries a scheme: a local path is spelled
+as a `file:` URL, never as a bare path.
 
 ## Explicit Store Creation
 

mkdocs.yml

@@ -62,6 +62,10 @@ nav:
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.create_array</code>': api/zarr/functions/create_array.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.create_group</code>': api/zarr/functions/create_group.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.create_hierarchy</code>': api/zarr/functions/create_hierarchy.md
+      - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.create_v2_array</code>': api/zarr/functions/create_v2_array.md
+      - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.create_v2_group</code>': api/zarr/functions/create_v2_group.md
+      - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.create_v3_array</code>': api/zarr/functions/create_v3_array.md
+      - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.create_v3_group</code>': api/zarr/functions/create_v3_group.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.dtype</code>': api/zarr/dtype.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.empty</code>': api/zarr/functions/empty.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.empty_like</code>': api/zarr/functions/empty_like.md
@@ -81,6 +85,7 @@ nav:
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.open_consolidated</code>': api/zarr/functions/open_consolidated.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.open_group</code>': api/zarr/functions/open_group.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.open_like</code>': api/zarr/functions/open_like.md
+      - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.open_url</code>': api/zarr/functions/open_url.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.print_debug_info</code>': api/zarr/functions/print_debug_info.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.registry</code>': api/zarr/registry.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-function"></code> <code>zarr.save</code>': api/zarr/functions/save.md

pyproject.toml

@@ -159,7 +159,7 @@ dev = [
     {include-group = "remote-tests"},
     {include-group = "docs"},
     "universal-pathlib",
-    "mypy==2.3.1",
+    "mypy==2.4.0",
 ]
 
 [tool.coverage.report]

src/zarr/__init__.py

@@ -13,6 +13,10 @@
     create_array,
     create_group,
     create_hierarchy,
+    create_v2_array,
+    create_v2_group,
+    create_v3_array,
+    create_v3_group,
     empty,
     empty_like,
     from_array,
@@ -27,6 +31,7 @@
     open_consolidated,
     open_group,
     open_like,
+    open_url,
     save,
     save_array,
     save_group,
@@ -35,6 +40,7 @@
     zeros_like,
 )
 from zarr.core.array import Array, AsyncArray
+from zarr.core.common import AUTO
 from zarr.core.config import config
 from zarr.core.group import AsyncGroup, Group
 
@@ -146,6 +152,7 @@ def set_format(log_format: str) -> None:
 
 
 __all__ = [
+    "AUTO",
     "Array",
     "AsyncArray",
     "AsyncGroup",
@@ -161,6 +168,10 @@ def set_format(log_format: str) -> None:
     "create_array",
     "create_group",
     "create_hierarchy",
+    "create_v2_array",
+    "create_v2_group",
+    "create_v3_array",
+    "create_v3_group",
     "empty",
     "empty_like",
     "from_array",
@@ -175,6 +186,7 @@ def set_format(log_format: str) -> None:
     "open_consolidated",
     "open_group",
     "open_like",
+    "open_url",
     "print_debug_info",
     "save",
     "save_array",