feat: URL pipeline core — URLPipeline objects, adapter ABC and registry

changes/4192.feature.md

@@ -0,0 +1,7 @@
+Added core support for URL pipelines (https://github.com/jbms/url-pipeline), `|`-chained
+URLs such as `s3://bucket/data.zip|zip:|zarr3:`. Parse one with
+`zarr.storage.URLPipeline.from_url` and pass it to any function that accepts a `StoreLike`;
+plain strings are never read as pipelines. A trailing `zarr:`, `zarr2:` or `zarr3:` segment
+gives the node path and selects (or detects) the format. Packages provide adapters for other
+schemes by subclassing `zarr.abc.url_pipeline.URLPipelineAdapter` and registering it under the
+`zarr.url_adapters` entry-point group or with `zarr.registry.register_url_adapter`.

docs/api/zarr/abc/index.md

@@ -11,3 +11,4 @@ Abstract base classes for extending Zarr-Python.
 - **[zarr.abc.metadata](./metadata.md)** - Creating metadata classes compatible with the Zarr API
 - **[zarr.abc.numcodec](./numcodec.md)** - Protocols and classes for modeling codec interface used by numcodecs
 - **[zarr.abc.store](./store.md)** - ABC for implementing Zarr stores and managing getting and setting bytes in a store
+- **[zarr.abc.url_pipeline](./url_pipeline.md)** - ABC for implementing [URL pipeline](https://github.com/jbms/url-pipeline) adapters

docs/api/zarr/abc/url_pipeline.md

@@ -0,0 +1,5 @@
+---
+title: url_pipeline
+---
+
+::: zarr.abc.url_pipeline

docs/user-guide/extending.md

@@ -85,6 +85,72 @@ implementation.
 Custom stores can be created by implementing the [`zarr.abc.store.Store`][] interface.
 See [developing custom stores](storage.md#developing-custom-stores) for more information.
 
+## Custom URL pipeline adapters
+
+In a [URL pipeline](storage.md#user-guide-url-pipelines) such as
+`s3://bucket/data.zip|zip:|zarr3:`, each sub-URL after the first names an *adapter*
+that reinterprets everything to its left. A package provides an adapter by subclassing
+[`zarr.abc.url_pipeline.URLPipelineAdapter`][] and implementing one classmethod, which
+returns an open store and the path addressed inside it as a
+[`StorePath`][zarr.storage.StorePath]:
+
+```python test="true" session="url-adapter"
+from zarr.abc.url_pipeline import URLPipelineAdapter
+from zarr.storage import PipelineContext, PipelineSegment, StorePath, WrapperStore
+
+
+class MyAdapter(URLPipelineAdapter):
+    @classmethod
+    async def open_pipeline_segment(
+        cls, segment: PipelineSegment, context: PipelineContext
+    ) -> StorePath:
+        # A wrapper adapter opens the resource to its left and wraps it, joining the
+        # preceding residual path with this segment's own path.
+        preceding = await context.resolve_preceding()
+        store = WrapperStore(preceding.store)
+        path = "/".join(part.strip("/") for part in (preceding.path, segment.body) if part)
+        return StorePath(store, path)
+```
+
+To resolve the preceding pipeline with another mode or options, use a derived context:
+`await dataclasses.replace(context, mode="r").resolve_preceding()`. Wrapper adapters
+compose: `memory://x|mypackage.myscheme:a|mypackage.myscheme:b` resolves to `a/b`.
+
+```python test="true" session="url-adapter"
+import zarr
+from zarr.registry import register_url_adapter
+from zarr.storage import URLPipeline
+
+register_url_adapter("mypackage.myscheme", MyAdapter)
+url = "memory://x|mypackage.myscheme:a|mypackage.myscheme:b"
+# keep a reference: in-memory data lives only while a store refers to it
+created = zarr.create_group(URLPipeline.from_url(url))
+group = zarr.open_group(URLPipeline.from_url(url), mode="r")
+assert group.path == "a/b"
+```
+
+Register the class under the `zarr.url_adapters` entry-point group with the scheme as
+the entry-point name, vendor-prefixed for nonstandard schemes. Adapters are imported
+only when a pipeline naming their scheme is resolved.
+
+```toml
+[project.entry-points."zarr.url_adapters"]
+"mypackage.myscheme" = "mypackage.zarr_adapter:MyAdapter"
+```
+
+An adapter can also be a *root* scheme (`mypackage.myscheme://org/repo`), with an empty
+`context.preceding`. Installing a root adapter for a scheme fsspec also handles takes
+that scheme over at the root of every pipeline. The schemes zarr handles itself
+(`file`, `memory`, `zarr`, `zarr2`, `zarr3`) cannot be registered, and entry points
+with those names are ignored.
+
+Every adapter sees all of the caller's `storage_options`, including the root's
+credentials: consume only keys namespaced for your adapter, strip them before resolving
+the rest of the pipeline, and never log the options. The returned store must honor
+`context.read_only`, and `open_pipeline_segment` must not block or call zarr's
+synchronous API. See [`PipelineContext`][zarr.storage.PipelineContext] for the full
+contract.
+
 ## Custom array buffers
 
 Zarr-python provides control over where and how arrays are stored in memory through

docs/user-guide/storage.md

@@ -100,9 +100,56 @@ print(group)
    group = zarr.open_group(UPath('s3://noaa-nwm-retro-v2-zarr-pds', anon=True), mode='r')
    ```
 
+- a [`URLPipeline`][zarr.storage.URLPipeline], the parsed form of a
+  [URL pipeline](#user-guide-url-pipelines) such as `s3://bucket/data.zip|zip:`, which
+  is resolved through registered adapters.
+
 - a [`Store`][zarr.abc.store.Store] or [`StorePath`][zarr.storage.StorePath] -
   see explicit store creation below.
 
+## URL Pipelines {#user-guide-url-pipelines}
+
+Zarr supports [URL pipelines](https://github.com/jbms/url-pipeline): `|`-chained URLs
+read left to right. The first sub-URL locates a resource with a conventional URL; each
+later sub-URL names an *adapter* that reinterprets everything to its left (e.g.
+`s3://bucket/data.zip|zip:|zarr3:`). Adapters are provided by packages; see
+[custom URL pipeline adapters](extending.md#custom-url-pipeline-adapters).
+
+Parse a pipeline with [`URLPipeline.from_url`][zarr.storage.URLPipeline.from_url] and
+pass it to any function that accepts a `StoreLike`. At present, this is the only way a string is
+read as a pipeline: a plain string store specification currently maintains its existing meaning, so a
+local path containing `|` still names a directory.
+
+```python exec="true" session="storage" source="above" result="ansi"
+from zarr.storage import URLPipeline
+
+pipeline = URLPipeline.from_url("memory://pipeline-demo|zarr2:")
+group = zarr.open_group(pipeline, mode="w")
+print(zarr.open_group(pipeline, mode="r").metadata.zarr_format)
+```
+
+A trailing `zarr2:` or `zarr3:` *format segment* selects the zarr format,
+and its body (`zarr3:path/to/node`) is the node path; `zarr:` gives a node path and
+detects the format. A conflicting `zarr_format` argument raises `ValueError`, including
+the default `zarr_format=3` of `zarr.create_array`, `Group.open`, `Group.from_store` and
+`Array.open`.
+
+The root sub-URL must carry a scheme, so a local path is spelled as an absolute `file:`
+URL (`file:/data/x.zarr`, with at most a `localhost` authority). Every `memory:` root
+addresses one managed in-memory store, and its body is a path within it (`memory:a` ≡
+`memory:/a` ≡ `memory://a`); data lives only while some store object refers to it.
+Neither root accepts `.` or `..` path components, and drive letters are only special on
+Windows. No percent-escape is decoded anywhere in a pipeline. Other roots, such as
+`s3://...`, are opened exactly as the same plain string would be.
+
+`storage_options` configure the root sub-URL, and every adapter in the pipeline can
+see them. When a pipeline may come from an untrusted source, keep credentials out of
+`storage_options` unless you trust every installed adapter, and don't accept `memory:`
+roots in a process that serves several users: the in-memory store is process-wide.
+
+For complex store configurations, explicit store creation, as described below, may be
+required.
+
 ## Explicit Store Creation
 
 In some cases, it may be helpful to create a store instance directly. Zarr-Python offers

mkdocs.yml

@@ -43,6 +43,7 @@ nav:
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.metadata</code>': api/zarr/abc/metadata.md
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.numcodec</code>': api/zarr/abc/numcodec.md
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.store</code>': api/zarr/abc/store.md
+        - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.abc.url_pipeline</code>': api/zarr/abc/url_pipeline.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.api</code>':
         - api/zarr/api/index.md
         - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr.api.asynchronous</code>': api/zarr/api/asynchronous.md

src/zarr/abc/url_pipeline.py

@@ -0,0 +1,55 @@
+"""
+Abstract base class for URL pipeline adapters
+(https://github.com/jbms/url-pipeline). See
+[`URLPipelineAdapter`][zarr.abc.url_pipeline.URLPipelineAdapter].
+"""
+
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from zarr.storage import PipelineContext, PipelineSegment, StorePath
+
+__all__ = ["URLPipelineAdapter"]
+
+
+class URLPipelineAdapter(ABC):
+    """
+    Handler for one URL pipeline scheme.
+
+    In a pipeline such as `s3://bucket/data.zip|zip:inner|zarr3:`, each sub-URL
+    after the first names an adapter that reinterprets everything to its left.
+    Subclasses implement
+    [`open_pipeline_segment`][zarr.abc.url_pipeline.URLPipelineAdapter.open_pipeline_segment]
+    and are registered under the `zarr.url_adapters` entry-point group, with
+    the URL scheme as the entry-point name:
+
+        [project.entry-points."zarr.url_adapters"]
+        "mypackage.myscheme" = "mypackage.zarr_adapter:MyAdapter"
+
+    Nonstandard schemes should be vendor-prefixed (`vendor.scheme`), per the
+    URL pipeline specification. An adapter is used either as an *adapter
+    segment* (`s3://bucket/repo|icechunk://tag.v1`), where the context carries
+    the preceding sub-URLs, or as a *root scheme* (`gh://org/repo`), where
+    `context.preceding` is empty.
+    """
+
+    @classmethod
+    @abstractmethod
+    async def open_pipeline_segment(
+        cls, segment: PipelineSegment, context: PipelineContext
+    ) -> StorePath:
+        """
+        Resolve `segment`, in the context of the pipeline to its left, into an
+        open store and the path the pipeline addresses inside it (empty for
+        the store root).
+
+        The store must honor `context.read_only`; the resolver also enforces
+        it. This coroutine runs on zarr's I/O event loop, so it must not
+        block, and must not call zarr's synchronous API (which raises
+        `SyncError`). Open the preceding pipeline with
+        [`resolve_preceding`][zarr.storage.PipelineContext.resolve_preceding].
+        """
+        ...

src/zarr/api/asynchronous.py

@@ -48,7 +48,7 @@
     ZarrUserWarning,
 )
 from zarr.storage import StorePath
-from zarr.storage._common import make_store_path
+from zarr.storage._common import make_store_path, resolve_zarr_format
 
 if TYPE_CHECKING:
     from collections.abc import Iterable
@@ -234,6 +234,7 @@ async def consolidate_metadata(
         consolidated metadata, this function raises a `TypeError`.
         See `Store.supports_consolidated_metadata`.
     """
+    zarr_format = resolve_zarr_format(store, zarr_format)
     store_path = await make_store_path(store, path=path)
 
     if not store_path.store.supports_consolidated_metadata:
@@ -403,6 +404,7 @@ async def open(
             mode = "r"
         else:
             mode = "a"
+    zarr_format = resolve_zarr_format(store, zarr_format)
     store_path = await make_store_path(store, mode=mode, path=path, storage_options=storage_options)
 
     # TODO: the mode check below seems wrong!
@@ -526,6 +528,7 @@ async def save_array(
     **kwargs
         Passed through to [`create`][zarr.api.asynchronous.create], e.g., compressor.
     """
+    zarr_format = resolve_zarr_format(store, zarr_format)
     if zarr_format is None:
         zarr_format = _default_zarr_format()
     if not isinstance(arr, NDArrayLike):
@@ -581,6 +584,7 @@ async def save_group(
         NumPy arrays with data to save.
     """
 
+    zarr_format = resolve_zarr_format(store, zarr_format)
     if zarr_format is None:
         zarr_format = _default_zarr_format()
 
@@ -796,6 +800,7 @@ async def create_group(
         The new group.
     """
 
+    zarr_format = resolve_zarr_format(store, zarr_format)
     if zarr_format is None:
         zarr_format = _default_zarr_format()
 
@@ -891,6 +896,7 @@ async def open_group(
         }
     )
 
+    zarr_format = resolve_zarr_format(store, zarr_format)
     store_path = await make_store_path(store, mode=mode, storage_options=storage_options, path=path)
     if attributes is None:
         attributes = {}
@@ -1098,6 +1104,7 @@ async def create(
     z : array
         The array.
     """
+    zarr_format = resolve_zarr_format(store, zarr_format)
     if zarr_format is None:
         zarr_format = _default_zarr_format()
 
@@ -1333,6 +1340,7 @@ async def open_array(
     """
 
     mode = kwargs.pop("mode", None)
+    zarr_format = resolve_zarr_format(store, zarr_format)
     store_path = await make_store_path(store, path=path, mode=mode, storage_options=storage_options)
 
     if "write_empty_chunks" in kwargs:

src/zarr/core/array.py

@@ -156,7 +156,7 @@
     _parse_bytes_bytes_codec,
     get_pipeline_class,
 )
-from zarr.storage._common import StorePath, make_store_path
+from zarr.storage._common import StorePath, make_store_path, resolve_zarr_format
 from zarr.storage._memory import MemoryStore
 from zarr.storage._utils import _relativize_path
 
@@ -469,6 +469,7 @@ async def _create(
         Deprecated in favor of [`zarr.api.asynchronous.create_array`][].
         """
 
+        zarr_format = resolve_zarr_format(store, zarr_format)
         dtype_parsed = parse_dtype(dtype, zarr_format=zarr_format)
         store_path = await make_store_path(store)
 
@@ -821,6 +822,7 @@ async def example():
         # <AsyncArray memory://... shape=(100, 100) dtype=int32>
         ```
         """
+        zarr_format = resolve_zarr_format(store, zarr_format)
         store_path = await make_store_path(store)
         metadata_dict = await get_array_metadata(store_path, zarr_format=zarr_format)
         # TODO: remove this cast when we have better type hints
@@ -4399,6 +4401,7 @@ async def from_array(
     """
     mode: Literal["a"] = "a"
     config_parsed = parse_array_config(config)
+    zarr_format = resolve_zarr_format(store, zarr_format)
     store_path = await make_store_path(store, path=name, mode=mode, storage_options=storage_options)
 
     (
@@ -4902,6 +4905,7 @@ async def create_array(
     else:
         mode: Literal["a"] = "a"
 
+        zarr_format = resolve_zarr_format(store, zarr_format)
         store_path = await make_store_path(
             store, path=name, mode=mode, storage_options=storage_options
         )

src/zarr/core/group.py

@@ -67,7 +67,7 @@
     ZarrUserWarning,
 )
 from zarr.storage import StoreLike, StorePath
-from zarr.storage._common import make_store_path
+from zarr.storage._common import make_store_path, resolve_zarr_format
 from zarr.storage._utils import _join_paths, _normalize_path_keys, normalize_path
 
 if TYPE_CHECKING:
@@ -532,6 +532,7 @@ async def from_store(
         overwrite: bool = False,
         zarr_format: ZarrFormat = 3,
     ) -> AsyncGroup:
+        zarr_format = resolve_zarr_format(store, zarr_format)
         store_path = await make_store_path(store)
         attributes = attributes or {}
         group = cls(
@@ -574,6 +575,7 @@ async def open(
             (``.zmetadata`` by default). Specify the custom key as ``use_consolidated``
             to load consolidated metadata from a non-default key.
         """
+        zarr_format = resolve_zarr_format(store, zarr_format)
         store_path = await make_store_path(store)
         if not store_path.store.supports_consolidated_metadata:
             # Fail if consolidated metadata was requested but the Store doesn't support it

src/zarr/errors.py

@@ -12,6 +12,7 @@
     "MetadataValidationError",
     "NegativeStepError",
     "NodeTypeValidationError",
+    "URLPipelineError",
     "UnknownCodecError",
     "UnstableSpecificationWarning",
     "VindexInvalidSelectionError",
@@ -101,6 +102,12 @@ class UnknownCodecError(BaseZarrError):
     """
 
 
+class URLPipelineError(BaseZarrError):
+    """
+    Raised when a URL pipeline cannot be parsed or resolved.
+    """
+
+
 class NodeTypeValidationError(MetadataValidationError):
     """
     Specialized exception when the node_type of the metadata document is incorrect.