[jsignell]

• Towards opening a zarr dataset taking so much time with dask #8902
  • it doesn't really close it, but it is a better alternative than chunks={} or chunks="auto". Since Remove special mapping of auto to {} in open_zarr #11010 got in xarra could eventually change the default on open_zarr to map it to "preserve" rather than to {} if dask is available.
• Tests added
• User visible changes (including notable bug fixes) are documented in whats-new.rst
• New functions/methods are listed in api.rst

Proposal

A new chunks option that is only allowed to use encoded chunks or multiples of them. No chunk splitting allowed.

Demo

Current behavior when chunks="auto"

ds = xr.open_zarr("gs://gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3", chunks="auto")

This PR introduces a new option: chunks="preserve"

ds = xr.open_zarr("gs://gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3", chunks="preserve")

Context

I originally set out to update the auto_chunks function in dask, but it felt like my goals were actually quite different. The goal of the dask auto_chunks function is to guarantee that the chunksize will be under a configurable limit while preserving the aspect ratio of previous_chunks (previous_chunks == encoding). This PR instead guarantees that encoded chunks are never split but it will multiply them by some factor to try to get the chunksize close to a targetsize. It doesn't try to preserve the aspect ratio of the chunks. Instead it goes after the dim where there is the greatest number of chunks and it tries to take those in bigger bites.

Also:

• I'm not quite sure how the interface should work and I am definitely not attached to the word "preserve".
• I originally put this in the DaskManager, but moved it to the base ChunkManagerEntrypoint class once I realized there was nothing dasky about it. I'm not sure if there is really supposed to be logic in methods on that class though.

[jsignell]

These tests are kind of slow.

[jsignell]

@dcherian is this something you would be able to review? I'd love to get more people trying it.

[jsignell]

Ok @dcherian this is now a PR that changes the behavior of chunks="auto". The changes to tests were minimal with this change, but if we think it's important to give people a way to get the prior behavior we can add an option so that people can set xr.set_options(use_dask_auto=True)

This would be what the commit looks like to add the option jsignell@84cca20

[jsignell]

I can make this not change the existing behavior at all if we want to do a slow rollout where people would do:

import xarray as xr

with xr.set_options(use_dask_auto=False):
    ds = xr.open_zarr("gs://gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3", chunks="auto")

[jsignell]

I was just reminded of this PR and remembered that I still care about it! I am going to update this branch and include the option setting so that people can get back to the previous behavior.

[dcherian]

Sorry, I can review when you're done. I don't think an option is warranted, by saying "auto:, the user has already opted in to magic.