opening a zarr dataset taking so much time with dask

What is your issue?

I have an era5 dataset stored in GCS bucket as zarr. It contains 273 weather related variables and 4 dimensions. It's an hourly stored data from 1940 to 2023.
When I try to open with ds = xr.open_zarr("gs://gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3") it takes 90 seconds to actually finish the open call.
The chunk scheme is { 'time': 1 }.

Thanks for opening your first issue here at xarray! Be sure to follow the issue template!
If you have an idea for a solution, we would really welcome a Pull Request with proposed changes.
See the Contributing Guide for more.
It may take us a while to respond here, but we really value your contribution. Contributors like you help make xarray better.
Thank you!

I've felt the pain on this particular store as well, it's a nice test case. 3PB total, ~300,000,000 chunks in total.

Looks like this is a dask problem though. All the time is spent in single-threaded code creating the array chunks.

If we skip dask with xr.open_zarr(..., chunks=None) it takes 1.5s.

We currently have a drop_variables arg. When you have a dataset with 273 variables and you only want a couple, the inverse keep_variables would be a lot easier. It looks like drop_variables gets applied before we create the dask chunks for the arrays, so reading the store once and on the second read adding drop_variables=[v for v in ds.data_vars if v != "geopotential"], I recover a ~1.5s read time.

@jrbourbeau

Nice profile!

@jrbourbeau dask/dask#10648 probably improves this by a lot. Can that be reviewed/merged please?

edit: xref dask/dask#10269

Interestingly things are almost a factor of 4x worse on both those PRs, but both are out of date so may be missing other recent improvements.

@slevang

@slevang, do you mind sharing how you are generating these profiles and associated graphs? I've struggled in the past to do this effectively with a dask cluster. This looks great!

This is snakeviz: https://jiffyclub.github.io/snakeviz/

^yep. Probably not useful for distributed profiling but I haven't really tried. It's just a visualization layer for cProfile.

In this case the creation of this monster task graph would be happening serially in the main process even if your goal was to eventually use a distributed client to run processing. The graph (only describing the array chunks) is close to a billion objects in this case, so would run into issues even trying to serialize that out to workers.

FYI something like:

ds = xr.open_zarr(
    "gs://gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3",
    chunks=dict(time=-1, level=-1, latitude="auto", longitude="auto"),
)

...may give a better balance between dask task graph size and chuck size.

(I do this "make the dask chunks bigger than the zarr chunks" a lot, because simple dask graphs can become huge with a 50TB dataset, since the default zarr encoding has a maximum chunk size of 2GB. Not sure it's necessarily the best way, very open to ideas...)

Edit: nevermind I actually have no idea where this profile came from. Disregard

I wonder if we can close this now that we've demonstrated some substantial improvements here:

This repository has consolidated metadata, so this ticket has always been about dask, and specifically how Xarray calls tokenize, and how Xarray checks for chunk alignment.

Try using uuids

Just went down a bit of a rabbit hole and wanted to just mention that to this point: #8902 (comment) picking an explicit chunks value will speed things up dramatically. For instance to make the dask tasks 10x the on-disk chunksize (which is 1) in the time dimension:

%%time
ds = xr.open_zarr(
    "gs://gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3",
    chunks=dict(time=10, level=-1, latitude=(), longitude=()),
)

This is partly an Xarray issue with us passing a normalized chunks tuple that is very very large. tokenizing that takes ages: #9897

One thought for anyone interested, we might skip "normalizing" chunks for int chunks (like all Zarrs & netCDFs in existence today) and pass them straight through; dask can handle them and it gets us out of this mess of tokenizing a tuple with half a million 1s. I may have looked in to this, but can't remember why I didn't do it.

I am kind of wondering if we can actually skip "normalizing" chunks always for Dask. Dask is going to to normalize anyways on its side I imagine.

I can't remember why but it's to do with handling the "auto" I think?

Zooming out a bit, I am wondering if a HighLevelGraph approach might let us not materialize the low level task graph at all for this kind of many data variables case. Especially given that you likely only want a few of the data variables.

I realized that this is actually just setting the key for the HighLevelGraph so that didn't really make any sense.

I just put up a PR that implements a new type of automatic chunking that is never allowed to split chunks. That makes opening this particular zarr store much faster.

Oops - #11017 isn't quite enough to close this. That is more of #11060's job

Navigation Menu

Repository navigation

opening a zarr dataset taking so much time with dask #8902

What is your issue?

8 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

opening a zarr dataset taking so much time with dask #8902

Description

What is your issue?

Activity

welcome commented on Apr 2, 2024

slevang commented on Apr 2, 2024

dcherian commented on Apr 2, 2024

slevang commented on Apr 2, 2024

riley-brady commented on Apr 3, 2024

dcherian commented on Apr 3, 2024

slevang commented on Apr 3, 2024

max-sixty commented on Apr 3, 2024

slevang commented on Apr 3, 2024

8 remaining items

jhamman commented on Mar 20, 2025

dcherian commented on Mar 20, 2025

jsignell commented on Dec 10, 2025

dcherian commented on Dec 10, 2025

dcherian commented on Dec 11, 2025

jsignell commented on Dec 11, 2025

dcherian commented on Dec 11, 2025

jsignell commented on Dec 12, 2025

jsignell commented on Dec 31, 2025

jsignell commented on Feb 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions