From 81bf2c1028f1a7d4937be607a452d26d03902dbd Mon Sep 17 00:00:00 2001 From: Erik van Sebille Date: Fri, 11 Sep 2026 08:40:48 +0200 Subject: [PATCH 1/2] Update order of performance suggestions --- .../examples/explanation_performance.md | 86 +++++++++---------- 1 file changed, 43 insertions(+), 43 deletions(-) diff --git a/docs/user_guide/examples/explanation_performance.md b/docs/user_guide/examples/explanation_performance.md index b039b7946..2c19bef52 100644 --- a/docs/user_guide/examples/explanation_performance.md +++ b/docs/user_guide/examples/explanation_performance.md @@ -36,34 +36,28 @@ This will make Parcels use `numpy` functions in the interpolation routines, whic | ---------------------------------- | ------------------------------------------------------ | | Very fast and simple to implement. | Will only work if the entire Dataset fits into memory. | -## Option 2: use (cached) zarr files +## Option 2: use ChunkCachedArrays -**Best for: large Datasets (more than a few GB) and particles distributed over a small part of the domain** +_Uses Parcels Backend: ChunkCachedArray_ -_Uses Parcels Backend: Zarr_ +**Best for: large Datasets (more than a few GB) and particles distributed over the entire domain** -If your Dataset is too large to fit into memory, but your particles are only distributed over a small part of the domain, it could be efficient to use cached zarr files. This can be done by using the (experimental) `zarr.CacheStore` in combination with the {py:func}`parcels.open_raw_zarr()` function. This will make Parcels only load the chunks that are needed for the particles, and cache these chunks in memory for future use. +If your Dataset is so large that it doesn't fit into memory, you can use the {py:func}`parcels.FieldSet.to_chunk_cached_arrays()`. +This constructs a cache where individual (dask) chunks of data are stored. +During a simulation, Parcels fetches data from the chunk cache and, only if the data is not loaded in the cache, retrieves data from disk to use and store in the cache. -```{code-block} python -source_store = zarr.storage.LocalStore(filenames) -cache_store = zarr.storage.MemoryStore() +This results in a smaller memory footprint than the windowed array approach (especially if the particles aren't distributed over the spatial domain). -store = CacheStore( - store=source_store, cache_store=cache_store, max_size=MAX_CACHE_SIZE -) -ds = parcels.open_raw_zarr(store) +```{code-block} python +fieldset.to_chunkcached_arrays() ``` ### Advantages and disadvantages -| Advantages | Disadvantages | -| -------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | -| Parcels will only load the chunks that are needed for the particles, which can be much less than the entire Dataset. | The hydrodynamic data will have to be stored in zarr format, which may require additional storage space. | -| | The fieldset data can't be changed after it is loaded, as dask operations are not supported on the raw zarr data. | - -```{note} -In our performance testing, we have found that using zarr files saved without any compression can be considerably faster than using compressed zarr files. -``` +| Advantages | Disadvantages | +| ----------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | +| Parcels will only hold accessed chunks of data in memory, which is much less than the entire Dataset. | With many particles that cover the entire domain, the caching layer (and finding which particles are in which chunks) can have more overhead. | +| Hydrodynamic files do not have to be reformatted and stored. | | ## Option 3: use Windowed Arrays @@ -91,7 +85,36 @@ fieldset.to_windowed_arrays() If your particles start at multiple times (e.g. [Delayed starts](./tutorial_delaystart.ipynb)), it's best to do the initial sampling with the FieldSet in Dask mode and only convert to WindowedArrays _after_ the initial sampling. ``` -## Option 4: use Dask +## Option 4: use (cached) zarr files + +**Best for: large Datasets (more than a few GB) and particles distributed over a small part of the domain** + +_Uses Parcels Backend: Zarr_ + +If your Dataset is too large to fit into memory, but your particles are only distributed over a small part of the domain, it could be efficient to use cached zarr files. This can be done by using the (experimental) `zarr.CacheStore` in combination with the {py:func}`parcels.open_raw_zarr()` function. This will make Parcels only load the chunks that are needed for the particles, and cache these chunks in memory for future use. + +```{code-block} python +source_store = zarr.storage.LocalStore(filenames) +cache_store = zarr.storage.MemoryStore() + +store = CacheStore( + store=source_store, cache_store=cache_store, max_size=MAX_CACHE_SIZE +) +ds = parcels.open_raw_zarr(store) +``` + +### Advantages and disadvantages + +| Advantages | Disadvantages | +| -------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | +| Parcels will only load the chunks that are needed for the particles, which can be much less than the entire Dataset. | The hydrodynamic data will have to be stored in zarr format, which may require additional storage space. | +| | The fieldset data can't be changed after it is loaded, as dask operations are not supported on the raw zarr data. | + +```{note} +In our performance testing, we have found that using zarr files saved without any compression can be considerably faster than using compressed zarr files. +``` + +## Option 5: use Dask **Best for: large Datasets (more than a few GB) and small ParticleSets (less than a few hundred particles)** @@ -110,26 +133,3 @@ The long-term plan for Parcels development is to make this Option 4 work well fo If you have ideas for how to make Parcels faster, we'd love to hear from you! Feel free to [open an issue](https://github.com/Parcels-code/Parcels/issues) or reach out to us on [Zulip](https://clam-community.github.io). ``` - -## Option 5: use ChunkCachedArrays - -_Uses Parcels Backend: ChunkCachedArray_ - -**Best for: large Datasets (more than a few GB) and particles distributed over the entire domain** - -If your Dataset is so large that it doesn't fit into memory, you can use the {py:func}`parcels.FieldSet.to_chunk_cached_arrays()`. -This constructs a cache where individual (dask) chunks of data are stored. -During a simulation, Parcels fetches data from the chunk cache and, only if the data is not loaded in the cache, retrieves data from disk to use and store in the cache. - -This results in a smaller memory footprint than the windowed array approach (especially if the particles aren't distributed over the spatial domain). - -```{code-block} python -fieldset.to_chunkcached_arrays() -``` - -### Advantages and disadvantages - -| Advantages | Disadvantages | -| ----------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | -| Parcels will only hold accessed chunks of data in memory, which is much less than the entire Dataset. | With many particles that cover the entire domain, the caching layer (and finding which particles are in which chunks) can have more overhead. | -| Hydrodynamic files do not have to be reformatted and stored. | | From 57ad187fc5dd962d09317929ab9f1489883325f2 Mon Sep 17 00:00:00 2001 From: Erik van Sebille Date: Fri, 11 Sep 2026 08:45:19 +0200 Subject: [PATCH 2/2] Fixing typeo in tutorial --- docs/user_guide/examples/explanation_performance.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user_guide/examples/explanation_performance.md b/docs/user_guide/examples/explanation_performance.md index 2c19bef52..873995c4e 100644 --- a/docs/user_guide/examples/explanation_performance.md +++ b/docs/user_guide/examples/explanation_performance.md @@ -49,7 +49,7 @@ During a simulation, Parcels fetches data from the chunk cache and, only if the This results in a smaller memory footprint than the windowed array approach (especially if the particles aren't distributed over the spatial domain). ```{code-block} python -fieldset.to_chunkcached_arrays() +fieldset.to_chunk_cached_arrays() ``` ### Advantages and disadvantages