Skip to content

Update order of performance suggestions #2896

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
erikvansebille merged 2 commits into from
Sep 11, 2026
Merged

Update order of performance suggestions #2896

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
81bf2c1
Update order of performance suggestions
erikvansebille Sep 11, 2026
57ad187
Fixing typeo in tutorial
erikvansebille Sep 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 43 additions & 43 deletions docs/user_guide/examples/explanation_performance.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,34 +36,28 @@ This will make Parcels use `numpy` functions in the interpolation routines, whic
| ---------------------------------- | ------------------------------------------------------ |
| Very fast and simple to implement. | Will only work if the entire Dataset fits into memory. |

## Option 2: use (cached) zarr files
## Option 2: use ChunkCachedArrays

**Best for: large Datasets (more than a few GB) and particles distributed over a small part of the domain**
_Uses Parcels Backend: ChunkCachedArray_

_Uses Parcels Backend: Zarr_
**Best for: large Datasets (more than a few GB) and particles distributed over the entire domain**

If your Dataset is too large to fit into memory, but your particles are only distributed over a small part of the domain, it could be efficient to use cached zarr files. This can be done by using the (experimental) `zarr.CacheStore` in combination with the {py:func}`parcels.open_raw_zarr()` function. This will make Parcels only load the chunks that are needed for the particles, and cache these chunks in memory for future use.
If your Dataset is so large that it doesn't fit into memory, you can use the {py:func}`parcels.FieldSet.to_chunk_cached_arrays()`.
This constructs a cache where individual (dask) chunks of data are stored.
During a simulation, Parcels fetches data from the chunk cache and, only if the data is not loaded in the cache, retrieves data from disk to use and store in the cache.

```{code-block} python
source_store = zarr.storage.LocalStore(filenames)
cache_store = zarr.storage.MemoryStore()
This results in a smaller memory footprint than the windowed array approach (especially if the particles aren't distributed over the spatial domain).

store = CacheStore(
store=source_store, cache_store=cache_store, max_size=MAX_CACHE_SIZE
)
ds = parcels.open_raw_zarr(store)
```{code-block} python
fieldset.to_chunk_cached_arrays()
```

### Advantages and disadvantages

| Advantages | Disadvantages |
| -------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| Parcels will only load the chunks that are needed for the particles, which can be much less than the entire Dataset. | The hydrodynamic data will have to be stored in zarr format, which may require additional storage space. |
| | The fieldset data can't be changed after it is loaded, as dask operations are not supported on the raw zarr data. |

```{note}
In our performance testing, we have found that using zarr files saved without any compression can be considerably faster than using compressed zarr files.
```
| Advantages | Disadvantages |
| ----------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| Parcels will only hold accessed chunks of data in memory, which is much less than the entire Dataset. | With many particles that cover the entire domain, the caching layer (and finding which particles are in which chunks) can have more overhead. |
| Hydrodynamic files do not have to be reformatted and stored. | |

## Option 3: use Windowed Arrays

Expand Down Expand Up @@ -91,7 +85,36 @@ fieldset.to_windowed_arrays()
If your particles start at multiple times (e.g. [Delayed starts](./tutorial_delaystart.ipynb)), it's best to do the initial sampling with the FieldSet in Dask mode and only convert to WindowedArrays _after_ the initial sampling.
```

## Option 4: use Dask
## Option 4: use (cached) zarr files

@erikvansebille erikvansebille Sep 11, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we have to_chunkcached_arrays(), do we even need the parcels.open_raw_zarr() feature anymore? What is the advantage of the latter over the former approach (the disadvantage is clear: the fieldset can't be changed after loading)

Should we remove parcels.open_raw_zarr()?

@VeckoTheGecko VeckoTheGecko Sep 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this was something I was also wondering about #2856


**Best for: large Datasets (more than a few GB) and particles distributed over a small part of the domain**

_Uses Parcels Backend: Zarr_

If your Dataset is too large to fit into memory, but your particles are only distributed over a small part of the domain, it could be efficient to use cached zarr files. This can be done by using the (experimental) `zarr.CacheStore` in combination with the {py:func}`parcels.open_raw_zarr()` function. This will make Parcels only load the chunks that are needed for the particles, and cache these chunks in memory for future use.

```{code-block} python
source_store = zarr.storage.LocalStore(filenames)
cache_store = zarr.storage.MemoryStore()

store = CacheStore(
store=source_store, cache_store=cache_store, max_size=MAX_CACHE_SIZE
)
ds = parcels.open_raw_zarr(store)
```

### Advantages and disadvantages

| Advantages | Disadvantages |
| -------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| Parcels will only load the chunks that are needed for the particles, which can be much less than the entire Dataset. | The hydrodynamic data will have to be stored in zarr format, which may require additional storage space. |
| | The fieldset data can't be changed after it is loaded, as dask operations are not supported on the raw zarr data. |

```{note}
In our performance testing, we have found that using zarr files saved without any compression can be considerably faster than using compressed zarr files.
```

## Option 5: use Dask

**Best for: large Datasets (more than a few GB) and small ParticleSets (less than a few hundred particles)**

Expand All @@ -110,26 +133,3 @@ The long-term plan for Parcels development is to make this Option 4 work well fo
If you have ideas for how to make Parcels faster, we'd love to hear from you!
Feel free to [open an issue](https://github.com/Parcels-code/Parcels/issues) or reach out to us on [Zulip](https://clam-community.github.io).
```

## Option 5: use ChunkCachedArrays

_Uses Parcels Backend: ChunkCachedArray_

**Best for: large Datasets (more than a few GB) and particles distributed over the entire domain**

If your Dataset is so large that it doesn't fit into memory, you can use the {py:func}`parcels.FieldSet.to_chunk_cached_arrays()`.
This constructs a cache where individual (dask) chunks of data are stored.
During a simulation, Parcels fetches data from the chunk cache and, only if the data is not loaded in the cache, retrieves data from disk to use and store in the cache.

This results in a smaller memory footprint than the windowed array approach (especially if the particles aren't distributed over the spatial domain).

```{code-block} python
fieldset.to_chunkcached_arrays()
```

### Advantages and disadvantages

| Advantages | Disadvantages |
| ----------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| Parcels will only hold accessed chunks of data in memory, which is much less than the entire Dataset. | With many particles that cover the entire domain, the caching layer (and finding which particles are in which chunks) can have more overhead. |
| Hydrodynamic files do not have to be reformatted and stored. | |