[d-v-b]

🤖 AI text below 🤖

The cast_value codec now checks that the array's fill value survives a round trip through the codec, as the cast_value spec requires. Previously out_of_range="clamp" or "wrap", or rounding, could encode the fill value to something that decodes to a different value, so the unwritten part of a partly written chunk read back as that value instead of the fill value. For example, a float32 array with fill value 1000.0 and an int8 target with clamping read 127.0 there. Creating such an array now raises a ValueError that names the fill value, both data types, and the value it decodes to. Opening an array already stored with such a fill value still works: it keeps its fill value, and a warning says what its partly written chunks hold and how to fix it. The check runs once when metadata is created or read, not for every chunk, and the codecs inside a shard are checked with the array's fill value rather than the data type's default scalar, so a shard's codec chain is no longer refused for a fill value it would never be asked to encode.

Spec

From the data_type section of the cast_value spec:

The fill value of the output array MUST be cast to the target data type using the same casting semantics as elements. … Implementations SHOULD validate at metadata construction time that the fill value can be successfully cast in both the forward (encode) and inverse (decode) directions using the codec's configuration. If the fill value cannot survive a round-trip cast, implementations MUST treat this as an error.

CastValue.resolve_metadata cast the fill value forward only, with the encode scalar_map, and never checked the inverse.

Reproduction (before this change, cast-value-rs 0.4.2)

import zarr
from zarr.storage import MemoryStore
a = zarr.create_array(MemoryStore(), shape=(4,), chunks=(2,), dtype="float32", fill_value=1000.0,
                      filters=[{"name": "cast_value", "configuration": {"data_type": "int8", "out_of_range": "clamp"}}],
                      compressors=None)
a[0] = 5.0
a[:]  # array([   5.,  127., 1000., 1000.], dtype=float32)

Element 1 sits in a partly written chunk, so it is stored as the encoded fill (127) and decodes to 127.0. Elements 2 and 3 are in a chunk that was never written, so they read the fill directly. After this change, create_array raises:

ValueError: The fill value 1000.0 does not survive a round trip through the cast_value codec: it encodes from float32 to 127 in int8, which decodes back to 127.0 in float32. Choose a fill value that the codec's data_type, rounding, out_of_range, and scalar_map preserve.

An array already stored that way opens with a ZarrUserWarning that carries the same text, followed by what it means for the stored chunks and how to fix the array, and reads exactly as before.

Where the check runs

The check is not in resolve_metadata, which the chunk pipeline calls for every chunk read or written and which ShardingCodec calls with the data type's default scalar when it validates its inner codecs. Instead:

• CastValue._check_fill_value(spec) does the round trip for one codec and raises.
• _check_fill_value_round_trip(metadata) threads the array's fill value through the codecs, as a chunk is encoded, and calls _check_fill_value on every cast_value codec with the fill value it would receive, descending into the codecs of a sharding codec.
• AsyncArray._create_metadata_v3, which every array creation goes through, calls it and lets the error propagate.
• ArrayV3Metadata.from_dict calls it after constructing the metadata and turns the error into a Reading warning through the existing metadata repair machinery, so an existing array stays readable.

resolve_metadata keeps casting the fill value forward only, as it did before, so the per-chunk cost is unchanged. It now wraps a failed forward cast in a ValueError that names the fill value and both data types, which _check_fill_value reuses.

Tests

In tests/test_codecs/test_cast_value.py:

Test	Case	Result
test_fill_value_roundtrip_accepted[...] (7 cases)	NaN through a scalar map, in-range clamp and wrap, int to float, integral float to int, exact float narrowing, signed zero float to float	accepted; the encoded fill is stored and reads back bit for bit
test_fill_value_partial_chunk_without_check_reads_wrong_value	float32 1000.0 to int8, clamp	with the check patched out, the unwritten element reads 127.0; creation raises; the stored array opens with a warning and reads the same values
test_fill_value_checked_inside_shards	shards with ScaleOffset(offset=100) then uint16 clamp, fill 100.0; shards with int8 clamp, fill 1000.0	the first is accepted although the default scalar 0.0 would be lost; the second raises
test_resolve_metadata_only_casts_forward	float32 1000.0 to int8, clamp	resolve_metadata returns 127 without raising
test_fill_value_clamp_out_of_range_rejected	float32 1000.0 to int8, clamp	raises, comes back 127.0
test_fill_value_wrap_out_of_range_rejected	int16 300 to int8, wrap	raises, comes back 44
test_fill_value_lost_to_rounding_rejected	float64 1.5 to int8, nearest-even	raises, comes back 2.0
test_fill_value_signed_zero_lost_rejected	float32 -0.0 to int8	raises, comes back 0.0
test_fill_value_nan_to_int_without_scalar_map_rejected	float32 NaN to int8, no map	raises, forward cast fails
test_fill_value_cannot_decode_rejected	int32 0 to float64, encode 0 to NaN, no decode map	raises, inverse cast fails

Results with uv sync --extra cast-value-rs --extra optional --group test (cast-value-rs 0.4.2):

• tests/ (full suite, -n auto, without tests/test_regression): 12520 passed, 1276 skipped, 4 xfailed
• prek run on the changed files: ruff check, ruff format, codespell, numpydoc validation and the rest pass; the mypy hook reports only the pre-existing type: ignore[no-any-return] in _do_cast as unused, because this environment has the cast-value-rs extra installed and the package ships type stubs, which the lockfile environment CI uses does not have

Bookkeeping

• Based on main at bd0dc84.
• Supersedes the first version of this PR, which did the check inside resolve_metadata. That refused to open arrays already stored with such a fill value, doubled the per-chunk casts, and rejected sharded ScaleOffset plus clamping chains through the default-scalar probe of ShardingCodec.validate (the sharded half of ShardingCodec.validate incorrectly assumes that Zdtype.default_scalar is always a valid value #4511).
• Relates to ShardingCodec.validate incorrectly assumes that Zdtype.default_scalar is always a valid value #4511 and fix: tolerate unencodable default_scalar in ShardingCodec.validate #4526: this PR does not change how ShardingCodec.validate probes its inner codecs, so an inner chain that cannot encode the default scalar at all (no clamping or wrapping) still fails there.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.72%. Comparing base (bd0dc84) to head (cd5fa7f).
⚠️ Report is 7 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4510      +/-   ##
==========================================
+ Coverage   94.71%   94.72%   +0.01%     
==========================================
  Files          94       94              
  Lines       13663    13700      +37     
==========================================
+ Hits        12941    12978      +37     
  Misses        722      722              

Files with missing lines	Coverage Δ
src/zarr/codecs/cast_value.py	98.93% <100.00%> (+0.19%)	⬆️
src/zarr/core/array.py	98.44% <100.00%> (+<0.01%)	⬆️
src/zarr/core/metadata/v3.py	96.95% <100.00%> (+0.04%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.