Repository navigation
Add lossless Zstandard snapshot compression - #8724
Jens Hedegaard Nielsen (jenshnielsen) wants to merge 5 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #8724 +/- ##
==========================================
+ Coverage 75.70% 75.74% +0.03%
==========================================
Files 308 309 +1
Lines 32767 32803 +36
==========================================
+ Hits 24806 24845 +39
+ Misses 7961 7958 -3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
814de55 to
0142b5e
Compare
0142b5e to
dc6a728
Compare
Mark the capture validation example as PowerShell so Sphinx does not parse it as Python and fail the documentation build for PR microsoft#8724. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Snapshot handling breaks reads for runs sharing a result-table name and read-only loading of version-9 databases, while case-sensitive checks allow compression and validation bypasses.
3 open issues
What changed in this PR
Adds lossless Zstandard compression to QCoDeS dataset snapshot storage while preserving JSON-facing APIs and legacy TEXT decoding.
Changes:
- Introduces versioned, checksummed snapshot encoding with configurable compressed or uncompressed writes.
- Integrates encoding and decoding into SQLite persistence, adds schema version 10, and declares a conditional Zstandard backport dependency.
- Documents storage compatibility and adds regression coverage for round trips, corruption, copying, exports, and measurement backends.
| File | Description |
|---|---|
tests/dataset/test_snapshot_storage.py |
Adds codec and dataset persistence regression coverage. |
tests/dataset/test_database_creation_and_upgrading.py |
Updates the expected latest schema version to 10. |
src/qcodes/dataset/sqlite/snapshot_storage.py |
Implements versioned Zstandard encoding and decoding. |
src/qcodes/dataset/sqlite/queries.py |
Applies snapshot storage policy, validates schema compatibility, and decodes reads. |
src/qcodes/dataset/sqlite/db_upgrades/__init__.py |
Registers version 10 without rewriting existing snapshots. |
src/qcodes/dataset/data_set.py |
Routes raw snapshot reads through the decoding query helper. |
src/qcodes/configuration/qcodesrc.json |
Enables compressed snapshot writes by default. |
src/qcodes/configuration/qcodesrc_schema.json |
Defines supported snapshot compression settings. |
pyproject.toml |
Adds the Zstandard backport for Python versions below 3.14. |
docs/dataset/dataset_design.rst |
Documents storage format, configuration, compatibility, and local validation. |
docs/changes/newsfragments/8724.new |
Announces compression and database compatibility implications. |
Review effort: Balanced
Mark the capture validation example as PowerShell so Sphinx does not parse it as Python and fail the documentation build for PR microsoft#8724. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
dc6a728 to
0f37ba2
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Compressed-snapshot databases remain at schema version 9, allowing older readers to open them and fail during reads despite the PR description’s promised compatibility gate.
3 open issues
1 resolved since last review
Review effort: Balanced
Store snapshots as versioned, checksummed Zstandard frames while preserving JSON APIs and legacy TEXT reads. Add a configurable write policy, schema version gate, conditional backport dependency, documentation and regression coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The version 10 upgrade only changed user_version, since SQLite already accepts BLOBs in runs.snapshot, and it prevented older QCoDeS from opening upgraded databases at all. Remove it and default dataset.snapshot_compression to none so new snapshots stay readable by released QCoDeS. Compressed snapshots remain readable in all cases and can be enabled explicitly; the default can be switched once a release supports reading them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Runs can share a result table name, for example with split raw data storage and a constant format string, so looking up the snapshot by table name raised 'Expected only one row'. Select by run ID, as before, and decode the stored value. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
get_db_overview copies extra_columns straight from the runs table, so requesting the snapshot column returned the compressed BLOB for runs written with snapshot_compression set to zstd. Decode it like every other snapshot read. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Uncompressed snapshots are readable by all QCoDeS versions, compressed snapshots only by QCoDeS 0.64.0 and later. Say so explicitly instead of referring to versions without compressed snapshot support. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
0f37ba2 to
9582151
Compare


Add opt-in, lossless Zstandard compression for dataset snapshots, and the ability to read compressed snapshots in all cases.
dataset.snapshot_compression:"none"(default) writes plain JSON TEXT exactly as before;"zstd"writes a versioned, checksummed Zstandard level 1 BLOB into the existingruns.snapshotcolumn.snapshot,snapshot_raw,get_metadata("snapshot")), copies, and xarray/NetCDF exports keep returning the complete JSON, never compressed bytes. Corrupt or unknown compressed data raises instead of silently falling back.compression.zstd; earlier versions use the new conditional dependencybackports.zstd.No database upgrade and no version gate, by design. SQLite already accepts BLOBs in
runs.snapshot, and a schema version bump would stop older QCoDeS from opening the database at all, even for runs without compressed snapshots. Compression is therefore opt-in: QCoDeS releases without this support can still open the database and read everything except compressed snapshots, which fail to decode. The default can be switched to"zstd"once a release that reads compressed snapshots is available. The trade-off is documented indocs/dataset/dataset_design.rst.Validation: snapshot, schema, copy, export, in-memory, raw-storage and config tests (364 passed, 31 skipped), Pyright and pre-commit hooks pass locally. A database written by this branch was also read with released QCoDeS 0.63.0: plain snapshots work fully; compressed ones fail only where the snapshot itself is decoded, plus in-memory runs.