Skip to content

Add lossless Zstandard snapshot compression - #8724

Open
Jens Hedegaard Nielsen (jenshnielsen) wants to merge 5 commits into
microsoft:mainfrom
jenshnielsen:feature/zstandard-snapshots
Open

Jens Hedegaard Nielsen (jenshnielsen) wants to merge 5 commits into
microsoft:mainfrom
jenshnielsen:feature/zstandard-snapshots

Conversation

@jenshnielsen

@jenshnielsen Jens Hedegaard Nielsen (jenshnielsen) commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Add opt-in, lossless Zstandard compression for dataset snapshots, and the ability to read compressed snapshots in all cases.

  • New config option dataset.snapshot_compression: "none" (default) writes plain JSON TEXT exactly as before; "zstd" writes a versioned, checksummed Zstandard level 1 BLOB into the existing runs.snapshot column.
  • Both formats are always read, whatever the setting, so one database may contain both. Existing snapshots are never rewritten.
  • Snapshot APIs (snapshot, snapshot_raw, get_metadata("snapshot")), copies, and xarray/NetCDF exports keep returning the complete JSON, never compressed bytes. Corrupt or unknown compressed data raises instead of silently falling back.
  • Python 3.14+ uses the standard library compression.zstd; earlier versions use the new conditional dependency backports.zstd.

No database upgrade and no version gate, by design. SQLite already accepts BLOBs in runs.snapshot, and a schema version bump would stop older QCoDeS from opening the database at all, even for runs without compressed snapshots. Compression is therefore opt-in: QCoDeS releases without this support can still open the database and read everything except compressed snapshots, which fail to decode. The default can be switched to "zstd" once a release that reads compressed snapshots is available. The trade-off is documented in docs/dataset/dataset_design.rst.

Validation: snapshot, schema, copy, export, in-memory, raw-storage and config tests (364 passed, 31 skipped), Pyright and pre-commit hooks pass locally. A database written by this branch was also read with released QCoDeS 0.63.0: plain snapshots work fully; compressed ones fail only where the snapshot itself is decoded, plus in-memory runs.

@codecov

codecov Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.56098% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 75.74%. Comparing base (4ad3e84) to head (9582151).

Files with missing lines Patch % Lines
src/qcodes/dataset/sqlite/queries.py 90.90% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8724      +/-   ##
==========================================
+ Coverage   75.70%   75.74%   +0.03%     
==========================================
  Files         308      309       +1     
  Lines       32767    32803      +36     
==========================================
+ Hits        24806    24845      +39     
+ Misses       7961     7958       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Jens Hedegaard Nielsen (jenshnielsen) added a commit to jenshnielsen/Qcodes that referenced this pull request Nov 16, 2026
Mark the capture validation example as PowerShell so Sphinx does not parse it as Python and fail the documentation build for PR microsoft#8724.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Snapshot handling breaks reads for runs sharing a result-table name and read-only loading of version-9 databases, while case-sensitive checks allow compression and validation bypasses.

3 open issues
What changed in this PR

Adds lossless Zstandard compression to QCoDeS dataset snapshot storage while preserving JSON-facing APIs and legacy TEXT decoding.

Changes:

  • Introduces versioned, checksummed snapshot encoding with configurable compressed or uncompressed writes.
  • Integrates encoding and decoding into SQLite persistence, adds schema version 10, and declares a conditional Zstandard backport dependency.
  • Documents storage compatibility and adds regression coverage for round trips, corruption, copying, exports, and measurement backends.
File Description
tests/​dataset/​test_snapshot_storage.py Adds codec and dataset persistence regression coverage.
tests/​dataset/​test_database_creation_and_upgrading.py Updates the expected latest schema version to 10.
src/​qcodes/​dataset/​sqlite/​snapshot_storage.py Implements versioned Zstandard encoding and decoding.
src/​qcodes/​dataset/​sqlite/​queries.py Applies snapshot storage policy, validates schema compatibility, and decodes reads.
src/​qcodes/​dataset/​sqlite/​db_upgrades/​__init__.py Registers version 10 without rewriting existing snapshots.
src/​qcodes/​dataset/​data_set.py Routes raw snapshot reads through the decoding query helper.
src/​qcodes/​configuration/​qcodesrc.json Enables compressed snapshot writes by default.
src/​qcodes/​configuration/​qcodesrc_schema.json Defines supported snapshot compression settings.
pyproject.toml Adds the Zstandard backport for Python versions below 3.14.
docs/​dataset/​dataset_design.rst Documents storage format, configuration, compatibility, and local validation.
docs/​changes/​newsfragments/​8724.new Announces compression and database compatibility implications.

Review effort: Balanced

Comment thread src/qcodes/dataset/data_set.py Resolved
Comment thread src/qcodes/dataset/sqlite/queries.py Resolved
Jens Hedegaard Nielsen (jenshnielsen) added a commit to jenshnielsen/Qcodes that referenced this pull request Nov 16, 2026
Mark the capture validation example as PowerShell so Sphinx does not parse it as Python and fail the documentation build for PR microsoft#8724.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Compressed-snapshot databases remain at schema version 9, allowing older readers to open them and fail during reads despite the PR description’s promised compatibility gate.

3 open issues
1 resolved since last review

Review effort: Balanced

Comment thread tests/dataset/test_snapshot_storage.py Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The public database overview API returns compressed snapshot BLOBs instead of JSON strings, breaking callers that parse snapshots with json.loads.

1 open issue
3 resolved since last review

Review effort: Balanced

Comment thread src/qcodes/dataset/sqlite/queries.py Resolved
Store snapshots as versioned, checksummed Zstandard frames while preserving JSON APIs and legacy TEXT reads. Add a configurable write policy, schema version gate, conditional backport dependency, documentation and regression coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The version 10 upgrade only changed user_version, since SQLite already accepts BLOBs in runs.snapshot, and it prevented older QCoDeS from opening upgraded databases at all. Remove it and default dataset.snapshot_compression to none so new snapshots stay readable by released QCoDeS. Compressed snapshots remain readable in all cases and can be enabled explicitly; the default can be switched once a release supports reading them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Runs can share a result table name, for example with split raw data storage and a constant format string, so looking up the snapshot by table name raised 'Expected only one row'. Select by run ID, as before, and decode the stored value.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
get_db_overview copies extra_columns straight from the runs table, so requesting the snapshot column returned the compressed BLOB for runs written with snapshot_compression set to zstd. Decode it like every other snapshot read.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Uncompressed snapshots are readable by all QCoDeS versions, compressed snapshots only by QCoDeS 0.64.0 and later. Say so explicitly instead of referring to versions without compressed snapshot support.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the opt-in snapshot-compression change.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@jenshnielsen
Jens Hedegaard Nielsen (jenshnielsen) marked this pull request as ready for review November 16, 2026 10:50

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants