Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/changes/newsfragments/8724.new
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
Snapshots can be stored losslessly using versioned, checksummed Zstandard
level 1 compression by setting ``dataset.snapshot_compression`` to ``"zstd"``.
Compressed and uncompressed snapshots are always readable, and snapshot APIs and
NetCDF exports return the complete JSON. Uncompressed snapshots, the default,
can be read by all QCoDeS versions; compressed snapshots can only be read by
QCoDeS 0.64.0 and later. No database upgrade is required.
49 changes: 49 additions & 0 deletions docs/dataset/dataset_design.rst
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,55 @@ SQLite database. This is not an intrinsic limitation of the dataset and
measurement layer. It is possible that at a future state support for writing
to a different backend will be added.

.. _sec:design_snapshot_storage:

Snapshot storage
================

The ``runs.snapshot`` column holds either JSON TEXT or a versioned, compressed
BLOB. Both are read transparently, independent of the current write policy;
SQL NULL continues to mean no snapshot. The write policy is controlled by::

qcodes.config.dataset.snapshot_compression = "zstd"

The default, ``"none"``, writes uncompressed JSON TEXT, which all QCoDeS
versions can read. ``"zstd"`` writes lossless Zstandard level 1 BLOBs, which
can only be read by QCoDeS 0.64.0 and later. The policy
applies to each new write, including overwrites, database copies and imported
metadata; it never rewrites existing snapshots, so one database may contain
both representations. No database schema change or upgrade is involved;
SQLite already allows BLOB values in this column.

The BLOB format is the ASCII bytes ``QCoDeS.snapshot``, a NUL byte, a version
byte (currently ``0x01``), then exactly one Zstandard frame with a checksum.
The frame contains the original JSON string encoded as UTF-8. Compression
does not parse or re-encode JSON, drop fields, deduplicate values, or change
NumPy/custom encoder, complex number, uncertainty or non-finite float semantics.
Unknown versions, truncated or corrupt frames, checksum failures and invalid
UTF-8 raise exceptions; there is no fallback to an empty or legacy snapshot.
JSON parsing remains the responsibility of the decoded snapshot API, so
malformed JSON can still be stored/read raw and raises when decoded.

``DataSet.snapshot`` returns complete decoded contents; ``snapshot_raw``,
``_snapshot_raw``, ``get_metadata("snapshot")`` and low-level raw run attributes
return JSON strings, never compressed bytes. Xarray and NetCDF exports also
contain JSON strings. In-memory datasets retain JSON strings and use this write
policy when persisting snapshot metadata. Results backends and foreground or
background result writers do not change the snapshot format or startup order.
CPython 3.14 and newer use ``compression.zstd``; earlier supported Python
versions use the declared ``backports.zstd`` runtime dependency.

Compressed snapshots are opt-in because QCoDeS versions before 0.64.0 cannot
decode them. Those versions still open the database, and runs with
uncompressed snapshots are unaffected. For runs written with ``"zstd"``, they
can still read the measured data and other metadata, but accessing
the snapshot, exporting to xarray or NetCDF and extracting the run into another
database fail; ``get_metadata("snapshot")`` returns the raw bytes; and runs
without a results table, such as those written by ``DataSetInMem``, cannot be
loaded at all. Direct SQL consumers must handle BLOBs explicitly rather than
passing them to SQLite JSON functions. Exporting to NetCDF with QCoDeS 0.64.0 or
later retains the snapshot as JSON for interchange with earlier versions.

.. _sec:design_split_storage:

Split Raw Data Storage
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ license = "MIT"
readme = "README.rst"
requires-python = ">=3.12"
dependencies = [
"backports.zstd>=1.0.0; python_version < '3.14'",
"broadbean>=0.11.0",
"h5netcdf>=1.3.0",
"h5py>=3.10.0",
Expand Down
1 change: 1 addition & 0 deletions src/qcodes/configuration/qcodesrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@
"sample": 0
},
"dataset": {
"snapshot_compression": "none",
"write_in_background": false,
"write_period": 5.0,
"use_threads": false,
Expand Down
6 changes: 6 additions & 0 deletions src/qcodes/configuration/qcodesrc_schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -303,6 +303,12 @@
"dataset": {
"type": "object",
"properties": {
"snapshot_compression": {
"type": "string",
"enum": ["none", "zstd"],
"default": "none",
"description": "Snapshot write policy: uncompressed JSON TEXT, readable by all QCoDeS versions, or lossless Zstandard level 1 BLOBs, readable by QCoDeS 0.64.0 and later. Reads support both independently of this setting."
},
"write_in_background": {
"type": "boolean",
"default": false,
Expand Down
9 changes: 4 additions & 5 deletions src/qcodes/dataset/data_set.py
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@
insert_many_values,
select_one_where,
)
from qcodes.dataset.sqlite.snapshot_storage import decode_snapshot
from qcodes.utils import (
NumpyJSONEncoder,
QCoDeSDeprecationWarning,
Expand Down Expand Up @@ -497,11 +498,9 @@ def snapshot(self) -> dict[str, Any] | None:
@property
def _snapshot_raw(self) -> str | None:
"""Snapshot of the run as a JSON-formatted string (or None)"""
snapshot_raw = select_one_where(
self.conn, "runs", "snapshot", "run_id", self.run_id
)
assert isinstance(snapshot_raw, (str, type(None)))
return snapshot_raw
stored = select_one_where(self.conn, "runs", "snapshot", "run_id", self.run_id)
assert isinstance(stored, (str, bytes, type(None)))
return decode_snapshot(stored)

@property
def snapshot_raw(self) -> str | None:
Expand Down
3 changes: 3 additions & 0 deletions src/qcodes/dataset/sqlite/db_overview.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
conn_from_dbpath_or_conn,
)
from qcodes.dataset.sqlite.query_helpers import is_column_in_table
from qcodes.dataset.sqlite.snapshot_storage import decode_snapshot

if TYPE_CHECKING:
from collections.abc import Sequence
Expand Down Expand Up @@ -283,6 +284,8 @@ def get_db_overview(
extra = {
col: row[n_fixed + i] for i, col in enumerate(valid_extra_columns)
}
if "snapshot" in extra:
extra["snapshot"] = decode_snapshot(extra["snapshot"])
# The keys of ``extra`` are only known at runtime (they are the
# user-supplied ``extra_columns``), so they cannot be part of
# the closed ``RunOverviewDict`` definition.
Expand Down
18 changes: 15 additions & 3 deletions src/qcodes/dataset/sqlite/queries.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@
sql_placeholder_string,
update_where,
)
from qcodes.dataset.sqlite.snapshot_storage import decode_snapshot, encode_snapshot
from qcodes.parameters import ParamSpecBase
from qcodes.utils import list_of_data_to_maybe_ragged_nd_array

Expand Down Expand Up @@ -1761,7 +1762,7 @@ def create_run(
)
if metadata:
add_data_to_dynamic_columns(conn, run_id, metadata)
if snapshot_raw:
if snapshot_raw is not None:
add_data_to_dynamic_columns(conn, run_id, {"snapshot": snapshot_raw})
_update_experiment_run_counter(conn, exp_id, run_counter)
if create_run_table:
Expand Down Expand Up @@ -1838,6 +1839,9 @@ def get_data_by_tag_and_table_name(
data = None
else:
raise
if tag == "snapshot":
assert isinstance(data, (str, bytes, type(None)))
return decode_snapshot(data)
return data


Expand Down Expand Up @@ -1960,6 +1964,14 @@ def update_columns(

"""
validate_dynamic_column_data(data)
if table_name == "runs" and "snapshot" in data:
snapshot = data["snapshot"]
if not isinstance(snapshot, str):
raise TypeError("Snapshot must be a raw JSON string")
data = {
**data,
"snapshot": encode_snapshot(snapshot, config.dataset.snapshot_compression),
Comment thread
jenshnielsen marked this conversation as resolved.
Resolved
}
update_where(conn, table_name, "rowid", row_id, **data)


Expand Down Expand Up @@ -2307,7 +2319,7 @@ def get_raw_run_attributes(
assert isinstance(name, str)

rawsnapshot = select_one_where(conn, "runs", "snapshot", "guid", guid)
assert isinstance(rawsnapshot, (str, type(None)))
assert isinstance(rawsnapshot, (str, bytes, type(None)))
output: RawRunAttributesDict = {
"run_id": run_id,
"experiment": experiment,
Expand All @@ -2320,7 +2332,7 @@ def get_raw_run_attributes(
"metadata": get_metadata_from_run_id(conn, run_id),
"parent_dataset_links": get_parent_dataset_links(conn, run_id),
"run_description": get_run_description(conn, run_id),
"snapshot": rawsnapshot,
"snapshot": decode_snapshot(rawsnapshot),
}

return output
Expand Down
66 changes: 66 additions & 0 deletions src/qcodes/dataset/sqlite/snapshot_storage.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
"""Versioned, lossless storage of snapshot JSON strings in SQLite."""

from __future__ import annotations

import sys

if sys.version_info >= (3, 14):
from compression import zstd
else:
from backports import zstd

_MAGIC = b"QCoDeS.snapshot\x00"
_VERSION_1 = _MAGIC + b"\x01"


def encode_snapshot(snapshot: str, compression: str) -> str | bytes:
"""Encode raw JSON without parsing or changing its encoder semantics.

Args:
snapshot: The original JSON string, including any whitespace.
compression: ``zstd`` for version 1 storage, or ``none`` for legacy TEXT.

Returns:
The original string or a versioned BLOB containing one checksummed frame.

Raises:
ValueError: If the write policy is unsupported.

"""
if compression == "none":
return snapshot
if compression != "zstd":
raise ValueError(f"Unsupported snapshot compression: {compression!r}")
return _VERSION_1 + zstd.compress(
snapshot.encode("utf-8"),
options={
zstd.CompressionParameter.compression_level: 1,
zstd.CompressionParameter.checksum_flag: 1,
},
)


def decode_snapshot(snapshot: str | bytes | None) -> str | None:
"""Return original JSON from legacy TEXT, versioned BLOB, or SQL NULL.

Args:
snapshot: The value stored in the snapshot column.

Returns:
The original JSON string, or None for SQL NULL.

Raises:
ValueError: If the format is unknown, truncated, or has trailing data.
zstd.ZstdError: If the compressed frame or its checksum is corrupt.
UnicodeDecodeError: If the decompressed bytes are not UTF-8.

"""
if snapshot is None or isinstance(snapshot, str):
return snapshot
if not isinstance(snapshot, bytes) or not snapshot.startswith(_VERSION_1):
raise ValueError("Unsupported snapshot storage format or version")
decompressor = zstd.ZstdDecompressor()
raw = decompressor.decompress(snapshot[len(_VERSION_1) :])
if not decompressor.eof or decompressor.unused_data:
raise ValueError("Incomplete snapshot frame or trailing data")
return raw.decode("utf-8")
19 changes: 19 additions & 0 deletions tests/dataset/test_db_overview.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@

import pytest

import qcodes as qc
from qcodes.dataset import (
get_db_overview,
load_or_create_experiment,
Expand Down Expand Up @@ -155,6 +156,24 @@ def test_get_db_overview_extra_columns(db_conn: AtomicConnection) -> None:
assert "does_not_exist" not in overview[1]


def test_get_db_overview_decodes_snapshot_extra_column(
db_conn: AtomicConnection,
) -> None:
exp = load_or_create_experiment("exp", sample_name="sample", conn=db_conn)
plain = new_data_set("plain", exp_id=exp.exp_id, conn=db_conn)
plain.add_snapshot('{"plain": true}')
qc.config.dataset.snapshot_compression = "zstd"
compressed = new_data_set("compressed", exp_id=exp.exp_id, conn=db_conn)
compressed.add_snapshot('{"compressed": true}')
new_data_set("no_snapshot", exp_id=exp.exp_id, conn=db_conn)

overview = get_db_overview(conn=db_conn, extra_columns=["snapshot"])

assert overview[1]["snapshot"] == '{"plain": true}' # type: ignore[typeddict-item]
assert overview[2]["snapshot"] == '{"compressed": true}' # type: ignore[typeddict-item]
assert overview[3]["snapshot"] is None # type: ignore[typeddict-item]


def test_get_db_overview_accepts_connection(db_conn: AtomicConnection) -> None:
_create_runs(db_conn, n_runs=1)

Expand Down
Loading
Loading