Repository navigation
Add lossless Zstandard snapshot compression #8724
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Jens Hedegaard Nielsen (jenshnielsen)
wants to merge
5
commits into
microsoft:main
Choose a base branch
from
jenshnielsen:feature/zstandard-snapshots
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
7c935a0
Add lossless Zstandard snapshot compression
jenshnielsen b092ab1
Make snapshot compression opt-in without a database upgrade
jenshnielsen 3c4bfb8
Read snapshots by run ID
jenshnielsen 635c703
Decode snapshots requested from get_db_overview
jenshnielsen 9582151
State which QCoDeS versions read each snapshot format
jenshnielsen File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| Snapshots can be stored losslessly using versioned, checksummed Zstandard | ||
| level 1 compression by setting ``dataset.snapshot_compression`` to ``"zstd"``. | ||
| Compressed and uncompressed snapshots are always readable, and snapshot APIs and | ||
| NetCDF exports return the complete JSON. Uncompressed snapshots, the default, | ||
| can be read by all QCoDeS versions; compressed snapshots can only be read by | ||
| QCoDeS 0.64.0 and later. No database upgrade is required. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| """Versioned, lossless storage of snapshot JSON strings in SQLite.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import sys | ||
|
|
||
| if sys.version_info >= (3, 14): | ||
| from compression import zstd | ||
| else: | ||
| from backports import zstd | ||
|
|
||
| _MAGIC = b"QCoDeS.snapshot\x00" | ||
| _VERSION_1 = _MAGIC + b"\x01" | ||
|
|
||
|
|
||
| def encode_snapshot(snapshot: str, compression: str) -> str | bytes: | ||
| """Encode raw JSON without parsing or changing its encoder semantics. | ||
|
|
||
| Args: | ||
| snapshot: The original JSON string, including any whitespace. | ||
| compression: ``zstd`` for version 1 storage, or ``none`` for legacy TEXT. | ||
|
|
||
| Returns: | ||
| The original string or a versioned BLOB containing one checksummed frame. | ||
|
|
||
| Raises: | ||
| ValueError: If the write policy is unsupported. | ||
|
|
||
| """ | ||
| if compression == "none": | ||
| return snapshot | ||
| if compression != "zstd": | ||
| raise ValueError(f"Unsupported snapshot compression: {compression!r}") | ||
| return _VERSION_1 + zstd.compress( | ||
| snapshot.encode("utf-8"), | ||
| options={ | ||
| zstd.CompressionParameter.compression_level: 1, | ||
| zstd.CompressionParameter.checksum_flag: 1, | ||
| }, | ||
| ) | ||
|
|
||
|
|
||
| def decode_snapshot(snapshot: str | bytes | None) -> str | None: | ||
| """Return original JSON from legacy TEXT, versioned BLOB, or SQL NULL. | ||
|
|
||
| Args: | ||
| snapshot: The value stored in the snapshot column. | ||
|
|
||
| Returns: | ||
| The original JSON string, or None for SQL NULL. | ||
|
|
||
| Raises: | ||
| ValueError: If the format is unknown, truncated, or has trailing data. | ||
| zstd.ZstdError: If the compressed frame or its checksum is corrupt. | ||
| UnicodeDecodeError: If the decompressed bytes are not UTF-8. | ||
|
|
||
| """ | ||
| if snapshot is None or isinstance(snapshot, str): | ||
| return snapshot | ||
| if not isinstance(snapshot, bytes) or not snapshot.startswith(_VERSION_1): | ||
| raise ValueError("Unsupported snapshot storage format or version") | ||
| decompressor = zstd.ZstdDecompressor() | ||
| raw = decompressor.decompress(snapshot[len(_VERSION_1) :]) | ||
| if not decompressor.eof or decompressor.unused_data: | ||
| raise ValueError("Incomplete snapshot frame or trailing data") | ||
| return raw.decode("utf-8") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.