LittDB compression by cody-littley · Pull Request #3769 · sei-protocol/sei-chain

cody-littley · 2026-07-16T14:28:45Z

Describe your changes and provide context

Adds optional compression for LittDB.

github-actions · 2026-07-16T14:30:31Z

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
`✅ passed`	`✅ passed`	`✅ passed`	`✅ passed`	Jul 20, 2026, 6:33 PM

codecov · 2026-07-16T14:39:11Z

Codecov Report

❌ Patch coverage is 69.93865% with 98 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.97%. Comparing base (ad863c7) to head (dab46c8).
⚠️ Report is 15 commits behind head on main.

Files with missing lines	Patch %	Lines
sei-db/db_engine/litt/metrics/littdb_metrics.go	0.00%	38 Missing ⚠️
sei-db/db_engine/litt/disktable/segment/segment.go	78.78%	15 Missing and 6 partials ⚠️
...ei-db/db_engine/litt/disktable/compression_loop.go	59.37%	8 Missing and 5 partials ⚠️
sei-db/db_engine/litt/types/compression.go	78.57%	10 Missing and 2 partials ⚠️
.../db_engine/litt/disktable/segment/metadata_file.go	75.00%	6 Missing and 2 partials ⚠️
sei-db/db_engine/litt/disktable/disk_table.go	94.73%	1 Missing and 1 partial ⚠️
...db_engine/litt/disktable/segment/segment_reader.go	50.00%	1 Missing and 1 partial ⚠️
sei-db/db_engine/litt/table_config.go	33.33%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3769      +/-   ##
==========================================
- Coverage   59.88%   58.97%   -0.91%     
==========================================
  Files        2288     2203      -85     
  Lines      190023   180436    -9587     
==========================================
- Hits       113786   106414    -7372     
+ Misses      66091    64664    -1427     
+ Partials    10146     9358     -788

Flag	Coverage Δ
sei-chain-pr	`64.88% <69.93%> (?)`
sei-db	`70.41% <ø> (ø)`
sei-db-state-db	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
sei-db/db_engine/litt/disktable/control_loop.go	`68.31% <100.00%> (+1.08%)`	⬆️
sei-db/db_engine/litt/disktable/flush_loop.go	`59.61% <100.00%> (ø)`
...ei-db/db_engine/litt/disktable/forward_iterator.go	`72.34% <100.00%> (+2.12%)`	⬆️
sei-db/db_engine/litt/disktable/keymap_manager.go	`75.47% <100.00%> (-0.16%)`	⬇️
sei-db/db_engine/litt/table.go	`100.00% <ø> (ø)`
sei-db/db_engine/litt/disktable/disk_table.go	`71.33% <94.73%> (+1.16%)`	⬆️
...db_engine/litt/disktable/segment/segment_reader.go	`68.42% <50.00%> (-1.95%)`	⬇️
sei-db/db_engine/litt/table_config.go	`88.88% <33.33%> (-11.12%)`	⬇️
.../db_engine/litt/disktable/segment/metadata_file.go	`70.90% <75.00%> (+3.16%)`	⬆️
sei-db/db_engine/litt/types/compression.go	`78.57% <78.57%> (ø)`
... and 3 more

... and 88 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

cursor · 2026-07-16T15:18:43Z

PR Summary

High Risk
Changes the storage engine write/read path, segment metadata versioning, crash recovery, and value size limits—errors could corrupt or misread existing data across restarts.

Overview
Adds optional per-table value compression for LittDB disk tables via TableConfig.Compression (default off; S2 supported). A new compression loop sits ahead of the control loop, encodes writes with a one-byte algorithm tag (store compressed only when smaller), and preserves message order so flush/shutdown stay correct behind in-flight writes.

On-disk format: segment metadata bumps to v4 with a persisted compression byte; legacy v3 segments still load as uncompressed. Values are written through WriteCompressed, decompressed on read, and max value size is tightened to 2³²−2 bytes to reserve space for the tag.

Semantics: secondary keys on compressed tables must be full-value aliases (no sub-range slicing). Iterator and keymap paths are updated accordingly; unflushed-cache batching tracks raw value bytes, not compressed on-disk sizes. OTel metrics cover compression latency and ratio. Extensive integration tests cover flush ordering, restart/toggle, recovery, GC, and iteration.

^{Reviewed by Cursor Bugbot for commit dab46c8. Bugbot is set up for automated code reviews on this repo. Configure here.}

cursor

Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 23cfd53. Configure here.}

yzang2019 · 2026-07-17T18:33:12Z

+	// CompressionS2 compresses each value with the S2 block codec (github.com/klauspost/compress/s2),
+	// a high-throughput, Snappy-compatible algorithm. The encoded block is self-describing: the
+	// decompressed length is recoverable from the block itself, so it is not stored separately.
+	CompressionS2 CompressionAlgorithm = 1


Do we plan to do any benchmark here for other alternatives ? I saw that MinLZ might be even faster? What about traditional ones like lz4, gzip, zstd?

I'm open to the idea of of considering alternate compression algorithms when we do formal benchmarking of compression.

Originally, I was planning on handing compression benchmarking off to Kartik. But since will be switching focus to integrations, it might be a little time until we get around to this. Opened a ticket so we don't forget about this. https://linear.app/seilabs/issue/STO-659/benchmark-littdb-compression

yzang2019 · 2026-07-17T18:44:14Z

+	compressed := make([][]byte, len(req.values))
+	var uncompressedBytes uint64
+	var compressedBytes uint64
+	for i, kv := range req.values {


The compression loop processes one batch at a time on a single goroutine, which is necessary to preserve flush ordering. Under high write throughput with large batches, compression could become the pipeline bottleneck. The design is correct as-is, but if this becomes a problem, we could parallelize compression within a batch (each value is independent) while keeping inter-batch ordering serial.

Added a godoc mentioning this, in case we end up getting bottlenecked here:

// Future work: if the compression thread ever becomes a bottleneck, consider the following tactics: // - Fan out batches, handle normal work serially. // Simple, but only helps with batch workloads. // - Fan out all requests, have this thread only responsible for maintaining ordering. // Complex but improves all worloads.

yzang2019 · 2026-07-17T18:45:10Z

+	case CompressionNone:
+		return src, nil
+	case CompressionS2:
+		decompressed, err := s2.Decode(nil, src)


Every Read on a compressed segment allocates a new buffer via s2.Decode(nil, src). For read-heavy workloads, this adds GC pressure. A sync.Pool of decompression buffers (sized to the expected decompressed length) would reduce allocations on the hot path.

We don't copy the data out of the buffers, so it would be very tricky to reuse these buffers. We'd have to expose the sync.Pool via API to the calling context, and require that callers to the DB recycle their buffers after they are are done with them. Since this would add a lot of complexity, I'm not sure it's worth doing , absent benchmark/profiling results that suggest golang GC pressure is degrading performance.

Thoughts?

… enabled

Cody Littley added 4 commits July 15, 2026 16:36

implement littDB compression

a0657db

cleanup

d037b88

unit tests

9c02ca0

godoc update

b386096

cody-littley requested a review from Kbhat1 July 16, 2026 14:28

cody-littley self-assigned this Jul 16, 2026

cody-littley added the non-app-hash-breaking label Jul 16, 2026

unit tests

23cfd53

cody-littley marked this pull request as ready for review July 16, 2026 15:17

cursor Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread sei-db/db_engine/litt/metrics/littdb_metrics.go

Comment thread sei-db/db_engine/litt/disktable/segment/metadata_file.go

Comment thread sei-db/db_engine/litt/disktable/segment/segment.go Outdated

This comment was marked as low quality.

Sign in to view

claude Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread sei-db/db_engine/litt/disktable/segment/segment.go

Comment thread sei-db/db_engine/litt/disktable/segment/metadata_file.go

yzang2019 approved these changes Jul 17, 2026

View reviewed changes

yzang2019 reviewed Jul 17, 2026

View reviewed changes

Cody Littley added 5 commits July 17, 2026 14:11

bugfix

448ea49

handle edge case where compressed data exceeds size of original data

817b3f5

added note about performance hotspot

82e6679

properly bound unflushed data cache memory footprint with compression…

ba73f62

… enabled

go fmt

dab46c8

This comment was marked as low quality.

Sign in to view

cody-littley added this pull request to the merge queue Jul 20, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 20, 2026

Uh oh!

Conversation

cody-littley commented Jul 16, 2026

Describe your changes and provide context

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

cursor Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as low quality.

Uh oh!

Uh oh!

Uh oh!

yzang2019 Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

cody-littley Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

yzang2019 Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cody-littley Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

yzang2019 Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

cody-littley Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

This comment was marked as low quality.

Uh oh!

This comment was marked as low quality.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jul 16, 2026 •

edited

Loading

codecov Bot commented Jul 16, 2026 •

edited

Loading

cursor Bot commented Jul 16, 2026 •

edited

Loading

yzang2019 Jul 17, 2026 •

edited

Loading