Skip to content

feat(router): report decrypted plaintext size in ListObjects responses - #51

Merged
louisgls merged 1 commit into
Intrinsec:mainfrom
pszafarczyk:ls-size
Jul 29, 2026
Merged

feat(router): report decrypted plaintext size in ListObjects responses#51
louisgls merged 1 commit into
Intrinsec:mainfrom
pszafarczyk:ls-size

Conversation

@pszafarczyk

Copy link
Copy Markdown

ListObjects/ListObjectsV2 previously reported the at-rest ciphertext size (plaintext + 28-byte AES-GCM-SIV envelope). Intercept bucket-level list requests, subtract the fixed encryption overhead from every , and clamp at 0. Bucket sub-resource GETs (acl, versioning, multipart listings, ?versions, …) are still forwarded unchanged.

  • cryptoutil: export EncryptionOverhead constant + invariant test
  • router: bucketOnlyPattern + isListObjects, wired into getHandler
  • handler: extract forwardUpstream; add handleListObjects (buffer, rewrite on 2xx, fix Content-Length)
  • listsize: byte-preserving regex rewrite + unit tests
  • e2e: assert v1 and v2 listings report plaintext size

Known limitation: objects not written through the proxy (legacy plaintext, server-side copies, multipart) are reported 28 bytes short / 0 after clamp, since list responses carry no per-object encryption metadata.

ListObjects/ListObjectsV2 previously reported the at-rest ciphertext size
(plaintext + 28-byte AES-GCM-SIV envelope). Intercept bucket-level list
requests, subtract the fixed encryption overhead from every <Size>, and
clamp at 0. Bucket sub-resource GETs (acl, versioning, multipart listings,
?versions, …) are still forwarded unchanged.

- cryptoutil: export EncryptionOverhead constant + invariant test
- router: bucketOnlyPattern + isListObjects, wired into getHandler
- handler: extract forwardUpstream; add handleListObjects (buffer, rewrite
on 2xx, fix Content-Length)
- listsize: byte-preserving <Size> regex rewrite + unit tests
- e2e: assert v1 and v2 listings report plaintext size

Known limitation: objects not written through the proxy (legacy plaintext,
server-side copies, multipart) are reported 28 bytes short / 0 after clamp,
since list responses carry no per-object encryption metadata.
@louisgls
louisgls merged commit 109b748 into Intrinsec:main Jul 29, 2026
@louisgls

Copy link
Copy Markdown
Collaborator

Merged in #55 as part of the v1.9.0 release. Reviewed it end to end while integrating — the approach holds up: buffering a list page (≤1000 keys) is safe, the sub-resource exclusion list is thorough, and recomputing Content-Length after the rewrite is the detail that most implementations get wrong. Verified against MinIO through the added e2e coverage.

Three observations for a follow-up, none blocking:

1. Path-style addressing only. bucketOnlyPattern is ^/([^/?]+)/?$, which requires at least one path segment. With virtual-host-style addressing (bucket.s3.example.com/) the request path is just /, so isListObjects returns false and sizes are not rewritten. The behaviour therefore depends on how the client is configured — our own deployment uses s3forcepathstyle: true so it is unaffected, but a client using the default addressing style would silently get ciphertext sizes.

2. Content-encoded responses. If an upstream ever returns a gzip-encoded list body, io.ReadAll yields the compressed bytes, the regex matches nothing, and the response passes through with unmodified sizes. Silent no-op rather than corruption, so the failure mode is benign, but it is invisible. Checking resp.Header.Get("Content-Encoding") and skipping (or logging) would make it explicit.

3. Mixed buckets. Documented in the changelog already, just restating the operational shape: in a bucket where some objects were not written through the proxy, those are reported 28 bytes short — or 0 once clamped, for anything under 28 bytes. Fine for a bucket written exclusively through the proxy (our Loki bucket), worth a warning in the README for anyone pointing this at an existing bucket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants