Document the Galexie data lake storage format by kaankacar · Pull Request #2827 · stellar/stellar-docs

kaankacar · 2026-09-08T00:16:46Z

🤖 Automated message from Kaan's Automated Triage Bot.

This adds a Data Lake Storage Format page for readers who consume a Galexie data lake directly, instead of through the ingest SDK or Stellar RPC. It summarizes SEP-54 and links to it as the specification: the object key layout, the reversed-sequence hex prefix, the zstd-compressed LedgerCloseMetaBatch payload, the .config.json manifest and the per-object metadata. Three existing pages now link to it: the Galexie overview, the Data Organization settings in the Galexie admin guide, and the RPC data lake integration guide.

Every fact came from a primary source or a live read:

SEP-54 is now a numbered SEP at ecosystem/sep-0054.md, status Draft, updated 2026-01-07.
The key format matches DataStoreSchema.GetObjectKeyFromSequenceNumber in go-stellar-sdk/support/datastore/schema.go.
.config.json and the field names match DatastoreManifest and manifestFilename in the same package.
The current suffix is .xdr.zst (ZstdCompressor.Name() returns zst); .xdr.zstd is the legacy form the code still reads.
LedgerCloseMetaBatch is in stellar-xdr Stellar-exporter.x and in stellar-xdr v28.0.0, which the Stellar CLI depends on.
The worked example key was fetched from the public bucket. It returns 200 and carries the metadata headers the page lists.

Refs #1605 (and not Closes), because SEP-54 is still status Draft. Please close the issue when you are satisfied with that half.

Add a Data Lake Storage Format page for readers who consume a Galexie data lake directly instead of through the ingest SDK or Stellar RPC. The page summarizes SEP-54 (Ledger Metadata Storage) and links to it as the specification: object key layout, the reversed-sequence hex prefix, the zstd-compressed LedgerCloseMetaBatch payload, the .config.json manifest and the per-object metadata. It also shows the layout of the public AWS Open Data lake with a worked read of one ledger. Link the new page from the Galexie overview, from the Data Organization settings in the Galexie admin guide and from the RPC data lake integration guide. Refs #1605

Copilot

Pull request overview

Adds documentation for directly consuming Galexie data lakes, aligned with SEP-54.

Changes:

Documents object layout, manifests, compression, metadata, and integrity considerations.
Adds a public AWS data lake example.
Links the guide from Galexie and RPC documentation.
Five code fences need text language identifiers (data-lake-format.mdx:20,30,36,88,107).

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`routes.txt`	Registers the new route.
`galexie/README.mdx`	Links to the storage guide.
`galexie/data-lake-format.mdx`	Adds the SEP-54-based guide.
`admin_guide/configuring.mdx`	Connects storage settings to the guide.
`rpc/admin-guide/data-lake-integration.mdx`	Links direct consumers to the guide.

Suppressed comments (4)

docs/data/indexers/build-your-own/galexie/data-lake-format.mdx:30

This code fence is missing a language identifier. Mark the partition-name example as text so syntax handling is explicit.

**docs/data/indexers/build-your-own/galexie/data-lake-format.mdx:36**
* This code fence is missing a language identifier. Mark these batch filenames as `text` to satisfy the docs fence convention.

docs/data/indexers/build-your-own/galexie/data-lake-format.mdx:88

This code fence is missing a language identifier. Mark the S3 paths as text so the fence declares its content type.

**docs/data/indexers/build-your-own/galexie/data-lake-format.mdx:107**
* This code fence is missing a language identifier. Mark the worked object key as `text` to comply with the documentation convention.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-09-08T00:21:07Z

PR Preview: https://developers-pr-2827.previews.kube001.services.stellar-ops.com

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

kaankacar · 2026-09-08T01:15:10Z

+
+## Configuration file
+
+Every data store holds a JSON configuration object at `<ledgers-path>/.config.json`. A client reads it to learn the batch and partition sizes before it builds any object key.


🤖 Automated message from Kaan's Automated Triage Bot.

Correct, and fixed in 5885d29. LoadSchema in support/datastore/configure.go falls back to the client's own ledgersPerFile and filesPerPartition when readManifest returns os.ErrNotExist.

The section now says SEP-54 requires the file, and adds a line for older stores that do not have it.

github-actions · 2026-09-08T01:14:41Z

PR Preview: https://developers-pr-2827.previews.kube001.services.stellar-ops.com

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

github-actions · 2026-09-08T01:19:58Z

PR Preview: https://developers-pr-2827.previews.kube001.services.stellar-ops.com

kaankacar · 2026-09-08T01:21:21Z

🤖 Automated message from Kaan's Automated Triage Bot.

Verified against SEP-54, the go-stellar-sdk datastore package and the live AWS bucket. I pushed two corrections: the Testnet path holds one data store per reset, and older stores have no .config.json.

All checks are green at 5885d29. Ready to merge.

ElliotFriend

this is a pretty good start. i've left some comments throughout the new page. i also want to get the eyes of someone from the data team on this, so our bases are covered.

ElliotFriend · 2026-09-08T17:26:26Z

+sidebar_position: 10
+---
+
+# Data Lake Storage Format


this line isn't actually necessary. it's the exact same text as the frontmatter's title field. in the absence of any <h1> tag (or the equivalent in markdown), docusaurus will automatically place an h1 element with the title's value.

ElliotFriend · 2026-09-08T17:28:23Z

+<ledgers-path>/<partition>/<batch>.xdr.zst
+```
+
+Galexie omits the partition directory when [`files_per_partition`](./admin_guide/configuring.mdx#data-organization-optional) is 1.


Suggested change

Galexie omits the partition directory when [`files_per_partition`](./admin_guide/configuring.mdx#data-organization-optional) is 1.

Galexie omits the partition directory when [`files_per_partition`](./admin_guide/configuring.mdx#data-organization-optional) is `1`.

ElliotFriend · 2026-09-08T17:31:48Z

+s3://aws-public-blockchain/v1.1/stellar/ledgers/pubnet
+```
+
+Testnet resets. Each reset starts a new data store, in a directory named after the reset date:


this line has a sentence fragment to begin it. this should be rephrased to maximize clarity.

ElliotFriend · 2026-09-08T17:34:37Z

+The Pubnet configuration file holds one ledger per batch and 64,000 batches per partition:
+
+```json title="v1.1/stellar/ledgers/pubnet/.config.json"
+{
+  "networkPassphrase": "Public Global Stellar Network ; September 2015",
+  "version": "1.0",
+  "compression": "zstd",
+  "ledgersPerBatch": 1,
+  "batchesPerPartition": 64000
+}
+```
+
+With those sizes, ledger 50000000 falls in the partition that starts at ledger 49984000, so its object key is:
+
+```text
+v1.1/stellar/ledgers/pubnet/FD054DFF--49984000-50047999/FD050F7F--50000000.xdr.zst
+```
+
+Download that ledger and print it as JSON with [`stellar xdr decode`](../../../../tools/cli/stellar-cli.mdx#stellar-xdr-decode):
+
+```sh
+curl -s https://aws-public-blockchain.s3.amazonaws.com/v1.1/stellar/ledgers/pubnet/FD054DFF--49984000-50047999/FD050F7F--50000000.xdr.zst \
+  | zstd -dc \
+  | stellar xdr decode --type LedgerCloseMetaBatch --input single --output json-formatted
+```


it feels like the testnet information above interrupts part of this configuration file section. the details about the AWS public lake could be its own section, but the config info here feels like it's more at home in the config section

sydneynotthecity

Overall I think it would be helpful for us to add when you should read directly from the data store yourself versus using RPC or ingest SDK. In this guide we don't make it clear when that would be the recommendation or preferred solution over other data services. Couple nits to improve readability of the page

sydneynotthecity · 2026-09-08T19:30:58Z

+
+Galexie stores ledger metadata in an object store as compressed XDR files.
+
+Most applications never read those files directly. Read them through the [ingest SDK](../ingest-sdk/README.mdx) or through [Stellar RPC](../../../apis/rpc/admin-guide/data-lake-integration.mdx) instead. Both hide the layout described here.


I think this could be written to something that is clearer, like:

Unless you are building for a highly specialized edge case, use the ingest SDK or Stellar RPC rather than reading these files directly. Both tools abstract away the underlying layout described here so your application doesn't have to manage it.

sydneynotthecity · 2026-09-08T19:34:55Z

+A data store does not necessarily hold the whole ledger history. List the oldest partition to find where its history starts.
+
+:::
+


It would be helpful if there was a full example of the full path + filename so a reader could see how all keys work together

sydneynotthecity · 2026-09-08T19:50:00Z

+
+Every batch in one data store holds the same number of ledgers. Refer to the [XDR](../../../../learn/fundamentals/data-format/xdr.mdx) documentation for more information on the encoding.
+
+Older data stores use the `.xdr.zstd` suffix instead of `.xdr.zst`. Both suffixes mean Zstandard. A client reads the suffix from an existing object rather than assuming one.


I would delete this, idk of any data stores with that standard and we updated all code to use the new zstd suffix

sydneynotthecity · 2026-09-08T19:52:08Z

+| `ledgersPerBatch` | integer | The number of ledgers in each batch file. |
+| `batchesPerPartition` | integer | The number of batch files in each partition directory. |
+
+The Galexie configuration file names the last two values [`ledgers_per_file` and `files_per_partition`](./admin_guide/configuring.mdx#data-organization-optional).


This hyperlinking is confusing. I would link file instead

sydneynotthecity · 2026-09-08T19:54:17Z

+
+The Galexie configuration file names the last two values [`ledgers_per_file` and `files_per_partition`](./admin_guide/configuring.mdx#data-organization-optional).
+
+Older data stores do not hold this file. A client that reads one must get the batch and partition sizes from its own configuration. The Go `datastore` package does that when the file is absent.


I would rephrase to reference the old versions of Galexie that do not include the config.json to make this less confusing. Something like "Data stores that are created by operating a version of Galexie X.X or earlier do not use a config.json file...."

kaankacar requested review from ElliotFriend and a balanced review from Copilot September 8, 2026 00:16

kaankacar mentioned this pull request Sep 8, 2026

Add documentation about galexie's file storage format #1605

Open

Copilot started reviewing on behalf of kaankacar September 8, 2026 00:17 View session

Copilot AI reviewed Sep 8, 2026

View reviewed changes

Comment thread docs/data/indexers/build-your-own/galexie/data-lake-format.mdx Outdated

github-actions Bot added the preview Preview builds for PRs by SDF employees. label Sep 8, 2026

Correct the Testnet data lake path and label the plain code fences

4e31e31

Copilot AI review requested due to automatic review settings September 8, 2026 01:10

github-actions Bot removed the preview Preview builds for PRs by SDF employees. label Sep 8, 2026

Copilot started reviewing on behalf of kaankacar September 8, 2026 01:10 View session

Copilot AI reviewed Sep 8, 2026

View reviewed changes

github-actions Bot added the preview Preview builds for PRs by SDF employees. label Sep 8, 2026

Note that older data stores have no config file

5885d29

Copilot AI review requested due to automatic review settings September 8, 2026 01:15

github-actions Bot removed the preview Preview builds for PRs by SDF employees. label Sep 8, 2026

Copilot started reviewing on behalf of kaankacar September 8, 2026 01:15 View session

Copilot AI reviewed Sep 8, 2026

View reviewed changes

github-actions Bot added the preview Preview builds for PRs by SDF employees. label Sep 8, 2026

kaankacar added the bot:ready-to-merge Bot verified and approved; waiting for a maintainer to merge label Sep 8, 2026

ElliotFriend reviewed Sep 8, 2026

View reviewed changes

sydneynotthecity reviewed Sep 8, 2026

View reviewed changes


		## Configuration file

		Every data store holds a JSON configuration object at `<ledgers-path>/.config.json`. A client reads it to learn the batch and partition sizes before it builds any object key.

	Galexie omits the partition directory when [`files_per_partition`](./admin_guide/configuring.mdx#data-organization-optional) is 1.
	Galexie omits the partition directory when [`files_per_partition`](./admin_guide/configuring.mdx#data-organization-optional) is `1`.


		Galexie stores ledger metadata in an object store as compressed XDR files.

		Most applications never read those files directly. Read them through the [ingest SDK](../ingest-sdk/README.mdx) or through [Stellar RPC](../../../apis/rpc/admin-guide/data-lake-integration.mdx) instead. Both hide the layout described here.

		A data store does not necessarily hold the whole ledger history. List the oldest partition to find where its history starts.

		:::


		Every batch in one data store holds the same number of ledgers. Refer to the [XDR](../../../../learn/fundamentals/data-format/xdr.mdx) documentation for more information on the encoding.

		Older data stores use the `.xdr.zstd` suffix instead of `.xdr.zst`. Both suffixes mean Zstandard. A client reads the suffix from an existing object rather than assuming one.


		The Galexie configuration file names the last two values [`ledgers_per_file` and `files_per_partition`](./admin_guide/configuring.mdx#data-organization-optional).

		Older data stores do not hold this file. A client that reads one must get the batch and partition sizes from its own configuration. The Go `datastore` package does that when the file is absent.

Uh oh!

Conversation

kaankacar commented Sep 8, 2026 • edited by ElliotFriend Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

github-actions Bot commented Sep 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

github-actions Bot commented Sep 8, 2026

Uh oh!

kaankacar commented Sep 8, 2026

Uh oh!

ElliotFriend left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sydneynotthecity left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kaankacar commented Sep 8, 2026 •

edited by ElliotFriend

Loading