Skip to content

feat: indexed aggregates behind --indexed-aggregates - #61

Closed
SgtPooki wants to merge 1 commit into
mainfrom
podsi
Closed

feat: indexed aggregates behind --indexed-aggregates#61
SgtPooki wants to merge 1 commit into
mainfrom
podsi

Conversation

@SgtPooki

Copy link
Copy Markdown
Collaborator

What changed

Adds an opt-in indexed-aggregate mode: plan --indexed-aggregates packs sub-pieces into FRC-0058 data segment aggregates, redirect-serve streams the assembled aggregate at /piece/{root} (each sub-piece hash-verified on the way out), and pdp-submit pulls that one stream and adds the root as its own single sub-piece — the 1:1 shape Curio's v0 add requires. The layout is built in pure JS (ipfs2foc-core/indexed-aggregate) from the @web3-storage/data-segment dep, with offsets and index serialization matched against go-data-segment source. Default (bare) aggregation is unchanged.

This targets the provider-side aggregate indexing being ported from Curio's mk20 pipeline into the PDPv0 path; keep on this branch until that lands and a calibration round-trip passes.

How to verify

pnpm -r typecheck && pnpm biome check . && pnpm -r test — the key tests recompute the piece commitment over the assembled bytes (unit and over real HTTP) and require it to equal the aggregate root, the same check the provider runs on pull.

Notes / risks

New indexed column on aggregates (pre-release, no migration). Indexed aggregates need --source-base; the stateless relay cannot assemble the stream.

@FilOzzy FilOzzy added this to FOC Jul 20, 2026
@github-project-automation github-project-automation Bot moved this to 📌 Triage in FOC Jul 20, 2026
@SgtPooki
SgtPooki marked this pull request as draft August 5, 2026 16:38
@SgtPooki SgtPooki moved this from 📌 Triage to ⌨️ In Progress in FOC Aug 5, 2026
@SgtPooki

Copy link
Copy Markdown
Collaborator Author

closing this because its using an old optimization attempt that is a little more complex than we want to offer to self-serve users. However, for those more technically minded and wanting to optimize certain avenues, this may be a reasonable approach for you.

@SgtPooki SgtPooki closed this Aug 18, 2026
@github-project-automation github-project-automation Bot moved this from ⌨️ In Progress to 🎉 Done in FOC Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: 🎉 Done

Development

Successfully merging this pull request may close these issues.

2 participants