[python] Add ACT storage benchmark and contiguous windows - #9466
Open
YannByron wants to merge 15 commits into
Open
[python] Add ACT storage benchmark and contiguous windows#9466YannByron wants to merge 15 commits into
YannByron wants to merge 15 commits into
Conversation
YannByron
force-pushed
the
m0/pr3-act-benchmark
branch
from
August 29, 2026 14:53
622ceae to
5d8890e
Compare
YannByron
marked this pull request as draft
August 29, 2026 14:54
YannByron
force-pushed
the
m0/pr3-act-benchmark
branch
2 times, most recently
from
September 1, 2026 03:24
fb77b0e to
7b2b5a4
Compare
YannByron
marked this pull request as ready for review
September 1, 2026 03:51
YannByron
marked this pull request as draft
September 1, 2026 03:52
YannByron
marked this pull request as ready for review
September 2, 2026 01:14
JingsongLi
reviewed
Sep 2, 2026
JingsongLi
reviewed
Sep 2, 2026
JingsongLi
reviewed
Sep 2, 2026
JingsongLi
reviewed
Sep 2, 2026
JingsongLi
reviewed
Sep 2, 2026
JingsongLi
reviewed
Sep 2, 2026
Add a generic snapshot-pinned window Dataset that builds a lightweight row-id index, reads projected payloads on demand, and handles continuity, tail policies, transforms, and multi-worker DataLoader use. Co-Authored-By: Codex <noreply@anthropic.com> Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5.6-sol Co-Authored-By: Codex <noreply@ai-tool.com> AI-Contributed/Feature: 497/497 AI-Contributed/UT: 314/314
Run both backends through one deterministic ACT harness and pin lazy Paimon windows to the normalization snapshot. Adapt the benchmark to the rebased lazy source contract, coalesce overlapping batch reads, and measure Python allocations outside wall-clock timing. Co-Authored-By: Codex <noreply@anthropic.com> Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5.6-sol Co-Authored-By: Codex <noreply@ai-tool.com> AI-Contributed/Feature: 1418/1418 AI-Contributed/UT: 363/363
Read observation images only at each window anchor and tolerate installed packages without a Git checkout. Improve public option documentation and CLI guidance. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5 AI-Contributed/Feature: 200/200 AI-Contributed/UT: 25/25
Fetch several physical batches at once while preserving the logical batch boundaries consumed by ACT training. Keep warmup isolation, cursor replay, and snapshot semantics unchanged. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-5 AI-Contributed/Feature: 102/102 AI-Contributed/UT: 105/105
Use eight logical batches per physical fetch, based on the current Vortex-backed RoboMIND layout. Preserve logical batch boundaries while documenting the per-worker memory tradeoff. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-5 AI-Contributed/Feature: 6/6 AI-Contributed/UT: 4/4
Reuse the guarded LeRobot dependency set for the ACT extra, clarify batch-fetch measurement terminology, and strengthen the timed plural-fetch assertion. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5 AI-Contributed/Feature: 136/136 AI-Contributed/UT: 54/54
Skip the optional ACT test module before importing unavailable dependencies and keep its synthetic RoboMIND fixture independent of Vortex. Co-Authored-By: Codex <noreply@openai.com> AI-Model: gpt-5 AI-Contributed/Feature: 0/0 AI-Contributed/UT: 23/23
Separate experiment preparation, backend execution, and result comparison while preserving a shared ACT workload contract. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-5.6-sol Co-Authored-By: Codex <noreply@ai-tool.com> Co-Authored-By: Codex <noreply@openai.com> AI-Contributed/Feature: 3667/3667 AI-Contributed/UT: 1330/1330
Avoid reading the full qpos horizon when ACT only consumes the anchor observation. Add a golden backend sample contract and explicit HDF5 index bounds coverage. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-5.6-sol Co-Authored-By: Codex <noreply@ai-tool.com> AI-Contributed/Feature: 13/13 AI-Contributed/UT: 99/99
Document contiguous ordering, anchor-column cardinality, batching, snapshot pinning, and map-style index behavior. Strengthen plural-access and pickle round-trip coverage. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-5.6-sol Co-Authored-By: Codex <noreply@ai-tool.com> AI-Contributed/Feature: 87/87 AI-Contributed/UT: 56/56
Add the required blank line before the nested dataset factory and remove the extra module-level blank line. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-5.6-sol Co-Authored-By: Codex <noreply@ai-tool.com> Co-Authored-By: Codex <noreply@openai.com> AI-Contributed/Feature: 2/2 AI-Contributed/UT: 0/0
Skip only the ACT runtime ownership checks when Torch or Pillow is absent, while keeping dependency-free comparison tests active. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-5.6-sol Co-Authored-By: Codex <noreply@ai-tool.com> Co-Authored-By: Codex <noreply@openai.com> AI-Contributed/Feature: 0/0 AI-Contributed/UT: 8/8
Measure four complete physical fetch groups by default so the first Paimon fetch does not dominate steady-state throughput. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-5.6-sol AI-Contributed/Feature: 4/4 AI-Contributed/UT: 9/9
Avoid coupling tests to tuned experiment values and fold redundant integration assertions into the paired backend parity test. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-5.6-sol AI-Contributed/Feature: 0/0 AI-Contributed/UT: 174/174
Resolve deferred MAP BLOB reads, snapshot option conflicts, and shared mutable window cells. Tighten fetch timing, runtime identity, and Python version reporting. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-5.6-sol AI-Contributed/Feature: 100/100 AI-Contributed/UT: 149/149
YannByron
force-pushed
the
m0/pr3-act-benchmark
branch
from
September 2, 2026 09:16
42be5b2 to
705683a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a lazy, snapshot-pinned contiguous-window Dataset and a reproducible ACT
storage benchmark for RoboMIND HDF5 and Paimon data.
HDF5 and Paimon runs are independent commands. They consume one resolved
experiment containing the same configuration, normalization, seed, episode
selection, snapshot, and logical window sequence. Comparison verifies that
contract, runtime environment, tensors, and loss traces before reporting
performance ratios.
Changes
ContiguousWindowDatasetwith deterministic group/order indexing,row-ID reads, snapshot pinning, tail policies, anchor-only projections,
transforms/adapters, and coalesced plural
__getitems__access.BLOBandMAP<..., BLOB>values lazily, isolate mutablevalues between overlapping windows, and normalize conflicting scan options
before snapshot reads.
prepare,run, andcompareACT commands with separate HDF5 andPaimon adapters and one shared model/training harness.
experiment into every result artifact.
source identity, tensor fingerprints, losses, timing, and Python allocation
metrics. Batch-fetch timing excludes tensor validation.
boundaries, and Python 3.10 requirement.
Testing
origin/masterat8d67f8c6f.python -m pytest pypaimon/tests/contiguous_window_dataset_test.py pypaimon/tests/act_benchmark_test.py pypaimon/tests/act_runner_test.py -q(42 passed).python -m pytest pypaimon/acceptance -q(3 passed).null/empty values, mutable overlapping/duplicate windows, latest-full scan
pinning, fetch-only timing, runtime identity, and the Python version floor.
738 files have valid license headers).python -m compileall -q pypaimon/benchmark/act pypaimon/multimodal.git diff --check.the window Dataset, and guarded optional dependencies.
A raw local
python -m pytest -qrun completed with4691 passed, 137 skipped. Its remaining failures were environment-specific: one missingDataFusion dependency passed after installing the CI version, the macOS
torchruntest timed out, and Java interoperability fixtures were not built.Notes
quality are not validated by this local CPU benchmark.
public API.