Skip to content

[python] Add ACT storage benchmark and contiguous windows - #9466

Open
YannByron wants to merge 15 commits into
apache:masterfrom
YannByron:m0/pr3-act-benchmark
Open

[python] Add ACT storage benchmark and contiguous windows#9466
YannByron wants to merge 15 commits into
apache:masterfrom
YannByron:m0/pr3-act-benchmark

Conversation

@YannByron

@YannByron YannByron commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Add a lazy, snapshot-pinned contiguous-window Dataset and a reproducible ACT
storage benchmark for RoboMIND HDF5 and Paimon data.

HDF5 and Paimon runs are independent commands. They consume one resolved
experiment containing the same configuration, normalization, seed, episode
selection, snapshot, and logical window sequence. Comparison verifies that
contract, runtime environment, tensors, and loss traces before reporting
performance ratios.

Changes

  • Add ContiguousWindowDataset with deterministic group/order indexing,
    row-ID reads, snapshot pinning, tail policies, anchor-only projections,
    transforms/adapters, and coalesced plural __getitems__ access.
  • Resolve scalar BLOB and MAP<..., BLOB> values lazily, isolate mutable
    values between overlapping windows, and normalize conflicting scan options
    before snapshot reads.
  • Add prepare, run, and compare ACT commands with separate HDF5 and
    Paimon adapters and one shared model/training harness.
  • Package default experiment parameters and write the complete resolved
    experiment into every result artifact.
  • Record dependency/build versions, CPU identity/count, Torch thread settings,
    source identity, tensor fingerprints, losses, timing, and Python allocation
    metrics. Batch-fetch timing excludes tensor validation.
  • Document the window API, benchmark workflow, result comparison, measurement
    boundaries, and Python 3.10 requirement.

Testing

  • Rebased onto origin/master at 8d67f8c6f.
  • python -m pytest pypaimon/tests/contiguous_window_dataset_test.py pypaimon/tests/act_benchmark_test.py pypaimon/tests/act_runner_test.py -q (42 passed).
  • python -m pytest pypaimon/acceptance -q (3 passed).
  • Reviewer regressions cover map-only and mixed scalar/map BLOBs with
    null/empty values, mutable overlapping/duplicate windows, latest-full scan
    pinning, fetch-only timing, runtime identity, and the Python version floor.
  • Ruff and Flake8 on the changed Python files.
  • Python license check (738 files have valid license headers).
  • python -m compileall -q pypaimon/benchmark/act pypaimon/multimodal.
  • git diff --check.
  • Built and inspected the wheel for ACT modules, the default experiment,
    the window Dataset, and guarded optional dependencies.

A raw local python -m pytest -q run completed with 4691 passed, 137 skipped. Its remaining failures were environment-specific: one missing
DataFusion dependency passed after installing the CI version, the macOS
torchrun test timed out, and Java interoperability fixtures were not built.

Notes

  • Ingestion and canonical-action backfill are outside timed benchmark runs.
  • The benchmark does not control the OS page cache.
  • GPU, multi-worker loading, distributed training, recovery, and policy
    quality are not validated by this local CPU benchmark.
  • This adds Python APIs and benchmark tooling; it does not change an existing
    public API.
  • This builds on the RoboMIND AgileX HDF5 pipeline merged in [python] Add RoboMIND AgileX HDF5 pipeline #9445.

@YannByron
YannByron force-pushed the m0/pr3-act-benchmark branch from 622ceae to 5d8890e Compare August 29, 2026 14:53
@YannByron
YannByron marked this pull request as draft August 29, 2026 14:54
@YannByron
YannByron force-pushed the m0/pr3-act-benchmark branch 2 times, most recently from fb77b0e to 7b2b5a4 Compare September 1, 2026 03:24
@YannByron
YannByron marked this pull request as ready for review September 1, 2026 03:51
@YannByron
YannByron marked this pull request as draft September 1, 2026 03:52
@YannByron
YannByron marked this pull request as ready for review September 2, 2026 01:14
Comment thread paimon-python/pypaimon/multimodal/window_dataset.py
Comment thread paimon-python/pypaimon/benchmark/act/runner.py Outdated
Comment thread docs/docs/pypaimon/robomind-act-benchmark.md
Comment thread paimon-python/pypaimon/benchmark/act/harness.py Outdated
Comment thread paimon-python/pypaimon/multimodal/window_dataset.py Outdated
Comment thread paimon-python/pypaimon/multimodal/window_dataset.py
YannByron and others added 15 commits September 2, 2026 16:30
Add a generic snapshot-pinned window Dataset that builds a lightweight row-id index, reads projected payloads on demand, and handles continuity, tail policies, transforms, and multi-worker DataLoader use.

Co-Authored-By: Codex <noreply@anthropic.com>

Co-Authored-By: Codex <noreply@openai.com>

AI-Model: gpt-5.6-sol
Co-Authored-By: Codex <noreply@ai-tool.com>
AI-Contributed/Feature: 497/497
AI-Contributed/UT: 314/314
Run both backends through one deterministic ACT harness and pin lazy Paimon windows to the normalization snapshot. Adapt the benchmark to the rebased lazy source contract, coalesce overlapping batch reads, and measure Python allocations outside wall-clock timing.

Co-Authored-By: Codex <noreply@anthropic.com>

Co-Authored-By: Codex <noreply@openai.com>

AI-Model: gpt-5.6-sol
Co-Authored-By: Codex <noreply@ai-tool.com>
AI-Contributed/Feature: 1418/1418
AI-Contributed/UT: 363/363
Read observation images only at each window anchor and tolerate installed packages without a Git checkout. Improve public option documentation and CLI guidance.

Co-Authored-By: Codex <noreply@openai.com>

AI-Model: gpt-5
AI-Contributed/Feature: 200/200
AI-Contributed/UT: 25/25
Fetch several physical batches at once while preserving the logical batch boundaries consumed by ACT training. Keep warmup isolation, cursor replay, and snapshot semantics unchanged.

Co-Authored-By: Codex <noreply@anthropic.com>
AI-Model: gpt-5
AI-Contributed/Feature: 102/102
AI-Contributed/UT: 105/105
Use eight logical batches per physical fetch, based on the current Vortex-backed RoboMIND layout. Preserve logical batch boundaries while documenting the per-worker memory tradeoff.

Co-Authored-By: Codex <noreply@anthropic.com>
AI-Model: gpt-5
AI-Contributed/Feature: 6/6
AI-Contributed/UT: 4/4
Reuse the guarded LeRobot dependency set for the ACT extra, clarify batch-fetch measurement terminology, and strengthen the timed plural-fetch assertion.

Co-Authored-By: Codex <noreply@openai.com>

AI-Model: gpt-5

AI-Contributed/Feature: 136/136
AI-Contributed/UT: 54/54
Skip the optional ACT test module before importing unavailable dependencies and keep its synthetic RoboMIND fixture independent of Vortex.

Co-Authored-By: Codex <noreply@openai.com>

AI-Model: gpt-5
AI-Contributed/Feature: 0/0
AI-Contributed/UT: 23/23
Separate experiment preparation, backend execution, and result comparison while preserving a shared ACT workload contract.

Co-Authored-By: Codex <noreply@anthropic.com>
AI-Model: gpt-5.6-sol
Co-Authored-By: Codex <noreply@ai-tool.com>
Co-Authored-By: Codex <noreply@openai.com>
AI-Contributed/Feature: 3667/3667
AI-Contributed/UT: 1330/1330
Avoid reading the full qpos horizon when ACT only consumes the anchor observation. Add a golden backend sample contract and explicit HDF5 index bounds coverage.

Co-Authored-By: Codex <noreply@anthropic.com>

AI-Model: gpt-5.6-sol
Co-Authored-By: Codex <noreply@ai-tool.com>
AI-Contributed/Feature: 13/13
AI-Contributed/UT: 99/99
Document contiguous ordering, anchor-column cardinality, batching, snapshot pinning, and map-style index behavior. Strengthen plural-access and pickle round-trip coverage.

Co-Authored-By: Codex <noreply@anthropic.com>

AI-Model: gpt-5.6-sol
Co-Authored-By: Codex <noreply@ai-tool.com>
AI-Contributed/Feature: 87/87
AI-Contributed/UT: 56/56
Add the required blank line before the nested dataset factory and remove the extra module-level blank line.

Co-Authored-By: Codex <noreply@anthropic.com>

AI-Model: gpt-5.6-sol
Co-Authored-By: Codex <noreply@ai-tool.com>
Co-Authored-By: Codex <noreply@openai.com>
AI-Contributed/Feature: 2/2
AI-Contributed/UT: 0/0
Skip only the ACT runtime ownership checks when Torch or Pillow is absent, while keeping dependency-free comparison tests active.

Co-Authored-By: Codex <noreply@anthropic.com>

AI-Model: gpt-5.6-sol
Co-Authored-By: Codex <noreply@ai-tool.com>
Co-Authored-By: Codex <noreply@openai.com>
AI-Contributed/Feature: 0/0
AI-Contributed/UT: 8/8
Measure four complete physical fetch groups by default so the first Paimon fetch does not dominate steady-state throughput.

Co-Authored-By: Codex <noreply@anthropic.com>
AI-Model: gpt-5.6-sol
AI-Contributed/Feature: 4/4
AI-Contributed/UT: 9/9
Avoid coupling tests to tuned experiment values and fold redundant integration assertions into the paired backend parity test.

Co-Authored-By: Codex <noreply@anthropic.com>
AI-Model: gpt-5.6-sol
AI-Contributed/Feature: 0/0
AI-Contributed/UT: 174/174
Resolve deferred MAP BLOB reads, snapshot option conflicts, and shared mutable window cells. Tighten fetch timing, runtime identity, and Python version reporting.

Co-Authored-By: Codex <noreply@anthropic.com>

AI-Model: gpt-5.6-sol

AI-Contributed/Feature: 100/100

AI-Contributed/UT: 149/149
@YannByron
YannByron force-pushed the m0/pr3-act-benchmark branch from 42be5b2 to 705683a Compare September 2, 2026 09:16
@YannByron YannByron changed the title [python] Add paired HDF5/Paimon ACT benchmark [python] Add ACT storage benchmark and contiguous windows Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants