Skip to content

[python] Add temporal alignment for multimodal scans - #9536

Draft
XiaoHongbo-Hope wants to merge 3 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/temporal-align
Draft

[python] Add temporal alignment for multimodal scans#9536
XiaoHongbo-Hope wants to merge 3 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/temporal-align

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Purpose

Multimodal training data is often stored as independently sampled streams. A
caller currently has to collect those streams and implement timestamp matching,
episode isolation, snapshot pinning, and payload lookup itself.

This draft proposes a typed, bounded temporal-alignment API for
MultimodalTable.scan():

steps = align(
    actions.scan().select(["episode_id", "event_time", "action"]),
    on="event_time",
    by="episode_id",
    sources={
        "camera": nearest(images.scan().select("image"), tolerance=20),
        "state": backward(states.scan().select("state"), tolerance=50),
    },
)

for batch in steps.to_arrow_batch_reader(batch_size=128):
    train(batch)

Source names are caller-defined output namespaces, not schema or domain names.
Using an explicit mapping supports dynamically generated source lists without
coupling this API to a UI specification.

The implementation:

  • requires an explicit group boundary so matches never cross episodes/clips;
  • supports exact, backward, forward, and nearest matching with inclusive
    tolerance and deterministic earlier-frame tie breaking;
  • pins every input scan to its current snapshot;
  • plans only group keys, timestamps, and _ROW_ID, then fetches selected
    payload rows in batches;
  • keeps BLOB values as descriptors so existing image/video readers can fetch or
    decode only selected samples;
  • emits validity, matched timestamp, and signed delta columns for auditing.

This is deliberately a bounded MVP. It does not add interpolation, training
windows, watermark/streaming semantics, a customer-specific JSON spec, or a
generic ScanQuery batch-reader API. The metadata index is currently
coordinator-local. Since this is a draft, feedback on the API shape and whether
the alignment executor belongs in PyPaimon is especially welcome.

Tests

python -m pytest \
  pypaimon/tests/multimodal_temporal_test.py \
  pypaimon/tests/multimodal_table_test.py -q

81 passed
python -m flake8 \
  pypaimon/multimodal/temporal.py \
  pypaimon/tests/multimodal_temporal_test.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant