[python] Persist LeRobot dataset metadata in Paimon by XiaoHongbo-Hope · Pull Request #9529 · apache/paimon

XiaoHongbo-Hope · 2026-09-01T09:44:13Z

Depends on #9589 for reconciled commit callback delivery.

Summary

import one non-empty LeRobot V3 dataset into a Paimon table group: frames, versions, episodes, tasks, and optional subtasks
preserve native LeRobot component schemas: frames keep task_index and optional subtask_index, while text mappings remain in tasks and subtasks
use BIGINT version_id as the common Paimon tag on every present component table
record optional subtask presence in the versions manifest
commit and tag all components, then append one versions row as the publication point
stream Episode Parquet shards through one batch writer so Paimon applies normal target-file-size rolling
validate frame controls, including task/subtask mappings and float32 timestamp quantization for long Episodes

This establishes the persisted LeRobot contract that #9498 can consume after rebasing.

Tests

LeRobot 0.4.4 integration and validation: 50 passed, 33 subtests passed
LeRobot, multimodal, and HDF5 regression: 149 passed, 2 skipped, 63 subtests passed
verified native task, subtask, and Episode schemas plus common numeric tags
verified 8 x 8 MiB Episode shards leave no live Arrow allocation after import
flake8, py_compile, and git diff --check pass

JingsongLi · 2026-09-01T15:39:14Z

I think the four-table split and the READY-last publication point are the right direction. My remaining concern is that the current schema combines two different version models without defining which one is authoritative for row membership.

Today every component row carries metadata_version, while the dataset manifest also records component snapshot IDs and creates tags. Consider two imports:

V1 / frames snapshot S1:
  A(V1), B(V1)

V2 / frames snapshot S2:
  A(V1), B(V1), A(V2), B(V2), C(V2)

Because the frame table is append-only, S2 is cumulative. Reading S2 or its tag alone returns both V1 and V2. The actual V2 view still requires dataset_id = D AND metadata_version = V2. This means metadata_version, rather than the snapshot, currently defines which rows belong to a dataset version; the snapshot/tag is only a retention and commit boundary.

That model is internally consistent only if every version is a complete materialization. Adding C to an existing dataset must write A, B, and C again. Writing only C under V2 would make the V2 filter return only the delta, unless the reader implements parent-version replay and INSERT/UPDATE/DELETE overlay semantics. The current implementation does not implement such a delta model.

I suggest choosing one explicit model before establishing this as the persisted contract:

Full row-version model
- Logical keys include the version:
  - frames: (dataset_id, metadata_version, index)
  - episodes: (dataset_id, metadata_version, episode_index)
  - tasks: (dataset_id, metadata_version, task_index)
- Every import is documented as a full materialization.
- Every reader must push down both dataset_id and metadata_version.
- Snapshot IDs/tags are documented as retention fences, not as sufficient version selectors.
- The API must return metadata_version, and the manifest needs a publication sequence or a separate head pointer because a UUID cannot define the latest version.
- Version deletion/GC is required because expiring an old snapshot does not remove old-version rows that remain live in the latest append snapshot.
Snapshot-state model
- Component rows represent current logical state and do not carry metadata_version.
- Stable logical keys are independent of the release:
  - frames: (dataset_id, stable_frame_id)
  - episodes: (dataset_id, stable_episode_id)
  - tasks: (dataset_id, task_id)
- A dataset version is one manifest row containing the exact frames, episodes, and tasks snapshot IDs.
- Appending a new episode writes only the new rows. The new snapshots logically contain old plus new state while reusing unchanged files.
- Updates and deletes touch only affected logical rows; old tagged snapshots still reproduce the previous version.
- The cross-table UUID remains useful, but only as a version_id or publication_id that resolves the manifest. The component snapshots define data membership.

I prefer the snapshot-state model because it matches Paimon snapshot/time-travel semantics and LeRobot recording behavior, which is primarily append-by-episode. It also avoids O(number of versions * full dataset size) storage for large BLOB-backed datasets.

A possible manifest is:

dataset_versions
  dataset_id
  version_id
  parent_version_id
  status
  frames_snapshot_id
  episodes_snapshot_id
  tasks_snapshot_id
  published_at

The publication flow can then be:

reserve (dataset_id, version_id) as PENDING
commit/update frames
commit/update episodes
commit/update tasks
pin the three snapshots
publish the manifest as READY
optionally move dataset_heads[dataset_id] to version_id

The public import result should contain at least dataset_id, version_id, and the component snapshot IDs. This is also important for empty imports, where the current API returns None even though a new metadata version was published.

The important point is not that UUIDs and snapshots cannot coexist. They can, but they need distinct roles: the UUID should identify the cross-table release, while exactly one mechanism must define which component rows belong to that release.

JingsongLi · 2026-09-02T07:39:55Z

I suggest using version_id directly as the Paimon tag name for all component tables.
Each LeRobot dataset would map to one table group:

frames
tasks
episodes
versions

The component tables keep their native LeRobot V3 schemas and do not need dataset_id or version_id columns:

frames:   <LeRobot V3 frame schema>
tasks:    <LeRobot V3 task schema>
episodes: <LeRobot V3 episode schema>

Each published version creates the same tag on all three tables:

frames@<version_id>
tasks@<version_id>
episodes@<version_id>

The versions table acts as the version manifest:

version_id
info_json
stats_json

Version id is a bigint.

Readers first resolve a READY row from versions, then read all three component tables using scan.tag-name = version_id. Snapshot IDs do not need to be stored in versions, because each table’s tag already resolves the version to its corresponding snapshot.

JingsongLi · 2026-09-02T09:10:18Z

 )
-print(snapshot_id)
+print(result.version_id)
+print(result.frames_snapshot_id)


Just version_id is enough?

JingsongLi · 2026-09-02T09:39:45Z


 ```python
-snapshot_id = conn.load_from_lerobot(
+version_id = conn.load_from_lerobot(


How to use this version in PyToch Dataloader?

How to use this version in PyToch Dataloader?

Like this dataset = PaimonLeRobotDataset( conn, "catalog.db.robot_data", version_id=version_id, )?

I think it is OK, if none you should find latest.

JingsongLi · 2026-09-02T11:38:33Z

+                or not isinstance(timestamp, numbers.Real)
+                or not math.isclose(
+                    float(timestamp), frame_index / fps,
+                    rel_tol=0.0, abs_tol=1e-4)):


LeRobot stores timestamp as float32, so this fixed absolute tolerance can reject valid long episodes as the float32 ULP grows. For example, at 30 FPS, frame_index = 61,441 is stored as 2048.033447265625, while frame_index / fps is 2048.0333333333333; the error is about 1.14e-4, so this check fails for otherwise valid LeRobot data. Please compare against an expected value quantized to the declared Arrow dtype, or use a dtype/ULP-aware tolerance.

JingsongLi · 2026-09-02T13:18:29Z

I suggest two additional schema-alignment changes:

Remove the denormalized task column from the frames table. LeRobot V3 frame data persists task_index, while the task text belongs to tasks.parquet. Repeating the full task string in every frame makes the frame schema non-native, duplicates storage, and creates a second source of truth. The reader should resolve task text from the version-tagged tasks table.
Support optional meta/subtasks.parquet instead of rejecting it. LeRobot 0.4.4 treats subtasks as part of the dataset metadata. When present, it should be stored in a __subtasks component table with its native schema and published under the same version_id tag as frames, tasks, and episodes. The version manifest should also make the optional component presence explicit.

…contained-import # Conflicts: # paimon-python/pypaimon/tests/multimodal_lerobot_test.py

JingsongLi · 2026-09-03T03:04:08Z

+            raise ValueError(
+                "LeRobot metadata table %s must be append-only." % identifier)
+        actual = _target_schema(table)
+        if not actual.equals(schema, check_metadata=False):


Ignoring schema metadata here hides a round-trip incompatibility. LeRobot writes tasks.parquet and subtasks.parquet from Pandas DataFrames, with the task or subtask text encoded as a named Pandas index. Schema.from_pyarrow_schema does not persist the top-level pandas schema metadata, so scanning or exporting this Paimon table and calling to_pandas() produces a RangeIndex instead of restoring that text index; LeRobot lookups such as tasks.iloc[task_idx].name then return the wrong value. Please either persist and reattach the component Arrow schema metadata, or explicitly reconstruct the index from the physical text column in the Paimon LeRobot reader and cover that round trip in a test.

JingsongLi · 2026-09-03T03:04:13Z

-        ))
-    return pa.schema(fields)
+    return pa.schema([
+        _feature_field(name, feature)


This derives the frame schema from arbitrary info.json descriptors but never enforces the mandatory LeRobot V3 control schema. LeRobot V3 defines timestamp as scalar float32 and frame_index, episode_index, index, and task_index as scalar int64; subtask_index should likewise be scalar int64 when present. Today a source declaring timestamp as float64 or frame_index as int32 is accepted and published as a non-native frame schema because the row validator only checks values. Please validate these required feature descriptors against the V3 defaults before creating the table.

JingsongLi · 2026-09-03T03:04:15Z

+def _validated_episode_tables(metadata):
+    rows = []
+    for table in _source_episode_tables(metadata):
+        rows.extend(table.select(_EPISODE_CONTROL_COLUMNS).to_pylist())


Although each Arrow shard is released before the next one is read, this still materializes every Episode control row as Python dictionaries in rows, and _episode_rows then builds a second full Python list retained in metadata. The FileIO path has already materialized another locator list in _RemoteLeRobotDataset, so peak heap remains O(total_episodes) and can reach multiple GB at the million-Episode scale that V3 targets. Please validate ordered Episode rows incrementally and keep the boundaries in Arrow, mmap, or compact arrays, or consume them directly during frame import, instead of retaining duplicate to_pylist() structures.

XiaoHongbo-Hope · 2026-09-03T09:15:21Z

I’m not sure whether we should clean up the table group automatically when an import fails. I prefer leaving it in place and letting the user drop it explicitly. Could you let me know your suggestion? @JingsongLi @YannByron

JingsongLi · 2026-09-03T09:27:30Z

I’m not sure whether we should clean up the table group automatically when an import fails. I prefer leaving it in place and letting the user drop it explicitly. Could you let me know your suggestion? @JingsongLi @YannByron

+1 to keep it.

JingsongLi · 2026-09-03T13:34:42Z

Two suggestions on the publication contract:

Keep __versions as a published-version manifest only. Commit frames, episodes, tasks, and optional subtasks, create the common version_id tag on every component, then append one manifest row as the final publication step. For the current new-target-only importer, the status column and the initial PENDING row are unnecessary. The schema can simply be (version_id, info_json, stats_json, has_subtasks). Failed imports may leave tables and tags for inspection; without a manifest row, the version remains unpublished.
Make the reader and tag contract explicit. Readers must only open versions present in __versions and read every required component through the matching version_id tag. A missing required tag must be an error, never a fallback to the latest snapshot. Published release tags must not be rebound to different snapshots, and their retention/deletion should be managed as a complete dataset release rather than independently per table.

JingsongLi · 2026-09-03T14:09:33Z

+1

XiaoHongbo-Hope force-pushed the codex/lerobot-self-contained-import branch 4 times, most recently from dd6ac5b to 445d4a9 Compare September 1, 2026 12:30

[python] Persist LeRobot dataset metadata in Paimon

286235b

XiaoHongbo-Hope force-pushed the codex/lerobot-self-contained-import branch from 445d4a9 to 286235b Compare September 1, 2026 15:38

XiaoHongbo-Hope added 6 commits September 1, 2026 09:12

[python] Publish LeRobot imports by component snapshots

ebf9140

[python] Make LeRobot imports retry-safe

7a23db9

[python] Preserve LeRobot import result on cleanup

580233a

[python] Clean up failed LeRobot table creation

f127d2a

[python] Handle concurrent multimodal table deletion

81540dc

[python] Preserve initial data across create races

0d9d053

[python] Align LeRobot versions with table tags

aa99328

JingsongLi reviewed Sep 2, 2026

View reviewed changes

[python] Return LeRobot version ID

222aa3e

JingsongLi reviewed Sep 2, 2026

View reviewed changes

[python] Guard initial LeRobot publication

dbc0922

JingsongLi reviewed Sep 2, 2026

View reviewed changes

XiaoHongbo-Hope added 5 commits September 2, 2026 07:07

[python] Align LeRobot task and subtask metadata

a8df2a5

[python] Stabilize LeRobot FileIO tests

b9fdcef

Merge remote-tracking branch 'apache/master' into codex/lerobot-self-…

caa84d9

…contained-import # Conflicts: # paimon-python/pypaimon/tests/multimodal_lerobot_test.py

[python] Stabilize LeRobot metadata publication

82430d4

[python] Stream LeRobot episode metadata

1b8c807

JingsongLi reviewed Sep 3, 2026

View reviewed changes

[python] Make LeRobot cleanup generation safe

5fd9084

XiaoHongbo-Hope added 11 commits September 2, 2026 22:42

[python] Validate native LeRobot control metadata

e98d1f2

[python] Preserve LeRobot task index metadata

7d06ab5

[python] Bound LeRobot import cleanup and episode state

47e5734

[python] Clean only owned LeRobot import tables

7a58c04

[python] Reconcile LeRobot cleanup generations

f90fd76

[python] Reconcile uncertain LeRobot table renames

847c507

[python] Avoid retrying ambiguous LeRobot table drops

05f423a

[python] Avoid table renames during LeRobot cleanup

60c858c

[python] Keep LeRobot root until companions drop

a33837a

[docs] Simplify LeRobot version description

a5f744f

[docs] Avoid promising transactional import cleanup

27f7d39

[python] Notify callbacks for duplicate commits

f99e287

XiaoHongbo-Hope added 4 commits September 3, 2026 02:36

[python] Keep failed LeRobot imports for inspection

e1ebdcb

[python] Clarify LeRobot required feature validation

d0bd8c0

[python] Reconcile LeRobot publication retries

e2806b4

[python] Separate generic fixes from LeRobot import

178e356

JingsongLi mentioned this pull request Sep 3, 2026

[python][torch] Add lazy contiguous window dataset #9580

Open

6 tasks

XiaoHongbo-Hope marked this pull request as ready for review September 3, 2026 12:45

XiaoHongbo-Hope requested a review from JingsongLi September 3, 2026 12:45

[python] Publish LeRobot versions after component tags

0704cf6

JingsongLi merged commit ffe8e51 into apache:master Sep 3, 2026
10 checks passed

Uh oh!

Conversation

XiaoHongbo-Hope commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests

Uh oh!

JingsongLi commented Sep 1, 2026

Uh oh!

JingsongLi commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JingsongLi commented Sep 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

XiaoHongbo-Hope commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JingsongLi commented Sep 3, 2026

Uh oh!

JingsongLi commented Sep 3, 2026

Uh oh!

JingsongLi commented Sep 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

XiaoHongbo-Hope commented Sep 1, 2026 •

edited

Loading

JingsongLi commented Sep 2, 2026 •

edited

Loading

XiaoHongbo-Hope commented Sep 3, 2026 •

edited

Loading