Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions docs/docs/pypaimon/lerobot.md
Original file line number Diff line number Diff line change
Expand Up @@ -402,6 +402,46 @@ By default, imports copy the original videos. Set
re-encoding (requires PyAV 14.2+), retaining decoding dependencies. Each clip is limited
to 64 MiB and 65,536 packets; unsupported timestamps fail the import.

## Export a LeRobot Dataset

Export one source Dataset from a three-table group to a local directory or an
`oss://bucket/prefix` LeRobot dataset.
If the group contains multiple Datasets, select one by its `_datasets.id` UUID.
Pass the same `tag_name` used for training to export a fixed snapshot.
The destination directory or OSS prefix must not exist.
Pass the table-group prefix, or the actual Frames table name for a group
created by `PaimonLeRobotWriter`.

```python
conn.export_to_lerobot("robot_data", "/data/export-v3", version="v3.0")
conn.export_to_lerobot(
"robot_merged", "/data/export-v21", version="v2.1",
dataset_id="019a91e1-f035-79b9-b04b-44734dc2eb74",
)
# Use version="v2.0" for a LeRobot v2.0 consumer.
conn.export_to_lerobot("robot_data", "oss://my-bucket/lerobot/export-v3")
```

OSS output uses the table's FileIO when it can access the target bucket. Pass
`output_options` with `fs.oss.endpoint`, `fs.oss.accessKeyId`,
`fs.oss.accessKeySecret`, and optional `fs.oss.securityToken` when the output
bucket has separate credentials. The exporter stages files in the worker's
temporary directory, uploads them to OSS, and writes `meta/info.json` last.
An incomplete upload leaves its prefix for inspection; choose a new prefix or
remove it before retrying.

V3.0 keeps source-local indices and copies each encoded MP4 once, including
videos shared by multiple Episodes. V2.0 and v2.1 write one Parquet and one MP4 per
Episode and camera; RGB video export requires PyAV and re-encodes selected frames.
V2 export rejects depth video and video with audio. V2.0 requires source-level
statistics; v2.1 additionally writes per-Episode statistics.
Both exports write source Dataset metadata and Episode statistics. The output
can be opened with the matching LeRobot version.
`meta/paimon_source.json` preserves the Paimon Dataset ID, optional name, and
extra dataset text fields, which have no standard LeRobot metadata fields.
Nullable `subtask_index` values are preserved; older LeRobot readers that
reject null subtasks will also reject those source frames after export.

## Capture LeRobot frames directly into Paimon

`PaimonLeRobotWriter` implements the write-side surface used by LeRobot's
Expand Down
21 changes: 21 additions & 0 deletions paimon-python/pypaimon/multimodal/connection.py
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,27 @@ def append_to_lerobot(
append=True,
)

def export_to_lerobot(
self,
table_name: str,
output_dir,
*,
version: str = "v3.0",
dataset_id: Optional[str] = None,
tag_name: Optional[str] = None,
batch_size: int = 1024,
output_options: Optional[Dict[str, str]] = None):
"""Export one source Dataset to a local directory or OSS prefix.

``version`` is ``v3.0``, ``v2.1``, or ``v2.0``. Multi-source groups require a
``dataset_id`` from ``<table_name>_datasets.id``.
"""
from pypaimon.multimodal.lerobot.export import export_to_lerobot
return export_to_lerobot(
self, table_name, output_dir, version=version,
dataset_id=dataset_id, tag_name=tag_name, batch_size=batch_size,
output_options=output_options)

def recover_lerobot_append(self, table_name: str):
"""Recover a three-table group after an interrupted append."""
from pypaimon.multimodal.lerobot.aggregate import _recover_incomplete_append
Expand Down
2 changes: 2 additions & 0 deletions paimon-python/pypaimon/multimodal/lerobot/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
"""LeRobot Dataset v3 integration for multimodal Paimon tables."""

from pypaimon.multimodal.lerobot.api import load_from_lerobot
from pypaimon.multimodal.lerobot.export import export_to_lerobot
from pypaimon.multimodal.lerobot.dataset import (
PaimonDatasetReader,
PaimonLeRobotDataset,
Expand All @@ -29,4 +30,5 @@
"PaimonLeRobotDataset",
"PaimonLeRobotWriter",
"load_from_lerobot",
"export_to_lerobot",
]
Loading
Loading