Skip to content

Describe the temporal coverings as TemporalParquet 2.0 struct columns #138

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
estebanzimanyi merged 1 commit into from
Sep 10, 2026
Merged

Describe the temporal coverings as TemporalParquet 2.0 struct columns #138

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 45 additions & 33 deletions docs/temporal-covering.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,34 +2,39 @@

`meta/temporal-covering.json` is the **single codegen source of truth**
(RFC #870 TemporalParquet / #913 Temporal Data Lake) for projecting a MEOS
temporal column into Parquet/Iceberg **covering columns**. The pipeline
folds it into `meos-idl.json` as `temporalCovering`. Every binding/engine
(PyMEOS, JMEOS, MobilityDuck, MobilitySpark, …) generates the **identical**
covering schema from this one mapping, so a temporal table prunes the same
way on every platform — no per-engine covering code to maintain.
temporal column into the **covering columns** of TemporalParquet 2.0.0. The
pipeline folds it into `meos-idl.json` as `temporalCovering`. Every
binding/engine (PyMEOS, JMEOS, MobilityDuck, MobilitySpark, …) generates the
**identical** covering schema from this one mapping, so a temporal table
prunes the same way on every platform — no per-engine covering code to
maintain.

## What it is

A temporal value is stored on disk as a canonical MEOS-WKB `BLOB`. Iceberg
and Parquet cannot prune on a `BLOB`. The covering descriptor names, per
temporal-type **class**, the primitive columns to *materialise alongside*
the value — the bounding box and SRID — which Iceberg collects as manifest
statistics and Parquet as row-group min/max. A bbox/time predicate then
prunes whole files and row groups with **no spatial-aware engine**
(GeoParquet 1.1 `covering.bbox`; MVB v3 measured this as ~10× faster than
the `ST_Intersects` path).
temporal-type **class**, the columns to *materialise alongside* the value —
struct columns at the root of the schema, named after the temporal column,
whose fields Iceberg collects as manifest statistics and Parquet as
row-group min/max. A bbox/time predicate on those fields then prunes whole
files and row groups with **no spatial-aware engine** (MVB v3 measured this
as ~10× faster than the `ST_Intersects` path).

The mapping is keyed by **class**, not by type — adding a type is one entry
in its class:

| Class | Box | Types | Covering columns |
|---|---|---|---|
| `spatial` | `STBOX` via `tspatial_to_stbox` | tgeompoint, tgeogpoint, tgeometry, tgeography, tcbuffer, tnpoint, tpose, trgeometry | `xmin xmax ymin ymax [zmin zmax] tmin tmax srid` |
| `number` | `TBOX` via `tnumber_to_tbox` | tint, tfloat, tbigint | `vmin vmax tmin tmax` |

| `spatial` | `STBOX` via `tspatial_to_stbox` | tgeompoint, tgeogpoint, tgeometry, tgeography, tcbuffer, tnpoint, tpose, trgeometry | `{col}_bbox` {`xmin`, `ymin`, [`zmin`,] `xmax`, `ymax`[, `zmax`]} · `{col}_tspan` {`tmin`, `tmax`} · `srid` |
| `number` | `TBOX` via `tnumber_to_tbox` | tint, tfloat, tbigint | `{col}_vspan` {`vmin`, `vmax`} · `{col}_tspan` {`tmin`, `tmax`} |
| `timeOnly` | — | tbool, ttext | `{col}_tspan` {`tmin`, `tmax`} |

`{col}_bbox` is a GeoParquet bounding box column: its fields are `DOUBLE`, in
the order shown, it has the repetition of its temporal column, and it holds a
value exactly when the temporal column does. `zmin`/`zmax` are emitted only
for 3D values (`when: hasZ`). `srid` is a plain column beside the coverings.
The canonical value column is unchanged and lossless; covering columns are
denormalised derivations of the value's box. `zmin`/`zmax` are emitted only
for 3D values (`when: hasZ`).
denormalised derivations of the value's box.

## In the catalog

Expand All @@ -41,41 +46,48 @@ codegen:
"valueCodec": { "asHexWkb": "temporal_as_hexwkb",
"fromHexWkb": "temporal_from_hexwkb" },
"byType": { "tgeompoint": { "class": "spatial", "box": {...},
"srid": "tspatial_srid", "columns": [...] }, ... },
"srid": "tspatial_srid",
"coverings": [...], "columns": [...] }, ... },
"symbols": ["stbox_xmin", "tbox_xmin", "tspatial_to_stbox", ...],
"count": 11
"count": 13
}
```

- `byType` — `"tgeompoint"` → its class, box converter, SRID accessor, and
covering columns (each with its MEOS bbox accessor and SQL type). A
generator reads this directly; it never re-derives the mapping.
- `byType` — `"tgeompoint"` → its class, box converter, SRID accessor, its
coverings (each with its key, its column name and its fields, every field
with its MEOS accessor and SQL type) and its plain columns. A generator
reads this directly; it never re-derives the mapping.
- `symbols` — every MEOS C symbol the descriptor depends on. The covering
parity audit (`tools/covering_parity.py`) checks each is exported by the
catalog and each covered type is a real `MeosType` — a miss is reported as
a worklist (add/export the accessor in MEOS), never a fabricated pass.

The parser rejects a `bbox` covering whose fields are not a GeoParquet
bounding box column's, in its order, and a class that declares a covering
twice.

## How a generator uses it

For a column `traj TGEOMPOINT`, emit alongside the WKB value column:
`generate_covering.py` projects the catalog onto the language-agnostic
contract: per type, each covering with its column name and its fields in
order, every field with the MEOS expression that derives it from `VALUE`.
For a column `traj TGEOMPOINT`, a generator substitutes `VALUE` with the
column reference and `{col}` with its name, and emits alongside the WKB
value column:

```sql
xmin = stbox_xmin(tspatial_to_stbox(traj)), xmax = stbox_xmax(...),
ymin = stbox_ymin(...), ymax = stbox_ymax(...),
tmin = stbox_tmin(...), tmax = stbox_tmax(...),
srid = tspatial_srid(traj)
traj_bbox = {xmin: stbox_xmin(tspatial_to_stbox(traj)),
ymin: stbox_ymin(...), xmax: stbox_xmax(...), ymax: stbox_ymax(...)},
traj_tspan = {tmin: stbox_tmin(tspatial_to_stbox(traj)), tmax: stbox_tmax(...)},
srid = tspatial_srid(traj)
```

(each engine in its own idiom — DuckDB generated columns, a Spark UDF
projection, a PyMEOS writer), plus the `temporal` and GeoParquet `geo` /
`covering.bbox` file metadata keys from `metadataKeys`.
(each engine in its own idiom — a DuckDB struct column, a Spark UDF
projection, a PyMEOS writer), and declares the coverings in the `temporal`
file metadata key under `covering`, with `bbox` in GeoParquet's form.

## Not yet covered

- **Time-only** (`tbool`, `ttext`): a `tmin`/`tmax` covering needs a span
lower/upper bound accessor; `temporal_to_tstzspan` is exported but a span
bound accessor is not. Surfaced as a MEOS export gap (close in MEOS C),
not filled binding-side.
- **Point-cloud / cell-index** (`tpcpoint`, `tpcpatch`, `th3index`,
`tquadbin`): fold into the `spatial` class once the catalog confirms a
uniform temporal→`STBOX` converter for these families.
55 changes: 34 additions & 21 deletions generator/covering.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,32 +2,46 @@

Projects the ``temporalCovering`` block of the MEOS catalog
(``meos-idl.json``, produced by ``parser/covering.py``) onto the canonical,
language-agnostic covering-column contract: per temporal type, the ordered
covering columns with the fully-composed MEOS expression that derives each
from the value.
language-agnostic covering-column contract of TemporalParquet 2.0.0: per
temporal type, each covering struct column with its name, its fields in
order, and the fully-composed MEOS expression that derives each field from
the value, plus the plain columns beside them.

Every binding generator (PyMEOS, JMEOS, MobilityDuck, MobilitySpark, …)
renders this same contract in its own idiom — a DuckDB ``GENERATED`` column,
a Spark UDF projection, a PyMEOS writer — so a temporal table prunes the
renders this same contract in its own idiom — a DuckDB struct column, a
Spark UDF projection, a PyMEOS writer — so a temporal table prunes the
same way on every platform (Iceberg manifest + Parquet row-group min/max).
The ``VALUE`` placeholder is the temporal column reference the binding
substitutes.
Two placeholders are the binding's to substitute: ``VALUE`` is the temporal
column reference, and ``{col}`` in a covering's column name is the temporal
column's name.

Pure ``dict`` → ``dict``; no libclang and no MEOS runtime.
Pure ``dict`` → ``dict``: it reads the catalog only and needs no MEOS runtime.
"""

from __future__ import annotations


def _column_expr(column: dict, box_from: str) -> str:
"""Compose the MEOS expression that derives one covering column from the
temporal value (``VALUE``). A ``box`` column is read off the value's box;
a ``value`` column is read off the value directly."""
"""Compose the MEOS expression that derives one field from the temporal
value (``VALUE``). A ``box`` field is read off the value's box; a
``value`` field is read off the value directly."""
if column["source"] == "value":
return f"{column['accessor']}(VALUE)"
return f"{column['accessor']}({box_from}(VALUE))"


def _field(field: dict, box_from: str) -> dict:
"""Project one covering field or plain column."""
entry = {
"name": field["name"],
"sqlType": field["sqlType"],
"expr": _column_expr(field, box_from),
}
if field.get("when"):
entry["when"] = field["when"]
return entry


def build_covering_projection(catalog: dict) -> dict:
"""Project ``temporalCovering`` onto the canonical covering-column contract."""
cov = catalog.get("temporalCovering")
Expand All @@ -38,20 +52,19 @@ def build_covering_projection(catalog: dict) -> dict:
for tname, spec in cov["byType"].items():
box = spec.get("box")
box_from = box["from"] if box else None
columns = []
for col in spec["columns"]:
entry = {
"name": col["name"],
"sqlType": col["sqlType"],
"expr": _column_expr(col, box_from),
coverings = [
{
"key": covering["key"],
"column": covering["column"],
"fields": [_field(f, box_from) for f in covering["fields"]],
}
if col.get("when"):
entry["when"] = col["when"]
columns.append(entry)
for covering in spec["coverings"]
]
types[tname] = {
"class": spec["class"],
"boxType": box["type"] if box else None,
"columns": columns,
"coverings": coverings,
"columns": [_field(c, box_from) for c in spec.get("columns", [])],
}

return {
Expand Down
74 changes: 45 additions & 29 deletions meta/temporal-covering.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,61 +1,75 @@
{
"_comment": "Temporal-covering descriptor — the single codegen source of truth for projecting a MEOS temporal column into Parquet/Iceberg covering columns (GeoParquet 1.1 `covering.bbox`). Every binding/engine generates the IDENTICAL covering schema from this mapping, so a temporal table prunes the same way on every platform (Iceberg manifest pruning + Parquet row-group min/max) with no spatial-aware engine. Curated canonical data keyed by temporal-type FAMILY (a `class`), not per type — adding a type is one entry in its class. The canonical MEOS-WKB value column is unchanged and lossless; the covering columns are denormalised derivations of the value's bounding box. RFC #870 (TemporalParquet) / #913 (Temporal Data Lake).",
"_comment": "Temporal-covering descriptor — the single codegen source of truth for projecting a MEOS temporal column into the covering columns of TemporalParquet 2.0.0. Every binding/engine generates the IDENTICAL covering schema from this mapping, so a temporal table prunes the same way on every platform (Iceberg manifest pruning + Parquet row-group min/max) with no spatial-aware engine. Each covering is a struct column at the root of the schema, named after its temporal column: `bbox` is a GeoParquet 2.0 bounding box column, and `tspan` and `vspan` are built the same way for the time and value bounds. Curated canonical data keyed by temporal-type FAMILY (a `class`), not per type — adding a type is one entry in its class. The canonical MEOS-WKB value column is unchanged and lossless; the covering columns are denormalised derivations of the value's bounding box.",
"provenance": {
"rfc": "MobilityDB RFC #870 (TemporalParquet) + #913 (Temporal Data Lake)",
"discussion": "MobilityDB#861 (edge-to-cloud SQL portability: one query, three platforms)",
"geoParquet": "GeoParquet 1.1 covering.bbox (geoparquet.org/releases/v1.1.0)",
"temporalParquet": "TemporalParquet 2.0.0 covering columns (github.com/MobilityDB/MobilityLakehouse/blob/main/spec/covering-columns.md)",
"geoParquet": "GeoParquet 2.0 bounding box column, declared under covering.bbox (github.com/opengeospatial/geoparquet/blob/main/format-specs/geoparquet.md)",
"benchmark": "MVB v3 — the scalar AND-chain on materialised covering columns prunes row groups identically to the spatial-aware path and ~10x faster, with no DuckDB spatial extension"
},
"version": "1.0.0",
"version": "2.0.0",
"valueCodec": {
"asHexWkb": "temporal_as_hexwkb",
"fromHexWkb": "temporal_from_hexwkb",
"note": "The canonical MEOS-WKB stays the lossless value column (BLOB); covering columns are denormalised and never the source of truth."
},
"metadataKeys": {
"temporal": "temporal",
"geo": "geo",
"covering": "bbox"
"geo": "geo"
},
"classes": {
"spatial": {
"doc": "Spatial temporal types — STBOX covering (x/y[/z] extent + time extent + SRID).",
"doc": "Spatial temporal types — STBOX covering: a GeoParquet bounding box column, the time extent, and the SRID.",
"box": {"type": "STBOX", "from": "tspatial_to_stbox"},
"srid": "tspatial_srid",
"types": ["tgeompoint", "tgeogpoint", "tgeometry", "tgeography", "tcbuffer", "tnpoint", "tpose", "trgeometry"],
"coverings": [
{"key": "bbox", "column": "{col}_bbox", "fields": [
{"name": "xmin", "sqlType": "double", "accessor": "stbox_xmin", "source": "box"},
{"name": "ymin", "sqlType": "double", "accessor": "stbox_ymin", "source": "box"},
{"name": "zmin", "sqlType": "double", "accessor": "stbox_zmin", "source": "box", "when": "hasZ"},
{"name": "xmax", "sqlType": "double", "accessor": "stbox_xmax", "source": "box"},
{"name": "ymax", "sqlType": "double", "accessor": "stbox_ymax", "source": "box"},
{"name": "zmax", "sqlType": "double", "accessor": "stbox_zmax", "source": "box", "when": "hasZ"}
]},
{"key": "tspan", "column": "{col}_tspan", "fields": [
{"name": "tmin", "sqlType": "timestamptz", "accessor": "stbox_tmin", "source": "box"},
{"name": "tmax", "sqlType": "timestamptz", "accessor": "stbox_tmax", "source": "box"}
]}
],
"columns": [
{"name": "xmin", "sqlType": "double", "accessor": "stbox_xmin", "source": "box"},
{"name": "xmax", "sqlType": "double", "accessor": "stbox_xmax", "source": "box"},
{"name": "ymin", "sqlType": "double", "accessor": "stbox_ymin", "source": "box"},
{"name": "ymax", "sqlType": "double", "accessor": "stbox_ymax", "source": "box"},
{"name": "zmin", "sqlType": "double", "accessor": "stbox_zmin", "source": "box", "when": "hasZ"},
{"name": "zmax", "sqlType": "double", "accessor": "stbox_zmax", "source": "box", "when": "hasZ"},
{"name": "tmin", "sqlType": "timestamptz", "accessor": "stbox_tmin", "source": "box"},
{"name": "tmax", "sqlType": "timestamptz", "accessor": "stbox_tmax", "source": "box"},
{"name": "srid", "sqlType": "int", "accessor": "tspatial_srid", "source": "value"}
{"name": "srid", "sqlType": "int", "accessor": "tspatial_srid", "source": "value"}
]
},
"number": {
"doc": "Numeric temporal types — TBOX covering (value range + time extent).",
"doc": "Numeric temporal types — TBOX covering: the value range and the time extent.",
"box": {"type": "TBOX", "from": "tnumber_to_tbox"},
"srid": null,
"types": ["tint", "tfloat", "tbigint"],
"columns": [
{"name": "vmin", "sqlType": "double", "accessor": "tbox_xmin", "source": "box"},
{"name": "vmax", "sqlType": "double", "accessor": "tbox_xmax", "source": "box"},
{"name": "tmin", "sqlType": "timestamptz", "accessor": "tbox_tmin", "source": "box"},
{"name": "tmax", "sqlType": "timestamptz", "accessor": "tbox_tmax", "source": "box"}
]
"coverings": [
{"key": "vspan", "column": "{col}_vspan", "fields": [
{"name": "vmin", "sqlType": "double", "accessor": "tbox_xmin", "source": "box"},
{"name": "vmax", "sqlType": "double", "accessor": "tbox_xmax", "source": "box"}
]},
{"key": "tspan", "column": "{col}_tspan", "fields": [
{"name": "tmin", "sqlType": "timestamptz", "accessor": "tbox_tmin", "source": "box"},
{"name": "tmax", "sqlType": "timestamptz", "accessor": "tbox_tmax", "source": "box"}
]}
],
"columns": []
},
"timeOnly": {
"doc": "Time-only temporal types — no spatial box; time extent only.",
"doc": "Time-only temporal types — no box; the time extent only.",
"box": null,
"srid": null,
"types": ["tbool", "ttext"],
"columns": [
{"name": "tmin", "sqlType": "timestamptz", "accessor": "temporal_start_timestamptz", "source": "value"},
{"name": "tmax", "sqlType": "timestamptz", "accessor": "temporal_end_timestamptz", "source": "value"}
]
"coverings": [
{"key": "tspan", "column": "{col}_tspan", "fields": [
{"name": "tmin", "sqlType": "timestamptz", "accessor": "temporal_start_timestamptz", "source": "value"},
{"name": "tmax", "sqlType": "timestamptz", "accessor": "temporal_end_timestamptz", "source": "value"}
]}
],
"columns": []
}
},
"deferred": {
Expand All @@ -66,8 +80,10 @@
},
"notes": [
"The covering columns are a denormalisation of the value's bounding box; the canonical MEOS-WKB BLOB remains the lossless source of truth.",
"Materialising the covering columns as primitive Parquet columns gives Iceberg manifest-level file pruning and Parquet row-group min/max pruning, with no spatial-aware engine.",
"zmin/zmax are emitted only for 3D values (`when: hasZ`); 2D values omit them or store null.",
"Each covering is a struct column at the root of the schema; `{col}` in its `column` is the name of the temporal column it covers. The statistics of its fields give Iceberg manifest-level file pruning and Parquet row-group min/max pruning, with no spatial-aware engine.",
"The `bbox` covering is a GeoParquet bounding box column: its fields are DOUBLE, in the order xmin, ymin, [zmin,] xmax, ymax[, zmax], it has the repetition of its temporal column, and it holds a value exactly when the temporal column does.",
"zmin/zmax are emitted only for 3D values (`when: hasZ`); the bbox of a 2D value has four fields.",
"`columns` are plain columns at the root beside the coverings; the spatial class carries `srid` there.",
"`source: box` accessors take the box returned by `class.box.from(value)`; `source: value` accessors take the temporal value directly.",
"This descriptor is type-agnostic per class exactly as `portable-aliases.json` is type-agnostic per operator family — codegen consumes it identically across every binding."
]
Expand Down
Loading
Loading