@@ -118,6 +118,11 @@ Common Options
118118 - ``none ``
119119 - ``none `` or ``full ``. Full mode collects per-batch min/max and null counts
120120 for conservative predicate pruning, including retained spill metadata.
121+ * - ``realtime.index.<index_name>.columns ``
122+ - Unset
123+ - Comma-separated fields indexed in memory by the default Arrow store.
124+ Available in append and append deduplicate modes; independent of
125+ ``realtime.store.stats-mode ``.
121126 * - ``realtime.deduplicate-key-fields ``
122127 - Unset
123128 - Selects append deduplicate mode. Although expressed as a comma-separated
@@ -401,6 +406,37 @@ context's reclaimed progress for relevant active stores. It does not restore
401406an old in-memory state and is not unrestricted historical time travel. For a
402407disk-only historical query, do not attach the real-time context.
403408
409+ In-memory Predicate Indexes
410+ --------------------------
411+
412+ The default Arrow store supports optional predicate indexes on key or value fields
413+ in append and append deduplicate modes. PK modes are unsupported. For example:
414+
415+ .. code-block :: text
416+
417+ realtime.enabled = true
418+ realtime.index.hash.columns = id,payload
419+ realtime.store.stats-mode = full
420+
421+ The built-in ``hash `` plugin is **experimental **; its implementation and supported
422+ capabilities may change. It supports equality and ``IN `` predicates on boolean,
423+ integer, floating-point, string, binary, and date fields. Decimal, timestamp, and
424+ nested types are currently unsupported. Use a comma-separated list to index
425+ multiple fields; indexing ``_REALTIME_OFFSET `` is unsupported.
426+
427+ Note that this index does not currently handle hash collisions.
428+
429+ Indexes and batch statistics work independently and can be enabled together.
430+ For ``id = 7 AND score > 100 ``, an index supporting equality on ``id `` selects
431+ candidate offsets; statistics can then skip batches whose maximum ``score ``
432+ is at most 100.
433+ Statistics cover entire batches, so mixed batches still need row-level
434+ filtering. Keep predicate fields in the read schema and enable
435+ ``ReadContextBuilder::EnablePredicateFilter(true) `` for exact filtering.
436+
437+ In-memory and disk indexes are configured independently. Use
438+ ``file-index.<type>.columns `` to enable indexes for committed files.
439+
404440Implementation by Mode
405441----------------------
406442
@@ -409,9 +445,10 @@ Append
409445
410446``RealtimeAppendOnlyWriter `` validates offsets and writes to an
411447``APPEND_ONLY `` store. The default ``ArrowRealtimeStore `` retains building
412- batches and immutable sealed segments in write order. Query readers project
413- the requested fields, optionally prune stored batches using statistics, and
414- apply the visible offset range before removing the hidden offset column.
448+ batches and immutable sealed segments in write order. Query readers use
449+ optional predicate indexes and batch statistics to prune candidates, project
450+ the requested fields, and apply the visible offset range before removing the
451+ hidden offset column.
415452
416453Preparing reads sealed segments through the ordinary ``AppendOnlyWriter ``,
417454removing the internal offset prefix before writing table files. Queries do not
@@ -474,8 +511,10 @@ are marked deleted, and within-batch duplicates are resolved in row order.
474511Add rows are appended to the store; delete events update only the key/DV state,
475512without persisting a delete row.
476513
477- Key indexes and DV bitmaps use copy-on-write snapshots, so later writes do not
478- change the state held by existing lookups, read views, or seal checkpoints.
514+ Deduplicate key-to-latest-offset lookups and DV bitmaps use copy-on-write
515+ snapshots, so later writes do not change the state held by existing lookups,
516+ read views, or seal checkpoints. The optional predicate indexes share
517+ append-only postings instead.
479518
480519Memory reads use the read view's DV to exclude rows whose offsets are marked
481520deleted.
@@ -518,8 +557,9 @@ still terminal for the owning writer/context.
518557Old views can retain the memory representation after spill or reclamation;
519558new views use the published representation. Spill files are deleted when the
520559store and all remaining handles/readers have released them. They are temporary
521- storage, not a recoverable log after process restart. Deduplicate key indexes
522- and DVs remain in memory; full-mode statistics also remain resident after spill.
560+ storage, not a recoverable log after process restart. Predicate indexes,
561+ deduplicate key indexes, DVs, and full-mode statistics remain in memory after
562+ spill. Predicate index memory is included in store memory accounting.
523563
524564``RealtimeContext::GetMetrics() `` and real-time writer metrics expose:
525565
@@ -587,7 +627,7 @@ Extension Points and Current Limits
587627Custom stores manage buffering, read views, and spill while following the
588628``RealtimeStore `` contracts. Paimon manages table files, indexes, merge semantics,
589629and commits. Append deduplicate also keeps key indexes and DVs in the framework;
590- custom delegates need ``RealtimeStoreTwoPhaseSeal `` to enable its spill.
630+ custom delegates need ``TwoPhaseSealableRealtimeStore `` to enable its spill.
591631
592632Current restrictions include:
593633
@@ -604,8 +644,8 @@ Current restrictions include:
604644* Append deduplicate requires a single non-null scalar key, a persisted offset
605645 column, exact offset file-index reads, and non-bitmap64 DVs. Existing ordinary
606646 append files are not migrated automatically.
607- * Spill covers Arrow data, not key indexes or DVs, and provides neither a hard
608- memory limit nor crash recovery.
647+ * Spill covers Arrow data, not predicate indexes, key indexes, or DVs, and
648+ provides neither a hard memory limit nor crash recovery.
609649
610650Code Map
611651--------
0 commit comments