Skip to content

Commit 63a3f85

Browse files
SteNicholasclaude
andcommitted
feat(global-index): support row filters in full-text search
Add FullTextSearchBuilder::WithFilter() so that a full-text search ranks only the rows matching a predicate. - FullTextSearchBuilder splits the filter into partition and data predicates and passes the data filter to the scan and the read. - DataEvolutionFullTextScan attaches to each split the intersecting files of the single-field scalar indexes on the filter fields, and IndexFullTextSearchSplit serializes them in version 2. - DataEvolutionFullTextRead resolves the data filter into matched rows with those files before ranking, and searches each split among them only. GlobalIndexScanImpl and GlobalIndexEvaluatorImpl report the fields that decided the result, so that an exact answer can be told from a superset. - The new FilteredRowIdReader verifies the rows of a superset against the data when global-index.filter.refine-from-data is set; otherwise they are excluded with a warning. - StaticFromTagStartingScanner reads tags from the branch of the scan instead of the main branch. - Add the options scalar-index.search-mode, of which only fast is supported so far, and global-index.filter.refine-from-data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent df6f7c0 commit 63a3f85

30 files changed

Lines changed: 2258 additions & 208 deletions

‎docs/source/user_guide/global_index.rst‎

Lines changed: 49 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -303,8 +303,7 @@ an index shard covers are searched, as in the default ``fast`` ``full-text-index
303303
Java Paimon, so rows written since the index was last built are not found until it is built for
304304
them. A search whose ``full-text-index.search-mode``, or else ``global-index.search-mode``, is
305305
``full`` or ``detail``, which also search the rows that no index covers, fails with
306-
``NotImplemented``. Row filters, which Java Paimon evaluates with scalar indexes before ranking, are
307-
not supported yet.
306+
``NotImplemented``.
308307

309308
.. code-block:: cpp
310309
@@ -338,6 +337,8 @@ candidates only: which of the rows tied at the ``limit``-th score of a shard bec
338337
to its index.
339338

340339
- ``SetPartitionFilter()`` restricts the search to some partitions, as in ``ScanContextBuilder``.
340+
- ``WithFilter()`` restricts the search to the rows matching a predicate; see
341+
`Row Filters`_ below.
341342
- The search reads the snapshot that the time-travel options select, as Java Paimon does: snapshot
342343
``scan.snapshot-id``, the snapshot of tag ``scan.tag-name``, the latest snapshot at or before
343344
``scan.timestamp`` or ``scan.timestamp-millis``, or else the latest snapshot. At most one of these
@@ -359,6 +360,45 @@ to its index.
359360
- A column with a primary-key full-text index (``pk-full-text.index.columns``) cannot be searched
360361
yet.
361362

363+
Row Filters
364+
~~~~~~~~~~~
365+
366+
``WithFilter()`` adds a row filter, as Java Paimon's ``withFilter`` does. Only the matching rows are
367+
ranked, so the result is the top ``limit`` rows among them, not a filtered subset of the unfiltered
368+
top ``limit``. The matching rows are handed to the existing index as include row ids, so BM25
369+
statistics stay those of the whole index shard. Repeated calls are ANDed. Conjuncts over partition
370+
fields alone restrict the partitions instead, ANDed with ``SetPartitionFilter()``. As in Java
371+
Paimon, any other conjunct still filters rows but also restricts the partitions to those it
372+
implies: ``(p = 1 AND a > 0) OR (p = 2 AND a > 0)`` searches only partitions ``p = 1`` and
373+
``p = 2``.
374+
375+
.. code-block:: cpp
376+
377+
auto filter = paimon::PredicateBuilder::Equal(/*field_index=*/1, "category",
378+
paimon::FieldType::INT, paimon::Literal(7));
379+
search_builder.WithFilter(filter);
380+
381+
The scan keeps the scalar global index files that index a field of the filter, and the read
382+
resolves the filter with them before ranking:
383+
384+
- If the indexes decide the whole filter, the rows they return are searched.
385+
- If they can only narrow the filter down to candidates, because a conjunct has no index or the
386+
filter has a ``Contains``, ``EndsWith`` or ``Like`` predicate, the candidates are excluded and a
387+
warning is logged. The search then never returns a non-matching row, but may return fewer rows
388+
than ``limit``. With ``global-index.filter.refine-from-data=true``, the read instead verifies the
389+
candidates by reading their filter columns and ``_ROW_ID`` in the planned snapshot, on the caller
390+
and possibly for every candidate.
391+
- Rows that no scalar index covers are excluded. If no index can evaluate the filter at all, no row
392+
is searched and a warning is logged. This is the default ``fast`` ``scalar-index.search-mode``,
393+
which falls back on ``global-index.search-mode``. A row filter in ``full`` or ``detail`` mode,
394+
which decide such rows by reading their filter columns, fails with ``NotImplemented``.
395+
396+
Only single-field indexes that do not support full-text search evaluate a filter. A filter with a
397+
``PredicateBuilder::FullTextMatch`` predicate fails with ``NotImplemented``.
398+
399+
Pinning the Data Read
400+
~~~~~~~~~~~~~~~~~~~~~
401+
362402
The row ids of the result refer to the snapshot the search was planned from. If the table may
363403
change before the rows are read, plan with ``NewFullTextScan()`` and pin the data scan to the
364404
snapshot of the plan, setting ``scan.mode`` too so that a ``scan.mode`` of the table cannot override
@@ -386,12 +426,13 @@ outlive the snapshot.
386426
``ExecuteLocal()`` plans and searches in one process. To plan in one process and search in others,
387427
plan with ``NewFullTextScan()``, ship the splits with ``FullTextSearchSplit::Serialize()`` and
388428
``FullTextSearchSplit::Deserialize()``, and search them with the ``NewFullTextRead()`` of a builder
389-
for the same table, column and query. The serialized form is versioned and specific to Paimon C++:
390-
Java Paimon serializes its splits with Java serialization, so splits cannot be exchanged with it.
391-
Each split records the snapshot it was planned from, and the read ranks the rows live in that
392-
snapshot. The read rejects a split whose index files do not index the column of the latest schema,
393-
such as after the column was replaced. A read returns the top rows of the splits it is given, so
394-
keep the ``limit`` highest-scoring rows of all reads if the splits are searched by several of them.
429+
for the same table, column, query and filter. The serialized form is versioned and specific to
430+
Paimon C++: Java Paimon serializes its splits with Java serialization, so splits cannot be exchanged
431+
with it. Each split records the snapshot it was planned from, and the read ranks the rows live in
432+
that snapshot. The read rejects a split whose index files do not index the column of the latest
433+
schema, such as after the column was replaced. A read returns the top rows of the splits it is
434+
given, so keep the ``limit`` highest-scoring rows of all reads if the splits are searched by several
435+
of them.
395436

396437
.. code-block:: cpp
397438

‎include/paimon/defs.h‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -623,6 +623,19 @@ struct PAIMON_EXPORT Options {
623623
/// index covers. Default value is "global-index.search-mode", or else "fast". Paimon C++
624624
/// supports only "fast" so far, and a full-text search in another mode fails.
625625
static const char FULL_TEXT_INDEX_SEARCH_MODE[];
626+
/// "scalar-index.search-mode" - Search mode for scalar index queries: "fast" evaluates only the
627+
/// rows that an index covers, while "full" and "detail" also evaluate the rows that no index
628+
/// covers. Default value is "global-index.search-mode", or else "fast". Paimon C++ supports
629+
/// only "fast" so far, and a full-text search with a row filter in another mode fails.
630+
static const char SCALAR_INDEX_SEARCH_MODE[];
631+
/// "global-index.filter.refine-from-data" - Whether a vector, hybrid or full-text search may
632+
/// read the filter columns of candidate rows to verify a row filter that the scalar global
633+
/// index can only answer with a superset, such as contains, ends-with or like on a BTree index
634+
/// or a conjunction with a member no index can evaluate. When false, such candidates are
635+
/// excluded from the search, which never returns a non-matching row but may return fewer than
636+
/// the requested top-k. When true, the read runs on the caller and may cover every candidate
637+
/// row. Default value is "false". Paimon C++ reads it only in full-text search so far.
638+
static const char GLOBAL_INDEX_FILTER_REFINE_FROM_DATA[];
626639
/// "pk-btree.index.columns" - Comma-separated columns indexed by primary-key BTree indexes.
627640
/// No default value.
628641
static const char PK_BTREE_INDEX_COLUMNS[];

‎include/paimon/table/source/full_text_read.h‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,10 +51,18 @@ class PAIMON_EXPORT FullTextRead {
5151
/// ranks only its search row ranges, and with deletion vectors enabled only the rows that are
5252
/// live in the snapshot the splits were planned from; BM25 statistics still come from its whole
5353
/// index shard.
54-
/// @param splits Splits planned by a `FullTextScan` of the same column, all from one snapshot.
54+
///
55+
/// With a row filter (`FullTextSearchBuilder::WithFilter()`), a split ranks only the rows that
56+
/// satisfy it, as resolved before ranking by the scalar indexes that the splits carry. Rows
57+
/// that no scalar index can evaluate are not ranked. Candidates that the indexes cannot decide
58+
/// exactly, as for `Contains`, `EndsWith` and `Like`, are excluded too, so the result may hold
59+
/// fewer than `limit` rows, unless `global-index.filter.refine-from-data` verifies them against
60+
/// the data of the planned snapshot on this thread.
61+
/// @param splits Splits planned by a `FullTextScan` of the same column and filter, all from one
62+
/// snapshot.
5563
/// @return The rows found, or an error if a split is invalid, its index files do not index the
5664
/// column of the latest schema (for example, after the column was replaced), or an
57-
/// index cannot be read.
65+
/// index or data file cannot be read.
5866
virtual Result<std::shared_ptr<ScoredGlobalIndexResult>> Read(
5967
const std::vector<std::shared_ptr<FullTextSearchSplit>>& splits) = 0;
6068
};

‎include/paimon/table/source/full_text_scan.h‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,9 @@ class PAIMON_EXPORT FullTextScan {
7676
/// without the backend of a `full-text` or `lucene-fts` index it finds. When the row ranges of
7777
/// index shards overlap, each row is ranked by one shard only: an index whose indexed field is
7878
/// the column comes before one that carries it as an extra field, and further ties are broken
79-
/// by index type and fields, then by the start and end of the shard.
79+
/// by index type and fields, then by the start and end of the shard. With a row filter, each
80+
/// split also carries the files of single-field scalar indexes that intersect its shard and
81+
/// index a field of the filter.
8082
/// @return The plan, or an error if more than one time-travel option is set, the option does
8183
/// not agree with `scan.mode`, the selected snapshot or tag does not exist, or reading
8284
/// the index manifests fails; `NotImplemented` for an index whose backend is not built.

‎include/paimon/table/source/full_text_search_builder.h‎

Lines changed: 19 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525
#include <vector>
2626

2727
#include "paimon/global_index/global_index_result.h"
28+
#include "paimon/predicate/predicate.h"
2829
#include "paimon/result.h"
2930
#include "paimon/table/source/full_text_read.h"
3031
#include "paimon/table/source/full_text_scan.h"
@@ -49,7 +50,8 @@ class Executor;
4950
///
5051
/// `ExecuteLocal()` plans and searches in this process. To run them in different processes, plan
5152
/// with `NewFullTextScan()`, serialize the splits with `FullTextSearchSplit::Serialize()`, and
52-
/// search them with the `NewFullTextRead()` of a builder for the same table, column and query.
53+
/// search them with the `NewFullTextRead()` of a builder for the same table, column, query and
54+
/// filter.
5355
///
5456
/// The row ids of the result refer to the snapshot the search was planned from, which
5557
/// `FullTextScan::Plan::SnapshotId()` returns. If the table may change in between, pin the data
@@ -68,6 +70,17 @@ class PAIMON_EXPORT FullTextSearchBuilder {
6870
FullTextSearchBuilder& SetPartitionFilter(
6971
const std::vector<std::map<std::string, std::string>>& partition_filters);
7072

73+
/// Restricts the search to the rows matching `predicate`, as Java
74+
/// `FullTextSearchBuilder#withFilter` does: only those rows are ranked, so the result is the
75+
/// top `limit` rows among them, and BM25 statistics stay those of the whole index shard. The
76+
/// fields of the predicate are matched by name. Repeated calls are ANDed. Conjuncts over
77+
/// partition fields alone restrict the partitions, ANDed with `SetPartitionFilter()`, and the
78+
/// rest is the row filter that `FullTextRead::Read()` resolves. A conjunct of the row filter
79+
/// also restricts the partitions to those it implies, as `(p = 1 AND a > 0) OR (p = 2 AND
80+
/// a > 0)` does to `p = 1 OR p = 2`.
81+
/// @param predicate The filter; a null predicate adds no filter.
82+
FullTextSearchBuilder& WithFilter(const std::shared_ptr<Predicate>& predicate);
83+
7184
/// Sets the number of highest-scoring rows to return. It must be positive to create a read.
7285
FullTextSearchBuilder& SetLimit(int32_t limit);
7386

@@ -100,9 +113,11 @@ class PAIMON_EXPORT FullTextSearchBuilder {
100113
/// Creates a scan that plans the search on the current settings. The query must be set, and
101114
/// its column must exist in the latest schema of the table.
102115
/// @return The scan, or an error if the settings are invalid or the table cannot be loaded;
103-
/// `NotImplemented` for a primary-key full-text index (`pk-full-text.index.columns`) or
104-
/// a search mode other than `fast` (`full-text-index.search-mode`, or else
105-
/// `global-index.search-mode`).
116+
/// `NotImplemented` for a primary-key full-text index (`pk-full-text.index.columns`), a
117+
/// search mode other than `fast` (`full-text-index.search-mode`, or else
118+
/// `global-index.search-mode`), a row filter whose scalar search mode is not `fast`
119+
/// (`scalar-index.search-mode`, or else `global-index.search-mode`), or a filter with a
120+
/// full-text match.
106121
Result<std::unique_ptr<FullTextScan>> NewFullTextScan() const;
107122

108123
/// Creates a read that searches the splits of a scan on the current settings. In addition to

‎src/paimon/CMakeLists.txt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -488,6 +488,7 @@ set(PAIMON_CORE_SRCS
488488
core/table/source/data_evolution_batch_scan.cpp
489489
core/table/source/data_evolution_full_text_read.cpp
490490
core/table/source/data_evolution_full_text_scan.cpp
491+
core/table/source/filtered_row_id_reader.cpp
491492
core/table/source/full_text_search_builder.cpp
492493
core/table/source/full_text_search_split.cpp
493494
core/table/source/primary_key_sorted_index_scan.cpp
@@ -1045,6 +1046,7 @@ if(PAIMON_BUILD_TESTS)
10451046
core/table/source/data_evolution_batch_scan_test.cpp
10461047
core/table/source/data_evolution_full_text_read_test.cpp
10471048
core/table/source/data_evolution_full_text_scan_test.cpp
1049+
core/table/source/filtered_row_id_reader_test.cpp
10481050
core/table/source/full_text_search_split_test.cpp
10491051
core/table/source/data_split_test.cpp
10501052
core/table/source/deletion_file_test.cpp

‎src/paimon/common/defs.cpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -164,6 +164,8 @@ const char Options::GLOBAL_INDEX_THREAD_NUM[] = "global-index.thread-num";
164164
const char Options::GLOBAL_INDEX_EXTERNAL_PATH[] = "global-index.external-path";
165165
const char Options::GLOBAL_INDEX_SEARCH_MODE[] = "global-index.search-mode";
166166
const char Options::FULL_TEXT_INDEX_SEARCH_MODE[] = "full-text-index.search-mode";
167+
const char Options::SCALAR_INDEX_SEARCH_MODE[] = "scalar-index.search-mode";
168+
const char Options::GLOBAL_INDEX_FILTER_REFINE_FROM_DATA[] = "global-index.filter.refine-from-data";
167169
const char Options::PK_BTREE_INDEX_COLUMNS[] = "pk-btree.index.columns";
168170
const char Options::PK_BITMAP_INDEX_COLUMNS[] = "pk-bitmap.index.columns";
169171
const char Options::PK_VECTOR_INDEX_COLUMNS[] = "pk-vector.index.columns";

‎src/paimon/core/core_options.cpp‎

Lines changed: 22 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -178,11 +178,10 @@ class ConfigParser {
178178
"sort engine", sort_engine);
179179
}
180180

181-
// Parse GlobalIndexSearchMode of full-text search, as Java `CoreOptions#indexSearchMode` does
182-
Status ParseFullTextIndexSearchMode(GlobalIndexSearchMode* search_mode) const {
183-
const char* key = config_map_.count(Options::FULL_TEXT_INDEX_SEARCH_MODE) > 0
184-
? Options::FULL_TEXT_INDEX_SEARCH_MODE
185-
: Options::GLOBAL_INDEX_SEARCH_MODE;
181+
// Parse GlobalIndexSearchMode of an index family, as Java `CoreOptions#indexSearchMode` does
182+
Status ParseIndexSearchMode(const char* family_key, GlobalIndexSearchMode* search_mode) const {
183+
const char* key =
184+
config_map_.count(family_key) > 0 ? family_key : Options::GLOBAL_INDEX_SEARCH_MODE;
186185
return ParseEnum(key,
187186
{{"fast", GlobalIndexSearchMode::FAST},
188187
{"full", GlobalIndexSearchMode::FULL},
@@ -450,6 +449,8 @@ struct CoreOptions::Impl {
450449
std::optional<int32_t> lookup_compact_max_interval;
451450
std::optional<int32_t> global_index_thread_num;
452451
GlobalIndexSearchMode full_text_index_search_mode = GlobalIndexSearchMode::FAST;
452+
GlobalIndexSearchMode scalar_index_search_mode = GlobalIndexSearchMode::FAST;
453+
bool global_index_filter_refine_from_data = false;
453454

454455
bool realtime_enabled = false;
455456
bool realtime_spill_enabled = false;
@@ -927,7 +928,14 @@ struct CoreOptions::Impl {
927928
PAIMON_RETURN_NOT_OK(
928929
parser.Parse<int32_t>(Options::GLOBAL_INDEX_THREAD_NUM, &global_index_thread_num));
929930
// Parse full-text-index.search-mode, falling back to global-index.search-mode, default fast
930-
PAIMON_RETURN_NOT_OK(parser.ParseFullTextIndexSearchMode(&full_text_index_search_mode));
931+
PAIMON_RETURN_NOT_OK(parser.ParseIndexSearchMode(Options::FULL_TEXT_INDEX_SEARCH_MODE,
932+
&full_text_index_search_mode));
933+
// Parse scalar-index.search-mode, falling back to global-index.search-mode, default fast
934+
PAIMON_RETURN_NOT_OK(parser.ParseIndexSearchMode(Options::SCALAR_INDEX_SEARCH_MODE,
935+
&scalar_index_search_mode));
936+
// Parse global-index.filter.refine-from-data, default false
937+
PAIMON_RETURN_NOT_OK(parser.Parse<bool>(Options::GLOBAL_INDEX_FILTER_REFINE_FROM_DATA,
938+
&global_index_filter_refine_from_data));
931939
// Parse global-index.external-path - global index root directory
932940
PAIMON_RETURN_NOT_OK(
933941
parser.Parse(Options::GLOBAL_INDEX_EXTERNAL_PATH, &global_index_external_path));
@@ -1920,6 +1928,14 @@ GlobalIndexSearchMode CoreOptions::GetFullTextIndexSearchMode() const {
19201928
return impl_->full_text_index_search_mode;
19211929
}
19221930

1931+
GlobalIndexSearchMode CoreOptions::GetScalarIndexSearchMode() const {
1932+
return impl_->scalar_index_search_mode;
1933+
}
1934+
1935+
bool CoreOptions::GlobalIndexFilterRefineFromData() const {
1936+
return impl_->global_index_filter_refine_from_data;
1937+
}
1938+
19231939
std::optional<std::string> CoreOptions::GetGlobalIndexExternalPath() const {
19241940
return impl_->global_index_external_path;
19251941
}

‎src/paimon/core/core_options.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -307,6 +307,10 @@ class PAIMON_EXPORT CoreOptions {
307307
/// Returns `full-text-index.search-mode`, or else `global-index.search-mode`, or else
308308
/// `GlobalIndexSearchMode::FAST`, as Java `CoreOptions#fullTextIndexSearchMode` does.
309309
GlobalIndexSearchMode GetFullTextIndexSearchMode() const;
310+
/// Returns `scalar-index.search-mode`, or else `global-index.search-mode`, or else
311+
/// `GlobalIndexSearchMode::FAST`, as Java `CoreOptions#scalarIndexSearchMode` does.
312+
GlobalIndexSearchMode GetScalarIndexSearchMode() const;
313+
bool GlobalIndexFilterRefineFromData() const;
310314

311315
const std::vector<std::string>& GetBlobFields() const;
312316
const std::vector<std::string>& GetBlobDescriptorFields() const;

0 commit comments

Comments
 (0)