Skip to content

Commit f2f5c9d

Browse files
author
zhoujianfeng.jeff
committed
chore: merge main into realtime partial-update branch
Retain both Arrow utility headers required by the partial-update tests and the upstream refresh regression test.
2 parents a0e32ac + 26753d5 commit f2f5c9d

50 files changed

Lines changed: 2902 additions & 226 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/source/user_guide/realtime.rst‎

Lines changed: 50 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,11 @@ Common Options
118118
- ``none``
119119
- ``none`` or ``full``. Full mode collects per-batch min/max and null counts
120120
for conservative predicate pruning, including retained spill metadata.
121+
* - ``realtime.index.<index_name>.columns``
122+
- Unset
123+
- Comma-separated fields indexed in memory by the default Arrow store.
124+
Available in append and append deduplicate modes; independent of
125+
``realtime.store.stats-mode``.
121126
* - ``realtime.deduplicate-key-fields``
122127
- Unset
123128
- Selects append deduplicate mode. Although expressed as a comma-separated
@@ -401,6 +406,37 @@ context's reclaimed progress for relevant active stores. It does not restore
401406
an old in-memory state and is not unrestricted historical time travel. For a
402407
disk-only historical query, do not attach the real-time context.
403408

409+
In-memory Predicate Indexes
410+
--------------------------
411+
412+
The default Arrow store supports optional predicate indexes on key or value fields
413+
in append and append deduplicate modes. PK modes are unsupported. For example:
414+
415+
.. code-block:: text
416+
417+
realtime.enabled = true
418+
realtime.index.hash.columns = id,payload
419+
realtime.store.stats-mode = full
420+
421+
The built-in ``hash`` plugin is **experimental**; its implementation and supported
422+
capabilities may change. It supports equality and ``IN`` predicates on boolean,
423+
integer, floating-point, string, binary, and date fields. Decimal, timestamp, and
424+
nested types are currently unsupported. Use a comma-separated list to index
425+
multiple fields; indexing ``_REALTIME_OFFSET`` is unsupported.
426+
427+
Note that this index does not currently handle hash collisions.
428+
429+
Indexes and batch statistics work independently and can be enabled together.
430+
For ``id = 7 AND score > 100``, an index supporting equality on ``id`` selects
431+
candidate offsets; statistics can then skip batches whose maximum ``score``
432+
is at most 100.
433+
Statistics cover entire batches, so mixed batches still need row-level
434+
filtering. Keep predicate fields in the read schema and enable
435+
``ReadContextBuilder::EnablePredicateFilter(true)`` for exact filtering.
436+
437+
In-memory and disk indexes are configured independently. Use
438+
``file-index.<type>.columns`` to enable indexes for committed files.
439+
404440
Implementation by Mode
405441
----------------------
406442

@@ -409,9 +445,10 @@ Append
409445

410446
``RealtimeAppendOnlyWriter`` validates offsets and writes to an
411447
``APPEND_ONLY`` store. The default ``ArrowRealtimeStore`` retains building
412-
batches and immutable sealed segments in write order. Query readers project
413-
the requested fields, optionally prune stored batches using statistics, and
414-
apply the visible offset range before removing the hidden offset column.
448+
batches and immutable sealed segments in write order. Query readers use
449+
optional predicate indexes and batch statistics to prune candidates, project
450+
the requested fields, and apply the visible offset range before removing the
451+
hidden offset column.
415452

416453
Preparing reads sealed segments through the ordinary ``AppendOnlyWriter``,
417454
removing the internal offset prefix before writing table files. Queries do not
@@ -474,8 +511,10 @@ are marked deleted, and within-batch duplicates are resolved in row order.
474511
Add rows are appended to the store; delete events update only the key/DV state,
475512
without persisting a delete row.
476513

477-
Key indexes and DV bitmaps use copy-on-write snapshots, so later writes do not
478-
change the state held by existing lookups, read views, or seal checkpoints.
514+
Deduplicate key-to-latest-offset lookups and DV bitmaps use copy-on-write
515+
snapshots, so later writes do not change the state held by existing lookups,
516+
read views, or seal checkpoints. The optional predicate indexes share
517+
append-only postings instead.
479518

480519
Memory reads use the read view's DV to exclude rows whose offsets are marked
481520
deleted.
@@ -518,8 +557,9 @@ still terminal for the owning writer/context.
518557
Old views can retain the memory representation after spill or reclamation;
519558
new views use the published representation. Spill files are deleted when the
520559
store and all remaining handles/readers have released them. They are temporary
521-
storage, not a recoverable log after process restart. Deduplicate key indexes
522-
and DVs remain in memory; full-mode statistics also remain resident after spill.
560+
storage, not a recoverable log after process restart. Predicate indexes,
561+
deduplicate key indexes, DVs, and full-mode statistics remain in memory after
562+
spill. Predicate index memory is included in store memory accounting.
523563

524564
``RealtimeContext::GetMetrics()`` and real-time writer metrics expose:
525565

@@ -587,7 +627,7 @@ Extension Points and Current Limits
587627
Custom stores manage buffering, read views, and spill while following the
588628
``RealtimeStore`` contracts. Paimon manages table files, indexes, merge semantics,
589629
and commits. Append deduplicate also keeps key indexes and DVs in the framework;
590-
custom delegates need ``RealtimeStoreTwoPhaseSeal`` to enable its spill.
630+
custom delegates need ``TwoPhaseSealableRealtimeStore`` to enable its spill.
591631

592632
Current restrictions include:
593633

@@ -604,8 +644,8 @@ Current restrictions include:
604644
* Append deduplicate requires a single non-null scalar key, a persisted offset
605645
column, exact offset file-index reads, and non-bitmap64 DVs. Existing ordinary
606646
append files are not migrated automatically.
607-
* Spill covers Arrow data, not key indexes or DVs, and provides neither a hard
608-
memory limit nor crash recovery.
647+
* Spill covers Arrow data, not predicate indexes, key indexes, or DVs, and
648+
provides neither a hard memory limit nor crash recovery.
609649

610650
Code Map
611651
--------

‎include/paimon/api.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
#include "paimon/file_store_commit.h" // IWYU pragma: export
2727
#include "paimon/file_store_write.h" // IWYU pragma: export
2828
#include "paimon/fs/file_system_factory.h" // IWYU pragma: export
29+
#include "paimon/io/offset_output_stream.h" // IWYU pragma: export
2930
#include "paimon/memory/memory_pool.h" // IWYU pragma: export
3031
#include "paimon/predicate/predicate.h" // IWYU pragma: export
3132
#include "paimon/read_context.h" // IWYU pragma: export
@@ -42,6 +43,9 @@
4243
#include "paimon/write_context.h" // IWYU pragma: export
4344

4445
// IWYU pragma: begin_exports
46+
#include "paimon/realtime/index/realtime_index.h"
47+
#include "paimon/realtime/index/realtime_index_factory.h"
48+
#include "paimon/realtime/index/realtime_index_reader.h"
4549
#include "paimon/realtime/realtime_context.h"
4650
#include "paimon/realtime/realtime_snapshot_offsets.h"
4751
#include "paimon/realtime/realtime_store.h"

‎include/paimon/file_store_write.h‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,9 @@ class RecordBatch;
4141
class WriteContext;
4242

4343
/// Interface for write operations in a file store.
44+
///
45+
/// A `RealtimeContext` supports only one active writer for each partition-bucket. Close the
46+
/// previous writer before creating a replacement that accesses the same partition-bucket.
4447
class PAIMON_EXPORT FileStoreWrite {
4548
public:
4649
/// Create an instance of `FileStoreWrite`.
@@ -134,6 +137,10 @@ class PAIMON_EXPORT FileStoreWrite {
134137
///
135138
/// The writer loads the snapshot's partition-bucket offsets and releases sealed memory that is
136139
/// fully covered by disk. Calling this method on a non-real-time writer returns an error.
140+
/// After a successful refresh, fully covered segments are also removed from this writer's
141+
/// pending prepare queue. Existing read views continue to pin the resources they reference.
142+
/// Snapshot installation and queue cleanup are serialized with `Seal()` and
143+
/// `PrepareCommitWithProgress()`.
137144
///
138145
/// With `WriteContextBuilder::WithCatalog()`, the latest snapshot can be loaded from the
139146
/// catalog. Older requested snapshots must be available on the file system.
@@ -162,8 +169,9 @@ class PAIMON_EXPORT FileStoreWrite {
162169

163170
/// Releases resources owned by this writer.
164171
///
165-
/// Closing a real-time writer with data not covered by a successful
166-
/// `PrepareCommitWithProgress()` returns an error and invalidates its `RealtimeContext`.
172+
/// Closing a real-time writer with data covered by neither a successful
173+
/// `PrepareCommitWithProgress()` nor a refreshed committed snapshot returns an error and
174+
/// invalidates its `RealtimeContext`.
167175
/// The caller must rebuild both objects and recover input from the durable snapshot offset.
168176
virtual Status Close() = 0;
169177
};

src/paimon/common/io/offset_output_stream.h renamed to include/paimon/io/offset_output_stream.h

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,11 +34,14 @@ namespace paimon {
3434
/// outlive this view.
3535
class PAIMON_EXPORT OffsetOutputStream : public OutputStream {
3636
public:
37+
/// Creates a relative-position view over a non-null borrowed stream.
3738
static Result<std::unique_ptr<OffsetOutputStream>> Create(OutputStream* output);
3839
~OffsetOutputStream() override = default;
3940

4041
Result<int64_t> Write(const char* buffer, int64_t size) override;
4142
Result<int64_t> GetPos() const override;
43+
/// Returns the current position of the innermost stream, unwrapping nested OffsetOutputStreams.
44+
Result<int64_t> GetUnderlyingPos() const;
4245
Result<std::string> GetUri() const override;
4346
Status Flush() override;
4447
Status Close() override;

‎include/paimon/realtime/arrow_realtime_store_factory.h‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,8 @@ class PAIMON_EXPORT ArrowRealtimeStoreFactory : public RealtimeStoreFactory {
2828
public:
2929
/// Creates an Arrow-backed append or primary-key store for one partition and bucket.
3030
/// The framework uses the append implementation as the delegate of its DEDUPLICATE wrapper.
31+
/// Append delegates resolve configured index plugins through the registry.
32+
/// Primary-key stores do not support query indexes.
3133
Result<std::shared_ptr<RealtimeStore>> Create(RealtimeStoreCreateRequest&& request) override;
3234
};
3335

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
#pragma once
21+
22+
#include <cstdint>
23+
#include <memory>
24+
25+
#include "paimon/realtime/offset_range.h"
26+
#include "paimon/result.h"
27+
#include "paimon/visibility.h"
28+
29+
struct ArrowArray;
30+
31+
namespace paimon {
32+
class RealtimeIndexReader;
33+
34+
/// Append-only in-memory index for one field, supporting one writer and concurrent readers.
35+
class PAIMON_EXPORT RealtimeIndex {
36+
public:
37+
virtual ~RealtimeIndex() = default;
38+
39+
/// Consumes [_REALTIME_OFFSET, field] with increasing offsets.
40+
virtual Status AddBatch(::ArrowArray* batch) = 0;
41+
42+
/// Creates a reader for [begin, end), sharing postings without copying.
43+
virtual Result<std::shared_ptr<RealtimeIndexReader>> CreateReader(
44+
const OffsetRange& offset_range) const = 0;
45+
46+
virtual uint64_t GetMemoryUsage() const = 0;
47+
};
48+
49+
} // namespace paimon
Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
#pragma once
21+
22+
#include <map>
23+
#include <memory>
24+
#include <string>
25+
26+
#include "paimon/factories/factory.h"
27+
#include "paimon/result.h"
28+
#include "paimon/visibility.h"
29+
30+
struct ArrowSchema;
31+
32+
namespace paimon {
33+
class MemoryPool;
34+
class RealtimeIndex;
35+
36+
/// Factory for field indexes configured by realtime.index.<identifier>.columns.
37+
class PAIMON_EXPORT RealtimeIndexFactory : public Factory {
38+
public:
39+
~RealtimeIndexFactory() override = default;
40+
41+
/// Registry suffix for real-time index plugins.
42+
static const char REALTIME_INDEX_IDENTIFIER_SUFFIX[];
43+
44+
/// Resolves identifier + "-realtime" and consumes the [_REALTIME_OFFSET, field] schema.
45+
static Result<std::shared_ptr<RealtimeIndex>> Get(
46+
const std::string& identifier, ::ArrowSchema* write_schema,
47+
const std::map<std::string, std::string>& options, const std::shared_ptr<MemoryPool>& pool);
48+
49+
/// Creates a new field index; consumes the [_REALTIME_OFFSET, field] schema.
50+
virtual Result<std::shared_ptr<RealtimeIndex>> Create(
51+
::ArrowSchema* write_schema, const std::map<std::string, std::string>& options,
52+
const std::shared_ptr<MemoryPool>& pool) const = 0;
53+
};
54+
55+
} // namespace paimon
Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
#pragma once
21+
22+
#include <memory>
23+
#include <optional>
24+
#include <vector>
25+
26+
#include "paimon/predicate/function_visitor.h"
27+
#include "paimon/utils/roaring_bitmap64.h"
28+
#include "paimon/visibility.h"
29+
30+
namespace paimon {
31+
32+
/// Visits predicates on one indexed field, returning realtime offsets.
33+
/// nullopt means no pruning; an empty bitmap means no candidates. False negatives are not allowed.
34+
/// Unsupported predicates default to no pruning. Plugins override only the supported visits.
35+
class PAIMON_EXPORT RealtimeIndexReader : public FunctionVisitor<std::optional<RoaringBitmap64>> {
36+
public:
37+
~RealtimeIndexReader() override = default;
38+
39+
Result<std::optional<RoaringBitmap64>> VisitIsNotNull() override;
40+
Result<std::optional<RoaringBitmap64>> VisitIsNull() override;
41+
Result<std::optional<RoaringBitmap64>> VisitEqual(const Literal& literal) override;
42+
Result<std::optional<RoaringBitmap64>> VisitNotEqual(const Literal& literal) override;
43+
Result<std::optional<RoaringBitmap64>> VisitLessThan(const Literal& literal) override;
44+
Result<std::optional<RoaringBitmap64>> VisitLessOrEqual(const Literal& literal) override;
45+
Result<std::optional<RoaringBitmap64>> VisitGreaterThan(const Literal& literal) override;
46+
Result<std::optional<RoaringBitmap64>> VisitGreaterOrEqual(const Literal& literal) override;
47+
Result<std::optional<RoaringBitmap64>> VisitIn(const std::vector<Literal>& literals) override;
48+
Result<std::optional<RoaringBitmap64>> VisitNotIn(
49+
const std::vector<Literal>& literals) override;
50+
Result<std::optional<RoaringBitmap64>> VisitStartsWith(const Literal& literal) override;
51+
Result<std::optional<RoaringBitmap64>> VisitEndsWith(const Literal& literal) override;
52+
Result<std::optional<RoaringBitmap64>> VisitContains(const Literal& literal) override;
53+
Result<std::optional<RoaringBitmap64>> VisitLike(const Literal& literal) override;
54+
};
55+
56+
} // namespace paimon

‎include/paimon/realtime/realtime_context.h‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,8 @@ class PAIMON_EXPORT RealtimeMetrics {
8888
/// reads. `RealtimeContext` itself is not a customization interface and must not be implemented by
8989
/// applications. Customize real-time storage and retrieval through `RealtimeStoreFactory` and
9090
/// `RealtimeStore` instead.
91+
/// Only one writer may actively use a partition-bucket in this context. Close that writer before
92+
/// creating its replacement; writers for different partition-buckets may share the context.
9193
///
9294
/// A context is valid only for one uninterrupted committed-progress history. Overwrite, truncate,
9395
/// partition drop, and rollback operations do not automatically clear process-local real-time

‎include/paimon/realtime/realtime_store.h‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,9 @@ struct PAIMON_EXPORT RealtimeQueryContext {
142142
/// Optional predicate using field indexes from `read_schema`. A non-null predicate allows the
143143
/// plugin to prune candidate rows. Exact filtering is applied by the Paimon read framework.
144144
std::shared_ptr<Predicate> predicate;
145+
/// Required [disk snapshot committed end, pinned memory end) bounds.
146+
/// Pin memory before selecting the disk snapshot; keep these bounds for the query.
147+
OffsetRange offset_range;
145148
/// Maximum number of rows in a returned query batch.
146149
int32_t read_batch_size = 1024;
147150
};
@@ -214,9 +217,9 @@ class PAIMON_EXPORT RealtimeStore {
214217

215218
/// Optional capability for `RealtimeStore` implementations to separate segment freezing from spill
216219
/// I/O.
217-
class PAIMON_EXPORT RealtimeStoreTwoPhaseSeal : public RealtimeStore {
220+
class PAIMON_EXPORT TwoPhaseSealableRealtimeStore : public RealtimeStore {
218221
public:
219-
~RealtimeStoreTwoPhaseSeal() override = default;
222+
~TwoPhaseSealableRealtimeStore() override = default;
220223

221224
/// Freezes the building segment without spill I/O.
222225
virtual Result<std::optional<std::shared_ptr<RealtimeSegmentHandle>>>

0 commit comments

Comments
 (0)