Skip to content
24 changes: 16 additions & 8 deletions docs/source/user_guide/realtime.rst
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,8 @@ The table schema and options select the mode; there is no separate
- Batches sorted by primary key and sequence number
- Merge disk runs, sealed batches, and building batches by primary key
* - PK DV
- Primary key; ``deletion-vectors.enabled=true``
- Primary key; ``merge-engine=deduplicate``;
``deletion-vectors.enabled=true``
- The same sorted PK store as PK MOR
- Apply committed DVs, then merge disk and memory by primary key
* - Append deduplicate
Expand All @@ -78,10 +79,12 @@ and exact residual filtering follow the mode-specific rules below.

Append accepts ``INSERT`` only. PK and append deduplicate also accept
``UPDATE_AFTER``, ``DELETE``, and ``UPDATE_BEFORE``. PK supports composite
primary keys, but its real-time merge engine is currently limited to
``merge-engine=deduplicate``. Append deduplicate currently accepts exactly one
non-null, non-nested key field and is experimental: its persisted index format
may change.
primary keys with ``merge-engine=deduplicate`` or ``merge-engine=partial-update``.
PK realtime partial-update requires deletion vectors disabled
(``deletion-vectors.enabled=false``). Partial update uses the existing merge
function and its sequence-group, field-aggregation, and delete options. Append deduplicate currently
accepts exactly one non-null, non-nested key field and is experimental: its
persisted index format may change.

PK DV and append deduplicate are different designs. PK DV retains primary-key
merge semantics; append deduplicate uses a persisted stable row offset and
Expand Down Expand Up @@ -459,11 +462,15 @@ number, row kind, and offset. Each sorted batch is an independent merge input;
multiple stored batches do not form a single globally sorted stream.

Query construction merges sorted disk sections and the memory readers, applies
deduplicate merge semantics, and drops final delete records. It collects all
the configured merge engine, and drops final delete records. It collects all
disk splits belonging to the active partition-bucket, rather than merging
memory against only one disk split. Preparing similarly merges the sealed
streams before handing them to ``MergeTreeWriter`` to write normal PK files.

Partial update reuses the existing framework merge logic and options. The store
continues to hold sorted batches and spill them without merging. Real-time
compaction remains disabled.

Only primary-key predicates are safe to push into unresolved versions. A
predicate on a value column must be evaluated after version merge: filtering
out the newest value first could incorrectly expose an older matching value.
Expand Down Expand Up @@ -629,8 +636,9 @@ Current restrictions include:
* Process-local batch union scans only. Streaming scans, limit pushdown,
global-index result splits, vector/full-text search, and count-only real-time
readers are unsupported.
* PK mode supports the deduplicate merge engine with default delete handling,
ascending sequence order, and ``changelog-producer=none``. Custom
* PK mode supports deduplicate with default delete handling, or partial update
with deletion vectors disabled and its existing delete and sequence-group
options. Both require ascending sequence order and ``changelog-producer=none``. Custom
``sequence.field``, floating-point keys, PK global indexes, lookup without DVs,
and read-optimized union scans are unsupported.
* Append deduplicate requires a single non-null scalar key, a persisted offset
Expand Down
8 changes: 4 additions & 4 deletions src/paimon/core/operation/key_value_file_store_write.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -175,10 +175,10 @@ Result<std::shared_ptr<BatchWriter>> KeyValueFileStoreWrite::CreateWriter(
if (!realtime_context_) {
return std::shared_ptr<BatchWriter>(std::move(writer));
}
return RealtimePrimaryKeyWriter::Create(partition_map, bucket, realtime_schema_layout_,
trimmed_primary_keys, key_comparator_, options_,
realtime_context_impl, realtime_store_state.value(),
restore_max_seq_number, writer, pool_);
return RealtimePrimaryKeyWriter::Create(
partition_map, bucket, realtime_schema_layout_, trimmed_primary_keys,
table_schema_->PrimaryKeys(), key_comparator_, options_, realtime_context_impl,
realtime_store_state.value(), restore_max_seq_number, writer, pool_);
}

Status KeyValueFileStoreWrite::RefreshCommittedSnapshot(int64_t snapshot_id) {
Expand Down
17 changes: 11 additions & 6 deletions src/paimon/core/realtime/realtime_primary_key_writer.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ Result<std::shared_ptr<RealtimePrimaryKeyWriter>> RealtimePrimaryKeyWriter::Crea
const std::map<std::string, std::string>& partition, int32_t bucket,
const std::shared_ptr<RealtimeSchemaLayout>& schema_layout,
const std::vector<std::string>& trimmed_primary_keys,
const std::vector<std::string>& primary_keys,
const std::shared_ptr<FieldsComparator>& key_comparator, const CoreOptions& options,
const std::shared_ptr<RealtimeContextImpl>& realtime_context,
const RealtimeStoreState& store_state, int64_t restored_max_sequence_number,
Expand All @@ -112,8 +113,10 @@ Result<std::shared_ptr<RealtimePrimaryKeyWriter>> RealtimePrimaryKeyWriter::Crea
restored_max_sequence_number == std::numeric_limits<int64_t>::max()) {
return Status::Invalid("PK restored sequence number is invalid");
}
if (options.GetMergeEngine() != MergeEngine::DEDUPLICATE) {
return Status::NotImplemented("PK realtime supports only the DEDUPLICATE merge engine");
if (options.GetMergeEngine() != MergeEngine::DEDUPLICATE &&
options.GetMergeEngine() != MergeEngine::PARTIAL_UPDATE) {
return Status::NotImplemented(
"PK realtime supports only the DEDUPLICATE and PARTIAL_UPDATE merge engines");
}
if (!schema_layout) {
return Status::Invalid("PK realtime schema layout is null");
Expand All @@ -133,8 +136,8 @@ Result<std::shared_ptr<RealtimePrimaryKeyWriter>> RealtimePrimaryKeyWriter::Crea
partition_bucket, restored_max_sequence_number));
return std::shared_ptr<RealtimePrimaryKeyWriter>(new RealtimePrimaryKeyWriter(
store_state.store, merge_tree_writer, realtime_context, partition_bucket, schema_layout,
arrow::schema(std::move(key_fields)), trimmed_primary_keys, key_comparator, options,
store_state.initial_offset, initial_max_sequence_number, memory_pool));
arrow::schema(std::move(key_fields)), trimmed_primary_keys, primary_keys, key_comparator,
options, store_state.initial_offset, initial_max_sequence_number, memory_pool));
}

RealtimePrimaryKeyWriter::RealtimePrimaryKeyWriter(
Expand All @@ -145,6 +148,7 @@ RealtimePrimaryKeyWriter::RealtimePrimaryKeyWriter(
const std::shared_ptr<RealtimeSchemaLayout>& schema_layout,
const std::shared_ptr<arrow::Schema>& key_schema,
const std::vector<std::string>& trimmed_primary_keys,
const std::vector<std::string>& primary_keys,
const std::shared_ptr<FieldsComparator>& key_comparator, const CoreOptions& options,
int64_t next_offset, int64_t last_sequence_number,
const std::shared_ptr<MemoryPool>& memory_pool)
Expand All @@ -157,6 +161,7 @@ RealtimePrimaryKeyWriter::RealtimePrimaryKeyWriter(
schema_layout_(schema_layout),
key_schema_(key_schema),
trimmed_primary_keys_(trimmed_primary_keys),
primary_keys_(primary_keys),
key_comparator_(key_comparator),
options_(options),
next_offset_(next_offset),
Expand Down Expand Up @@ -263,8 +268,8 @@ Status RealtimePrimaryKeyWriter::FlushSegment(
realtime_primary_key_readers) {
PAIMON_ASSIGN_OR_RAISE(
std::unique_ptr<MergeFunction> merge_function,
PrimaryKeyTableUtils::CreateMergeFunction(
schema_layout_->UserSchema(), trimmed_primary_keys_, options_, memory_pool_));
PrimaryKeyTableUtils::CreateMergeFunction(schema_layout_->UserSchema(), primary_keys_,
options_, memory_pool_));
sorted_readers.push_back(std::make_unique<MergedKeyValueRecordReader>(
std::move(realtime_primary_key_reader), key_comparator_,
std::make_shared<ReducerMergeFunctionWrapper>(std::move(merge_function))));
Expand Down
3 changes: 3 additions & 0 deletions src/paimon/core/realtime/realtime_primary_key_writer.h
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ class RealtimePrimaryKeyWriter final : public BatchWriter {
const std::map<std::string, std::string>& partition, int32_t bucket,
const std::shared_ptr<RealtimeSchemaLayout>& schema_layout,
const std::vector<std::string>& trimmed_primary_keys,
const std::vector<std::string>& primary_keys,
const std::shared_ptr<FieldsComparator>& key_comparator, const CoreOptions& options,
const std::shared_ptr<RealtimeContextImpl>& realtime_context,
const RealtimeStoreState& store_state, int64_t restored_max_sequence_number,
Expand Down Expand Up @@ -78,6 +79,7 @@ class RealtimePrimaryKeyWriter final : public BatchWriter {
const std::shared_ptr<RealtimeSchemaLayout>& schema_layout,
const std::shared_ptr<arrow::Schema>& key_schema,
const std::vector<std::string>& trimmed_primary_keys,
const std::vector<std::string>& primary_keys,
const std::shared_ptr<FieldsComparator>& key_comparator,
const CoreOptions& options, int64_t next_offset,
int64_t last_sequence_number,
Expand All @@ -96,6 +98,7 @@ class RealtimePrimaryKeyWriter final : public BatchWriter {
std::vector<std::shared_ptr<RealtimeSegmentHandle>> sealed_segments_;
std::shared_ptr<arrow::Schema> key_schema_;
std::vector<std::string> trimmed_primary_keys_;
std::vector<std::string> primary_keys_;
std::shared_ptr<FieldsComparator> key_comparator_;
CoreOptions options_;
int64_t next_offset_;
Expand Down
2 changes: 2 additions & 0 deletions src/paimon/core/table/source/data_table_batch_scan.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ DataTableBatchScan::DataTableBatchScan(bool pk_table, const CoreOptions& core_op
snapshot_reader_->WithLevelFilter(
[top_level](int32_t level) -> bool { return level == top_level; });
snapshot_reader_->EnableValueFilter();
} else if (realtime_pk_scan && core_options.GetMergeEngine() == MergeEngine::PARTIAL_UPDATE) {
snapshot_reader_->EnableValueFilterForLevels([](int32_t) -> bool { return false; });
} else if (pk_table && (core_options.DeletionVectorsEnabled() ||
core_options.GetMergeEngine() == MergeEngine::FIRST_ROW)) {
if (realtime_pk_scan) {
Expand Down
13 changes: 10 additions & 3 deletions src/paimon/core/utils/primary_key_table_utils.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -103,13 +103,20 @@ Status PrimaryKeyTableUtils::ValidateRealtimeOptions(const CoreOptions& options,
if (options.GetBucket() <= 0) {
return Status::NotImplemented("PK realtime requires fixed buckets");
}
if (options.GetMergeEngine() != MergeEngine::DEDUPLICATE) {
return Status::NotImplemented("PK realtime supports only the DEDUPLICATE merge engine");
if (options.GetMergeEngine() != MergeEngine::DEDUPLICATE &&
options.GetMergeEngine() != MergeEngine::PARTIAL_UPDATE) {
Comment thread
HaHaJeff marked this conversation as resolved.
Resolved
return Status::NotImplemented(
"PK realtime supports only the DEDUPLICATE and PARTIAL_UPDATE merge engines");
Comment thread
lxy-9602 marked this conversation as resolved.
Resolved
}
if (options.GetMergeEngine() == MergeEngine::PARTIAL_UPDATE &&
options.DeletionVectorsEnabled()) {
return Status::NotImplemented(
"PK realtime partial-update does not support deletion vectors");
}
if (options.DataEvolutionEnabled()) {
return Status::NotImplemented("PK realtime does not support data evolution");
}
if (options.IgnoreDelete()) {
if (options.GetMergeEngine() == MergeEngine::DEDUPLICATE && options.IgnoreDelete()) {
return Status::NotImplemented("PK realtime requires default delete behavior");
}
if (!options.GetSequenceField().empty()) {
Expand Down
24 changes: 15 additions & 9 deletions src/paimon/core/utils/primary_key_table_utils_test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
#include <vector>

#include "arrow/type.h"
#include "fmt/format.h"
#include "gtest/gtest.h"
#include "paimon/common/types/data_field.h"
#include "paimon/common/types/row_kind.h"
Expand Down Expand Up @@ -58,16 +59,22 @@ std::shared_ptr<TableSchema> PkSchema(
} // namespace

TEST(PrimaryKeyTableUtilsTest, TestSupportedRealtimeOptions) {
ASSERT_OK_AND_ASSIGN(CoreOptions options,
CoreOptions::FromMap({{Options::BUCKET, "1"},
{Options::DELETION_VECTORS_ENABLED, "true"}}));
ASSERT_OK(PrimaryKeyTableUtils::ValidateRealtimeOptions(options, *PkSchema()));
const std::vector<std::map<std::string, std::string>> supported_options = {
{{Options::BUCKET, "1"}, {Options::DELETION_VECTORS_ENABLED, "true"}},
{{Options::BUCKET, "1"},
{Options::MERGE_ENGINE, "partial-update"},
{Options::DELETION_VECTORS_ENABLED, "false"}},
};
for (const std::map<std::string, std::string>& option_map : supported_options) {
ASSERT_OK_AND_ASSIGN(CoreOptions options, CoreOptions::FromMap(option_map));
ASSERT_OK(PrimaryKeyTableUtils::ValidateRealtimeOptions(options, *PkSchema()));
}
}

TEST(PrimaryKeyTableUtilsTest, TestUnsupportedRealtimeOptions) {
const std::vector<std::map<std::string, std::string>> unsupported_options = {
{{Options::BUCKET, "0"}},
{{Options::BUCKET, "1"}, {Options::MERGE_ENGINE, "partial-update"}},
{{Options::BUCKET, "1"}, {Options::MERGE_ENGINE, "aggregation"}},
{{Options::BUCKET, "1"}, {Options::DATA_EVOLUTION_ENABLED, "true"}},
{{Options::BUCKET, "1"}, {Options::SEQUENCE_FIELD, "seq"}},
};
Expand Down Expand Up @@ -112,7 +119,7 @@ TEST(PrimaryKeyTableUtilsTest, TestRealtimeRejectsDeleteAndSequenceOrderingOptio
"supports only ascending sequence.field.sort-order");
}

TEST(PrimaryKeyTableUtilsTest, TestRealtimeReportsSpecificLookupErrors) {
TEST(PrimaryKeyTableUtilsTest, TestRealtimeReportsSpecificUnsupportedOptions) {
const std::vector<std::pair<std::map<std::string, std::string>, std::string>> cases = {
{{{Options::BUCKET, "1"}, {Options::FORCE_LOOKUP, "true"}},
"PK realtime does not support lookup"},
Expand All @@ -123,9 +130,8 @@ TEST(PrimaryKeyTableUtilsTest, TestRealtimeReportsSpecificLookupErrors) {
};
for (const auto& [option_map, expected_message] : cases) {
ASSERT_OK_AND_ASSIGN(CoreOptions options, CoreOptions::FromMap(option_map));
Status status = PrimaryKeyTableUtils::ValidateRealtimeOptions(options, *PkSchema());
ASSERT_TRUE(status.IsNotImplemented());
ASSERT_EQ(status.message(), expected_message);
ASSERT_NOK_WITH_MSG(PrimaryKeyTableUtils::ValidateRealtimeOptions(options, *PkSchema()),
fmt::format("NotImplemented: {}", expected_message));
}
}

Expand Down
Loading
Loading