Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions docs/source/api/file_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,15 @@ Interface
:members:
:undoc-members:

.. doxygenstruct:: paimon::FileIndexInput
:members:

.. doxygenstruct:: paimon::DataFileIndexInput
:members:

.. doxygenclass:: paimon::FileIndexWrite
:members:

.. doxygenclass:: paimon::FileIndexReader
:members:
:undoc-members:
Expand Down
34 changes: 34 additions & 0 deletions docs/source/user_guide/file_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,40 @@
File Index
==========

Attach indexes to existing data files
------------------------------------

``FileIndexWrite::PrepareCommit`` supports managed append tables with fixed buckets.
It merges serialized payloads into one external index file per data file and returns
commit messages. Inputs use original file field names and physical row positions,
including DV-deleted rows. Given a serialized bitmap stream ``payload``:

.. code-block:: cpp

#include "paimon/file_index/file_index_write.h"

std::vector<paimon::DataFileIndexInput> inputs = {
{"data-example.parquet", {{"original_field_name", "bitmap", payload}}}};
PAIMON_ASSIGN_OR_RAISE(
std::vector<std::shared_ptr<paimon::CommitMessage>> messages,
paimon::FileIndexWrite::PrepareCommit(
table_path, branch, options, base_snapshot_id, inputs,
file_system, pool, executor));

Use the payloads' snapshot as ``base_snapshot_id``; ``std::nullopt`` selects latest
once. Null file system, pool or executor selects internal defaults. Input streams
are borrowed synchronously and are not closed.

Existing targets are reused without reading or writing index payloads, and commit
messages are still prepared. Otherwise, index files are published from UUID temporary
files using ``Rename``. If the target appears meanwhile, it is reused and the temporary
file is cleaned up best-effort. Existing contents are not validated.
Concurrent publication requires atomic rename without overwrite from the file system.

Commit the messages with ``FileStoreCommit`` to the same table and branch.
Serialize backfill commits with compaction/overwrite per partition/bucket,
keeping base metadata valid through commit.

Full-text predicates and root TopK
----------------------------------

Expand Down
85 changes: 85 additions & 0 deletions include/paimon/file_index/file_index_write.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

#pragma once

#include <cstdint>
#include <map>
#include <memory>
#include <optional>
#include <string>
#include <vector>

#include "paimon/commit_message.h"
#include "paimon/result.h"
#include "paimon/visibility.h"

namespace paimon {
class Executor;
class FileSystem;
class InputStream;
class MemoryPool;

/// Serialized index payload using the data file's original field name.
struct PAIMON_EXPORT FileIndexInput {
std::string field_name;
std::string index_type;
/// Raw payload read from [0, Length()); nullptr denotes an empty index.
/// Non-null inputs must be nonempty. Borrowed synchronously, never closed or retained.
std::shared_ptr<InputStream> input;
};

/// Index updates for one live data file in the base snapshot.
struct PAIMON_EXPORT DataFileIndexInput {
std::string data_file_name;
std::vector<FileIndexInput> indexes;
};

/// Prepares file-index updates for fixed-bucket append tables.
/// Serialize backfill commits with compaction/overwrite per partition/bucket;
/// keep base metadata valid through commit.
class PAIMON_EXPORT FileIndexWrite {
public:
FileIndexWrite() = delete;
~FileIndexWrite() = delete;

/// Writes one external container per file, preserving indexes except supplied field/type pairs.
/// Ignores the embedding threshold; preserves other extra files and historical index files.
/// Reuses existing targets without reading or writing index payloads; still prepares messages.
/// Otherwise publishes UUID temporary files via Rename. If the target appears meanwhile,
/// reuses it and cleans up the temporary file best-effort. Existing contents are not validated.
/// Concurrent publication requires atomic rename without overwrite from the file system.
/// Payloads must use physical row positions, including DV-deleted rows. Do not use input
/// streams concurrently. Null file system, pool or executor uses the corresponding default.
///
/// @param branch Target branch; empty selects main.
/// @param options Table option overrides; bucket count cannot change.
/// @param base_snapshot_id Base snapshot; nullopt pins latest once for the batch.
/// @param files Unique live file names and field/type pairs; empty index lists are skipped.
/// @return Messages grouped by partition/bucket for commit to the same table and branch.
/// New files are cleaned up best-effort on failure; callers handle commit recovery and orphans.
static Result<std::vector<std::shared_ptr<CommitMessage>>> PrepareCommit(
const std::string& table_path, const std::string& branch,
const std::map<std::string, std::string>& options,
const std::optional<int64_t>& base_snapshot_id,
const std::vector<DataFileIndexInput>& files,
const std::shared_ptr<FileSystem>& file_system, const std::shared_ptr<MemoryPool>& pool,
const std::shared_ptr<Executor>& executor);
};

} // namespace paimon
2 changes: 2 additions & 0 deletions src/paimon/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -407,6 +407,7 @@ set(PAIMON_CORE_SRCS
core/operation/commit/retry_waiter.cpp
core/operation/commit_context.cpp
core/operation/expire_snapshots.cpp
core/operation/file_index_write.cpp
core/operation/file_store_commit.cpp
core/operation/file_store_commit_impl.cpp
core/operation/file_store_scan.cpp
Expand Down Expand Up @@ -1026,6 +1027,7 @@ if(PAIMON_BUILD_TESTS)
core/operation/commit_metrics_test.cpp
core/operation/commit_context_test.cpp
core/operation/expire_snapshots_test.cpp
core/operation/file_index_write_test.cpp
core/operation/file_store_commit_impl_test.cpp
core/operation/file_store_commit_test.cpp
core/operation/append_only_file_store_scan_test.cpp
Expand Down
3 changes: 3 additions & 0 deletions src/paimon/common/file_index/file_index_format_utils.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@
namespace paimon {
Result<FileIndexFormatUtils::FileIndexEntries> FileIndexFormatUtils::ReadIndexEntries(
const std::shared_ptr<InputStream>& input, const std::shared_ptr<MemoryPool>& pool) {
if (!input || !pool) {
return Status::Invalid("File index metadata requires an input stream and memory pool");
}
DataInputStream in(input);
PAIMON_ASSIGN_OR_RAISE(int64_t magic, in.ReadValue<int64_t>());
if (magic != FileIndexFormat::MAGIC) {
Expand Down
23 changes: 23 additions & 0 deletions src/paimon/core/io/data_file_path_factory.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,13 @@

#include "paimon/core/io/data_file_path_factory.h"

#include <limits>
#include <optional>
#include <utility>

#include "fmt/format.h"
#include "paimon/common/utils/path_util.h"
#include "paimon/common/utils/string_utils.h"
#include "paimon/common/utils/uuid.h"
#include "paimon/core/io/data_file_meta.h"
#include "paimon/macros.h"
Expand Down Expand Up @@ -66,6 +69,26 @@ std::string DataFilePathFactory::ToFileIndexPath(const std::string& file_path) c
return PathUtil::JoinPath(parent, PathUtil::GetName(file_path) + INDEX_PATH_SUFFIX);
}

std::string DataFilePathFactory::CreateNewFileIndexFilePath(const std::string& file_path) {
std::string file_name = PathUtil::GetName(file_path);
size_t dot = file_name.rfind('.');
size_t dash = file_name.rfind('-');
std::string parent = PathUtil::GetParentDirPath(file_path);
if (dash != std::string::npos && dash < dot) {
auto number =
StringUtils::StringToValue<int32_t>(file_name.substr(dash + 1, dot - dash - 1));
if (number) {
int32_t next = *number == std::numeric_limits<int32_t>::max()
? std::numeric_limits<int32_t>::min()
: *number + 1;
return PathUtil::JoinPath(parent, fmt::format("{}{}{}", file_name.substr(0, dash + 1),
next, INDEX_PATH_SUFFIX));
}
}
return PathUtil::JoinPath(parent,
fmt::format("{}-1{}", file_name.substr(0, dot), INDEX_PATH_SUFFIX));
}

std::string DataFilePathFactory::ToAlignedPath(const std::string& file_name,
const std::shared_ptr<DataFileMeta>& aligned) const {
auto external_path = aligned->ExternalPathDir();
Expand Down
1 change: 1 addition & 0 deletions src/paimon/core/io/data_file_path_factory.h
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ class DataFilePathFactory : public PathFactory {
}

std::string ToFileIndexPath(const std::string& file_path) const;
static std::string CreateNewFileIndexFilePath(const std::string& file_path);
std::string ToAlignedPath(const std::string& file_name,
const std::shared_ptr<DataFileMeta>& aligned) const;

Expand Down
21 changes: 21 additions & 0 deletions src/paimon/core/io/data_file_path_factory_test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,9 @@
#include "paimon/core/io/data_file_path_factory.h"

#include <optional>
#include <string>
#include <utility>
#include <vector>

#include "gtest/gtest.h"
#include "paimon/common/data/binary_row.h"
Expand Down Expand Up @@ -115,6 +117,25 @@ TEST_F(DataFilePathFactoryTest, TestToFileIndexPath) {
ASSERT_EQ(index_path, "/tmp/example.txt.index");
}

TEST_F(DataFilePathFactoryTest, TestCreateNewFileIndexFilePath) {
for (const auto& [before, after] : std::vector<std::pair<std::string, std::string>>{
{"example.txt.index", "example.txt-1.index"},
{"data-uuid-0.orc.index", "data-uuid-0.orc-1.index"},
{"data-uuid-0.orc-1.index", "data-uuid-0.orc-2.index"},
{"example-9.index", "example-10.index"},
{"example.index", "example-1.index"},
{"example-other.index", "example-other-1.index"},
{"example-2147483647.index", "example--2147483648.index"},
{"example-2147483648.index", "example-2147483648-1.index"}}) {
for (const std::string& parent :
std::vector<std::string>{"/tmp", "file:/external/bucket-0"}) {
ASSERT_EQ(PathUtil::JoinPath(parent, after),
DataFilePathFactory::CreateNewFileIndexFilePath(
PathUtil::JoinPath(parent, before)));
}
}
}

TEST_F(DataFilePathFactoryTest, TestToAlignedPath) {
auto file_meta = std::make_shared<DataFileMeta>(
"data-0.txt", /*file_size=*/645, /*row_count=*/5, BinaryRow::EmptyRow(),
Expand Down
66 changes: 55 additions & 11 deletions src/paimon/core/manifest/file_entry.h
Original file line number Diff line number Diff line change
Expand Up @@ -17,18 +17,23 @@
*/
#pragma once

#include <algorithm>
#include <cstddef>
#include <cstdint>
#include <limits>
#include <memory>
#include <optional>
#include <string>
#include <variant>
#include <vector>

#include "fmt/format.h"
#include "fmt/ranges.h"
#include "paimon/common/data/binary_row.h"
#include "paimon/common/utils/linked_hash_map.h"
#include "paimon/common/utils/murmurhash_utils.h"
#include "paimon/core/manifest/file_kind.h"
#include "paimon/memory/bytes.h"
#include "paimon/status.h"

namespace paimon {
Expand All @@ -40,16 +45,25 @@ class FileEntry {
/// file.
struct Identifier {
Identifier(const BinaryRow& _partition, int32_t _bucket, int32_t _level,
const std::string& _file_name, const std::optional<std::string>& _external_path)
const std::string& _file_name,
const std::vector<std::optional<std::string>>& _extra_files,
const std::shared_ptr<Bytes>& _embedded_index,
const std::optional<std::string>& _external_path)
: partition(_partition),
bucket(_bucket),
level(_level),
file_name(_file_name),
extra_files(_extra_files),
embedded_index(_embedded_index),
external_path(_external_path) {}

bool operator==(const Identifier& other) const {
return partition == other.partition && bucket == other.bucket && level == other.level &&
file_name == other.file_name && external_path == other.external_path;
file_name == other.file_name && extra_files == other.extra_files &&
(embedded_index == other.embedded_index ||
(embedded_index && other.embedded_index &&
*embedded_index == *other.embedded_index)) &&
external_path == other.external_path;
}

size_t HashCode() const {
Expand All @@ -61,31 +75,61 @@ class FileEntry {
hash_ = MurmurHashUtils::HashUnsafeBytes(reinterpret_cast<const void*>(&level),
/*offset=*/0, sizeof(level),
/*seed=*/hash_);
auto hash_bytes = [this](const char* data, size_t length) {
while (length > 0) {
int32_t chunk_size = static_cast<int32_t>(
std::min<size_t>(length, std::numeric_limits<int32_t>::max()));
hash_ = MurmurHashUtils::HashUnsafeBytes(data, /*offset=*/0, chunk_size,
/*seed=*/hash_);
data += chunk_size;
length -= chunk_size;
}
};
hash_bytes(file_name.data(), file_name.size());
for (const auto& extra_file : extra_files) {
bool present = extra_file.has_value();
hash_ = MurmurHashUtils::HashUnsafeBytes(&present, /*offset=*/0,
sizeof(present), /*seed=*/hash_);
if (extra_file) {
hash_bytes(extra_file->data(), extra_file->size());
}
}
bool has_embedded_index = embedded_index != nullptr;
hash_ = MurmurHashUtils::HashUnsafeBytes(
reinterpret_cast<const void*>(file_name.data()),
/*offset=*/0, file_name.size(),
/*seed=*/hash_);
&has_embedded_index, /*offset=*/0, sizeof(has_embedded_index), /*seed=*/hash_);
if (embedded_index) {
hash_bytes(embedded_index->data(), embedded_index->size());
}
bool has_external_path = external_path.has_value();
hash_ = MurmurHashUtils::HashUnsafeBytes(&has_external_path, /*offset=*/0,
sizeof(has_external_path), /*seed=*/hash_);
if (external_path) {
hash_ = MurmurHashUtils::HashUnsafeBytes(
reinterpret_cast<const void*>(external_path.value().data()),
/*offset=*/0, external_path.value().size(),
/*seed=*/hash_);
hash_bytes(external_path->data(), external_path->size());
}
}
return hash_;
}

std::string ToString() const {
std::vector<std::string> extra_file_names;
for (const auto& extra_file : extra_files) {
extra_file_names.push_back(extra_file.value_or("null"));
}
std::string embedded_index_str =
embedded_index ? fmt::format("{} bytes", embedded_index->size()) : "null";
std::string external_path_str =
external_path == std::nullopt ? "null" : external_path.value();
return fmt::format("{{{}, {}, {}, {}, {}}}", partition.ToString(), bucket, level,
file_name, external_path_str);
return fmt::format("{{{}, {}, {}, {}, [{}], {}, {}}}", partition.ToString(), bucket,
level, file_name, fmt::join(extra_file_names, ", "),
embedded_index_str, external_path_str);
}

BinaryRow partition;
int32_t bucket;
int32_t level;
std::string file_name;
std::vector<std::optional<std::string>> extra_files;
std::shared_ptr<Bytes> embedded_index;
std::optional<std::string> external_path;

private:
Expand Down
Loading
Loading