Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 52 additions & 14 deletions docs/source/user_guide/prefetch.rst
Original file line number Diff line number Diff line change
Expand Up @@ -74,23 +74,60 @@ single-reader path.
Sequential file preparation
===========================

Append/raw reads construct a sliding window of at most four data-file reader
stacks per split, including construction tasks still in flight. The initial
window is built on the dedicated reader-build executor before ``CreateReader``
returns. As a file reaches EOF, its reader is closed, its metrics are retained,
and the next file is submitted for construction while prepared files are read.
Append/raw reads construct a sliding window of data-file reader stacks per
split, including construction tasks still in flight. The C++-specific option
``read.preparation-window`` accepts a positive 32-bit integer and defaults to
``4``. This limit is independent of the reader-build executor's
parallelism, batch prefetch and warmup settings. A value of ``1`` retains only
the current reader. Larger windows let the next files prepare while the
current file is read. The dedicated reader-build executor constructs the
initial window before ``CreateReader(split)`` returns. At EOF the split reader
closes the finished file reader, records its metrics and submits the next
file for construction.
This bounds open streams and format metadata independently of the number of
files in the split. Per-file metadata and owned creation inputs still scale
with the split's file count. Merge-on-read keeps its eager input construction.

Plan validation still runs during ``CreateReader``. File-open and reader-build
errors in the initial window are returned there; errors for later files are
returned through ``NextBatch`` or ``NextBatchWithBitmap`` when that file is
reached. These errors are terminal. The returned reader retains its creation
context, schemas, file system and memory pools, so it may outlive the object
that created it. Closing or destroying it cancels queued construction, waits
for already running construction, and releases the prepared readers without
scheduling the unopened tail.
The following example configures a window of eight readers per split.

.. code-block:: cpp

paimon::ReadContextBuilder builder(table_path);
builder.AddOption(paimon::Options::READ_PREPARATION_WINDOW, "8");

For ``CreateReader(split)``, plan validation still runs during creation.
Creation returns file-open and reader-build errors from the initial window.
For later files, ``NextBatch`` and ``NextBatchWithBitmap`` return the error
when reading reaches the affected file. These errors are terminal.
The returned reader retains its creation context, schemas, file system and
memory pools, so it may outlive the object that created it. Closing or
destroying it cancels queued construction, waits for already running
construction, and releases the prepared readers without scheduling the
unopened tail.

In the append-only and primary-key implementations, ``CreateReader(splits)``
opens splits in input order only when they are reached. The previous split
is closed before the next is opened. A list of raw disk splits therefore
retains at most one preparation window. A real-time split can group
multiple disk inputs and open them together when it is reached. Closing
before the first batch opens no data files.
The returned reader owns its creation state and may outlive the ``TableRead``
and the input split list. Errors opening a split or its initial files are
terminal when reading reaches that split.

``CreateReader(splits)`` validates disk split metadata, including disk splits
inside real-time splits. It checks for missing inputs, misaligned deletion
files, and mismatches between indexed files, row ranges and scores.
Primary-key real-time merges require ordinary data splits as disk inputs.
Creation also checks each real-time split's version, partition and bucket,
offset bounds and pinned table schema before consuming any tickets.
If validation fails, valid tickets remain available for retry.
Successful creation consumes the tickets and retains the views across
ticket expiry and committed-state refreshes. Close, EOF and read errors
release the remaining creation state and retained views.

Branch-fallback and query-authorization readers open their split lists
during creation.

Warmup
======
Expand All @@ -106,7 +143,8 @@ compaction as well, but does not cross split boundaries.

For lazy append/raw readers, warmup uses the next file once its construction
has completed; reading the current file never waits for lookahead construction.
The construction window is independent of the warmup level below.
The construction window is configured independently of the warmup level below.
With ``read.preparation-window=1``, no next-file reader is retained for warmup.

How far the next file is prepared is set with
``ReadContextBuilder::SetWarmupLevel()``. Append compaction builds its own read
Expand Down
5 changes: 5 additions & 0 deletions include/paimon/defs.h
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,11 @@ struct PAIMON_EXPORT Options {
/// The default value is 1024.
static const char READ_BATCH_SIZE[];

/// "read.preparation-window" - C++-specific limit on file readers per append/raw split,
/// including readers under construction. Accepts positive 32-bit integers and defaults to 4.
/// Build parallelism, batch prefetch and warmup are configured independently.
static const char READ_PREPARATION_WINDOW[];
Comment thread
SteNicholas marked this conversation as resolved.
Resolved

/// "write.batch-size" - Write batch size for any file format if it supports.
/// The default value is 1024.
static const char WRITE_BATCH_SIZE[];
Expand Down
11 changes: 5 additions & 6 deletions include/paimon/table/source/table_read.h
Original file line number Diff line number Diff line change
Expand Up @@ -48,13 +48,12 @@ class PAIMON_EXPORT TableRead {
/// @return A Result containing a unique pointer to the `TableRead` instance.
static Result<std::unique_ptr<TableRead>> Create(std::unique_ptr<ReadContext> context);

/// Creates a `BatchReader` instance for reading data.
/// The append-only and primary-key implementations open one split at a time in input order.
/// Their readers may outlive this TableRead. Creation rejects null splits and validates
/// real-time views before consuming any tickets. The reader retains those views and returns
/// a terminal error if a split fails to open when reached.
///
/// This method creates a BatchReader that will be responsible for reading data from the
/// provided splits.
///
/// @param splits A vector of shared pointers to `Split` instances representing the
/// data to be read.
/// @param splits The splits to read.
/// @return A Result containing a unique pointer to the `BatchReader` instance.
/// @note `BatchReader`s created by the same `TableRead` are not thread-safe for
/// concurrent reading.
Expand Down
1 change: 1 addition & 0 deletions src/paimon/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -503,6 +503,7 @@ set(PAIMON_CORE_SRCS
core/table/source/snapshot/snapshot_reader.cpp
core/table/source/startup_mode.cpp
core/table/source/table_read.cpp
core/table/source/table_read_utils.cpp
core/table/source/table_scan.cpp
core/table/source/data_evolution_batch_scan.cpp
core/table/source/data_evolution_full_text_read.cpp
Expand Down
1 change: 1 addition & 0 deletions src/paimon/common/defs.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ const char Options::SCAN_MANIFEST_ENTRY_LAZY_DECODE_ENABLED[] =
"scan.manifest-entry.lazy-decode.enabled";
const char Options::PREFETCH_IO_METRICS_ENABLED[] = "prefetch.io-metrics.enabled";
const char Options::READ_BATCH_SIZE[] = "read.batch-size";
const char Options::READ_PREPARATION_WINDOW[] = "read.preparation-window";
const char Options::WRITE_BATCH_SIZE[] = "write.batch-size";
const char Options::WRITE_BUFFER_SIZE[] = "write-buffer-size";
const char Options::WRITE_BUFFER_SPILLABLE[] = "write-buffer-spillable";
Expand Down
46 changes: 35 additions & 11 deletions src/paimon/common/reader/concat_batch_reader.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@

#include <algorithm>
#include <chrono>
#include <exception>
#include <new>
#include <utility>

#include "arrow/c/abi.h"
Expand All @@ -43,22 +45,35 @@ ConcatBatchReader::ConcatBatchReader(std::vector<std::unique_ptr<BatchReader>>&&
}
}

ConcatBatchReader::ConcatBatchReader(size_t reader_count, std::shared_ptr<Preparation> preparation,
size_t preparation_window,
const std::shared_ptr<arrow::MemoryPool>& arrow_pool,
const std::shared_ptr<Executor>& build_executor)
: arrow_pool_(arrow_pool),
finished_reader_metrics_(std::make_shared<MetricsImpl>()),
preparation_(std::move(preparation)),
build_executor_(build_executor),
preparation_window_(preparation_window),
reader_count_(reader_count) {}

ConcatBatchReader::~ConcatBatchReader() {
ConcatBatchReader::Close();
}

Result<std::unique_ptr<ConcatBatchReader>> ConcatBatchReader::CreateLazy(
size_t reader_count, ReaderFactory factory,
size_t reader_count, ReaderFactory factory, int32_t preparation_window,
const std::shared_ptr<arrow::MemoryPool>& arrow_pool,
const std::shared_ptr<Executor>& build_executor) {
if (!factory || !build_executor) {
return Status::Invalid("lazy reader requires a factory and a build executor");
}
auto reader = std::make_unique<ConcatBatchReader>(std::vector<std::unique_ptr<BatchReader>>(),
arrow_pool);
reader->preparation_ = std::make_shared<Preparation>(std::move(factory));
reader->build_executor_ = build_executor;
reader->reader_count_ = reader_count;
if (preparation_window <= 0) {
return Status::Invalid("reader preparation window must be positive");
}
auto preparation = std::make_shared<Preparation>(std::move(factory));
auto reader = std::unique_ptr<ConcatBatchReader>(
new ConcatBatchReader(reader_count, std::move(preparation),
static_cast<size_t>(preparation_window), arrow_pool, build_executor));
reader->FillWindow();
for (auto& state : reader->readers_) {
PAIMON_RETURN_NOT_OK(reader->ResolveReader(&state, /*wait=*/true));
Expand All @@ -67,7 +82,7 @@ Result<std::unique_ptr<ConcatBatchReader>> ConcatBatchReader::CreateLazy(
}

void ConcatBatchReader::FillWindow() {
while (next_reader_ < reader_count_ && readers_.size() < kPreparationWindow) {
while (next_reader_ < reader_count_ && readers_.size() < preparation_window_) {
readers_.emplace_back();
readers_.back().future =
Via(build_executor_.get(),
Expand All @@ -85,15 +100,25 @@ void ConcatBatchReader::FillWindow() {
}

Status ConcatBatchReader::ResolveReader(ReaderState* state, bool wait) {
if (state->future.valid() &&
(wait || state->future.wait_for(std::chrono::seconds(0)) == std::future_status::ready)) {
if (!state->future.valid() ||
(!wait && state->future.wait_for(std::chrono::seconds(0)) != std::future_status::ready)) {
return state->failure;
}
try {
auto result = state->future.get();
if (!result.ok()) {
state->failure = result.status();
} else {
state->reader = std::move(result).value();
state->file_reader = dynamic_cast<FileBatchReader*>(state->reader.get());
}
} catch (const std::bad_alloc&) {
state->failure = Status::OutOfMemory("out of memory preparing file reader");
} catch (const std::exception& e) {
state->failure = Status::UnknownError("file reader preparation failed: ", e.what());
} catch (...) {
state->failure =
Status::UnknownError("file reader preparation failed with an unknown exception");
}
return state->failure;
}
Expand All @@ -117,8 +142,7 @@ void ConcatBatchReader::Close() {
try {
(void)ResolveReader(&state, /*wait=*/true);
} catch (...) {
// Construction exceptions captured by Via must not escape destruction or prevent
// the remaining tasks from being drained.
// Keep draining even if recording a preparation failure runs out of memory.
}
CloseAndReleaseReader(&state);
}
Expand Down
20 changes: 14 additions & 6 deletions src/paimon/common/reader/concat_batch_reader.h
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@

#include <atomic>
#include <cstddef>
#include <cstdint>
#include <deque>
#include <functional>
#include <future>
Expand All @@ -29,32 +30,33 @@
#include <vector>

#include "arrow/api.h"
#include "paimon/common/executor/reader_build_executor.h"
#include "paimon/metrics.h"
#include "paimon/reader/batch_reader.h"
#include "paimon/result.h"
#include "paimon/status.h"

namespace paimon {
class Executor;
class FileBatchReader;
class MemoryPool;

/// This reader is to concatenate a list of BatchReaders and read them sequentially. The input list
/// is already sorted by key and sequence number, and the key intervals do not overlap each other.
/// Reads children in input order and releases each reader at EOF.
class ConcatBatchReader : public BatchReader {
public:
/// Must own the state needed to build the indexed reader beyond its caller's lifetime.
/// Called at most once per index, possibly concurrently. A null reader means a skipped file.
using ReaderFactory = std::function<Result<std::unique_ptr<BatchReader>>(size_t)>;

static constexpr size_t kPreparationWindow = kReaderBuildMaxParallelNum;
static constexpr size_t kDefaultPreparationWindow = 4;

/// Builds the initial window concurrently and refills it as readers reach EOF. Later
/// construction failures are terminal read errors when the failed reader is reached.
/// The positive preparation window bounds retained readers, including in-flight builds,
/// independently of the build executor's parallelism.
static Result<std::unique_ptr<ConcatBatchReader>> CreateLazy(
size_t reader_count, ReaderFactory factory,
size_t reader_count, ReaderFactory factory, int32_t preparation_window,
const std::shared_ptr<arrow::MemoryPool>& arrow_pool,
const std::shared_ptr<Executor>& build_executor = GetReaderBuildExecutor());
const std::shared_ptr<Executor>& build_executor);

ConcatBatchReader(std::vector<std::unique_ptr<BatchReader>>&& readers,
const std::shared_ptr<arrow::MemoryPool>& arrow_pool);
Expand Down Expand Up @@ -84,6 +86,11 @@ class ConcatBatchReader : public BatchReader {
std::atomic<bool> cancelled = false;
};

ConcatBatchReader(size_t reader_count, std::shared_ptr<Preparation> preparation,
size_t preparation_window,
const std::shared_ptr<arrow::MemoryPool>& arrow_pool,
const std::shared_ptr<Executor>& build_executor);

void FillWindow();
Status ResolveReader(ReaderState* state, bool wait);
void WarmupRange();
Expand All @@ -94,6 +101,7 @@ class ConcatBatchReader : public BatchReader {
std::deque<ReaderState> readers_;
std::shared_ptr<Preparation> preparation_;
std::shared_ptr<Executor> build_executor_;
size_t preparation_window_ = 0;
size_t reader_count_ = 0;
size_t next_reader_ = 0;
Status failure_;
Expand Down
Loading
Loading