Skip to content

fix(parquet): preserve application Arrow CPU capacity - #574

Closed
SteNicholas wants to merge 1 commit into
apache:mainfrom
SteNicholas:PAIMON-562
Closed

SteNicholas wants to merge 1 commit into
apache:mainfrom
SteNicholas:PAIMON-562

Conversation

@SteNicholas

@SteNicholas SteNicholas commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Purpose

Linked issue: close #562

Parquet readers resized Arrow's global CPU pool during construction, including when using the
default options. This overwrote capacity configured by the embedding application or spill
setup and made readers with different settings depend on construction order. Parallel
prefetch construction repeated the same process-wide change for each sub-reader.

The change separates reader column parallelism from ownership of the shared CPU pool:

  • ParquetFileBatchReader: removes global CPU-pool resizing from reader construction and
    adds parquet.read.use-threads to control parallel column decoding. Threaded readers use
    the existing Arrow CPU pool; creating, reading and closing readers preserve its capacity.
  • Legacy option compatibility: deprecates parquet.read.executor.thread-count as a
    zero/nonzero fallback when the boolean option is absent. Zero disables column threading;
    positive values enable it without setting global capacity. The boolean option takes
    precedence, and both supplied options are validated, including an overridden legacy value.
  • CPU capacity ownership: leaves capacity configuration to the application. The existing
    positive spill-thread setting remains an explicit process-wide setter applied by
    WriteContextBuilder::Finish(); subsequent Parquet readers preserve that capacity.
    Paimon's outer prefetch executor remains independent of Arrow's column-decoding pool.
  • Concurrent-reader benchmarks: compare one and four readers using default, zero,
    nonzero and mixed legacy settings. Counters cover aggregate throughput, mean reader
    open/scan/close latency and effective CPU capacity, alongside process CPU time.

Arrow 17's synchronous batch-reader API does not accept a scoped CPU executor. The change
retains that read path and its decoding behavior. The default still enables column threading,
but no longer forces a global capacity of three; applications needing a fixed CPU budget
configure the shared pool before starting concurrent work.

Tests

UT:

  • src/paimon/format/parquet/parquet_file_batch_reader_test.cpp:

    • TestCreateArrowReaderProperties: consolidates default, zero, positive and explicit
      boolean settings into table-driven cases. It checks option precedence, unsigned 32-bit
      boundaries, invalid values and unchanged application CPU capacity while retaining the
      existing batch-size, pre-buffer and cache-option assertions.
    • TestConcurrentReadersPreserveGlobalCpuPool: constructs readers with different settings
      concurrently, reads them alongside Arrow IPC spill write/read operations, and checks
      returned data and capacity through reader close and destruction. It covers both disabled
      and enabled spill threading and restores the original global capacity.
    • TestPrefetchReadersPreserveGlobalCpuPool: exercises prefetch sub-reader construction
      and decoded warmup with one and four outer workers, read-ahead enabled and disabled, and
      default, legacy and boolean options. It checks complete output and capacity preservation
      with a single-worker Arrow CPU pool, including after reader destruction.
  • src/paimon/format/parquet/CMakeLists.txt: links the test target to static Paimon so
    spill configuration, Parquet readers and capacity assertions use the same Arrow CPU pool.

IT:

No standalone integration tests are added. The reader tests exercise real Parquet files
through ParquetReaderBuilder and PrefetchFileBatchReaderImpl, and real Arrow IPC spill
files using the threading policy from WriteContextBuilder.

API and Format

No public API signatures, stored file formats or table protocols change.
include/paimon/write_context.h clarifies the existing spill setter's process-wide scope.

The new parquet.read.use-threads option defaults to enabled. The deprecated numeric option
continues to control whether column threading is enabled when the boolean option is absent,
but positive values no longer configure a thread count. Applications that relied on those
values to size the global pool must move capacity configuration to application startup.

Documentation

docs/source/user_guide/read.rst documents application ownership of Arrow CPU capacity,
legacy-option migration, spill interaction, the independent prefetch executor and the
constraint against submitting synchronous threaded reads to the same saturated Arrow pool.
docs/source/examples/benchmark.rst documents the concurrent-reader policies, counters
and before/after methodology. It calls for separate processes per comparison case and
distinguishes a change in effective CPU budget from a fixed-budget speedup.

Generative AI tooling

Generated-by: Codex (GPT-6)

Parquet readers resized Arrow's global CPU pool during construction,
overriding application and spill settings and making concurrent readers
depend on construction order.

- Remove per-reader CPU pool resizing and add parquet.read.use-threads
  to control column parallelism, addressing apache#562.
- Retain legacy thread-count values as a zero/nonzero fallback, with
  parquet.read.use-threads taking precedence. Validate both options;
  positive legacy values no longer set global capacity.
- Document application ownership of CPU capacity, explicit spill
  configuration and the independent prefetch executor.
- Add concurrent-reader benchmarks for throughput, latency and process
  CPU time.
@lxy-9602

Copy link
Copy Markdown
Member

In the current shared-library build with bundled Arrow, this setting only changes the Parquet plugin’s internal Arrow CPU pool, not the application’s or core spill’s pool. Could you share the concrete issue you encountered that motivates this change?

@SteNicholas

Copy link
Copy Markdown
Member Author

Thanks @lxy-9602 for pointing this out. You're right for the Linux shared-library build with bundled Arrow: core and the Parquet plugin each link Arrow with its symbols hidden, so this setter changes the plugin's internal CPU pool, not the application's or core spill's pool. My description of the effect as unconditionally process-wide was too broad.

To answer the motivation question directly, this proposal came from source inspection, as noted in #562. I don't have a production failure or measured performance regression to cite here.

The concrete behavior motivating it is within the same Parquet plugin: constructing reader A with a thread count of 6 sets the shared plugin pool to 6, then constructing reader B with default options resets it to 3 while A may still be reading. Thus, a per-reader option changes the effective pool capacity for other readers, depending on construction order. This is a configuration side effect; it does not by itself establish a data race or performance regression.

I also checked the test setup: changing the test dependency from paimon_shared to paimon_static makes the spill and Parquet checks use the same Arrow instance. Those tests cover that linkage arrangement, but do not substantiate the cross-component interference claim for the bundled shared-library build. The PR's motivation and documentation need to distinguish these configurations, and validation should cover the shared-library setup, including how its isolated plugin pool is configured once the legacy numeric option stops setting capacity.

@lxy-9602

Copy link
Copy Markdown
Member

We generally set the thread count via query options. Although the readers overwrite the plugin pool capacity, they still use the same thread count in our current setup, so we haven’t seen issues with this behavior.

The current pr modifies the default CPU capacity and could impact query performance—especially for wide schemas with thousands of columns. Without a production problem driving this change, I’d prefer to keep the current behavior for now. Alternatively, we could preserve the existing default behavior and introduce an option to disable pool resizing. What do you think, @lucasfang?

@lucasfang

Copy link
Copy Markdown
Member

We generally set the thread count via query options. Although the readers overwrite the plugin pool capacity, they still use the same thread count in our current setup, so we haven’t seen issues with this behavior.

The current pr modifies the default CPU capacity and could impact query performance—especially for wide schemas with thousands of columns. Without a production problem driving this change, I’d prefer to keep the current behavior for now. Alternatively, we could preserve the existing default behavior and introduce an option to disable pool resizing. What do you think, @lucasfang?

I agree that we should preserve the current default behavior. Since Arrow symbols are hidden within the plugin, arrow::SetCpuThreadPoolCapacity(executor_thread_count) only affects the plugin’s internal Arrow thread pool.

This PR effectively switches to Arrow’s default thread count, which could change query performance for existing workloads, especially those with thousands of columns. Without a concrete production issue motivating that change, I’d prefer to keep the existing behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Avoid reconfiguring Arrow's global CPU thread pool per Parquet reader

3 participants