Repository navigation
fix(parquet): preserve application Arrow CPU capacity - #574
SteNicholas wants to merge 1 commit into
Conversation
Parquet readers resized Arrow's global CPU pool during construction, overriding application and spill settings and making concurrent readers depend on construction order. - Remove per-reader CPU pool resizing and add parquet.read.use-threads to control column parallelism, addressing apache#562. - Retain legacy thread-count values as a zero/nonzero fallback, with parquet.read.use-threads taking precedence. Validate both options; positive legacy values no longer set global capacity. - Document application ownership of CPU capacity, explicit spill configuration and the independent prefetch executor. - Add concurrent-reader benchmarks for throughput, latency and process CPU time.
2e5ab73 to
f09030a
Compare
|
In the current shared-library build with bundled Arrow, this setting only changes the Parquet plugin’s internal Arrow CPU pool, not the application’s or core spill’s pool. Could you share the concrete issue you encountered that motivates this change? |
|
Thanks @lxy-9602 for pointing this out. You're right for the Linux shared-library build with bundled Arrow: core and the Parquet plugin each link Arrow with its symbols hidden, so this setter changes the plugin's internal CPU pool, not the application's or core spill's pool. My description of the effect as unconditionally process-wide was too broad. To answer the motivation question directly, this proposal came from source inspection, as noted in #562. I don't have a production failure or measured performance regression to cite here. The concrete behavior motivating it is within the same Parquet plugin: constructing reader A with a thread count of 6 sets the shared plugin pool to 6, then constructing reader B with default options resets it to 3 while A may still be reading. Thus, a per-reader option changes the effective pool capacity for other readers, depending on construction order. This is a configuration side effect; it does not by itself establish a data race or performance regression. I also checked the test setup: changing the test dependency from |
|
We generally set the thread count via query options. Although the readers overwrite the plugin pool capacity, they still use the same thread count in our current setup, so we haven’t seen issues with this behavior. The current pr modifies the default CPU capacity and could impact query performance—especially for wide schemas with thousands of columns. Without a production problem driving this change, I’d prefer to keep the current behavior for now. Alternatively, we could preserve the existing default behavior and introduce an option to disable pool resizing. What do you think, @lucasfang? |
I agree that we should preserve the current default behavior. Since Arrow symbols are hidden within the plugin, arrow::SetCpuThreadPoolCapacity(executor_thread_count) only affects the plugin’s internal Arrow thread pool. This PR effectively switches to Arrow’s default thread count, which could change query performance for existing workloads, especially those with thousands of columns. Without a concrete production issue motivating that change, I’d prefer to keep the existing behavior. |
Purpose
Linked issue: close #562
Parquet readers resized Arrow's global CPU pool during construction, including when using the
default options. This overwrote capacity configured by the embedding application or spill
setup and made readers with different settings depend on construction order. Parallel
prefetch construction repeated the same process-wide change for each sub-reader.
The change separates reader column parallelism from ownership of the shared CPU pool:
ParquetFileBatchReader: removes global CPU-pool resizing from reader construction andadds
parquet.read.use-threadsto control parallel column decoding. Threaded readers usethe existing Arrow CPU pool; creating, reading and closing readers preserve its capacity.
parquet.read.executor.thread-countas azero/nonzero fallback when the boolean option is absent. Zero disables column threading;
positive values enable it without setting global capacity. The boolean option takes
precedence, and both supplied options are validated, including an overridden legacy value.
positive spill-thread setting remains an explicit process-wide setter applied by
WriteContextBuilder::Finish(); subsequent Parquet readers preserve that capacity.Paimon's outer prefetch executor remains independent of Arrow's column-decoding pool.
nonzero and mixed legacy settings. Counters cover aggregate throughput, mean reader
open/scan/close latency and effective CPU capacity, alongside process CPU time.
Arrow 17's synchronous batch-reader API does not accept a scoped CPU executor. The change
retains that read path and its decoding behavior. The default still enables column threading,
but no longer forces a global capacity of three; applications needing a fixed CPU budget
configure the shared pool before starting concurrent work.
Tests
UT:
src/paimon/format/parquet/parquet_file_batch_reader_test.cpp:TestCreateArrowReaderProperties: consolidates default, zero, positive and explicitboolean settings into table-driven cases. It checks option precedence, unsigned 32-bit
boundaries, invalid values and unchanged application CPU capacity while retaining the
existing batch-size, pre-buffer and cache-option assertions.
TestConcurrentReadersPreserveGlobalCpuPool: constructs readers with different settingsconcurrently, reads them alongside Arrow IPC spill write/read operations, and checks
returned data and capacity through reader close and destruction. It covers both disabled
and enabled spill threading and restores the original global capacity.
TestPrefetchReadersPreserveGlobalCpuPool: exercises prefetch sub-reader constructionand decoded warmup with one and four outer workers, read-ahead enabled and disabled, and
default, legacy and boolean options. It checks complete output and capacity preservation
with a single-worker Arrow CPU pool, including after reader destruction.
src/paimon/format/parquet/CMakeLists.txt: links the test target to static Paimon sospill configuration, Parquet readers and capacity assertions use the same Arrow CPU pool.
IT:
No standalone integration tests are added. The reader tests exercise real Parquet files
through
ParquetReaderBuilderandPrefetchFileBatchReaderImpl, and real Arrow IPC spillfiles using the threading policy from
WriteContextBuilder.API and Format
No public API signatures, stored file formats or table protocols change.
include/paimon/write_context.hclarifies the existing spill setter's process-wide scope.The new
parquet.read.use-threadsoption defaults to enabled. The deprecated numeric optioncontinues to control whether column threading is enabled when the boolean option is absent,
but positive values no longer configure a thread count. Applications that relied on those
values to size the global pool must move capacity configuration to application startup.
Documentation
docs/source/user_guide/read.rstdocuments application ownership of Arrow CPU capacity,legacy-option migration, spill interaction, the independent prefetch executor and the
constraint against submitting synchronous threaded reads to the same saturated Arrow pool.
docs/source/examples/benchmark.rstdocuments the concurrent-reader policies, countersand before/after methodology. It calls for separate processes per comparison case and
distinguishes a change in effective CPU budget from a fixed-budget speedup.
Generative AI tooling
Generated-by: Codex (GPT-6)