[924060929]

What problem does this PR solve?

Issue Number: None

Problem Summary: External scans can monopolize shared FIFO workers, block on metadata submitted to the same pool, repeat path/backend-assignment work, and load cold Iceberg manifests serially. This change uses query-owned bounded task groups, separate split and metadata workers, cooperative continuations and bounded completion-order manifest prefetch. Early stop and exceptions close query-owned tasks and resources, including planning failures before a scan can register its finish callback. Path reuse, bounded consistent-hash candidate lookup and compatible range compaction stay with scheduling to avoid moving the bottleneck into scan bookkeeping. File-format selection and snapshot-specific partition type adaptation preserve per-file semantics without duplicate SDK planning or changes to shared cache values.

Each backend's bounded Split channel wakes on data, completion, cancellation and failure, eliminating the empty terminal request's previous 100 ms polling delay. An earlier four-query JFR reproduction found 78 delayed EOF requests in 400 SQL executions; the fixed path has none in the subsequent replay.

Initial RPC delivery is separate from background lookahead: split_initial_batch_size (default 64) limits the first response per backend, additionally capped by its request size. Further responses use the BE request size. Producers continue before BE consumption, limited by the configured queues and cooperative fairness. The response bound counts real splits before transport-envelope compaction. There is no producer-side wait for a first BE request.

The query-level enable_external_table_batch_mode switch is checked at the engine entry before either partition-batch capability checks or file-streaming estimates. Setting it to false consistently selects synchronous planning; previously the partition-batch path could bypass the switch. Provider implementations still determine whether asynchronous planning is supported. Partition/file thresholds and producer/queue defaults are unchanged.

Scan-local partition adaptation preserves delete matching after numeric promotion and source-schema cleanup, even when data and delete Manifest entries have different cache warmth. UNKNOWN partition targets use consistent numeric Java representations before delete-key comparison, without rewriting shared cache entries. Regression tests exercise both cache orders, identity/truncate partitions, concurrent historical/current projections and different-partition exclusion.

Split channels and their capacity waiters are indexed by backend ID. This avoids repeatedly hashing and boxing a Backend's endpoint/health fields for each producer batch, which was visible in CPU/alloc flamegraphs at split_batch_size=1. The immutable ID also keeps one bounded channel when a backend handle's metadata changes. Batch generation size, FIFO delivery, backpressure, wakeup ordering, cancellation and fair scheduling are retained. A focused queue/lifecycle suite has 63 passing tests, including three-backend streaming and the same-ID/different-handle capacity-wakeup boundary.

Main benefits and independent SQL measurements

This PR addresses two different concerns: bounded scheduling and query lifecycle correctness, and overlapping cold metadata I/O. Query groups rotate ready work and bound queued tasks; separate SPLIT and METADATA workers avoid parent/child pool starvation; cancellation prevents queued work and cache-to-SDK fallback from restarting a finished query. These are scheduling/resource guarantees, not a claim that every SQL statement becomes faster.

The clearest standalone latency gain is Iceberg cold full scans: approximately 32–34% less complete-SQL time. Hive already has partition-parallel listing, so warm Hive single-query latency remains approximately flat. With the same Linux AArch64 Release BE, one backend, six CPUs, warm BE footer/data cache, split_batch_size=1 and split_queue_capacity=4096:

Single-query complete SQL median, ms	Unoptimized master baseline runs	This PR alone	Interpretation
Hive, 1,000 partitions, warm LIMIT 10	8.60–9.28	8.81	Approximately flat
Hive, 10,000 partitions, warm full scan	752.84–783.66	748.83	Small difference; no large speedup claimed
Iceberg, 1,000 manifests, cold LIMIT 10	925.44–933.77	920.35	Approximately flat; partition-view work remains
Iceberg, 1,000 manifests, cold full scan	2109.54–2168.54	1425.94	32–34% less elapsed time

Baseline: frozen unoptimized master e0bf4874070. The standalone measurement package records 9b6d902ffa1 (built source d61d21b943a before its equivalent EOF-fix amendment). Each single-query cell has 40 measured samples across forward/reverse runs after prewarm. Cold means FE catalog/Manifest invalidation, not cold JVM or BE data. These historical measurements predate the subsequent initial-delivery refinements, rebase and review repairs; they establish the standalone implementation's measured benefit and are not a fresh A/B of the current head. The total PR gain includes Manifest prefetch and scan-path improvements; it does not isolate the fairness mechanism itself.

The focused initial-delivery experiments below compare two already-optimized FE versions. For example, Hive four-query LIMIT group median improves 15.38 → 11.82 ms (23.1%), while the controlled slow-listing/delayed-BE full scan stays 388.52 → 386.96 ms. These incremental results must not be added to, or substituted for, the standalone master comparison above.

Ready split batches are drained together within each BE request, capped at 64 producer batches and the number that fits the remaining request. This amortizes queue operations and per-batch channel checks without waiting to fill a response or changing producer buffering. Capacity callbacks execute after the queue lock is released; callback-published errors and cancellation are checked before returning the drained data. The scalar path still handles empty queues and partial batches.

A four-phase fixed-Release-BE comparison on a warm 10,000-file Iceberg table narrows single-query batch-one overhead against each phase's batch128 control from 5.8–6.4% to 0.7–2.8%. Four-query full scans show no stable change. The initial four-query LIMIT difference of approximately 1 ms does not reproduce in a separate 3,200-SQL reverse-order repetition; sub-millisecond differences remain noisy. CPU samples inside getNextBatch are essentially unchanged (392 versus 389), while allocation samples fall from 444 to 174, mostly boxing and iterators. These are statistical samples, not exact invocation/byte counts. Across both formal comparisons, 4,800 SQL statements pass result checks; separate cache-hit profiles, Hive correctness and Iceberg cold LIMIT startup checks also pass. Both isolated PR and integration worktrees pass 66 focused unit tests and full build.sh --fe including Checkstyle. This comparison changes only SplitSource.class; plugins and the Release BE are identical.

Release note

External Hive and Iceberg scans support bounded fair task scheduling, query cancellation and concurrent cold Manifest loading. The new split_initial_batch_size session setting controls first-response size independently of split_batch_size and split_queue_capacity. File formats, pinned snapshots and full-scan results are preserved.

Check List (For Author)

Current-head repair validation at 28fb25d01bd: StmtExecutorTest and IcebergManifestReadContextTest passed (69 tests); build.sh --fe -j4 and Checkstyle passed. Earlier review-repair validation at 17859e3ff14: 1,521 Iceberg package tests (zero failures/errors, one environment-dependent skip), two query-lifecycle tests, and build.sh --fe -j4 including Checkstyle passed. Coverage includes cancellation before first scheduler acquisition, all four cache-to-SDK fallback paths, shared query read admission and identity/truncate data/delete manifests after source-schema cleanup. Previously reported live SQL/regression and performance experiments are historical and were not rerun after every review repair. CI status remains tracked by the checks on this PR.

• Test: Clean build.sh --fe, including Checkstyle; 750 scoped FE/SPI/Hive/Iceberg tests in 53 classes, zero failures/errors/skips. Updated Hive streaming regression and Iceberg scheduled-split regression both passed without skips. Matched Release-BE experiment: 1,200 four-query LIMIT SQL and 150 full scans, all result checked. Separate 400-query RPC/EOF diagnostic; footer/page-cache profile verification.
• Behavior changed: Yes, external scan scheduling, bounded metadata prefetch, transport compaction and initial delivery size.
• Does this need documentation: Yes, developer scheduling documentation included.

Supporting experiments: initial delivery, full-scan overlap and RPC diagnostics

Focused initial-delivery experiment

Same Linux AArch64 Release BE, one backend, six engine CPUs, warm BE data/footer cache; num_partitions_in_batch_mode=1, split_batch_size=1, split_queue_capacity=4096. Hive fixture: 1,000 partitions, one file per partition. The reference FE already includes the other scheduling/scan changes. It is the exact FE before the initial-delivery change, not unoptimized master or SR.

Each fresh FE follows identical 45-second full-scan/LIMIT prewarm. Four-query LIMIT results have 100 groups / 400 SQL per version, without agents, profilers, builds or unit tests during timing. Every LIMIT length and full COUNT/SUM result is checked. Full scans use 10 measured SQL per cell; cold cells run forward/reverse order.

Complete SQL milliseconds	Before initial-delivery change	Independent initial delivery
Four-query LIMIT, group median	15.38	11.82
Four-query LIMIT, group mean	16.57	11.94
Four-query LIMIT, group p95	26.94	14.10
Four-query LIMIT, per-SQL median	13.95	11.15
Warm full scan	57.57	57.24
Cold directory listing, local S3	104.68	101.17
Cold directory listing, first fetch delayed 300 ms	379.52	377.03
Cold listing with additional 20 ms/request, no fetch delay	344.54	345.52
Cold listing with additional 20 ms/request and 300 ms first-fetch delay	388.52	386.96

The cold-listing catalog disables only the file-list cache over the same table, with warmed partition metadata and BE data/footer caches. First-fetch delay is injected at SplitSource.getNextBatch entry before demand callbacks; the slower-listing case injects 20 ms before each real directory request. These controlled delay-injection measurements include the sleeps and do not represent ordinary production latencies or all resource effects of a slow BE startup.

A rejected producer startup barrier had the same LIMIT group median (11.85 ms), but the slower-listing/delayed-fetch full scan took 651.26 ms. Independent initial delivery restores this to 386.96 ms. During the same 300 ms delay, both the pre-change reference and final implementation complete another 896 directory listings (median); the barrier completed zero. The final source contains no startup barrier. The barrier's per-SQL LIMIT median was 10.77 ms versus 11.15 ms now; group mean, p95 and maximum all improve. Both distributions are retained in validation rather than claiming uniform per-SQL improvement.

Separate diagnostic replay of 400 LIMIT SQL has 400 Split RPCs: 399 first responses contain 64 real file ranges and one contains 25. Every first response satisfies the bound, with no extra RPC needed in these LIMIT cases and no delayed EOF request. Diagnostic query timings are excluded from the table. A full-scan profile records 1,000 footer hits / zero footer reads and 1,000 page hits / zero page misses.

Backend candidate lookup was independently measured with production ConsistentHash/Backend classes, 256 virtual nodes, JDK17/macOS ARM64: 0.21–0.26 microseconds at 2–128 backends, including four threads. That is a function benchmark, not multi-backend complete SQL. No SR performance matrix was rerun. The subsequent Iceberg matrix is recorded below with its own fixed BE identity.

Iceberg first-delivery follow-up

Matched previous/current FE (3b97eee547c / 3e94facccfb), keeping the same Linux AArch64 Release BE throughout the timed runs (SHA256 prefix 26a52681f1ed). Only initial-delivery behavior differs between these FE variants; the rest of the implementation is held fixed. This BE differs from the one used in the Hive table above, so absolute values should not be mixed between the tables.

Doris Manifest Cache enabled; SDK manifest cache disabled. Both use producer batch1, queue4096, manifest concurrency8, pruningfalse. Cold means catalog/manifest invalidation with hot BE data/footer caches and warm JVM. Before each cold four-query group, invalidate exactly once, outside timing; all four then run concurrently. Previous groups' background tasks drain first. Every COUNT/SUM and LIMIT length is checked. Single-query40 samples/cell; four-query30 groups/cell, forward/reverse order.2560 measured SQL.

Manifests	Cache	SQL	Single before ms	Single after ms	4-query GROUP before ms	4-query GROUP after ms
100	warm	limit	8.36	6.65	15.52	11.41
100	cold	limit	16.77	16.40	17.67	17.72
100	warm	full	78.31	81.03	192.53	195.17
100	cold	full	111.02	107.41	219.79	222.71
1000	warm	limit	8.17	6.78	13.96	9.78
1000	cold	limit	15.52	15.91	16.62	16.73
1000	warm	full	79.51	78.68	190.70	196.50
1000	cold	full	244.96	241.17	292.82	301.91

Warm LIMIT improves17–20% single-query and26–30% group completion. Cold LIMIT is near-flat. Some full-scan points increase1–3.5%, which are disclosed rather than calling all cases faster.

An additional same-JVM parameter-only ABBA comparison (cap64 versus unlimited,20 samples/groups per mode/cell;1600 SQL) shows full scans mostly within-1.2% to+2.1%. Its1000-manifest cold single full scan had one13.6% median increase; a focused random-order repetition of50 samples/mode did not reproduce it (unlimited223.60ms versus64's223.25ms). The median-difference bootstrap95% interval is-12.90 to+8.28ms. These runs do not establish zero overhead or equal tails; they did not reproduce a consistent large full-scan regression. No product code change is made from the small mixed differences.

Separate diagnostics directly count actual Manifest-cache loader executions: cold100/1000, warm0/0. Both full-scan profiles show10000 footer hits/zero footer reads and10000 page hits/zero misses. No agent, profiler, build or UT runs during the4260 measured SQL. Supplementary RPC diagnostics are excluded from the timing claims.

Focused multi-backend small-batch follow-up

Balanced assignments now consult the existing weight-count index before constructing redistribution heaps. Successful offers and consumers with no capacity waiter bypass the query-wide capacity monitor. On a failed offer, callback registration precedes the retry under that monitor, preserving wakeups when capacity is released before registration. Listener-set mutations and error fanout remain protected, and callbacks execute outside the monitor.

A four-phase previous/current FE experiment holds the remaining code fixed and alternates reference/current/current/reference packages. The fixed Linux AArch64 Release BE uses three backends, two configured cores each, a shared six-CPU budget, 4 GiB per BE and footer-cache capacity 20000. Both variants use partition/file thresholds 1, initial response 64, producer batch1 and queue4096. All 3200 measured SQL results pass; each table cell has 80 queries/groups. Separate profiles verify zero footer reads and zero page misses on every backend. No profiler, agent, build or unit test overlaps timing.

Complete SQL median, ms	Previous implementation	Current implementation
Hive 10000 files, single LIMIT 10	5.464	5.131
Hive 10000 files, four-query LIMIT group	11.815	11.155
Hive 10000 files, single COUNT/SUM	55.577	55.486
Hive 10000 files, four-query COUNT/SUM group	159.172	160.824

The demonstrated benefit is reduced FE resource/lock cost: equal-workload sampling of 240 full scans reduces active CPU samples from 6385 to 4373 (31.5%) and allocation samples from 39174 to 25396 (35.2%). Recorded accumulated waits at capacity notification fall from 295.112 ms to zero above the 10-us recording threshold. Samples and accumulated parallel waits are not complete-SQL time. Full scans remain approximately flat, with the four-query median increasing 1.0%; this experiment does not establish an end-to-end full-scan speedup. The current same-version batch128/queue32 control takes 152.504 ms for four-query full scans, leaving 5.5% batch1 overhead. LIMIT is preserved. Batch128 LIMIT controls also improve in this experiment, so the full observed LIMIT difference should not be attributed solely to batch1 handling.

Both isolated PR and integration worktrees pass 120 focused policy/split/concurrency tests and full FE build/Checkstyle. Added tests cover redistribution across cumulative weight thresholds, monitor-independent uncontended paths and deterministic capacity release before waiter registration. Source and modified-class identities match between the independently built worktrees. Configuration defaults remain unchanged.

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[924060929]

run buildall

[924060929]

/review

[github-actions]

Reviewed #69298 at 7de569daeb9521790075380d85a36e81968bf400 against the supplied authoritative aggregate diff. All 77 changed paths were covered by the main review and two reviewers with complementary scopes; an additional reviewer checked four cross-boundary risks.

The scheduling and scan hot-path changes broadly implement their stated goal. Four P2 issues are reported inline: retained manifests fail after source-schema cleanup, recovered schema names can collide with partition names, detached prefetch can outlive its FileIO owner, and one existing producer test retains malformed scan envelopes. No existing inline threads or independently confirmed existing P0/P1 comments were present. No additional user focus was supplied.

The invocation chain reviewed was:

statement -> captured query groups -> SPLIT producer -> Hive/Iceberg source
                                           |             |
                                           |       METADATA workers
                                           v             |
                                  assignment + BE queues <-+
                                           |
                                  SplitSource / compaction
                                           |
                                  BE descriptor iterator -> per-file reader
finish/cancel -> stop groups/producers -> statement/read-lease cleanup

Applicable code-review skill checkpoints:

Checkpoint	Conclusion
Goal and proof	Fair task groups, separate metadata workers, bounded completion-order prefetch and smaller transport envelopes are implemented. Checked-in tests cover their main behavior; the four reported cases need fixes.
Scope and abstraction	Broad but related changes span scheduling, lifetime and scan transport. Shared helpers and SPI keep generic planning connector-agnostic.
Concurrency and protected state	Worker, planner, cancellation and RPC threads use captured sessions, atomics/concurrent queues, group admission locks, producer scheduleLock and assignment/readiness/consumer locks. The detached-load ownership issue is reported inline.
Work under locks	External reads, future waits and user callbacks are outside registry locks; backend-policy mutation remains serialized separately from readiness publication.
Lock order and deadlock	Reviewed admission/listener/queue/cleanup paths showed no additional concrete lock cycle.
Lifecycle and initialization	Captured groups reject resurrection, queued work cancels, and source cleanup is cooperative. Running metadata callables still need ownership through actual exit. No C++ static-initialization change.
Configuration	Three FE worker/capacity settings are restart-scoped and captured at creation. The forwarded session manifest concurrency is validated to 1..64 and shared across query scans.
Symbol/protocol/storage compatibility	Connector API major advances 18 to 19 with surface/version baselines; no new persisted format. Existing BE readers support nested range descriptors.
Thrift/protobuf definitions	None changed. Existing optional per-range fields and query-option defaults were checked at consumption sites.
Parallel paths	Both Coordinator and ThriftPlansBuilder compact transport copies; streaming enforces the original split count before compaction. Native/JNI and row-id paths consume per-range formats.
Special conditions	Backpressure/cancellation conditions have concrete triggers and comments. URI-prefix reuse and backend candidate clamping preserve reviewed semantics. Dropped-source recovery has the two inline defects.
Test coverage	Positive/negative, cancellation, failure, queue-capacity, single-worker, race, serialization, path and schema-promotion cases exist. The stale producer fixture introduces an intermittent test failure.
Changed expected results	Hive fixture totals and Iceberg count/sum/union/limit outputs are consistent by inspection. Output-generation provenance was not independently reproduced.
Observability	Fair-executor pool/active/queue/group/completion metrics and existing generation/cache profiles and exception context cover the new paths. No separate proven gap found.
Transactions, persistence and failover	Read planning and write-format inference are changed; no EditLog, committed-data visibility, rowset version or MoW delete-bitmap protocol change. Read resource ownership was traced through statement cleanup.
Writes and atomicity	No new data-write or commit protocol. Write-side ORC inference remains; no additional atomicity defect identified.
FE-to-BE values	Formats, credentials, partitions and connector descriptors remain in existing fields and survive both transport paths.
Performance and memory	Producer work, task queues, prefetch results and query read permits are bounded. Warm-cache reads avoid futures; path reuse and envelope compaction avoid repeated wrapper work. Scanner concurrency is preserved by the wrapper floor.
Other issues and final recheck	Retry/Flight/planning-only cleanup, continuation admission, schema evolution/delete matching, mixed formats/JNI and test fixtures were rechecked. Every candidate and initial risk has an explicit ledger disposition.

Validation was static only, as required: no builds, tests, cluster execution or source edits. Author-reported performance and test results were not independently rerun. There are no unresolved candidates or uncovered changed paths within this static scope. The review converged after 2 rounds: every assigned reviewer returned NO_NEW_VALUABLE_FINDINGS on the same frozen set.

Review converged after 2 round(s).

[hello-stephen]

TPC-H: Total hot run time: 28431 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7de569daeb9521790075380d85a36e81968bf400, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17584	5241	5149	5149
q2	2213	296	248	248
q3	10180	1381	820	820
q4	4678	463	345	345
q5	7455	794	522	522
q6	182	188	154	154
q7	753	805	604	604
q8	9245	1333	1066	1066
q9	5394	4164	4180	4164
q10	6831	1281	1024	1024
q11	451	249	228	228
q12	643	400	280	280
q13	18075	3576	3022	3022
q14	272	272	249	249
q15	q16	731	706	653	653
q17	1193	746	586	586
q18	6980	6143	6085	6085
q19	1101	958	613	613
q20	377	327	226	226
q21	4989	2137	2323	2137
q22	353	297	256	256
Total cold run time: 99680 ms
Total hot run time: 28431 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5731	5695	5673	5673
q2	723	606	585	585
q3	5216	5343	4910	4910
q4	2181	2295	1772	1772
q5	5229	5157	5710	5157
q6	258	193	158	158
q7	2344	2072	1878	1878
q8	3094	2766	2745	2745
q9	7872	7970	7798	7798
q10	3734	3681	3291	3291
q11	628	590	455	455
q12	735	780	556	556
q13	3553	4069	3236	3236
q14	308	314	304	304
q15	q16	709	719	670	670
q17	8407	7630	7085	7085
q18	13092	12318	13092	12318
q19	955	830	831	830
q20	2195	2170	1974	1974
q21	5959	5052	5203	5052
q22	512	488	420	420
Total cold run time: 73435 ms
Total hot run time: 66867 ms

[hello-stephen]

TPC-DS: Total hot run time: 150261 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7de569daeb9521790075380d85a36e81968bf400, data reload: false

query5	4302	583	425	425
query6	423	208	193	193
query7	4799	448	238	238
query8	312	160	144	144
query9	8822	3092	3072	3072
query10	473	300	256	256
query11	5926	3855	3546	3546
query12	144	100	86	86
query13	1249	430	316	316
query14	6042	3929	3647	3647
query14_1	3408	3396	3403	3396
query15	213	192	176	176
query16	944	439	402	402
query17	845	697	524	524
query18	2428	409	293	293
query19	185	154	136	136
query20	81	81	81	81
query21	208	120	100	100
query22	13037	12997	12818	12818
query23	13388	12744	12414	12414
query23_1	12587	12575	12418	12418
query24	7399	837	455	455
query24_1	463	471	474	471
query25	508	401	362	362
query26	1260	248	135	135
query27	2801	429	267	267
query28	4621	1923	1918	1918
query29	1576	588	457	457
query30	305	215	181	181
query31	1066	908	785	785
query32	147	88	92	88
query33	514	282	234	234
query34	907	835	496	496
query35	742	779	652	652
query36	849	807	706	706
query37	119	91	86	86
query38	1801	1702	1670	1670
query39	788	798	717	717
query39_1	725	726	735	726
query40	214	110	96	96
query41	65	67	95	67
query42	88	82	82	82
query43	338	360	304	304
query44	1283	670	681	670
query45	176	177	158	158
query46	820	961	551	551
query47	3034	2912	2943	2912
query48	293	296	215	215
query49	575	395	294	294
query50	671	273	210	210
query51	10640	10416	10440	10416
query52	79	80	69	69
query53	187	207	154	154
query54	265	200	173	173
query55	75	68	66	66
query56	226	208	213	208
query57	1646	1630	1567	1567
query58	280	259	253	253
query59	2063	2088	1914	1914
query60	290	223	220	220
query61	162	140	141	140
query62	393	327	282	282
query63	201	153	154	153
query64	2735	908	821	821
query65	3302	3262	3252	3252
query66	1801	436	323	323
query67	20101	20086	19917	19917
query68	2988	1008	622	622
query69	417	291	259	259
query70	861	771	764	764
query71	301	248	211	211
query72	2797	2524	2270	2270
query73	515	536	292	292
query74	4608	4485	4339	4339
query75	2316	2277	1892	1892
query76	2317	1008	577	577
query77	341	379	254	254
query78	9794	9666	9148	9148
query79	887	851	508	508
query80	519	460	359	359
query81	534	322	271	271
query82	578	138	102	102
query83	304	193	176	176
query84	309	124	103	103
query85	799	424	365	365
query86	327	239	224	224
query87	1979	1974	1811	1811
query88	3634	2661	2660	2660
query89	314	282	243	243
query90	1848	163	160	160
query91	156	155	126	126
query92	95	87	88	87
query93	990	994	568	568
query94	445	305	269	269
query95	574	341	352	341
query96	677	480	236	236
query97	2414	2425	2293	2293
query98	163	154	147	147
query99	764	750	649	649
Total cold run time: 233742 ms
Total hot run time: 150261 ms

[hello-stephen]

ClickBench: Total hot run time: 24.35 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7de569daeb9521790075380d85a36e81968bf400, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.05
query3	0.26	0.09	0.09
query4	1.60	0.10	0.10
query5	0.26	0.24	0.23
query6	1.19	0.69	0.68
query7	0.04	0.00	0.00
query8	0.06	0.03	0.04
query9	0.41	0.32	0.35
query10	0.58	0.59	0.60
query11	0.21	0.14	0.14
query12	0.18	0.14	0.14
query13	0.50	0.51	0.49
query14	1.05	1.01	1.02
query15	0.62	0.60	0.59
query16	0.36	0.35	0.35
query17	1.13	1.10	1.14
query18	0.22	0.19	0.20
query19	2.11	1.96	1.97
query20	0.02	0.01	0.02
query21	15.43	0.20	0.14
query22	4.95	0.06	0.05
query23	16.26	0.33	0.12
query24	3.06	0.63	0.45
query25	0.13	0.10	0.06
query26	1.48	0.22	0.14
query27	0.05	0.05	0.04
query28	3.51	0.73	0.35
query29	12.52	4.21	3.30
query30	0.30	0.14	0.10
query31	2.77	0.56	0.31
query32	3.21	0.60	0.52
query33	3.14	3.23	3.19
query34	15.40	3.98	3.38
query35	3.34	3.33	3.37
query36	0.58	0.45	0.42
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.15	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 97.57 s
Total hot run time: 24.35 s

[924060929]

run buildall

[924060929]

/review

[github-actions]

Static review of all 77 changed paths at eadfc6f identified one additional P2: an outer cloud replan retry can permanently retain newly allocated scheduler groups after repopulating a closed statement scope. The inline comment describes the trigger, ownership chain and fix direction.

The four existing P2 reports remain applicable and are not duplicated: retained-manifest schema cleanup (4317059423), recovery-schema name collisions (4317059437), metadata callable/table-owner lifetime (4317059444), and malformed producer-test envelopes (4317059450). There are no existing P0/P1 findings to carry as blocking IDs. No additional user-provided focus was supplied.

1. Goal and proof: the code implements query-owned fair scheduling, separate metadata workers, cooperative split production, bounded completion-ordered prefetch and smaller transport envelopes. Tests cover these mechanisms; the new retry leak and known P2 cases qualify the correctness conclusion.
2. Scope and clarity: the changes are broad but follow the documented scheduling/hot-path goal and connector SPI boundary. Per-file format replaces read-side format inference; writer inference remains available.
3. Concurrency: traced planning, SPLIT/METADATA workers, BE-fetch RPCs and cancellation/finalization. CAS/queues own task publication; admission, producer, readiness, backend-policy and per-backend wait locks protect their respective state. Tasks, IO and callbacks execute outside scheduler registry locks. No additional deadlock, lost wakeup or concurrent source iteration was verified.
4. Lifecycle: captured groups reject resurrection through an old session; normal cleanup, planning failure, inner retries and deferred Flight were traced. Outer replan scope reuse is the new inline finding; running FileIO owner retention is the existing report. No static initialization issue applies.
5. Configuration: three FE thread/queue settings are nonmutable and captured at executor creation; no dynamic-resize promise is made. Session manifest concurrency is forwarded, checked in [1,64] and shared through a query-scoped semaphore.
6. Compatibility: connector API advances 18.0 to 19.0 with its version assertion and plugin surface baseline. New SPI defaults/forwarding are present. ConnectorMetadata methods, persisted formats and existing wire descriptors are unchanged.
7. Thrift/protobuf definitions: none changed; cross-branch field-ID comparison is inapplicable.
8. Parallel paths: checked Hive/Iceberg, cache-on/off, synchronous/streaming/COUNT, metadata tables, both FE dispatch builders and BE local/remote split readers. Per-file formats precede reader/schema setup, including ORC and row-ID fetch.
9. Special conditions: startup demand bypasses the single-lane case; terminal changes wake consumers; admission rechecks cover early dequeues. Partition adaptation uses field IDs, preserves nulls and rejects lossy narrowing. Known schema-recovery cases remain reported above.
10. Test coverage: reviewed changed unit and regression cases for queue pressure, fairness, cancellation, failure, schema promotions, deletes, mixed formats, serialization and LIMIT/full/multiple scans. Outer replan group cleanup needs coverage; the existing malformed fixture and three Iceberg edge cases also remain gaps.
11. Expected results: Hive expectations match the added scheduling configuration. Iceberg IDs 0..191 imply 192 rows/distinct IDs and sum 18336; UNION gives 384 and 36672, and LIMIT counts are 10. Actual execution and output-generation provenance were not independently established.
12. Observability: pool size, active/queued tasks, group count and completions have metrics; existing scan profiles, atomic manifest counters and contextual errors remain. No additional necessary observability omission was verified.
13. Transactions/persistence: no EditLog, committed-data visibility or persisted-version protocol changes. Read-transaction cleanup remains classloader-pinned; the lifetime issues are identified separately.
14. Writes/atomicity: no new data mutation or commit protocol; write-format inference preserves its previous resolution/failure behavior. No additional crash/atomicity regression was verified.
15. FE/BE propagation: no new wire variable. Compaction preserves all inner descriptors, parameter/volume/location boundaries and separate split sources; both BE readers drain every inner range and remote EOF requires an empty response.
16. Performance/resources: bounded task queues, one pending producer batch, manifest windows/read budgets, capped hash candidates and per-producer path reuse address the intended costs. Scanner parallelism is retained. The new permanent group retention is the resource exception; reported benchmarks were not rerun.
17. Further review: independently challenged continuation overlap, callback reentry, option defaults, historical cache load orders, delete matching, ORC initialization and both retry loops. Only the outer-replan scheduler leak adds a distinct finding beyond the existing threads.

Review converged after 2 rounds: all three latest reviewers returned NO_NEW_VALUABLE_FINDINGS on the unchanged one-comment set. All changed paths, candidates and risk focuses have explicit dispositions; no unresolved review question or uncovered changed path remains. Validation was static only: no builds, tests, runtime stress or benchmark reruns, and no repository source changes.

Review converged after 2 round(s).

[hello-stephen]

FE UT Coverage Report

Increment line coverage 85.95% (477/555) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

TPC-H: Total hot run time: 28445 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eadfc6f447199375305fb429e1235d45f8beda75, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17570	5106	5063	5063
q2	2116	352	255	255
q3	10205	1384	828	828
q4	4690	466	336	336
q5	7435	801	537	537
q6	184	183	154	154
q7	750	792	598	598
q8	9254	1333	1097	1097
q9	5530	4222	4220	4220
q10	6819	1286	1028	1028
q11	444	261	227	227
q12	636	403	275	275
q13	18098	3517	2992	2992
q14	280	268	245	245
q15	q16	730	722	657	657
q17	1197	741	581	581
q18	7022	6167	6117	6117
q19	1127	954	589	589
q20	391	344	228	228
q21	4944	2160	2300	2160
q22	339	289	258	258
Total cold run time: 99761 ms
Total hot run time: 28445 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5620	5611	5639	5611
q2	654	570	522	522
q3	5234	5388	4947	4947
q4	1997	2157	1554	1554
q5	5289	5187	5437	5187
q6	257	202	153	153
q7	2300	2098	1892	1892
q8	3044	2796	2807	2796
q9	7994	8072	8019	8019
q10	3732	3669	3227	3227
q11	635	606	440	440
q12	695	680	510	510
q13	3589	3897	3251	3251
q14	328	331	299	299
q15	q16	716	714	653	653
q17	8311	7300	6897	6897
q18	12980	12214	12828	12214
q19	918	828	831	828
q20	2164	2155	1900	1900
q21	5678	4582	4677	4582
q22	491	475	435	435
Total cold run time: 72626 ms
Total hot run time: 65917 ms

[hello-stephen]

TPC-DS: Total hot run time: 150530 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eadfc6f447199375305fb429e1235d45f8beda75, data reload: false

query5	4318	577	440	440
query6	436	204	191	191
query7	4800	428	226	226
query8	316	165	146	146
query9	8773	3080	3072	3072
query10	469	308	276	276
query11	6056	3834	3563	3563
query12	146	89	89	89
query13	1238	448	332	332
query14	6167	3944	3648	3648
query14_1	3384	3467	3464	3464
query15	205	193	182	182
query16	979	453	407	407
query17	856	645	534	534
query18	2423	424	312	312
query19	201	160	136	136
query20	85	85	85	85
query21	211	130	104	104
query22	12940	12967	12801	12801
query23	13346	12829	12505	12505
query23_1	12674	12561	12620	12561
query24	7375	851	459	459
query24_1	477	478	466	466
query25	534	426	381	381
query26	1261	255	140	140
query27	2770	442	262	262
query28	4642	1922	1949	1922
query29	1569	600	466	466
query30	291	212	186	186
query31	1008	930	836	836
query32	144	96	92	92
query33	535	302	244	244
query34	976	841	501	501
query35	742	750	630	630
query36	807	788	731	731
query37	128	110	85	85
query38	1769	1721	1704	1704
query39	817	797	747	747
query39_1	721	729	734	729
query40	217	119	97	97
query41	72	68	68	68
query42	87	82	83	82
query43	351	365	317	317
query44	1305	694	743	694
query45	179	169	161	161
query46	880	950	558	558
query47	3000	2681	2808	2681
query48	313	303	214	214
query49	566	396	291	291
query50	661	266	205	205
query51	10460	10274	10137	10137
query52	75	78	65	65
query53	181	203	149	149
query54	238	193	211	193
query55	68	68	64	64
query56	220	210	220	210
query57	1675	1623	1560	1560
query58	286	267	247	247
query59	2107	2184	1940	1940
query60	275	239	222	222
query61	145	145	143	143
query62	380	321	299	299
query63	193	154	156	154
query64	2763	906	741	741
query65	3303	3249	3255	3249
query66	1768	425	309	309
query67	20212	20262	19942	19942
query68	2936	998	607	607
query69	384	306	234	234
query70	830	769	763	763
query71	298	234	214	214
query72	2690	2547	2346	2346
query73	511	516	294	294
query74	4624	4489	4340	4340
query75	2324	2284	1911	1911
query76	2291	1011	619	619
query77	350	375	270	270
query78	9715	9749	9211	9211
query79	946	898	517	517
query80	517	405	337	337
query81	527	317	267	267
query82	582	130	110	110
query83	299	193	185	185
query84	305	122	95	95
query85	819	432	363	363
query86	339	238	230	230
query87	2000	1963	1778	1778
query88	3678	2706	2682	2682
query89	321	286	238	238
query90	1884	168	168	168
query91	153	155	123	123
query92	92	88	88	88
query93	1028	985	585	585
query94	446	311	275	275
query95	586	389	317	317
query96	676	489	242	242
query97	2444	2437	2293	2293
query98	162	149	155	149
query99	767	774	642	642
Total cold run time: 234734 ms
Total hot run time: 150530 ms

[hello-stephen]

ClickBench: Total hot run time: 24.26 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit eadfc6f447199375305fb429e1235d45f8beda75, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.05
query3	0.25	0.09	0.09
query4	1.61	0.11	0.11
query5	0.23	0.24	0.24
query6	1.17	0.68	0.70
query7	0.04	0.01	0.01
query8	0.05	0.04	0.04
query9	0.40	0.35	0.37
query10	0.57	0.61	0.58
query11	0.20	0.14	0.15
query12	0.19	0.15	0.16
query13	0.48	0.49	0.50
query14	1.03	1.03	1.02
query15	0.64	0.61	0.61
query16	0.35	0.35	0.36
query17	1.19	1.13	1.15
query18	0.20	0.18	0.19
query19	2.15	1.93	2.02
query20	0.01	0.01	0.02
query21	15.48	0.19	0.13
query22	4.94	0.05	0.06
query23	16.26	0.32	0.12
query24	3.14	0.48	0.38
query25	0.10	0.07	0.05
query26	1.08	0.21	0.16
query27	0.05	0.03	0.05
query28	3.28	0.77	0.38
query29	12.47	4.17	3.26
query30	0.29	0.13	0.09
query31	2.77	0.54	0.32
query32	3.23	0.59	0.51
query33	3.15	3.16	3.28
query34	15.42	4.01	3.39
query35	3.33	3.32	3.33
query36	0.54	0.42	0.42
query37	0.10	0.07	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.18	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 96.92 s
Total hot run time: 24.26 s

[924060929]

run buildall

[924060929]

/review

[github-actions]

Reviewed all 78 changed paths at 8b6086f, including production code, deleted code, tests, documentation and expected results. No additional distinct inline findings were substantiated. This is a static review, not an approval of the unresolved issues already discussed.

Five existing P2 findings still apply and are not reposted: outer replan scope/task-group leak, producer test envelopes incompatible with compaction, prefetch callable outliving its table/FileIO owner, recovery-schema partition-name collision, and retained manifests after historical-schema cleanup. No existing P0/P1 finding was present or independently confirmed, so existing_blocking_comment_ids is empty.

Critical checkpoint conclusions:

• Goal and evidence: separate SPLIT/METADATA workers, cooperative producer continuations, bounded manifest prefetch and transport compaction implement the intended scheduling design. Added unit and regression tests exercise the intended behavior; the five known defects above remain.
• Scope and reuse: scheduling, per-producer path reuse, backend candidate bounds and transport helpers form a coherent scan-planning change. The removed read-format cache is replaced by existing per-file format metadata; write-side format resolution remains intact.
• Concurrency and locking: traced request, producer, metadata, RPC and cancellation threads through queue admission, listener registration, readiness and terminal notifications. Remote loads and future waits occur outside the manifest pending lock; callback and queue lock ordering did not establish another defect. Scheduling is cooperative and cannot preempt an SDK call that does not return.
• Lifecycle: checked queued/running futures, backpressure suspension, early LIMIT, planning failure, EXPLAIN, retry and deferred Arrow Flight cleanup. Existing owner-lifetime and outer-retry findings cover the verified remaining failures; no additional lifetime issue was substantiated.
• Configuration: worker counts are startup settings; the bounded manifest concurrency and initial batch size are validated session settings and forwarded where required. Initial delivery limits real splits before compaction and does not suspend producer lookahead.
• Compatibility: connector API major version advances to 19.0 with the changed public surface baseline; ConnectorMetadata's baseline surface is unchanged. Both FE plan builders preserve existing FE/BE descriptor fields, and BE already flattens multiple ranges per envelope. No Thrift/protobuf definitions or persisted format changes occur.
• Parallel paths: reviewed synchronous/streaming/COUNT planning, Hive partition lanes, native/JNI metadata reads, V1/V2 scanners, mixed ORC/Parquet files, row-id fetches and write-format callers. Per-range physical formats and Iceberg field-id mappings remain authoritative.
• Conditions and errors: examined admission/full/closed states, cancellation during permit waits, empty or filtered scans, unsupported formats and lossy historical conversions. Failures propagate through their established error channels; known schema-recovery defects are covered by the existing threads.
• Test coverage and expected results: inspected admission, fairness, cancellation, wakeup, bounded-window, promotion/delete, transport serialization and path-identity tests, plus Hive and Iceberg regression cases. Expected aggregate results are internally consistent. The remaining malformed fixture is the existing test finding linked above.
• Observability: new pool/active/queued/completed/group metrics complement existing scan profiles and cache counters; no additional observability defect was established.
• Persistence and failover: no EditLog, internal partition-version, rowset or delete-bitmap changes. Snapshot pinning and delete-file metadata were traced through cache adaptation before residual/delete matching; the two known historical-spec recovery issues remain.
• Writes, transactionality and crashes: write-format property/inference behavior remains, and no write commit or atomicity protocol changes were introduced. Cancellation/resource failure analysis is covered by the lifecycle conclusions above.
• FE/BE variable passing: transport compaction retains per-file descriptors and existing fields in both plan builders; new session controls are consumed in FE rather than requiring new BE wire fields.
• Performance and memory: inspected bounded task queues/read windows, short-held locks, per-producer path reuse and physical-backend candidate clamping. Synchronous compaction retains the scanner concurrency upper bound. No additional static performance regression was substantiated; performance claims were not independently benchmarked.
• Further issues and user focus: no additional focus was supplied. The final missed-path sweep found no unresolved static candidate or uncovered changed path.

Converged in 1 review round: all three assigned reviewers, including the separate risk reviewer, returned NO_NEW_VALUABLE_FINDINGS on the unchanged empty new-comment set. Builds, tests and runtime/remote-I/O timing validation were not performed under this review-only contract. Delivery verification is separate from convergence.

Review converged after 1 round(s).

[hello-stephen]

FE UT Coverage Report

Increment line coverage 86.45% (504/583) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

TPC-H: Total hot run time: 28212 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8b6086f26c73ca8e693f10a8e886c9e976d5e684, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17567	5186	5012	5012
q2	2091	316	252	252
q3	10258	1379	808	808
q4	4685	456	329	329
q5	7428	792	531	531
q6	181	182	151	151
q7	756	795	605	605
q8	9244	1325	1056	1056
q9	5595	4196	4164	4164
q10	6813	1278	1032	1032
q11	414	248	230	230
q12	646	395	279	279
q13	18063	3504	2999	2999
q14	271	263	237	237
q15	q16	716	708	655	655
q17	1193	742	574	574
q18	7056	6149	6083	6083
q19	1103	943	591	591
q20	370	337	228	228
q21	4354	2132	2262	2132
q22	346	301	264	264
Total cold run time: 99150 ms
Total hot run time: 28212 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5606	5571	5579	5571
q2	706	583	540	540
q3	5207	5312	4871	4871
q4	1984	2150	1541	1541
q5	5243	5153	5299	5153
q6	272	209	161	161
q7	2360	2016	1855	1855
q8	3172	2699	2731	2699
q9	7994	8023	7850	7850
q10	3745	3664	3202	3202
q11	612	446	424	424
q12	867	720	538	538
q13	3549	3902	3267	3267
q14	304	316	296	296
q15	q16	723	701	631	631
q17	8248	7519	6846	6846
q18	13027	12215	12796	12215
q19	901	838	779	779
q20	2135	2141	1919	1919
q21	5657	4684	4658	4658
q22	512	437	440	437
Total cold run time: 72824 ms
Total hot run time: 65453 ms

[hello-stephen]

TPC-DS: Total hot run time: 150104 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8b6086f26c73ca8e693f10a8e886c9e976d5e684, data reload: false

query5	4299	584	445	445
query6	423	209	188	188
query7	4809	447	217	217
query8	317	164	148	148
query9	8737	3053	3068	3053
query10	482	312	244	244
query11	5998	3832	3556	3556
query12	146	96	87	87
query13	1246	433	307	307
query14	6124	3939	3652	3652
query14_1	3408	3398	3375	3375
query15	208	197	180	180
query16	938	427	351	351
query17	841	653	573	573
query18	2418	411	333	333
query19	183	155	129	129
query20	82	80	77	77
query21	205	120	102	102
query22	13092	12920	12843	12843
query23	13335	12788	12352	12352
query23_1	12547	12442	12556	12442
query24	7379	820	450	450
query24_1	452	452	473	452
query25	493	396	324	324
query26	1255	273	137	137
query27	2769	432	263	263
query28	4650	1958	1909	1909
query29	1512	571	411	411
query30	291	216	181	181
query31	986	918	816	816
query32	129	89	91	89
query33	526	282	233	233
query34	891	846	486	486
query35	746	737	647	647
query36	842	854	746	746
query37	122	97	75	75
query38	1760	1708	1684	1684
query39	808	808	744	744
query39_1	732	722	738	722
query40	213	123	94	94
query41	67	63	60	60
query42	84	83	79	79
query43	345	364	305	305
query44	1291	669	673	669
query45	175	175	157	157
query46	852	959	556	556
query47	3017	2917	2913	2913
query48	326	302	215	215
query49	578	388	293	293
query50	669	270	202	202
query51	10458	10274	10237	10237
query52	78	78	72	72
query53	183	201	151	151
query54	231	188	169	169
query55	69	63	61	61
query56	243	216	226	216
query57	1648	1622	1539	1539
query58	278	259	249	249
query59	2129	2196	2047	2047
query60	298	243	220	220
query61	139	145	141	141
query62	371	334	301	301
query63	189	158	164	158
query64	2712	931	762	762
query65	3283	3245	3259	3245
query66	1759	421	313	313
query67	20137	20086	20024	20024
query68	3231	1002	613	613
query69	381	294	240	240
query70	851	806	774	774
query71	304	243	214	214
query72	2815	2705	2414	2414
query73	509	520	292	292
query74	4615	4497	4322	4322
query75	2313	2303	1910	1910
query76	2325	1011	575	575
query77	342	383	267	267
query78	9885	9683	9102	9102
query79	975	937	536	536
query80	614	421	339	339
query81	521	319	271	271
query82	581	134	103	103
query83	326	197	186	186
query84	314	127	110	110
query85	814	422	367	367
query86	317	231	228	228
query87	1994	1980	1838	1838
query88	3626	2663	2616	2616
query89	330	283	239	239
query90	1894	156	159	156
query91	170	151	127	127
query92	98	86	88	86
query93	1017	986	573	573
query94	434	315	255	255
query95	588	387	297	297
query96	669	478	230	230
query97	2410	2464	2281	2281
query98	162	170	144	144
query99	778	763	638	638
Total cold run time: 234181 ms
Total hot run time: 150104 ms

[hello-stephen]

ClickBench: Total hot run time: 24.42 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8b6086f26c73ca8e693f10a8e886c9e976d5e684, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.05
query3	0.26	0.09	0.08
query4	1.62	0.10	0.10
query5	0.24	0.24	0.23
query6	1.19	0.68	0.68
query7	0.04	0.01	0.00
query8	0.06	0.04	0.04
query9	0.40	0.34	0.36
query10	0.61	0.58	0.59
query11	0.20	0.14	0.14
query12	0.19	0.15	0.15
query13	0.49	0.49	0.49
query14	1.02	1.01	1.02
query15	0.65	0.61	0.60
query16	0.34	0.36	0.34
query17	1.09	1.12	1.15
query18	0.21	0.20	0.19
query19	2.12	2.00	1.99
query20	0.02	0.01	0.01
query21	15.42	0.20	0.13
query22	4.80	0.06	0.05
query23	16.27	0.33	0.12
query24	3.09	0.67	0.35
query25	0.14	0.06	0.06
query26	1.24	0.21	0.15
query27	0.04	0.04	0.04
query28	3.77	0.79	0.36
query29	12.48	4.11	3.30
query30	0.28	0.14	0.10
query31	2.77	0.56	0.32
query32	3.23	0.59	0.51
query33	3.18	3.23	3.24
query34	15.45	3.99	3.39
query35	3.35	3.35	3.35
query36	0.55	0.45	0.44
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.15	0.15
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 97.42 s
Total hot run time: 24.42 s

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 65.61% (372/567) 🎉
Increment coverage report
Complete coverage report

[924060929]

run buildall

[924060929]

/review

[github-actions]

Reviewed exact head 62a461e31a827144ac72ad413f28edff70c7e088. No new reportable defect was found after complete static review of all 86 changed paths and related callers/consumers. The earlier inline issues were revalidated as fixed, including P1 partition/delete matching, pre-registration planning cleanup, and COUNT cancellation test arguments. No existing P0/P1 finding was confirmed to remain applicable.

Code-review checkpoint conclusions:

1. Purpose and correctness: bounded cooperative scan scheduling, independent metadata workers and first-response sizing follow the actual statement-to-BE flow; inspected tests cover split conservation, fairness and failure/cancellation behavior.
2. Scope and abstraction: engine scheduling stays connector-agnostic through the SPI; related path/assignment optimizations and format-cache removal support the same scan-planning change.
3. Concurrency and locks: traced planning, producer, metadata, completion, cancellation and RPC threads. Policy, readiness, admission and backend wait-channel state have separate protection; callbacks and remote I/O execute outside registry locks. No concrete deadlock or lost-wakeup defect was found.
4. Lifetimes: checked EXPLAIN/planning failure, prepared execution, both retry loops and deferred Arrow Flight. Groups terminate without resurrection; active shared readers retain the exact table/FileIO owner until callable exit, while cancelled followers detach. No static-initialization issue applies.
5. Configuration: FE worker/queue settings are immutable startup settings. Session initial-batch and manifest-concurrency settings are validated, forwarded and captured for query use; the read budget is shared across the statement's scans.
6. Compatibility: the connector API major and recorded public surface move together to 21.0. Existing wire descriptors retain per-file metadata and formats; no persistent storage format or function-symbol change was identified.
7. Thrift/protobuf IDs: no definitions changed; cross-branch field-ID checks are not applicable.
8. Parallel paths: inspected partition and whole-scan streaming, synchronous planning, both COUNT paths, cache-disabled SDK fallback, pinned/time-travel schemas, system tables, rewrite filtering, both FE transport builders and BE V1/V2 native/JNI consumers.
9. Conditions: cancellation guards, serial-first reads, demand-controlled prefetch, batch-mode gating and compaction boundaries have concrete triggering scenarios. Empty and fully filtered scans still progress; invalid formats/conversions fail visibly.
10. Test coverage: inspected unit/race/negative tests and Hive/Iceberg regressions, including source-schema expiration, mixed cache warmth, exact-owner retention, queue capacity and EOF. No new static test defect was found.
11. Changed expected results: aggregate and ordered outputs agree with the fixture arithmetic. Static inspection cannot prove how output files were generated.
12. Observability: separate executor metrics, split profiles, concurrent cache counters and error channels expose the changed paths; no missing critical signal was established.
13. Transactions/persistence/failover: no EditLog, visible-version or transaction-commit protocol changes; connector read-transaction cleanup remains query-owned.
14. Writes/atomicity/crashes: no new data mutation path; writer-side format inference retains its behavior. Cancellation/error exits preserve resource cleanup in the inspected flow.
15. FE/BE values: both synchronous senders and batch RPC delivery preserve all inner descriptors. BE iterates every compacted range, retains scanner concurrency and treats only an empty response as EOF; physical range formats override Iceberg's scan-level default.
16. Performance/resources: pending work and prefetch windows are bounded; warm reads avoid executor traffic and shared-load followers release metadata workers. Parser reuse, candidate capping and variance checks remove repeated work. Fairness remains cooperative across blocking SDK calls.
17. Additional concerns: the supplied focus file contains no additional user focus. Final missed-case checks found no further verified issue or unresolved static coverage gap.

Validation limits: this invocation performed static inspection only, as required. No builds, tests, live SQL or performance measurements were run, and historical PR benchmark/test claims were not independently reproduced.

Review converged after 2 round(s).

[hello-stephen]

FE UT Coverage Report

Increment line coverage 87.01% (536/616) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

TPC-H: Total hot run time: 28640 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 62a461e31a827144ac72ad413f28edff70c7e088, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17595	5155	5058	5058
q2	2086	306	292	292
q3	10233	1384	805	805
q4	4677	460	339	339
q5	7437	797	526	526
q6	178	188	154	154
q7	744	779	601	601
q8	9242	1312	1068	1068
q9	5478	4211	4188	4188
q10	6910	1270	1018	1018
q11	452	251	239	239
q12	637	401	273	273
q13	18094	3558	2996	2996
q14	266	272	248	248
q15	q16	726	708	673	673
q17	1176	744	582	582
q18	7077	6169	6110	6110
q19	1108	959	601	601
q20	389	331	227	227
q21	5380	2362	2530	2362
q22	394	336	280	280
Total cold run time: 100279 ms
Total hot run time: 28640 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5919	5820	5977	5820
q2	718	649	566	566
q3	5516	5750	5196	5196
q4	2013	2148	1569	1569
q5	5874	5729	5704	5704
q6	237	192	144	144
q7	2249	2086	1832	1832
q8	3013	2556	2634	2556
q9	7633	7446	7487	7446
q10	3620	3553	3150	3150
q11	610	424	398	398
q12	669	671	501	501
q13	3321	3548	3044	3044
q14	293	291	277	277
q15	q16	661	685	621	621
q17	7881	7022	7057	7022
q18	13010	12162	12938	12162
q19	889	777	767	767
q20	2149	2184	1925	1925
q21	5621	4686	4750	4686
q22	499	473	428	428
Total cold run time: 72395 ms
Total hot run time: 65814 ms

[hello-stephen]

TPC-DS: Total hot run time: 151399 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 62a461e31a827144ac72ad413f28edff70c7e088, data reload: false

query5	4307	576	449	449
query6	424	222	194	194
query7	4794	450	233	233
query8	325	183	166	166
query9	8762	3122	3108	3108
query10	482	316	251	251
query11	6026	3849	3573	3573
query12	148	92	88	88
query13	1264	441	312	312
query14	6154	4068	3731	3731
query14_1	3393	3464	3482	3464
query15	211	200	197	197
query16	944	438	418	418
query17	863	666	536	536
query18	2427	450	294	294
query19	183	160	128	128
query20	84	80	79	79
query21	202	127	109	109
query22	12982	12998	12863	12863
query23	13440	12693	12429	12429
query23_1	12503	12608	12589	12589
query24	7444	858	466	466
query24_1	483	480	484	480
query25	513	394	340	340
query26	1266	262	142	142
query27	2760	435	279	279
query28	4624	1924	1945	1924
query29	1555	590	445	445
query30	295	222	183	183
query31	1009	923	820	820
query32	152	89	95	89
query33	522	297	231	231
query34	886	847	519	519
query35	740	737	644	644
query36	822	814	722	722
query37	123	100	80	80
query38	1796	1735	1651	1651
query39	784	796	741	741
query39_1	741	724	737	724
query40	216	115	92	92
query41	65	61	63	61
query42	85	83	81	81
query43	350	392	317	317
query44	1312	679	685	679
query45	184	175	163	163
query46	829	952	568	568
query47	2714	2899	2865	2865
query48	296	330	232	232
query49	580	405	303	303
query50	678	276	216	216
query51	10573	10365	10315	10315
query52	81	80	72	72
query53	189	210	156	156
query54	278	202	200	200
query55	74	70	66	66
query56	251	233	237	233
query57	1703	1628	1593	1593
query58	297	278	266	266
query59	2141	2172	1959	1959
query60	322	260	231	231
query61	169	170	167	167
query62	395	342	287	287
query63	197	167	174	167
query64	2893	1041	881	881
query65	3403	3269	3263	3263
query66	1790	420	302	302
query67	19999	20286	19942	19942
query68	2910	1008	604	604
query69	389	277	251	251
query70	871	785	779	779
query71	287	236	206	206
query72	2760	2599	2328	2328
query73	506	539	299	299
query74	4633	4577	4392	4392
query75	2319	2298	1922	1922
query76	2310	1036	610	610
query77	347	384	281	281
query78	9757	9697	9161	9161
query79	982	834	522	522
query80	1247	436	356	356
query81	595	321	280	280
query82	600	139	104	104
query83	318	215	193	193
query84	314	126	100	100
query85	867	416	378	378
query86	402	237	223	223
query87	1993	1928	1826	1826
query88	3613	2668	2655	2655
query89	325	278	237	237
query90	1927	171	175	171
query91	168	153	132	132
query92	105	90	93	90
query93	990	1001	575	575
query94	625	305	266	266
query95	575	395	291	291
query96	655	473	237	237
query97	2442	2425	2321	2321
query98	159	150	149	149
query99	768	771	655	655
Total cold run time: 235063 ms
Total hot run time: 151399 ms

[hello-stephen]

ClickBench: Total hot run time: 24.3 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 62a461e31a827144ac72ad413f28edff70c7e088, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.05
query3	0.25	0.09	0.09
query4	1.61	0.11	0.09
query5	0.27	0.24	0.24
query6	1.18	0.68	0.68
query7	0.03	0.01	0.00
query8	0.05	0.04	0.03
query9	0.39	0.34	0.34
query10	0.63	0.62	0.58
query11	0.20	0.14	0.14
query12	0.18	0.15	0.14
query13	0.50	0.51	0.49
query14	1.03	1.01	1.02
query15	0.63	0.62	0.62
query16	0.35	0.35	0.35
query17	1.13	1.13	1.14
query18	0.20	0.19	0.19
query19	2.07	1.89	1.99
query20	0.02	0.02	0.01
query21	15.50	0.19	0.14
query22	4.95	0.06	0.05
query23	16.27	0.31	0.12
query24	3.12	0.46	0.50
query25	0.10	0.06	0.06
query26	0.99	0.21	0.17
query27	0.05	0.03	0.05
query28	3.76	0.81	0.35
query29	12.46	4.07	3.23
query30	0.29	0.13	0.10
query31	2.77	0.56	0.32
query32	3.23	0.60	0.49
query33	3.15	3.29	3.21
query34	15.47	3.99	3.40
query35	3.36	3.32	3.35
query36	0.56	0.44	0.43
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.15	0.14
query41	0.08	0.02	0.02
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 97.34 s
Total hot run time: 24.3 s

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 66.40% (409/616) 🎉
Increment coverage report
Complete coverage report

[924060929]

run buildall

[924060929]

/review

[github-actions]

Static review of PR #69298 at b756054 identified one new verified P2 issue: dropped partition-source recovery orders schema IDs numerically and can mislabel a retained LONG manifest as INTEGER, causing a cold streaming scan to fail. All 86 changed paths and their relevant upstream/downstream mechanisms were covered. The review converged in 3 rounds, including an unchanged-payload final pass on the complete one-finding set by the complete reviewer and both focused risk reviewers.

All eleven supplied existing threads were revalidated without reposting them. In particular, P1 comment 4322080662 is fixed: scan-local UNKNOWN partition projections normalize Integer/Long and Float/Double representations before both data/delete matching, with mixed warm/cold schema-cleanup coverage. No existing P0/P1 finding remains applicable. The prior COUNT reflection arguments, pre-scan and replan group cleanup, scheduler-cancellation fallback, delete-manifest budget, running-reader ownership, test envelopes and retained-schema/name-collision concerns are also addressed at this head.

Required checkpoint	Conclusion
Goal and supporting tests	Query-owned SPLIT/METADATA scheduling, cooperative producers and bounded manifest prefetch follow the stated goal. Dedicated unit and Hive/Iceberg regression cases cover the principal paths; inspected only. Recovery on nonmonotonic schema IDs remains defective as described inline.
Scope and clarity	The large change remains coherent around external scan scheduling, bookkeeping and delivery. Admission, read lifetime and partition adaptation have separate helpers.
Concurrency, lock scope and order	Traced planner, SPLIT/METADATA workers, fetch RPCs, cancellation and shared-load callbacks. Listener registration/rechecks prevent missed capacity notifications; producer scheduling coalesces wakeups. Remote I/O and callbacks stay outside short scheduler registry locks. No substantiated deadlock or duplicate producer execution found.
Lifecycle and initialization	Captured groups remain terminal after close. Readers retain the exact table owner until the callable exits. Planning failures, EXPLAIN, prepared executions and retries receive cleanup; deferred Flight retains its callbacks until finalization. No C++ static initialization change.
Configuration and dynamic behavior	FE worker/capacity settings are startup settings with lazy executor creation. Session concurrency and initial fetch size are validated and forwarded. Initial delivery limits preserve background lookahead.
API and format compatibility	Connector plugin API advances from 21.0 to 22.0 with the updated surface baseline/version test; ConnectorMetadata's method surface is unchanged. Existing wire descriptors carry each file's physical format and metadata. No storage-format change.
Thrift/protobuf IDs	No gensrc definitions change, so cross-branch field/enum-ID comparison is inapplicable.
Parallel paths	Checked synchronous/streaming, materialized/representative/COUNT fallback, data/delete manifests and equality-field discovery, Hive listing, both coordinator builders, and native/JNI consumers.
Conditional checks and errors	Cancellation propagates instead of starting SDK fallback; ordinary cache failures retain the existing active-query fallback. Warm probes avoid executor/permit work. Historical narrowing guards fail explicitly; source recovery incorrectly treats schema identifiers as chronological, as described inline.
Test coverage	Inspected queue saturation, wakeups, cancellation, completion races, owner/permit release, schema evolution, mixed cache warmth, request boundaries and full/LIMIT/multiple-scan cases. A retained post-promotion manifest with nonmonotonic schema IDs needs coverage. No tests were executed.
Changed expected results	Added Hive results agree with the reused fixture; Iceberg count/distinct 192, sum 18336, UNION 384/36672 and LIMIT 10 agree with the generated IDs. Output-generation provenance was not independently established.
Observability	Fair-pool size, active/queued tasks, query groups and completion metrics use the existing thread-pool metric framework. Existing cache/scan profiling and failure causes are retained.
Transactions, persistence and failover	Read-transaction release stays in query finalization, now including planning-only attempts. No EditLog, durable metadata or visible-version protocol changes. Scheduler groups are transient.
Writes, atomicity and crash behavior	Read scheduling introduces no write-commit protocol. Write-format inference remains property-first with its existing fallback. Shared-cache publication keeps its generation fence; cancellation retains running-reader ownership.
FE-BE propagation	New values control FE planning and use session forwarding. BE already consumes every descriptor and selects range-local formats; compacted response size is not EOF. No new BE wire variable.
Performance and remaining issues	Queues and completed manifest windows remain bounded; shared-load followers release workers. Path reuse and bounded candidate lookup avoid repeated work. Compaction preserves descriptors, stream boundaries and the conservative BE scanner-concurrency floor. The one verified recovery regression is reported inline; no further verified issue was found.

Core invariants: scan-local partition adaptation precedes residual/delete evaluation and leaves shared cache payloads unchanged, but its source-type assumption fails for the retained-manifest scenario reported inline. No native partition-version or MoW bitmap change is involved. Ownership and bounded FE allocations were traced; no BE allocation path changes. Failures remain explicit rather than silently dropping scan work.

User focus: no additional focus points were supplied. All five initial risk areas have evidence-based dispositions: R4 yields the accepted inline finding; R1-R3 and R5 were dismissed with evidence. No unresolved candidate or uncovered changed path remains. Validation remains static-only: builds, tests, runtime concurrency stress and output-generation provenance were not verified in this invocation.

Review converged after 3 round(s).

[github-actions]

[P2] Recover dropped sources without ordering schema IDs

For a v2 table with retained schema history S100(p INT) then S20(p LONG), keep a live manifest written under S20, then remove identity(p) and drop p while retaining both schemas. Schema IDs need only be unique (specification); the SDK parser preserves the supplied IDs. This sort selects S100 and labels the manifest's physical Long(7) partition as INTEGER. The SDK reads physical LONG as Long and stores it in the expected partition struct, so forScan to the current UNKNOWN spec reaches IcebergPartitionValueAdapter's Integer cast and throws ClassCastException. With manifest caching enabled, a cold streaming scan fails during lazy iteration, outside streamingFileScanTasks' fallback. Recover a compatible source type from metadata history or the manifest header instead of the greatest numeric ID. This differs from the prior UNKNOWN finding: one entirely cold data manifest fails without deletes, mixed cache warmth or schema expiration.

[924060929]

Fixed in 3c8c571. The cache now resolves partition source types from the snapshot that added the manifest and its schema, without ordering schema IDs. If that snapshot or schema has expired, it lets Iceberg read the physical type from the manifest header. The new cold-manifest regression reproduced ClassCastException on the previous code for both retained and expired writer snapshots; both pass after the fix, along with all 31 IcebergManifestCacheTest cases.

[hello-stephen]

TPC-H: Total hot run time: 28704 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b756054b97d083f173c18b3627650281810e965b, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17621	5149	5058	5058
q2	2143	353	246	246
q3	10177	1361	806	806
q4	4677	475	350	350
q5	7431	807	532	532
q6	181	183	150	150
q7	740	804	603	603
q8	9264	1306	1020	1020
q9	5414	4194	4157	4157
q10	6819	1283	1033	1033
q11	438	249	231	231
q12	641	394	276	276
q13	18114	3519	3011	3011
q14	276	276	248	248
q15	q16	723	701	672	672
q17	1199	746	572	572
q18	7016	6198	6156	6156
q19	1119	966	605	605
q20	389	345	222	222
q21	5014	2465	2467	2465
q22	374	344	291	291
Total cold run time: 99770 ms
Total hot run time: 28704 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6015	5900	6043	5900
q2	782	593	588	588
q3	5586	5761	5294	5294
q4	2051	2148	1552	1552
q5	5702	5507	5810	5507
q6	243	191	149	149
q7	2238	2009	1812	1812
q8	3014	2667	2624	2624
q9	7927	8072	7875	7875
q10	3728	3579	3134	3134
q11	573	414	379	379
q12	662	670	499	499
q13	3225	3597	3015	3015
q14	284	288	276	276
q15	q16	674	695	620	620
q17	7672	7006	6871	6871
q18	13006	12242	12886	12242
q19	890	783	777	777
q20	2160	2208	1934	1934
q21	5520	4663	4694	4663
q22	520	452	404	404
Total cold run time: 72472 ms
Total hot run time: 66115 ms

[hello-stephen]

TPC-DS: Total hot run time: 151290 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b756054b97d083f173c18b3627650281810e965b, data reload: false

query5	4327	570	451	451
query6	427	206	182	182
query7	4799	427	223	223
query8	318	166	155	155
query9	8757	3120	3133	3120
query10	471	325	246	246
query11	6022	3851	3554	3554
query12	146	89	86	86
query13	1253	432	360	360
query14	6071	3917	3674	3674
query14_1	3466	3439	3373	3373
query15	206	194	182	182
query16	939	436	409	409
query17	857	645	530	530
query18	2417	428	308	308
query19	207	163	136	136
query20	87	85	85	85
query21	214	130	113	113
query22	13115	13115	12780	12780
query23	13480	12899	12333	12333
query23_1	12576	12644	12495	12495
query24	7384	810	469	469
query24_1	482	478	480	478
query25	536	430	360	360
query26	1277	260	141	141
query27	2768	433	271	271
query28	4642	1972	1964	1964
query29	1575	607	446	446
query30	300	215	188	188
query31	1016	933	821	821
query32	152	97	99	97
query33	540	310	238	238
query34	906	840	493	493
query35	729	752	646	646
query36	802	810	733	733
query37	127	100	81	81
query38	1776	1724	1685	1685
query39	798	801	763	763
query39_1	739	727	772	727
query40	210	116	93	93
query41	65	63	62	62
query42	84	85	80	80
query43	344	369	309	309
query44	1282	689	693	689
query45	183	169	163	163
query46	824	966	558	558
query47	2619	2965	2800	2800
query48	298	298	213	213
query49	563	392	285	285
query50	675	264	203	203
query51	10798	10725	10650	10650
query52	79	77	70	70
query53	190	208	150	150
query54	231	188	187	187
query55	72	66	70	66
query56	236	230	207	207
query57	1654	1586	1571	1571
query58	290	268	259	259
query59	2127	2152	1969	1969
query60	287	245	217	217
query61	149	153	143	143
query62	386	335	298	298
query63	193	157	157	157
query64	2748	914	801	801
query65	3335	3282	3267	3267
query66	1763	410	306	306
query67	20248	20118	20118	20118
query68	2937	999	610	610
query69	372	290	248	248
query70	864	778	768	768
query71	297	235	220	220
query72	2756	2686	2367	2367
query73	507	514	288	288
query74	4601	4500	4334	4334
query75	2319	2278	1918	1918
query76	2322	1004	586	586
query77	329	374	272	272
query78	9743	9716	9237	9237
query79	1008	858	536	536
query80	531	425	339	339
query81	532	311	270	270
query82	575	131	111	111
query83	305	195	182	182
query84	313	118	103	103
query85	806	418	379	379
query86	324	240	230	230
query87	2015	1952	1800	1800
query88	3656	2708	2689	2689
query89	335	281	241	241
query90	1893	173	173	173
query91	156	151	127	127
query92	90	88	89	88
query93	1003	1009	583	583
query94	445	304	267	267
query95	593	332	306	306
query96	672	477	232	232
query97	2429	2438	2315	2315
query98	164	158	145	145
query99	769	770	644	644
Total cold run time: 234023 ms
Total hot run time: 151290 ms

[924060929]

run buildall

[924060929]

/review

[hello-stephen]

ClickBench: Total hot run time: 24.43 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b756054b97d083f173c18b3627650281810e965b, data reload: false

query1	0.01	0.01	0.00
query2	0.10	0.05	0.05
query3	0.26	0.09	0.09
query4	1.61	0.10	0.10
query5	0.24	0.24	0.24
query6	1.17	0.68	0.68
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.39	0.34	0.36
query10	0.57	0.62	0.57
query11	0.20	0.16	0.15
query12	0.18	0.14	0.14
query13	0.49	0.51	0.51
query14	1.01	1.02	1.00
query15	0.63	0.62	0.62
query16	0.33	0.35	0.35
query17	1.10	1.13	1.12
query18	0.19	0.19	0.20
query19	2.08	1.96	1.96
query20	0.02	0.01	0.01
query21	15.50	0.18	0.14
query22	4.92	0.06	0.05
query23	16.27	0.31	0.13
query24	2.94	0.42	0.41
query25	0.14	0.07	0.05
query26	1.08	0.21	0.16
query27	0.06	0.03	0.04
query28	3.18	0.68	0.35
query29	12.47	4.19	3.23
query30	0.28	0.13	0.10
query31	2.77	0.55	0.32
query32	3.23	0.61	0.52
query33	3.22	3.21	3.24
query34	15.42	3.99	3.41
query35	3.34	3.35	3.35
query36	0.54	0.46	0.44
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.15	0.15
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.56 s
Total hot run time: 24.43 s

[github-actions]

Reviewed #69298 at 3c8c571. No new substantiated findings. Converged in 1 review round after complete static coverage, independent engine/connector reviews and a separate risk investigation.

Existing threads were revalidated and deduplicated. The prior P1 UNKNOWN-partition delete-matching issue (4322080662) is fixed by common data/delete normalization; no existing P0/P1 finding remains confirmed on this head. The earlier schema recovery, source-name collision, running-reader lease, delete-read budget, scheduler cancellation, attempt cleanup/retry and test-signature/envelope concerns are also addressed. No additional user focus was supplied.

Critical checkpoint conclusions:

1. Goal/proof: cooperative split scheduling, separate metadata workers, query cancellation, bounded manifest prefetch and compact transport are implemented along the traced call chains. Added tests exercise the intended behavior; they were inspected only.
2. Scope/clarity: the cross-layer changes serve these scheduling and scan-planning goals; common admission, cache and adaptation helpers are reused.
3. Concurrency/locks: traced request, cancellation, split, metadata, completion and RPC threads. Queue registration rechecks, atomic task ownership and producer coalescing cover the investigated races; remote work and callbacks stay outside scheduler registry locks. No concrete new deadlock, lost wakeup or duplicate production was established.
4. Lifecycle: captured groups close on cancellation and nondeferred attempt completion, including pre-scan failures. Both retry paths replace scopes; prepared execution gets fresh state; deferred Flight retains its original cleanup. Actual shared readers retain their table owner until callable exit.
5. Configuration: FE worker/capacity knobs are startup settings; session initial-batch and manifest-concurrency values are validated and forwarded. No dynamic-update promise is introduced.
6. Compatibility: plugin API major/surface is consistently updated to 22.0; existing nested file-range transport is preserved. No persisted format change.
7. Thrift/protobuf IDs: no definitions changed; cross-release field-ID comparisons are inapplicable.
8. Parallel paths: examined partition/whole-scan sources, cache-on/off, materialized/streaming/both COUNT fallbacks, historical/metadata/mixed-format reads, both FE dispatch builders and both BE scanner versions. Cancellation does not restart SDK planning.
9. Special conditions: first-read/demand gating, filtered scans, UNKNOWN recovery, lossless historical projections, empty partitions and initial-fetch limits have concrete conditions and coverage. No further defect found.
10. Tests: inspected all changed unit/regression cases, including latch-driven admission/cancellation, owner retention, schema evolution, delete matching, exact delivery and descriptor roundtrips. No new statically demonstrable test failure found.
11. Test outputs: Hive and Iceberg counts/sums agree with fixture SQL. Actual output-generation provenance is unverified.
12. Observability: scheduler pool/active/queued/group/completed metrics and concurrent cache tallies expose the changed paths; errors preserve causes and manifest context.
13. Persistence/transactions: no EditLog, replay, visibility-version or commit/failover protocol changes; statement/read-resource cleanup remains ordered and idempotent.
14. Writes/atomicity: write-format inference preserves property precedence and fallback behavior. Transport compaction leaves assignment objects intact; no new publication or atomicity protocol.
15. FE/BE variables: no new wire field; existing session forwarding and explicit per-range physical formats remain. Both dispatch paths supply existing query options.
16. Performance/memory: task work, per-query queues, prefetch windows and per-producer path retention are bounded. Warm cache avoids scheduling; format discovery no longer enumerates data manifests. Compaction preserves the scanner-count bound and every nested range. No benchmark claim is made.
17. Further recheck: revisited callback-before-return, continuation/cancel schedules, prepared/retry/Flight ownership, retained/expired writer schemas, both cache-warmth orders, special numeric partition keys and actual ORC/Parquet dispatch. No unresolved candidate remains.

Limits: static review only, as required. No builds, tests, cluster/external-service execution, race stress or benchmarks were run. Runtime test outcomes and generated-output provenance are unverified; there are no outstanding assigned static coverage gaps.

Review converged after 1 round(s).

[924060929]

run buildall

[924060929]

/review

[github-actions]

Static review of #69298 at aa3fa71, against the supplied merge-base diff. All 86 changed paths were covered by the main review, two complete-review scopes and a separate integration-risk review. No new verified finding remains.

Existing review context: P1 comment 4322080662 is repaired. Both data and delete partitions pass through the scan-local adapter before matching, and UNKNOWN numeric targets now use consistent representations; the mixed-cache/schema-cleanup tests cover the reported scenario. It is neither reposted nor carried as a current blocker. The existing writer-schema, source-name/schema-expiration, read-budget, cancellation, callable-owner and execution-cleanup threads were also checked against their repairs.

Critical checkpoint conclusions:

1. Goal and proof: the code implements query-owned bounded split/metadata scheduling, cooperative production, concurrent cold-manifest loading, cancellation and independent initial delivery. Focused unit/concurrency and Hive/Iceberg regression sources exercise these behaviors; this review does not establish runtime success.
2. Scope and clarity: scheduler, cache, query-lifetime, path/backend bookkeeping and transport changes form one external-scan flow. No unrelated change was identified.
3. Concurrency and locks: traced planner/request threads, SPLIT/METADATA workers, shared-load completions, cancellation and backend fetch requests. Backend policy, readiness, capacity listeners and consumer channels have separate protection; remote reads and user callbacks run outside registry locks. Queue reservation/publication, reentrant admission callbacks, continuation boundaries and terminal wakeups yielded no verified race or lock cycle.
4. Lifecycle: captured groups cannot be resurrected through a cleared scope. EXPLAIN/planning failures, both retry paths, cancellation before coordinator publication and deferred Flight cleanup were traced. Running manifest callables retain their exact table/FileIO owner independently of Future cancellation. No C++ static initialization change is involved.
5. Configuration: the three FE scheduler settings are restart-only; the initial-fetch and manifest-concurrency session values are validated and forwarded/captured. The read budget is shared across a statement's scans/catalogs; no promised dynamic update is lost.
6. Compatibility: the connector plugin API version and public-surface baseline advance to 23.0. Existing transport structures and per-range formats are preserved; offline default schedulers remain available. No storage-format change was found.
7. Thrift/protobuf IDs: no gensrc definition is changed, so no field/enum compatibility change requires cross-branch comparison.
8. Parallel paths: checked synchronous, partition-batch and file-streaming planning, warm/cold data/delete reads, both COUNT variants, SDK fallback, metadata/native formats, legacy Coordinator and Nereids ThriftPlansBuilder, plus row-id fetch.
9. Special conditions: the batch switch is checked before capability/estimate work; cancellation remains terminal before cache-to-SDK fallback. Physical manifest recovery and scan-local conversion have concrete schema-evolution triggers. No speculative success recovery was identified.
10. Test coverage: reviewed shared-load success/failure/invalidation, bounded windows/completion order, queue admission/wakeups, cancellation and permit/owner release, retries, immutable adaptation, serialization and functional suites. Concurrency tests use explicit coordination. Builds, tests and external-service scenarios were not run under this static-only contract.
11. Changed results: checked Hive fixture aggregates/order and all new Iceberg cold/warm/queue-capacity tags. Iceberg IDs 0..191 give count/distinct/sum 192/192/18336 and UNION ALL 384/36672. No inconsistent expected result was found; the diff alone cannot certify how output files were generated.
12. Observability: pool/group/queue/active/completed metrics and existing scan profiles remain available; manifest counters support concurrent loads and failures retain context. No demonstrated observability omission remains.
13. Transactions and persistence: no EditLog, commit protocol or failover mechanism changes. Scope teardown preserves existing transaction-finalization ordering; write-side format inference remains present.
14. Data modification, atomicity and crash handling: no new data-write path. Cache generation fences remain in place, partition adaptation does not mutate shared payloads, and read leases preserve active I/O through cleanup.
15. FE/BE propagation: no new wire field is needed. Both serialization paths retain file descriptors. Actual BE consumers iterate packed ranges, preserve scanner capacity, prefer each range's format and rebuild row-id fetch projection maps independently.
16. Performance: reviewed bounded retention, warm-hit scheduling avoidance, shared-follower worker release, prefix reuse, candidate bounds and batching/compaction. No concrete new hot-path regression was substantiated. Runtime fairness and the PR's benchmark claims were not independently measured.
17. Other/core invariants: checked partition recovery, legal promotion families, residual/delete matching, sequence/row-lineage metadata and failure propagation. No new visible-version, MoW/delete-bitmap or BE allocation change is involved, and no additional verified defect remains.

User focus: no additional focus was supplied. All initial risk items and reviewer candidates have explicit dispositions; no changed-path coverage gap or unresolved code candidate remains. Validation is limited to static inspection. Convergence and delivery are reported separately by this invocation's review marker.

Review converged after 2 round(s).

[hello-stephen]

FE UT Coverage Report

Increment line coverage 86.85% (535/616) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

TPC-H: Total hot run time: 28343 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aa3fa71e7016590937dc9a6ff402bd5e97f3db42, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17596	5172	5035	5035
q2	2086	306	285	285
q3	10231	1366	811	811
q4	4674	464	335	335
q5	7437	791	532	532
q6	179	183	150	150
q7	739	785	605	605
q8	9239	1319	1102	1102
q9	5451	4170	4143	4143
q10	6814	1295	1035	1035
q11	426	245	225	225
q12	644	394	275	275
q13	18099	3541	3018	3018
q14	272	267	252	252
q15	q16	738	720	674	674
q17	1188	746	579	579
q18	7074	6198	6114	6114
q19	1112	959	605	605
q20	375	327	220	220
q21	4604	2099	2230	2099
q22	349	296	249	249
Total cold run time: 99327 ms
Total hot run time: 28343 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5594	5614	5563	5563
q2	673	581	542	542
q3	5202	5401	4922	4922
q4	1981	2113	1514	1514
q5	5302	5169	5467	5169
q6	254	209	162	162
q7	2421	2105	1876	1876
q8	3018	2780	2796	2780
q9	8003	8033	7864	7864
q10	3691	3668	3235	3235
q11	615	449	413	413
q12	692	691	513	513
q13	3494	3734	3333	3333
q14	309	310	275	275
q15	q16	686	702	644	644
q17	8063	7484	7297	7297
q18	13070	12293	12973	12293
q19	918	803	810	803
q20	2183	2142	1907	1907
q21	5556	4590	4689	4590
q22	477	462	426	426
Total cold run time: 72202 ms
Total hot run time: 66121 ms

[hello-stephen]

TPC-DS: Total hot run time: 150540 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aa3fa71e7016590937dc9a6ff402bd5e97f3db42, data reload: false

query5	4304	622	442	442
query6	423	209	188	188
query7	4807	472	224	224
query8	321	172	149	149
query9	8788	3091	3133	3091
query10	467	322	258	258
query11	5948	3852	3587	3587
query12	146	93	87	87
query13	1256	454	347	347
query14	6175	3961	3644	3644
query14_1	3406	3381	3431	3381
query15	209	204	184	184
query16	935	433	399	399
query17	870	676	537	537
query18	2422	431	306	306
query19	196	177	137	137
query20	87	83	84	83
query21	209	137	110	110
query22	12905	13015	12764	12764
query23	13585	12795	12494	12494
query23_1	12459	12442	12461	12442
query24	7380	852	465	465
query24_1	477	474	469	469
query25	548	427	369	369
query26	1277	272	142	142
query27	2743	461	286	286
query28	4598	1948	1921	1921
query29	1570	640	463	463
query30	299	224	185	185
query31	1017	937	810	810
query32	139	94	95	94
query33	525	329	250	250
query34	924	906	497	497
query35	755	774	653	653
query36	814	803	745	745
query37	127	102	92	92
query38	1770	1730	1680	1680
query39	780	803	745	745
query39_1	747	722	742	722
query40	215	119	106	106
query41	78	68	69	68
query42	90	87	82	82
query43	353	390	309	309
query44	1371	701	704	701
query45	182	182	174	174
query46	879	1019	555	555
query47	2953	3030	2584	2584
query48	315	313	221	221
query49	567	408	293	293
query50	727	284	207	207
query51	10575	10369	10592	10369
query52	77	80	70	70
query53	193	223	153	153
query54	237	194	191	191
query55	73	70	63	63
query56	224	205	202	202
query57	1649	1594	1562	1562
query58	272	264	255	255
query59	2167	2094	1910	1910
query60	288	245	228	228
query61	152	155	148	148
query62	392	327	282	282
query63	193	165	159	159
query64	2738	966	785	785
query65	3318	3273	3271	3271
query66	1793	435	307	307
query67	20232	20126	20055	20055
query68	3070	1057	620	620
query69	385	282	257	257
query70	847	755	781	755
query71	310	247	206	206
query72	2777	2578	2255	2255
query73	535	576	303	303
query74	4641	4504	4339	4339
query75	2461	2313	1903	1903
query76	2300	1108	618	618
query77	373	386	282	282
query78	9991	9741	9167	9167
query79	1045	933	515	515
query80	612	432	340	340
query81	537	326	269	269
query82	572	133	104	104
query83	276	196	176	176
query84	310	115	99	99
query85	808	444	367	367
query86	318	234	231	231
query87	2014	1944	1793	1793
query88	3667	2676	2655	2655
query89	349	290	244	244
query90	1913	171	154	154
query91	153	152	129	129
query92	92	91	89	89
query93	1055	1037	580	580
query94	456	311	259	259
query95	579	385	304	304
query96	669	495	238	238
query97	2446	2446	2338	2338
query98	164	151	155	151
query99	763	754	655	655
Total cold run time: 235016 ms
Total hot run time: 150540 ms

[hello-stephen]

ClickBench: Total hot run time: 24.06 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aa3fa71e7016590937dc9a6ff402bd5e97f3db42, data reload: false

query1	0.01	0.00	0.01
query2	0.09	0.04	0.04
query3	0.27	0.09	0.09
query4	1.61	0.08	0.10
query5	0.24	0.24	0.23
query6	1.16	0.69	0.69
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.39	0.33	0.35
query10	0.62	0.56	0.56
query11	0.20	0.15	0.14
query12	0.17	0.15	0.14
query13	0.51	0.48	0.48
query14	1.04	1.03	1.01
query15	0.63	0.60	0.60
query16	0.35	0.34	0.36
query17	1.13	1.11	1.17
query18	0.21	0.20	0.19
query19	2.10	1.98	1.93
query20	0.02	0.01	0.01
query21	15.48	0.19	0.13
query22	4.92	0.06	0.05
query23	16.28	0.32	0.12
query24	3.15	0.50	0.27
query25	0.11	0.06	0.04
query26	0.97	0.21	0.16
query27	0.05	0.03	0.04
query28	3.66	0.82	0.35
query29	12.55	4.16	3.27
query30	0.28	0.13	0.10
query31	2.76	0.54	0.32
query32	3.22	0.60	0.51
query33	3.27	3.25	3.17
query34	15.46	3.99	3.39
query35	3.35	3.36	3.35
query36	0.56	0.44	0.46
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.15	0.14
query41	0.09	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 97.43 s
Total hot run time: 24.06 s

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 62.71% (402/641) 🎉
Increment coverage report
Complete coverage report