Skip to content

[fix](thirdparty) Keep accepting the pre-#67158 Paimon prebuilt fingerprint - #67305

Open
u70b3 wants to merge 1 commit into
apache:masterfrom
u70b3:paimon-prebuilt-fp-compat
Open

[fix](thirdparty) Keep accepting the pre-#67158 Paimon prebuilt fingerprint#67305
u70b3 wants to merge 1 commit into
apache:masterfrom
u70b3:paimon-prebuilt-fp-compat

Conversation

@u70b3

@u70b3 u70b3 commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Related PR: #67158

Problem Summary:

#67158 rotated PAIMON_LEGACY_COMPATIBLE_SEMANTIC_FINGERPRINT to the new semantic fingerprint 9887cf1ec…, but did not keep the previous semantic marker cb82e41ba46f534e611cdd52e66b53c227d49bf8 accepted. The prebuilt currently shipped in the compile image (apache/doris:build-env-ldb-toolchain-latest, digest sha256:5d474521b72d…) is stamped with exactly that old semantic marker, and it is in no accept list. As a result, every CI build that merges current master and validates the prebuilt (COMPILE and performance pipelines) aborts before compiling:

Paimon build fingerprint does not match selected inputs
Arrow/Paimon thirdparty libraries need to be rebuilt ...
Cannot rebuild thirdparty libraries: /var/local/thirdparty/build-thirdparty.sh is missing.

Evidence that the installed marker is cb82e41ba… (elimination): the identical image digest passed the same pipeline one day before #67158 merged (pre-#67158 checkout: expected == cb82e41ba == installed marker) and fails after it (installed ∉ {9887cf1ec, 2bbf52e7…, dbb6ca6e…}). Example failing TeamCity builds: COMPILE 1034233 and 1034257, performance 1034228 and 1034249.

Fix: since #67158 only reorders Paimon codec dependencies and does not change the installed artifacts, list the previous semantic marker in PAIMON_LEGACY_BUILD_FINGERPRINTS so the in-circulation prebuilt stays accepted.

Release note

None

Check List (For Author)

  • Test

    • Manual test (add detailed scripts or steps below)
      • Gate check with the repo's own functions: arrow_paimon_fingerprint_matches cb82e41ba… $(paimon_build_fingerprint) … is rejected on master and accepted with this change; a bogus marker is still rejected in both cases.
      • bash thirdparty/test/arrow-paimon-lifecycle-test.sh → PASS.
    • Previous test can cover this change. (thirdparty/test/arrow-paimon-lifecycle-test.sh)
  • Behavior changed:

  • Does this need documentation?

    • No.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

… fingerprint

apache#67158 rotated PAIMON_LEGACY_COMPATIBLE_SEMANTIC_FINGERPRINT to the new
semantic fingerprint, but the prebuilt currently shipped in the compile
image is still stamped with the previous semantic marker cb82e41ba, which
is in no accept list. Every CI build that merges current master and
validates the prebuilt (COMPILE, performance pipelines) now aborts before
compiling with "Paimon build fingerprint does not match selected inputs"
and cannot rebuild inside the install-only image prefix.

Dependency ordering does not change the installed artifacts, so keep the
previous semantic marker accepted by listing it in
PAIMON_LEGACY_BUILD_FINGERPRINTS.
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@u70b3

u70b3 commented Aug 29, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16720 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 340cc23f08811249e9fbbe29aacd23d7ff316554, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17567	3010	2997	2997
q2	2088	259	228	228
q3	10239	854	526	526
q4	4670	248	205	205
q5	7677	550	381	381
q6	135	116	92	92
q7	540	497	401	401
q8	9376	931	862	862
q9	3700	2423	2361	2361
q10	6564	839	711	711
q11	400	194	185	185
q12	642	262	202	202
q13	18168	1535	1131	1131
q14	160	144	133	133
q15	q16	441	386	359	359
q17	1304	924	842	842
q18	3058	2249	2242	2242
q19	2510	910	771	771
q20	376	279	191	191
q21	5613	1675	1846	1675
q22	327	259	225	225
Total cold run time: 95555 ms
Total hot run time: 16720 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3404	3284	3263	3263
q2	501	392	351	351
q3	2227	2314	2174	2174
q4	1171	1155	879	879
q5	2191	2127	2105	2105
q6	163	126	88	88
q7	1009	924	875	875
q8	1573	1382	1392	1382
q9	3115	3074	3068	3068
q10	1846	1774	1611	1611
q11	347	270	246	246
q12	441	426	345	345
q13	1456	1532	1150	1150
q14	164	166	149	149
q15	q16	391	398	358	358
q17	3557	3230	3255	3230
q18	4793	4410	4757	4410
q19	928	863	844	844
q20	969	952	818	818
q21	3732	3022	3170	3022
q22	390	346	320	320
Total cold run time: 34368 ms
Total hot run time: 30688 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 81204 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 340cc23f08811249e9fbbe29aacd23d7ff316554, data reload: false

query5	4262	405	346	346
query6	377	139	125	125
query7	4952	396	234	234
query8	292	126	115	115
query9	8676	2871	2844	2844
query10	428	213	194	194
query11	5370	1015	919	919
query12	111	70	69	69
query13	1192	450	326	326
query14	5998	2182	2081	2081
query14_1	1950	1937	1919	1919
query15	177	117	116	116
query16	941	366	364	364
query17	802	464	355	355
query18	2331	328	239	239
query19	167	140	111	111
query20	70	68	71	68
query21	200	98	88	88
query22	5298	5256	5223	5223
query23	6647	6180	5807	5807
query23_1	5978	5975	6185	5975
query24	7254	1076	779	779
query24_1	738	761	766	761
query25	399	268	260	260
query26	1236	233	126	126
query27	2786	407	255	255
query28	4716	1471	1496	1471
query29	906	430	335	335
query30	249	150	128	128
query31	811	412	325	325
query32	122	67	71	67
query33	437	208	170	170
query34	983	864	482	482
query35	413	391	333	333
query36	552	543	533	533
query37	118	84	68	68
query38	996	827	799	799
query39	493	478	476	476
query39_1	468	489	460	460
query40	197	90	74	74
query41	52	50	49	49
query42	72	71	74	71
query43	236	239	206	206
query44	1008	542	564	542
query45	103	104	94	94
query46	757	862	506	506
query47	741	771	714	714
query48	303	306	224	224
query49	528	228	185	185
query50	700	243	201	201
query51	8144	8157	8116	8116
query52	72	69	57	57
query53	183	194	142	142
query54	220	170	169	169
query55	72	61	53	53
query56	203	183	152	152
query57	685	669	664	664
query58	193	252	157	157
query59	1176	1253	1085	1085
query60	232	176	168	168
query61	109	116	109	109
query62	356	195	169	169
query63	167	155	145	145
query64	2687	676	540	540
query65	1619	1556	1659	1556
query66	1842	268	191	191
query67	9742	9840	9664	9664
query68	2732	1170	708	708
query69	341	227	188	188
query70	673	627	612	612
query71	247	169	160	160
query72	2365	1693	1532	1532
query73	634	581	334	334
query74	1574	1201	1121	1121
query75	1165	1089	949	949
query76	2280	706	527	527
query77	252	247	221	221
query78	3825	3778	3131	3131
query79	1246	793	607	607
query80	964	315	269	269
query81	465	151	126	126
query82	592	120	92	92
query83	294	214	189	189
query84	308	110	87	87
query85	1017	368	291	291
query86	358	183	173	173
query87	1024	965	895	895
query88	2743	2084	2090	2084
query89	290	196	174	174
query90	1751	133	122	122
query91	131	122	99	99
query92	80	67	68	67
query93	1242	1130	754	754
query94	522	250	220	220
query95	517	250	294	250
query96	858	574	271	271
query97	1067	1139	994	994
query98	138	134	134	134
query99	412	346	308	308
Total cold run time: 173832 ms
Total hot run time: 81204 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.61 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 340cc23f08811249e9fbbe29aacd23d7ff316554, data reload: false

query1	0.00	0.01	0.00
query2	0.08	0.04	0.03
query3	0.25	0.11	0.10
query4	1.61	0.10	0.09
query5	0.17	0.16	0.17
query6	1.26	0.70	0.68
query7	0.03	0.00	0.01
query8	0.05	0.03	0.03
query9	0.28	0.21	0.21
query10	0.33	0.34	0.34
query11	0.16	0.11	0.12
query12	0.14	0.12	0.11
query13	0.29	0.30	0.30
query14	0.43	0.44	0.44
query15	0.37	0.34	0.35
query16	0.20	0.23	0.23
query17	0.66	0.70	0.70
query18	0.17	0.16	0.17
query19	1.16	1.13	1.16
query20	0.02	0.01	0.01
query21	15.43	0.15	0.13
query22	5.06	0.04	0.04
query23	16.18	0.25	0.10
query24	3.02	0.33	0.24
query25	0.09	0.06	0.04
query26	0.79	0.16	0.12
query27	0.03	0.03	0.03
query28	3.66	0.56	0.25
query29	12.45	3.14	2.55
query30	0.25	0.11	0.10
query31	2.76	0.39	0.17
query32	3.50	0.30	0.22
query33	1.34	1.46	1.53
query34	15.41	2.16	1.77
query35	1.73	1.71	1.71
query36	0.47	0.29	0.28
query37	0.06	0.04	0.04
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.11	0.08	0.07
query41	0.08	0.03	0.02
query42	0.03	0.03	0.03
query43	0.03	0.02	0.03
Total cold run time: 90.22 s
Total hot run time: 14.61 s

@u70b3

u70b3 commented Aug 29, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16584 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 340cc23f08811249e9fbbe29aacd23d7ff316554, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17612	3040	3026	3026
q2	2156	253	228	228
q3	10195	854	509	509
q4	4677	245	202	202
q5	7678	538	400	400
q6	138	115	95	95
q7	516	498	387	387
q8	9239	879	868	868
q9	3427	2381	2324	2324
q10	6526	846	706	706
q11	417	195	178	178
q12	630	257	195	195
q13	18116	1513	1147	1147
q14	161	148	139	139
q15	q16	430	396	365	365
q17	1331	885	799	799
q18	3098	2214	2236	2214
q19	1266	892	796	796
q20	392	295	199	199
q21	5592	1574	1854	1574
q22	324	268	233	233
Total cold run time: 93921 ms
Total hot run time: 16584 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3389	3318	3311	3311
q2	515	399	377	377
q3	2166	2347	2156	2156
q4	1176	1152	884	884
q5	2155	2084	2074	2074
q6	172	124	84	84
q7	1017	915	883	883
q8	1579	1397	1410	1397
q9	3109	3093	3076	3076
q10	1835	1768	1581	1581
q11	357	266	259	259
q12	456	428	342	342
q13	1480	1522	1149	1149
q14	184	180	157	157
q15	q16	380	393	356	356
q17	3604	3264	3301	3264
q18	4784	4361	4704	4361
q19	864	818	861	818
q20	983	971	832	832
q21	3773	3010	3245	3010
q22	395	339	326	326
Total cold run time: 34373 ms
Total hot run time: 30697 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 81310 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 340cc23f08811249e9fbbe29aacd23d7ff316554, data reload: false

query5	4264	411	331	331
query6	385	135	120	120
query7	4965	424	231	231
query8	320	133	114	114
query9	8682	2868	2870	2868
query10	396	209	176	176
query11	5366	1026	934	934
query12	116	70	68	68
query13	1183	447	316	316
query14	6096	2193	2080	2080
query14_1	1966	1935	1943	1935
query15	168	119	110	110
query16	951	372	366	366
query17	805	447	382	382
query18	2337	330	238	238
query19	166	136	113	113
query20	71	69	68	68
query21	216	100	85	85
query22	5234	5178	5102	5102
query23	6528	6206	5960	5960
query23_1	6071	6133	5880	5880
query24	7298	1092	755	755
query24_1	798	768	762	762
query25	464	307	262	262
query26	1233	219	130	130
query27	2799	425	261	261
query28	4676	1493	1492	1492
query29	982	442	360	360
query30	259	154	133	133
query31	837	395	330	330
query32	127	72	78	72
query33	461	226	179	179
query34	1055	813	476	476
query35	402	402	332	332
query36	571	577	562	562
query37	127	78	75	75
query38	1022	841	837	837
query39	514	479	481	479
query39_1	478	476	450	450
query40	208	93	83	83
query41	61	56	56	56
query42	75	72	71	71
query43	240	244	211	211
query44	1047	545	549	545
query45	107	108	102	102
query46	753	854	532	532
query47	754	761	710	710
query48	322	333	233	233
query49	538	228	181	181
query50	792	268	185	185
query51	8123	8079	8056	8056
query52	73	67	62	62
query53	198	197	149	149
query54	228	166	149	149
query55	70	57	55	55
query56	193	158	152	152
query57	752	663	640	640
query58	204	153	168	153
query59	1200	1203	1096	1096
query60	240	190	177	177
query61	113	116	135	116
query62	381	225	167	167
query63	175	143	135	135
query64	2771	694	627	627
query65	1653	1612	1544	1544
query66	1888	267	196	196
query67	9862	9628	9606	9606
query68	2769	1254	767	767
query69	352	213	205	205
query70	663	613	603	603
query71	254	183	165	165
query72	2315	1736	1530	1530
query73	681	633	322	322
query74	1564	1212	1127	1127
query75	1163	1093	952	952
query76	2287	747	531	531
query77	249	257	211	211
query78	3805	3727	3167	3167
query79	1543	863	593	593
query80	1192	319	266	266
query81	491	153	133	133
query82	714	121	94	94
query83	283	211	188	188
query84	292	107	89	89
query85	818	358	299	299
query86	416	213	169	169
query87	994	956	879	879
query88	2799	2118	2065	2065
query89	288	198	171	171
query90	1978	126	129	126
query91	132	115	99	99
query92	74	61	68	61
query93	1350	1103	720	720
query94	691	235	203	203
query95	515	315	227	227
query96	824	604	268	268
query97	1039	1020	1087	1020
query98	148	132	135	132
query99	432	348	305	305
Total cold run time: 176018 ms
Total hot run time: 81310 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.37 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 340cc23f08811249e9fbbe29aacd23d7ff316554, data reload: false

query1	0.01	0.00	0.01
query2	0.08	0.04	0.04
query3	0.26	0.11	0.10
query4	1.59	0.10	0.10
query5	0.17	0.15	0.16
query6	1.18	0.69	0.69
query7	0.03	0.01	0.00
query8	0.05	0.03	0.03
query9	0.30	0.21	0.22
query10	0.35	0.34	0.36
query11	0.16	0.11	0.12
query12	0.14	0.11	0.12
query13	0.30	0.31	0.30
query14	0.44	0.45	0.44
query15	0.36	0.35	0.35
query16	0.21	0.23	0.23
query17	0.69	0.75	0.62
query18	0.18	0.17	0.17
query19	1.23	1.07	1.10
query20	0.02	0.01	0.01
query21	15.43	0.16	0.11
query22	5.06	0.05	0.04
query23	16.12	0.24	0.10
query24	3.06	0.31	0.26
query25	0.10	0.03	0.04
query26	0.78	0.17	0.11
query27	0.04	0.04	0.03
query28	3.54	0.60	0.29
query29	12.47	3.12	2.55
query30	0.27	0.11	0.12
query31	2.76	0.37	0.18
query32	3.46	0.31	0.24
query33	1.49	1.35	1.56
query34	15.37	2.16	1.71
query35	1.72	1.68	1.72
query36	0.47	0.29	0.28
query37	0.06	0.04	0.04
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.11	0.07	0.07
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 90.27 s
Total hot run time: 14.37 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.99% (34432/45309)
Line Coverage 60.96% (387745/636081)
Region Coverage 57.10% (325426/569961)
Branch Coverage 57.89% (148388/256306)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 62.73% (29344/46781)
Line Coverage 47.70% (306910/643444)
Region Coverage 43.35% (248016/572144)
Branch Coverage 44.89% (115374/257039)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.99% (34432/45309)
Line Coverage 60.96% (387739/636081)
Region Coverage 57.09% (325385/569961)
Branch Coverage 57.89% (148376/256306)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.00% (34433/45309)
Line Coverage 60.95% (387713/636081)
Region Coverage 57.08% (325346/569961)
Branch Coverage 57.89% (148366/256306)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.99% (34432/45309)
Line Coverage 60.95% (387690/636081)
Region Coverage 57.06% (325224/569961)
Branch Coverage 57.88% (148352/256306)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.00% (34434/45309)
Line Coverage 60.97% (387797/636081)
Region Coverage 57.06% (325237/569961)
Branch Coverage 57.88% (148360/256306)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants