Skip to content

build(deps): validate shared third-party patch state - #570

Merged
lxy-9602 merged 7 commits into
apache:mainfrom
zjw1111:codex/thirdparty-patch-state
Nov 19, 2026
Merged

lxy-9602 merged 7 commits into
apache:mainfrom
zjw1111:codex/thirdparty-patch-state

Conversation

@zjw1111

@zjw1111 zjw1111 commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Purpose

Linked issue: close #569

A dependency version or patch change can currently reuse sources and installed libraries marked successful by an older build. Introduce shared validation of source versions, archive checksums, ordered patches, and registered build inputs. Reject changed or incomplete state with explicit cleanup paths; preserve incremental builds when inputs are unchanged and never delete files automatically.

Migrate Arrow, ORC, Jieba, OSS SDK v2, and Vortex to the shared mechanism. Align ORC and Vortex build directories with the other dependencies and remove PAIMON_VORTEX_SOURCE_DIR. Use lsdiff from patchutils to enumerate patched files and remove PAIMON_OPTIONAL_INSTALL so installation requires built artifacts.

Tests

The patch lifecycle test script is kept locally and is not included in this PR. Local checks with Unix Makefiles and Ninja cover patch content changes with preserved timestamps; patch additions, removal, and ordering; source and build input changes; partial patch failures and process interruption; build/install retry; inconsistent completion records; legacy artifacts and cleanup paths; source damage and direct-step/install guards; directory migration; and unchanged-build reuse.

Additional cases cover file additions/deletions, unsafe patch paths, subproject registration, cleanup followed by a clean rebuild, and the Vortex adapter's Cargo working directory and output layout using simulated Cargo. Installation checks validate completion state rather than individual installed files.

Regression cases also cover multi-file diffs with unmatched brackets and header-like hunk content; missing or failing lsdiff; local directory URLs and archive checksums; truncated verification manifests; and rejecting clean-first or source URL changes before overwriting patched sources.

Manual verification with the real ORC patch

Built and installed ORC 2.1.1 using cmake_modules/orc.diff in an isolated build directory. Each build used the same command:

cmake --build /tmp/paimon-orc-real-1rez0ug8/build --target orc_ep -j 4

After the first successful build, changed an added comment in orc.diff that is applied to MemoryPool.hh. The next build failed during CMake regeneration, before changing the existing library or installed header. Actual error output (call stack omitted):

orc_ep: Source version, patches/order, or build inputs changed.  Previous
identity 1b47193abb0b4257a7e97d279ed4d73acd7fcfcd79912138a805e2c294fb843b,
requested identity
48d0bc3e12c0d5d9af2a810d89eba4316d1e87760a2c3b6eeac7e753d12164c6

Do not remove only .patched or a patch stamp.  Remove these build-owned
paths, then re-run CMake and build:

  /tmp/paimon-orc-real-1rez0ug8/build/orc_ep-prefix/src/orc_ep
  /tmp/paimon-orc-real-1rez0ug8/build/orc_ep-prefix/src/orc_ep-build
  /tmp/paimon-orc-real-1rez0ug8/build/orc_ep-prefix/src/orc_ep-stamp
  /tmp/paimon-orc-real-1rez0ug8/build/orc_ep-prefix/tmp
  /tmp/paimon-orc-real-1rez0ug8/build/orc_ep-install
  /tmp/paimon-orc-real-1rez0ug8/build/CMakeFiles/orc_ep-complete
  /tmp/paimon-orc-real-1rez0ug8/build/paimon-thirdparty/orc_ep

Verified source archives may be retained.  No files were deleted
automatically.

Kept the modified patch, deleted exactly the seven reported paths, then reconfigured and rebuilt successfully. The top-level CMake cache and downloaded source archive were retained. The rebuild ended with:

-- Installing: /tmp/paimon-orc-real-1rez0ug8/build/orc_ep-install/lib/liborc.a
-- Installing: /tmp/paimon-orc-real-1rez0ug8/build/orc_ep-install/lib/cmake/orc/orcTargets.cmake
-- Installing: /tmp/paimon-orc-real-1rez0ug8/build/orc_ep-install/lib/cmake/orc/orcTargets-debug.cmake
[9/10] Performing paimon-verify step for 'orc_ep'
[10/10] Completed 'orc_ep'

Verified that both the extracted source and installed MemoryPool.hh contained the changed comment, and that patch and installation completion records matched the new identity. Another unchanged build ran only the guard and preserved source and artifact contents and timestamps. Restored orc.diff to its original contents after the experiment.

API and Format

No public C++ API, storage format, or protocol changes. PAIMON_VORTEX_SOURCE_DIR and PAIMON_OPTIONAL_INSTALL are removed; patched dependencies use checksum-verified source archives and single-configuration CMake generators. Existing dependency builds may require the cleanup reported by the new checks.

Documentation

Update the build guide with the patchutils requirement and dependency reuse, cleanup, and retry instructions. Install patchutils explicitly in the CI build matrix, GCC 8 and release-candidate jobs, and in both Ubuntu and CentOS development containers.

Generative AI tooling

Generated-by: OpenAI Codex (codex-cli 0.167.0)

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lxy-9602
lxy-9602 merged commit f450d0a into apache:main Nov 19, 2026
31 of 34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Validate third-party patch state before reusing dependency builds

2 participants