Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion .github/actions/setup-ccache/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,12 @@ runs:
steps:
- name: Install ccache
shell: bash
run: sudo apt-get update && sudo apt-get install -y ccache
run: |
if [[ "${RUNNER_OS}" == "macOS" ]]; then
brew install ccache
else
sudo apt-get update && sudo apt-get install -y ccache
fi

- name: Restore ccache
uses: actions/cache@v4
Expand Down
41 changes: 41 additions & 0 deletions .github/actions/setup-release-assets/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

name: Prepare release assets
description: Cache and verify pinned GitHub release archives before building
runs:
using: composite
steps:
- uses: actions/cache@v4
with:
path: ${{ runner.temp }}/paimon-release-assets
key: release-assets-v1-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('third_party/versions.txt', 'ci/scripts/setup_rust.sh') }}
- shell: bash
run: |
set -eu
source third_party/versions.txt
cache_dir="${RUNNER_TEMP}/paimon-release-assets"
for entry in "${DEPENDENCIES[@]}"; do
read -r var archive url <<< "${entry}"
case "${var}" in
PAIMON_ZLIB_URL|PAIMON_ZSTD_URL|PAIMON_PROTOBUF_URL|PAIMON_ARROW_URL)
checksum_var="${var%_URL}_BUILD_SHA256_CHECKSUM"
bash ci/scripts/download_release_asset.sh "${url}" "${cache_dir}/${archive}" "${!checksum_var}"
echo "${var}=${cache_dir}/${archive}" >> "${GITHUB_ENV}"
;;
esac
done
echo "PAIMON_RELEASE_ASSET_DIR=${cache_dir}" >> "${GITHUB_ENV}"
104 changes: 104 additions & 0 deletions .github/workflows/build_and_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -43,13 +43,96 @@ jobs:
- name: Run CMake module tests
shell: bash
run: ci/scripts/test_cmake_modules.sh
- name: Test release asset downloads
shell: bash
run: bash ci/scripts/test_download_release_asset.sh
- name: Run packaging argument tests
shell: bash
run: ci/scripts/test_packaging_args.sh
- name: Run asan_symbolize tests
shell: bash
run: ci/scripts/test_asan_symbolize.sh

macos-build-and-test:
name: ${{ matrix.name }}
runs-on: macos-15
timeout-minutes: 180
strategy:
fail-fast: false
matrix:
include:
- name: appleclang-release-arm64
cc: /usr/bin/clang
cxx: /usr/bin/clang++
build_type: Release
- name: appleclang-debug-arm64
cc: /usr/bin/clang
cxx: /usr/bin/clang++
build_type: Debug
- name: gcc15-release-macos-arm64
cc: /opt/homebrew/bin/gcc-15
cxx: /opt/homebrew/bin/g++-15
Comment on lines +72 to +74

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The GCC macOS entry also needs C++ dependencies built against GCC's standard library. Dependency resolution remains AUTO, and this job's configure log already selects Homebrew's system fmt. That library uses libc++, whereas g++-15 uses libstdc++. After supplying the include path, a minimal fmt::format("value={}", 42) program reproduces a link failure for fmt::v12::vformat[abi:cxx11] with GCC; the same program links with AppleClang. Thus fixing the missing header alone still leaves this entry unable to link. Please explicitly build fmt, and any other selected C++ binary dependencies, with the selected GCC toolchain rather than consuming incompatible Homebrew bottles.

build_type: Release
gcc_formula: gcc@15
env:
CC: ${{ matrix.cc }}
CXX: ${{ matrix.cxx }}
PAIMON_BUILD_JOBS: '3'
CARGO_BUILD_JOBS: '3'
steps:
- name: Checkout paimon-cpp
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
with:
lfs: true
- name: Install build dependencies
shell: bash
run: |
brew install cmake ninja protobuf openssl@3 curl
echo "$(brew --prefix protobuf)/bin" >> "$GITHUB_PATH"
echo "CMAKE_PREFIX_PATH=$(brew --prefix openssl@3):$(brew --prefix curl)" >> "$GITHUB_ENV"
- name: Install GCC
if: matrix.gcc_formula
shell: bash
env:
GCC_FORMULA: ${{ matrix.gcc_formula }}
run: brew install "${GCC_FORMULA}"
- name: Show compiler versions
shell: bash
run: |
"${CC}" --version
"${CXX}" --version
- name: Setup ccache
uses: ./.github/actions/setup-ccache
with:
cache-key-prefix: ccache-${{ matrix.name }}
- name: Prepare release assets
uses: ./.github/actions/setup-release-assets
- name: Install Rust and protobuf tools
shell: bash
run: ci/scripts/setup_rust.sh
- name: Install flatc and libclang (Vortex)
shell: bash
run: ci/scripts/setup_flatc.sh
- name: Build, test and verify installation
shell: bash
run: |
ci/scripts/build_paimon.sh \
--source_dir "$(pwd)" \
--build_type "${{ matrix.build_type }}" \
--install_smoke
Comment on lines +119 to +122

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All three new macOS jobs fail before reaching CTest or the installation smoke test. In the current run, compilation of paimon_file_index_objlib fails with fmt/format.h not found. Configuration selects system fmt, but that object target does not receive its include directory: add_paimon_lib() applies STATIC_LINK_LIBS to the final libraries, and the internal paimon_shared dependency does not propagate compile usage requirements to this object target. Please wire fmt's usage requirements into the object-library compilation, or explicitly select bundled fmt for these jobs, and verify that the actual macOS matrix reaches its tests.

- name: Upload test logs
if: failure()
uses: actions/upload-artifact@v4
with:
name: ${{ matrix.name }}-test-logs
path: |
build/build/test-logs/
build/Testing/Temporary/
build/CMakeFiles/CMakeConfigureLog.yaml
- name: Show ccache statistics
if: always()
run: ccache -s

build-and-test:
name: ${{ matrix.name }}
runs-on: ${{ matrix.runner || 'ubuntu-24.04' }}
Expand Down Expand Up @@ -88,6 +171,12 @@ jobs:
cc: gcc-14
cxx: g++-14
build_args: --build_type Release
- name: gcc13-release-aarch64
runner: ubuntu-24.04-arm
cc: gcc-13
cxx: g++-13
gcc_version: '13.3.0'
build_args: --build_type Release
Comment on lines +174 to +179

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The GCC 13.3 ARM64 regression job still cannot compile the current head. Its build log fails in src/paimon/rest/rest_token_file_system_test.cpp:638 at std::optional<std::shared_ptr<FileSystem>> cached = cache->GetIfPresent(token);, with the same -Werror=maybe-uninitialized diagnostic on the shared-pointer payload. This configuration enables REST, so the optional-initialization fixes in the realtime-store and deletion-vector code leave another build blocker before CTest can run. Please include this remaining initialization in the workaround and rerun the actual GCC 13.3.0 ARM64 Release job with strict warnings retained.

- name: clang-debug-aarch64
runner: ubuntu-24.04-arm
- name: asan-ubsan-aarch64
Expand All @@ -109,13 +198,28 @@ jobs:
uses: ./.github/actions/setup-ccache
with:
cache-key-prefix: ccache-${{ matrix.name }}
- name: Prepare release assets
uses: ./.github/actions/setup-release-assets
- name: Install Rust and protobuf tools
shell: bash
run: ci/scripts/setup_rust.sh
- name: Install build and debugging dependencies
run: |
sudo apt-get update
sudo apt-get install -y gdb libcurl4-openssl-dev libssl-dev
- name: Install and verify regression compiler
if: matrix.gcc_version
shell: bash
env:
CC: ${{ matrix.cc }}
CXX: ${{ matrix.cxx }}
EXPECTED_GCC_VERSION: ${{ matrix.gcc_version }}
run: |
sudo apt-get install -y "${CC}" "${CXX}"
"${CC}" --version
"${CXX}" --version
test "$("${CC}" -dumpfullversion)" = "${EXPECTED_GCC_VERSION}"
test "$("${CXX}" -dumpfullversion)" = "${EXPECTED_GCC_VERSION}"
- name: Install flatc and libclang (Vortex)
shell: bash
run: ci/scripts/setup_flatc.sh
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/gcc8_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,8 @@ jobs:
ls -la
- name: Checkout paimon-cpp
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
- name: Prepare release assets
uses: ./.github/actions/setup-release-assets
- name: Install Rust and protobuf tools
shell: bash
run: ci/scripts/setup_rust.sh
Expand Down
39 changes: 37 additions & 2 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -345,11 +345,46 @@ endif()

if(PAIMON_ENABLE_LANCE)
find_program(PAIMON_LANCE_PROTOC protoc REQUIRED)
corrosion_import_crate(MANIFEST_PATH crates/lance_ffi/Cargo.toml CRATES
paimon_lance_ffi)
set(PAIMON_LANCE_CORROSION_OPTIONS)
if(APPLE)
# Rust's host tools and Apple's native crypto dependencies require Apple's toolchain.
list(APPEND PAIMON_LANCE_CORROSION_OPTIONS NO_LINKER_OVERRIDE)
endif()
corrosion_import_crate(MANIFEST_PATH
crates/lance_ffi/Cargo.toml
CRATES
paimon_lance_ffi
${PAIMON_LANCE_CORROSION_OPTIONS})
corrosion_experimental_cbindgen(TARGET paimon_lance_ffi HEADER_NAME
paimon_lance_ffi.h)
corrosion_set_env_vars(paimon_lance_ffi "PROTOC=${PAIMON_LANCE_PROTOC}")
if(APPLE)
target_link_libraries(paimon_lance_ffi INTERFACE "-framework Security"
"-framework CoreFoundation")
find_program(PAIMON_LANCE_CLANG clang
PATHS /usr/bin
NO_DEFAULT_PATH REQUIRED)
find_program(PAIMON_LANCE_CLANGXX clang++
PATHS /usr/bin
NO_DEFAULT_PATH REQUIRED)
# cc-rs gives the hyphenated target variables priority over Corrosion's defaults.
# Preserve Apple's crypto-capable CPU baseline for these native dependencies.
corrosion_set_env_vars(paimon_lance_ffi
"CC_${Rust_CARGO_TARGET_CACHED}=${PAIMON_LANCE_CLANG}"
"CXX_${Rust_CARGO_TARGET_CACHED}=${PAIMON_LANCE_CLANGXX}"
"CFLAGS="
"CXXFLAGS="
"CFLAGS_${Rust_CARGO_TARGET_CACHED}="
"CXXFLAGS_${Rust_CARGO_TARGET_CACHED}=")
if(CMAKE_OSX_SYSROOT)
corrosion_set_env_vars(paimon_lance_ffi "SDKROOT=${CMAKE_OSX_SYSROOT}")
endif()
if(CMAKE_OSX_DEPLOYMENT_TARGET)
corrosion_set_env_vars(paimon_lance_ffi
"MACOSX_DEPLOYMENT_TARGET=${CMAKE_OSX_DEPLOYMENT_TARGET}"
)
endif()
endif()
endif()

if(PAIMON_ENABLE_LUCENE)
Expand Down
6 changes: 6 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,12 @@ Before submitting, please verify:
- **CMake** ≥ 3.22
- **Python 3** (for linting scripts and pre-commit)

See the [build guide](docs/source/building.rst) for platform requirements and
the CI compiler matrix, feature flags, and platform-specific exclusions.
The CI entry point is `ci/scripts/build_paimon.sh`; it enables the available
optional modules and runs CTest. It defaults to Debug; use
`--build_type Release` to reproduce a Release configuration.

---

## Code Style
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,8 @@ Paimon C++ currently provides:

Building requires a C++17-compatible compiler and CMake 3.22 or newer.
Linux `x86_64` and `aarch64` builds are supported and verified in CI.
CI also includes macOS 15 ARM64 builds with AppleClang in Debug and Release,
and Homebrew GCC 15 in Release.
See the [build guide](docs/source/building.rst) for platform support, system requirements,
dependency configuration, optional components, and instructions for building and running tests.

Expand Down
3 changes: 2 additions & 1 deletion build_support/asan_symbolize.py
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,8 @@ def open_atos(self):
self.pipe = subprocess.Popen(cmdline,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
stderr=subprocess.PIPE,
encoding='utf-8', errors='surrogateescape')

def symbolize(self, addr, binary, offset):
"""Overrides Symbolizer.symbolize."""
Expand Down
7 changes: 5 additions & 2 deletions ci/scripts/build_paimon.sh
Original file line number Diff line number Diff line change
Expand Up @@ -145,9 +145,10 @@ if [[ "${enable_tsan}" == "true" ]]; then
fi
# CI always builds natively, so the host architecture is the target architecture.
host_arch=$(uname -m)
if [[ "${host_arch}" != "x86_64" ]]; then
if [[ "$(uname -s)" != "Linux" ||
( "${host_arch}" != "x86_64" && "${host_arch}" != "aarch64" && "${host_arch}" != "arm64" ) ]]; then
ENABLE_LUMINA="OFF"
echo "=== Lumina disabled: no prebuilt artifacts for ${host_arch} ==="
echo "=== Lumina disabled: no prebuilt artifacts for $(uname -s)-${host_arch} ==="
fi
if [[ "${verify_full_text_archives}" == "true" && "${ENABLE_FULL_TEXT}" != "ON" ]]; then
echo "--verify_full_text_archives requires the full-text index, which this build disables" >&2
Expand All @@ -158,6 +159,8 @@ CMAKE_ARGS=(
"-G Ninja"
"-DCMAKE_BUILD_TYPE=${build_type}"
"-DPAIMON_BUILD_TESTS=ON"
"-DPAIMON_ENABLE_AVRO=ON"
"-DPAIMON_ENABLE_ORC=ON"
"-DPAIMON_ENABLE_MOSAIC=ON"
"-DPAIMON_ENABLE_LANCE=${ENABLE_LANCE}"
"-DPAIMON_ENABLE_VORTEX=ON"
Expand Down
54 changes: 54 additions & 0 deletions ci/scripts/download_release_asset.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Fetch a pinned GitHub release asset. Use the release API if the browser
# download endpoint times out, and publish a file only after verification.
set -euo pipefail

url=$1
output=$2
sha256=$3

verify() {
if command -v sha256sum >/dev/null; then
echo "${sha256} $1" | sha256sum -c -
else
echo "${sha256} $1" | shasum -a 256 -c -
fi
}

if [[ -f "${output}" ]] && verify "${output}"; then
exit 0
fi

mkdir -p "$(dirname "${output}")"
tmp=$(mktemp "${output}.XXXXXX")
trap 'rm -f "${tmp}"' EXIT
curl_args=(--proto '=https' --tlsv1.2 -fsSL --retry 3 --connect-timeout 15 --max-time 180)
if ! curl "${curl_args[@]}" "${url}" -o "${tmp}" || ! verify "${tmp}"; then
release=${url#https://github.com/}
repo=${release%%/releases/download/*}
asset=${release##*/}
tag=${release#*/releases/download/}
tag=${tag%/*}
asset_url=$(curl "${curl_args[@]}" "https://api.github.com/repos/${repo}/releases/tags/${tag}" |
python3 -c 'import json, sys
print(next(a["url"] for a in json.load(sys.stdin)["assets"] if a["name"] == sys.argv[1]))' "${asset}")
curl "${curl_args[@]}" -H 'Accept: application/octet-stream' "${asset_url}" -o "${tmp}"
verify "${tmp}"
fi
mv -f "${tmp}" "${output}"
Loading
Loading