[Bug] be/test does not compile on macOS: Linux-only APIs and Darwin type identity in recently added tests #68404

Activity

1. Doris-Breakwater commented on Sep 23, 2026

   

   Doris-Breakwater

   on Sep 23, 2026

   Collaborator

   More actions

   Initial maintainer triage

   Assessment: valid and actionable build-portability regression (high confidence). I inspected the exact reported revision 1a3772ff041630444eab49b98eaccbc90fabbe8e. I could not execute a Darwin build on this Linux host, but the reported diagnostics match the source and Darwin ABI/API differences. The impact is currently limited to building the monolithic doris_be_test target on macOS; the production BE build is not implicated.

   The issue is open, unassigned, and currently has no labels.

   Verified facts

   • be/test/CMakeLists.txt glob-adds these sources to one doris_be_test executable, while .github/workflows/be-ut-mac.yml configures MAKE_TEST=OFF. Therefore any one compile failure blocks every filtered local UT run, and current macOS CI cannot detect it.
   • On an arm64 Darwin target, Clang defines int64_t/uint64_t as long long/unsigned long long, while size_t is unsigned long. This directly explains the printf, initializer-list, and std::min failures. Use PRIi64/PRIu64 (or type-safe formatting) rather than hard-coding %lld/%llu, which would merely move the mismatch to platforms where int64_t is long.
   • snii_vs_v3_benchmark_test.cpp unconditionally calls Linux-only posix_fadvise(..., POSIX_FADV_DONTNEED), and passes unsigned char* to Darwin's mincore(..., char*) signature.
   • The libc++ heterogeneous-lookup error is consistent with the test's Equal: it has (string_view, uint32_t) but not (uint32_t, string_view). The production OwnedVocabEq in spimi_term_buffer.h already provides both directions and is the appropriate local pattern.
   • ip_address_cidr.h uses std::from_chars but does not include <charconv>; successful Linux builds currently depend on a transitive include.
   • The Faiss failure has a concrete CMake dependency-path explanation. function_map_inner_product_test.cpp includes function_inner_product.h, which includes function_array_distance.h and therefore Faiss headers. On non-Apple builds, doris_be_test links the vector_search_test target and inherits its transitive ann_index -> faiss usage requirements. The Apple branch instead links $<TARGET_FILE:vector_search_test> via -force_load; that archive path does not propagate target usage requirements to compilation of doris_be_test.
   • The __COUNTER__ error is separate and not macOS-specific. The token is defined in runtime_profile.h; it is expanded by SCOPED_TIMER/SCOPED_RAW_TIMER call sites in the two reported headers. New Clang versions diagnose that extension through -Wcounter-extension under -Wpedantic, and Doris promotes it with -Werror.

   Recommended direction

   1. Prefer portable source fixes for the integer formatting/types, initializer lists, std::min, symmetric comparator, and direct <charconv> include. These are small correctness/build-hygiene fixes and preserve useful macOS coverage.
   2. Treat snii_vs_v3_benchmark_test.cpp specially. A Darwin no-op around posix_fadvise would compile but would violate the benchmark's explicit “cold page cache” contract. Either implement a verified Darwin-equivalent cold-read path (and keep the existing mincore residency check meaningful), or exclude only this disabled benchmark TU on macOS.
   3. Fix the Faiss dependency at the target/CMake level rather than excluding function_map_inner_product_test.cpp. The Apple whole-archive linkage should retain vector_search_test/ann_index/faiss usage requirements, or doris_be_test should receive the required dependency explicitly.
   4. Handle __COUNTER__ as a toolchain-compatibility item, ideally with narrowly scoped handling of -Wcounter-extension for Clang rather than relaxing -Wpedantic globally.
   5. Add at least a compile-only doris_be_test step to macOS CI. Running all UTs can remain a separate policy decision, but compiling the target is needed to prevent the same class of regression.

   Missing information / acceptance checks

   No additional information is required to accept and assign this issue. If a maintainer cannot reproduce the two configuration-sensitive failures, the only useful additions would be the corresponding ninja -v compile commands for the Faiss and __COUNTER__ TUs.

   Before closing, verify a clean ./run-be-ut.sh -j 4 build on arm64 macOS with the reported Clang 23 toolchain, run representative filters for the enabled affected tests, and confirm the Linux BE UT build remains clean. The author's prepared patch is a reasonable starting point, but the cold-cache semantics and Apple CMake usage-requirement propagation should be reviewed as above.

   Breakwater-GitHub-Analysis-Slot: slot_088b0c303f7b
2. robll-v1 commented on Sep 23, 2026

   

   robll-v1

   on Sep 23, 2026

   ContributorAuthor

   More actions

   Thanks for the triage — confirming the two configuration-sensitive failures with concrete evidence, plus one correction.

   1) Faiss include on Apple: confirmed, and it is the -force_load code path

   be/test/CMakeLists.txt:168-180 is exactly as described: the Apple branch passes the archive as a raw link flag

   if (APPLE)
       target_link_libraries(doris_be_test ${TEST_LINK_LIBS}
           -Wl,-force_load,$<TARGET_FILE:vector_search_test>)
       add_dependencies(doris_be_test vector_search_test)
   else()
       target_link_libraries(doris_be_test ${TEST_LINK_LIBS}
           -Wl,--whole-archive vector_search_test -Wl,--no-whole-archive)
   endif()

   so vector_search_test's INTERFACE usage requirements (ann_index -> faiss include dirs) never reach doris_be_test compilation on Apple, while the non-Apple branch passes vector_search_test as a target name and does propagate them.

   The failing TU's recorded command (from be/ut_build_ASAN/compile_commands.json created by run-be-ut.sh) abridges to:

   ... -I<repo>/be/src -I<repo>/be/test ... -isystem <repo>/thirdparty/installed/include ...
   

   with no faiss include directory anywhere, matching that explanation.

   2) __COUNTER__: tag name and blast radius

   With clang 23.1.1 the diagnostic is -Wc2y-extensions on this revision, not -Wcounter-extension:

   $ clang++ -std=gnu++20 -Wpedantic -Werror -c counter.cpp
   counter.cpp:1:18: error: '__COUNTER__' is a C2y extension [-Werror,-Wc2y-extensions]
   
   $ clang++ -std=gnu++20 -Wpedantic -Werror -Wno-counter-extension -c counter.cpp
   error: unknown warning option '-Wno-counter-extension'; did you mean '-Wno-openacc-extension'? [-Werror,-Wunknown-warning-option]
   

   Also, this is not limited to be/test: with clang 23 the production BE build fails the same way — ./build.sh --be stops at be/src/runtime/memory/lru_cache_policy.h:177 and be/src/storage/segment/variant/hierarchical_data_iterator.h:174 (SCOPED_TIMER/SCOPED_RAW_TIMER expand __COUNTER__), and a syntax-only sweep of be/src reports ~642 sites in total.

   One implementation note for whoever fixes it: adding -Wno-* to CMAKE_CXX_FLAGS/EXTRA_CXX_FLAGS is ineffective, because be/CMakeLists.txt appends -Wpedantic later on the command line and that re-enables the diagnostic group (last match wins). A #pragma clang diagnostic ignored or a flag appended after the project's -Wpedantic is required.

   3) snii_vs_v3_benchmark_test.cpp

   Agreed — a Darwin no-op posix_fadvise would silently invalidate the benchmark's cold-page-cache contract, so it is not an acceptable upstream fix. I only used that shim locally as a temporary probe to enumerate the remaining failing TUs. Excluding this disabled benchmark on macOS (or implementing a verified Darwin cold-read path) is the right call.

   4) macOS CI

   Also agree that at least a compile-only doris_be_test step in the macOS workflow would be worth it — that is the only thing that would have caught this class of regression, given the job deliberately sets -DMAKE_TEST=OFF since #65796.

   Happy to prepare a PR along the lines of your option 1 (PRIu64-style formatting, symmetric comparator mirroring OwnedVocabEq in spimi_term_buffer.h, explicit std::min<uint64_t>, #include <charconv> in ip_address_cidr.h), and to either fix the Apple -force_load linkage or exclude snii_vs_v3_benchmark_test.cpp on macOS as you prefer.
3. robll-v1 commented on Oct 9, 2026

   

   robll-v1

   on Oct 9, 2026

   ContributorAuthor

   More actions

   This is fixed by #68520 (merged 2026-09-28): be/test now builds and runs on macOS —
   the seven TUs reported here are covered, plus run-be-ut.sh now initializes the
   contrib/faiss and contrib/openblas submodules it needs. Verified locally on macOS 27
   (arm64) with a clang-23 toolchain; the only remaining local diagnostics are clang-23
   -Werror items that need a newer toolchain-specific suppression, not code changes.
   Suggest closing this issue.