Repository navigation
[doc](build) Install LLVM 22 and LLVM 20 for building Doris on macOS - #4180
Merged
Merged
Conversation
apache/doris#68595 moves the macOS BE compiler to llvm@22, the first LLVM whose AddressSanitizer runtime starts on macOS 26.4 and later, and keeps the third-party libraries on llvm@20, which thirdparty/build-thirdparty.sh now requires on macOS. The macOS build guide installed only llvm@20. Following it after that change, env.sh finds no llvm@22 and the BE silently builds with Apple clang, while a guide that listed only llvm@22 would make every third-party source build stop at the llvm@20 check. Install both, with a note on what each is for, in the English and Chinese guides. Also drop `export PATH="/opt/homebrew/opt/llvm/bin:$PATH"`: it names the unversioned llvm formula, which the guide does not install, and env.sh puts its own LLVM and /usr/bin ahead of the user's PATH, so the line never affected which compiler the build uses. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
yiguolei
pushed a commit
to apache/doris
that referenced
this pull request
Sep 30, 2026
…OS 26.4+ (#68595) ### What problem does this PR solve? Issue Number: N/A Related PR: apache/doris-website#4180 (the macOS build guide change that goes with this PR) Problem Summary: **Context.** On macOS the only entry point for the BE unit tests is `run-be-ut.sh`, and it defaults to `BUILD_TYPE_UT=ASAN`. Since macOS 26.4 that default cannot start a single process: every binary linked against llvm.org's `libclang_rt.asan_osx_dynamic` deadlocks during runtime initialisation, before `main()`. `env.sh` selects that same toolchain on macOS (`llvm@20`), so a developer who follows the documented setup gets a build that hangs rather than one that fails. How the macOS toolchain gets chosen matters for the rest of this description: - **Selection.** `env.sh` writes `custom_env_mac.sh`, which puts `$(brew --prefix)/opt/<formula>/bin` on `PATH` for every formula in its `CELLARS` list. Unless `DORIS_CLANG_HOME` is already set, `env.sh` then takes the first `clang` on `PATH` and derives `CC`/`CXX` from it. - **Installation.** `CELLARS` only selects; it installs nothing. Developers install Homebrew formulae by following the macOS build guide, and each CI workflow installs from its own `brew install` list. - **Third-party libraries.** `thirdparty/installed` comes from one of two places: - the published `doris-thirdparty-prebuilt-darwin-*.tar.xz`, which `apache/doris-thirdparty`'s `build-target.yml` builds by running `thirdparty/build-thirdparty.sh` from apache/doris master; - a local run of that same script. `build.sh` also starts one by itself when `thirdparty/installed` lacks a library it checks for. **1. The problem, and what it cost** *The deadlock.* `sample` on the stalled process shows the whole chain: ``` __malloc_init (libsystem_malloc) <- early libSystem init calls malloc_default_zone() `- wrap_malloc_default_zone (asan runtime) `- AsanInitFromRtl -> AsanInitInternal -> InitializeShadowMemory `- MemoryRangeIsAvailable -> MemoryMappingLayout::Next -> get_dyld_hdr() `- dyld_shared_cache_iterate_text_swift <- macOS 26.4 reimplemented this in Swift `- _Block_copy -> malloc `- __sanitizer_mz_malloc (asan's own malloc) `- AsanInitFromRtl() <- re-enters init `- StaticSpinMutex::LockSlow <- spins on the lock it already holds ``` The cause is an OS-side change, not a Doris one. It also does not mean ASAN is broken on macOS 26: Apple's own clang sanitizer runtime runs fine on the same host, and only llvm.org's compiler-rt is affected. Upstream fixed it by weak-importing `_dyld_get_dyld_header` and using it, when present, instead of walking the shared cache (llvm/llvm-project#182943, main `2e7d07a`; backport #188913, release/22.x `7b6514c`). **The fix shipped only in 22.1.8.** 20.1.8 is the last 20.x release and 21.1.8 the last 21.x, so no version of `llvm@20` can ever be made to work. *Why it looks like a hang and not an error.* - `be/CMakeLists.txt` reaches `storage/index/ann` unconditionally, and that directory's `cmake-protect` target calls `add_subdirectory()` on `contrib/openblas`. - OpenBLAS runs an instrumented `getarch` probe from its **configure** step (`contrib/openblas/cmake/prebuild.cmake:1513`, `execute_process(COMMAND .../getarch 0 ...)`), and `execute_process` has no timeout. - So every ASAN build stops at `-- Running getarch` and never moves again. It sits at about 90% CPU and prints no diagnostic. `run-be-ut.sh` on macOS was therefore unusable at its default setting, and the failure gave the developer nothing to act on. *The second obstacle, once the toolchain is bumped.* - clang 22 added `-Wc2y-extensions` and folds it into `-Wpedantic`. - `__COUNTER__` only reached the C standard in C2y (WG14 N3457). `be/src/runtime/runtime_profile.h:73-85` uses it to give two `SCOPED_TIMER` / `SCOPED_RAW_TIMER` expansions on the same line distinct names. - As a result, a clang 22 build of any TU that includes that header fails under `-Werror`. Three representative unity TUs were enough to hit it, so the failure is not confined to one module. *The third obstacle: clang 22 cannot build the third-party libraries.* The first revision of this PR moved every macOS toolchain reference to `llvm@22`, including the third-party build. CI failed both macOS third-party jobs (7m49s, 6m23s) on the first library that exercises a new clang 22 error: ``` unixODBC 2.3.7 src/SQLBrowseConnectW.c:424:82: error: incompatible pointer types passing 'SQLSMALLINT *' (aka 'short *') to parameter of type 'int *' [-Wincompatible-pointer-types] ``` clang 16, clang 20 and Apple clang 21 report this as a warning; clang 22 makes it an error. unixODBC is only the *first* failure: the build aborts there, so every library after it is untested against clang 22. The macOS third-party libraries therefore have to stay on `llvm@20` while the BE moves to `llvm@22`. Leaving the third-party build alone does not achieve that, because it sources `env.sh`: - With `CELLARS` naming `llvm@22`, a machine without `llvm@22` gets a non-existent directory on `PATH`. - `command -v clang` then falls through to `/usr/bin/clang` (Apple clang), which is worse than either explicit choice. - The job that publishes the prebuilt archive in `apache/doris-thirdparty` is exactly such a machine: it installs only `llvm@20` and sets no `DORIS_CLANG_HOME`. *Two latent defects clang 22 then surfaced.* The bump is not a pure version change. clang 22's stricter diagnostics stop the build on two pre-existing bugs, and both are worth fixing on their own merits. | # | Site | Diagnostic | What is actually wrong | |---|---|---|---| | 1 | `be/src/load/group_commit/wal/wal_dirs_info.cpp:98` | `-Wunused-result` | `LOG(INFO) << "… err: {}", e.what();` — the `,` is the comma operator, not an argument separator, so the statement is `(LOG(INFO) << "…{}") , (e.what())`. The `{}` is never substituted, and `e.what()` is evaluated and then discarded: **the error message has never been logged**, only the literal `{}`. A repo-wide scan for the same shape finds exactly this one site (`cloud/src/common/logging.h` matches only inside macro definitions and is not a bug). | | 2 | `common/cpp/sync_point.cpp:208,210` | `-Wthread-safety-analysis` | The function holds `std::unique_lock lock(mutex_)`, then releases and re-takes the mutex with a raw `mutex_.unlock()` / `mutex_.lock()` pair around the callback, which bypasses the lock's ownership tracking. If the callback throws, the re-lock is skipped while `~unique_lock` still believes it owns the mutex, so the destructor unlocks a mutex that is not held (UB) and `num_callbacks_running_` is never decremented. `lock.unlock()` / `lock.lock()` still runs the callback unlocked, as intended, and keeps the ownership state correct on every path. | **2. What this PR does, and why it helps** - **`env.sh`**: the macOS `CELLARS` list moves from `llvm@20` to `llvm@22`, with a comment recording that 22.1.8 is the minimum and why. This is the single line that decides which clang a macOS developer builds the BE with. - **`thirdparty/build-thirdparty.sh`**: on Darwin, right after sourcing `env.sh` and next to the existing environment sanitization, the script unconditionally points `DORIS_CLANG_HOME`, `CC` and `CXX` at `$(brew --prefix llvm@20)` and puts its `bin/` first on `PATH`. If `llvm@20` is not installed, it stops with a `brew install llvm@20` hint rather than falling back to another compiler. - The override has to come after `env.sh` and has to be unconditional. `env.sh` sources `custom_env.sh`, which may export `DORIS_CLANG_HOME` for the BE, and `build.sh` has already exported the BE's `llvm@22` values before it starts this script. - Every macOS third-party build runs this script: `build.sh`'s automatic rebuild, a manual run, the pull request check in `build-thirdparty.yml`, and the `apache/doris-thirdparty` job that publishes `doris-thirdparty-prebuilt-darwin-*.tar.xz`. This one place therefore keeps all of them, and the published archive, on `llvm@20`. - `.github/workflows/build-thirdparty.yml` is unchanged: its macOS jobs already install `llvm@20`. - **`be/CMakeLists.txt`**: add `-Wno-c2y-extensions` for clang 19 and newer, in its own `add_compile_options` call **after** `-Wpedantic`. - The position matters: a later `-Wpedantic` turns the group back on. That is also why passing the flag through `EXTRA_CXX_FLAGS` does not work: that variable lands near the front of the command line. - The version gate matters because the `c2y-extensions` group only exists from clang 19 on. clang rejects an unknown `-Wno-` option like any other unknown warning option (`-Wunknown-warning-option`, fatal under `-Werror`), and this file accepts clang 16 and newer. - For clang 19 and newer the command line is exactly what it was. - **`.github/workflows/be-ut-mac.yml`**: install `llvm@22` for the BE and keep `llvm@20` next to it. - The job builds the BE on top of the downloaded darwin-arm64 prebuilt. When that archive lacks a library `build.sh` checks for, `build.sh` rebuilds the third-party libraries from source, and that rebuild now needs `llvm@20`. The archive falls behind like this whenever master changes the set of checked libraries before the next archive is published. - Both formulae are keg-only, so installing both does not change which one the BE uses. - **`be/src/load/group_commit/wal/wal_dirs_info.cpp` and `common/cpp/sync_point.cpp`**: the two fixes from the table above. Each is one line, the smallest change that removes the defect rather than suppressing the warning. No `-Wno-unused-result` / `-Wno-thread-safety-analysis` is added, because those diagnostics point at real bugs and should keep firing. - **apache/doris-website#4180**: the macOS build guide installs `llvm@22` and `llvm@20` instead of only `llvm@20`. Without it, a developer who follows the guide after this PR has no `llvm@22`, and the BE silently builds with Apple clang. What this PR deliberately does not do is add `llvm@20` to `env.sh`'s `CELLARS`: - `CELLARS` installs nothing. - `build-thirdparty.sh` does not use `PATH` to find `llvm@20`. - The `CELLARS` loop prepends each entry to `PATH`. An `llvm@20` entry after `llvm@22` would put the BE back on `llvm@20` and bring the deadlock back. **3. The classes, and how they call each other** ``` env.sh CELLARS := llvm@22 (selects a compiler; installs nothing) |- generates custom_env_mac.sh: $(brew --prefix)/opt/<cellar>/bin prepended to PATH |- DORIS_CLANG_HOME := dirname($(command -v clang))/.. -> CC / CXX / ASAN_SYMBOLIZER_PATH '- is sourced by: |- build.sh, run-be-ut.sh -> BE: llvm@22 '- thirdparty/build-thirdparty.sh:55 '- :100-110 on Darwin: DORIS_CLANG_HOME / CC / CXX := $(brew --prefix llvm@20) llvm@20 missing -> exit 1, "brew install llvm@20" -> third-party libraries: llvm@20 run by: build.sh (library missing from thirdparty/installed), manual runs, build-thirdparty.yml (PR check), doris-thirdparty build-target.yml (publisher) brew install lists, i.e. who installs which LLVM: macOS build guide (apache/doris-website#4180) llvm@22 llvm@20 .github/workflows/be-ut-mac.yml llvm@22 llvm@20 .github/workflows/build-thirdparty.yml llvm@20 apache/doris-thirdparty build-target.yml llvm@20 be/CMakeLists.txt if (COMPILER_CLANG) |- add_compile_options(-Wpedantic ...) <- enables the c2y group '- if (CMAKE_CXX_COMPILER_VERSION >= 19) add_compile_options(-Wno-c2y-extensions) <- must stay after the line above | be/src/runtime/runtime_profile.h:73-85 '- MACRO_CONCAT(SCOPED_TIMER, __COUNTER__) <- the only __COUNTER__ use in be/src, be/test | be/src/storage/index/ann/cmake-protect/CMakeLists.txt:48 '- add_subdirectory(contrib/openblas) '- cmake/prebuild.cmake:1513 execute_process(getarch) <- where the hang surfaced ``` ### Release note None ### Check List (For Author) - Test: Manual test on macOS 26.5.1 (arm64, dyld-1378), plus CI. - `clang -fsanitize=address` hello world exits 0 with llvm@22 and still deadlocks with llvm@20 on the same host. - `sh run-be-ut.sh` with no environment overrides now selects `Clang-22.1.8` and clears the `-- Running getarch` point that previously hung forever: zero FAILED targets, and `doris_be_test` links. - `sh run-be-ut.sh --run --filter=FormatRoundTest.*` starts the ASAN binary and passes 8 tests. - `-Wno-c2y-extensions` gate, tested on a `__COUNTER__` TU under `-Wpedantic -Werror`: clang 16.0.6 rejects the flag (`unknown warning option`), which is why it is gated; clang 20.1.8 accepts it; clang 22.1.8 fails without it and passes with it. - `build-thirdparty.sh` compiler override, checked under `bash -x` with both `custom_env.sh` and `build.sh` exporting the `llvm@22` values: the effective `CC`/`CXX` and the first clang on `PATH` are llvm@20. With `llvm@20` missing, the script exits 1 with the brew hint before building anything. - `be-ut-mac.yml`: the "Build BE" step script passes `bash -n` on bash 3.2 and 5.3, and its `cellars` array holds both `llvm@22` and `llvm@20`. The workflow runs only on pushes to master and on schedule, so this PR cannot run it. - `build-support/check-build-hygiene.sh` passes. - CI: `Build Third Party Libraries (macOS)`, `(macOS-arm64)` and `(Linux)` pass at `a955357a30f` (run 36542107899), which already contains the `build-thirdparty.sh` override. The first revision, which moved these builds to llvm@22, failed both macOS jobs on unixODBC. - Regression test / unit test: N/A (toolchain, flag and workflow change). - Behavior changed: Yes, for macOS builds only. - The BE builds with llvm@22 instead of llvm@20. - The third-party libraries, including the published prebuilt, stay on llvm@20. Building them from source now requires llvm@20 installed next to llvm@22; without it, the build stops with a `brew install llvm@20` hint. - Existing macOS developers need to run `brew install llvm@22`. Without it, `env.sh` falls back to Apple clang in `/usr/bin` for newly configured build directories. - Does this need documentation: Yes, apache/doris-website#4180. --------- Co-authored-by: Claude Code <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
apache/doris#68595 splits the macOS toolchain in two:
llvm@22, whichenv.shselects. LLVM 22.1.8 is the first release whose AddressSanitizer runtime no longer deadlocks at startup on macOS 26.4 and later.llvm@20. clang 22 turns-Wincompatible-pointer-typesinto an error and stops at unixODBC, sothirdparty/build-thirdparty.shrequiresllvm@20on macOS and exits if it is missing.The macOS build guide installs only
llvm@20. After that change, a developer who follows it has nollvm@22:env.shfinds nothing under$(brew --prefix)/opt/llvm@22/bin, and the BE silently builds with Apple clang from/usr/bin. Listing onlyllvm@22would be wrong the other way.build.shrebuilds the third-party libraries from source wheneverthirdparty/installedlacks one it checks for, and that build stops at thellvm@20check. This is not rare: on an Intel Mac today, the darwin-x86_64 prebuilt predates the paimon-rust libraries that master checks for, so every build on master rebuilds from source.Changes
In
community/source-install/compilation-mac.mdand its zh-CN counterpart:brew install … llvm@20 …becomesbrew install … llvm@22 llvm@20 …, followed by a note on what each version is for.export PATH="/opt/homebrew/opt/llvm/bin:$PATH"is removed. It points to the unversionedllvmformula, which the guide does not install. It also never affected which compiler the build uses, becauseenv.shputs its own LLVM directory and/usr/binahead of the user'sPATH. The new note says the build scripts find both LLVMs through Homebrew.Merge order
Merge this together with apache/doris#68595 or after it. Installing both versions before then does no harm, since master's
env.shstill selectsllvm@20, but the new note describes the behavior after #68595.Versions and languages
Validation
git diff --checkis clean, and the English and Chinese pages carry the same content.