Skip to content

fix(release): keep macos pgo instrumentation rust-only - #11129

Merged
jdx merged 1 commit into
mainfrom
claude/fix-macos-pgo
Jul 20, 2026
Merged

jdx merged 1 commit into
mainfrom
claude/fix-macos-pgo

Conversation

@jdx

@jdx jdx commented Jul 20, 2026 •

Copy link
Copy Markdown
Owner

Fixes the build-tarball-macos-arm64 failure that has broken every release since PGO landed in #11031:

LLVM Profile Error: Runtime and instrumentation version mismatch : expected 10, but get 8
ERROR: no .profraw files written to .../target/pgo-data/profraw after training

Root cause

The cc crate mirrors -Cprofile-generate/-Cprofile-use from RUSTFLAGS onto the C compiler command line (-fprofile-generate/-fprofile-use). The macOS tarball builds vendored C code (openssl/vendored, vfox/vendored-lua), so those objects were instrumented by Apple clang, whose profile format follows Xcode's LLVM — not rustc's:

  • release runner: Xcode 16 / Apple clang 16 → compiler-rt raw profile version 8
  • rustc 1.97 (LLVM 22) profiler_builtins → raw profile version 10

At link, the C objects' version-8 __llvm_profile_raw_version won symbol coalescing (verified in the CI-built binary: __TEXT,__const value 0x…08), while the Rust profile runtime doing the writes expects 10 → every profile write aborts → training produces zero .profraw files.

This is why #11071 (toolchain pinning) couldn't fix it — the rogue instrumentation comes from the C compiler, not a mismatched rust toolchain — and why it doesn't reproduce on machines where Xcode's LLVM happens to match rustc's generation (e.g. clang 21 also uses version 10).

Fix

Export -fno-profile-generate -fno-profile-use via CFLAGS/CXXFLAGS on Darwin in scripts/pgo.bash. cc appends environment flags after the rustc-inherited ones, so the overrides win and PGO stays Rust-only. C-side PGO could never work here anyway: Apple clang 16 can't consume profdata merged by rustup's LLVM-22 llvm-profdata, so phase 3b would have failed next.

Linux PGO is untouched (it's green today; gcc's gcov-style flags don't hit this conflict).

Validation

Ran on the actual namespace-profile-endev-macos-arm64 runner via workflow_dispatch on this branch (with the release workflow temporarily trimmed to the macOS tarball path, since tarball jobs don't run on regular PRs):

  • Repro of the bug (pre-fix, with link forensics): run 29762515264 — final binary carries version-8 metadata from the C objects, reproduces the exact error
  • Fix validation (full MISE_PGO=1 scripts/build-tarball.sh aarch64-apple-darwin incl. codesigning): run 29762962943 — ✅ 54 .profraw collected, 84.5 MB merged profile, optimized rebuild, signed tarballs produced

Note: the cargo-release registry failure on the release branch is a separate issue (upstream re-published v1.1.0 with incomplete assets, so /releases/latest resolves to a release with no Linux builds) and is not addressed here.

🤖 Generated with Claude Code


Note

Low Risk
Small, Darwin-only change to release build flags; Linux PGO is untouched and the override is documented to win over inherited profile flags.

Overview
Fixes macOS PGO release builds that failed during training with an LLVM profile runtime/instrumentation version mismatch and zero .profraw files.

On Darwin, scripts/pgo.bash now exports CFLAGS/CXXFLAGS with -fno-profile-generate and -fno-profile-use before the instrumented build. That stops the cc crate from forwarding rustc’s PGO flags to Apple clang for vendored C (e.g. OpenSSL, Lua), so only Rust stays instrumented and profile writes succeed. Linux behavior is unchanged.

Reviewed by Cursor Bugbot for commit 03bdfa3. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by CodeRabbit

  • Bug Fixes
    • Improved macOS profiling reliability by preventing incompatible instrumentation in bundled native components.
    • Resolved an issue that could cause profiling runs to fail when generating runtime profile data during later profiling phases.

The cc crate mirrors -Cprofile-generate/-Cprofile-use from RUSTFLAGS
into C compiler flags, so vendored C code (openssl, lua) was being
instrumented by Apple clang, whose profile-runtime format follows
Xcode's LLVM rather than rustc's. On the release runners (Xcode 16,
raw profile version 8) this clashes with rustc 1.97 (version 10) and
every profile write aborted with:

  LLVM Profile Error: Runtime and instrumentation version mismatch

leaving zero .profraw files and failing build-tarball-macos-arm64 on
every release since PGO landed. Override with -fno-profile-generate/
-fno-profile-use via CFLAGS/CXXFLAGS, which cc appends after the
rustc-inherited flags.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 20, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 59ff8914-fe95-4d6b-bd82-947f3550ef4b

📥 Commits

Reviewing files that changed from the base of the PR and between 6c7b16f and 03bdfa3.

📒 Files selected for processing (1)
  • scripts/pgo.bash

📝 Walkthrough

Walkthrough

The PGO script now detects macOS and disables Rust/cc profiling instrumentation for C and C++ compilation by adding compiler flags to CFLAGS and CXXFLAGS.

Changes

macOS PGO instrumentation

Layer / File(s) Summary
Disable macOS C/C++ profiling instrumentation
scripts/pgo.bash
On Darwin, appends -fno-profile-generate and -fno-profile-use to CFLAGS and CXXFLAGS.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Poem

A rabbit hopped through flags so bright,
“No clang profiles tonight!”
C and C++ tucked in their burrow,
While PGO runs without sorrow.
Ears up—macOS gets it right!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: keeping macOS PGO instrumentation Rust-only during release builds.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@greptile-apps

greptile-apps Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This fix resolves a macOS release-tarball regression introduced when PGO landed: the cc crate was propagating -fprofile-generate/-fprofile-use from RUSTFLAGS to Apple clang, which produces profile data in a different raw-version format than rustc's LLVM, causing every profile write to abort with a version-mismatch error and no .profraw files collected.

  • Adds a Darwin-specific block in scripts/pgo.bash that appends -fno-profile-generate -fno-profile-use to CFLAGS/CXXFLAGS before Phase 1, so the cc crate's C flag forwarding is cancelled out and PGO instrumentation stays Rust-only.
  • The fix is placed before Phase 1 and applies to Phase 3b as well (since the same export persists), correctly suppressing both -fprofile-generate and -fprofile-use propagation across all build phases.

Confidence Score: 5/5

Safe to merge — the change is a minimal, targeted two-line export guarded by uname -s, validated with a full end-to-end run on the actual CI runner.

The fix is two lines, Darwin-only, validated end-to-end on the real runner (54 profraw collected, optimized binary produced). Linux PGO is unaffected. The only finding is a cosmetic leading-space nit.

No files require special attention.

Important Files Changed

Filename Overview
scripts/pgo.bash Adds a Darwin-specific CFLAGS/CXXFLAGS block before Phase 1 to suppress C-side PGO instrumentation, preventing the Apple clang profile-version mismatch that was breaking every macOS release tarball build.

Fix All in Claude Code

Reviews (1): Last reviewed commit: "fix(release): keep macos pgo instrumenta..." | Re-trigger Greptile

Comment thread scripts/pgo.bash
Comment on lines +97 to +98
export CFLAGS="${CFLAGS:-} -fno-profile-generate -fno-profile-use"
export CXXFLAGS="${CXXFLAGS:-} -fno-profile-generate -fno-profile-use"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 When CFLAGS (or CXXFLAGS) is unset, ${CFLAGS:-} expands to an empty string, so the exported value begins with a leading space (e.g., -fno-profile-generate -fno-profile-use). This is harmless for clang/gcc in practice, but the ${VAR:+$VAR } idiom avoids it cleanly.

Suggested change
export CFLAGS="${CFLAGS:-} -fno-profile-generate -fno-profile-use"
export CXXFLAGS="${CXXFLAGS:-} -fno-profile-generate -fno-profile-use"
export CFLAGS="${CFLAGS:+$CFLAGS }-fno-profile-generate -fno-profile-use"
export CXXFLAGS="${CXXFLAGS:+$CXXFLAGS }-fno-profile-generate -fno-profile-use"

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Claude Code

@github-actions

Copy link
Copy Markdown
Contributor

Hyperfine Performance

mise x -- echo

Command Mean [ms] Min [ms] Max [ms] Relative
mise-2026.7.7 x -- echo 23.3 ± 1.5 20.5 30.4 1.00
mise x -- echo 24.1 ± 1.5 19.8 29.4 1.04 ± 0.09

mise env

Command Mean [ms] Min [ms] Max [ms] Relative
mise-2026.7.7 env 24.2 ± 1.5 20.1 31.3 1.03 ± 0.09
mise env 23.5 ± 1.5 19.4 29.0 1.00

mise hook-env

Command Mean [ms] Min [ms] Max [ms] Relative
mise-2026.7.7 hook-env 25.4 ± 1.5 21.3 30.6 1.05 ± 0.10
mise hook-env 24.2 ± 1.7 20.2 30.5 1.00

mise ls

Command Mean [ms] Min [ms] Max [ms] Relative
mise-2026.7.7 ls 22.3 ± 1.6 17.9 29.0 1.04 ± 0.12
mise ls 21.4 ± 1.9 16.7 28.2 1.00

xtasks/test/perf

Command mise-2026.7.7 mise Variance
install (cached) 199ms 201ms +0%
ls (cached) 83ms 79ms +5%
bin-paths (cached) 84ms 85ms -1%
task-ls (cached) 171ms 170ms +0%

@jdx
jdx enabled auto-merge (squash) July 20, 2026 18:24
@jdx
jdx merged commit 996b1b9 into main Jul 20, 2026
34 checks passed
@jdx
jdx deleted the claude/fix-macos-pgo branch July 20, 2026 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant