Skip to content

perf(release): add PGO builds for natively-trainable release targets - #11031

Merged
jdx merged 2 commits into
mainfrom
perf/pgo-release-builds
Jul 16, 2026
Merged

jdx merged 2 commits into
mainfrom
perf/pgo-release-builds

Conversation

@jdx

@jdx jdx commented Jul 16, 2026 •

Copy link
Copy Markdown
Owner

Summary

Ports jdx/aube's benchmarks/pgo.bash three-phase PGO pipeline to mise, wired into the release tarball builds. Follow-up to #11025 / #11027 / #11029 — this attacks the remaining first-touch page-in cost with profile-guided code layout, the last lever identified in that series.

  • scripts/pgo.bash — instrument (-Cprofile-generate) → train → rebuild (-Cprofile-use). Training is hermetic and offline: throwaway HOME + isolated MISE_*_DIRs exercising settings/config load, toolset resolution, hook-env (full and per-prompt early-exit fast path), env rendering, tasks, and exec. No registry infrastructure needed, unlike aube's Verdaccio workload. Carries over aube's hard-won guards: profraw sanity check, llvm-profdata silent-no-op check, LLVM_PROFILE_FILE override and profile-dir bind-mount for cross's container path translation.
  • scripts/build-tarball.sh — MISE_PGO=1 routes through pgo.bash with the identical feature set/profile; output lands at the same path.
  • release.yml — pgo: 1 on the targets whose instrumented binaries can execute on their build runner: linux-x64 (gnu+musl, cross-built, trained on the amd64 host) and macos-arm64 (native). arm/armv7 Linux can't run their binaries on the runner, macos-x64 would train skewed under Rosetta (aube's precedent), Windows PGO is untested — all stay plain serious.
  • Cross.toml — pass RUSTFLAGS through to the container (without this, the PGO flags silently never reach cargo — aube hit exactly this).

Measured effect (macOS arm64, same-source A/B, serious profile, hyperfine -N, warm)

non-PGO PGO
mise version 4.3 ms 4.1 ms
mise current 6.9 ms 6.3 ms (1.10×)
hook-env early-exit fast path (per shell prompt) 3.6 ms 3.3 ms
binary size 84.5 MB 81.3 MB (−4%)
startup pages touched 1182 1079

Honest framing: 6–10% on the hot paths, not a silver bullet. For context, the whole series has taken mise version on this machine from 8.1 ms → 4.1 ms. Linux is unvalidated (no local toolchain on the test box) but expected to be in the same range; the bigger Linux win is BOLT, which layers on top of this and is the natural follow-up (aube's phase 4 is the template, bolt-19 runner dependency, Linux GNU only — BOLT's instrumentation runtime SIGSEGVs on musl per aube's notes).

Cost & rollback

  • One extra serious-profile (fat LTO) build per PGO target per release: linux job budget raised 45→80 min, macos was already 90.
  • PGO failures fail the build loudly (no silent fallback to non-PGO). Backing out a target = deleting its pgo: 1 matrix line.
  • The first release with this merged is worth watching — local validation covered the full pipeline end-to-end (190 profraw files, 98 MB merged profile) but only for the native macOS path; the cross-container path follows aube's working config.

Verification

  • Full pipeline ran locally end-to-end on macOS arm64 with the exact tarball feature set and deployment target; A/B numbers above from the resulting binary
  • Trained binary passes the same benchmark suite used throughout this series (version/current/ls/exec/hook-env against a fixture project)
  • mise run lint clean (actionlint, shellcheck, shfmt)

🤖 Generated with Claude Code


Note

Medium Risk
Changes how primary release binaries are compiled for key platforms; PGO failures fail the build with no silent fallback, and the cross+PGO path is less validated locally than native macOS.

Overview
Adds profile-guided optimization (PGO) to release tarballs for targets where the instrumented binary can be trained on the CI runner.

scripts/pgo.bash implements a three-phase flow: instrument with -Cprofile-generate, run a hermetic offline workload (startup paths including hook-env fast path), merge profiles with llvm-profdata, then rebuild with -Cprofile-use. It includes cross-container bind-mounts for profile paths, LLVM_PROFILE_FILE overrides, and hard failures when no .profraw is produced.

scripts/build-tarball.sh routes through pgo.bash when MISE_PGO is set (using cross on Linux), leaving the same serious output path as before.

release.yml enables PGO for linux-x64 (gnu + musl) and macos-arm64, installs llvm-tools on those matrix rows, passes MISE_PGO, and raises Linux job/retry timeouts (~45→80/75 min) for the extra build phase.

Cross.toml passthroughs RUSTFLAGS only for the x86_64 Linux gnu/musl targets so PGO flags reach cargo inside cross containers.

Other Linux arch targets, macOS x64 (Rosetta skew), and Windows stay on plain serious builds.

Reviewed by Cursor Bugbot for commit 005258e. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by CodeRabbit

  • New Features

    • Release builds now use profile-guided optimization for supported Linux and macOS architectures.
    • Optimized binaries are built using representative application workloads to improve runtime performance.
  • Bug Fixes

    • Cross-platform release builds now apply build settings only to their intended targets, improving build consistency and reliability.

Ports jdx/aube's three-phase rustc PGO pipeline to mise: build with
-Cprofile-generate, train, rebuild with -Cprofile-use. The training
workload is hermetic and offline — throwaway HOME with isolated
MISE_*_DIRs exercising the startup-dominated hot paths: settings and
config load, toolset resolution, hook-env (full run and the per-prompt
early-exit fast path), env rendering, task listing, and exec.

Enabled for the targets whose instrumented binaries can execute on
their build runner: x86_64 linux gnu/musl (cross-built, trained on the
amd64 host) and native macos-arm64. arm/armv7 Linux, macos-x64, and
Windows keep plain serious builds.

Measured on macOS arm64 against a same-source serious build (hyperfine
-N, warm, isolated env):

- mise version: 4.3ms -> 4.1ms
- mise current: 6.9ms -> 6.3ms (1.10x)
- hook-env early-exit fast path (per shell prompt): 3.6ms -> 3.3ms
- binary size: 84.5MB -> 81.3MB; startup pages touched: 1182 -> 1079

Costs one extra serious-profile build per PGO target, so the linux
tarball job budget goes from 45 to 80 minutes. Backing a target out is
deleting its `pgo: 1` matrix line. Cross.toml now passes RUSTFLAGS
through so the profile-generate/-use flags reach cargo inside cross
containers, and pgo.bash bind-mounts the profile dir at the same path
so -Cprofile-use resolves there.

BOLT post-link optimization (Linux-only, layered on PGO) is a potential
follow-up; aube's phase-4 flow is the template.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 16, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: f4a3911b-1e49-4070-af75-0dbdc1777892

📥 Commits

Reviewing files that changed from the base of the PR and between 94cc66d and 005258e.

📒 Files selected for processing (4)
  • .github/workflows/release.yml
  • Cross.toml
  • scripts/build-tarball.sh
  • scripts/pgo.bash

📝 Walkthrough

Walkthrough

Release tarball workflows now enable PGO for selected Linux and macOS targets. The tarball script invokes a new three-phase PGO process, while Cross configuration scopes host RUSTFLAGS passthrough to supported Linux targets.

Changes

Profile-guided release builds

Layer / File(s) Summary
PGO driver and optimization phases
scripts/pgo.bash
Adds instrumented compilation, hermetic offline training, profile merging, optional early exit, and optimized recompilation.
Tarball build integration
scripts/build-tarball.sh, Cross.toml
Selects cargo or cross for PGO, passes the target and features, and scopes RUSTFLAGS passthrough to GNU and musl x86_64 targets.
Release matrix and workflow wiring
.github/workflows/release.yml
Enables PGO for selected Linux and native macOS ARM64 builds, conditionally installs LLVM tools, passes MISE_PGO, and increases build timeouts.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ReleaseWorkflow
  participant build_tarball
  participant pgo_bash
  participant mise
  participant llvm_profdata
  ReleaseWorkflow->>build_tarball: Set MISE_PGO and install LLVM tools
  build_tarball->>pgo_bash: Pass target and build features
  pgo_bash->>mise: Build instrumented binary
  pgo_bash->>mise: Run offline training workload
  mise-->>pgo_bash: Write profraw files
  pgo_bash->>llvm_profdata: Merge profiles
  pgo_bash->>mise: Rebuild with profile-use flags
Loading

Suggested labels: release

Poem

I’m a rabbit with profiles in tow,
Through warm-up runs and LLVM glow.
I hop through builds, both swift and bright,
Then tuck merged data in for flight.
Tarballs roll from tunnel to moon—
PGO hums a carrot tune!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: adding PGO builds for release targets that can be trained natively.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@greptile-apps

greptile-apps Bot commented Jul 16, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR wires a three-phase profile-guided optimization (PGO) pipeline into the release tarball builds for linux-x64 (gnu + musl) and macos-arm64, delivering 6–10% startup improvements on hot paths like hook-env. A new scripts/pgo.bash handles the full instrument → train → rebuild cycle with a hermetic offline training workload.

  • scripts/pgo.bash: Instruments with -Cprofile-generate, runs a hermetic training workload (3 passes, isolated MISE_*_DIRs, LLVM_PROFILE_FILE override for cross-container path translation, profraw-count and merged-profile sanity checks), then rebuilds with -Cprofile-use.
  • scripts/build-tarball.sh: Routes through pgo.bash when MISE_PGO is set, selecting cross vs cargo based on OS, with the same feature set and output path as the existing builds.
  • Cross.toml / release.yml: RUSTFLAGS passthrough is scoped to the two x86_64 cross targets; PGO matrix rows install llvm-tools and raise the Linux job budget from 45 → 80 min.

Confidence Score: 5/5

Safe to merge; the PGO pipeline is well-guarded and failures are loud, not silent.

The implementation is careful and well-validated: profraw count check, merged-profile existence check, per-target RUSTFLAGS scoping, and hermetic training workload with isolated state dirs. The only finding is that pgo.bash hardcodes $REPO_ROOT/target/ without checking CARGO_TARGET_DIR, which has no effect in CI where that variable is unset.

No files require special attention. scripts/pgo.bash is the most complex new file and is worth a read, but it is well-commented and carries over proven guards from aube's pipeline.

Important Files Changed

Filename Overview
scripts/pgo.bash New three-phase PGO pipeline: instrumented build → hermetic offline training workload → profile-guided rebuild. Well-guarded with profraw count check, merged-profile existence check, LLVM_PROFILE_FILE override for cross containers, and per-pass isolated state dirs. Minor: hardcodes REPO_ROOT/target/ paths without respecting CARGO_TARGET_DIR.
scripts/build-tarball.sh Adds an MISE_PGO branch that selects the correct build tool (cross for Linux) and delegates to pgo.bash with the same feature set and target triple; falls through to the existing cross/cargo paths otherwise. Binary output path is unchanged so glibc checks and packaging are unaffected.
Cross.toml Adds per-target RUSTFLAGS passthrough for the two x86_64 PGO targets (gnu and musl). Correctly scoped at the target level rather than globally to avoid silently forwarding host RUSTFLAGS into non-PGO cross builds.
.github/workflows/release.yml Enables PGO for linux-x64 (gnu + musl) and macos-arm64 matrix rows; installs llvm-tools for those rows; raises Linux job timeout from 45 → 80 min and retry timeout to 75 min. macOS step timeout stays at 90 min (author validated PGO fits within that budget on the arm64 runner).

Reviews (2): Last reviewed commit: "perf(release): scope RUSTFLAGS passthrou..." | Re-trigger Greptile

Comment thread Cross.toml Resolved
Comment thread scripts/pgo.bash Resolved
@github-actions

github-actions Bot commented Jul 16, 2026 •

Copy link
Copy Markdown
Contributor

Hyperfine Performance

mise x -- echo

Command Mean [ms] Min [ms] Max [ms] Relative
mise-2026.7.7 x -- echo 25.5 ± 1.4 21.4 29.7 1.00
mise x -- echo 25.6 ± 2.6 19.7 66.6 1.01 ± 0.12

mise env

Command Mean [ms] Min [ms] Max [ms] Relative
mise-2026.7.7 env 24.9 ± 1.5 21.0 29.6 1.05 ± 0.09
mise env 23.8 ± 1.6 18.7 29.0 1.00

mise hook-env

Command Mean [ms] Min [ms] Max [ms] Relative
mise-2026.7.7 hook-env 26.3 ± 1.5 21.6 30.3 1.05 ± 0.10
mise hook-env 25.0 ± 1.7 19.5 30.1 1.00

mise ls

Command Mean [ms] Min [ms] Max [ms] Relative
mise-2026.7.7 ls 22.6 ± 1.4 17.7 27.9 1.06 ± 0.10
mise ls 21.3 ± 1.6 16.8 26.6 1.00

xtasks/test/perf

Command mise-2026.7.7 mise Variance
install (cached) 194ms 194ms +0%
ls (cached) 77ms 74ms +4%
bin-paths (cached) 83ms 79ms +5%
task-ls (cached) 160ms 157ms +1%

Review feedback: global [build.env] passthrough would silently inject
host RUSTFLAGS into every cross build, including non-PGO arm targets.
Scope it to the two x86_64 targets that PGO actually builds through
cross. Also alias the phase-3b output as PGO_FINAL_BIN so the success
log doesn't misleadingly reference the instrumented binary.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 005258e. Configure here.

Comment thread scripts/pgo.bash
RUSTFLAGS="${RUSTFLAGS:-} -Cprofile-generate=$PGO_PROFRAW_DIR" \
"$PGO_BUILD_TOOL" build --profile="$PGO_PROFILE" $target_arg --bin mise "$@"

INSTRUMENTED_BIN="$REPO_ROOT/target/${target_dir_part}${PGO_PROFILE}/mise"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ignores CARGO_TARGET_DIR for binary

Low Severity

scripts/pgo.bash always resolves the instrumented binary under REPO_ROOT/target/…, while cargo/cross honor CARGO_TARGET_DIR and scripts/build-tarball.sh packages from that same variable. With a non-default CARGO_TARGET_DIR, phase 1 can build in one tree while training checks or runs a binary from another, including a stale target/…/mise left from an earlier build.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 005258e. Configure here.

@jdx
jdx enabled auto-merge (squash) July 16, 2026 04:02
@jdx
jdx merged commit 0e32a14 into main Jul 16, 2026
35 checks passed
@jdx
jdx deleted the perf/pgo-release-builds branch July 16, 2026 04:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant