ci: reorganize mesh-llm builds for 50% speedups using warm build caches

[ndizazzo]

Summary

• Removes a duplicate ~7 minute release build from the Linux and macOS CI jobs. They used to run cargo test --release and then cargo build --release -p mesh-llm --bin mesh-llm back-to-back. The second step is redundant: cargo test --release already compiles target/release/mesh-llm as part of building the workspace, and every downstream integration test (ci-smoke-test.sh, ci-split-test.sh, ci-moe-mesh-test.sh, ci-client-auto-test.sh, CLI smoke) already consumes that same binary.
• Moves cargo fmt --check to an early-fail gate, before unit tests and the llama.cpp build. Formatting issues now fail the job in seconds instead of ~15 minutes in. The || true suffix is also removed so the check actually blocks.
• Fixes pre-existing rustfmt violations that had been silently masked by || true, in a separate style-only commit. Surfacing these is the point of making the fmt check blocking.

Why the duplicate build is safe to remove

cargo test --release at the workspace root compiles every [[bin]] target of every workspace member in release mode (it has to, in order to build and link the test harnesses against the same crate). After that step, target/release/mesh-llm exists and is byte-for-byte the binary the subsequent tests were already using. The old cargo build --release -p mesh-llm --bin mesh-llm step was a full second-compile pass that accomplished nothing new.

Rust caching — findings (no change in this PR)

The user asked whether there is an sccache-equivalent for Rust that we could adopt. Current state:

• Swatinem/rust-cache@v2 is wired up and caches the Rust target/ dir + registry. This is the primary Rust cache today.
• mozilla-actions/sccache-action@v0.0.9 is installed and SCCACHE_GHA_ENABLED: "true" is set, but RUSTC_WRAPPER: sccache is not set, and CARGO_INCREMENTAL is not pinned. So sccache currently only accelerates the llama.cpp C/C++ build via -DCMAKE_C_COMPILER_LAUNCHER=sccache. It does not cache any Rust compilation.

Enabling sccache for Rust is possible (RUSTC_WRAPPER: sccache + CARGO_INCREMENTAL: 0) but interacts non-trivially with Swatinem/rust-cache — they cache at different layers and combining them can either compound or fight each other depending on the workload. I did not change this in this PR because (a) it is risky and should be measured, and (b) this PR's goal is to eliminate an obvious 7 minute duplicate. Worth a follow-up PR that A/B tests the two caching strategies.

Validation

Locally on this branch:

```
$ cargo fmt --manifest-path mesh-llm/Cargo.toml -- --check
(clean)
```

The four files touched in `83907bf` are pure whitespace/line-wrapping changes produced by `cargo fmt` and contain no behavioral changes.

[Copilot]

Pull request overview

This PR aims to speed up CI by removing an allegedly redundant release build step and making cargo fmt --check an early, blocking failure gate, alongside rustfmt-only cleanup in a few Rust sources.

Changes:

• Move cargo fmt --check earlier in Linux/macOS CI and make it blocking (remove || true).
• Remove the explicit cargo build --release -p mesh-llm --bin mesh-llm step from Linux/macOS CI jobs.
• Apply rustfmt output to several Rust files previously masked by non-blocking formatting.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
.github/workflows/ci.yml	Reorders CI to fail fast on formatting and removes a dedicated release build step.
mesh-llm/src/protocol/mod.rs	rustfmt-only formatting in protocol tests.
mesh-llm/src/mesh/mod.rs	rustfmt-only formatting in config apply path.
mesh-llm/src/lib.rs	rustfmt-only formatting.
mesh-llm/src/inference/launch.rs	rustfmt-only formatting in warning text + tests.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ndizazzo]

Good catch on both threads — you were right that cargo test --release does not reliably produce target/release/mesh-llm on a cold cache (the smoke test empirically hit the missing-binary failure). Applied suggestion B in 4a05723: explicit cargo build --release -p mesh-llm --bin mesh-llm as the single source of the binary, plus cargo test --release -p mesh-llm --lib --tests so the bin target is excluded from the test step and not recompiled. Binary is now built exactly once per job and the ~7 minute duplicate compile is still eliminated.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 5 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 5 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ndizazzo]

Final scoreboard

Pushed all four levers, validated against baseline run 24066466965 (main branch) vs run 24075815657 (this PR after cache warmup):

Job	Baseline	This PR (warm)	Δ	%
linux	16m 23s	9m 09s	-7m 14s	-44%
macos	18m 45s	10m 51s	-7m 54s	-42%
Linux CUDA	21m 42s	9m 38s	-12m 04s	-56%
Linux ROCm	9m 06s	9m 30s	+24s	noise
Linux Vulkan	6m 03s	6m 00s	-3s	noise
Workflow wall clock	21m 42s	10m 51s	-10m 51s	-50%

PR runs now go green in half the time.

Levers applied (commit-by-commit)

1. 83907bf style: cargo fmt on pre-existing rustfmt violations (4 files; whitespace only). Precondition for the early-fail fmt gate to actually pass without || true.

2. 4a05723 ci: explicit release binary build + cargo test --lib --tests. Replaces the original cargo test --release (which doesn't reliably emit target/release/mesh-llm) with an explicit cargo build --release step + --lib --tests for the test step. Net: same single bin compile, fmt early-fail enabled.

3. 827fb55 ci: run unit tests in DEBUG profile. Release-mode test compiles take ~3x longer than debug. Saved ~3-5 min on linux+macos.

4. 67d4861 ci(cuda): single GPU arch (sm_89) for CI validation only. The CI job does not exercise CUDA binaries (CLI smoke is mesh-llm --version only) so a single arch proves the build works. release.yml keeps the full 75;80;86;89;90;120 list for user-facing artifacts. Carrack measurement: 6-arch llama.cpp 10m 32s vs 1-arch 2m 51s = 3.70x reduction.

5. f31a2a9 ci: rust-cache workspace config fix. linux + macos jobs had workspaces: mesh-llm which caches a non-existent mesh-llm/target/ while cargo writes to ./target/ (workspace root). The linux_cuda/rocm/vulkan jobs already had the correct . -> target config. This was costing us ~4 min on linux and ~3.5 min on macos per run from missed cache restore.

6. b158a1d ci: documentation comments on the rust-cache fix to prevent future regressions.

Cache warm-up gotcha

Both Lever 1 (cuda_arch change) and Lever 4 (rust-cache workspace fix) invalidated their respective cache namespaces. The first 1-2 runs after either commit were SLOWER than baseline because everything had to cold-populate. Run #3 hit steady state with sccache 99.81% hit rate on CUDA. If you ever change cuda_arch again expect a similar ~2-run warmup cycle.

Carrack experiment summary

Exp	what	-j	wall
E2	llama.cpp CUDA cold 1-arch	24	2m 51s
E5	llama.cpp CUDA cold 6-arch	24	10m 32s (3.70x)
E4	llama.cpp CUDA cold 1-arch	4	9m 58s (CI sim)
E3b	llama.cpp CUDA WARM rebuild	24	14s (99% cache hit)
E6	cargo build --release cold	24	52s

Carrack workspace cleaned up after measurement.

What's NOT in this PR (deferred)

• L3: parallel matrix split (cargo + llama.cpp on separate runners). Considered but the per-run gain after Levers 1+4 is small (~6 min cargo vs ~10 min llama.cpp; max would be ~10 min) and the complexity cost is high (artifact upload/download, two cache namespaces). Defer until needed.
• sccache for Rust (RUSTC_WRAPPER=sccache). Swatinem/rust-cache is now working correctly and gets us most of the benefit. Adding sccache on top risks cache conflicts.
• mold linker on linux. Modest gain (~30s-1min), not necessary to hit the 50% target.

The PR is shippable as-is.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 5 changed files in this pull request and generated 6 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ndizazzo]

Follow-up: Debug profile lever — macOS down to 5m 34s (-49%)

Previous scoreboard had macOS at 10m 51s as the new bottleneck. Investigated locally on Apple hardware (M4 Pro, isolated CARGO_HOME) following the same carrack methodology.

Root cause found

The macOS job (and linux) did two full Rust compiles per run:

1. cargo build --release -p mesh-llm --bin mesh-llm (3m 34s)
2. cargo test -p mesh-llm --lib --tests (debug, 2m 49s)

Two separate profiles = separate target/ subdirs = zero artifact sharing. And release has incremental = false by default, so every warm rebuild re-codegens the whole crate.

Local experiments (M4 Pro, isolated CARGO_HOME, measured via /usr/bin/time -p)

Exp	Command	Wall	User
E1	cold cargo build --release -p mesh-llm --bin mesh-llm	1m 45s	676s
E3	warm release after touch lib.rs	43s	251s
E4	warm debug test --no-run after E1	1m 01s	272s
E5	warm test --release --no-run after E1	1m 07s	580s
E6a	cold debug build -p mesh-llm --bin mesh-llm	52s	219s
E6b	warm debug test after E6a	40s	154s
E7	warm debug after touch lib.rs	4.65s	3s

E5 rules out release-mode tests (dead end: 2x user CPU). E7 shows debug rebuild is ~10x faster than release rebuild on lib edits — because debug has incremental = true while release has incremental = false.

Fix (commit 95d951c)

Build both steps in dev/debug profile so they share one target/ subdir. Integration tests now exec target/debug/mesh-llm. mesh-llm is a thin orchestrator around llama-server (hot loop is inside llama-server C++, still built optimized), so debug vs release binary perf is negligible for smoke tests. CUDA/ROCm/Vulkan jobs keep release — they validate backend compilation for the release pipeline.

Gotcha: GitHub Actions cache scoping

First run after the commit showed a REGRESSION (+162s linux, +90s macos). Root cause: pull_request runs cache under refs/pull/209/merge but the first post-commit run was cold. Subsequent warm run now shows true numbers. (Same pattern as Lever 1/4 — documented for future readers.)

Warm-case results (run 24095554088)

Job	Baseline (b158a1d)	New warm	Δ
macos	10m 51s	5m 34s	-5m 17s (-49%)
linux	9m 09s	6m 18s	-2m 51s (-31%)
Linux CUDA	9m 38s	10m 03s	+25s (noise)
Linux ROCm	9m 30s	9m 40s	+10s (noise)
Linux Vulkan	6m 00s	5m 58s	noise
Workflow wall clock	10m 51s	10m 03s	-48s (-7%)

macOS step detail

Step	Baseline	Warm	Δ
rust-cache restore	27s	13s	-14s
Build mesh-llm	214s	38s	-176s (-82%)
Unit tests	169s	99s	-70s (-41%)

Linux step detail

Step	Baseline	Warm	Δ
rust-cache restore	21s	18s	-3s
Build mesh-llm	146s	32s	-114s (-78%)
Unit tests	115s	113s	-2s (test execution dominates)

Cumulative progress on PR #209

Run	macos	linux	Linux CUDA	Wall clock
main baseline (2075bf7)	18m 45s	16m 23s	21m 42s	21m 42s
PR levers 1-4 (b158a1d)	10m 51s	9m 09s	9m 38s	10m 51s
PR + debug (95d951c)	5m 34s	6m 18s	10m 03s	10m 03s
vs main	-70%	-62%	-54%	-54%

Next bottleneck

The workflow wall clock is now bound by Linux CUDA at 10m 03s. macOS is no longer the critical path. Attacking CUDA further would likely require changing scripts/build-linux.sh (Rust build profile inside the CUDA container) or splitting CUDA into parallel jobs — deferring to a follow-up since the main user-visible PRs already run well under 11 minutes.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 8 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Bortlesboat]

The cache workspace fix and format-check changes are correct. One thing to check: the crate has tests in and that were previously covered by the workspace-wide . The new flag drops those.

[michaelneale]

@ndizazzo any risks - in the past I have been bitten by testing debug vs release but as long as release bin is tested at least once?

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ndizazzo]

@ndizazzo any risks - in the past I have been bitten by testing debug vs release but as long as release bin is tested at least once?

@michaelneale Low risk, but not zero. PR CI intentionally uses debug for speed + incremental, and I think we're covered via the release workflow, where release-profile binaries are built and validated.

As long as we still exercise the release binary in the release path at least once, I’m fairly comfortable with this change.

I'm also thinking that our docker builds can run on main with a release path, so that's another angle we cover (once #220 gets finalized)

[ndizazzo]

@michaelneale making an executive decision to get this in. I was blocked on MLX I didn't want to conflict with that work, but we're losing the efficiency this brings. I'll pull changes into the MLX branch and update for compatibility.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.