Update CI and release workflows to use packaged artifacts and gated inference smoke tests - #235
Update CI and release workflows to use packaged artifacts and gated inference smoke tests#235i386 wants to merge 11 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Updates CI and release automation to build and package platform binaries first, then run inference smoke tests as a gated follow-up before publishing.
Changes:
- CI now packages/upload binaries (CPU + GPU lanes) and runs inference smoke tests in a separate job using the packaged artifacts.
- Release workflow now uploads a packaged Linux inference-binaries artifact, runs gated inference smokes, and publishes via
gh release. - Adds
release-build-amd*/release-bundle-amd*Justfile aliases that delegate to existing ROCm recipes.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| Justfile | Adds AMD-named aliases for ROCm build/bundle recipes used by workflows. |
| .github/workflows/release.yml | Splits build vs smoke testing, gates publish on smoke tests, and switches publishing to gh release. |
| .github/workflows/ci.yml | Splits packaging vs inference smokes, uploads packaged binaries, and runs smokes from uploaded artifacts. |
ndizazzo
left a comment
ndizazzo
left a comment
There was a problem hiding this comment.
Not sure what happened here - it undid a LOT of work, sending this one back to bake more
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…s to exclude from release-* publish pattern Agent-Logs-Url: https://github.com/michaelneale/mesh-llm/sessions/4da65737-73e4-4ddc-81cd-405fa439a18c Co-authored-by: i386 <50156+i386@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
| if [ ! -f llama.cpp/build/bin/llama-moe-split ]; then | ||
| echo "Required binary llama.cpp/build/bin/llama-moe-split is missing; cannot package Linux inference binaries." >&2 | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
Package Linux inference binaries checks only -f for llama-moe-split, but the later smoke test requires it to be executable (-x). To avoid uploading a tarball that passes packaging but fails at runtime, align the check here with the smoke job (verify it exists and is executable, or chmod +x and then verify).
| fi | |
| fi | |
| chmod +x llama.cpp/build/bin/llama-moe-split | |
| if [ ! -x llama.cpp/build/bin/llama-moe-split ]; then | |
| echo "Required binary llama.cpp/build/bin/llama-moe-split is not executable; cannot package Linux inference binaries." >&2 | |
| exit 1 | |
| fi |
| cp linux-binaries/mesh-llm target/release/mesh-llm | ||
| cp linux-binaries/rpc-server llama.cpp/build/bin/rpc-server | ||
| cp linux-binaries/llama-server llama.cpp/build/bin/llama-server | ||
| if [ ! -x linux-binaries/llama-moe-split ]; then | ||
| echo "Required binary linux-binaries/llama-moe-split is missing or not executable after artifact extraction" >&2 | ||
| exit 1 | ||
| fi | ||
| cp linux-binaries/llama-moe-split llama.cpp/build/bin/llama-moe-split |
There was a problem hiding this comment.
In Stage binaries for inference smokes, only llama-moe-split is validated as executable. If the tarball ever loses exec bits for mesh-llm, rpc-server, or llama-server, failures will be less obvious and show up later in the scripts. Consider adding test -x (or chmod +x + test -x) for all staged binaries here to fail fast with a clear error.
| cp linux-binaries/mesh-llm target/release/mesh-llm | |
| cp linux-binaries/rpc-server llama.cpp/build/bin/rpc-server | |
| cp linux-binaries/llama-server llama.cpp/build/bin/llama-server | |
| if [ ! -x linux-binaries/llama-moe-split ]; then | |
| echo "Required binary linux-binaries/llama-moe-split is missing or not executable after artifact extraction" >&2 | |
| exit 1 | |
| fi | |
| cp linux-binaries/llama-moe-split llama.cpp/build/bin/llama-moe-split | |
| for spec in \ | |
| "linux-binaries/mesh-llm:target/release/mesh-llm" \ | |
| "linux-binaries/rpc-server:llama.cpp/build/bin/rpc-server" \ | |
| "linux-binaries/llama-server:llama.cpp/build/bin/llama-server" \ | |
| "linux-binaries/llama-moe-split:llama.cpp/build/bin/llama-moe-split" | |
| do | |
| src="${spec%%:*}" | |
| dst="${spec#*:}" | |
| if [ ! -x "$src" ]; then | |
| echo "Required binary $src is missing or not executable after artifact extraction" >&2 | |
| exit 1 | |
| fi | |
| cp "$src" "$dst" | |
| chmod +x "$dst" | |
| if [ ! -x "$dst" ]; then | |
| echo "Staged binary $dst is not executable" >&2 | |
| exit 1 | |
| fi | |
| done |
| inference_smoke_tests: | ||
| name: Inference Smoke Tests | ||
| runs-on: ubuntu-latest | ||
| needs: | ||
| - build | ||
| steps: |
There was a problem hiding this comment.
inference_smoke_tests depends on the matrix build job, so it won’t start until both Linux and macOS builds finish even though it only consumes the Linux inference-binaries artifact. If the goal is to fail fast on inference regressions (and shorten time-to-release), consider splitting Linux CPU into a dedicated job (or otherwise structuring the workflow) so inference smokes can start as soon as the Linux artifact is ready.
| cp target/debug/mesh-llm ci-artifacts/linux-cuda-binaries/ | ||
| cp llama.cpp/build/bin/rpc-server ci-artifacts/linux-cuda-binaries/ | ||
| cp llama.cpp/build/bin/llama-server ci-artifacts/linux-cuda-binaries/ | ||
| cp llama.cpp/build/bin/llama-moe-split ci-artifacts/linux-cuda-binaries/ 2>/dev/null || true |
There was a problem hiding this comment.
cp llama.cpp/build/bin/llama-moe-split ... 2>/dev/null || true suppresses both stderr and the failure exit code, so a missing/failed llama-moe-split build will quietly produce an incomplete artifact. Prefer making this explicit: either fail fast if the binary is expected in this lane, or gate the copy with a clear if [[ -f ... ]] branch that logs that the binary is intentionally omitted (and consider reflecting that in the artifact name/content).
| cp llama.cpp/build/bin/llama-moe-split ci-artifacts/linux-cuda-binaries/ 2>/dev/null || true | |
| if [[ -f llama.cpp/build/bin/llama-moe-split ]]; then | |
| cp llama.cpp/build/bin/llama-moe-split ci-artifacts/linux-cuda-binaries/ | |
| else | |
| echo "INFO: llama-moe-split not present in this Linux CUDA build; omitting it from packaged artifacts." | |
| fi |
| cp target/release/mesh-llm ci-artifacts/linux-rocm-binaries/ | ||
| cp llama.cpp/build/bin/rpc-server ci-artifacts/linux-rocm-binaries/ | ||
| cp llama.cpp/build/bin/llama-server ci-artifacts/linux-rocm-binaries/ | ||
| cp llama.cpp/build/bin/llama-moe-split ci-artifacts/linux-rocm-binaries/ 2>/dev/null || true |
There was a problem hiding this comment.
This lane also suppresses errors when copying llama-moe-split (2>/dev/null || true), which can hide unexpected build regressions and upload an incomplete tarball. Prefer either requiring the binary (fail if missing) or conditionally including it with an explicit log message when absent.
| cp llama.cpp/build/bin/llama-moe-split ci-artifacts/linux-rocm-binaries/ 2>/dev/null || true | |
| if [[ -f llama.cpp/build/bin/llama-moe-split ]]; then | |
| cp llama.cpp/build/bin/llama-moe-split ci-artifacts/linux-rocm-binaries/ | |
| else | |
| echo "INFO: llama-moe-split not present in llama.cpp/build/bin; skipping optional binary" | |
| fi |
| cp target/release/mesh-llm ci-artifacts/linux-vulkan-binaries/ | ||
| cp llama.cpp/build/bin/rpc-server ci-artifacts/linux-vulkan-binaries/ | ||
| cp llama.cpp/build/bin/llama-server ci-artifacts/linux-vulkan-binaries/ | ||
| cp llama.cpp/build/bin/llama-moe-split ci-artifacts/linux-vulkan-binaries/ 2>/dev/null || true |
There was a problem hiding this comment.
llama-moe-split is copied with 2>/dev/null || true, which hides missing-binary errors and can result in a silently incomplete linux-vulkan-binaries.tgz. If the binary is optional here, make the omission explicit with a conditional + log; if it’s required, fail the step when it’s not present/executable.
| cp llama.cpp/build/bin/llama-moe-split ci-artifacts/linux-vulkan-binaries/ 2>/dev/null || true | |
| if [ -x llama.cpp/build/bin/llama-moe-split ]; then | |
| cp llama.cpp/build/bin/llama-moe-split ci-artifacts/linux-vulkan-binaries/ | |
| echo "Included optional binary: llama-moe-split" | |
| else | |
| echo "Optional binary not found or not executable, skipping: llama.cpp/build/bin/llama-moe-split" | |
| fi |
Summary
Testing