Skip to content

ci: drop duplicate matrix legs and consolidate caches - #2614

Closed
ilblackdragon wants to merge 2 commits into
stagingfrom
ci/consolidate-redundant-jobs
Closed

ilblackdragon wants to merge 2 commits into
stagingfrom
ci/consolidate-redundant-jobs

Conversation

@ilblackdragon

Copy link
Copy Markdown
Member

Summary

Combines five reductions targeting the duplicate compile work the analysis flagged: clippy and `cargo check` were running the same compile multiple times across matrix legs and platforms.

# Change
1 Drop `windows-build` from `test.yml` — its `cargo check` matrix is fully subsumed by `clippy-windows`, which already type-checks `--all --benches --tests --examples` on the same configs.
2 Collapse PR-blocking clippy to `--all-features` only (Linux + Windows). `default` and `libsql-only` move to `clippy-extra` / `clippy-windows-extra` jobs, gated to `push` on `main`/`staging`.
3 Share rust-cache slot between clippy and tests via `shared-key: build` / `shared-key: build-windows`. They produce overlapping artifacts for the same feature set; one slot beats two near-duplicates.
6 Replace `cargo install cargo-component --locked || true` with a composite action wrapping `taiki-e/install-action` (precompiled binary). Applied to `test.yml`, `coverage.yml`, `release.yml`.
7 Add `paths-ignore` for top-level docs, LICENSE, PNGs, issue/PR templates, and `docs/**`. Conservative — any `.md` actually `include_str!()`'d by the binary (workspace seeds, engine prompts, channel READMEs) is not filtered.

Interaction with #2609 / #2610

Independent and complementary. This PR includes `save-if` guards on every rust-cache step (so it's self-contained if those land later) and changes cache keys (`clippy` → `build` for Linux; `clippy-windows` + `windows-build` → `build-windows` for Windows). Expect trivial line-level merge conflicts in the rust-cache blocks; the intent layers cleanly:

Also: gates `no-panics` to PR-only (it diffs against the PR base SHA, empty on push) and updates the `code-style` roll-up to accept `skipped` for the push-only / PR-only conditional jobs.

Expected impact per PR

  • 3 Windows runners removed (`windows-build` matrix gone)
  • Clippy goes from 3 Linux + 3 Windows legs → 1 Linux + 1 Windows on PRs
  • Tests reuse the clippy-warmed `target/` slot when their matrix runs
  • `cargo-component` install drops from minutes to seconds across 5 jobs

Test plan

  • CI on this PR: confirm `Clippy (all-features)` and `Clippy Windows (all-features)` are the only clippy jobs running; `clippy-extra` and `clippy-windows-extra` show `skipped`; `Windows Build (...)` is gone
  • `Code Style (fmt + ...)` roll-up reports success despite skipped legs
  • After merge, push to `staging` runs the full clippy matrix (`clippy-extra` / `clippy-windows-extra` succeed) and refreshes the `build` / `build-windows` cache slots
  • Subsequent PR shows clippy with cache restore from the `build` slot

Watch-outs

  • The `clippy` (all-features) job uses `--all-features` while `tests` (all-features) uses `--no-default-features --features postgres,libsql,html-to-markdown,bedrock,import` (excluding the test-only `integration` feature). Sharing the cache slot still helps because the bulk of the compile graph overlaps, but the diff will be rebuilt — by design, not a regression.
  • `paths-ignore` triggers a workflow-level skip when only filtered paths change. The Main ruleset's required checks (`Run Tests`, `Code Style (fmt + clippy)`) currently don't strictly enforce on developer PRs (the latter's name doesn't even match the actual job name), and the Staging ruleset has no required checks. So docs-only PRs to staging won't be blocked. If branch protection tightens later, we'll need a no-op stub workflow to satisfy required checks on filtered paths.

Combines five reductions targeting the same root cause: clippy and
cargo check were running the same compile work multiple times across
matrix legs and platforms.

- Drop `windows-build` from test.yml. Its `cargo check` matrix is
  fully subsumed by `clippy-windows`, which already type-checks
  `--all --benches --tests --examples` on the same configs.

- Collapse PR-blocking clippy to `--all-features` only (Linux + Windows).
  Lint findings are almost never feature-gated, and the remaining
  configs (`default`, `libsql-only`) now run as `clippy-extra` /
  `clippy-windows-extra` jobs gated to `push` events on long-lived
  branches, where they verify feature combinations without slowing PRs.

- Share rust-cache slot between clippy and tests via
  `shared-key: build` (Linux) and `shared-key: build-windows`. Clippy
  and `cargo test` produce overlapping artifacts for the same feature
  set, so a single slot serves both jobs and avoids duplicating
  multi-GB target/ caches.

- Replace `cargo install cargo-component --locked || true` with a
  composite action backed by `taiki-e/install-action`. Drops a
  multi-minute source build to seconds and stops swallowing failures.
  Applied to test.yml, coverage.yml, and release.yml.

- Add `paths-ignore` filters for top-level docs, LICENSE files,
  PNG assets, issue/PR templates, and `docs/**` (mintlify site).
  Conservative scope: any `.md` file actually `include_str!()`'d
  by the binary (workspace seeds, engine prompts, channel READMEs)
  remains unfiltered.

Includes `save-if` guards on every rust-cache step so PR jobs are
restore-only — complementary to #2609 and #2610. Gates `no-panics`
to PR events (it diffs against the PR base SHA, unavailable on push)
and teaches the `code-style` roll-up to accept `skipped` for the
push-only / PR-only conditional jobs.
Copilot AI review requested due to automatic review settings April 18, 2026 05:08
@github-actions github-actions Bot added size: L 200-499 changed lines scope: ci CI/CD workflows risk: medium Business logic, config, or moderate-risk modules contributor: core 20+ merged PRs and removed size: L 200-499 changed lines labels Apr 18, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new GitHub composite action to install cargo-component using precompiled binaries via taiki-e/install-action, which significantly reduces CI setup time compared to source-based installation. Feedback suggests pinning the cargo-component tool to a specific version to ensure build reproducibility and avoid non-deterministic behavior in the CI environment.

- name: Install cargo-component
uses: taiki-e/install-action@62b0f2dec647a8e604c6a0fda0e38530180dce20 # v2
with:
tool: cargo-component

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To ensure build reproducibility and avoid unexpected CI failures when a new version of cargo-component is released, consider pinning the tool to a specific version (e.g., cargo-component@0.20.0). This aligns with the project's practice of using --locked for deterministic builds, as seen in the local setup scripts. Without a version specifier, install-action will fetch the latest available version, which may introduce non-deterministic behavior in the build process.

References
  1. When applying a best practice, such as using --locked for reproducible builds, it should be applied consistently across the codebase. Pinning tool versions in CI is the equivalent best practice for binary installers to ensure a deterministic environment.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in #2566 — the composite action now pins to cargo-component@0.21.1 (latest stable as of 2026-04-07) and includes a comment about bump cadence via dependabot's cargo-component* group. Commit: 86d4419. (This PR is being closed in favor of #2566 — see PR description.)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Reduces duplicated compile work in GitHub Actions by shrinking CI matrices, consolidating rust-cache usage across jobs, and speeding up cargo-component installation via a composite action.

Changes:

  • Add paths-ignore filters to skip CI on docs-only/template/image changes.
  • Consolidate Swatinem rust-cache usage via shared-key and save-if gating, and remove windows-build from test.yml.
  • Replace ad-hoc cargo install cargo-component with a composite action using taiki-e/install-action.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
.github/workflows/test.yml Adds paths-ignore, shares rust-cache slot with code_style, adds save gating, removes windows-build, uses composite installer.
.github/workflows/code_style.yml Adds paths-ignore + push trigger, collapses PR clippy coverage, shares caches, adjusts roll-up for skipped conditional jobs.
.github/workflows/coverage.yml Uses composite action for cargo-component installation.
.github/workflows/release.yml Uses composite action for cargo-component installation.
.github/actions/install-cargo-component/action.yml New composite action to install cargo-component via prebuilt binaries.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/test.yml Outdated
Comment on lines +79 to +80
shared-key: build
save-if: ${{ github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/staging') }}

Copilot AI Apr 18, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tests is a 3-leg matrix, but the rust-cache uses a single shared-key: build and is allowed to save on push. Because GitHub caches are immutable, whichever leg finishes first will “win” the cache slot, likely leaving it populated by a subset build (e.g. libsql-only) and preventing the all-features leg from ever refreshing the shared cache. Consider making only the all-features leg eligible to save (and keep the other legs restore-only), or split cache slots per feature-set.

Suggested change
shared-key: build
save-if: ${{ github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/staging') }}
#
# Only the broadest feature-set leg is allowed to save this immutable
# shared cache; the narrower matrix legs remain restore-only so they
# cannot win the slot with a subset build.
shared-key: build
save-if: ${{ matrix.name == 'all-features' && github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/staging') }}

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved differently in #2566. After merging this work into #2566, tests keeps the per-leg key: ${{ matrix.name }} (no shared-key: build), and #2566's dynamic matrix means PRs only run the all-features leg while pushes run the full matrix on independent keys — no race possible. (This PR is being closed in favor of #2566.)

with:
key: clippy-${{ matrix.name }}
shared-key: build
save-if: ${{ github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/staging') }}

Copilot AI Apr 18, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clippy-extra shares shared-key: build with the PR-blocking clippy job and is also configured to save on push. Since caches can’t be overwritten, an extra (subset) leg can win the race and permanently prevent the all-features clippy job from refreshing the shared cache on that push. Recommend making clippy-extra restore-only (no save) and letting only the Clippy (all-features) job save build on pushes.

Suggested change
save-if: ${{ github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/staging') }}
save-if: false

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved in #2566. The merged design has a single clippy job (no clippy-extra split) with #2566's dynamic matrix. The shared-key: clippy slot is now guarded by save-if: matrix.name == 'all-features' && push to main/staging — only the all-features leg can write the slot, so subset legs never race against it. (This PR is being closed in favor of #2566.)

with:
key: clippy-windows-${{ matrix.name }}
shared-key: build-windows
save-if: ${{ github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/staging') }}

Copilot AI Apr 18, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clippy-windows-extra shares shared-key: build-windows with the Clippy Windows (all-features) job and is allowed to save on push. Because cache keys are immutable, a subset leg can win and block the all-features job from refreshing the slot, reducing cache usefulness for future PRs. Recommend making clippy-windows-extra restore-only and letting only the all-features Windows job save the shared slot on pushes.

Suggested change
save-if: ${{ github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/staging') }}
save-if: false

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved in #2566. clippy-windows-extra doesn't exist in the merged result — clippy-windows keeps its per-leg key: windows-${{ matrix.name }} shared with windows-build, so each variant has its own slot and there's no race. (This PR is being closed in favor of #2566.)

Comment on lines +226 to +229
# Windows compilation is covered by the clippy-windows job in
# code_style.yml (clippy implies cargo check), so no Windows job is needed
# here.
needs: [tests, heavy-integration-tests, telegram-tests, wasm-wit-compat, docker-build, version-check, bench-compile]

Copilot AI Apr 18, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This workflow drops windows-build based on Windows compilation being covered by code_style.yml’s clippy job, but test.yml is also used as a reusable workflow (e.g. via staging-ci.yml), which does not run code_style.yml. As a result, those callers will lose any Windows compile signal entirely. If that coverage is still desired for batch/staging runs, consider adding a Windows leg back under workflow_call, or have the caller workflow also invoke the relevant Windows lint/check workflow.

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Resolved in #2566 by keeping windows-build (with #2566's dynamic matrix: SLIM=1 leg on PRs, FULL=3 legs on push and workflow_call from staging-ci.yml). So scheduled batches and direct callers retain full Windows compile signal. (This PR is being closed in favor of #2566.)

…undant-jobs

# Conflicts:
#	.github/workflows/code_style.yml
#	.github/workflows/test.yml
@github-actions github-actions Bot added the size: M 50-199 changed lines label Apr 18, 2026
@ilblackdragon ilblackdragon mentioned this pull request Apr 18, 2026
6 tasks
@ilblackdragon

Copy link
Copy Markdown
Member Author

Closing in favor of #2566. The full set of changes from this PR (drop windows-build → kept under #2566's dynamic matrix instead, cargo-component composite action with pinned version, path-based skip via #2566's changes job pattern) plus all 5 Copilot/Gemini review comments have been folded into #2566 at commit 86d4419. Per-comment replies left in this PR's review threads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor: core 20+ merged PRs risk: medium Business logic, config, or moderate-risk modules scope: ci CI/CD workflows size: M 50-199 changed lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants