Optimize CI caches by silverwind · Pull Request #37387 · go-gitea/gitea

silverwind · 2026-04-23T14:57:24Z

Cache includes go, lint and unittests. Integration tests with their standalone binaries are uncacheable with their current architecture. Summary from Claude below.

Every Go job uses a new composite action (.github/actions/go-cache) that restores and saves the Go module cache, a shared build cache, and the golangci-lint cache. A cache-seeder workflow runs on push: main to pre-populate those slots; PRs read them via GitHub's default-branch fallback, so the common case is warm from the first commit.

Also dropped -coverprofile from test-unit (it silently disabled Go's test result cache), and -race from test-pgsql and test-mysql (kept on test-unit and test-sqlite).

Warm-cache runtime (last run before this PR vs latest run here)

Job	Before	After	Δ
test-pgsql	27m53s	17m25s	−10m28s
test-mysql	30m36s	19m42s	−10m54s
test-mssql	21m20s	18m1s	−3m19s
test-sqlite	21m32s	19m50s	−1m42s
test-unit	19m14s	5m54s	−13m20s
test-e2e	3m57s	4m17s	+20s
lint-backend	5m45s	1m9s	−4m36s
lint-go-windows	5m56s	3m54s	−2m2s
lint-go-gogit	4m56s	1m10s	−3m46s
workflow critical path	~30m36s	~19m50s	~−35%

This PR was written with the help of Claude Opus 4.7

All five jobs in pull-db-tests.yml relied on setup-go@v6's default cache, which keys solely on hash(go.sum). Each job builds with a different combination of TAGS, GOEXPERIMENT and -race, producing disjoint Go build-cache entries — but GitHub Actions cache is first-writer-wins on a given key, so only one job's entries survived, making the restore a near-total miss for the other four. Disable setup-go's built-in cache and add a per-job actions/cache step keyed by job name, so each configuration gets its own slot. Co-Authored-By: Claude (Opus 4.7) <noreply@anthropic.com>

The test-unit job invoked 'make unit-test-coverage', which passes -coverprofile to 'go test'. That flag is not in Go's cacheable-flags allowlist, so it silently disables the test result cache for every unit-test package on every run. The coverage.out output is unused: no workflow calls 'make coverage' (its only consumer), and no codecov/coveralls integration exists. The matching test-mysql job comment ('at the moment, no coverage is really handled') confirms coverage has been dead for some time. Switch to 'make test-backend', preserving the 20m timeout via GOTESTFLAGS. On warm runs with an intact build cache, unchanged unit-test packages are now skipped entirely. Co-Authored-By: Claude (Opus 4.7) <noreply@anthropic.com>

With a fixed key, actions/cache skips the save step on an exact-key restore ('Cache hit occurred on the primary key ..., not saving cache.'). That traps $GOCACHE at whatever state first populated it — any newly-added entries (including Go's test result cache, now that -coverprofile is gone) are discarded at job end. Add ${{ github.run_id }} to the primary key and move the original key into restore-keys. Every run saves a fresh snapshot; subsequent runs restore the latest one via the prefix fallback. Old entries age out via GitHub's 7-day cache TTL and 10 GB per-repo LRU. Co-Authored-By: Claude (Opus 4.7) <noreply@anthropic.com>

Cosmetic. Each job's prefix word is unique in our cache namespace, so no cross-job partial match is possible. Co-Authored-By: Claude (Opus 4.7) <noreply@anthropic.com>

- Module cache (~/go/pkg/mod) now lives in its own actions/cache step with a deterministic key shared across every Go job (all workflows). Identical content for a given go.sum regardless of TAGS/race/tool, so one snapshot serves all. Cuts ~500 MB of duplicated storage per job. - Build cache (~/.cache/go-build) stays per-job and keeps the run_id rotation so newly-populated entries (including the test result cache) actually persist. Single restore-key prefix; the bare-prefix fallback removed as unnecessary. - Added runner.arch to all keys for safety. - Extended the same pattern to test-e2e (which also compiles the gitea binary) and to the compliance Go jobs (lint-backend, lint-go-windows, lint-go-gogit, checks-backend, backend). - Added a golangci-lint cache (~/.cache/golangci-lint) to the three lint-go* jobs, keyed on go.sum + .golangci.yml. 'make lint-go' uses 'go run', which picks up the cache automatically without any Makefile change. Co-Authored-By: Claude (Opus 4.7) <noreply@anthropic.com>

Race detection roughly doubles test-execution time. Integration tests rarely surface races that unit tests don't: the 239-file integration suite has only 13 occurrences of go func / sync primitives across 6 files, and zero t.Parallel() calls. Recent race fixes in this repo (e.g. container blob upload, go-gitea#36524) were developed with targeted unit tests, not caught by integration runs. Keep -race on test-unit (where all t.Parallel() tests live) and test-sqlite (preserves one integration-level race signal without a DB container). Drop it from pgsql and mysql, which are the critical-path jobs. Co-Authored-By: Claude (Opus 4.7) <noreply@anthropic.com>

Both jobs have small gobuild caches relative to their value: - checks-backend: cache was 244 MB, job runs ~2 min cold - test-e2e: Go compile is a small fraction of runtime (dominated by playwright setup and browser tests) Keep the shared gomod cache on both (single slot, benefits everyone). Trading ~1 GB of cache storage for ~30s-1min extra compile on two non-critical-path jobs. The 'actions' compliance lint job was already uncached; left untouched. Co-Authored-By: Claude (Opus 4.7) <noreply@anthropic.com>

Replace the per-workflow actions/cache@v5 blocks with a shared .github/actions/go-cache composite action. All 11 Go jobs across pull-db-tests, pull-compliance, and pull-e2e-tests now call it with a job name (and optionally build-cache / lint-cache / build-cache-rotate). Storage shape after this: - gomod cache: deterministic key shared across every Go job; one save per unique go.sum. - gobuild cache: stable key for every job except test-unit, which rotates with github.run_id so Go's test result cache can accumulate (~60% speedup on that job once populated). - golangci-lint cache: stable key per lint job, saves once per (go.sum + .golangci.yml) change. Effect: most PR runs write ~1.8 GB of fresh cache (test-unit only) instead of ~14 GB across all jobs. Steady-state fits comfortably under GitHub's 10 GB per-repo cap. Co-Authored-By: Claude (Opus 4.7) <noreply@anthropic.com>

- Stable gobuild cache key drops per-job segment; all non-rotating Go jobs share one slot (test-unit still rotates). Go's content-addressing keeps multi-config entries safely coexisting. - Add restore-keys prefix fallbacks on stable gobuild and golangci-lint keys so go.sum bumps and release-branch PRs still get partial cache restores via main's default-branch scope. - New cache-seeder.yml: on push to main, populates gobuild (three TAGS configs in one $GOCACHE, saved to the shared slot) and the three golangci-lint caches. PRs then read those seeds via GitHub's cross-scope fallback. Storage footprint in main's scope: ~3 GB seeded (gomod ~900 MB + shared gobuild ~1.5 GB + 3 lint ~900 MB), comfortably under the 10 GB per-repo cap. Co-Authored-By: Claude (Opus 4.7) <noreply@anthropic.com>

Yamllint rejects the column-aligned inline mappings; collapse to single spaces after commas. Co-Authored-By: Claude (Opus 4.7) <noreply@anthropic.com>

Signed-off-by: silverwind <me@silverwind.io>

actionlint requires 'description' in composite action metadata; removing it fails lint-actions. Co-Authored-By: Claude (Opus 4.7) <noreply@anthropic.com>

Rotating cache's restore-keys previously only fell back to other test-unit-specific keys, leaving first-ever PR runs of test-unit cold on the build cache even when the main-branch seeder had populated the shared 'gobuild-${os}-${arch}-${gosum}' slot. Add two more fallback prefixes so the rotating key can prefix-match the seeded shared key and restore its compile-cache content. Collapsed the blank lines between cache steps for compactness. Co-Authored-By: Claude (Opus 4.7) <noreply@anthropic.com>

Copilot

Pull request overview

Introduces a new Go caching strategy for CI intended to reduce workflow runtime by reusing Go module/build and golangci-lint caches across jobs and runs, and adjusts test invocations/flags to improve cache hit rates.

Changes:

Add a composite action (.github/actions/go-cache) and wire it into PR workflows, disabling setup-go’s built-in cache.
Add a cache-seeder workflow on push to main to pre-populate caches for PR fallback restores.
Adjust unit test commands/flags (drop coverage target; add GOTESTFLAGS) and remove RACE_ENABLED from MySQL/Postgres integration jobs.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
.github/workflows/pull-e2e-tests.yml	Disables setup-go cache and adds go-cache action for the e2e job (build cache disabled).
.github/workflows/pull-db-tests.yml	Adds go-cache usage across DB/unit jobs; adjusts unit test make targets/flags; removes race for mysql/pgsql.
.github/workflows/pull-compliance.yml	Adds go-cache usage for backend build/lint/check jobs (including golangci-lint cache).
.github/workflows/cache-seeder.yml	New workflow to warm caches on `main` pushes by building and running lints with caches enabled.
.github/actions/go-cache/action.yml	New composite action implementing Go module/build and golangci-lint caches via `actions/cache`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Addressing maintainer feedback: 'name:' clashes with top-level action metadata and reads ambiguously at call sites. 'cache-name:' labels exactly what the value represents. Also adds a header comment on cache-seeder explaining the per-ref cache scoping and why the seeder only runs on push-to-main. Co-Authored-By: Claude (Opus 4.7) <noreply@anthropic.com>

Co-Authored-By: Claude (Opus 4.7) <noreply@anthropic.com>

Signed-off-by: silverwind <me@silverwind.io>

Co-Authored-By: Claude (Opus 4.7) <noreply@anthropic.com>

silverwind · 2026-04-26T00:27:35Z

uses introduced in this PR are now hash-pinned in preparation for #37050.

* origin/main: Optimize CI caches (go-gitea#37387)

* origin/main: (176 commits) Refactor pull request view (3) (go-gitea#37439) Update 1.26.1 changelog in main (go-gitea#37442) Make GetPossibleUserByID can handle deleted user (go-gitea#37430) Fix fetch action redirect (go-gitea#37437) Refactor integration test DecodeJSON calls to use generic return value (go-gitea#37432) Integrate renovate bot for all dependency updates (go-gitea#37050) Refactor pull request view (2) (go-gitea#37428) Use MarkLongPolling instead of hard-coded route path (go-gitea#37427) Optimize CI caches (go-gitea#37387) Update AGENTS.md (go-gitea#37420) Update Nix flake (go-gitea#37425) [skip ci] Updated translations via Crowdin remove excessive quote from terraform instructions (go-gitea#37424) Improve testing init, clean up webhook tests (go-gitea#37412) Fix color regressions, add `priority` color (go-gitea#37417) [skip ci] Updated translations via Crowdin Stabilize e2e logout propagation test (go-gitea#37403) refactor: serve site manifest via `/assets/site-manifest.json` endpoint (go-gitea#37405) feat(security): set X-Content-Type-Options: nosniff by default (go-gitea#37354) Refactor pull request view (1) (go-gitea#37380) ... # Conflicts: # templates/repo/diff/box.tmpl

* main: (33 commits) refactor: use named `Permission` field in `Repository` struct instead of anonymous embedding (go-gitea#37441) Refactor pull request view (3) (go-gitea#37439) Update 1.26.1 changelog in main (go-gitea#37442) Make GetPossibleUserByID can handle deleted user (go-gitea#37430) Fix fetch action redirect (go-gitea#37437) Refactor integration test DecodeJSON calls to use generic return value (go-gitea#37432) Integrate renovate bot for all dependency updates (go-gitea#37050) Refactor pull request view (2) (go-gitea#37428) Use MarkLongPolling instead of hard-coded route path (go-gitea#37427) Optimize CI caches (go-gitea#37387) Update AGENTS.md (go-gitea#37420) Update Nix flake (go-gitea#37425) [skip ci] Updated translations via Crowdin remove excessive quote from terraform instructions (go-gitea#37424) Improve testing init, clean up webhook tests (go-gitea#37412) Fix color regressions, add `priority` color (go-gitea#37417) [skip ci] Updated translations via Crowdin Stabilize e2e logout propagation test (go-gitea#37403) refactor: serve site manifest via `/assets/site-manifest.json` endpoint (go-gitea#37405) feat(security): set X-Content-Type-Options: nosniff by default (go-gitea#37354) ...

…-review-feedback * origin/main: (144 commits) Add API endpoint to reply to pull request review comments (go-gitea#36683) Add CurrentURL template variable back (go-gitea#37444) refactor: use named `Permission` field in `Repository` struct instead of anonymous embedding (go-gitea#37441) Refactor pull request view (3) (go-gitea#37439) Update 1.26.1 changelog in main (go-gitea#37442) Make GetPossibleUserByID can handle deleted user (go-gitea#37430) Fix fetch action redirect (go-gitea#37437) Refactor integration test DecodeJSON calls to use generic return value (go-gitea#37432) Integrate renovate bot for all dependency updates (go-gitea#37050) Refactor pull request view (2) (go-gitea#37428) Use MarkLongPolling instead of hard-coded route path (go-gitea#37427) Optimize CI caches (go-gitea#37387) Update AGENTS.md (go-gitea#37420) Update Nix flake (go-gitea#37425) [skip ci] Updated translations via Crowdin remove excessive quote from terraform instructions (go-gitea#37424) Improve testing init, clean up webhook tests (go-gitea#37412) Fix color regressions, add `priority` color (go-gitea#37417) [skip ci] Updated translations via Crowdin Stabilize e2e logout propagation test (go-gitea#37403) ... # Conflicts: # models/project/column.go # routers/web/repo/issue_page_meta.go

GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Apr 23, 2026

silverwind marked this pull request as draft April 23, 2026 15:00

silverwind changed the title ~~CI: give each Go test job its own build cache key~~ Optimize CI caches Apr 23, 2026

silverwind added type/testing performance/speed performance issues with slow downs labels Apr 23, 2026

This comment was marked as outdated.

Sign in to view

silverwind mentioned this pull request Apr 23, 2026

CI: drop -coverprofile from unit-tests to enable Go test result cache #37389

Closed

silverwind and others added 10 commits April 23, 2026 21:30

restart ci

2c590a1

CI: drop trailing hyphens from Go cache restore-keys

536377d

Cosmetic. Each job's prefix word is unique in our cache namespace, so no cross-job partial match is possible. Co-Authored-By: Claude (Opus 4.7) <noreply@anthropic.com>

restart ci

a91e03e

silverwind force-pushed the ci-go-cache-per-job-key branch from d82d9ef to f896743 Compare April 23, 2026 19:41

silverwind marked this pull request as ready for review April 23, 2026 19:50

silverwind and others added 6 commits April 23, 2026 22:18

CI: fix lint-yaml in cache-seeder matrix

12f0856

Yamllint rejects the column-aligned inline mappings; collapse to single spaces after commas. Co-Authored-By: Claude (Opus 4.7) <noreply@anthropic.com>

Apply suggestion from @silverwind

80ce1eb

Signed-off-by: silverwind <me@silverwind.io>

CI: restore description field on composite action

9313735

actionlint requires 'description' in composite action metadata; removing it fails lint-actions. Co-Authored-By: Claude (Opus 4.7) <noreply@anthropic.com>

restart ci

a6193b1

silverwind added skip-changelog This PR is irrelevant for the (next) changelog, for example bug fixes for unreleased features. and removed performance/speed performance issues with slow downs labels Apr 23, 2026

silverwind requested a review from Copilot April 23, 2026 21:47

Merge branch 'main' into ci-go-cache-per-job-key

c650c36

Copilot AI reviewed Apr 23, 2026

View reviewed changes

Comment thread .github/actions/go-cache/action.yml

Comment thread .github/actions/go-cache/action.yml

Comment thread .github/workflows/cache-seeder.yml

Comment thread .github/workflows/pull-db-tests.yml

GiteaBot added the lgtm/need 1 This PR needs approval from one additional maintainer to be merged. label Apr 23, 2026

wxiaoguang reviewed Apr 24, 2026

View reviewed changes

Comment thread .github/actions/go-cache/action.yml Outdated

wxiaoguang reviewed Apr 24, 2026

View reviewed changes

Comment thread .github/workflows/cache-seeder.yml

wxiaoguang marked this pull request as draft April 24, 2026 07:36

silverwind and others added 2 commits April 24, 2026 12:07

CI: document PR cache consumption path in seeder header

2d6206b

Co-Authored-By: Claude (Opus 4.7) <noreply@anthropic.com>

silverwind commented Apr 24, 2026

View reviewed changes

Comment thread .github/workflows/cache-seeder.yml

Apply suggestion from @silverwind

42d438a

Signed-off-by: silverwind <me@silverwind.io>

silverwind marked this pull request as ready for review April 24, 2026 10:22

silverwind and others added 4 commits April 24, 2026 12:22

Merge branch 'main' into ci-go-cache-per-job-key

9820fae

CI: reword boolean input descriptions to 'Whether to …' form

ad3bf75

Co-Authored-By: Claude (Opus 4.7) <noreply@anthropic.com>

Merge branch 'main' into ci-go-cache-per-job-key

39fa0ce

CI: pin third-party actions to commit hashes

e0e4e59

Co-Authored-By: Claude (Opus 4.7) <noreply@anthropic.com>

bircni approved these changes Apr 26, 2026

View reviewed changes

GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Apr 26, 2026

Merge branch 'main' into ci-go-cache-per-job-key

de9cad5

silverwind enabled auto-merge (squash) April 26, 2026 10:01

silverwind added the reviewed/wait-merge This pull request is part of the merge queue. It will be merged soon. label Apr 26, 2026

Merge branch 'main' into ci-go-cache-per-job-key

d233488

silverwind merged commit ebf30ac into go-gitea:main Apr 26, 2026
26 checks passed

silverwind deleted the ci-go-cache-per-job-key branch April 26, 2026 10:25

GiteaBot added this to the 1.27.0 milestone Apr 26, 2026

GiteaBot removed the reviewed/wait-merge This pull request is part of the merge queue. It will be merged soon. label Apr 26, 2026

silverwind added a commit to silverwind/gitea that referenced this pull request Apr 26, 2026

Merge remote-tracking branch 'origin/main' into renovate

0132ab4

* origin/main: Optimize CI caches (go-gitea#37387)

silverwind mentioned this pull request Apr 26, 2026

Make integration tests cacheable #37434

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Optimize CI caches#37387

Optimize CI caches#37387
silverwind merged 27 commits intogo-gitea:mainfrom
silverwind:ci-go-cache-per-job-key

silverwind commented Apr 23, 2026 •

edited

Loading

Uh oh!

This comment was marked as outdated.

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

silverwind commented Apr 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

silverwind commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Warm-cache runtime (last run before this PR vs latest run here)

Uh oh!

This comment was marked as outdated.

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

silverwind commented Apr 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

silverwind commented Apr 23, 2026 •

edited

Loading