Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't cache m2 repo in GitHub Actions for PRs and take advantage of saved space #43559

Closed
yrodiere opened this issue Sep 27, 2024 · 4 comments
Closed
Assignees
Labels
area/housekeeping Issue type for generalized tasks not related to bugs or enhancements area/infra internal and infrastructure related issues

Comments

@yrodiere
Copy link
Member

yrodiere commented Sep 27, 2024

Description

We're currently using the GitHub Actions cache for very dubious things, especially on PRs:

  • m2 repo cache
  • Gradle setup
  • maybe more

Ideally we'd cache them, but it's 1 to 2GB per PR, which is not sustainable considering the global 10GB limit for all branches and PRs:

  1. We're likely only ever be caching up to 3 PRs at any given time.
  2. This is likely causing cache evictions for more important caches on the main branch, leading to cache misses for all builds (main or PR).

We should stop doing that, and take advantage of that saved space to cache other, more space-efficient things on PRs.

Implementation ideas

We should prevent cache writes for Maven dependencies / gradle setup on PRs. There's a lookup-only option on the cache action that does just that [EDIT: Wrong, it does something else]. We'd only write to the cache from branch builds, restricting ourselves to read on PRs (so PRs benefit from the main branch's cache). This would save several GBs from our current cache usage, between 2 and 4GB depending when you check (you can check here).

Using that saved space, we should enable cache read/write for the Develocity local build cache on PRs. This hopefully would take less that 4GB, and could result in much faster PR builds/testing, especially for small, incremental pushes to PRs.

See https://quarkusio.zulipchat.com/#narrow/stream/187038-dev/topic/Persistent.20Develocity.20build.20cache.20in.20PRs/near/435358207

@yrodiere yrodiere added the area/housekeeping Issue type for generalized tasks not related to bugs or enhancements label Sep 27, 2024
@quarkus-bot
Copy link

quarkus-bot bot commented Sep 27, 2024

/cc @gwenneg (cache)

@quarkus-bot
Copy link

quarkus-bot bot commented Sep 27, 2024

You added a link to a Zulip discussion, please make sure the description of the issue is comprehensive and doesn't require accessing Zulip

This message is automatically generated by a bot.

@yrodiere yrodiere added area/infra internal and infrastructure related issues and removed area/cache labels Sep 27, 2024
@yrodiere
Copy link
Member Author

/cc @gwenneg (cache)

Sorry for the ping: the bot is wrong, this is not related to quarkus-cache extensions.

@gsmet gsmet self-assigned this Sep 27, 2024
gsmet added a commit to gsmet/quarkus that referenced this issue Sep 27, 2024
We use the ones coming from the push events.
Related to quarkusio#43559
gsmet added a commit to gsmet/quarkus that referenced this issue Sep 27, 2024
This can be useful when pushing small iterations to a pull request.

It is limited to pull requests only and only for the Initial JDK build.
We will check if it improves things and the size of the cache.

Related to quarkusio#43559
gsmet added a commit to gsmet/quarkus that referenced this issue Sep 27, 2024
This can be useful when pushing small iterations to a pull request.

It is limited to pull requests only and only for the Initial JDK build.
We will check if it improves things and the size of the cache.

Related to quarkusio#43559
gsmet added a commit to gsmet/quarkus that referenced this issue Sep 28, 2024
This can be useful when pushing small iterations to a pull request.

It is limited to pull requests only and only for the Initial JDK build.
We will check if it improves things and the size of the cache.

Related to quarkusio#43559
gsmet added a commit to gsmet/quarkus that referenced this issue Sep 30, 2024
We use the ones coming from the push events.
Related to quarkusio#43559
gsmet added a commit to gsmet/quarkus that referenced this issue Sep 30, 2024
This can be useful when pushing small iterations to a pull request.

It is limited to pull requests only and only for the Initial JDK build.
We will check if it improves things and the size of the cache.

Related to quarkusio#43559
mskacelik pushed a commit to mskacelik/quarkus that referenced this issue Oct 2, 2024
We use the ones coming from the push events.
Related to quarkusio#43559
mskacelik pushed a commit to mskacelik/quarkus that referenced this issue Oct 2, 2024
This can be useful when pushing small iterations to a pull request.

It is limited to pull requests only and only for the Initial JDK build.
We will check if it improves things and the size of the cache.

Related to quarkusio#43559
gsmet added a commit to gsmet/quarkus that referenced this issue Oct 21, 2024
We use the ones coming from the push events.
Related to quarkusio#43559

(cherry picked from commit e788d0e)
gsmet added a commit to gsmet/quarkus that referenced this issue Oct 21, 2024
This can be useful when pushing small iterations to a pull request.

It is limited to pull requests only and only for the Initial JDK build.
We will check if it improves things and the size of the cache.

Related to quarkusio#43559

(cherry picked from commit 287ce49)
@gsmet
Copy link
Member

gsmet commented Oct 28, 2024

I think we are now in a much better state.

@gsmet gsmet closed this as completed Oct 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/housekeeping Issue type for generalized tasks not related to bugs or enhancements area/infra internal and infrastructure related issues
Projects
None yet
Development

No branches or pull requests

2 participants