Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI disk space issues #2307

Open
Nemo157 opened this issue Nov 8, 2023 · 5 comments
Open

CI disk space issues #2307

Nemo157 opened this issue Nov 8, 2023 · 5 comments

Comments

@Nemo157
Copy link
Member

Nemo157 commented Nov 8, 2023

https://github.com/rust-lang/docs.rs/actions/runs/6799427301 just failed with an out of disk space error.

Looking at the restore cache step of https://github.com/rust-lang/docs.rs/actions/runs/6799427301/job/18486983082 it's restoring a 3GB cache, which is both slow and probably contributing to the disk space issues. (It appears to fail while attempting to create the new cache to upload).

@Nemo157
Copy link
Member Author

Nemo157 commented Nov 8, 2023

(The cache also appears to be only partially effective, there were ~280 dependencies built for a PR that doesn't touch the lockfile).

@Nemo157
Copy link
Member Author

Nemo157 commented Nov 8, 2023

Running with debug logging, I see quite a few duplicate -sys crates included in the cache, the cache action only removes files after a week of no access, we may need to use something more aggressive about cleaning it up.

@syphar
Copy link
Member

syphar commented Nov 8, 2023

It seems like this is a systemic issue with Swatinem/rust-cache#158 , and not sure how to solve it.

we probably could use cargo sweep and delete unused files directly or a day after they aren't used any more.

Also, when it's coming from all the -sys crates, it's related to the amount of dependency updates we do, which means we also could purge the cache once, and then see if it happens more often.

@Nemo157
Copy link
Member Author

Nemo157 commented Nov 8, 2023

Since we are doing infrequent mass dependency updates, I think what we could do is not restore during updates, use that as a sync point to purge the cache and start a new one. I'll take a look at the action a bit more tomorrow and see if that's something that could easily be supported by it. (Preferably while still keeping the incremental lockfile update support for PRs that only affect a few dependencies, so somehow keyed off just the mass updates).

@Nemo157
Copy link
Member Author

Nemo157 commented Nov 9, 2023

I can't see any reasonable way to do it from just our workflow config, opened Swatinem/rust-cache#177

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants