Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only build attributes not found in any cache. #40

Open
Mic92 opened this issue Apr 16, 2020 · 8 comments
Open

Only build attributes not found in any cache. #40

Mic92 opened this issue Apr 16, 2020 · 8 comments
Labels
enhancement New feature or request

Comments

@Mic92
Copy link
Contributor

Mic92 commented Apr 16, 2020

This would speed up CI builds significantly and save bandwidth.
Currently the cachix-action will pull built packages:
https://github.com/Mic92/nur-packages/runs/589727774
In the example above there were 500 MB pulled without a single package built.

This is how it can be implemented:

  1. Get all packages: nix-env -f ./non-broken.nix -qaP \* --out-path --xml --meta > packages.xml
  2. Get all build hashes: grep -oP '(?<=/nix/store/)[^-]*' < packages.xml (obviously this should be done by proper xml parsing)
  3. Only build those attributes where curl -I https://<binary-cache>/<hash>.narinfo does not return a 404
@domenkozar
Copy link
Member

Similar idea for upstream NixOS/nix#3428

@Mic92
Copy link
Contributor Author

Mic92 commented Apr 16, 2020

Once there would be a wrapper around nix-build, error logging could be improved as well. Right now it nix-build is quite verbose and it is hard to search for the error that broke a particular package because the build log is quite mix of all builds. It would be better if there was a nix build output that would just iterate over full logs of each failed package individually at the end of execution.

@domenkozar domenkozar added the enhancement New feature or request label Apr 21, 2020
@domenkozar
Copy link
Member

This should probably be a wrapper or (I prefer) part of nix-build. Either way I'll leave it open here since it's relevant.

@Mic92
Copy link
Contributor Author

Mic92 commented Apr 23, 2020

This is wrapper I came up with: https://github.com/Mic92/ci-nix-build
It's a single static binary (2mb). Still look for a better name.

@domenkozar
Copy link
Member

Nice! :)

I think it's going to be tricky to get argument passing to nix-build to work, but it's quite nice for 90% of the cases where nix-build does the job.

@Mic92
Copy link
Contributor Author

Mic92 commented Apr 23, 2020

I will add support for some build arguments later.

@zimbatm
Copy link
Contributor

zimbatm commented Apr 23, 2020

nice! I have a name suggestion: nix-build-uncached. I started adding a nix-build --uncached attribute to Nix itself but this is still at a very early stage.

@Mic92
Copy link
Contributor Author

Mic92 commented Apr 24, 2020

https://github.com/Mic92/nix-build-uncached it is. Even if your flag lands in Nix it is still going to take a while until it ends in a released version. Until than nix-build-cached will pay the builds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants