infra: make it possible to cache docker containers build #12284
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: David Korczynski <[email protected]>
DonggeLiu
approved these changes
Jul 30, 2024
oliverchang
approved these changes
Aug 2, 2024
|
/gcbrun skip |
DavidKorczynski
added a commit
to google/oss-fuzz-gen
that referenced
this pull request
Aug 10, 2024
First touch on #499 Depends on: google/oss-fuzz#12284 The way this work is by saving a cached version of `build_fuzzers` post running of `compile` and then modifying the Dockerfiles of a project to use this cached build image + an adjusted build script. For example, for brotli the Dockerfile is originally: ```sh FROM gcr.io/oss-fuzz-base/base-builder RUN apt-get update && apt-get install -y cmake libtool make RUN git clone --depth 1 https://github.com/google/brotli.git WORKDIR brotli COPY build.sh $SRC/ COPY 01.c /src/brotli/c/fuzz/decode_fuzzer.c ``` a Dockerfile is then created which relies on the cached version, and it loosk like: ```sh FROM cached_image_brotli # RUN apt-get update && apt-get install -y cmake libtool make # # RUN git clone --depth 1 https://github.com/google/brotli.git # WORKDIR brotli # COPY build.sh $SRC/ # COPY 01.c /src/brotli/c/fuzz/decode_fuzzer.c # COPY adjusted_build.sh $SRC/build.sh ``` `adjusted_build.sh` is then the script that only builds fuzzers. This means we can also use `build_fuzzers`/`compile` workflows as we know it. More specifically, this PR: - Makes it possible to build Docker images of fuzzer build containers. Does this by running `build_fuzzers`, saving the docker container and then commit the docker container to an image. This image will have a projects' build set up post running of `compile`. This is then used when building fuzzers by OFG. - Supports only ASAN mode for now. Should be easy to extend to coverage too. - Currently builds images first and then uses them locally. We could extend, probably on another step of this, to use containers pushed by OSS-Fuzz itself. - Only does the caching if a "cache-build-script" exists (added a few for some projects) which contains the build instructions post-build process. It should be easy to extend such that we can rely on some DB of auto-generated build scripts as well (ref: google/oss-fuzz#11937) but I think it's nice to have both the option of us creating the scripts ourselves + an auto-generated DB. --------- Signed-off-by: David Korczynski <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.