feat(infra): Implement parallel base image builds #14112

hunsche · 2025-10-10T20:40:22Z

What this PR does

This PR refactors the base image build process to support parallel builds for multiple Ubuntu versions (legacy, 20.04, 24.04), aligning with the project's multi-version image strategy.

The primary goal is to enable a flexible and controlled migration path from the legacy image to newer Ubuntu versions, while ensuring stability and consistency.

Why this PR is important

Previously, the base image build was a monolithic process tied to a single version. This change introduces the necessary infrastructure to:

Build, test, and publish images for all supported Ubuntu versions simultaneously.
Safely promote a new Ubuntu version (e.g., ubuntu-20-04) to become the new default (:v1 tag) via a single configuration change, without race conditions.
Ensure that all versioned tags (e.g., :ubuntu-24-04) are also multi-architecture, providing a consistent experience across all images.

How the changes were implemented

Parallel Build Trigger: The main base_builder Cloud Function in infra/build/functions/base_images.py was refactored. It now iterates through the SUPPORTED_VERSIONS and triggers a separate, parallel Google Cloud Build run for each version.
Configurable Default Version: A DEFAULT_VERSION constant was introduced. The logic that applies the primary :v1 tag and builds the multi-arch manifest for it is now tied to this variable, making the promotion of a new default version a simple, one-line change.
Generalized Manifest Creation: The logic to create multi-architecture manifests was generalized and is now applied to all build versions, ensuring that tags like :ubuntu-20-04 and :ubuntu-24-04 are also multi-arch.
Simplified Tagging: The redundant :legacy tag was removed. The build designated as DEFAULT_VERSION now produces the :v1 tag. To ensure compatibility with chained builds, it also produces the :latest tag. All other builds produce only their specific version tag (e.g., :ubuntu-20-04).
Local Testing Capabilities: The script now supports --dry-run and --no-push flags, allowing for robust local testing and validation of the build logic without affecting the image registry.
Bug Fixes:
- report_generator.py was enhanced to correctly handle build failures where no projects are run.
- build_lib.py was modified to return the full build object, enabling build tracking and logging of GCB URLs.
- Corrected a path resolution bug for the base-clang-full image variant.

How to test

Run the script locally with the dry run flag to inspect the generated build steps:
```
python3 infra/build/functions/base_images.py --dry-run
```
Trigger a real build without publishing the images to the registry:
```
python3 infra/build/functions/base_images.py --no-push
```
Monitor the build links logged to the console to confirm success.

Verification

The new logic was tested by triggering a real build on GCB using the --no-push flag. All three versioned builds completed successfully, confirming the fix.

Legacy (v1) Build: SUCCESS (View Log)
Ubuntu 20.04 Build: SUCCESS (View Log)
Ubuntu 24.04 Build: SUCCESS (View Log)

Related Task

Taskflow: b/441792502

Refactors the base image build process to support parallel builds for multiple Ubuntu versions (legacy, 20.04, 24.04), aligning with the multi-version image strategy. Key changes: - The main `base_builder` Cloud Function now iterates through supported versions and triggers a separate, parallel GCB build for each. - Implemented a configurable `DEFAULT_VERSION` variable to allow for safe and easy promotion of a new default image (which receives the `:v1` tag). - Generalized the multi-architecture manifest generation to apply to all versioned tags (e.g., `:ubuntu-24-04`), not just the default. - The `legacy` build now correctly produces both `:v1` and `:latest` tags to ensure compatibility with subsequent build steps. - Added local testing capabilities (`--dry-run`, `--no-push`) to validate build logic without deploying. - Enhanced the `report_generator.py` script to correctly handle build failures where no projects are run. - Modified `build_lib.py` to return the full build object, enabling build tracking.

hunsche · 2025-10-10T21:13:47Z

/gcbrun trial_build.py json-c zlib libarchive bad_example --fuzzing-engines libfuzzer --sanitizers address

evverx · 2025-10-11T02:49:19Z

infra/build/functions/base_images.py


+# Define the supported Ubuntu versions for base images.
+# 'legacy' refers to the unversioned, default image.
+SUPPORTED_VERSIONS = ('legacy', 'ubuntu-20-04', 'ubuntu-24-04')


I experimented with ubuntu-24-04 a bit and I wonder how mismatches are planned to be handled eventually? For example I appended the ubuntu-24-04 tag and then build_check went sideways with

BAD BUILD: /tmp/not-out/tmp89q_bpe8/fuzz-packet seems to have either startup crash or exit: sysctl: permission denied on key "vm.mmap_rnd_bits", ignoring /tmp/not-out/tmp89q_bpe8/fuzz-packet -- -rss_limit_mb=2560 -timeout=25 -seed=1337 -runs=4 < /dev/null /tmp/not-out/tmp89q_bpe8/fuzz-packet: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /tmp/not-out/tmp89q_bpe8/fuzz-packet) /tmp/not-out/tmp89q_bpe8/fuzz-packet: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /tmp/not-out/tmp89q_bpe8/fuzz-packet) /tmp/not-out/tmp89q_bpe8/fuzz-packet: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by /tmp/not-out/tmp89q_bpe8/fuzz-packet)

because the fuzz target was built on Ubuntu 24.04 but build_check pulled the latest base-runner with Ubuntu 20.04. I wonder if the idea is to look at Dockerfile, extract the tag and then pull the corresponding base-runner images or something like that? With no tags in Dockerfiles is it safe to assume that the underlying latest images keep rolling forward and point to the latest Ubuntu releases available on OSS-Fuzz eventually?

Thanks for the excellent point! You've correctly identified a key challenge with the version mismatch.

We are still finalizing the tests for this migration, but your suggestion is exactly the plan: to make build_check and other tools aware of the chosen version to ensure the entire pipeline (build, check, and execution) uses the corresponding images.

For now, we recommend sticking with the current default version. In the coming weeks, we'll officially enable the option to select ubuntu-24-04 in project.yaml. By then, all the tooling will be ready to handle versioning intelligently.

This is now implemented in PR #14128.

The check_build command (and other runner commands like reproduce and run_fuzzer) is now aware of the base_os_version. It dynamically selects the correct base-runner image tag to match the environment the fuzzers were built in.

This guarantees that we won't have GLIBC mismatches, as the entire pipeline now uses the corresponding versioned images from build to test.

Got it. I left a comment in #14128 (comment) though.

Also looking at #14128 it seems latest is going to point to Ubuntu 20.04 and I think it's kind of confusing in that I'd expect latest to point to the latest release. I understand the reason behind the decision but it would be great if it was documented eventually probably. I'd certainly use latest and hope that it would roll forward :-)

The variable `BASE_IMAGES` was renamed to `BASE_IMAGE_DEFS` in the `base_images` module, but this change was not propagated to the trial build script, causing an `AttributeError`. This commit updates `build_and_push_test_images.py` to use the correct `BASE_IMAGE_DEFS` variable, resolving the error and allowing trial builds to run successfully.

hunsche · 2025-10-13T13:15:51Z

/gcbrun trial_build.py zlib bad_example --fuzzing-engines libfuzzer --sanitizers address

hunsche · 2025-10-13T13:17:27Z

/gcbrun trial_build.py zlib bad_example --fuzzing-engines libfuzzer --sanitizers address

hunsche · 2025-10-13T13:48:23Z

/gcbrun trial_build.py zlib bad_example --fuzzing-engines libfuzzer --sanitizers address

vitaliset · 2025-10-13T15:29:42Z

infra/build/functions/base_images.py

+    """
+    Resolves the path to the Dockerfile, preferring a version-specific
+    one if it exists, otherwise falling back to the legacy Dockerfile.
+    """


Can you fix it to be one line or one line + paragraph? go/pystyle#function-docs

vitaliset · 2025-10-13T15:29:50Z

infra/build/functions/base_images.py

 # See the License for the specific language governing permissions and
-# limitations under the License.
-#
+# limitations under the License.n#


Please revert this change.

Done, reverted.

vitaliset · 2025-10-13T15:30:45Z

infra/build/functions/base_images.py

+  """
+    Returns the steps to push a manifest pointing to ARM64 and AMD64
+    versions of an |image| with a specific |target_tag|.
+    """


vitaliset · 2025-10-13T15:30:52Z

infra/build/functions/base_images.py

+  """
+    Returns steps to create and push a multi-architecture manifest for
+    the base-builder and base-runner images with a specific tag.
+    """


vitaliset · 2025-10-13T15:31:01Z

infra/build/functions/base_images.py

+  """
+    Executes a build in GCB and pushes the resulting images, or prints the
+    configuration if in dry_run mode.
+    """


vitaliset · 2025-10-13T15:31:23Z

infra/build/functions/base_images.py

+  """
+    Cloud function entry point. Triggers parallel base image builds for each
+    supported Ubuntu version.
+    """


vitaliset · 2025-10-13T15:32:23Z

infra/build/functions/base_images.py

+  # This allows the script to be run locally for testing or deployment.
+  # By default, it performs a real build.
+  # To perform a dry run, pass the '--dry-run' flag.
+  # To prevent pushing images to the registry, pass '--no-push'.


Can you add it to the module docstring? Maybe with a run command example.

Done. Moved to the module docstring with an example.

vitaliset · 2025-10-13T15:32:54Z

infra/build/functions/report_generator.py

      any_results_found = True
      if data.get('failed_builds', 0) > 0:
        any_failures = True
+      # Use .get() for safe access to prevent potential KeyErrors.


Stale comment.

Done, removed.

vitaliset · 2025-10-13T15:33:07Z

infra/build/functions/report_generator.py

  generate_final_summary(all_results)

-  if any_failures:
+  # Define failure conditions explicitly for clarity.


Stale comment.

Done, removed.

vitaliset · 2025-10-13T15:34:30Z

infra/build/functions/build_lib.py

-  build_id = build_info['metadata']['build']['id']
-
-  return build_id
+  return build_info['metadata']['build']


I think we can revert this.

Kept as is, as returning the full build object provides access to other useful properties like logUrl.

hunsche · 2025-10-13T15:54:24Z

/gcbrun trial_build.py zlib bad_example --fuzzing-engines libfuzzer --sanitizers address

Responds to code review comments by: - Reformatting all function docstrings to meet the style guide. - Removing stale and redundant comments. - Moving local run instructions to the module docstring for better visibility. - Adding a clarifying comment for the SUPPORTED_VERSIONS variable.

hunsche · 2025-10-13T15:58:37Z

/gcbrun trial_build.py zlib bad_example --fuzzing-engines libfuzzer --sanitizers address

This reverts commit 4107371.

Reverts #14112 Reverting as the ":latest" images no longer seem to be pushed.

With google/oss-fuzz#14112 and google/oss-fuzz#14128, we can now use Ubuntu 24.04. Let's bump the image version.

With google/oss-fuzz#14112 and google/oss-fuzz#14128, we can now use Ubuntu 24.04. Let's bump the image version. Note, the i386 build failure mentioned in the removed comment is related to https://bugs.launchpad.net/ubuntu/+source/linux-signed-azure/+bug/2071445 actions/runner-images#9977 and has been already fixed.

hunsche and others added 2 commits October 10, 2025 20:33

Merge branch 'master' into feat/parallel-base-image-builds

0fb10d2

hunsche requested review from ViniciustCosta, javanlacerda and vitaliset October 10, 2025 20:42

style: Apply yapf formatter

e4bb26b

evverx mentioned this pull request Oct 10, 2025

Update ossfuzz base image to newer Ubuntu (22.04 or 24.04) ? #11835

Open

evverx reviewed Oct 11, 2025

View reviewed changes

style: Apply yapf formatter

3c5df94

Merge branch 'master' into feat/parallel-base-image-builds

79779ba

vitaliset approved these changes Oct 13, 2025

View reviewed changes

hunsche force-pushed the feat/parallel-base-image-builds branch from 3bd68e4 to 09dbb73 Compare October 13, 2025 15:58

hunsche merged commit 4107371 into master Oct 13, 2025
19 checks passed

hunsche deleted the feat/parallel-base-image-builds branch October 13, 2025 17:20

oliverchang added a commit that referenced this pull request Oct 16, 2025

Revert "feat(infra): Implement parallel base image builds (#14112)"

cfe1159

This reverts commit 4107371.

oliverchang mentioned this pull request Oct 16, 2025

Revert "feat(infra): Implement parallel base image builds" #14138

Merged

oliverchang added a commit that referenced this pull request Oct 16, 2025

Revert "feat(infra): Implement parallel base image builds" (#14138)

37d1fcd

Reverts #14112 Reverting as the ":latest" images no longer seem to be pushed.

yuwata added a commit to yuwata/systemd that referenced this pull request Oct 19, 2025

ci/oss-fuzz: switch to Ubuntu 24.04

f1b8d04

With google/oss-fuzz#14112 and google/oss-fuzz#14128, we can now use Ubuntu 24.04. Let's bump the image version.

yuwata mentioned this pull request Oct 19, 2025

ci/oss-fuzz: switch to Ubuntu 24.04 systemd/systemd#39371

Merged

feat(infra): Implement parallel base image builds #14112

feat(infra): Implement parallel base image builds #14112

Uh oh!

Conversation

hunsche commented Oct 10, 2025

What this PR does

Why this PR is important

How the changes were implemented

How to test

Verification

Related Task

Uh oh!

hunsche commented Oct 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hunsche commented Oct 13, 2025

Uh oh!

hunsche commented Oct 13, 2025

Uh oh!

hunsche commented Oct 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hunsche commented Oct 13, 2025

Uh oh!

hunsche commented Oct 13, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone