fix(function-autoscaler): build linux/arm64 images - #533
Merged
Conversation
function-autoscaler was the only one of nineteen Bazel modules producing a single-architecture image index. Three independent defects had to be fixed together; each one alone still fails the build. The oci.pull platform list named "linux/arm64", but nvcr.io/nvidia/distroless/cc publishes its aarch64 manifest under the v8 variant and rules_oci matches the platform string exactly, so the pull failed outright. http-invocation pins the same image and digest as arm64/v8 and has always built both architectures. openssl-sys checks <TARGET>_OPENSSL_LIB_DIR before the unprefixed name. Only the unprefixed form was set, so an aarch64 build silently resolved to the x86_64 library path. Explicit per-target entries replace that fallback. libssl-dev is Multi-Arch: same and splits its headers: the arch-independent ones are shared under /usr/include/openssl/ while opensslconf.h and configuration.h are per-arch under /usr/include/<triplet>/openssl/. Host gcc searches its own triplet directory implicitly, which is why amd64 never needed this, but the zig toolchain used for arm64 is hermetic and searches neither. OPENSSL_INCLUDE_DIR holds a single directory, so the triplet path is supplied through the cc crate's target-suffixed CFLAGS. Vendored OpenSSL is not an alternative: openssl-src's bundled source tree is not materialised in the crate_universe sandbox and the build fails with "openssl-src-.../openssl: No such file or directory", with or without a build_script_data annotation. Requires a bazel-ci image carrying libssl-dev:arm64. Ubuntu's aarch64 libssl.so.3 requires at most GLIBC_2.38, which the registered linux_arm64_gnu.2.38 zig toolchain provides. Co-authored-by: Balaji Ganesan <bganesan@nvidia.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0.14.0 adds libssl-dev:arm64, which the function-autoscaler arm64 build in the preceding commit needs in order to resolve aarch64 openssl. These are the fallback values behind the BAZEL_CI_IMAGE repository variable; when that variable is set it takes precedence and must be bumped alongside. Merge only after the image is published and mirrored to ghcr. Pinning a tag that does not exist yet fails every job in the matrix at container pull, before any step runs. Co-authored-by: Balaji Ganesan <bganesan@nvidia.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
📝 WalkthroughWalkthroughThe pull request updates default Bazel CI images to 0.14.0 and extends function-autoscaler build and OCI configuration to support linux/arm64 alongside linux/amd64. It adds OCI image-index architecture validation with nested and flat layout regression tests. ChangesCI image updates
Function autoscaler arm64 support
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant create_oci_image
participant image_index_platforms_test.sh
participant OCIImageIndex
create_oci_image->>image_index_platforms_test.sh: Pass generated OCI layout and required architectures
image_index_platforms_test.sh->>OCIImageIndex: Resolve index, manifest, and config blobs
OCIImageIndex-->>image_index_platforms_test.sh: Return declared architectures
image_index_platforms_test.sh-->>create_oci_image: Report platform coverage result
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/control-plane-services/function-autoscaler/MODULE.bazel`:
- Around line 113-152: Add repository-native regression coverage for the
autoscaler configuration around crate.annotation, including a CI build targeting
//platforms:linux_arm64 and verification that the published image index contains
both AMD64 and ARM64 variants. If such tests cannot be added, document the
omission and rationale in the pull request description.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: afadc083-d2a7-4f95-83c5-d2fcd85daf0e
📒 Files selected for processing (5)
.github/workflows/bazel.yml.github/workflows/chart-push-manual.yml.github/workflows/image-push-manual.ymlsrc/control-plane-services/function-autoscaler/MODULE.bazelsrc/control-plane-services/function-autoscaler/rules/oci/private/common.bzl
balajinvda
requested review from
borao and
sbaum1994
and removed request for
Max-NV and
nvaghela-oss
July 29, 2026 19:34
This service published an amd64-only image index for months and the build stayed green throughout. Dropping a platform is a configuration edit, not a compile error: DEFAULT_PLATFORMS lost an entry and every downstream target happily built the smaller index. Only someone reading the registry would have noticed. create_oci_image now generates a platform-coverage test alongside every image. It walks the OCI layout the same way the Java contract test does (index.json -> manifest list -> per-platform manifest -> config) and checks both directions: a required architecture missing, and an architecture present that nobody declared. Following the descriptor through to the config blob also catches an entry filed under one platform while pointing at an image built for another, which pushes without complaint and then runs the wrong binary. The expected architectures are a separate REQUIRED_ARCHES constant rather than something derived from DEFAULT_PLATFORMS. Deriving them would make the test restate whatever the build already decided, so removing arm64 would also remove it from the expectation and the test would pass. That is precisely the regression being guarded against. The generated test is deliberately not tagged manual. The image targets are, so inheriting their tags would keep it out of `bazel test //...` and leave the guard inert. image_index_platforms_selftest exercises the guard against synthetic layouts, because the guard itself only ever sees well-formed indexes in a healthy build and nothing else would catch it passing when it should fail. Writing it was worthwhile: it caught the guard reporting success on an index whose arm64 manifest blob was missing. Co-authored-by: Balaji Ganesan <bganesan@nvidia.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@src/control-plane-services/function-autoscaler/rules/oci/private/test_image_index_platforms.sh`:
- Around line 28-53: Extend make_layout to support a flat OCI index case where
index.json contains the manifests array directly and no separate list blob is
created. Add a self-test fixture invoking this mode so
image_index_platforms_test.sh exercises both the existing nested layout and the
spec-standard flat layout.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 3ef9b5ba-f49f-4283-b209-b5ed158d6b0a
📒 Files selected for processing (5)
src/control-plane-services/function-autoscaler/MODULE.bazelsrc/control-plane-services/function-autoscaler/rules/oci/private/BUILD.bazelsrc/control-plane-services/function-autoscaler/rules/oci/private/common.bzlsrc/control-plane-services/function-autoscaler/rules/oci/private/image_index_platforms_test.shsrc/control-plane-services/function-autoscaler/rules/oci/private/test_image_index_platforms.sh
🚧 Files skipped from review as they are similar to previous changes (1)
- src/control-plane-services/function-autoscaler/MODULE.bazel
sbaum1994
approved these changes
Jul 29, 2026
balajinvda
enabled auto-merge
July 29, 2026 20:03
…orm guard The guard walked index.json -> nested index blob -> manifest -> config, and the self-test built fixtures in exactly that shape. Both encoded the same assumption, so the self-test could not have caught the assumption being wrong. That shape is what rules_oci emits today: oci/private/image_index.sh.tpl sets index.json's .manifests to a single application/vnd.oci.image.index.v1+json descriptor, putting the platform descriptors one level down. The OCI spec also permits them inline in index.json, and other tooling emits that flatter form. Rather than depend on which shape the ruleset happens to produce, the guard now detects it: platform architectures present in index.json means the descriptors are inline, otherwise it follows the single descriptor to the nested blob. This matters because the failure mode is quiet. Against an unexpected layout the walk finds no architectures at all, which reads as a missing platform rather than as a guard that no longer works, and the obvious "fix" for that noise would be to stop failing on an empty result. The self-test now covers the flat form independently, including the missing-arm64 and mislabelled-config cases, so neither layout is validated only by a fixture that shares its assumptions. Co-authored-by: Balaji Ganesan <bganesan@nvidia.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
borao
approved these changes
Jul 29, 2026
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@src/control-plane-services/function-autoscaler/rules/oci/private/image_index_platforms_test.sh`:
- Around line 59-65: The JSON normalization in image_index_platforms_test.sh
must remove tabs and carriage returns as well as spaces and newlines before the
architecture guard matches. In
src/control-plane-services/function-autoscaler/rules/oci/private/test_image_index_platforms.sh
lines 50-54, add a fixture mode producing valid tab- or CRLF-formatted flat
JSON; in lines 133-152, invoke the guard with that formatted fixture to cover
the normalization path.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 9e14d9f3-a418-4e53-890c-f24e2fe3df36
📒 Files selected for processing (2)
src/control-plane-services/function-autoscaler/rules/oci/private/image_index_platforms_test.shsrc/control-plane-services/function-autoscaler/rules/oci/private/test_image_index_platforms.sh
Contributor
Author
|
🎉 This PR is included in version nvcf-function-autoscaler-v1.18.10 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Closes #527.
function-autoscaler is the only one of nineteen Bazel modules publishing a
single-architecture image index. Every other service already ships amd64 and
arm64.
The service was deliberately pinned to amd64 with a code comment blaming
openssl-sys. That comment was accurate about the symptom but the cause was
three separate defects stacked on top of each other, and fixing any one alone
still fails the build. That is why earlier attempts kept concluding the problem
was somewhere else.
What changed
oci.pullnamed the platformlinux/arm64, butnvcr.io/nvidia/distroless/ccpublishes its aarch64 manifest under the v8 variant and rules_oci matches the
platform string exactly, so the pull failed outright. http-invocation pins the
same image and the same digest as
linux/arm64/v8and has always built botharchitectures. This defect alone blocks the build regardless of openssl.
openssl-sys checks
<TARGET>_OPENSSL_LIB_DIRbefore the unprefixed name. Onlythe unprefixed form was set, so an aarch64 build silently resolved to the
x86_64 library path rather than failing loudly. Explicit per-target entries
replace that fallback.
libssl-devisMulti-Arch: sameand splits its headers: 142arch-independent headers are shared at
/usr/include/openssl/, whileopensslconf.handconfiguration.hare per-arch under/usr/include/<triplet>/openssl/. Host gcc searches its own triplet directoryimplicitly, which is why amd64 never needed this; the zig toolchain used for
arm64 is hermetic and searches neither.
OPENSSL_INCLUDE_DIRholds a singledirectory and cannot express the pair, so the triplet path is supplied through
the cc crate's target-suffixed
CFLAGS.DEFAULT_PLATFORMSin the service's forkedrules/ociregainslinux_arm64.Alternatives ruled out, with evidence
Vendored OpenSSL:
openssl-src's bundled source tree is not materialised in thecrate_universe sandbox. The build fails with
openssl-src-.../openssl: No such file or directory, with and without abuild_script_dataannotation. An earlier test that appeared to succeed wasinvalid because
OPENSSL_NO_VENDOR=1was still set.Porting Cassandra TLS from openssl to rustls (#529): works, but it is a change
to production TLS code to solve a build-configuration problem, and it would have
been the wrong trade here. Superseded by this PR.
Replacing zig with
crossbuild-essential-arm64: its default include path coversthe triplet directory, so defect 3 would resolve itself. Rejected because zig is
a Bazel-fetched dependency that installs nothing in the CI image, whereas the
cross compiler is a real image install and would need a hand-written
cc_toolchain_config(the repo has none), diverging this module from theeighteen others that all register zig.
Customer Release Notes
function-autoscaler container images are now published for linux/arm64 in
addition to linux/amd64.
Plan Summary
Not applicable.
Usage
Not applicable.
Testing
Not verifiable on the current CI image, which is why this is a draft. The build
needs
libssl-dev:arm64, added in bazel-ci 0.14.0.Two package properties were verified directly against the Ubuntu arm64 debs,
since both are load-bearing and neither is obvious:
readelfon aarch64libssl.so.3shows a maximum requirement ofGLIBC_2.38,matching the registered
linux_arm64_gnu.2.38toolchain. No symbol-versionwall at link time.
libssl-devarm64 deb contains exactly two files under the tripletinclude directory and 142 under the shared one, which is what defect 3 above
is about.
Notes
Draft until bazel-ci 0.14.0 is published and mirrored to ghcr. The second commit
bumps the three fallback pins; merging before the tag exists fails every matrix
job at container pull.
Nothing enforces that a forked
rules/ocilists both platforms, so this classof regression can recur silently in another module. Worth a follow-up guard.
References
Closes #527.
Related Merge Requests/Pull Requests
Requires the bazel-ci image MR adding arm64 libssl. Supersedes #529.
Dependencies
None.
Summary by CodeRabbit
New Features
Bug Fixes
Chores