fix(bazel): repin distroless_static and make the nvsnap base multi-arch - #580
Merged
Conversation
`bazel build //...` fails at analysis on any checkout without a warm Bazel
repository cache:
ERROR: src/compute-plane-services/nvsnap/BUILD.bazel:66:13:
nvsnap_server_image_pre_transitioned depends on
@@rules_oci++oci+distroless_static which failed to fetch
WARNING: Download from https://gcr.io/v2/distroless/static/manifests/
sha256:dd7614b5... failed: GET returned 404 Not Found
The pinned digest was garbage-collected upstream. gcr.io answers it with
"manifest unknown" while the tag itself still resolves, so this is not
authentication or network. Digests are immutable but the manifests behind them
are not retained forever once untagged.
CI never noticed because it restores the Bazel repository cache, so the fetch
was served from cache and a dead pin looked healthy. That is fragile in a
specific, already-live way: the Actions cache is over its quota and evicting
continuously, so this was one eviction away from breaking main with an error
that reads like a network blip. It had already blocked an external Black Duck
scan, which runs cold and hit it immediately.
check-oci-pins verifies every oci.pull digest still resolves. It is a scheduled
check rather than a per-change one, because a pin rots without anyone editing
it.
The check carries a self-test for its reference handling. While investigating
this I twice reported healthy pins as dead, because appending @digest to an
image reference that already carries a tag produces an invalid reference that
skopeo rejects. It also retries before declaring a pin dead, so a registry
timeout is not mistaken for a missing image, and it fails if it parses no pins
at all rather than reporting success over an empty set.
Co-authored-by: Balaji Ganesan <bganesan@nvidia.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Repinning the digest exposed a second failure one layer down:
ERROR: configurable attribute "actual" in @@rules_oci++oci+distroless_static
doesn't match this configuration: could not find an image matching the
target platform.
ERROR: Analysis of target
//src/compute-plane-services/nvsnap:nvsnap_agent_image_multi_arch failed
go_oci_image always emits a _multi_arch target over DEFAULT_PLATFORMS, which is
arm64 and x86_64. @distroless_static was pulled for linux/amd64 only, so the
arm64 branch of the transition had no matching image. nvsnap was the only
service whose base disagreed with the macro.
`bazel build //...` never showed this because those targets are tagged manual
and wildcard builds skip them. `cquery deps(//...)` ignores manual, so an
external scan hit it immediately while CI stayed green.
With both fixes the exact Black Duck command now succeeds on Linux against
//..., returning 347 java_import targets where it previously produced none.
Co-authored-by: Balaji Ganesan <bganesan@nvidia.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe OCI module configuration repins the distroless static image and adds arm64 support. A Python CI utility parses OCI pins, validates digest resolution and declared platform coverage with ChangesOCI pin management
Estimated code review effort: 4 (Complex) | ~45 minutes Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant check_oci_pins
participant MODULE_bazel
participant skopeo
participant OCIRegistry
check_oci_pins->>MODULE_bazel: parse OCI image, digest, and platforms
check_oci_pins->>skopeo: inspect image@digest
skopeo->>OCIRegistry: request raw manifest
OCIRegistry-->>skopeo: return manifest or resolution error
skopeo-->>check_oci_pins: return result after retries
check_oci_pins->>check_oci_pins: report DEAD, NOPLAT, or ok
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Pulling distroless_static for linux/arm64/v8 in addition to linux/amd64 changes module resolution, so the committed lockfile has to move with it. Leaving it stale makes Bazel either rewrite it mid-build or fail outright, depending on lockfile mode. Co-authored-by: Balaji Ganesan <bganesan@nvidia.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tools/ci/check-oci-pins`:
- Around line 24-28: Move the SKOPEO command-availability guard below the
--self-test option branch in the script. Ensure --self-test reaches its
strip_tag-only execution without requiring skopeo, while normal pin verification
still validates SKOPEO before proceeding.
- Around line 81-92: Extend the OCI pin check’s Python extraction in
check-oci-pins to parse each oci.pull platforms list alongside image and digest,
then inspect the resolved OCI index manifests and verify every requested
OS/architecture/variant descriptor is present, including linux/arm64/v8
matching. Add a focused self-test demonstrating that a missing arm64/v8 entry
fails while existing digest validation remains intact.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: faa25c78-e1c6-4974-a693-3d0693617170
📒 Files selected for processing (2)
MODULE.bazeltools/ci/check-oci-pins
Verifying that a digest resolves would not have caught the second defect this pull request fixes. nvsnap's base resolved perfectly; it simply lacked the arm64 entry the shared macro asks for, and that fails later at analysis with "could not find an image matching the target platform". The check now parses each pin's platforms list and asserts the index advertises every one, matching os/arch/variant exactly because rules_oci compares the platform string exactly. Rewritten in Python: this is manifest JSON work, and the matching logic needs to be unit-testable without a registry. The self-test covers the case that matters, an index missing arm64/v8, plus arm64 not matching arm64/v8, single-image manifests satisfying nothing, and the reference handling that produced earlier false positives. It runs before the skopeo dependency check so it works on a machine without skopeo. Comment stripping is in there for a reason worth recording: function-autoscaler documents why it uses arm64/v8 by quoting the wrong value, "linux/arm64", in a comment inside the platforms list. Scanning for quoted strings without removing comments read that as a declaration and reported a correct pin as broken. The first run of this check produced exactly that false positive, so the fix ships with a regression test in its shape. Co-authored-by: Balaji Ganesan <bganesan@nvidia.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
balajinvda
enabled auto-merge
July 30, 2026 19:11
kristinapathak
approved these changes
Jul 30, 2026
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tools/ci/check-oci-pins`:
- Around line 149-174: Update subprocess.run in fetch_manifest to enforce a
per-call timeout for each skopeo inspect attempt, using the existing retry flow
to handle timeout failures and continue or return according to the established
behavior. Ensure a stalled registry cannot block the retry loop indefinitely.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 6936cb1d-868e-4e3f-9da0-74d1e8057227
⛔ Files ignored due to path filters (1)
MODULE.bazel.lockis excluded by!**/*.lock,!**/MODULE.bazel.lock
📒 Files selected for processing (1)
tools/ci/check-oci-pins
subprocess.run without a timeout blocks indefinitely, so a stalled registry or proxy would hang the retry loop and take the whole scheduled sweep with it, losing every result rather than one pin's. Each inspect is now capped at 60s. A timeout is treated like any other transient failure: retried, and if it persists reported as unresolved rather than as a dead pin, since a hung connection is not evidence that an image is gone. Co-authored-by: Balaji Ganesan <bganesan@nvidia.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The Black Duck scan of this repository has been failing, and the cause is two
defects here rather than anything in the scanner. Both were found by reproducing
its command locally:
1. A dead base image digest. The root module pinned a
gcr.io/distroless/staticdigest that upstream had garbage-collected:The tag still resolves; the manifest behind that digest does not. Digests are
immutable, but registries garbage-collect untagged manifests, so a pin rots
without anyone editing it.
2. A base/macro platform mismatch, hidden behind the first. With the digest
repinned, analysis failed one layer deeper:
go_oci_imagealways emits a_multi_archtarget overDEFAULT_PLATFORMS(arm64 and x86_64), but
@distroless_staticwas pulledlinux/amd64only, sothe arm64 branch of the transition had no image. nvsnap was the only service
whose base disagreed with the shared macro.
Why CI never caught either
bazel build //...skips these targets because they are taggedmanual.cquery 'deps(//...)'ignoresmanualand analyses them, so an external scanhits what our own wildcard build never touches.
The dead pin was additionally masked because CI restores the Bazel repository
cache, so the fetch was served from cache and the pin looked healthy. That was
live risk, not theory: the Actions cache is over quota and evicting
continuously, so this was one eviction away from breaking main with an error
that reads like a network blip.
What changed
distroless_staticis repinned to a current digest and pulled for botharchitectures, matching the macro and every other service. Its
use_repogainsthe
_linux_arm64_v8entry.tools/ci/check-oci-pinsverifies that everyoci.pulldigest still resolves,across all 23 pins in the repository. It is meant to run on a schedule rather
than per change, since a pin rots with no commit touching it.
Testing
The scanner's exact command now succeeds on Linux against
//...:Previously it produced no output at all.
nvsnap_agent_image_multi_archalsoanalyses cleanly.
check-oci-pinsreports all 23 pins resolving. It ships with a self-test(
--self-test) covering its reference handling, because while investigatingthis I twice reported healthy pins as dead: appending
@digestto a referencethat already carries a tag yields an invalid reference that skopeo rejects. The
check also retries before declaring a pin dead, so a registry timeout is not
mistaken for a missing image, and it fails if it parses zero pins rather than
reporting success over an empty set.
Notes
Worth knowing for the SBOM effort: a scan of
//...from the repository rootcovers only the root Bazel module. The ~20 nested modules each have their own
MODULE.bazeland are not in that target universe, so this unblocks the scanbut does not by itself make it complete. Our CI already enumerates those
modules and their workdirs.
References
None
Dependencies
None.
Summary by CodeRabbit
Bug Fixes
Chores