Skip to content

ci(security): fail the build on a fixable critical in an image (#2018) - #2238

Merged
ktursunov merged 4 commits into
mainfrom
ci/image-critical-gate
Aug 5, 2026
Merged

ci(security): fail the build on a fixable critical in an image (#2018)#2238
ktursunov merged 4 commits into
mainfrom
ci/image-critical-gate

Conversation

@Gregory91G

@Gregory91G Gregory91G commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Closes the second half of #2018 — "a published image with fixable critical CVEs fails a check". The findings half is already done: #2012 and #2013 are closed and #2014 landed this morning as #2168.

Two gates, both --severity CRITICAL --ignore-unfixed, matching the issue's acceptance (criticals only, fixable only).

Publish gate — build-images.yml

The per-arch build pushes by digest with no tag (push-by-digest=true); tags appear later, in merge-<service>, via docker/metadata-action. That gap is where the gate goes: a new composite action scans the digests immediately before Create multi-arch manifest and push.

A rejected build therefore gets no build tag and does not move :latest, so bump-descriptors and publish-chart have nothing to reference and the gitops poller never sees a new version. The blob stays in the registry, unreferenced.

Covers the seven merge jobs behind the fourteen images the nightly scans (insight-jira-enrich rides the connector matrix in merge-image). insight-ui-tests is out, matching its absence from that matrix.

Drift gate — trivy-images.yml

A verdict step, placed last and reading the JSON the scan already produced.

Putting --exit-code 1 on the scan itself does not work: trivy convert runs inside the same step under set -euo pipefail, so a failure aborts before the SARIF exists, the upload skips itself on hashFiles(...) != '', and the summary exits on its own [ -f trivy-image.json ] guard. You would lose the alerts and the summary for exactly the images that failed.

Why Trivy's severity and not the alert level

GitHub derives its level from the CVSS score in security-severity, not from Trivy's label. CVE-2026-59873 carried CRITICAL with a CVSS of 7.5 from all three sources Trivy holds, so it appeared as high in the Security tab. A gate reading Code Scanning severities would have missed it.

--severity is undocumented on trivy convert (only --exit-code is listed) but is honoured — one more reason the Trivy image stays pinned by digest.

Cost

Measured locally. On a cold cache the largest image (source-gitlab-insight, 209 MB compressed) takes 30 s with Trivy's default scanners and 26 s with --scanners vuln, which is what the gate uses — the default set adds a secret pass the gate never reads.

The bigger term is the second digest in the same job. The run mounts the same /tmp/trivy-cache the nightly workflow uses, so it is a cache hit instead of another database download and image analysis: measured on the toolbox, 79 s → 1.6 s.

That puts a merge job at roughly 30–45 s added — ~25–40 s for the first digest, a couple of seconds for the second. The seven jobs run in parallel and the last full build-images.yml took 29 min, so the wall-clock cost is well under a minute. A merge job itself currently takes ~34 s, so in relative terms it roughly doubles the cheapest jobs in the workflow. My link to GHCR is slower than a runner's, so treat these as upper bounds.

Test plan

  • actionlint: 7 distinct findings before and after, identical set — no new lint.
  • All three files parse as YAML.
  • Gate shell exercised on each path against the real registry: clean image at CRITICAL → exit 0; the same image at HIGH (10 present) → exit 1; empty digests directory → exit 1, so a missing artefact fails loudly instead of passing silently.
  • trivy convert verdict verified on real scan output: --severity CRITICAL --exit-code 1 gives 0 on the post-fix(images): bump the toolbox to Node 24 and upgrade npm in place (#2166) #2168 toolbox JSON and 1 on the pre-fix(images): bump the toolbox to Node 24 and upgrade npm in place (#2166) #2168 one; without --severity the same clean JSON gives 1, which proves the flag is applied.
  • Baseline before enabling: insight-toolbox 0 critical / 10 high after fix(images): bump the toolbox to Node 24 and upgrade npm in place (#2166) #2168 (was 1 / 24), insight-gateway 0 / 0, insight-front 0 / 0, source-salesforce-insight 0 / 1.
  • workflow_dispatch on trivy-images.yml after merge, to confirm the baseline across all fourteen images at once. fail-fast: false, so any red job names its image.
  • Re-verified after the cache/scanner tweak: clean image at CRITICAL → 0, same image at HIGH → 1, empty digests directory → 1.
  • First main build after merge passes the publish gate.

Not addressed here

Recorded so they are not mistaken for done: the nightly still scans :latest while deployments pin the build tag; --ignore-unfixed means neither gate can ever fire on a critical without a released fix; the fourteen-image matrix is still hand-maintained; and Trivy's SARIF still collapses findings that share a path across the seven connector images.

Summary by CodeRabbit

  • New Features

    • Added automated vulnerability gates for container images during image publishing.
    • Scans operating system and library packages for fixable vulnerabilities at configured severity levels.
    • Added validation to ensure image digests and registry credentials are available before scanning.
  • Bug Fixes

    • Image scan workflows now fail when fixable critical vulnerabilities are detected, while retaining existing summaries and security reports.

Two gates, both on CRITICAL with --ignore-unfixed, matching what the issue
asks for.

Publish side: a new composite action scans the push-by-digest blobs in each
merge job, before docker/metadata-action's tags are applied. A rejected build
therefore keeps no build tag and does not move :latest, so no descriptor bump
and no chart can reference it. It covers the seven jobs that produce the
fourteen images the nightly scans; insight-ui-tests stays out, as it is out of
that matrix.

Drift side: trivy-images.yml gets a verdict step. It runs last and reads the
JSON the scan already wrote, so the job summary and the SARIF upload still
happen for a failing image -- putting --exit-code 1 on the scan itself would
abort before `trivy convert` and lose both.

Gating on Trivy's own severity rather than on the Code Scanning level is
deliberate: GitHub derives its level from the CVSS score, so CVE-2026-59873
(#2014) was CRITICAL for Trivy and high in the Security tab. Note that
`--severity` is undocumented on `trivy convert` yet honoured, which is one
more reason the image stays pinned by digest.

Baseline measured before enabling: insight-toolbox 0 critical after #2168,
insight-gateway, insight-front and source-salesforce-insight also 0.

Signed-off-by: Gregory Gogin <grigoriy.gogin@constructor.tech>
@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@Gregory91G, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 19 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 3d59154a-2946-44d9-849e-e40fdc9bcd3f

📥 Commits

Reviewing files that changed from the base of the PR and between 52fbc12 and b76a4f4.

📒 Files selected for processing (2)
  • .github/actions/image-cve-gate/action.yml
  • .github/workflows/build-images.yml
📝 Walkthrough

Walkthrough

Changes

Image CVE enforcement

Layer / File(s) Summary
Digest scan action
.github/actions/image-cve-gate/action.yml
Adds a reusable composite action that validates image digests and scans them with authenticated, pinned Trivy.
Image merge job gates
.github/workflows/build-images.yml
Runs the CVE gate before manifest creation for backend, frontend, service, and connector images.
Existing scan report enforcement
.github/workflows/trivy-images.yml
Reports CRITICAL findings from the existing Trivy JSON and fails for fixable vulnerabilities.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related issues

Possibly related PRs

Suggested reviewers: ktursunov, cyberantonz

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: CI fails when an image contains a fixable critical vulnerability.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ci/image-critical-gate

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
.github/actions/image-cve-gate/action.yml (1)

54-57: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Reuse the Trivy cache across digest scans.

Each docker run --rm starts without a host mount for /root/.cache, so Trivy may redo vulnerability cache setup for every digest image. Mount /tmp/trivy-cache:/root/.cache before the digest loop, and create it with mkdir -p /tmp/trivy-cache.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/actions/image-cve-gate/action.yml around lines 54 - 57, Update the
Trivy digest-scan docker run command to reuse a host cache by creating
/tmp/trivy-cache before the digest loop and mounting it at /root/.cache
alongside the existing workspace volume. Apply this to the docker invocation
using TRIVY_IMAGE without changing the scan behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/build-images.yml:
- Around line 385-390: Add a pinned actions/checkout step immediately before
each local image-cve-gate invocation in .github/workflows/build-images.yml at
lines 385-390, 496-501, 607-612, 732-737, 866-871, 1042-1047, and 1309-1314,
ensuring the repository is checked out before the local composite action is
loaded.

---

Nitpick comments:
In @.github/actions/image-cve-gate/action.yml:
- Around line 54-57: Update the Trivy digest-scan docker run command to reuse a
host cache by creating /tmp/trivy-cache before the digest loop and mounting it
at /root/.cache alongside the existing workspace volume. Apply this to the
docker invocation using TRIVY_IMAGE without changing the scan behavior.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: cd63d666-7efd-4060-acc0-06606491232d

📥 Commits

Reviewing files that changed from the base of the PR and between d892b17 and 52fbc12.

📒 Files selected for processing (3)
  • .github/actions/image-cve-gate/action.yml
  • .github/workflows/build-images.yml
  • .github/workflows/trivy-images.yml

Comment thread .github/workflows/build-images.yml
Gregory91G and others added 3 commits August 5, 2026 13:47
Two measured tweaks to the image gate. The docker run now mounts the same
/tmp/trivy-cache the nightly workflow uses, so the second digest in a job is a
cache hit rather than another database download and image analysis: 79s -> 1.6s
on the toolbox. And --scanners vuln drops the default secret pass, which the
gate never reads, for about 13% off a cold run.

Signed-off-by: Gregory Gogin <grigoriy.gogin@constructor.tech>
A `uses: ./…` reference is resolved against the workspace when the runner loads
the action, before any of its steps run, so the checkout inside the composite
was too late -- the merge jobs have no working tree and every one of them would
have failed to find action.yml. The checkout moves to the caller, where it also
covers the .trivyignore the scan reads.

Signed-off-by: Gregory Gogin <grigoriy.gogin@constructor.tech>
@Gregory91G
Gregory91G requested a review from cyberantonz August 5, 2026 13:05
@ktursunov
ktursunov added this pull request to the merge queue Aug 5, 2026
Merged via the queue into main with commit 88484bb Aug 5, 2026
43 checks passed
@ktursunov
ktursunov deleted the ci/image-critical-gate branch August 5, 2026 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants