Add timeout to Grype DB update with graceful fallback to prevent indefinite readiness probe failures

[Copilot]

Grype DB updates can hang silently during the 24-hour refresh cycle, causing pods to fail readiness probes indefinitely with no recovery mechanism.

Changes

• Add 15-minute timeout to DB update operation using context.WithTimeout
• Execute update in goroutine with buffered channel to enable timeout handling
• Intelligent fallback behavior to prevent crashloop on slow networks:
  • Scheduled updates: On timeout, keeps existing DB and continues operating (retries in 24h)
  • Initial download: On timeout, restarts pod (no fallback available)
• Graceful degradation instead of always crashing on timeout

// Create a context with timeout to prevent stuck updates
// 15 minutes allows for slow network connections while still catching truly stuck downloads
updateCtx, cancel := context.WithTimeout(ctx, 15*time.Minute)
defer cancel()

// Track if we have an existing DB to fall back on
hasExistingDB := g.dbStatus != nil

// Buffered channel (size 1) prevents goroutine from blocking if timeout occurs
resultCh := make(chan updateResult, 1)

go func() {
    store, dbStatus, err := grype.LoadVulnerabilityDB(g.distCfg, g.installCfg, true)
    resultCh <- updateResult{store: store, dbStatus: dbStatus, err: err}
}()

select {
case result := <-resultCh:
    // Handle success/failure
case <-updateCtx.Done():
    if hasExistingDB {
        // Keep using old DB, log warning, stay ready
        logger.L().Ctx(ctx).Warning("grype DB update timed out after 15 minutes, continuing with existing DB")
        return true
    } else {
        // No DB to fall back on, must restart
        logger.L().Ctx(ctx).Error("grype DB initial download timed out after 15 minutes")
        tools.DeleteContents(g.installCfg.DBRootDir)
        os.Exit(0)
    }
}

Note: grype.LoadVulnerabilityDB does not accept context for cancellation, so the goroutine completes in background. The buffered channel ensures it won't block.

Behavior

• Stuck download during scheduled update: Logs warning, continues with existing DB, retries in next 24h cycle
• Stuck download during initial startup: Cleans cache and restarts pod
• Slow but functional network: 15-minute timeout accommodates slow downloads, preventing crashloop
• Update success: Updates DB normally
• Update error: Cleans cache and restarts pod (existing behavior)

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Readiness probe fail stuck due to Grype DB unsucessful update</issue_title>
<issue_description># Description
Readiness probe fails and pods get stuck due to Grype DB unsuccessful updates. The issue occurs intermittently during the automatic 24-hour Grype DB refresh cycle. When this happens, no logs are generated indicating “updating grype db” or any retry attempts. The pod remains stuck, causing readiness probe failures.

Environment

OS: <your OS here>
Version: v0.3.82

Steps To Reproduce

1. Deploy Kubescape with Kubevuln integration enabled also custom endpoint we have for listing its a mirror of grype listings endpoint.
2. Let it run for a few days until the scheduled Grype DB update is triggered (every 24 hours).
3. Observe the logs during the DB update window.
4. At random cycles, the update process hangs silently with no retry attempts and Pod readiness probe fails, leaving the pod stuck.

Expected behavior

Grype DB should update successfully every 24 hours. If the update fails or stalls, Kubevuln should log an error and retry downloading the database instead of remaining stuck.

Actual Behavior

• Random failures during DB updates.
• No logs generated showing the update attempt.
• No retry mechanism triggered.
• Pods status is Ready but stuck due to readiness probe failures.
• As soon as deployment is restarted manually it works fine again.

Additional context

The issue was first noticed a few days after deployment when Grype DB updates failed silently. It seems the update process isn’t handling download errors and gets stuck with a lock, preventing the readiness probe from recovering. I tested by intentionally taking down our DB listing endpoint before deployment start, it clearly fails to download, falls back to the previous DB, and restarts gracefully until the endpoint is back. But when the deployment is already running and I take down the endpoint, it fails silently without retries. However, since our endpoints are normally available 24/7, I’m not sure where the issue lies but it should either retry and then restart itself when if readiness fails for a while.
</issue_description>

<agent_instructions>add a context timeout to restart the update or the Pod when it's stuck updating the grype DB</agent_instructions>

Comments on the Issue (you are @copilot in this section)

@matthyx actually the readiness probe should fail in that case... but maybe we could add a context timeout to restart the update or the Pod when it's stuck like that do you want to work on that @Ripeey ?

• Fixes Readiness probe fail stuck due to Grype DB unsucessful update #303

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Summary by CodeRabbit

• Bug Fixes
  • Vulnerability database updates now run asynchronously with a 15-minute timeout
  • Database gracefully falls back to the previous version if an update times out, ensuring continuous operation
  • Enhanced error handling with automatic restart when needed

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

📝 Walkthrough

Walkthrough

The Ready path now executes grype DB updates asynchronously with a 15-minute timeout enforced via context. A goroutine handles the LoadVulnerabilityDB call using a buffered channel to capture results. Success updates store and status; timeout with existing DB logs warning and reuses the current DB; timeout without existing DB logs error and restarts the process; errors trigger logging, cache cleanup, and process restart.

Changes

Cohort / File(s)	Summary
Asynchronous DB Update with Timeout adapters/v1/grype.go	Refactored synchronous LoadVulnerabilityDB call to run asynchronously in a goroutine with a buffered channel. Enforces 15-minute timeout via context. Implements fallback logic: timeout with existing DB logs warning and reuses DB; timeout without existing DB or on error logs and restarts process. Introduces hasExistingDB flag to distinguish recovery behaviors.

Sequence Diagram

sequenceDiagram
    actor Caller as Caller/Scheduler
    participant Ready as Ready()
    participant Context as Context (15m timeout)
    participant Goroutine as Goroutine
    participant LoadDB as LoadVulnerabilityDB()
    participant Store as Store
    participant Logger as Logger
    participant Process as Process

    Caller->>Ready: Check readiness & update DB
    Ready->>Context: Create context with 15min timeout
    Ready->>Goroutine: Launch async LoadVulnerabilityDB
    
    par Timeout Monitor
        Context-->>Ready: Await result or timeout
    and Async Work
        Goroutine->>LoadDB: Load vulnerability DB
        alt Success
            LoadDB-->>Goroutine: Return DB
            Goroutine->>Store: Update store & status
            Goroutine->>Logger: Log success
            Goroutine-->>Ready: Send result on channel
            Ready->>Ready: Return true
        else Error/Failure
            LoadDB-->>Goroutine: Return error
            Goroutine->>Logger: Log error
            Goroutine->>Store: Cleanup cache
            Goroutine->>Process: os.Exit(0)
        end
    end
    
    alt Timeout Occurred
        Context->>Ready: Timeout fired
        alt Existing DB Present
            Ready->>Logger: Log warning (reusing)
            Ready->>Store: Refresh lastDbUpdate
            Ready->>Ready: Return true
        else No Existing DB
            Ready->>Logger: Log error
            Ready->>Store: Cleanup cache
            Ready->>Process: os.Exit(0)
        end
    end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Poem

🐰 A rabbit's tale of async flows,
Where Grype DB through timeouts goes,
Fifteen minutes, channels wide,
Fallback dances safe inside,
Now readiness hops free! ✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: adding a timeout to Grype DB updates with graceful fallback behavior to prevent readiness probe failures.
Linked Issues check	✅ Passed	The implementation addresses all key objectives from issue #303: adds timeout to prevent stuck updates, implements graceful fallback for existing DBs, restarts on initial failures, and includes retry mechanism via 24h cycle.
Out of Scope Changes check	✅ Passed	All changes are scoped to the Grype DB update mechanism and directly address the readiness probe failure issue; no unrelated modifications detected.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[matthyx]

@copilot what happens if downloading the grypeDB takes longer than 5 minutes because of poor internet access? Will we end up in a crashloop?

[Copilot]

@copilot what happens if downloading the grypeDB takes longer than 5 minutes because of poor internet access? Will we end up in a crashloop?

Good catch! I've addressed this by:

1. Increased timeout to 15 minutes - accommodates slow networks while still catching stuck downloads
2. Smart fallback behavior - if timeout occurs during scheduled update (when an existing DB is available), the pod continues using the old DB instead of crashing
3. Only crashes on initial download timeout - when there's no DB to fall back to

This prevents crashloop on slow networks while still handling stuck downloads. The pod will retry the update in the next 24-hour cycle.

Changes in commit d291b49

[matthyx]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@adapters/v1/grype.go`:
- Around line 178-186: The call to g.DBVersion(ctx) while g.mu.Lock() is held
causes deadlock because DBVersion acquires g.mu.RLock(); replace that call with
a lock-free access to the status (or add an internal helper that assumes the
caller already holds the lock). Specifically, inside the updateCtx.Done()
handling where g.mu.Lock() is held, read g.dbStatus.From directly (or create a
new dbVersionLocked() helper that parses g.dbStatus.From without acquiring any
mutex) and use that value in the logger and when setting g.lastDbUpdate to avoid
calling DBVersion while holding the write lock.

🧹 Nitpick comments (1)

adapters/v1/grype.go (1)

187-195: Minor style consideration: unreachable code after os.Exit.

While os.Exit(0) terminates the process and the code is technically correct, some static analysis tools may flag the lack of an explicit return. Consider adding a return false after os.Exit(0) for clarity, though this is optional since the code is unreachable.

Optional: add unreachable return for static analysis

 			logger.L().Info("restarting pod due to grype DB initial download timeout")
 			os.Exit(0)
+			return false // unreachable, but satisfies static analysis
 		}

📜 Review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 666ace5 and d291b49.

📒 Files selected for processing (1)

• adapters/v1/grype.go

🧰 Additional context used

🧬 Code graph analysis (1)

adapters/v1/grype.go (1)

internal/tools/tools.go (1)

• DeleteContents (94-106)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)

• GitHub Check: pr-created / test / Create cross-platform build
• GitHub Check: build-and-push-image

🔇 Additional comments (4)

adapters/v1/grype.go (4)

137-140: LGTM on timeout context setup.

The 15-minute timeout is reasonable for slow networks while still catching truly stuck downloads. The deferred cancel() ensures proper cleanup.

---

142-143: LGTM on existing DB tracking.

Correctly captures whether a fallback DB exists before starting the async update.

---

145-153: LGTM on channel design.

The buffered channel (size 1) correctly prevents the goroutine from blocking if timeout occurs, and the comments clearly explain this design decision.

---

155-161: Acknowledge the goroutine completion behavior.

Since grype.LoadVulnerabilityDB doesn't accept a context, the goroutine continues running after timeout. This is an acceptable trade-off given the constraint. In the timeout-with-existing-DB scenario, the orphaned goroutine will eventually complete and the result will be discarded—which is fine since we've already decided to continue with the existing DB.

One consideration: if the background download succeeds after timeout, the downloaded DB files remain on disk but aren't loaded into memory until the next 24h cycle. This is benign but worth noting.

_{✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.}

[coderabbitai]

⚠️ Potential issue | 🔴 Critical

Deadlock: g.DBVersion(ctx) acquires read lock while write lock is held.

At line 183, g.DBVersion(ctx) is called while g.mu.Lock() is already held (acquired at line 130). The DBVersion method (lines 112-118) attempts to acquire g.mu.RLock(), which will deadlock since Go's sync.RWMutex does not support re-entrant locking.

🔧 Proposed fix: access dbStatus directly without calling DBVersion

 		case <-updateCtx.Done():
 			if hasExistingDB {
 				// We have an existing DB, keep using it instead of crashing
 				// This prevents crashloop in case of slow but functional network
+				parts := strings.Split(g.dbStatus.From, "%3A")
+				existingVersion := parts[len(parts)-1]
 				logger.L().Ctx(ctx).Warning("grype DB update timed out after 15 minutes, continuing with existing DB",
-					helpers.String("existingDBVersion", g.DBVersion(ctx)))
+					helpers.String("existingDBVersion", existingVersion))
 				// Update lastDbUpdate to prevent immediate retry, will retry in next 24h cycle
 				g.lastDbUpdate = now
 				return true

Alternatively, create an internal helper that doesn't acquire the lock:

func (g *GrypeAdapter) dbVersionLocked() string {
	parts := strings.Split(g.dbStatus.From, "%3A")
	return parts[len(parts)-1]
}

🤖 Prompt for AI Agents

In `@adapters/v1/grype.go` around lines 178 - 186, The call to g.DBVersion(ctx)
while g.mu.Lock() is held causes deadlock because DBVersion acquires
g.mu.RLock(); replace that call with a lock-free access to the status (or add an
internal helper that assumes the caller already holds the lock). Specifically,
inside the updateCtx.Done() handling where g.mu.Lock() is held, read
g.dbStatus.From directly (or create a new dbVersionLocked() helper that parses
g.dbStatus.From without acquiring any mutex) and use that value in the logger
and when setting g.lastDbUpdate to avoid calling DBVersion while holding the
write lock.

[matthyx]

@copilot can you check?

[coderabbitai]

Seems like the humans are having a chat. I'll hop back into my burrow for now. If you need me again, just tag @coderabbitai in a new comment, and I'll come hopping out!