Skip to content

Add timeout to Grype DB update with graceful fallback to prevent indefinite readiness probe failures #320

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
matthyx merged 5 commits into from
Jan 16, 2026
Merged

Add timeout to Grype DB update with graceful fallback to prevent indefinite readiness probe failures #320

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9be0f58
Initial plan
Copilot Jan 16, 2026
d0a087c
Add context timeout to Grype DB update to prevent stuck updates
Copilot Jan 16, 2026
ab6534c
Address code review comments: add comments and use consistent exit code
Copilot Jan 16, 2026
d291b49
Increase timeout to 15min and prevent crashloop by keeping existing D…
Copilot Jan 16, 2026
adec623
Add dbVersionLocked helper and use it
matthyx Jan 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 70 additions & 16 deletions adapters/v1/grype.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,20 +108,26 @@ func NewGrypeAdapterFixedDB() (*GrypeAdapter, func(), error) {
return g, terminate, nil
}

func (g *GrypeAdapter) dbVersionLocked() string {
if g.dbStatus == nil {
return ""
}
parts := strings.Split(g.dbStatus.From, "%3A")
return parts[len(parts)-1]
}

// DBVersion returns the vulnerabilities DB checksum which is used to tag CVE manifests
func (g *GrypeAdapter) DBVersion(context.Context) string {
g.mu.RLock()
defer g.mu.RUnlock()

parts := strings.Split(g.dbStatus.From, "%3A")
return parts[len(parts)-1]
return g.dbVersionLocked()
}

// Ready returns the status of the vulnerabilities DB
func (g *GrypeAdapter) Ready(ctx context.Context) bool {
// DB update is in progress
if !g.mu.TryRLock() {
// FIXME this gets stuck forever if the db update times out
return false
}
g.mu.RUnlock() // because TryRLock doesn't unlock
Expand All @@ -134,19 +140,67 @@ func (g *GrypeAdapter) Ready(ctx context.Context) bool {
defer span.End()
logger.L().Info("updating grype DB",
helpers.String("listingURL", g.distCfg.LatestURL))
var err error
g.store, g.dbStatus, err = grype.LoadVulnerabilityDB(g.distCfg, g.installCfg, true)
if err != nil {
logger.L().Ctx(ctx).Error("failed to update grype DB", helpers.Error(err))
err := tools.DeleteContents(g.installCfg.DBRootDir)
logger.L().Debug("cleaned up cache", helpers.Error(err),
helpers.String("DBRootDir", g.installCfg.DBRootDir))
logger.L().Info("restarting to release previous grype DB")
os.Exit(0)

// Create a context with timeout to prevent stuck updates
// 15 minutes allows for slow network connections while still catching truly stuck downloads
updateCtx, cancel := context.WithTimeout(ctx, 15*time.Minute)
defer cancel()

// Track if we have an existing DB to fall back on
hasExistingDB := g.dbStatus != nil

// Run the DB update in a goroutine to respect the timeout
type updateResult struct {
store vulnerability.Provider
dbStatus *vulnerability.ProviderStatus
err error
}
// Buffered channel (size 1) prevents goroutine from blocking if timeout occurs
// The goroutine will complete in background but won't leak since it will successfully send
resultCh := make(chan updateResult, 1)

go func() {
// Note: grype.LoadVulnerabilityDB does not accept context, so the goroutine
// will continue to completion even if timeout occurs. The buffered channel
// ensures the goroutine can complete without blocking.
store, dbStatus, err := grype.LoadVulnerabilityDB(g.distCfg, g.installCfg, true)
resultCh <- updateResult{store: store, dbStatus: dbStatus, err: err}
}()

select {
case result := <-resultCh:
if result.err != nil {
logger.L().Ctx(ctx).Error("failed to update grype DB", helpers.Error(result.err))
err := tools.DeleteContents(g.installCfg.DBRootDir)
logger.L().Debug("cleaned up cache", helpers.Error(err),
helpers.String("DBRootDir", g.installCfg.DBRootDir))
logger.L().Info("restarting to release previous grype DB")
os.Exit(0)
}
g.store = result.store
g.dbStatus = result.dbStatus
g.lastDbUpdate = now
logger.L().Info("grype DB updated")
return true
case <-updateCtx.Done():
if hasExistingDB {
// We have an existing DB, keep using it instead of crashing
// This prevents crashloop in case of slow but functional network
logger.L().Ctx(ctx).Warning("grype DB update timed out after 15 minutes, continuing with existing DB",
helpers.String("existingDBVersion", g.dbVersionLocked()))
// Update lastDbUpdate to prevent immediate retry, will retry in next 24h cycle
g.lastDbUpdate = now
return true
Comment on lines +185 to +193
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Jan 16, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Deadlock: g.DBVersion(ctx) acquires read lock while write lock is held.

At line 183, g.DBVersion(ctx) is called while g.mu.Lock() is already held (acquired at line 130). The DBVersion method (lines 112-118) attempts to acquire g.mu.RLock(), which will deadlock since Go's sync.RWMutex does not support re-entrant locking.

🔧 Proposed fix: access dbStatus directly without calling DBVersion 
 		case <-updateCtx.Done():
 			if hasExistingDB {
 				// We have an existing DB, keep using it instead of crashing
 				// This prevents crashloop in case of slow but functional network
+				parts := strings.Split(g.dbStatus.From, "%3A")
+				existingVersion := parts[len(parts)-1]
 				logger.L().Ctx(ctx).Warning("grype DB update timed out after 15 minutes, continuing with existing DB",
-					helpers.String("existingDBVersion", g.DBVersion(ctx)))
+					helpers.String("existingDBVersion", existingVersion))
 				// Update lastDbUpdate to prevent immediate retry, will retry in next 24h cycle
 				g.lastDbUpdate = now
 				return true

Alternatively, create an internal helper that doesn't acquire the lock:

func (g *GrypeAdapter) dbVersionLocked() string {
	parts := strings.Split(g.dbStatus.From, "%3A")
	return parts[len(parts)-1]
}
🤖 Prompt for AI Agents 
In `@adapters/v1/grype.go` around lines 178 - 186, The call to g.DBVersion(ctx)
while g.mu.Lock() is held causes deadlock because DBVersion acquires
g.mu.RLock(); replace that call with a lock-free access to the status (or add an
internal helper that assumes the caller already holds the lock). Specifically,
inside the updateCtx.Done() handling where g.mu.Lock() is held, read
g.dbStatus.From directly (or create a new dbVersionLocked() helper that parses
g.dbStatus.From without acquiring any mutex) and use that value in the logger
and when setting g.lastDbUpdate to avoid calling DBVersion while holding the
write lock.

Copy link
Copy Markdown
Contributor

@matthyx matthyx Jan 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot can you check?

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Jan 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like the humans are having a chat. I'll hop back into my burrow for now. If you need me again, just tag @coderabbitai in a new comment, and I'll come hopping out!

} else {
// No existing DB to fall back on, must restart
logger.L().Ctx(ctx).Error("grype DB initial download timed out after 15 minutes")
err := tools.DeleteContents(g.installCfg.DBRootDir)
logger.L().Debug("cleaned up cache after timeout", helpers.Error(err),
helpers.String("DBRootDir", g.installCfg.DBRootDir))
logger.L().Info("restarting pod due to grype DB initial download timeout")
os.Exit(0)
}
}
g.lastDbUpdate = now
logger.L().Info("grype DB updated")
return true
}

return g.dbStatus.Error == nil
Expand Down Expand Up @@ -228,7 +282,7 @@ func (g *GrypeAdapter) ScanSBOM(ctx context.Context, sbom domain.SBOM) (domain.C
Name: sbom.Name,
SBOMCreatorVersion: sbom.SBOMCreatorVersion,
CVEScannerVersion: g.Version(),
CVEDBVersion: g.DBVersion(ctx),
CVEDBVersion: g.dbVersionLocked(),
Annotations: sbom.Annotations,
Labels: sbom.Labels,
Content: vulnerabilityResults,
Expand Down
Loading