Skip to content

fix(monitor): skip dirs without underscore in Update instead of panicking - #2371

Merged
hami-robot[bot] merged 1 commit into
Project-HAMi:masterfrom
Nakshatra480:fix/update-panic-on-underscore-free-dir
Aug 6, 2026
Merged

fix(monitor): skip dirs without underscore in Update instead of panicking#2371
hami-robot[bot] merged 1 commit into
Project-HAMi:masterfrom
Nakshatra480:fix/update-panic-on-underscore-free-dir

Conversation

@Nakshatra480

@Nakshatra480 Nakshatra480 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Update() in pkg/monitor/nvidia/cudevshr.go called strings.Split(entry.Name(), "_")[1] without checking that the split produced at least two parts. Any directory under the hook path whose name contains no underscore, a stray directory from another tool, a partial write, or operator error causes an index out of range panic that crashes vGPUmonitor and silences all GPU metrics on that node.

  • pkg/monitor/nvidia/cudevshr.go: replaced both strings.Split(...)[idx] calls with a single strings.SplitN(..., "_", 2); directories that don't match the expected <podUID>_<containerName> format are skipped with a warning log instead of panicking
  • pkg/monitor/nvidia/cudevshr_test.go: added "dir without underscore in name is skipped" regression case to Test_ContainerLister_Update
  • All existing Test_ContainerLister_Update subtests continue to pass

Which issue(s) this PR fixes:
Part of #2126 (LFX observability hardening - vGPU monitor stability)

Special notes for your reviewer:
The strings.Split(...)[0] on line 189 was safe (always produces at least one element), but the [1] on line 216 was not. The fix consolidates both into one SplitN call so the validation happens once before either index is used.

Does this PR introduce a user-facing change?
Yes vGPUmonitor no longer panics when a directory without an underscore appears in the hook path; GPU metrics continue to be reported normally.

AI Disclosure:
AI assistance was used for code inspection and draft formatting; all logic, test cases, and verification were manually checked and validated.

Summary by CodeRabbit

  • Bug Fixes
    • Improved NVIDIA container monitoring by ignoring malformed directory entries.
    • Prevented entries missing pod or container identifiers from creating container mappings.
    • Added validation for more consistent monitoring data parsing.

@hami-robot
hami-robot Bot requested review from DSFans2014 and lengrongfu August 5, 2026 04:22
@github-actions github-actions Bot added the kind/bug Something isn't working label Aug 5, 2026
@hami-robot hami-robot Bot added the size/M label Aug 5, 2026
@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 5db711f4-bea7-450b-ada6-db5d6980f349

📥 Commits

Reviewing files that changed from the base of the PR and between e6d0902 and 206222f.

📒 Files selected for processing (2)
  • pkg/monitor/nvidia/cudevshr.go
  • pkg/monitor/nvidia/cudevshr_test.go
🚧 Files skipped from review as they are similar to previous changes (2)
  • pkg/monitor/nvidia/cudevshr_test.go
  • pkg/monitor/nvidia/cudevshr.go

📝 Walkthrough

Walkthrough

ContainerLister.Update now validates container directory names before creating mappings. It skips names without a pod UID or container name and reuses the parsed components for metadata assignment. Tests cover missing and trailing separators.

Changes

Container directory validation

Layer / File(s) Summary
Validate and reconcile container directories
pkg/monitor/nvidia/cudevshr.go, pkg/monitor/nvidia/cudevshr_test.go
Update parses directory names with SplitN, skips malformed entries, and assigns validated pod and container values. Tests cover names without an underscore and names with a trailing underscore.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

Suggested reviewers: lengrongfu, dsfans2014

Poem

A rabbit checks each name in line,
Pod and container parts align.
Bad paths hop away,
Good mappings stay.
Tests guard the burrow gate.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the fix for skipping malformed directories without an underscore and preventing a panic.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/monitor/nvidia/cudevshr_test.go`:
- Around line 436-446: Update the malformed-name subtests “dir without
underscore in name is skipped” and the corresponding “uid” case to create
matching pods and valid cache files, using UIDs “nodashes” and “uid”
respectively. Ensure each test reaches the malformed metadata path, assert that
Update() succeeds, and retain the expectation that l.containers remains empty.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 0e40b033-25f2-49df-b8cd-58c556704b24

📥 Commits

Reviewing files that changed from the base of the PR and between e6d0902 and 01846ba.

📒 Files selected for processing (2)
  • pkg/monitor/nvidia/cudevshr.go
  • pkg/monitor/nvidia/cudevshr_test.go

Comment thread pkg/monitor/nvidia/cudevshr_test.go
…king

strings.Split(entry.Name(), "_")[1] in Update() assumes every directory
under the hook path follows the <podUID>_<containerName> format. A stray
directory without an underscore causes an index out of range panic, which
crashes vGPUmonitor and silences all GPU metrics on that node.

Replace both unguarded Split calls with a single SplitN(..., 2) whose
result is validated before use; directories that do not match the expected
format are now skipped with a warning.

Signed-off-by: Nakshatra Sharma <nakshatra.sharma3012@gmail.com>
@Nakshatra480
Nakshatra480 force-pushed the fix/update-panic-on-underscore-free-dir branch from 01846ba to 206222f Compare August 5, 2026 06:13
@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@archlitchi archlitchi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@hami-robot

hami-robot Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: archlitchi, Nakshatra480

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hami-robot hami-robot Bot added the approved label Aug 6, 2026
@codecov

codecov Bot commented Aug 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag Coverage Δ
unittests 61.75% <100.00%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
pkg/monitor/nvidia/cudevshr.go 86.74% <100.00%> (+0.32%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@hami-robot
hami-robot Bot merged commit b13e155 into Project-HAMi:master Aug 6, 2026
17 checks passed
@Nakshatra480

Copy link
Copy Markdown
Contributor Author

Thanks @archlitchi for merging this pr 😊

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants