Skip to content

add troubleshoot for GPU-Operator 25.10+ - #716

Merged
FouoF merged 4 commits into
Project-HAMi:masterfrom
FouoF:update-faq
Aug 11, 2026
Merged

add troubleshoot for GPU-Operator 25.10+#716
FouoF merged 4 commits into
Project-HAMi:masterfrom
FouoF:update-faq

Conversation

@FouoF

@FouoF FouoF commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Checklist:

  • npm run lint and npm run format:check pass
  • npm run build succeeds for both en and zh
  • Chinese translation updated if English docs changed (or noted why not)
  • Commits are signed off (git commit -s)

Summary by CodeRabbit

Documentation

  • Added troubleshooting guidance for NVIDIA GPU Operator 25.10 and later, including Device Plugin and workload startup failures.
  • Documented RuntimeClass, CDI, runtime, driver path, Helm, and device injection diagnostics.
  • Added guidance for resolving conflicts with the GPU Operator Device Plugin.
  • Corrected FAQ and prerequisite links in English and Chinese documentation.
  • Updated metrics configuration guidance and clarified GPU runtime requirements and vendor terminology.

Signed-off-by: Jifei Wang <jifei.wang@dynamia.ai>
@netlify

netlify Bot commented Aug 4, 2026

Copy link
Copy Markdown

Deploy Preview for project-hami ready!

Name Link
🔨 Latest commit 55997f7
🔍 Latest deploy log https://app.netlify.com/projects/project-hami/deploys/6a756cdfdcea1f00088aaae5
😎 Deploy Preview https://deploy-preview-716--project-hami.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Added English and Chinese troubleshooting documentation for HAMi Device Plugin and workload failures with NVIDIA GPU Operator 25.10+, including runtime, CDI, RuntimeClass, driver, injection, and Device Plugin configuration guidance.

Changes

GPU Operator troubleshooting

Layer / File(s) Summary
Runtime and injection behavior
docs/troubleshooting/troubleshooting.md, i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/troubleshooting.md, i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md, versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md
Documents GPU Operator 25.10+ runtime and CDI behavior, HAMi injection modes, RuntimeClass requirements, driver paths, and disabling the GPU Operator Device Plugin.
Device Plugin diagnostics
docs/troubleshooting/troubleshooting.md, i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/troubleshooting.md, i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md
Adds diagnosis and remediation steps for HAMi Device Plugin startup failures, including logs, runtime and CDI checks, Helm settings, driver paths, restart steps, prerequisites, and version compatibility.
Workload startup diagnostics
docs/troubleshooting/troubleshooting.md, i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/troubleshooting.md, i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md, versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md
Adds event inspection, error classification, injection-mode guidance, runtime configuration checks, and CDI validation for HAMi-scheduled Pod startup failures.
FAQ and versioned documentation updates
docs/faq/faq.md, i18n/zh/docusaurus-plugin-content-docs/current/faq/faq.md, i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/faq/faq.md, versioned_docs/version-v2.9.0/faq/faq.md
Updates internal links, vendor descriptions, troubleshooting references, and the monitoring key from devicePlugin.monitorPort to devicePlugin.service.httpPort.

Estimated code review effort: 2 (Simple) | ~15 minutes

Possibly related PRs

Suggested labels: kind/documentation

Suggested reviewers: rootsongjc, windsonsea

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the main change: adding troubleshooting documentation for GPU Operator 25.10 and later.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot added the kind/documentation Improvements or additions to documentation label Aug 4, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/troubleshooting/troubleshooting.md`:
- Around line 122-126: Update the Helm value lookup in both troubleshooting
flows: docs/troubleshooting/troubleshooting.md lines 122-126 and
i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/troubleshooting.md
lines 114-118. Add the --all option to helm get values hami -n kube-system so
computed values, including default injection settings, are shown.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 9bb69af9-a19a-4f2d-a543-49579fc4987c

📥 Commits

Reviewing files that changed from the base of the PR and between c0e0f77 and 144525f.

📒 Files selected for processing (4)
  • docs/faq/faq.md
  • docs/troubleshooting/troubleshooting.md
  • i18n/zh/docusaurus-plugin-content-docs/current/faq/faq.md
  • i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/troubleshooting.md

Comment thread docs/troubleshooting/troubleshooting.md
@mesutoezdil

mesutoezdil commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

just 1 thing: "For new clusters, we recommend GPU Operator..." uses first-person, which the style guide disallows: https://github.com/Project-HAMi/website/blob/master/docs/contributor/contribute-docs.md#writing-style.
The zh translation already avoids it ("推荐使用 GPU Operator"), pls match that in the English text.

Signed-off-by: Jifei Wang <jifei.wang@dynamia.ai>

@spencercjh spencercjh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you need to copy these changes to the corresponding files in the v2.9.0 (latest stable version) directory.

Signed-off-by: Jifei Wang <jifei.wang@dynamia.ai>
@hami-robot hami-robot Bot added size/XL and removed size/L labels Aug 6, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@versioned_docs/version-v2.9.0/faq/faq.md`:
- Line 230: Update the FAQ entries at
versioned_docs/version-v2.9.0/faq/faq.md:230-230 and
i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/faq/faq.md:227-227 to
distinguish the hami-device-plugin monitor container port configured by
devicePlugin.monitorPort (9394 by default) from the Service/NodePort configured
by devicePlugin.service.httpPort (31992). State both ports clearly in English
and Chinese, using the Service port for Prometheus/scraping examples and the
monitor port as the Pod/container target.

In `@versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md`:
- Around line 61-72: Update the CDI and RuntimeClass verification commands in
the troubleshooting section of
versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md (lines 61-72)
and its Chinese counterpart
i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md
(lines 53-64) to validate spec.cdi.enabled=true and the nvidia RuntimeClass
.handler=nvidia using JSONPath, while preserving the existing HAMi Device Plugin
runtimeClass check.
- Around line 88-98: Correct the devicePlugin.nvidiaHookPath value from
nvidia-ctk to nvidia-cdi-hook in
versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md (lines 88-98)
and
i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md
(lines 80-90), preserving the existing GPU Operator path.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 84889e4e-1d55-4314-b963-b330534ed26b

📥 Commits

Reviewing files that changed from the base of the PR and between 4789849 and 2fc81de.

📒 Files selected for processing (4)
  • i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/faq/faq.md
  • i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md
  • versioned_docs/version-v2.9.0/faq/faq.md
  • versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md

Comment thread versioned_docs/version-v2.9.0/faq/faq.md Outdated
Comment thread versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md
Comment thread versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md
Signed-off-by: Jifei Wang <jifei.wang@dynamia.ai>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (5)
versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md (3)

32-32: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Make the Helm remediation explicit and reproducible.

The two one-line Helm upgrade commands silently fetch the latest chart and merge previous release values without showing the chart version. Pin the compatible chart version and add --reuse-values in both remediation blocks.

  • versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md#L32-L32
  • i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md#L27-L27
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md` at line 32,
Update both Helm remediation commands in
versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md:32-32 and
i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md:27-27
to pin the compatible chart version and include --reuse-values, keeping the
commands otherwise equivalent and synchronized.

18-18: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Fix the broken prerequisites links in both copies.

The installation directory is a sibling of troubleshooting, so the current ./installation/... path is incorrect.

  • versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md#L18-L18: Change the link to ../installation/online-installation.
  • i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md#L19-L19: Change the link to ../installation/online-installation.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md` at line 18,
Update the prerequisites link in
versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md at lines 18-18
and
i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md
at lines 19-19 from ./installation/online-installation to
../installation/online-installation.

18-18: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Make the nvidia runtime requirement conditional on non-CDI cases.

GPU Operator 25.10+ enables CDI by default and standard Device Plugin workloads do not use nvidia as the container runtime by default. Keep the nvidia requirement only for legacy envvar/management-container paths, and update the matching Chinese text the same way.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md` at line 18,
Update the runtime requirement text in
versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md:18-18 to make
nvidia required only for legacy envvar or management-container paths, not CDI or
standard Device Plugin workloads. Apply the equivalent wording change in
i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md:19-19.
i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/faq/faq.md (2)

178-178: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Fix the relative documentation links.

This file is under the faq/ directory. Therefore, ./core-concepts/..., ./userguide/..., and ./developers/... resolve below faq/. Use ../ for sibling documentation directories.

Proposed link corrections
- [GPU 虚拟化](./core-concepts/gpu-virtualization)
+ [GPU 虚拟化](../core-concepts/gpu-virtualization)

- [动态 MIG 支持](./userguide/nvidia-device/dynamic-mig-support)
+ [动态 MIG 支持](../userguide/nvidia-device/dynamic-mig-support)

- [GPU 利用率指标](./developers/gpu-utilization-metrics)
+ [GPU 利用率指标](../developers/gpu-utilization-metrics)

- [Grafana Dashboard](./userguide/monitoring/grafana-dashboard)
+ [Grafana Dashboard](../userguide/monitoring/grafana-dashboard)

Also applies to: 194-198, 221-221, 227-227

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/faq/faq.md` at line
178, Update the relative documentation links in the FAQ content, including the
references near the cited sections, so sibling directories such as
core-concepts, userguide, and developers use ../ instead of ./; preserve the
linked targets and link text.

194-194: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Improve the Chinese wording for flexible memory sizing.

需要灵活的显存大小 describes the memory size as “flexible”. Use a verb phrase that states the workload can configure memory flexibly.

Proposed wording
-工作负载需要灵活的显存大小
+工作负载需要灵活地配置显存大小
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/faq/faq.md` at line
194, Update the Chinese wording in the HAMi vGPU usage sentence so “需要灵活的显存大小”
is replaced with a verb phrase expressing that the workload can flexibly
configure memory, while preserving the surrounding MIG comparison and link
unchanged.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/faq/faq.md`:
- Line 181: Update the relative links in docs/faq/faq.md:181-181,
i18n/zh/docusaurus-plugin-content-docs/current/faq/faq.md:178-178, and
versioned_docs/version-v2.9.0/faq/faq.md:181-181 to use parent-relative paths
for core-concepts, userguide, and developers targets (../...). Apply the same
link corrections in all three FAQ copies.

In `@i18n/zh/docusaurus-plugin-content-docs/current/faq/faq.md`:
- Line 194: Update the sentence near the HAMi vGPU guidance to replace “灵活的显存大小”
with a grammatically correct modifier such as “灵活地调整显存大小,” while preserving the
surrounding meaning and structure.

---

Outside diff comments:
In `@i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/faq/faq.md`:
- Line 178: Update the relative documentation links in the FAQ content,
including the references near the cited sections, so sibling directories such as
core-concepts, userguide, and developers use ../ instead of ./; preserve the
linked targets and link text.
- Line 194: Update the Chinese wording in the HAMi vGPU usage sentence so
“需要灵活的显存大小” is replaced with a verb phrase expressing that the workload can
flexibly configure memory, while preserving the surrounding MIG comparison and
link unchanged.

In `@versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md`:
- Line 32: Update both Helm remediation commands in
versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md:32-32 and
i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md:27-27
to pin the compatible chart version and include --reuse-values, keeping the
commands otherwise equivalent and synchronized.
- Line 18: Update the prerequisites link in
versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md at lines 18-18
and
i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md
at lines 19-19 from ./installation/online-installation to
../installation/online-installation.
- Line 18: Update the runtime requirement text in
versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md:18-18 to make
nvidia required only for legacy envvar or management-container paths, not CDI or
standard Device Plugin workloads. Apply the equivalent wording change in
i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md:19-19.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: b02c1294-a744-4341-895e-b245435f77ae

📥 Commits

Reviewing files that changed from the base of the PR and between 2fc81de and 55997f7.

📒 Files selected for processing (8)
  • docs/faq/faq.md
  • docs/troubleshooting/troubleshooting.md
  • i18n/zh/docusaurus-plugin-content-docs/current/faq/faq.md
  • i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/troubleshooting.md
  • i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/faq/faq.md
  • i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/troubleshooting.md
  • versioned_docs/version-v2.9.0/faq/faq.md
  • versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md
🚧 Files skipped from review as they are similar to previous changes (2)
  • docs/troubleshooting/troubleshooting.md
  • i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/troubleshooting.md

Comment thread docs/faq/faq.md
Comment thread i18n/zh/docusaurus-plugin-content-docs/current/faq/faq.md

@spencercjh spencercjh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM now.

@rootsongjc rootsongjc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@hami-robot

hami-robot Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: FouoF, rootsongjc, spencercjh
Once this PR has been reviewed and has the lgtm label, please assign windsonsea for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@FouoF
FouoF merged commit ae002b2 into Project-HAMi:master Aug 11, 2026
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants