Bug: vgpu-monitor fails to parse cache file in v2.8.1 - "not valid UTF-8"

[simva]

Bug Report: vgpu-monitor fails to parse cache file in v2.8.1 - "not valid UTF-8"

Environment

• HAMi Version: v2.8.1
• Kubernetes Version: v1.34.6
• GPU Model: NVIDIA GeForce RTX 2080 Ti
• OS: Ubuntu (Linux/amd64)

Description

After upgrading to v2.8.1, vgpu-monitor fails to collect per-pod GPU metrics with the following error:

E0428 metrics.go:407] Failed to send memoryTotal metric for device 0 in Pod work-1234/pod-dir-test, Container gpu-test: failed to create metric: label value "\x80\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00GPU-609c" is not valid UTF-8

E0428 metrics.go:358] Failed to collect metrics for container gpu-test in Pod work-1234/pod-dir-test: failed to create metric: label value "\x80\x00...GPU-609c" is not valid UTF-8

Root Cause (suspected)

The cache file written by libvgpu.so (HAMi-core) in /usr/local/vgpu/containers/<uuid>_<container>/ contains binary data (struct) followed by the GPU UUID string. The vgpu-monitor in v2.8.1 appears to read the entire binary content as a UTF-8 label value, causing the parse failure.

This worked correctly in v2.8.0.

Steps to Reproduce

1. Install HAMi v2.8.1
2. Create a pod requesting nvidia.com/gpu: 1 with nvidia.com/gpumem and nvidia.com/gpucores
3. Run any CUDA workload inside the pod (e.g. torch.randn(1000,1000).cuda())
4. Check vgpu-monitor logs: kubectl logs -n kube-system <hami-device-plugin-pod> -c vgpu-monitor

Expected Behavior

Per-pod GPU metrics should be collected and exposed:

vGPU_device_memory_limit_in_bytes{podname="...", ...}
vGPU_device_memory_usage_in_bytes{podname="...", ...}
Device_utilization_desc_of_container{podname="...", ...}

Actual Behavior

• vgpu-monitor logs show not valid UTF-8 errors
• Per-pod metrics are completely missing from /metrics endpoint
• Only host-level metrics (HostCoreUtilization, HostGPUMemoryUsage) are exposed

Workaround

Downgrade to v2.8.0:

helm upgrade hami hami-charts/hami -n kube-system \
  --reuse-values \
  --set global.imageTag=v2.8.0

This restores correct per-pod metrics collection.

Additional Notes

• v2.8.1 release note only mentions a vLLM fix, so this regression may be unintentional
• The issue is in the interaction between libvgpu.so cache file format and vgpu-monitor parsing logic

[Shouren]

@simva I think #1792 should fix this issue

[archlitchi]

i'll soon release v2.8.2 to fix that problem

[archlitchi]

please try v2.8.2(https://github.com/Project-HAMi/HAMi/releases/tag/v2.8.2) and let me know if it solves this problem