Fix partial cache tail latency by correcting the cache chunk size calc

[cemakd]

What type of PR is this?
/kind bug

What this PR does / why we need it:
The GKE Data Cache was exhibiting extreme tail latency on partially
cached volumes, performing significantly worse than non-cached volumes.
Initial investigation into LVM cache policies and atime-induced
write-back thrashing did not reveal the root cause.

A series of fio benchmarks comparing cached and uncached volumes
uncovered a bug where the LVM cache was being configured with a
massive 1 GB chunk size. This was traced to a failure in the
fetchPvSizeGiB function in cache.go. The pvs command syntax was
malformed, and a subsequent string parsing error on its empty output
caused the driver to fall back to the default maxChunkSize of 1 GB.

This 1 GB chunk size led to extreme "read and write amplification," where a
single 4KB cache miss would trigger an inefficient 1 GB read from the
backing disk, explaining the catastrophic per-miss penalty.

This change corrects the pvs command syntax and the string parsing
logic within the driver. With this fix, the cache is now correctly
configured with a reasonable chunk size (e.g., 384 KiB), which resolves
the extreme read and write amplification and restores performance to expected
levels.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Fix partial data cache tail latency by correcting the cache chunk size calc

[mattcary]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cemakd, mattcary

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [mattcary]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sunnylovestiramisu]

Waiting for e2e testing edge cases confirmation

/hold

[k8s-ci-robot]

@cemakd: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-gcp-compute-persistent-disk-csi-driver-e2e-arm	648f174	link	false	/test pull-gcp-compute-persistent-disk-csi-driver-e2e-arm
pull-gcp-compute-persistent-disk-csi-driver-e2e-windows-2019	648f174	link	false	/test pull-gcp-compute-persistent-disk-csi-driver-e2e-windows-2019

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[cemakd]

/unhold e2e verification complete, the new chunk size is 2,368 KiB (or ~2.31 MiB) on a cluster with data-cache-count=6

[cemakd]

/cherry-pick release-1.21

[k8s-infra-cherrypick-robot]

@cemakd: new pull request created: #2177

Details

In response to this:

/cherry-pick release-1.21

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.