test(inference): mark prefix caching CUDA graph test flaky

[ko3n1g]

• I, the PR author, have personally reviewed every line of this PR.

Background / motivation

TestPrefixCachingCudaGraphs::test_prefix_caching_cuda_graphs intermittently invalidates CUDA graph capture in the dev inference unit-test bucket. The failure is independent of the triggering changes:

PR	Change scope	Failure evidence
#6110	NeMo Run dependency pin	[decode-transformer], run 30479289096 / job 90671101491
#6049	Functional inference recipes	[mixed-transformer], run 30463155293 / job 90637047921
#6120	Package-version strings	[mixed-transformer], run 30427570429 / job 90655889236
#6109	Documentation only	[mixed-transformer], run 30407317207 / job 90536399941

Each job reports CUBLAS_STATUS_INTERNAL_ERROR from cublasSetStream, followed by cudaErrorStreamCaptureInvalidated at capture end.

What changed

• Mark the parametrized prefix-caching CUDA graph test with pytest.mark.flaky_in_dev.
• Link the quarantine to 🐛 CI failure: tests/unit_tests/inference/engines/test_prefix_caching_cuda_graphs.py::TestPrefixCachingCudaGraphs::test_prefix_caching_cuda_graphs #6130.

Details

The dev unit-test runner selects not flaky_in_dev, so this prevents the intermittent capture failure from blocking standard PR CI. The broader flaky marker is not added because the observed failures are all in the dev / latest job; LTS coverage remains enabled.

Fixes #6130.

Tested

• uv sync --locked --only-group linting — completed from the locked environment.
• BASE_REF=main CHECK_ONLY=true SKIP_DOCS=false bash tools/autoformat.sh with the linting environment on PATH — black, isort, pylint, and ruff passed; the wrapper exited 0. Its non-blocking mypy invocation was unavailable in the repository's linting-only group.
• Parsed the test module with Python ast and verified the function decorator list includes pytest.mark.flaky_in_dev ahead of all six parametrized cases.
• git diff --check — passed.
• Full GPU pytest was not run locally: the available environment does not include PyTorch, and collection stops in tests/unit_tests/conftest.py with ModuleNotFoundError: torch.

Contribution pre-checks

• I have added relevant unit tests — not applicable; this change quarantines an existing flaky test.
• I have added relevant functional tests — not applicable.
• I have added proper typing to my code — not applicable; no executable code changed.
• I have added relevant documentation — issue 🐛 CI failure: tests/unit_tests/inference/engines/test_prefix_caching_cuda_graphs.py::TestPrefixCachingCudaGraphs::test_prefix_caching_cuda_graphs #6130 records the failure and evidence.
• I have run tools/autoformat.sh on my PR.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.