Cleanups for OBJ S3

[aranadive]

What?

Two follow-ups to #2241 (generic S3-over-RDMA in the OBJ plugin):

1. Keep cuObject/RDMA headers out of s3_accel/client.h to fix a wheel-build OOM.
2. Review cleanups that came in after plugins/obj: wire generic S3-over-RDMA into the accelerated engine #2241 merged.

Why?

• s3_accel/client.h included object/rdma/rdma.h, so every file including it compiled the full cuObject header set. That pushed compiler memory high enough to OOM the wheel build.
• Review cleanups:
  • The cuObject accelerated tests skipped on any construction failure, which hid real errors.

How?

• client.h forward-declares the RDMA types; rdma.h is included only in the .cpp files that need it.
• Empty accel type → "s3" is now resolved in the engine registry instead of createAccelEngine.
• cuObject accelerated tests skip when NIXL_OBJ_ENDPOINT_OVERRIDE is unset (like the Dell tests) and only skip a construction failure when the fast path is unavailable; anything else fails.
• const nixl_b_params_t * on the param getters, plus comment fixes.

Summary by CodeRabbit

• New Features

  • Object acceleration requests without an explicitly specified engine now use the S3 engine when available. If it is unavailable, the request reports that no engine type was specified.

• Bug Fixes

  • Zero-byte RDMA writes are now treated as successful, preventing unnecessary retries.
  • Accelerated-object tests now skip when the endpoint is unset or the fast path is unavailable, while reporting other engine construction failures.

[github-actions]

👋 Hi aranadive! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: ai-dynamo/nixl/.coderabbit.yaml
• Review profile: ASSERTIVE
• Plan: Enterprise
• Run ID: af16528a-d77f-4df4-8351-c52a97abbffa

---

📥 Commits

Reviewing files that changed from the base of the PR and between 78cbeed and 7e33ca9.

---

📒 Files selected for processing (5)

• src/plugins/obj/obj_engine_registry.cpp
• src/plugins/obj/s3_accel/engine_impl.cpp
• src/plugins/obj/s3_accel/engine_impl.h
• src/utils/object/engine_utils.h
• test/gtest/plugins/obj_cuobj_plugin.cpp

---

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

---

📝 Walkthrough

Walkthrough

Empty accelerator types now resolve through the registry to the registered s3 entry. Engine parameter helpers accept const parameters. S3 Accel declarations and includes changed. RDMA retries now stop for zero-byte writes. Accelerated-object tests handle endpoint absence and construction errors differently.

Changes

Object Engine and RDMA

Layer / File(s)	Summary
Engine type resolution and tests src/utils/object/engine_utils.h, src/plugins/obj/obj_engine_registry.cpp, src/plugins/obj/obj_backend.cpp, src/plugins/obj/s3_accel/engine_impl.cpp, test/gtest/plugins/obj_cuobj_plugin.cpp, test/gtest/plugins/meson.build	Engine parameter helpers accept const pointers. Empty engine types resolve to the registered s3 entry. Accelerated-object tests skip when the endpoint is unset or the fast path is unavailable, and fail on other construction errors. The test target links against gmock_dep.
S3 Accel declarations and includes src/plugins/obj/s3_accel/client.*, src/plugins/obj/s3_accel/engine_impl.*	S3 Accel headers use forward declarations and revised include placement. The readiness comment states that rdmaEngReady() delegates to supportsRdma().
RDMA retry and signing guidance src/utils/object/rdma/rdmaRetry.cpp, src/utils/object/rdma/s3_control_plane_http.cpp	RDMA retries stop on zero-byte writes. The signV4 comment describes operation-specific headers and the headers added and signed by the signer.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

---

Merge Risk: ⚪ Minimal · up to 7e33c

This PR keeps RDMA headers out of the S3 Accel client header, resolves an empty engine type to the registered s3 engine, treats zero-byte RDMA writes as successful, and tightens when the accelerated-object tests skip. The earlier test build failure is addressed by declaring the Google Mock dependency. No merge-blocking issues remain.

Pre-merge checks | 4 | 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 36.84% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 19 functions across 10 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Title check		The title is concise and directly related to the pull request. It identifies cleanup changes for the OBJ S3 implementation.
Description check		The description includes the required What, Why, and How sections. It explains the header cleanup, registry behavior, test changes, and related rationale.

---

---

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

---

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[aranadive]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-build-container-pr · commit 19338283

TL;DR: The container build fails compiling test/gtest/plugins/obj_cuobj_plugin.cpp with fatal error: gmock/gmock.h: No such file or directory, because the new ASSERT_THAT/testing::HasSubstr gmock usage added in the #2241 review-cleanup commit was never matched by adding gmock_dep to the plugins_gtest target. Add gmock_dep to the dependency list in test/gtest/plugins/meson.build.

Full analysis

Summary: Stages "Build image" (x86_64 node 193 and aarch64 node 208) failed at Dockerfile STEP 68/74 (meson setup … && ninja) — the plugins_gtest target could not compile obj_cuobj_plugin.cpp.

Root cause: test/gtest/plugins/obj_cuobj_plugin.cpp:20 includes <gmock/gmock.h> (needed for the ASSERT_THAT(constructionError_, testing::HasSubstr("fast path is unavailable")) assertion at line 93, introduced by the review-cleanup commit for #2241). The plugins_gtest executable in test/gtest/plugins/meson.build:56 lists only gtest_dep and not gmock_dep. In this image gmock is not a system package — the meson log shows Run-time dependency GMock found: YES (building self), i.e. gmock is built from the vendored googletest subproject, so its include directory is only propagated through gmock_dep. The compile line in the log confirms no gmock include path is present, hence the fatal error and ninja: build stopped: subcommand failed → Error: building at STEP "RUN rm -rf build && … ninja": exit status 1.

This file is only compiled when cuobj_dep.found() (meson.build:48-50), which is why only the two variants that found cuobjclient-13.4 failed while the other two "Build image" stages passed.

Implicated commit: b7fdc710 — "plugins/obj: follow-up review cleanups for generic S3-over-RDMA (#2241)", Adit Ranadive (built as PR #2356 head [REDACTED:Hex High Entropy String]); the gmock-dependent assertion pattern originates in f080903d (Harshavardhana).

File: test/gtest/plugins/meson.build:56 (missing dep) — triggered by test/gtest/plugins/obj_cuobj_plugin.cpp:20 and :93

Suggested fix: Add gmock_dep to the plugins_gtest dependency list:

plugins_test_exe = executable('plugins_gtest',
    sources : plugins_gtest_sources,
    include_directories: [nixl_inc_dirs, utils_inc_dirs, plugins_inc_dirs, '.'],
    cpp_args : plugins_gtest_cpp_flags,
    dependencies : [nixl_dep, gtest_dep, gmock_dep, absl_strings_dep, absl_time_dep,
                    plugin_deps, obj_backend_interface] + plugins_gtest_cuda_deps,
    link_with: [nixl_build_lib],
    install : true
)

gmock_dep is already defined and guarded in the parent test/gtest/meson.build:24-29, so it is in scope. Alternatively, if the gmock matcher is not worth the extra dependency, replace ASSERT_THAT(constructionError_, testing::HasSubstr(...)) with a plain gtest check such as ASSERT_NE(constructionError_.find("fast path is unavailable"), std::string::npos) and drop the <gmock/gmock.h> include. Note azure_blob_gtest likely has the same latent gap if it ever adopts gmock matchers.

Related: PR #2356 (this build), PR #2241 (origin of the gmock-based assertion); no existing issue found for this error signature.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id de05280c-ddf3-4a78-963b-4e4621310278 in the triage console for the audit trail.

[aranadive]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 9234b465

TL;DR: The build and image push succeeded; the job failed in the "Allocate DL EP Environment" stage because salloc on the gb200nvl72_cx8 partition waited out its full --immediate=3600 window without any node freeing up and gave up — a cluster capacity/queue issue, not a defect in PR #2356.

Full analysis

Summary: Stage "Allocate DL EP Environment" failed after 60 minutes with salloc: error: Unable to allocate resources: Connection timed out, aborting the parallel branch build_helper_dl_ep/aarch64/1.

Root cause: The pipeline requests an exclusive single-node allocation on the DL cluster:

salloc -N 1 -p gb200nvl72_cx8 --job-name=nixl-ci-dl-gpu-ep-1534 \
       --immediate=3600 --time=02:00:00 --no-shell --account=blackwell

Slurm accepted it (Pending job allocation 2396456), logged job 2396456 queued and waiting for resources, and then after exactly the --immediate=3600 deadline (20:26:09Z → 21:26:18Z, a precise 3600 s) returned Unable to allocate resources: Connection timed out. That message plus the exact match to the --immediate value means the request was never satisfiable within the window — the partition had no free node for an hour (note salloc: lua: Setting job to exclusive mode as no gpu count was specified, so the job demanded a whole GB200 node). This is a scheduling/capacity failure external to the build: everything before it passed, including the full NIXL + nixlbench compile and the push of nixl-ci-dl-gpu-ep-test:1534.

The one-hour gap in the log is the intended blocking salloc wait, not a hung process, so this is not a hang and raising the pipeline time limit would not help.

Secondary issue visible in the cleanup stage: slurm.stopAllForBuild reported WARNING: No job ID files found in /mnt/pvc/nixl-ci-dl-gpu-ep/. The jobIdFile (job_id_ep_1534.txt) is only written once an allocation succeeds, so a job that is queued and then abandoned leaves nothing for the cleanup stage to scancel. Here salloc itself errored out so 2396456 was released by Slurm, but the same code path would leak a pending allocation if the stage were aborted mid-wait.

Implicated commit: none — no code commit is implicated. The build of [REDACTED:Hex High Entropy String] compiled and installed cleanly; the failure is in the Slurm allocation step. Nearest related CI work is 2dd278bf / 1f87bd7b (NirWolfer), which added partition-scoped GPU pipelines and node exclusions, but neither touches this partition's allocation parameters.

File: Jenkins shared library swx-jenkins-lib → slurm.allocation invocation for job nixl-ci-dl-gpu-ep, parameters partition: gb200nvl72_cx8, immediateTimeout: 3600, jobIdFile: /mnt/pvc/nixl-ci-dl-gpu-ep/job_id_ep_1534.txt (the pipeline definition is not in this repo's .ci/jenkins tree).

Suggested fix:

1. Simply retry the build — the artifacts all built fine and this is a transient queue-contention failure. If gb200nvl72_cx8 is persistently saturated, ask the DL cluster admins what is holding the nodes; quote pending allocation 2396456 under account blackwell.
2. Make the failure cheaper and clearer rather than longer: wrap the allocation stage in a retry with a shorter immediateTimeout (e.g. 2 × 900 s instead of one 3600 s blocking wait) so the pipeline doesn't burn an hour of agent time on a full queue, and so a transient opening is picked up on the second attempt.
3. Distinguish "queue full" from "controller unreachable" in the stage output — both surface as Connection timed out. Have the wrapper run squeue -j <jobid>/sinfo -p gb200nvl72_cx8 on failure so the log records whether nodes were busy, drained, or the controller was down.
4. Write the Slurm job ID to jobIdFile as soon as salloc reports Pending job allocation <id>, not only after the allocation is granted, so slurm.stopAllForBuild can scancel a still-queued job instead of logging "No job ID files found".

Related: PR #2356 (this build's PR — unaffected by the failure); #2339 / #2182 (NirWolfer, prior partition-specific GPU pipeline and node-exclusion changes) are the closest precedent for adjusting partition allocation behaviour. No existing issue tracks gb200nvl72_cx8 allocation timeouts — worth opening one if this recurs.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 7722e706-f391-4566-a6b7-35f46601f9e9 in the triage console for the audit trail.

[aranadive]

/ok to test 7e33ca9

[aranadive]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 7e33ca9e

TL;DR: The Allocate Environment stage failed because scctl could not log in — the Rancher auth endpoint (10.220.0.21:443) was unreachable (TCP connect timeout after 4 retries), so no slurm allocation was ever created and the pipeline aborted. This is a CI infrastructure/network failure, unrelated to commit 7e33ca9 or PR #2356; re-run the build once Rancher/network connectivity is restored.

Full analysis

Summary: nixl-ci-gpu #3943 failed in the Allocate Environment stage (node 164) after the build/Docker-image stages all succeeded; scctl could not obtain an auth token and the parallel branch build_helper/x86_64/1 aborted with exit code 1.

Root cause: Pure infra failure in the slurm allocation helper, not in nixl code:

• 18:05:07 — Calling slurm.allocation with args=[partition:mizu, headNode:scctl, nodes:1, jobTimeout:02:20:00, jobName:nixl-ci-gpu-3943, ...]
• 18:11:59 (≈6m52s of silence, i.e. the whole stage runtime spent in network retries) — Error: failed to get token: error sending rancher token request to https://rancher-swx-k8s-mec.mec01.labs.mlnx.nvidia.com/v3-public/azureADProviders/azuread?action=login: request failed after 4 retries: ... dial tcp 10.220.0.21:443: connect: connection timed out

The gap is explained by scctl's own 4 retries against an unreachable Rancher endpoint, so this is a blocked/unreachable service rather than an application hang. Because login never succeeded, sbatch was never issued: the teardown stage reports WARNING: No job ID files found in /mnt/pvc/nixl-ci-gpu/, and querying slurm_metrics.slurm_jobs_all_local for name = 'nixl-ci-gpu-3943' returns no rows (verified twice; the table clearly covers this cluster — neighbouring build nixl-ci-gpu-3914 appears there as job 101867 on mizu01). So no allocation was ever made and no node or GPU was involved in the failure. Everything before this point — UCX install, meson configure, the full 316-target ninja build, nixlbench, and the image push to artifactory as nixl-ci-gpu-test:3943 — completed successfully.

Implicated commit: none — the failure is in the Jenkins swx-jenkins-lib slurm.allocation step / scctl v0.12.2 login path, not in the repository. The tested commit [REDACTED:Hex High Entropy String] built cleanly.

File: not a repo file — pipeline stage "Allocate Environment" (stage ID 164), slurm.allocation in shared library swx-jenkins-lib; failing endpoint rancher-swx-k8s-mec.mec01.labs.mlnx.nvidia.com → 10.220.0.21:443

Suggested fix:

1. Simply retry/re-trigger build #3943 — the image is already built and pushed, and nothing in the PR is implicated.
2. Ask CI/IT to check reachability of 10.220.0.21:443 (Rancher swx-k8s-mec AzureAD login endpoint) from the nbu-swx-nixl Jenkins k8s namespace; a firewall/route change or a down Rancher ingress is the likely trigger.
3. Pipeline hardening: wrap the slurm.allocation/scctl login call in a Jenkins retry(2) with a short backoff, and classify this specific error (failed to get token: ... connection timed out) as an infra-abort so the ~7-minute silent stall surfaces as a retryable infra error instead of a generic script returned exit code 1. Also consider lowering scctl's connect timeout so the stage fails fast instead of burning seven minutes on TCP retries.

Note: the stage ran under withCredentials with $SERVICE_USER_PASSWORD; Jenkins masked it correctly and no credential material is exposed in the log — no rotation needed.

Related: none found (issue/PR search for this error signature returned no results)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 914f17ef-ddc5-4339-96ea-ac87c0bde56b in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-non-gpu · commit 7e33ca9e

TL;DR: The "Test CPP" stage of one build matrix variant failed because 42 gtest cases that passed exited with code 42 — the gtest harness treats any WARNING log as a test failure, and nixlTelemetry's constructor emits telemetry: no invariant CPU counter available... on CI agents whose CPU/container doesn't expose an invariant TSC. Fix by demoting that one-time message below WARNING (or registering a LogIgnoreGuard for it in the telemetry test fixtures).

Full analysis

Summary: Stage 496 ("Test CPP", one of six parallel matrix variants) failed with FAILED TESTS (42/302) — all 42 are telemetry/Prometheus/MpExporter tests that printed [ PASSED ] 1 test and then returned with exit code 42.

Root cause: Every failing test log shows the same pattern and nothing else:

W1008 17:56:24.006266 4893 telemetry.cpp:124] telemetry: no invariant CPU counter available;
       per-transfer timing falls back to steady_clock (higher overhead)
ATTENTION: Unexpected NIXL warning or error detected!
[  PASSED  ] 1 test.
ATTENTION: Problem count is 1
[38/302] telemetryTest.BasicInitialization returned with exit code 42 (489 ms)

Two pieces combine:

1. src/core/telemetry/telemetry.cpp:123-126 — the nixlTelemetry constructor unconditionally logs NIXL_WARN when nixlTime::fastClockUsesHwCounter() returns false. This is purely informational (it reports a performance fallback, not an error).
2. test/gtest/common.cpp:188-213 — LogProblemCounter::Send() counts any absl record whose severity is not kInfo as a "problem", and the gtest main exits 42 when the count is non-zero. There is no LogIgnoreGuard registered for this message in telemetryTest, prometheusTelemetryTest, telemetryStressTest, MpExporterTest, or ucx_telemetry*/TestTransferTelemetry.

So on any agent where the invariant TSC is unavailable, every test that constructs a nixlTelemetry fails despite passing. That is environment-dependent, which matches the build exactly: the identical gtest binary's "Test CPP" stage passed in the other five matrix variants (nodes 470, 525, 441, 483, 538) and only failed on the pod nixl-ci-non-gpu-nixl-ci-non-gpu-base-ubuntu22-3406-1j828-g8w85. Note also telemetryStressTest.RepeatedLifecycleShakeOut reports Problem count is 30 — one warning per telemetry construction — confirming the message is not rate-limited.

This failure is unrelated to the PR's content (obj/2241 review cleanups); nothing in the failing output touches the OBJ plugin.

Implicated commit: 355fbd65 (e-eygin, "telemetry: measure per-transfer durations with a CPU-counter stopwatch (#1890)") introduced the warning; c7e97a41 (Colin Hirsch, "TEST/GTEST: Fail on unexpected error or warning log. (#1288)") made warnings fail tests. Neither is in this branch — the combination is latent and surfaces only on hosts without an invariant counter.

File: src/core/telemetry/telemetry.cpp:123-126 (emitter); test/gtest/common.cpp:188-213 (the exit-42 policy)

Suggested fix: Preferred — demote the message so it can't fail tests, since it describes a capability fallback rather than a fault:

if (!nixlTime::fastClockUsesHwCounter()) {
    NIXL_INFO << "telemetry: no invariant CPU counter available; per-transfer timing "
                 "falls back to steady_clock (higher overhead)";
}

and make it fire once per process (e.g. a std::call_once / LOG_FIRST_N) so repeated-construction tests don't spam it. If the WARNING severity must be kept for production visibility, add a gtest::LogIgnoreGuard{"no invariant CPU counter available"} to the shared fixture base used by telemetryTest, prometheusTelemetryTest, telemetryStressTest, MpExporterTest, and TestTransferTelemetry, so it is tolerated on agents lacking the counter. A third, weaker option is pinning the non-GPU CI pool to hosts exposing constant_tsc/nonstop_tsc, but that leaves the latent fragility in place.

Related: #1890 (introduced the CPU-counter stopwatch and this warning), #1288 (gtest fails on unexpected warning/error), #1521 ("Fix IB warning test failure in CI" — same class of problem, fixed by ignoring an environment-dependent warning)

[aranadive]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-non-gpu · commit 5deb70a5

TL;DR: The Test CPP leg failed because 42 gtest cases exited with code 42 — the gtest harness fails on any unexpected NIXL warning, and nixlTelemetry's constructor emits telemetry: no invariant CPU counter available... on CI hosts where the kernel clocksource isn't TSC. Fix by demoting that message to INFO/VLOG, or adding a LogIgnoreGuard for it in test/gtest/main.cpp.

Full analysis

Summary: Stage 538 (Test CPP, one of six parallel matrix legs) failed: gtest-parallel reported FAILED TESTS (42/302), all telemetry-related, each "returned with exit code 42".

Root cause: Every one of the 42 failures passed its assertions — the log shows [ PASSED ] 1 test followed by:

W1009 20:45:34.643342 4868 telemetry.cpp:124] telemetry: no invariant CPU counter
available; per-transfer timing falls back to steady_clock (higher overhead)
ATTENTION: Unexpected NIXL warning or error detected!
... returned with exit code 42

Exit code 42 comes from test/gtest/main.cpp:105-109: LogProblemCounter turns any unignored NIXL WARNING/ERROR log record into a suite failure (introduced by c7e97a4, "TEST/GTEST: Fail on unexpected error or warning log"). The warning itself is emitted unconditionally in nixlTelemetry's constructor (src/core/telemetry/telemetry.cpp:123-126) whenever nixlTime::fastClockUsesHwCounter() is false.

That predicate is computed in src/utils/common/nixl_duration.cpp:91-120: on x86_64 it requires both hasInvariantTsc() (CPUID.80000007H:EDX[8]) and kernelClocksourceIsTsc() — reading /sys/devices/system/clocksource/clocksource0/current_clocksource == "tsc". On a virtualized/containerized Jenkins agent where the kernel has demoted the clocksource to hpet/acpi_pm/kvm-clock, the stopwatch falls back to steady_clock and the warning fires on every agent construction — so the failure is host-dependent, which is exactly why only 1 of 6 otherwise-identical matrix legs failed while the other five passed Test CPP.

This is unrelated to the PR under test (obj/S3 review cleanups); nothing in the build or the OBJ plugin is implicated. It's a latent environment-sensitive test-harness bug.

Implicated commit: 355fbd6 — "telemetry: measure per-transfer durations with a CPU-counter stopwatch (#1890)", e-eygin (added both the calibration logic and the NIXL_WARN); made fatal to tests by c7e97a4 — "TEST/GTEST: Fail on unexpected error or warning log (#1288)", Colin Hirsch.

File: src/core/telemetry/telemetry.cpp:123-126 (warning emitted); src/utils/common/nixl_duration.cpp:97 (clocksource gate); test/gtest/main.cpp:105-109 (exit code 42).

Suggested fix: The message is purely informational — a slower-but-correct timing path, not a problem — so it should not be a WARNING. Preferred: demote it to NIXL_INFO / VLOG in src/core/telemetry/telemetry.cpp:124, and consider emitting it once per process (e.g. std::call_once) rather than per nixlTelemetry construction — note telemetryStressTest.RepeatedLifecycleShakeOut logged it 30 times in a single test.

If the WARNING severity must be kept, add an ignore guard alongside the existing ib_regex in test/gtest/main.cpp RunAllTests() (~line 87):

const std::regex fastclock_regex(
    "telemetry: no invariant CPU counter available");
ligs.emplace_back(fastclock_regex);

Mitigation note: this will keep flapping unpredictably across the non-GPU matrix until fixed, since it depends on which agent the leg lands on.

Related: PR #2356 (this build); PR #1890 (introduced the stopwatch and the warning); PR #1288 (made unexpected warnings fail the suite); the ib_regex entry in test/gtest/main.cpp:86-87 carries a // TODO: Remove after the CI issues spuriously triggering this message are fixed comment — the same class of problem.