[jeffnvidia]

Temporary CI diagnostic for PR #11685.

This branch keeps the exact #11685 merge production code unchanged and retains the previously validated selection diagnostics. It removes only the two mock tests newly added by #11685, restoring test_ucp_proto_mock.cc exactly to the base revision, while leaving the original failing AM RNDV test unchanged.

Purpose: determine whether the Azure failure is caused by production protocol selection or by topology/mock state leaking from the new tests into a later randomized test.

This PR is not intended for review or merge. It will be closed and the temporary branch deleted after the Azure result is captured.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

blocker — leftover debug/diagnostic code must be removed. The dump_am_rndv_selection() function and its call at line 509 are pure debug scaffolding: they contain no assertions and only emit UCS_TEST_MESSAGE lines tagged PR11685_DIAG. This debug output: adds ~75 lines of test-diagnostic noise with no verification value; references the wrong PR number (PR11685 in PR #11861); spams the test log on every test_ucp_proto_am_rndv run. Remove the entire dump_am_rndv_selection function (lines 403–479) and its call at line 509.

[svc-nvidia-pr-review]

Remove the call to dump_am_rndv_selection() — part of the debug scaffolding that must be removed before merge.

[svc-nvidia-pr-review]

PROTO_INFO=y looks like a leftover debug flag. am_explicit_scheme_disables_force_provenance only asserts on check_am_rndv_remote_proto_config(0, 0) (op_id_flags and rkey config flag counts). PROTO_INFO=y just dumps protocol info at init and does not affect any assertion. It appears to be debug leftover paired with the PR11685_DIAG dump. Please drop it unless it is actually needed.

[svc-nvidia-pr-review]

Non-blocking (core fix): Setting mem_info.sys_dev = UCS_SYS_DEVICE_ID_UNKNOWN for the remote-proto select param is sound — the rkey sys_dev is in the remote topology namespace and must not be used as a local device when estimating the peer's protocol. The stored proto_config->select_param is taken from the outer select_param_copy in ucp_proto_select_elem_init, so this change only affects lane modeling during probe, not the selection hash key.

[svc-nvidia-pr-review]

Non-blocking: Overriding select_param.sys_dev with the staging (frag_mem_info->sys_dev) device is the necessary companion fix. With the rkey sys_dev now UNKNOWN, the mtype probe passes frag_mem_info as reg_mem_info, and ucp_proto_common_get_lane_perf/ucp_proto_common_filter_min_frag assert reg_mem_info.sys_dev == select_param->sys_dev under SEND_ZCOPY. This override keeps that invariant. The modified select_param is not propagated into the stored lookup key, as the code comment states.

[svc-nvidia-pr-review]

Non-blocking: The put_ppln pre-check flow is correct: the first smoke_test(true) runs before mock setup; the new pre-check runs with mocks installed but m_fail_target_proto=false, so close_peer() and op replacement are skipped and only m_target_proto_seen is set. m_is_peer_closed stays false for the real run. This addresses the fact that the capability check in init() doesn't cover staging-device reachability.

[svc-nvidia-pr-review]

Non-blocking: Entry is correctly alphabetically ordered.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (Tests new on worker 2) · commit e8b95559

TL;DR: The gtest run failed on a single timing-sensitive test, test_config.performance, which asserts that one car_opts (UCX config) parse completes within a hard-coded 0.05 s wall-clock budget; on the loaded shared Azure worker that budget was exceeded, so this is a flaky perf assertion, not a functional regression from PR #11861. Fix by making the measurement noise-tolerant (warm-up + best-of-N retries, or raise the limit).

Full analysis

Summary: make test in build-test/test/gtest exited 1 — 8569/8570 tests passed, the only failure being test_config.performance ("Time limit exceeded" from UCS_TEST_TIME_LIMIT(0.05)), causing Bash exited with code '2'.

Root cause: Evidence from the log: the run completed normally ([==========] 8570 tests from 1208 test suites ran. (3500301 ms total), [ FAILED ] 1 test ... test_config.performance) with no hang, crash, or watchdog kill — timestamps show continuous progress up to teardown at 12:54:57.

test_config.performance (test/gtest/ucs/test_config.cc:781-795) sets 300 extra env vars and then wraps a single car_opts construction in UCS_TEST_TIME_LIMIT(0.05). That constructor calls car_opts::parse() (test_config.cc:397-417), which inside the timed region does:

• two setenv/unsetenv operations (ucs::scoped_setenv ucx_config_dir("UCX_CONFIG_DIR", TEST_CONFIG_DIR)),
• ucs_config_parse_config_files() — actual filesystem reads of ucx.conf/ucx_test.conf from the workspace (/scrap/azure/agent-04/AZP_WORKSPACE/2/s/...),
• ucs_config_parser_fill_opts() with a 300-entry-larger environment.

So the assertion measures disk I/O plus environment scanning against a 50 ms budget, with no averaging and no retry: UCS_TEST_TIME_LIMIT (test/gtest/common/test_helpers.h:159-169) only checks ucs::perf_retry_count > 0 to enable the check (enabled by JENKINS_TEST_PERF, cf. PR #2131) — it never actually retries, unlike the UCT/UCP perf tests. One transient CPU/IO stall on this shared worker (this job is one of several running on swx-rdmz-ucx-new-01 / agent-04, and the PR under test is specifically about test isolation/concurrency) is enough to fail the whole 58-minute gtest run.

Implicated commit: unknown — pre-existing flakiness; test/gtest/ucs/test_config.cc was last touched by 597005c (Guy Ealey Morag, 2026-04-23) and src/ucs/config/parser.c also last in April 2026, neither related to the timing budget. Nothing in commit [REDACTED:Hex High Entropy String]'s area is implicated.

File: test/gtest/ucs/test_config.cc:792 (UCS_TEST_TIME_LIMIT(0.05)), with the measured code at test/gtest/ucs/test_config.cc:397-417; macro at test/gtest/common/test_helpers.h:159-169

Suggested fix: Make the assertion robust instead of raising the wall-clock limit blindly:

1. Add a warm-up parse before the timed block so config-file reads are already in page cache, e.g. { car_opts warmup(UCS_DEFAULT_ENV_PREFIX, NULL); } before UCS_TEST_TIME_LIMIT.
2. Amortize noise: loop the parse N times (e.g. 100) inside the timed block and scale the limit accordingly, or take the best/median of several attempts, so a single scheduler stall cannot fail the test.
3. Optionally make UCS_TEST_TIME_LIMIT honour ucs::perf_retry_count/perf_retry_interval by retrying the block up to perf_retry_count times and only failing if all attempts exceed the limit — this is the semantics -p/-i already imply and would fix all UCS_TEST_TIME_LIMIT users at once.
4. As a stopgap (precedent: PR GTEST/UCS: Increase test_config.perf time limit #2292 "GTEST/UCS: Increase test_config.perf time limit"), raise 0.05 → ~0.2 s.

This failure should not block PR #11861; re-run the job to confirm it is green.

Related: #2292 (previous increase of this exact test's time limit), #2131 (JENKINS_TEST_PERF gating of perf checks)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 482a4a48-8167-4478-907c-f2bba8a6c133 in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

minor — Only set_gtest_common_test_flags is switched to the fixed seed 37290; run_gtest_release still exports GTEST_RANDOM_SEED=0. Also, the specific value has no explanation. Consider a brief comment on why a fixed seed is needed (reproducing the ordering-dependent failure this PR addresses), and whether run_gtest_release should use the same seed for consistency.

Suggestion:

	export GTEST_RANDOM_SEED=37290  # fixed non-zero seed for reproducible shuffle

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

why hardcode the gtest random seed here? this changes the shuffle order for all gtests and looks unrelated to the rndv sys_dev fix. also run_gtest_release still uses GTEST_RANDOM_SEED=0, so the seed policy is now inconsistent. can this go in a separate PR, or is a specific seed needed to reproduce the fixed bug? Additionally, -s 11689 only reaches run_gtest_make (not run_specific_tests, which calls set_gtest_common_test_flags without GTEST_EXTRA_ARGS).

[svc-ucx]

🤖 CI Triage Agent — UCX PR (AddressSanitizer new on worker 0) · commit f0e7095c

TL;DR: The only failing test in the whole 8532-test ASan run is the newly added test_rcache_inv_q_growth.inv_q_bounded_with_interval_tree, whose hard-coded resource budgets (≤512 KB VmRSS growth / ≤100 interval-tree nodes over 40 000 munmap iterations) are not satisfiable in an AddressSanitizer build; the fix is to drop the RSS-based assertion (or skip the test under sanitizers/valgrind) and assert only on ucs_interval_tree_count().

Full analysis

Summary: make test in the "AddressSanitizer new on worker 0" job exited 1 because test_rcache_inv_q_growth.inv_q_bounded_with_interval_tree failed ([ PASSED ] 8531 tests, [ FAILED ] 1 test, make: *** [Makefile:4713: test] Error 1).

Root cause: The test is a resource-usage regression test, not a functional one. It performs 40 000 munmap+mmap iterations on an rcache created with UCS_RCACHE_FLAG_SYNC_EVENTS and max_unreleased = SIZE_MAX, so every unmap event goes through ucs_rcache_unmapped_callback() → ucs_interval_tree_insert() (src/ucs/memory/rcache.c:636-650), and then asserts:

• EXPECT_LE(q_delta, 100) (interval-tree node count), and
• EXPECT_LE(rss_delta, 512 /* KB */) — a VmRSS delta read from /proc/self/status.

Under AddressSanitizer neither budget is realistic: every interval-tree node allocation carries redzones plus a retained malloc stack trace, freed nodes go into ASan's quarantine instead of back to the OS, and ASan's allocator never trims — 40 000 insert/merge/free cycles inflate VmRSS well past 512 KB even though the interval tree itself stays compact. VmRSS is also a process-global metric, and this ASan binary runs the entire gtest suite in one process, so unrelated background allocations land inside the measurement window. The drain assertion (EXPECT_EQ(0u, inv_q_length())) is sound — ucs_rcache_check_inv_queue() (rcache.c:538-561) pops the whole tree — so the failure is one of the two threshold checks, most plausibly the RSS one.

Implicated commit: [REDACTED:Hex High Entropy String] (PR #11861, branch agent/pr11685-azure-test-isolation) — test_rcache_inv_q_growth does not exist in any upstream commit touching test/gtest/ucs/test_rcache.cc (latest upstream is 311e350 "UCS/RCACHE: Change inv_q type to interval tree (#11248)", nbellalou), so the test class was introduced by this PR.

File: test/gtest/ucs/test_rcache.cc:1100-1138 (thresholds at lines 1106-1107, failing checks at lines 1124 and 1130); related product code src/ucs/memory/rcache.c:636-650

Suggested fix:

1. Remove the VmRSS measurement from the test — RSS is not a deterministic property of the rcache and cannot be bounded in an instrumented (ASan) or shared-process run. Keep only the data-structure invariant, which is what the interval-tree fix actually guarantees:

   size_t q_delta = inv_q_length() - q_before;
   EXPECT_LE(q_delta, max_expected_nodes);   /* interval tree merges ranges */
   ...
   EXPECT_EQ(0u, inv_q_length());            /* one get() drains the tree */

2. If an RSS check is really wanted, gate it out for sanitizer/valgrind builds, e.g.

   #if defined(__SANITIZE_ADDRESS__) || defined(__has_feature) /* asan */
       UCS_TEST_SKIP_R("RSS accounting is unreliable under AddressSanitizer");
   #endif
       if (RUNNING_ON_VALGRIND) { UCS_TEST_SKIP_R("valgrind"); }

   (src/ucs/sys/checker.h currently only provides RUNNING_ON_VALGRIND; an ASan predicate would need to be added there.)
3. Also reduce num_iters (40 000 iterations of mlock/munmap plus UCM event dispatch is very slow under ASan) — a few thousand iterations already demonstrate bounded tree growth.

Related: upstream interval-tree work this test guards: 311e350 (#11248 "UCS/RCACHE: Change inv_q type to interval tree"), c0b9ee3 (#11213 "Interval tree fixes"), 6b25652 (#11120 "Introduce interval tree data structure"); no existing issue/PR mentions test_rcache_inv_q_growth.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id bb2986f5-e901-42f1-ad06-03b99fa07b05 in the triage console for the audit trail.