ci: stop non-etcd sanitizer stages from spawning the etcd/gRPC comm thread

[NirWolfer]

Summary

nixl_posix_test (and other single-process install-dir binaries) failed flakily under the ASan/UBSan sanitizer pipeline, aborting with no useful output on the teed logs. This fixes it without dropping any coverage by stopping those binaries from opening an unnecessary etcd/gRPC connection under the sanitizers.

Root cause

The failure is the GCC 13.3 libsanitizer thread-registry bug — the same toolchain issue nixl_gusli_test and MetadataExchangeTestFixture are already excluded for — but the trigger was incidental:

1. start_etcd_server (run for the gtest metadata-exchange tests) exports NIXL_ETCD_ENDPOINTS for the rest of the script.
2. detectEtcd() (nixl_agent.cpp) makes every nixlAgent start an etcd comm thread whenever that var is set.
3. That thread opens a gRPC channel whose WorkStealingThreadPool calls pthread_create, which flakily trips the libsanitizer CHECK.

A POSIX local file-backend test has no need for metadata exchange — the etcd connection was purely a side effect of the leaked env var.

Captured backtrace (via a temporary diagnostic that wrote sanitizer reports to files):

AddressSanitizer: CHECK failed: sanitizer_thread_registry.cpp:161
  #5  pthread_create (asan interceptor)
  #6  grpc_core::Thread::Thread
  #7  grpc ... WorkStealingThreadPool::StartThread
  #19 grpc_channel_create
  #22 etcd::detail::create_grpc_channel
  #23 etcd::SyncClient::SyncClient
  #25 nixlEtcdClient                       (nixl_listener.cpp:222)
  #27 nixlAgentData::commWorkerInternal     (nixl_listener.cpp:464)
  #28 nixlAgentData::commWorker             (nixl_listener.cpp:451)

Across a single 20-iteration diagnostic run the per-iteration exit codes were ~45% crashes (rc=134 SIGABRT / rc=139 SIGSEGV), the rest clean — consistent with a flaky toolchain race, not a NIXL bug.

Fix

• Run the only etcd-dependent install-dir binary (nixl_etcd_example) first, while the endpoints are still set.
• unset NIXL_ETCD_ENDPOINTS NIXL_ETCD_NAMESPACE NIXL_ETCD_PEER_URLS before the remaining single-process binaries.

With no endpoints set, those agents never start the comm thread, so they never reach the libsanitizer pthread_create path — making the stages deterministic while keeping full coverage (nothing is excluded; nixl_posix_test still runs its complete -n 128 -s 1048576 suite).

Notes

• nixl_etcd_example genuinely needs gRPC threads, so it remains the one stage still exposed to the same toolchain bug (it's the standalone equivalent of the already-excluded MetadataExchange). It's isolated up front now; if it proves flaky it can be excluded the same way.
• No product code changes — CI script only.

Summary by CodeRabbit

• Chores
  • Adjusted the sanitizer test sequence to run the etcd-backed example earlier in the workflow.
  • Prevented leftover etcd-related settings from affecting later test steps, improving test isolation and reliability.

[github-actions]

👋 Hi NirWolfer! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[NirWolfer]

/build

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 71649f4a-3971-4c19-9e06-4874b0f3dad8

📥 Commits

Reviewing files that changed from the base of the PR and between a9f456b and af895f8.

📒 Files selected for processing (1)

• .gitlab/test_sanitizer.sh

---

📝 Walkthrough

Walkthrough

In .gitlab/test_sanitizer.sh, the nixl_etcd_example stage is moved to execute before desc_example, agent_example, and nixl_example. After nixl_etcd_example runs, three environment variables (NIXL_ETCD_ENDPOINTS, NIXL_ETCD_NAMESPACE, NIXL_ETCD_PEER_URLS) are unset to prevent etcd/gRPC thread startup in subsequent single-process sanitizer stages.

Changes

Sanitizer Script Fix

Layer / File(s)	Summary
etcd stage reorder and env cleanup .gitlab/test_sanitizer.sh	nixl_etcd_example is moved before desc_example, agent_example, and nixl_example. After nixl_etcd_example completes, NIXL_ETCD_ENDPOINTS, NIXL_ETCD_NAMESPACE, and NIXL_ETCD_PEER_URLS are unset to prevent etcd/gRPC thread startup in later single-process stages.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Poem

A bunny reordered the tests with care,
etcd runs first, then cleans up its snare,
unset those endpoints, namespace too,
gRPC threads won't sneak through!
Sanitizer stages hop in the right queue. 🐇✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the CI sanitizer change and the affected non-etcd stages.
Description check	✅ Passed	The description covers what changed, why it was needed, and how the fix works, with supporting root cause details.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}