[NirWolfer]

On 2026-10-07, between 18:01 and 20:12 UTC, three AWS EFA validation runs (#2305, #2299, #2326) failed while installing build dependencies with E: Unable to fetch some archives. Every IPv4 address of archive.ubuntu.com and security.ubuntu.com timed out from the pod. Other downloads in the same pod were unaffected: content.mellanox.com delivered the 835 MB DOCA host package in 5 s, and the CUDA repo answered normally. Runs that passed in that window also downloaded from the Ubuntu archive at between 81 B/s and 1.7 MB/s, against 11–19 MB/s on a normal day. Across the last 400 AWS runs these are the only apt failures.

The pod reaches Canonical's archive over the internet. Canonical runs a regional mirror on EC2 in each AWS region (<region>.ec2.archive.ubuntu.com, on EC2 addresses in eu-central-1) that carries the same suites, including -security. Pointing apt at it keeps that traffic inside the region.

Changes:

• contrib/aws-efa/aws_test.sh: before the first apt-get update, the pod command rewrites http://archive.ubuntu.com/ and http://security.ubuntu.com/ under /etc/apt to http://${AWS_DEFAULT_REGION:-eu-central-1}.ec2.archive.ubuntu.com/. It works with both the deb822 (24.04) and sources.list (22.04) formats, and does nothing if the image has neither URL. .gitlab/build.sh is unchanged, so other pipelines are not affected.
• contrib/aws-efa/README.md: note the mirror.

Not changed: apt on 24.04 already retries each download 3 times with backoff by default, and the IPv6 attempts fail immediately (Network is unreachable), so neither Acquire::Retries nor Acquire::ForceIPv4 would have changed these failures. The apt-get upgrade -y after the DOCA repo is added stays; it moves the RDMA packages to the DOCA versions (#889).

Summary by CodeRabbit

• Documentation
  • Clarified that package installation in the container uses the Ubuntu mirror for the configured AWS region.
• Chores
  • Updated the AWS test setup to use the regional Ubuntu mirror when installing Git.

[github-actions]

👋 Hi NirWolfer! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: ai-dynamo/nixl/.coderabbit.yaml
• Review profile: ASSERTIVE
• Plan: Enterprise
• Run ID: fc5326d8-b408-4118-8409-26106b91fb37

---

📥 Commits

Reviewing files that changed from the base of the PR and between b69182f and d713b56.

---

📒 Files selected for processing (2)

• contrib/aws-efa/README.md
• contrib/aws-efa/aws_test.sh

---

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

---

---

📝 Walkthrough

Walkthrough

The AWS EFA test setup now configures apt to use an in-region EC2 Ubuntu mirror before updating package sources. The README documents this mirror.

Changes

AWS EFA apt mirror

Layer / File(s)	Summary
Configure and document the apt mirror contrib/aws-efa/aws_test.sh, contrib/aws-efa/README.md	The test command builds the mirror URL from AWS_DEFAULT_REGION, defaulting to eu-central-1, and replaces the default Ubuntu apt sources before running apt-get update. The README documents the in-region mirror. Git installation and remote test commands remain in their existing order.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: ovidiusm

---

Merge Risk: ⚪ Minimal · up to d713b

The change points apt in the AWS EFA test pod at the in-region EC2 Ubuntu mirror and documents it. No actionable merge-blocking risk remains.

Pre-merge checks | 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check		The title clearly and concisely identifies the main change: using the in-region EC2 Ubuntu mirror in the AWS EFA pod.
Description check		The description explains what changed, why the change is needed, and how the implementation works. It also covers compatibility, scope, and relevant non-changes, although it does not use the template …
Docstring Coverage		No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

---

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[NirWolfer]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-test-sanitizers · commit d713b56a

TL;DR: The build failed on one flaky gtest, MDManagerEtcdFixture.InvalidateLocalRemovesRemote, which timed out after 15.6 s because the dst agent never saw the etcd DELETE — its prefix watcher is armed after the metadata is loaded (and at the then-current revision), so a deletion landing in that window is missed. The fix is to arm the watcher before/at the fetch revision in nixlEtcdMetadataBackend::fetchRemote, not after loadRemoteMD.

Full analysis

Summary: Stage 198 "Test Sanitizer" (x86_64 asan_ubsan) failed: the meson sanitizer gtest suite reported 1 FAILED TEST — MDManagerEtcdFixture.InvalidateLocalRemovesRemote — while every other binary (ucx_backend_test, nixl_posix_test, ucx_backend_multi, serdes_test, test_plugin) passed.

Root cause: A race in the ETCD metadata backend, not in the commit under test. From the artifact meson-logs/testlog.junit.xml:

../test/gtest/md_manager.cpp:314: Failure
Expected equality of these values:
  waitForRemoteMD(dst.agent.get(), src.name, {DRAM_SEG}, NIXL_ERR_NOT_FOUND)
    Which is: 0
  NIXL_ERR_NOT_FOUND
    Which is: -4
[  FAILED  ] MDManagerEtcdFixture.InvalidateLocalRemovesRemote (15608 ms)

checkRemoteMD returned NIXL_SUCCESS (0) for the full 15 s sanitizer timeout — the dst agent never invalidated its cached remote MD. In nixlEtcdMetadataBackend::fetchRemote, the worker task calls ctx_.loadRemoteMD(blob, loaded_name) (line 401) and only then client_->setupAgentWatcher(remote_name) (line 413). The test's first waitForRemoteMD(..., NIXL_SUCCESS) is satisfied as soon as loadRemoteMD completes, so it proceeds to src.agent->invalidateLocalMD(nullptr) while dst's watcher may not exist yet. setupAgentWatcher constructs etcd::Watcher with no start revision, i.e. watching from now, so an rmdir that lands in that gap produces no event, invalidatedAgents_ stays empty, processInvalidatedAgents never runs, and dst's cache is never invalidated. Sanitizer instrumentation widens the gap, which is why the asan_ubsan branch failed while the parallel branch (stage 185) passed.

Corroborating evidence that no event was delivered: the gtest system-err contains no Watcher failed..., no Watcher ... received unexpected number of events, and no Failed to invalidate remote metadata for agent — the watcher simply never fired.

Secondary latent bug in the same area: rmdir(prefix, true) deletes both the anchor key and the metadata key, but the callback rejects any response where response.events().size() != 1 (line 278), dropping the invalidation entirely if etcd batches both deletes into one response.

Implicated commit: 2ff0105f — "Metadata manager pr5 (#1930)", aschwartz12 — introduced this fetch/watch ordering. The commit under test, [REDACTED:Hex High Entropy String] (NirWolfer, "CI: use the in-region EC2 Ubuntu mirror in the AWS EFA pod", PR #2385) only touches contrib/ apt configuration and is unrelated; this is a pre-existing flake, not a regression from the PR.

File: src/core/nixl_etcd_metadata_backend.cpp:401-413 (ordering), src/core/nixl_etcd_metadata_backend.cpp:278 (event-count guard); test assertion at test/gtest/md_manager.cpp:314.

Suggested fix:

1. In fetchRemote, call client_->setupAgentWatcher(remote_name) before ctx_.loadRemoteMD(...), so the watcher is armed before the metadata becomes observable to callers.
2. Better, close the window properly: have fetchOrWaitForMetadataFromEtcd return the etcd revision it read (response.index(), already available in waitForMetadataFromEtcd) and pass it as the Watcher start index in setupAgentWatcher, so a DELETE occurring between the fetch and the watch is replayed rather than lost.
3. Replace the events().size() != 1 rejection with a loop over response.events(), enqueueing on any DELETE_ — a prefix rmdir legitimately deletes two keys.
4. Until fixed, this test is a known flake under sanitizers; re-running PR CI: use the in-region EC2 Ubuntu mirror in the AWS EFA pod #2385 should go green, and the sanitizer job should not be treated as a gate for that PR.

Related: PR #1930 (introduced the path), PR #2349 ("test: lengthen the md_manager remote-MD wait under sanitizers" — an earlier timeout-widening band-aid for this same fixture, showing the flake predates this build), PR #303 (original etcd watcher MD fetch/invalidate flow), PR #2385 (the unrelated commit under test).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 0f557488-a204-4590-b131-564120d745df in the triage console for the audit trail.