Skip to content

fix(e2e): fix 3 stale assertions in newly-wired nightly E2E tests #2656

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cv merged 5 commits into from
Apr 29, 2026
Merged

fix(e2e): fix 3 stale assertions in newly-wired nightly E2E tests #2656

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c6edcfc
fix(e2e): update stale gateway-reuse grep in double-onboard test
jyaunches Apr 29, 2026
2b51511
fix(e2e): assert gateway reuse via container ID instead of log grep
jyaunches Apr 29, 2026
9ad4edd
fix(e2e): simulate interrupted state instead of relying on policy-mod…
jyaunches Apr 29, 2026
9fb78d1
fix(e2e): same interrupted-state fix for onboard-resume test
jyaunches Apr 29, 2026
1019da9
merge(main): resolve conflicts for PR 2656
cv Apr 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 10 additions & 6 deletions test/e2e/test-double-onboard.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,7 @@ fi
section "Phase 3: Second onboard ($SANDBOX_A — same name, recreate)"
info "Running nemoclaw onboard with NEMOCLAW_RECREATE_SANDBOX=1..."

GATEWAY_ID_BEFORE=$(docker ps -qf "name=openshell-cluster-nemoclaw" | head -1)

@coderabbitai coderabbitai Bot Apr 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Broad filter (may include partial matches):"
docker ps --format '{{.ID}}\t{{.Names}}' --filter 'name=openshell-cluster-nemoclaw'

echo
echo "Exact filter (only exact container name):"
docker ps --format '{{.ID}}\t{{.Names}}' --filter 'name=^/openshell-cluster-nemoclaw$'

Repository: NVIDIA/NemoClaw

Length of output: 168

🏁 Script executed:

# First, check if the file exists and read the relevant sections
wc -l test/e2e/test-double-onboard.sh

Repository: NVIDIA/NemoClaw

Length of output: 95

🏁 Script executed:

# Read lines around 277 (the first occurrence mentioned)
sed -n '270,285p' test/e2e/test-double-onboard.sh

Repository: NVIDIA/NemoClaw

Length of output: 631

🏁 Script executed:

# Read lines around 320 and 332 (other occurrences)
sed -n '315,340p' test/e2e/test-double-onboard.sh

Repository: NVIDIA/NemoClaw

Length of output: 1057

🏁 Script executed:

# Check for SPDX header and shebang at the start of the file
head -5 test/e2e/test-double-onboard.sh

Repository: NVIDIA/NemoClaw

Length of output: 248

🏁 Script executed:

# Check if file is executable
ls -la test/e2e/test-double-onboard.sh

Repository: NVIDIA/NemoClaw

Length of output: 141

🌐 Web query:

Docker ps filter name exact match vs partial match documentation

💡 Result:

The docker ps (or docker container ls) command with --filter name=VALUE (or -f name=VALUE) performs a partial substring match by default, not an exact match. This is explicitly stated in the official Docker documentation. For partial match: - docker ps --filter "name=nostalgic" matches containers like i_am_nostalgic, nostalgic_stallman, and nostalgic_shockley. For exact match: - Use regex anchors: docker ps --filter "name=^exactname$" to match only containers named precisely "exactname". The ^ anchors to the start and $ to the end of the name string. This behavior is consistent across Docker CLI reference pages and manpages. Note that container names are unique, but the filter matches substrings unless regex is used. There is no built-in exact-match flag without regex; workarounds like piping to grep exist but are not native to the filter.

Citations:

🏁 Script executed:

# Check lines 289-294
sed -n '285,300p' test/e2e/test-double-onboard.sh

Repository: NVIDIA/NemoClaw

Length of output: 646

🏁 Script executed:

# Find all instances of the problematic docker ps command in the file
rg 'docker ps.*openshell-cluster-nemoclaw' test/e2e/test-double-onboard.sh -n

Repository: NVIDIA/NemoClaw

Length of output: 391

Use exact Docker name filtering to prevent nondeterministic container ID selection.

The docker ps -qf "name=openshell-cluster-nemoclaw" filter on lines 277, 289, 320, and 332 performs partial substring matching by default. Combined with | head -1, this can select the wrong container if similarly named containers exist (e.g., "openshell-cluster-nemoclaw-backup"), causing flaky assertions.

Use regex anchors to match the exact container name:

Proposed fix 
+gateway_container_id() {
+  docker ps --filter "name=^openshell-cluster-nemoclaw$" --format '{{.ID}}'
+}
+
-GATEWAY_ID_BEFORE=$(docker ps -qf "name=openshell-cluster-nemoclaw" | head -1)
+GATEWAY_ID_BEFORE="$(gateway_container_id)"
@@
-GATEWAY_ID_AFTER=$(docker ps -qf "name=openshell-cluster-nemoclaw" | head -1)
+GATEWAY_ID_AFTER="$(gateway_container_id)"
@@
-GATEWAY_ID_BEFORE3=$(docker ps -qf "name=openshell-cluster-nemoclaw" | head -1)
+GATEWAY_ID_BEFORE3="$(gateway_container_id)"
@@
-GATEWAY_ID_AFTER3=$(docker ps -qf "name=openshell-cluster-nemoclaw" | head -1)
+GATEWAY_ID_AFTER3="$(gateway_container_id)"
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@test/e2e/test-double-onboard.sh` at line 277, The docker container name
filter uses substring matching which can return the wrong container; update the
docker ps calls that use docker ps -qf "name=openshell-cluster-nemoclaw" (e.g.,
the assignment to GATEWAY_ID_BEFORE and the other three occurrences) to use an
exact-name regex anchor: docker ps -qf "name=^/openshell-cluster-nemoclaw$" so
the filter matches the exact container name only.

PHASE3_START="$(phase_start_time)"
run_onboard "$SANDBOX_A" "1"
output2="$RUN_ONBOARD_OUTPUT"
Expand All @@ -324,10 +325,11 @@ else
dump_diagnostics "Phase 3"
fi

if grep -q "Reusing healthy NemoClaw gateway" <<<"$output2"; then
pass "Healthy gateway reused on second onboard"
GATEWAY_ID_AFTER=$(docker ps -qf "name=openshell-cluster-nemoclaw" | head -1)
if [ -n "$GATEWAY_ID_BEFORE" ] && [ "$GATEWAY_ID_BEFORE" = "$GATEWAY_ID_AFTER" ]; then
pass "Healthy gateway reused on second onboard (container $GATEWAY_ID_BEFORE)"
else
fail "Healthy gateway was not reused on second onboard"
fail "Gateway container changed on second onboard (before=$GATEWAY_ID_BEFORE after=$GATEWAY_ID_AFTER)"
fi

if grep -q "Port 8080 is not available" <<<"$output2"; then
Expand All @@ -354,6 +356,7 @@ fi
section "Phase 4: Third onboard ($SANDBOX_B — different name)"
info "Running nemoclaw onboard with new sandbox name..."

GATEWAY_ID_BEFORE3=$(docker ps -qf "name=openshell-cluster-nemoclaw" | head -1)
PHASE4_START="$(phase_start_time)"
run_onboard "$SANDBOX_B"
output3="$RUN_ONBOARD_OUTPUT"
Expand All @@ -370,10 +373,11 @@ else
dump_diagnostics "Phase 4"
fi

if grep -q "Reusing healthy NemoClaw gateway" <<<"$output3"; then
pass "Healthy gateway reused on third onboard"
GATEWAY_ID_AFTER3=$(docker ps -qf "name=openshell-cluster-nemoclaw" | head -1)
if [ -n "$GATEWAY_ID_BEFORE3" ] && [ "$GATEWAY_ID_BEFORE3" = "$GATEWAY_ID_AFTER3" ]; then
pass "Healthy gateway reused on third onboard (container $GATEWAY_ID_BEFORE3)"
else
fail "Healthy gateway was not reused on third onboard"
fail "Gateway container changed on third onboard (before=$GATEWAY_ID_BEFORE3 after=$GATEWAY_ID_AFTER3)"
fi

if grep -q "Port 8080 is not available" <<<"$output3"; then
Expand Down
Loading