test(e2e): close platform and workflow parity decisions - #5823
Merged
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds the ChangesCloud onboarding and live e2e support
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related issues
Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Contributor
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in the Show a code coverage summary of the most covered files.
TypeScript / code-coverage/cliThe overall coverage in the Show a code coverage summary of the most covered files.
Updated |
Contributor
Vitest E2E Scenario Results — ❌ Some jobs failedRun: 28203534004
|
Contributor
E2E Advisor RecommendationRequired E2E: Dispatch hint: Full advisor summaryE2E Recommendation AdvisorBase: Required E2E
Optional E2E
New E2E recommendations
Dispatch hint
|
Contributor
Vitest E2E Scenario RecommendationRequired Vitest E2E scenarios: Dispatch required Vitest E2E scenarios:
Full Vitest E2E advisor summaryVitest E2E Scenario AdvisorBase: Required Vitest E2E scenarios
Optional Vitest E2E scenarios
Relevant changed files
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
test/e2e-scenario/live/registry-scenarios.test.ts (1)
142-156: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDrive the experimental checks from the run plan instead of duplicating the paths here.
buildLiveScenarioRunPlan()now owns this onboarding-specific list, but this branch re-declares the same paths again. That makes the planner and executor easy to drift out of sync if a check is added, renamed, or reordered later. Please read the list from the already-built run plan and gate on whether it is present here instead.As per PR objectives, the registry scenario is supposed to consume the live-plan checks list.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/e2e-scenario/live/registry-scenarios.test.ts` around lines 142 - 156, The experimental checks for the cloud-langchain-deepagents-code onboarding are being duplicated in the registry scenario instead of using the live run plan. Update the registry scenario branch in registry-scenarios.test.ts to read the checks from the existing buildLiveScenarioRunPlan() output (the live-plan onboarding-specific list) and use that list for the fs.existsSync validation and runE2eCloudExperimentalChecks call, so the executor stays aligned with the planner and only gates on whether the plan contains checks.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/e2e-scenario/live/gpu-e2e.test.ts`:
- Around line 45-58: The current model selection in gpu-e2e.test.ts is
order-dependent and can silently skip the assertion for non-small-context
models. Update the test logic around the provider/model scan so it explicitly
selects the active Ollama model instead of the first available model, and keep
the small-context expectation tied to that model. In the same area, replace the
early return on contextWindow > 28_000 with a failing assertion so the test
breaks if the scenario stops using a small-context model.
---
Nitpick comments:
In `@test/e2e-scenario/live/registry-scenarios.test.ts`:
- Around line 142-156: The experimental checks for the
cloud-langchain-deepagents-code onboarding are being duplicated in the registry
scenario instead of using the live run plan. Update the registry scenario branch
in registry-scenarios.test.ts to read the checks from the existing
buildLiveScenarioRunPlan() output (the live-plan onboarding-specific list) and
use that list for the fs.existsSync validation and runE2eCloudExperimentalChecks
call, so the executor stays aligned with the planner and only gates on whether
the plan contains checks.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 6387a3fb-49ed-447b-a367-ab0461df0149
📒 Files selected for processing (7)
test/e2e-scenario/fixtures/phases/onboarding.tstest/e2e-scenario/live/gpu-e2e.test.tstest/e2e-scenario/live/registry-scenarios.test.tstest/e2e-scenario/live/run-plan.tstest/e2e-scenario/scenarios/runtime-support.tstest/e2e-scenario/scenarios/scenarios/baseline.tstest/e2e-scenario/support-tests/e2e-scenario-matrix.test.ts
Contributor
PR Review Advisor — Changes requestedMerge posture: Do not merge yet Action checklist
Findings index
Review findings by urgency: 0 required fixes, 2 items to resolve/justify, 0 in-scope improvements
|
Contributor
Vitest E2E Scenario Results — ❌ Some jobs failedRun: 28203681619
|
Contributor
Vitest E2E Scenario Results — ❌ Some jobs failedRun: 28204042928
|
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/e2e/e2e-cloud-experimental/checks/06-deepagents-code-python-egress.sh`:
- Around line 64-65: The blocked-host check in python_probe handling is too
permissive because it ignores probe failures and passes when REACHED: is absent.
Update the e2e script logic around python_probe and the grep check so a
blocked-host test only passes on an explicit positive blocked signal (for
example, a dedicated BLOCKED/denied marker from python_probe), and treat probe
execution failures or missing Python as test failures instead of success.
In `@test/langchain-deepagents-code-image.test.ts`:
- Around line 222-224: Scope the path assertions in the test to the `github` and
`pypi` policy blocks instead of checking the full YAML string, because the
current `expect(policy).toContain(...)` checks can pass due to matching
`managed_inference` entries. Update
`test/langchain-deepagents-code-image.test.ts` around the existing `policy`
assertions to parse or target the specific `github` and `pypi` sections, and
then verify `/usr/bin/python3*`, `/usr/local/bin/python3*`, and
`/usr/local/lib/python3.13/**` only within those blocks so the test fails if the
new binary entries are removed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 55e5bdeb-e46a-41b5-9265-442b1c966b4d
📒 Files selected for processing (3)
agents/langchain-deepagents-code/policy-additions.yamltest/e2e/e2e-cloud-experimental/checks/06-deepagents-code-python-egress.shtest/langchain-deepagents-code-image.test.ts
Contributor
Vitest E2E Scenario Results — ❌ Some jobs failedRun: 28204517774
|
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/e2e/e2e-cloud-experimental/checks/06-deepagents-code-python-egress.sh`:
- Line 35: The egress probe in the sandbox_exec Python snippet is treating every
urllib.request.urlopen exception as a policy block, which can hide successful
connections that return HTTPError or unrelated network failures. Update the
probe logic in the deepagents code egress check so it distinguishes an actual
sandbox denial from other exceptions by handling urlopen outcomes more
specifically and only emitting BLOCKED when the failure clearly indicates
outbound traffic was denied.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: ec782d00-3bbb-474d-965c-16c719f29236
📒 Files selected for processing (2)
test/e2e/e2e-cloud-experimental/checks/06-deepagents-code-python-egress.shtest/langchain-deepagents-code-image.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- test/langchain-deepagents-code-image.test.ts
Contributor
Vitest E2E Scenario Results — ✅ All selected jobs passedRun: 28204874388
|
Contributor
Vitest E2E Scenario Results — ✅ All selected jobs passedRun: 28205441400
|
Contributor
Vitest E2E Scenario Results — ✅ All selected jobs passedRun: 28206026948
|
cv
approved these changes
Jun 26, 2026
# Conflicts: # agents/langchain-deepagents-code/policy-additions.yaml # test/e2e/e2e-cloud-experimental/checks/06-deepagents-code-python-egress.sh # test/langchain-deepagents-code-image.test.ts
Collaborator
|
Addressing the remaining PR Review Advisor warning for interactive OpenClaw TUI first-turn auto-compaction parity:
The previous Deep Agents |
21 tasks
This was referenced Jun 26, 2026
This was referenced Jun 27, 2026
21 tasks
Hadar301
pushed a commit
to Hadar301/NemoClaw-OpenShift
that referenced
this pull request
Jul 12, 2026
## Summary Restore issue NVIDIA#5800 parity package `P0-E` for merged bash-suite deltas only. ## Related Issues Refs NVIDIA#5800 Refs NVIDIA#5098 Refs NVIDIA#5197 Refs NVIDIA#5245 Refs NVIDIA#5508 Refs NVIDIA#5587 ## Scope gate - Package: `P0-E — Platform, tunnel, cloud-experimental, and docs parity decisions` - Included PRs all merged and touched `test/e2e`: yes — NVIDIA#5197, NVIDIA#5245, NVIDIA#5508, NVIDIA#5587 - Out of scope: unmerged/non-bash PRs; shell lane retirement / PR NVIDIA#5756 cleanup ## Parity map | ID | Source PR | Contract | Inference classification | Vitest assertion / waiver | Status | | --- | --- | --- | --- | --- | --- | | E1 | NVIDIA#5197 | Deep Agents Code cloud-experimental Landlock check: `.deepagents` and `/tmp` writable, `/usr` and `/etc` read-only. | `hosted-compatible capable` | `ubuntu-repo-cloud-langchain-deepagents-code` live scenario now runs `05-deepagents-code-landlock-readonly.sh` after onboarding. | covered | | E2 | NVIDIA#5197 | Deep Agents Code arbitrary Python egress permits GitHub/PyPI and blocks Tavily/LangSmith/MCP/example.com absent explicit policy. | `hosted-compatible capable` | `ubuntu-repo-cloud-langchain-deepagents-code` live scenario now runs `06-deepagents-code-python-egress.sh` after onboarding. | covered | | E3 | NVIDIA#5245 | Trace timing signal is trusted summary only; raw target-controlled traces are not uploaded. | `none` | Existing `test/e2e-script-workflow.test.ts` and sanitizer tests cover trusted trace summary, scorecard comparison, and redaction boundary. | covered | | E4 | NVIDIA#5508 | GPU Local Ollama small-context config carries reduced OpenClaw compaction reserve. | `none` / local inference | `test/e2e-scenario/live/gpu-e2e.test.ts` now reads `/sandbox/.openclaw/openclaw.json` and asserts `reserveTokens` / `reserveTokensFloor` for small windows. | covered | | E5 | NVIDIA#5508 | Interactive OpenClaw TUI first turn must not fail auto-compaction. | `none` / local inference | No Vitest TUI fixture exists yet; config-level assertion covers the deterministic contract and the interactive smoke remains platform/manual-runner follow-up. Owner: NemoClaw maintainers; date: 2026-06-25; tracked in NVIDIA#5800 closeout. | waived/follow-up | | E6 | NVIDIA#5587 | Cloudflared exact-version override validates Debian version syntax before APT install. | `hosted-compatible capable` for live tunnel job; resolver itself `none` | Existing `test/cloudflared-version-resolver.test.ts`; tunnel workflow boundary already requires resolver use before `apt-get install cloudflared=${cf_version}`. | covered | ## Inference mode support - Default mode for touched live targets: `hosted-compatible capable` for Deep Agents Code and tunnel lifecycle; local Ollama/GPU uses local inference; trace/resolver tests are `none`. - Real inference support preserved: yes for Deep Agents Code scenario via `NVIDIA_INFERENCE_API_KEY`; yes for tunnel lifecycle via existing hosted-compatible route; local Ollama GPU remains local inference. - Modes validated in this PR: local unit/support tests plus live scenario selection without secret; selective hosted/GPU workflow dispatch required after PR opens. - If not validated with real inference: local machine lacks `NVIDIA_INFERENCE_API_KEY`, GPU runner, and Docker daemon; selective GitHub Actions will validate runner/secret boundaries. ## Validation - [x] `npm run build:cli` - [x] `npx vitest run test/e2e-scenario/support-tests/e2e-scenario-matrix.test.ts test/e2e-scenario/support-tests/e2e-scenario-registry.test.ts test/langchain-deepagents-code-image.test.ts test/ollama-local-openclaw-config-propagation.test.ts test/cloudflared-version-resolver.test.ts --silent=false --reporter=default` - [x] `NEMOCLAW_RUN_E2E_SCENARIOS=1 npx vitest run --project e2e-scenarios-live test/e2e-scenario/live/registry-scenarios.test.ts -t '^ubuntu-repo-cloud-langchain-deepagents-code$' --silent=false --reporter=verbose` (local result: skipped due missing `NVIDIA_INFERENCE_API_KEY`, confirms scenario wiring) - [ ] hosted/public selective E2E workflow, if required by classification: pending PR dispatch ## Follow-ups / waivers - E5: Interactive TUI first-turn smoke remains waived until a Vitest TUI fixture exists. Owner: NemoClaw maintainers; date: 2026-06-25; deterministic config contract is covered here. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a new cloud onboarding path for LangChain Deep Agents Code, including dedicated fixtures and live scenario coverage. * Surfaced new “cloud experimental” checks in live run planning and scenario matrix output. * **Bug Fixes** * Improved GPU live E2E validation by asserting the reserve-token compaction policy. * Updated Deep Agents Code expected gateway health handling to be optional. * **Security/Policy** * Expanded Deep Agents Code Python egress allowlists for approved GitHub/PyPI hosts. * **Tests** * Updated E2E checks and assertions (including Python egress probe behavior) and adjusted scenario/support-matrix expectations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Carlos Villela <cvillela@nvidia.com>
wscurran
added
area: ci
This was referenced Aug 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Restore issue #5800 parity package
P0-Efor merged bash-suite deltas only.Related Issues
Refs #5800
Refs #5098
Refs #5197
Refs #5245
Refs #5508
Refs #5587
Scope gate
P0-E — Platform, tunnel, cloud-experimental, and docs parity decisionstest/e2e: yes — feat(agents): add Deep Agents Code harness #5197, feat(ci): add nightly onboard trace timing summaries #5245, fix(inference): keep first Local Ollama TUI turn under OpenClaw compaction budget (#5468) #5508, test(e2e): migrate test-tunnel-lifecycle.sh to vitest #5587Parity map
.deepagentsand/tmpwritable,/usrand/etcread-only.hosted-compatible capableubuntu-repo-cloud-langchain-deepagents-codelive scenario now runs05-deepagents-code-landlock-readonly.shafter onboarding.hosted-compatible capableubuntu-repo-cloud-langchain-deepagents-codelive scenario now runs06-deepagents-code-python-egress.shafter onboarding.nonetest/e2e-script-workflow.test.tsand sanitizer tests cover trusted trace summary, scorecard comparison, and redaction boundary.none/ local inferencetest/e2e-scenario/live/gpu-e2e.test.tsnow reads/sandbox/.openclaw/openclaw.jsonand assertsreserveTokens/reserveTokensFloorfor small windows.none/ local inferencehosted-compatible capablefor live tunnel job; resolver itselfnonetest/cloudflared-version-resolver.test.ts; tunnel workflow boundary already requires resolver use beforeapt-get install cloudflared=${cf_version}.Inference mode support
hosted-compatible capablefor Deep Agents Code and tunnel lifecycle; local Ollama/GPU uses local inference; trace/resolver tests arenone.NVIDIA_INFERENCE_API_KEY; yes for tunnel lifecycle via existing hosted-compatible route; local Ollama GPU remains local inference.NVIDIA_INFERENCE_API_KEY, GPU runner, and Docker daemon; selective GitHub Actions will validate runner/secret boundaries.Validation
npm run build:clinpx vitest run test/e2e-scenario/support-tests/e2e-scenario-matrix.test.ts test/e2e-scenario/support-tests/e2e-scenario-registry.test.ts test/langchain-deepagents-code-image.test.ts test/ollama-local-openclaw-config-propagation.test.ts test/cloudflared-version-resolver.test.ts --silent=false --reporter=defaultNEMOCLAW_RUN_E2E_SCENARIOS=1 npx vitest run --project e2e-scenarios-live test/e2e-scenario/live/registry-scenarios.test.ts -t '^ubuntu-repo-cloud-langchain-deepagents-code$' --silent=false --reporter=verbose(local result: skipped due missingNVIDIA_INFERENCE_API_KEY, confirms scenario wiring)Follow-ups / waivers
Summary by CodeRabbit