fix(e2e): reconcile credential generation window state - #10391
Closed
sandl99 wants to merge 64 commits into
Closed
Conversation
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
…or-image-changes # Conflicts: # test/platform/images/vllm-docker-storage.test.ts
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
sandl99
changed the base branch from
fix/e2e-local-build-for-image-changes
to
main
August 26, 2026 14:39
Signed-off-by: San Dang <sdang@nvidia.com>
20 tasks
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com> # Conflicts: # .github/workflows/e2e.yaml
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Contributor
|
PR review advisory complete for commit |
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Signed-off-by: San Dang <sdang@nvidia.com>
Collaborator
Author
|
Closing this draft as obsolete for the active #10155 failure queue. Current main commit f15c80d from #10113 passes openshell-credential-generation-window: main job. The focused run for PR commit 8d9508f also passed: focused PR run. The bounded diagnostic and provider-detach state checks remain useful as future E2E hardening, but they are not required to fix the current main failure. Any retained hardening should return as a focused change from current main. |
sandl99
pushed a commit
that referenced
this pull request
Aug 27, 2026
… 0.0.106 (#10273) ## Summary A sandbox reads its provider environment once, at boot, and the agent process inherits that read for the life of the container. Any channel credential that only becomes injectable after boot therefore never reaches the running agent, and no restart recovers it — only recreating the sandbox does. This change makes every messaging credential injectable before the agent starts, and stops the agent config from shadowing the injected value once it arrives. ## Related Issue Part of #10079. It does not close that issue: WeChat and Teams on Hermes are untouched here and are described below. ## Changes - **Bind the credential in the policy preset and apply that preset at boot.** The provider profiles are endpointless, so the binding is the only thing that makes the token injectable, and `requiredAtCreate` is what puts the preset in the boot policy rather than a post-boot apply. Without both, OpenShell withholds the credential entirely (`withholding static provider credential handle from endpointless profile`). Bindings this PR adds: - Telegram — both agents. - Teams — OpenClaw. - Slack — OpenClaw; the Hermes side landed on `main` as #10271. Discord already carried the binding on both agents before this branch. - **Pass the sandbox name through both policy preflights.** Channel presets bind `{sandboxName}-<channel>-bridge`, so composing one without a sandbox name throws. Two paths dropped the name after resolving it: - `preflightPolicyRequirements` resolves it for the sandbox inspection. - `prepareSandboxCreatePolicy` has it on the create intent, and is the path the external-authority onboarding flow takes. #10314 fixed the sibling site inside `materializeSandboxCreatePlan`; these two were still uncovered. Four tests composed presets directly and mirrored the old shape, which let the composition error escape the test body and kill a whole vitest shard. - **Stop persisting the canonical placeholder in agent config.** OpenShell 0.0.106 refuses the canonical form once a credential is identity-bound, so the shape that used to work is now the one shape the credential endpoint rejects. Removed: - OpenClaw config — `botToken` for Telegram, `botToken` and `appToken` for Slack, `appPassword` for Teams. - Hermes `~/.hermes/.env` — the Telegram, Slack, and Discord token lines. - The Slack manifest's legacy `slackRuntimeEnvAliases` normalization, which existed only to rewrite those placeholders. Each agent now reads the key from its process environment, which OpenShell fills with the revision-scoped placeholder at boot. - **Prune stale credential keys from the Hermes env file.** Hermes loads `~/.hermes/.env` with `override=True`, so a leftover canonical placeholder from an earlier onboarding shadows the injected process value and the channel stays unauthenticated. Four gaps kept that line alive: - Cleanup lived only in `applyAgentConfigAtOpenShell`, whose sole production caller returns early for any non-OpenClaw plan. The Hermes runtime applier merged env lines and never removed any. - `readEnvLineKey` read `export KEY` as the key, so an export-prefixed assignment matched nothing. - Deletion keys came from the persisted plan, so a binding naming an unrelated key could remove an operator-owned line. - A plan encoded before the credential moved to a policy binding still carries the token in `agentRender`, and rebuild refreshes only host forwards and runtime setup, so the render reintroduced the line the cleanup had just removed. The rules now live in one module both appliers use: read the key from either assignment form, take deletion authority from the channel manifest rather than persisted state, treat a rendered key as wanted only while the manifests still assign a credential to it, and visit an owned target even when the plan renders nothing into it. WeChat and Teams render their Hermes credential under a different key than the provider env key, so the assignment metadata, not the provider key, decides what survives. Each rule was checked by removing it and confirming the new tests fail. - **Wait for the first gateway mint before creating the sandbox.** `provider refresh configure` returns while the credential is still the create-time sentinel and the refresh worker mints on its own sweep, so the sandbox was booting inside that window and pinning a revision whose value is the sentinel. The poll itself accepted any status table it could parse and counted attempts only, so two failure modes also passed through: - A nonzero `provider refresh status` can still print a stale `refreshed` row, which was read as success. - Attempts do not bound the wait; one probe with no timeout can hang and the loop never reaches its cap. It now requires exit status 0 before trusting a row, gives each probe a command timeout, and stops at an overall deadline. Current requirement and consumer: Google Chat, the only channel with a gateway-minted credential. Failing closed stays correct: creating the sandbox before the first mint pins the create-time sentinel for the life of the container. The `configureMessagingBridgeRefreshes` tests cover the success and the never-minted path, and the optional `sleep` dependency is a test injection point, not a configuration surface. - **Make the Google Chat outbound preload forward the injected placeholder verbatim.** Rewriting it to the canonical form produced `credential_unavailable` on every send. - **Keep preserved Hermes env lines anchored to an enabled channel.** They were dropped whenever no enabled channel happened to render a `~/.hermes/.env` entry — which is now the common case, since the token lines are gone. - **Add two drift guards over the real policy files.** A preset that declares `credential_binding` must be `requiredAtCreate`, and a host and port declared twice must carry distinct path selectors. Each guard was checked by reintroducing the defect and confirming it fails. - **Align the Discord render assertion added by #10277.** That PR fixed the OpenClaw half; the Hermes Discord policy already bound every endpoint to `{sandboxName}-discord-bridge`, so rendering the canonical placeholder into `~/.hermes/.env` wrote the one shape the credential endpoint refuses. - **Refresh the reviewed managed-startup bundle.** `managed-startup-image-runtime.bundle` embeds the channel manifests, so the manifest changes above made `bundle:reviewed:check` fail in `static-checks`. Regenerated from the merged tree; the delta is 8 blocks, all of them the credential renders removed above plus the two `requiredAtCreate` flags. Three overlapping fixes landed on `main` while this PR was open and are merged in here: #10271 (the Hermes Slack `path` selector), #10277 (the OpenClaw half of Discord), and #10314 (binding the Discord create-path providers). This branch keeps only an explanatory comment on `slack/policy/hermes.yaml`; the behavior there is main's. #10314 fixed the `materializeSandboxCreatePlan` call site; the two preflight call sites it left uncovered are fixed here. ## Channel coverage after this change | Channel | OpenClaw | Hermes | Status | |---|---|---|---| | Slack | fixed | fixed | live, bot replied — Hermes policy selector landed separately as #10271 | | Discord | fixed | fixed | live, bot replied — OpenClaw half landed separately as #10277 | | Google Chat | fixed | fixed | live, bot replied | | Telegram | fixed | fixed | live, bot replied on both | | Teams | fixed | not covered | withholding log observed, no live run | | WeChat | not covered | not covered | not measured | | WhatsApp | unaffected | unaffected | injects no provider credential (QR pairing) | Every `fixed` row except Teams was confirmed by an actual bot reply on a freshly wiped host, not by test output alone. For Telegram, both agents were run against OpenShell 0.0.106: each sandbox booted with the revision-scoped placeholder in its agent process, the policy matched the redacted `/bot[CREDENTIAL]/` path, and the bot answered — with no denial and no credential error across five hours of OpenClaw polling and twenty minutes of Hermes polling. Out of scope here: - **WeChat** — injects a provider credential with no endpoints on the profile and no `credential_binding`. Telegram's shape, so the same withholding is expected, but it was not measured, so it is not claimed. - **Teams on Hermes** — Hermes reads `TEAMS_CLIENT_SECRET`, the provider injects `MSTEAMS_APP_PASSWORD`. A name mismatch, not the ordering defect. ## Known gaps, deliberately out of scope - **Ready-sandbox reuse does not migrate messaging config.** Both reuse branches in `sandbox-create/orchestration.ts` revalidate policy, seed presets, upsert providers, restore the dashboard, and return. A sandbox that booted without the injected provider environment cannot be repaired by pruning `~/.hermes/.env` — it needs a recreate decision in the existing drift guard beside `credentialRotation.changed`, which is a new drift signal rather than a cleanup change. Nearest coverage: the create and rebuild paths this PR fixes. - **`remove-channel` on a legacy plan leaves that channel's placeholder line behind.** `removePlanChannel()` drops the credential binding and the render together, so cleanup has no ownership evidence for the key. The residue is a placeholder rather than a credential, is inert once the provider is removed, and is pruned if the channel is added again. ## Type of Change - [x] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: outstanding; this change touches messaging credentials, network policy presets, and the onboarding provider path, so it needs a maintainer sensitive-path review before merge. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: four checks are red on this branch and none of them is reachable from it. `CLI` fails its coverage gate on `src/lib/policy/commands.ts` at 88.88% against the 100% threshold that #9511 declares for `src/lib/policy/{commands,merge}.ts`, and `Required Checks` fails only because `CLI` does. `PR / Agent runtimes / Test activation` and both `PR / OpenClaw / MCP Discovery` runs fail on the same assertion, `Sandbox policy authority validation failed after creation`, in `managed-image-activation-e2e.test.ts` and `mcp-bridge.test.ts`. All four were red on #10332's own PR run before it merged, with a byte-identical coverage error, and #10332 both rewrote `src/lib/policy/commands.ts` and added its `commands.test.ts`. Bucketing open PRs by base confirms the boundary: `ac3ebe9aa` (#10384, the direct parent of #10332) passes those checks, while `1293457d3` (#10332 itself, #10392), `1effafb3f` (#10391), and `6062006e6` (this PR, #10397) all fail. This branch changes nothing under `src/lib/policy/`, and the failing image runs configure no messaging channel, so no preset from this PR is composed on that path. ## DGX Station Hardware Evidence Not applicable — `scripts/prepare-dgx-station-host.sh` is unchanged. - [ ] Tested on DGX Station - Tested commit: - Station profile/scenario: - Result: - Supporting evidence: ## Verification - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run validate:pr` passed after refreshing `origin/main` when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: `npx vitest run --project cli src/lib/messaging src/lib/onboard/sandbox-create-plan.test.ts src/lib/onboard/messaging-bridge-provider.test.ts src/lib/onboard/policy-authority/preflight.test.ts src/lib/actions/sandbox/policy-channel-remove-flow.test.ts` — 69 files, 785 pass; `npx vitest run --project integration test/runtime/messaging test/runtime/policy test/generation test/channels/channels-add-bridge-lifecycle.test.ts test/onboard-external-policy-authority-composition.test.ts` — 77 files, 1359 pass, and 6 failures in `whatsapp-qr-compact.test.ts` that come from `qrcode` not being installed on this host; `npm run typecheck:cli`, `npm --prefix nemoclaw run typecheck`, `npm run checks:repository`, and `npm --prefix tools/mcp-tool-discovery-runtime run bundle:reviewed:check` all pass. CI confirms the branch itself: all 12 `CLI / Shard` jobs, `Static Checks`, `Build and type-check`, `Installer Integration`, and `Plugin` pass on the merged head. - [ ] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — command/result: not applicable; this changes messaging manifests, policy presets, and one onboarding step, not the runtime, the test harness, or repo-wide validation. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- Signed-off-by: Hung Le <hple@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Security & Reliability** * Messaging credentials are injected at runtime instead of written to configuration files. * Stale credential entries are removed while unrelated environment settings are preserved. * Google Chat authentication supports revision-scoped credentials and dynamic refresh. * **Messaging Channels** * Updated Telegram, Teams, Slack, Discord, and Google Chat credential handling. * Slack access distinguishes Socket Mode from Web API traffic. * Added credential-bound network policies for Telegram and Teams. * **Onboarding** * Credential setup now waits for successful token issuance and reports clear failures. * Channel policies support sandbox-specific credential providers. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com> Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> Co-authored-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com> Co-authored-by: Prekshi Vyas <34834085+prekshivyas@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
wscurran
added
area: e2e
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make the credential-generation window E2E report bounded request-failure evidence and judge provider detach by the resulting attachment state. Stack the change on #10263 so exact-revision E2E selects the workload source from the commit tree and builds changed managed-image inputs locally.
Related Issue
Addresses the credential-generation window failures in #10155.
Depends on #10263 for exact-PR workload-source selection.
Changes
PREMATURE_CLOSE, then emit the acknowledgement and final summary instead of waiting indefinitely.Type of Change
Quality Gates
DGX Station Hardware Evidence
Verification
Signed-off-by:line and every commit appears asVerifiedin GitHubpre-commit,commit-msg, andpre-pushhooks passednpx vitest run --project integration test/credentials/openshell-credential-generation-window.test.ts— 7 passednpx vitest run --project e2e-support --maxWorkers=1 test/e2e/support/mcp-bridge-hermes-lifecycle.test.tswith repository-pinned PyYAML 6.0.3 — 10 passednpm run test:e2e-phases:check— 133 tests across 88 filesnpm run typecheck:clinpm run checks:repositorynpm testfor broad runtime/test-harness changes;npm run checkfor repo-wide validation/coverage changes — command/result:npm run docsbuilds without warnings (doc changes only)Signed-off-by: San Dang sdang@nvidia.com