Skip to content

v2 egress: default-allow mitmproxy gateway + account-level managed deploy keys + auth-token/provider reconcile - #345

Merged
kingsleydon merged 43 commits into
mainfrom
integrate/v2-cli-runtime-fixes
Jul 9, 2026
Merged

v2 egress: default-allow mitmproxy gateway + account-level managed deploy keys + auth-token/provider reconcile#345
kingsleydon merged 43 commits into
mainfrom
integrate/v2-cli-runtime-fixes

Conversation

@kingsleydon

Copy link
Copy Markdown
Contributor

Summary

CLI-side of the v2 hosted Invisible Gateway launch batch, plus the clean auth-token lifecycle and provider-projection reconcile. Pairs with the clawdi-hosted PR (invisible gateway image + single-key mint + launch alerting) — merge coordinated.

What's in it

  • Invisible Gateway (transparent MITM)invisible-gateway.ts (nftables meta skuid REDIRECT, default-DROP fail-closed, atomic nft -f, ct-mark matched redirect), transparent sidecar (mitm-sidecar/main.go: SO_ORIGINAL_DST, upstream TLS verified, secret by-ref, no key logging). Removes proxy-env from the hosted path (local clawdi run keeps its per-run forward proxy). Managed key is sidecar-only (agent env gets a placeholder; real key never in agent env).
  • Clean auth-token lifecycle — single canonical <runRoot>/secrets/auth-token, env-first-then-file; watch unit bound to the manifest URL from controlPlane.manifestUrl. source.auth retained (tolerated) for CLI self-upgrade version-skew.
  • Provider projection reconcile — switching provider now removes the stale (managed/BYOK) projected entry from OpenClaw/Hermes config; user-authored entries preserved.
  • nft table name / UID centralized (no hardcoded 10001); net.isIP for address validation.

Validation

  • bun run check (biome ci, 598 files): clean
  • bun run typecheck (turbo, all packages): 4/4 pass
  • bun test runtime + invisible-gateway + mitm-env + mitm-profiles + ai-provider + run: 201 pass / 0 fail
  • Adversarial security review: found + fixed 1 critical (transparent-port bypass), 2 high (boot fail-open ordering, flush ruleset scope), 1 test-bug; re-validated on dev CVM (direct_leaks=0 across bypass / sidecar-restart / CLI-self-upgrade)
  • Cleanliness review: 6 real items fixed (dead proxy path scoped, stale config, nft dedup, scrub list, exception convention, IP parsing)
  • Gateway dev-CVM e2e (pre-integration): both runtimes, managed AI intercepted + billed, fail-closed, Hermes self-managed uv/python under /home/clawdi

In-flight (not blocking review)

Full integrated-code prod-shape e2e on dev CVM is running (BYOK rebind resolution + both-runtime function matrix + Telegram pairing). Recommend merge after that e2e is green.

Owner-decided: ONE deploy key per compute (single-key), aligns with the single canonical CLAWDI_AUTH_TOKEN.

🤖 Generated with Claude Code

kingsleydon and others added 23 commits July 8, 2026 08:44
Keep the smart runtimeManifestUrlEnv() (controlPlane.manifestUrl -> sourcePath
-> env) from the auth-token fix; drop the gateway branch's env-only duplicate
that shadowed it and tripped TS1117.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 8, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
clawdi Skipped Skipped Jul 9, 2026 6:40am

Request Review

@kingsleydon

Copy link
Copy Markdown
Contributor Author

Paired hosted PR: Clawdi-AI/clawdi-hosted#722 (invisible gateway image + single-key mint + launch alerting). Merge coordinated — deploy image + CLI together, after the integrated-code dev-CVM e2e is green.

@kingsleydon

Copy link
Copy Markdown
Contributor Author

Self code-review (final integrated form) — no blocking findings

Reviewed the merged diff (not just sub-branches), risk-weighted on the security core:

invisible-gateway.ts — fail-closed verified: output_filter policy accept but a terminal meta skuid <agent> counter drop catches every non-allowed agent packet (transparentPort-direct, QUIC/UDP≠53, ICMP, arbitrary ports → dropped). Redirect matched by conntrack mark (ct mark) + oifname lo, so the old transparent-port bypass is closed. Same-table delete table + add table in one nft -f = atomic replace, no gap on re-converge. Full input validation (uid/port/nft-identifier/net.isIP).

egress boot (clawdi-runtime-egress-common) — early-drop guard and the full gateway use the SAME table name → the CLI's full apply atomically upgrades the boot drop-all-agent table into the full ruleset. No coexistence (no double-drop of redirected traffic), no window: fail-closed from t=0 through converge. UID derived from id -u, single source /etc/clawdi/invisible-gateway.env.

Prior review coverage (holds in final form): managed key sidecar-only (root:600, out of agent env — placeholder only), sidecar verifies upstream TLS + never logs the key, proxy-env removed from the hosted path (local clawdi run keeps its own), single deploy key per compute, no-data-no-patch-audit asserts agent can't read the MITM secret.

Verification: biome ci (598 files) clean; turbo typecheck 4/4; 201 CLI tests; adversarial security review (1 critical + 2 high fixed, direct_leaks=0 on dev CVM); cleanliness review (6 items fixed).

Residual (non-blocking): DNS→resolver is an accepted+documented exfil path (fast-follow allowlist for untrusted signup).

Verdict: approve — merge after the in-flight integrated-code dev-CVM e2e is green (BYOK rebind + both-runtime matrix + Telegram pairing), coordinated with clawdi-hosted#722.

@kingsleydon kingsleydon changed the title v2 hosted: invisible gateway CLI + clean auth-token + provider reconcile v2 egress: default-allow mitmproxy gateway + account-level managed deploy keys + auth-token/provider reconcile Jul 9, 2026
@kingsleydon

Copy link
Copy Markdown
Contributor Author

Reworked: fail-closed invisible gateway → default-allow mitmproxy (validated end-to-end)

The egress interception was over-engineered (nftables fail-closed default-DROP + hand-rolled Go MITM sidecar). Owner redesigned it to default-allow + profile-driven transparent MITM via runtime-fetched mitmproxy, plus a corrected deploy-key model.

What changed here (clawdi = CLI + cloud-api):

  • Deleted the Go MITM sidecar + fail-closed nftables. New: generic Python mitmproxy addon (default-allow; provider host-match Authorization overwrite; http channel placeholder→rewrite+inject; passthrough unmatched), minimal nft redirect (no drop), runtime-fetched pinned+SHA256-verified mitmdump from downloads.mitmproxy.org (NOT baked in image), single-source config.
  • cloud-api: managed platform API keys — the deploy/cloud-sync key is now account-level (environment_id=None) + managed=true: hidden from user GET /keys, user DELETE → 403, admin-only revoke. Manifest/managed-AI resolves by account (user_id) + manifest environment_id query, not key env-binding.
  • Retained: clean auth-token lifecycle, provider projection reconcile.

Validated (full live e2e, correct harness — local cloud-api + cloudflared tunnel, dev CVM runs only the agent pod, managed AI → faraday TEST): managed AI OpenClaw+Hermes (Faraday usage 301/303), channels {OpenClaw,Hermes}×{Telegram,Discord} + Telegram+Discord concurrent, Discord WSS passthrough, default-allow (real upstream cert), BYOK passthrough, security (secrets absent from agent env; sidecar secret+CA unreadable by clawdi; logs redacted), degraded (kill mitmdump → default-allow, no billing bypass), restart resume, CLI self-upgrade reconcile, account-level managed key (managed=true env_id=NULL; user can't list/revoke; admin can), perf ~50MB/s. All PASS.
Full test suites: CLI 596 + cloud-api 120 + hosted 91 + addon 9, biome/typecheck/ruff green.

🤖 Generated with Claude Code

The account-level managed deploy-key work added an environment_id query
param to list_channels + the managed key field; regenerate the committed
TS client so the generated-api CI gate + downstream typecheck pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Go MITM sidecar was deleted; remove the setup-go / go test / sidecar
bundle build + verify steps from client-ci and cli-publish so the CLI test
+ publish jobs stop referencing packages/cli/native/mitm-sidecar/go.mod.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
kingsleydon and others added 3 commits July 9, 2026 06:12
The 2026-07-08 transport research doc still described the deleted Go-sidecar +
forward-proxy-env model (9 dead file:line cites to native/mitm-sidecar/*). Mark
it SUPERSEDED with a banner pointing at the current source of truth (mitmproxy
addon + transparent-mitm.ts) and fix managed-runtime.md's stale native-sidecar
row + proxy-env paragraph to the default-allow transparent nft-redirect model
(CA-trust env only, no proxy env injection).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
clawdi's doc convention is plain kebab-case, no date in the filename (designs/,
plans/, scenarios/, top-level arch docs; ADRs use a numeric prefix). Three docs
used a trailing date (one also ALL-CAPS) — the anomaly. Drop the date suffix +
lowercase; switch the transport doc banner to the guide's documented HISTORICAL
form (agent-docs-guide.md). No inbound links referenced the old names.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 'keeps hosted managed provider key out of the agent env' test predated the
managed_by classifier (isClawdiManagedProviderProjection = managed_by ===
'clawdi', emitted by cloud-api as n:'clawdi'). Its inline provider omitted the
field, so the fixture was classified BYOK instead of managed and never exercised
the placeholder path — OPENAI_API_KEY came out undefined. Add managed_by:'clawdi'
so the fixture matches the real cloud-api contract: the agent env gets the
placeholder while the real key stays out. Prior CI runs on this branch were
cancelled mid-flight, so this real failure only surfaced on the first
uninterrupted run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kingsleydon
kingsleydon merged commit 6493775 into main Jul 9, 2026
12 checks passed
@kingsleydon
kingsleydon deleted the integrate/v2-cli-runtime-fixes branch July 9, 2026 06:49
@kingsleydon kingsleydon mentioned this pull request Jul 9, 2026
kingsleydon added a commit that referenced this pull request Jul 9, 2026
Ships the default-allow transparent mitmproxy egress gateway + account-level
managed provider key handling merged in #345. beta.37 predates that merge, so
v2 hosted runtimes tracking the clawdi@beta dist-tag need a new beta to pick up
the gateway at runtime (the image stays stable; the CLI is runtime-installed).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant