diff --git a/.github/workflows/e2e-install.yml b/.github/workflows/e2e-install.yml index a279d5f90..37b9b9c53 100644 --- a/.github/workflows/e2e-install.yml +++ b/.github/workflows/e2e-install.yml @@ -50,6 +50,25 @@ jobs: # Budget: native arm64 finishes in 10-15 min; qemu is 30-40 min; give # headroom for docker pulls on cold runners. timeout-minutes: 90 + # Where the secrets live. ONE job on purpose: a scheduled/manual job of + # its own would repeat the fifteen steps below, and a caller job that + # `uses:` a reusable workflow cannot set an Environment (only the called + # workflow's own jobs can, and on a pull_request the PR head supplies that + # file too) — so the one job names its Environment by event instead. A pull_request run + # references `e2e-pull-request`, which holds no secrets and needs no + # protection (the job checks out and runs the PR head's own config and + # specs; see "Write .env.test" below). Every other run — the schedule, a + # dispatch — references `e2e-credentials`, whose deployment-branch policy + # (beta and main only) is what fences the secrets: a run from any other + # ref is refused before a step starts, whatever this file says, because + # on a pull_request the PR head supplies this file too. The owner + # configures both under Settings → Environments: create + # `e2e-pull-request` empty; create `e2e-credentials`, put the provider + # keys and TELEGRAM_BOT_TOKEN in it, restrict its deployment branches to + # beta and main, then DELETE the repository-level copies — with those + # still there, a same-repo PR that edits the step-level `if:` below reads + # them anyway, and that `if:` is hygiene, not the fence. + environment: ${{ github.event_name == 'pull_request' && 'e2e-pull-request' || 'e2e-credentials' }} env: CLAWBOX_PORT: "8080" # Default target for the in-app updater test. On PRs targeted at this @@ -98,12 +117,25 @@ jobs: platforms: arm64 # Real AI-provider credentials are injected via repo secrets. The test - # steps that need them skip gracefully when the env var is empty, so + # steps that need them skip gracefully when the file is absent, so # forks without secrets still get meaningful coverage. Values are # passed through env vars and written via printf so a secret that # happens to contain the EOF token or other heredoc-breaking bytes # can't truncate the resulting .env file. + # + # Never on a pull_request run: this job checks out the PR HEAD and then + # runs its playwright.config.ts, global-setup.ts and specs on the host + # with this file beside them, so a same-repo PR branch could read the + # file and send it anywhere. A same-repo PR now behaves exactly like a + # fork PR; the credentialed specs run on the schedule and on dispatch. + # (A gate in this file is only hygiene — the PR head supplies the + # workflow too — which is why these secrets also belong in a GitHub + # Environment restricted to beta/main, not at repository scope.) + # Nothing reads CLAWBOX_UPGRADE_TARGET_BRANCH from the file — the + # updater spec reads process.env, set by the job-level env above — so + # the whole step is skipped rather than split. - name: Write .env.test + if: ${{ github.event_name != 'pull_request' }} env: ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} diff --git a/CLAUDE.md b/CLAUDE.md index 795dde97c..2f5808e42 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -25,6 +25,7 @@ Bun runtime (package management + builds), Node.js 22 (production runtime), Next - **Unit tests**: Vitest (`vitest.config.ts`, `vitest.workspace.ts`) — tests in `src/tests/` - **E2E tests**: Playwright (`playwright.config.ts`) - Test coverage for: config store, network utils, auth, OAuth, system info, updater, gateway proxy, middleware, API routes +- **CI**: `.github/workflows/e2e-install.yml` writes repository secrets into `e2e-install/.env.test` only when the event is NOT a `pull_request` (a same-repo PR head supplies the Playwright code that reads that file — and the workflow file itself, so the `if:` is hygiene and the real fence is the owner keeping those secrets in a GitHub Environment restricted to beta/main); same-repo PRs therefore behave like fork PRs and the credentialed specs skip. `scripts/issue-triage.mjs` and `scripts/pr-review.mjs` validate the model's JSON against their SCHEMA on BOTH transports (`assertMatchesSchema` in `scripts/lib/triage-output.mjs`: enum/type violations throw loudly, unknown keys are dropped, free text is truncated), derive every label from fixed tables, and pass free text through `plain()` before it reaches a bot comment — the `issues` workflow runs main's copy, so the fix is inert until it reaches main. ## Architecture @@ -45,17 +46,17 @@ Next.js rewrites in `next.config.ts` proxy gateway paths (`/api/*`, `/assets/*`, - **Memory embedder**: `embed/status`, `embed/install`, and the proxy `local-ai/embed/v1/*` — the memory-search embedding model (Qwen3-Embedding-0.6B, Q8_0) on ClawBox's own llama.cpp, run as the SYSTEM unit `clawbox-embed.service` (`config/clawbox-embed.service`, `scripts/start-embed-server.sh`, numbers in `src/lib/embed-runtime-ids.ts`, pinned by `embed-server-pin.test.ts`). It replaced the copy ollama hosted, which measured 2.8 GB resident whenever the agent was in use because ollama launched it with a 2,048-token batch — `n_ubatch × n_vocab × 4 B` of compute buffer, 1.2 GB, for inputs that never exceeded 484 tokens. The unit runs `-c/-b/-ub 1024` with q8_0 KV (~2 GB predicted on the GPU; `EMBED_N_GPU_LAYERS=0` measured 1.4 GB CPU-only with queries FASTER and indexing 13× slower), under a cgroup cap from `config/clawbox-resource-limits.env`, and is a system unit precisely so it can carry one — the web server's unit has none, and a user unit would share the desktop's. OpenClaw's `memory.search` is pointed at the PROXY (`provider: openai-compatible`, `remote.baseUrl` = `/setup-api/local-ai/embed/v1`, `apiKey` = `data/.local-ai-token`), never at port 8081: the proxy is what wakes the unit on the first request (`ensureLocalAiReady("embed")`, `sudo systemctl start`) and stops it ten idle minutes later, and it has to be an always-listening endpoint because OpenClaw's embedding client retries a refused connection three times inside two seconds and then answers keyword-only. The proxy also restores the model's query instruction: OpenClaw's ollama adapter prefixed every search query with `"Instruct: Given a user query, retrieve relevant memory notes and documents\nQuery:"` and its openai-compatible adapter does not, so `memory.search.queryInputType/documentInputType` make it label each request and `src/lib/embed-query-instruction.ts` prefixes the query ones — without this recall quietly degrades while the index stays "healthy". `src/lib/embed-input-fit.ts` trims an input that would overflow the batch (OpenClaw chunks by CHARACTERS, so a 1,600-character CJK chunk can be 1,600 tokens) through llama-server's `/tokenize`, the way ollama's `truncate: true` did, instead of letting the document drop from the index. `ensureLocalAiReady("embed")` refuses the wake below `EMBED_WAKE_MIN_AVAILABLE_MB` (2,300) of MemAvailable — the unit's own cap cannot stop a wake from squeezing the gateway or a build — and `armIdleStopIfRunning("embed")` at web-server boot re-arms the idle stop for a unit that outlived a restart. Provisioning is `scripts/ensure-local-embeddings.sh` (the GGUF via `hf download`, then the config in ONE merged `config set … --strict-json --merge` — a per-key sequence was killed midway on the box, because the bearer write restarts the gateway and during an update that restart is a `systemctl stop` of the cgroup the script runs in — read back from the file before success is logged, then `memory index --force` because the provider change alters the index identity and vector search pauses until a rebuild), run by install.sh's `ensure_local_embeddings` after the web server is up and detached by `gateway-pre-start.sh` on every gateway start. The script and `gateway-pre-start.sh` both export `OPENCLAW_CONFIG_PATH`/`OPENCLAW_STATE_DIR` and `unset OPENCLAW_HOME` before any `openclaw` runs: the CLI reads `OPENCLAW_HOME` as the ACCOUNT home and puts its tree at `$OPENCLAW_HOME/.openclaw`, while ClawBox has always used that name for the `.openclaw` directory itself, and the updater exporting it into the pre-start made every CLI write land in a second tree at `~/.openclaw/.openclaw/` with a clean log. The updater passes `CLAWBOX_OPENCLAW_HOME` instead (never `OPENCLAW_HOME`), and pre-start's `remove_stray_state_tree` deletes such a nested tree when it carries neither a workspace nor credentials; `step_embed_model` caches the model as root first so a first request never downloads inside OpenClaw's 120 s batch budget. Ollama stays installed for its chat-provider backend (retired from every picker; `local-ai/ollama` proxy, `ollama/*` routes, `setup/reset`'s model wipe are unchanged) but has no row on Settings → Local AI any more: the Memory search row IS the engine row ("Qwen 3 via llama.cpp"), `control: "none"` because the proxy owns the lifecycle, `managedBy: "clawkeep"` for the Memory Shard pointer. `post_update` stops ollama after the switch (stop, never disable) and `free_memory_for_build` stops the embed unit the way it stops ollama — and since TASK-724 both stops go through `pause_engine_unit`, which records whether the unit was RUNNING, with `resume_paused_engines` starting back exactly those and checking each after a settle (every one is `Type=simple`, so systemd calls it active the instant it forks). It runs at the end of `step_post_update` — the update's last act, and the stop whose missing start was the defect — and at the end of `do_rebuild` except when the caller announced `--reboot-follows`, since starting four model servers seconds before a reboot restores nothing that survives it. An update that stopped an engine and reported `completed` used to leave it stopped until something happened to ask for it; an engine the owner or the idle standby had already put away is still left alone, and `src/instrumentation.ts` now re-arms the ollama idle stop at web-server boot the way it already did the embedder's, because a root shell cannot arm a timer that lives in the web server. - **Ollama** (chat-provider backend, no UI): `ollama/status`, `ollama/pull`, `ollama/search`, `ollama/delete` — local model management. `search` offers only sizes under the ollama.service memory cap (`OLLAMA_MAX_MODEL_PARAM_B` in `resource-limits.ts`, answered as `maxParamBillions`); `pull` follows the client's abort so a cancelled download stops. The runtime's idle standby (`systemctl stop`, never `disable`, ten minutes after the last proxied use) still applies through `local-ai-runtime.ts`. - **Apps**: `apps/store`, `apps/install`, `apps/uninstall`, `apps/icon/[appId]`, `apps/settings`, `apps/skill-info` — app store integration. ClawHub namespaces skills as `@owner/slug` and `openclaw skills install` refuses a bare slug more than one publisher uses, so `install` resolves the publisher first (`src/lib/clawhub-url.ts` — GET clawhub.ai/api/v1/skills/): one owner installs; an ambiguous slug is settled only by the store listing's own `developer`, otherwise the route answers 409 `{ code: "ambiguous", matches: [{ ownerHandle, ref, url }] }` and the caller re-posts `{ appId, owner }` or `{ appId: "@owner/slug" }`; a slug ClawHub lacks is 404 `not_found`. Every install failure is a non-2xx with `ok: false`, an honest `code` and `retryable` (the CLI's trust gate is left alone — no `--acknowledge-clawhub-risk` — so a `review_required` release tells the owner to install it from the Terminal). Locally everything stays keyed by the bare slug, and the icon is downloaded only after the install succeeded. `install`, `uninstall` and the webapp registry (`src/lib/webapp-registry.ts`, behind webapp deploys) are the only writers of `installed_apps`/`installed_meta`; `uninstall` also drops the window's KV keys and `app__settings` on every edition, and — only where the edition HAS OpenClaw (`openclawSkillRoot()`, null on the hermes SKU alone) — the skill directory and `skills.entries.` from openclaw.json. That split is why `uninstall` is deliberately the one app route without `openclawAppsGuard()`: the MCP `app_uninstall` tool posts to it on Hermes to remove a webapp whose store the UI has hidden, so it must stay reachable there without ever resolving a delete target under a workspace that edition does not have (TASK-551). Its answer carries `skillRemoved: true | false | null` (`null` = no skill half to report on: the hermes SKU, or a web app) so a caller can tell a removal from a no-op, and it refuses the whole uninstall — 503 `ok: false`, `retryable: true` — rather than dropping the desktop entry for a skill it could not reach, because half an uninstall leaves the owner nothing to retry from: `config_unreadable` when openclaw.json exists and cannot be read (nothing is touched — that is not the same as an edition that has none), `skill_remove_failed` when the folder is there and will not go (`fs.rm` deletes as it walks, so part of it may already be gone, the message says so, and the skill-info cache is rescanned before answering); the outer catch carries the same contract (`uninstall_failed`, 500) and reports `skillRemoved`, because everything after the skill delete can still throw over a folder that is already gone; an id that is not one plain path segment is 400 `invalid_app_id`, `retryable: false`, before anything is touched. Every id that reaches a path on these routes is REBUILT from the alphabet rather than tested and passed through (`safeAppId` in `src/lib/webapp-icon.ts`, `safeProjectId` behind `projectDir`/`webappPath` in `src/lib/code-projects.ts`, `safeSkillName` in `src/lib/openclaw-skill-info.ts`) — a `.test()` guard leaves the caller's string in play, and CodeQL rightly flags every path built from it (`js/path-injection`). The 1-64 `[A-Za-z0-9_-]` bound is a rule about what may ARRIVE: `install` applies it (as `zInstalledAppId`, `skill-info`, the webapps route and the icon route already did), `uninstall` deliberately does not, so an app installed before that bound existed can still be removed. Whether the app is a WEB APP — `installed_meta.webappUrl`, else the deployed `data/webapps/` this uninstall removed — decides two things and neither is whether the skill half RUNS: an unreadable OpenClaw config does not refuse a web app (it has no skill half, so on a `dual` box a config belonging to the idle harness must not strand the agent's own pages), and an id the desktop knows as a web app answers `null` rather than `false`, since "there was no skill of that name on disk" is an absence report about something that never existed. One id can be both — `webapp_create` REPLACES `installed_meta[]` for any id with no collision check — so the removal itself is never skipped; and on the one path where it CANNOT run (a web app the `installed_meta` KNOWS about, removed while that config is unreadable — one recognised only by its deployed directory is still refused, since the refusal is answered before anything is looked at) the answer carries `skillHalfChecked: false` beside `ok: true`, present only when false, because `null` there would claim there was no skill half when the route simply could not look — `app_uninstall` and the desktop's own toast both relay that as "its skills were not checked". The enable switch (`settings` `_setEnabled`) is a direct JSON write of `skills.entries..enabled` (the gateway hot-reloads it; the CLI cost 10–17 s per click), and `skill-info?appId=` reads `enabled` back from the same file — its list comes from `openclaw skills list --json` served stale-while-revalidate (`src/lib/openclaw-skill-info.ts`; the routes rescan behind an install/uninstall), answering 404 `not_installed` for a skill that is gone and 503 `skills_unavailable` when the CLI failed with nothing cached. `store?slug=` adds `ownerHandle` (and `clawhubMatches`) from ClawHub and rewrites `clawhubUrl` to the real page, or removes it. `icon/[appId]` persists to `data/icons/` only for installed apps and remembers an upstream 404 for an hour. -- **Files**: `files/` — file list, read, write, upload, mkdir, delete. Its `resolve` action answers `relPath` beside `absPath` — the browse-relative path the Files app navigates by — which is how the Coding Agent's "Open in Files" turns a project folder's absolute path into a Files window opened there (`dispatchOpenApp("files", { forceNew, meta: { path } })` → `FilesApp initialPath`). +- **Files**: `files/` — file list, read, write, upload, mkdir, delete. Its `resolve` action answers `relPath` beside `absPath` — the browse-relative path the Files app navigates by — which is how the Coding Agent's "Open in Files" turns a project folder's absolute path into a Files window opened there (`dispatchOpenApp("files", { forceNew, meta: { path } })` → `FilesApp initialPath`). Uploads are bounded by what the box can spare, not by a size cap (large uploads are the point of the PUT path): `src/lib/bounded-body.ts` meters the body against a budget that re-measures free space on the DESTINATION filesystem every 64 MiB minus `DISK_FREE_RESERVE_BYTES` (`src/lib/disk-reserve.ts`, 512 MiB — the in-app update's build is what a full disk kills; project-import's reserve is the same constant), a chunked or understated body is cut at the reserve with 507 `disk_full` and the partial file unlinked, and the multipart POST carries Busboy limits (`files: 1`, `parts: 4`) with the attachments route's teardown discipline — busboy's `fileSize` TRUNCATES rather than errors, so the `limit` handler destroys the write and answers 507 instead of `ok: true` over a cut file; count limits answer 413 `too_many_parts`; a statfs that fails skips the reserve for that request rather than refusing every upload. The `[...path]` PUT (rename) and DELETE refuse a PROTECTED CONTAINER with 400 `protected_container` (`isProtectedContainer` in `src/lib/file-guard.ts`, judged on the typed path AND its `canonicalPath`): `data/` itself, the checkout, `~/.config` and the browse root — `isProtectedFilePath(DATA_DIR)` stays false so the listing keeps filtering entry by entry, but a rename of `data/` to `data-copy` used to move every credential store out from under the guard in one request. - **Browser**: `browser/` — Chromium automation via CDP (launch, navigate, click, type, screenshot). WHICH Chromium it drives is the owner's setting, read for EVERY session: with the Coding Agent's real browser on (`coding_agent_real_browser`, absent means on) it uses the device's own window, brought up the way `browser/manage`'s open does, and falls back to a headless Chromium of its own only when that is impossible (a foreign Chromium on port 18800, a launch that failed); with it off it goes straight to the headless one and never touches the owner's screen. Every answer carries `browser: "desktop" | "headless"` — a screenshot from the two is identical, and a run that verified its work where nobody could watch must not report that the owner saw it. The page handles it hands out live in `src/lib/browser-sessions.ts` rather than in the route, because a delegated coding run drives this Chromium too: a run gets a page of its OWN (borrowing `context.pages().at(-1)` steered the tab the owner was reading, and the ten-minute idle sweep then closed it), every session records the run that opened it, and the runner closes exactly those pages when the run settles. `browser/manage` is the device half — install Chromium, the agent link, open/close the desktop window — and answers `serviceSafe: false` for a snap build `clawbox-browser.service` cannot start, refusing the launch at once instead of "check /tmp/clawbox-browser.log" ten seconds later; the same GET also carries `setupComplete`, `autoOpen` and `startUrl`, so the app's face and its state cannot come from two different moments. `browser/setup` (owner-only AND same-origin) writes those three: the auto-open switch exists because `ui_open_app("browser")` is a tool call, and a tool that could switch it back on would make the owner's "no" temporary. systemd starts Chromium, not the web server, so the start page reaches it on disk — `writeBrowserLaunchEnv` (`src/lib/browser-setup.ts`) puts it in `~/.cache/clawbox/browser.env`, which `scripts/launch-browser.sh` sources the way it already sources the VNC display — and only an `http(s)` address survives `normalizeStartUrl`, because Chromium would happily open `file:///etc/shadow` on the screen the agent screenshots. - **Gateway**: `gateway/`, `gateway/health`, `gateway/ws-config` — gateway proxying with HTML injection - **Telegram**: `telegram/configure`, `telegram/status` — Telegram bot config - **Email**: `email/configure`, `email/status`, `email/test`, `email/send`, `email/messages`, `email/pending` — a mail account (any provider; Gmail app-password guide in the UI) with ONE of three modes: send only, read on demand (`email/messages` backs the `email_list`/`email_read` MCP tools over ClawBox's own IMAP client — no polling, EXAMINE + BODY.PEEK so reading never marks mail seen), or answer senders (Hermes' native adapter, allowlist-only). A separate "ask me before sending" gate turns `email/send` into a queued draft; `email/pending` approves or deletes it and refuses the MCP bearer, because the agent is the party it gates (`coding-agent/enable` refuses it for the same reason). Its `approve_batch` action is what the chat's batch card posts: one consent for a NAMED set of drafts, each carrying the content fingerprint it was shown with, so nothing queued while the owner was reading can ride along — and it answers per draft, 207 unless every one of them reached somebody — a copy an identical message already covered, and a draft another surface SENT while the card sat on screen, are counted apart from the failures, because neither is anything gone wrong; a draft that was deleted, refused or left unconfirmed elsewhere stays among them, because under all three the message reached nobody. A row for a send that failed carries the RECEIPT's ending (`failed` for a refusal the mail server spoke, `unconfirmed` for a silence this box cannot read), never the SMTP kind, so no surface can turn a message that may have gone out into a definite "not sent". Its `reject_batch` is the mirror: one gesture deletes a NAMED SET, each entry carrying the fingerprint it was shown with, so "send nothing" reaches the store instead of hiding the card until the next poll re-offered the same drafts. Its GET answers `{ pending, outcomes }` in one response — the queue and the receipts for what has left it — so no surface can catch a draft in neither list and render a live Approve button over a message that is already sent. `email/chat-approval` (owner-only, off by default) connects a SECOND Telegram bot that ClawBox polls itself and never hands to the harness, so a queued draft can be approved with a button in the owner's chat: the tap arrives from Telegram, not from the agent. `email/chat-reply` is the same consent with no BotFather token, and unlike `chat-approval` it is **ON by default** (owner's ruling 2026-09-04: "when I tell it in Telegram to send, it will send" — a switch he has to find first would not be that; and unlike the second bot it asks him for no credential, so there is nothing to configure): the owner replies "send " in the conversation he already has with the box, and each harness's OWN inbound hook — `before_dispatch` on OpenClaw, `pre_gateway_dispatch` on Hermes, both registered by the plugin ClawBox already ships — claims that message before the model sees it and posts it here. The consent it rides on is the one already given: it only ever asks about a draft, and a draft only exists because "ask me before sending" is on. The gate is the sender id the harness reports, checked against the harness's own owner allowlist, exactly as the button path checks who pressed; the code names ONE draft and its content fingerprint, so "send" can never mean "whatever is queued now". There is still no approve verb on the tool surface (`src/lib/owner-session.ts`), and the code is deliberately withheld from the `email_send` tool result. - **Discord**: `discord/configure`, `discord/status`, `discord/members` — Discord bot config (token validated live against the Discord API before it is saved), gateway connection state, and the guild-member allowlist picker -- **Setup**: `setup/status`, `setup/complete`, `setup/reset` — setup flow state, factory reset +- **Setup**: `setup/status`, `setup/complete`, `setup/reset` — setup flow state, factory reset — `setup/complete` checks `requireSession` in the handler itself (allowBootstrap false; CLAWBOX_TEST_MODE keeps the e2e harness's cookie-less door), the way `setup/reset` does, so it no longer relies on the middleware allow-list alone. - **Update**: `update/run`, `update/status` — git-based system updates -- **Preferences**: `preferences/` — persistent user preferences (language, installed apps, etc.) +- **Preferences**: `preferences/` — persistent user preferences (language, installed apps, etc.) — a POST naming an `installed_*` key is refused 403 `owner_only` unless `hasOwnerSession` (cookie, never the MCP bearer): `installed_meta[id].launch = "window"` makes the desktop `window.open` the app's URL top-level, and the middleware admits the bearer to this route like any other, so a bearer holder could plant an entry; install/uninstall and `webapp-registry.ts` write the store directly and the desktop reads the prefix, so no legitimate caller loses anything. - **KV Store**: `kv/` — key-value store for UI state -- **Code**: `code/` — code project management (init, file ops, search, build/deploy) +- **Code**: `code/` — code project management (init, file ops, search, build/deploy) — `search` is substring only: a body with `regex` is answered 400 `regex_unsupported` rather than compiled, because a caller-supplied pattern ran on the request thread with cost exponential in the LINE (`(a+)+$` over 26 characters was 10 s on the Orin), nothing on the device sent it, and the OpenClaw agent's `grep` tool is a killable child with a deadline. - **Coding agent**: `coding-agent/status`, `coding-agent/enable`, `coding-agent/run`, `coding-agent/runs`, `coding-agent/stop`, `coding-agent/kill`, `coding-agent/artifacts`, `coding-agent/projects`, `coding-agent/media/image`, `coding-agent/media/audio`, `coding-agent/tree`, `coding-agent/team`, `coding-agent/team/stop` — a headless Claude Code run (`claude-ds -p`) the assistant delegates a whole task to; runs live in the web server and persist in `data/coding-agent-runs.json`. `enable` is owner-only (refuses the MCP bearer) because it is the consent for a delegated shell; `run` answers 409 while the switch is off. Each run gets an evidence folder (`data/coding-agent-artifacts//`) for the screenshots and test output it saves; when a run settles, its closing message is filed there as `report.md` too (unless the run wrote its own), and the app renders that — and any `.md` a run wrote — as Markdown through the chat's own renderer; `artifacts` serves one such file (images and the WAVs a run can now speak for itself inline — the run's page plays them with a plain `