Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
87ab114
add secure Linux screen capture contract
KesleyDavid Aug 13, 2026
21cd3fd
add preview-only Linux screen UI
KesleyDavid Aug 13, 2026
91942c3
document and smoke-test Linux screen preview
KesleyDavid Aug 13, 2026
12787ce
document validated Wayland preview
KesleyDavid Aug 13, 2026
efa404f
handle cancelled Wayland capture cleanly
KesleyDavid Aug 13, 2026
6aae453
add safe Linux CUA diagnostics
KesleyDavid Aug 13, 2026
c623262
supervise private Linux CUA runtime
KesleyDavid Aug 13, 2026
410539d
isolate Linux CUA routing and approvals
KesleyDavid Aug 13, 2026
0903909
add explicit Linux local control UX
KesleyDavid Aug 13, 2026
c9353f5
document and smoke-test Linux CUA beta
KesleyDavid Aug 13, 2026
fc5abb7
harden certified Linux CUA handoff
KesleyDavid Aug 13, 2026
8b53628
isolate Linux package smoke runtime
KesleyDavid Aug 13, 2026
40076c3
document pinned Linux CUA installation
KesleyDavid Aug 13, 2026
3de84c8
enable guarded GNOME Wayland CUA control
KesleyDavid Aug 14, 2026
a9de63c
surface GNOME Wayland local control states
KesleyDavid Aug 14, 2026
8ec7728
smoke-test packaged GNOME Wayland control
KesleyDavid Aug 14, 2026
152abee
document the guarded Wayland control beta
KesleyDavid Aug 14, 2026
5c925b2
scope Linux filesystem tests to Linux
KesleyDavid Aug 14, 2026
7d78aec
scope Linux runtime lifecycle tests to Linux
KesleyDavid Aug 14, 2026
38c6c98
keep Linux private-state tests platform-correct
KesleyDavid Aug 14, 2026
d09a079
address Ubuntu desktop review feedback
KesleyDavid Aug 14, 2026
4107cfa
keep POSIX socket fixtures short
KesleyDavid Aug 14, 2026
21a774e
scope Linux group permission proofs
KesleyDavid Aug 14, 2026
933eb43
guard capability state against stale loads
KesleyDavid Aug 14, 2026
1df2efc
prove Linux native feature gates
KesleyDavid Aug 14, 2026
ebc6dfe
harden Linux CUA lifecycle and privacy
KesleyDavid Aug 14, 2026
db571cf
address final Ubuntu PR review
KesleyDavid Aug 14, 2026
b187718
clarify Linux preview policy
KesleyDavid Aug 14, 2026
303d7dd
address stacked Ubuntu review findings
KesleyDavid Aug 14, 2026
ab27593
tighten Ubuntu desktop contracts
KesleyDavid Aug 14, 2026
53092fa
strengthen Linux runtime lifecycle tests
KesleyDavid Aug 14, 2026
b1e6872
stage pinned Linux CUA release
KesleyDavid Aug 14, 2026
72e86fa
resolve bundled CUA in Linux packages
KesleyDavid Aug 14, 2026
9e8c280
package and verify bundled Linux CUA
KesleyDavid Aug 14, 2026
02b810d
document bundled Linux CUA trust model
KesleyDavid Aug 14, 2026
6b757b7
make Linux package checks portable
KesleyDavid Aug 14, 2026
10df11e
address bundled CUA review findings
KesleyDavid Aug 14, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 22 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,18 +52,36 @@ jobs:
with:
node-version: 24
cache: pnpm
- name: Install package validation tools
run: sudo apt-get update && sudo apt-get install -y desktop-file-utils xvfb dbus-x11
- name: Install package validation and native smoke tools
run: >-
sudo apt-get update && sudo apt-get install -y
at-spi2-core dbus-x11 desktop-file-utils libxi6 libxkbcommon0 squashfs-tools xvfb
- run: pnpm install --frozen-lockfile
- run: pnpm package:linux
- name: Stage the pinned CUA runtime
run: pnpm build:cua:linux
- name: Package from the verified offline CUA stage
run: pnpm package:linux:offline
- run: node scripts/verify-linux-package.mjs
- name: Configure Chromium sandbox for the unpacked app
run: |
sudo chown root:root release/linux-unpacked/chrome-sandbox
sudo chmod 4755 release/linux-unpacked/chrome-sandbox
test "$(stat -c '%U:%G %a' release/linux-unpacked/chrome-sandbox)" = "root:root 4755"
- name: Launch packaged app and verify lifecycle
run: dbus-run-session -- xvfb-run -a node scripts/smoke-linux-package.mjs
env:
OMB_KEEP_SMOKE_DIR: "1"
run: pnpm smoke:linux-package
- name: Upload smoke diagnostics on failure
uses: actions/upload-artifact@v4
if: failure()
with:
name: openmausbot-ubuntu-smoke-diagnostics
path: |
${{ runner.temp }}/omb-linux-smoke-*
${{ runner.temp }}/omb-linux-smoke-runtime-*
if-no-files-found: warn
include-hidden-files: true
retention-days: 7
- uses: actions/upload-artifact@v4
if: always()
with:
Expand Down
15 changes: 13 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,8 +101,19 @@ The SPI in [`server/contracts.ts`](server/contracts.ts) is deliberately small. A
- Renderer code must consume the desktop capability contract rather than infer support from Electron,
the user agent, or the presence of a preload bridge. Screen preview, dictation, and local control are
independent capabilities.
- Test Ubuntu platform claims on a real GNOME session. Xvfb proves packaging and lifecycle, not Wayland
portal behavior or local computer control.
- Test Ubuntu platform claims on a real GNOME session. Xvfb proves packaging and fake-driver orchestration, not
Wayland portal behavior or real CUA inspection/input delivery.
- Linux local control must remain explicit: global opt-in plus per-bot **This computer**. Linux Auto, provider
full-auto/bypass modes, remembered grants, and cloud approvals must never authorize the user's desktop.
- Keep CUA discovery shell-free and pin accepted archive, inner-file, manifest, and driver contracts. Packaged Linux
builds must prefer their reviewed outside-ASAR runtime and fail closed instead of executing ambient PATH code;
source/dev builds may use the validated explicit/user-local paths. Never add a runtime downloader/self-updater or
silently install GNOME extensions. GNOME/Wayland readiness must require its exact compositor/helper/portal health
contract; never infer it from `WAYLAND_DISPLAY` or XWayland.
- Native release changes must update the checked-in Cua license report/SBOM, preserve MIT/OFL/MPL notices, pass the
malicious-archive tests, and prove identical hashes in `linux-unpacked`, `.deb`, and AppImage artifacts. AppImage
must additionally prove post-copy hashing in its private `0700` execution stage; never weaken the general path
validator to accept a root-owned group-writable SquashFS path when its toolchain emits `0775` rather than `0755`.
- **Never build command strings for a shell.** No `shell: true`, no spawning through `cmd.exe` with
quoted strings — model names, personas, and MCP config JSON travel through argv, and cmd.exe
metacharacter expansion is a real injection class. On Windows, resolve `.cmd` shims to their JS
Expand Down
32 changes: 24 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,8 @@ already have:
custom CLI binary (a versioned build or wrapper) in **Settings → Engines**.
- **Local first.** One small harness server on `127.0.0.1` owns every agent process. Transcripts, keys, and
events live in `~/.openmausbot`, not a cloud.
- **Agents with hands.** Each bot can get a real computer — a cloud Linux desktop it drives while you watch
live, or your own Mac — plus 500+ apps through Composio.
- **Agents with hands.** Each bot can use a cloud Linux desktop, an isolated Local VM, or your own computer,
plus 500+ apps through Composio. Host control is available on macOS and as an explicit Ubuntu GNOME beta.

## Features

Expand Down Expand Up @@ -173,7 +173,7 @@ flowchart LR
| API | `server/index.ts` | Bots, turns, approvals, model catalog, computer lifecycle, connectors, config — HTTP + SSE. |
| Voice | `server/tts/` | ElevenLabs, bring your own key. Runs on the harness so the key never reaches the UI; markdown is rewritten into something worth hearing before it is spoken. |
| App | `src/` | The chat shell. Server-backed store, one reducer, zero client-side transports. |
| Desktop | `electron/` | macOS, Windows, and Ubuntu shells with an embedded harness and explicit platform capabilities; Apple speech, local screen capture, and the current CUA bridge remain macOS-only. |
| Desktop | `electron/` | macOS, Windows, and Ubuntu shells with an embedded harness and platform capabilities; Apple speech stays macOS-only, while a release-pinned bundled CUA runtime enables guarded Ubuntu GNOME local control. |

## Quick start

Expand Down Expand Up @@ -208,7 +208,7 @@ Package the desktop application:
```sh
pnpm package:mac # macOS: DMG + ZIP; requires Swift/Xcode tools
pnpm package:win # Windows: installer + ZIP
pnpm package:linux # Ubuntu x64: .deb + AppImage; no Swift required
pnpm package:linux # Ubuntu x64: .deb + AppImage + verified CUA runtime
```

### Desktop capability status
Expand All @@ -217,12 +217,24 @@ pnpm package:linux # Ubuntu x64: .deb + AppImage; no Swift required
|---|---|---|---|
| Packaged app, embedded harness, local agent CLIs | Supported | Beta | Beta |
| Composio and Box/cloud computers | Supported | Beta | Beta |
| Local screen preview and computer control | Supported | Planned | Planned after compositor validation |
| Explicit preview-only local screen capture | Supported | Beta | Beta |
| Bot control of this computer | Supported | Beta: opt-in, bundled Cua 0.19.3 | Beta: GNOME only, opt-in, bundled Cua 0.19.3; separately installed WinRects v8 helper |
| Native on-device dictation | Supported | Planned | Planned |

Unavailable native features fail closed on Ubuntu without blocking chat or cloud features. Linux local computer
control, Wayland capture/automation, dictation, and ARM64 are tracked in
[#29](https://github.com/milind-soni/OpenMausBot/issues/29) and are not claimed by the baseline package.
The Linux preview is user-initiated and never enables local bot control or Auto routing. Packaged Linux builds ship
the exact Cua Driver 0.19.3 runtime outside ASAR; control still requires explicit app opt-in and an explicit per-bot
**This computer** selection, and every local action asks for approval. GNOME/Wayland additionally requires the
versioned WinRects v8 helper and a
passing prompt-free AT-SPI/capture/portal health report. Other Wayland compositors fail closed without blocking
chat or cloud features. See the [Ubuntu Desktop guide](docs/linux-desktop.md) and
tracking issues [#29](https://github.com/milind-soni/OpenMausBot/issues/29) and
[#79](https://github.com/milind-soni/OpenMausBot/issues/79) / [#109](https://github.com/milind-soni/OpenMausBot/issues/109) / [#113](https://github.com/milind-soni/OpenMausBot/issues/113).

The Linux packager downloads only the tag-pinned upstream archive during the build, verifies its size, SHA-256,
complete member allowlist, and inner executable hashes, then packages only the CLI and cursor-theme sidecar. The
installed app never downloads or self-updates native automation code. Cua's MIT notice, Inter's SIL OFL, a generated
third-party license report, and a CycloneDX inventory ship with the runtime. See
[`third_party/cua-driver/`](third_party/cua-driver/) for the reviewed provenance record.

These credentials are optional — local chat works without them. Paste a key once in **App Settings** (gear
in the sidebar footer) when you want to enable its integration:
Expand Down Expand Up @@ -275,6 +287,10 @@ small; adding a provider is one file in [`server/drivers/`](server/drivers/) plu

[MIT](LICENSE) © 2026 Milind Soni and contributors.

Packaged Cua Driver components retain their upstream MIT, SIL OFL 1.1, MPL-2.0, and other dependency terms;
the corresponding notices, license texts, source locations, and SBOM are in
[`third_party/cua-driver/`](third_party/cua-driver/) and ship beside the native runtime.

OpenMausBot is an independent, open-source project inspired by Grok Bot. It is
not affiliated with, endorsed by, or associated with xAI; "Grok" is a trademark
of its respective owner.
44 changes: 30 additions & 14 deletions docs/computer-use-integration.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,18 @@
# Computer use & browser use in OpenMausBot

Decision doc, 2026-08-12. How bots in OpenMausBot get local computer use and
browser use, out of the box, with no separate installs. Based on a survey of
OSS chat-app MCP hosts, macOS control servers, browser-automation stacks, and
the local `cua` / `axstream` code on this machine.
browser use. macOS and packaged Ubuntu x64 builds use an out-of-the-box,
release-pinned provider; source/dev Ubuntu may use a separately installed provider. Based
on a survey of OSS chat-app MCP hosts, macOS control servers,
browser-automation stacks, and the local `cua` / `axstream` code on this
machine.

## TL;DR architecture

```
```text
Electron main process
├── EmbeddedCuaDriverHost ──spawns──▶ cua-driver (bundled Rust binary, Resources/)
one TCC prompt, named OpenMausBot │ unix socket (private)
├── CUA host ──spawns──▶ cua-driver (bundled on macOS and packaged Ubuntu x64)
platform permission boundary │ unix socket (private)
├── WebContentsView pool (embedded browser, persist: partitions per bot)
│ driven via webContents.debugger (CDP) — zero-install browser use
└── server/ harness (drivers spawn agent CLIs with --mcp-config)
Expand All @@ -21,8 +23,11 @@ Electron main process
- **Plugins = MCP servers over stdio.** The Plugins panel toggles which MCP
servers get injected into each bot's `--mcp-config`. Same pattern as Claude
Desktop / Cherry Studio / LibreChat.
- **Computer use = bundled `cua-driver`** (Rust, single static Mach-O,
23MB arm64 / 48MB universal — from `mywork/cua/libs/cua-driver/rust`).
- **Local desktop use = `cua-driver`**. macOS packages the Rust Mach-O in app
Resources; Ubuntu x64 packages the certified 0.19.3 ELF plus its cursor-theme
sidecar outside ASAR. Both remain paired with the application release. This
applies to the Ubuntu 24.04 GNOME/Xorg beta and guarded GNOME/Wayland beta;
remote/cloud boxes and the isolated Local VM remain separate providers.
NOT Swift — the Swift file everyone remembers
(`examples/embedded-host-macos/ExampleAgentHarness.swift`) is a 165-line
reference host showing the embedding pattern, not the driver.
Expand All @@ -31,18 +36,29 @@ Electron main process
`webContents.debugger` CDP transport. No Chrome dependency, no 281MB
Playwright download, and the user watches the bot browse inside the chat.

## Computer use: CUA only — bundle cua-driver, spawn from Electron main
## Local desktop use: CUA only — Electron owns the driver lifecycle

**Decision (Milind, 2026-08-12): CUA is the ONLY computer-use provider.
**Decision (Milind, 2026-08-12): CUA is the ONLY local desktop-control provider.
No cliclick, no robotjs/nut.js, no Python computer-server, no fallbacks.**
Everything that touches the user's screen/mouse/keyboard goes through the
bundled `cua-driver` binary. Alternatives evaluated and rejected:
All local desktop-control and input actions go through the validated
`cua-driver` binary. Linux screen preview uses the supported Xorg or
user-initiated XDG portal capture path and is not a control provider. This rule
does not replace remote/cloud boxes or the isolated Local VM provider. Local
alternatives evaluated and rejected:

The Ubuntu GNOME beta uses the same official CUA provider with the Phase 5
supply-chain contract tracked in [#113](https://github.com/milind-soni/OpenMausBot/issues/113): pinned archive
and inner hashes, exact archive allowlist, outside-ASAR resources, full notices/SBOM, no runtime download/update,
and fail-closed packaged discovery. Electron still owns a private embedded daemon/socket, and the harness only
receives the validated MCP proxy contract. Xorg is tracked in [#79](https://github.com/milind-soni/OpenMausBot/issues/79);
GNOME/Wayland additionally requires WinRects v8 plus the exact Cua health-report contract tracked in
[#109](https://github.com/milind-soni/OpenMausBot/issues/109).

| Option | Verdict |
| --- | --- |
| cua `computer-server` (Python/FastAPI) | ✗ 200MB+ frozen Python, second TCC prompt under wrong identity |
| axstream / cliclick / robotjs-class | ✗ rejected — CUA-only policy |
| **cua-driver binary, embedded mode** | ✓ THE provider: zero deps, 20+ tools, its own stdio MCP proxy + socket daemon + TS SDK (`@trycua/cua-driver`), agent-cursor overlay, permission tooling |
| **cua-driver binary, embedded mode** | ✓ THE provider: one contract, 20+ tools, its own stdio MCP proxy + socket daemon + TS SDK (`@trycua/cua-driver`), agent-cursor overlay, permission tooling |

### The rules (from `cua/libs/cua-driver/rust/Skills/cua-driver/EMBEDDING.md` — read it end to end)

Expand All @@ -64,7 +80,7 @@ bundled `cua-driver` binary. Alternatives evaluated and rejected:
grant change, destroy clients → `restart()` → reconnect (macOS caches TCC
per process).

### Packaging
### macOS packaging target

- Ship the binary at `OpenMausBot.app/Contents/Resources/cua-driver`,
**outside the ASAR**, executable bit preserved (electron-builder
Expand Down
Loading
Loading