feat(skills): add tldraw-offline agent scripting skill - #66896
Merged
Conversation
Optional skill for driving the tldraw offline desktop app via its local
HTTP control API (the same curl-based path the app's own agent skills use
for Codex/Claude Code/Cursor/Gemini) — read the canvas, make live edits,
and write embedded document scripts.
Grounded in the app's bundled script-context.d.ts and agent playbook, and
in the real tldraw SDK v5 shape schema:
- document-script contract: export default function ({ editor, helpers, signal })
- HTTP API: /api/search, /api/doc/:id/exec, /api/doc/:id/script-workspace,
/api/doc/:id/script-status (bearer token from server.json, re-read per call)
- shape schema table validated against @tldraw/tlschema (scripts/validate_shapes.mjs, 3/3)
- interactive-UI example (scripts/counter.js) + diagram-generation (scripts/main.js)
- honest verification boundary: click->state logic verified via /exec dispatch
(0->1->2->1->0), with documented host caveats (inotify watcher, Electron
background-click rejection)
Tests: tests/skills/test_tldraw_offline_skill.py (15 passing).
alt-glitch
added
type/feature
The skill claimed Chromium/Electron 'reject synthetic clicks' so computer-use can't drive the canvas. The Cua team disproved this on the exact v1.11.0 AppImage (Linux/X11): background delivery returns background_unavailable, but that's the first rung, not a wall — cua-driver returns escalation:'foreground' and its X11 XTest path (x11_xtest_fg) with delivery_mode:'foreground' clicks through, dismissing the consent dialog and landing canvas clicks. Corrected the note to say: climb to foreground on background_unavailable, don't conclude Electron is unclickable. Ref: #67052.
23 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an optional skill that teaches Hermes to drive the tldraw offline desktop app through its local HTTP control API — the same curl-based path the app itself installs for Codex/Claude Code/Cursor/Gemini — to read a canvas, make live edits, and author embedded document scripts.
Changes
optional-skills/creative/tldraw-offline/SKILL.md— the skill: the local HTTP API (/api/search,/api/doc/:id/exec,/api/doc/:id/script-workspace,/api/doc/:id/script-status; bearer token fromserver.json, re-read per shell call), the({ editor, helpers, signal })document-script contract, the shape-prop schema, packaging a self-running.tldraw(manifest digest + trust store), and pitfalls.scripts/main.js— diagram-generation example (idempotent boxes + bound arrows).scripts/counter.js— interactive-UI example (MINUS/RESET/PLUS buttons driving live state).scripts/validate_shapes.mjs— offline schema check against@tldraw/tlschema.tests/skills/test_tldraw_offline_skill.py— 15 tests (structure, ctx contract, HTTP-API coverage, prop-table/validator consistency, interactive example).Grounding & validation
Everything is grounded in the app's own bundled
script-context.d.tsand agent playbook (extracted from the v1.11.0 release), and in the real tldraw SDK v5 schema — not reconstructed API.@tldraw/tlschemav5/api/doc/:id/execdispatch0 → 1 → 2 → 1 → 0Honest verification boundary
The counter's click→state logic was verified using tldraw's own documented method — a simulated
editor.dispatchclick + state read via/exec— not a real GUI mouse. Two caveats are documented in the skill: (a) the script only runs once the app's file-watcher applies it (an inotify-starved host showsstate: "not-watching"and the script never executes — a host limit, not a script bug); (b) Chromium/Electron reject synthetic background clicks, so computer-use no-focus automation cannot drive the canvas (irrelevant to the real product path, which is/exec, but noted so nobody tries to "verify" via background GUI clicks).Infographic