feat(cua-driver): record human demonstrations as skill-ready trajectories - #2016
feat(cua-driver): record human demonstrations as skill-ready trajectories#2016ddupont808 wants to merge 9 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Linux visual regression artifactsMatrix jobs now run independently. Download visual artifacts from this workflow run.
|
1900bb1 to
074910d
Compare
Separate observed human input from executable trajectory recording, keep capture controls private, and produce skill-ready artifacts locally on stop. Salvaged from #2016 Co-authored-by: Dillon DuPont <ddupont@mit.edu>
ac31322 to
03e5820
Compare
injaneity
changed the title
9c4d228 to
fc08d02
Compare
fc08d02 to
a42f0eb
Compare
New crate for capturing human input on a single window during a demonstration recording, built so capture is impossible without a visible indicator: - event/coalesce/gate: HumanEvent model + privacy-preserving redaction; keystroke coalescer (typing -> one redacted Text event; command keys/hotkeys -> semantic Key events); IndicatorHeartbeat/CaptureGate coupling so the input hook only buffers an event while the recording border is provably painting and covering the point. - windows.rs: WH_KEYBOARD_LL/WH_MOUSE_LL backend that drops OS-injected, out-of-window, and indicator-dark events before buffering; gesture detection. - indicator.rs: layered click-through topmost glowing border that tracks the target window, drives the heartbeat from presented frames, and goes dark via a tamper watchdog when hidden/moved/covered. - demonstration.rs: RAII guard that builds the indicator first, derives the gate from it, then installs the hook; mac/linux are Unsupported stubs. 20 unit tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tories - recording.rs: demonstration mode. start can show the glowing border and capture window-scoped human input on a target window, written as turn-*/ folders (source:"human") interleaved with agent actions on one timeline. begin_demonstration/record_human_turn allocate turn numbers under a brief lock and write outside it; start/stop tear down the demo OFF the lock to avoid deadlocking the writer thread. RecordingState gains demonstration/human_turns; action.json gains a source field. - recording_markdown.rs: render a token-cheap TRAJECTORY.md (readable action prose, screenshots referenced by relative path, only a few key frames embedded inline) + SUMMARY.json, so reading a trajectory doesn't blow up context. - recording_tools.rs: start_recording gains demonstration/pid/window_id/ capture_raw_text; new process_recording tool renders the markdown and, when ANTHROPIC_API_KEY is set + author_skill, calls the API to draft a SKILL.md. 5 new tests (markdown + demonstration lifecycle/no-deadlock, incl. a real Windows border+capture run). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…odel
- Skills/cua-driver/DEMONSTRATION.md: how to turn a recorded demonstration into
a reusable SKILL.md in the Open Agent Skills Standard (minimal name+trigger
description frontmatter, ## Inputs, semantic ## Steps with {placeholders},
## Verification; generalize coordinates and redacted text into intent).
Registered in SKILL_FILES so it ships with the skill pack.
- RECORDING.md: demonstration mode section + the recording-border security model
(capture is gated on a visible, covering, live indicator; window-scoped;
redacted by default; tamper watchdog).
- SKILL.md: cross-links.
- docs/DEMONSTRATION_RECORDING_PLAN.md: design + status + follow-ups.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… & CLI
- indicator.rs: the recording border now glows OUTWARD from the window's
bounding box (no inset gap) — brightest right at the window edge, fading over
18px into a soft red blurred border + shadow; window interior stays
transparent and click-through.
- recording_tools.rs: intuitive one-call surface so an agent doesn't need the
start_recording flag combo:
start_demonstration(window_id, pid) -> border on, capture begins
stop_demonstration() -> stops, processes, returns TRAJECTORY.md
stop_demonstration treats 'no input captured' as a soft success, and supports
author_skill (Anthropic, when ANTHROPIC_API_KEY is set).
- CLI: 'cua-driver demonstration start <window_id> <pid> [dir] [--raw] | stop
[--author-skill] | status', wrapping the tools over the daemon socket.
- demo_record.rs example: full record -> process e2e harness.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Keep human observations separate from executable trajectories, make text capture content-free, use concrete platform capture, and narrow artifact generation to demonstrations. Co-authored-by: Zane Chee <44902825+injaneity@users.noreply.github.com>
Keep text and pointer observations in causal order, coalesce scroll bursts, reject invalid targets early, report evidence loss, and write a versioned demonstration manifest. Co-authored-by: Zane Chee <44902825+injaneity@users.noreply.github.com>
Revalidate target ownership throughout capture, preserve hook events during lifecycle locking, fail if idle timers cannot start, and disclose that screenshots can contain visible content. Co-authored-by: Zane Chee <44902825+injaneity@users.noreply.github.com>
Scope demonstration stops to their owning session, bind cleanup to the current runtime lifecycle, refresh the lockfile, and retain the tested DWM visible-frame border alignment. Salvaged from #2016. Co-authored-by: Dillon DuPont <ddupont@mit.edu>
b069e67 to
18d469b
Compare
Scope demonstration stops to their owning session, bind cleanup to the current runtime lifecycle, refresh the lockfile, and retain the tested DWM visible-frame border alignment. Co-authored-by: Zane Chee <44902825+injaneity@users.noreply.github.com>
18d469b to
f9622d0
Compare
|
I built and verified the missing macOS backend directly on top of this PR's exact head ( I opened #3248 into Evidence: 26 input-capture tests, 10 core demonstration tests, fmt/clippy/full compile-check, two fail-closed security review cycles, and a signed live macOS TextEdit test with a visible indicator, 27 physical events, zero drops, redacted typed content, and clean stop/removal. Temporary TCC grants were revoked afterward. @ddupont808, if you prefer cherry-picks or another collaboration shape instead of the child PR, I am happy to adjust. |
|
One concrete collaboration option, since this parent PR is now conflicting with current |
|
Thank you for merging #3248 into this branch. The parent attribution check now fails because squash commit |
|
Following up on the contributor-preserving landing option above: this branch remains conflicting with current |
|
I have opened the contributor-preserving current-main draft landing candidate: #3441. It keeps Dillon's eight original commits and human coauthor trailers intact with |
2 participants
what
adds human demonstration recording to cua-driver on windows:
pidandwindow_id;TRAJECTORY.md,SUMMARY.json,DEMONSTRATION.json, and relative screenshot links;DEMONSTRATION.mdguide to turn the artifact into a reusable skill.architecture
human capture is separate from ordinary trajectory recording:
RecordingSessioncontinues to record executable cua-driver tool calls.DemonstrationManagerowns one external human capture session, its writer, output directory, and MCP-session ownership.human_*action names and are not presented as automatically replayable tool calls.stop_demonstrationperforms local deterministic processing; cua-driver does not call a model API.this keeps the public workflow to two tools:
start_demonstrationandstop_demonstration.capture and privacy boundaries
text entered (redacted), without literal text, length, or character classes;this is a user-space recording indicator, not a hardware security LED. it cannot prove that another window or a secure desktop has not obscured it.
scope removed from this draft
start_recording;ureqdependency;verification
passed after rebasing onto current
main:cargo test -p input-capture— 11 tests;cargo check -p cua-driver-core --tests;cargo check -p cua-driver;cargo check -p input-capture --target x86_64-pc-windows-msvc;cargo check -p cua-driver-core --target x86_64-pc-windows-msvc;git diff --check.windows capture is implemented. macOS and Linux return
Unsupportedfor now.this update preserves Dillon DuPont's original capture, indicator, markdown, and skill-guide contribution and its authorship history while narrowing the implementation to the demonstration-recording workflow.
windows gui verification
Verified on Windows Server 2022. The fixed indicator geometry uses
DWMWA_EXTENDED_FRAME_BOUNDS, withGetWindowRectfallback and maximized work-area clamping.Live GUI sequence: normal window → corner resize → maximize → restore → stop. The border tracked the visible frame, stayed visible on all four maximized edges, and disappeared on stop.
Demo: 11.87 seconds, H.264, 1280x800, 30 fps.
pr-2016-demonstration-recording-dwm-aligned.mp4
cargo test -p input-capture: 14 passed.cargo test -p cua-driver-core demonstration: 9 passed.cargo build -p cua-driver: passed.This verifies the Windows GUI behavior on normal, resized, maximized, and restored windows.
current-main integration and verification
Rebased onto
mainat86e4f8c913521534ea61f0e4639b122ae8ca4a0bon 2026-07-29. The current runtime lifecycle owns demonstration cleanup, andstop_demonstrationis scoped to the owning session.Local verification on the rebased tree:
cargo fmt --all --check: passed;cargo check --locked -p cua-driver-core --tests: passed;cargo test -p input-capture: 11 passed;cargo test -p cua-driver-core demonstration --lib: 10 passed;git diff --check: passed.The identical implementation tree passed Windows and Linux unit/compile, Nix package/policy/unit builds, contract parity on all three platforms, generated bindings, docs, docs links, formatting, SPDX, and release metadata. Contributor attribution was rerun after correcting the integration commit metadata.