feat(skills): add screen-recording awareness skill #8011
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| --- | ||
| name: "Screen Recording" | ||
| description: "Capture screen recordings during computer-use sessions. Records the display or a specific window as H.264 MP4 video with optional audio." | ||
| user-invocable: false | ||
| disable-model-invocation: false | ||
| metadata: | ||
| vellum: | ||
| emoji: "🎥" | ||
| os: ["darwin"] | ||
|
Comment on lines
+6
to
+9
There was a problem hiding this comment.
The skill loader only parses Useful? React with 👍 / 👎. |
||
| --- | ||
|
|
||
| # Screen Recording | ||
|
|
||
| You have access to a screen recording capability that captures what happens on the user's Mac during computer-use sessions. | ||
|
|
||
| ## How It Works | ||
|
|
||
| - **Automatic in QA mode**: When a QA/test session starts, recording begins automatically before any destructive actions (clicks, typing, etc.) | ||
| - **Can be requested explicitly**: Sessions can be configured with `requiresRecording: true` to enable recording outside of QA mode | ||
|
There was a problem hiding this comment.
This instruction says Useful? React with 👍 / 👎. |
||
| - **Recording gate**: When recording is required, destructive actions are blocked until the first video frame is confirmed captured | ||
|
|
||
| ## Recording Details | ||
|
|
||
| - **Format**: H.264 MP4 video at 30 fps | ||
| - **Resolution**: 1920×1080 | ||
| - **Bitrate**: 4 Mbps video, 128 kbps AAC audio (when audio is enabled) | ||
| - **Capture scope**: Either the full display or a specific window | ||
| - **Storage**: Files are saved to `~/Library/Application Support/vellum-assistant/recordings/` | ||
| - **Naming**: `qa-recording-{timestamp}.mp4` | ||
|
|
||
| ## Health Checks | ||
|
|
||
| A first-frame handshake verifies the capture pipeline is healthy within 5 seconds of starting. If no frames arrive: | ||
| - **Required recording**: The session fails immediately with a clear error | ||
| - **Optional recording**: A warning is shown but the session continues | ||
|
|
||
| ## After Recording | ||
|
|
||
| When a recording completes: | ||
| 1. The video file is saved to disk | ||
| 2. A file-backed attachment is created (metadata in DB, file stays on disk) | ||
| 3. The attachment is linked to the originating chat message | ||
| 4. The video appears inline in the conversation for playback | ||
|
|
||
| ## Retention | ||
|
|
||
| Recordings have an expiration timestamp (default: 7 days, configurable). An automatic cleanup worker runs periodically (every 6 hours) and deletes expired recording files from disk. | ||
|
|
||
| ## Limitations | ||
|
|
||
| - Requires Screen Recording permission in System Settings > Privacy & Security | ||
| - Only available on macOS | ||
| - One recording at a time per session | ||
| - Recording must be stopped explicitly (or stops when the session ends) | ||
| - Large recordings consume disk space until cleanup runs | ||
|
|
||
| ## When to Mention Recording | ||
|
|
||
| - Tell users their QA session is being recorded when relevant | ||
| - Offer to analyze recordings using the media-processing skill (keyframe extraction, event detection) | ||
| - Mention retention period if users ask about storage or cleanup | ||
| - If recording fails, explain the specific error (permission denied, no display found, etc.) | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔴 Multi-line YAML metadata not parsed by line-based frontmatter parser, losing os constraint and emoji
The
metadatafield in the new SKILL.md uses multi-line nested YAML syntax, but the frontmatter parser (assistant/src/skills/frontmatter.ts:28-62) is a simple line-by-line key-value parser that splits on:. It does not support multi-line YAML or nested structures.Root Cause and Impact
When the parser encounters:
It parses each line independently, producing:
metadata→""(empty string)vellum→""(empty string)emoji→"🎥"os→["darwin"]Since
metadatais an empty string, the check atassistant/src/config/skills.ts:309(if (metadataRaw)) evaluates tofalseand skips JSON parsing entirely. Themetadatafield on the skill will beundefined.This causes two problems:
os: ["darwin"]constraint is lost — the skill will appear available on all platforms instead of only macOS. The OS eligibility check atassistant/src/config/skills.ts:149-158will never fire becausevellum.osis undefined.🎥is lost — the skill won't display its intended icon.Every other skill in the codebase uses single-line JSON for metadata (e.g.,
metadata: {"vellum": {"emoji": "🍎", "os": ["darwin"]}}inassistant/src/config/bundled-skills/macos-automation/SKILL.md).Was this helpful? React with 👍 or 👎 to provide feedback.