feat(cua-driver-rs)(macos): expose get_window_state element bounds - #1820
feat(cua-driver-rs)(macos): expose get_window_state element bounds#1820cyq1017 wants to merge 2 commits into
Conversation
|
@cyq1017 is attempting to deploy a commit to the Cua Team on Vercel. A member of the Team first needs to authorize it. |
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThis PR adds on-screen rectangle tracking to AX elements across the macOS accessibility layer. The ChangesScreen Rectangle Tracking Pipeline
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs`:
- Around line 77-90: The node serialization in get_window_state (the
nodes.iter() filter_map using node.screen_rect, bounds, scale_x, scale_y) must
skip nodes that do not intersect the captured window and clip those that
partially overlap: compute the intersection between node.screen_rect
[x,y,width,height] and bounds, skip if intersection width or height <= 0,
otherwise use the intersection rect (left = max(x,bounds.x), top =
max(y,bounds.y), inter_w = min(x+width, bounds.x+bounds.width)-left, inter_h =
min(y+height, bounds.y+bounds.height)-top) and then convert to window-local
pixels using ((left - bounds.x) * scale_x).round() etc for x/y and (inter_w *
scale_x).round() etc for width/height before serializing; update the filter_map
around node.element_index and node.screen_rect accordingly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 455a218a-553d-45a6-8f9e-42b259ea1766
📒 Files selected for processing (3)
libs/cua-driver/rust/crates/platform-macos/src/ax/cache.rslibs/cua-driver/rust/crates/platform-macos/src/ax/tree.rslibs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs
| nodes | ||
| .iter() | ||
| .filter_map(|node| { | ||
| let idx = node.element_index?; | ||
| let [x, y, width, height] = node.screen_rect?; | ||
| Some(serde_json::json!({ | ||
| "element_index": idx, | ||
| "x": ((x - bounds.x) * scale_x).round() as i64, | ||
| "y": ((y - bounds.y) * scale_y).round() as i64, | ||
| "width": (width * scale_x).round() as i64, | ||
| "height": (height * scale_y).round() as i64, | ||
| })) | ||
| }) | ||
| .collect() |
There was a problem hiding this comment.
Skip or clip nodes that fall outside the captured window.
walk_tree still includes non-window children when window_id is set (libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs Lines 147-160), so not every actionable AXNode belongs to the screenshot. This helper currently serializes those nodes anyway, which can produce negative/off-image elements[] coordinates and violate the “window-local screenshot pixels” contract. Filter to rects that intersect the window bounds and clip partial overlaps before scaling.
Suggested fix
fn build_structured_elements(
nodes: &[AXNode],
bounds: &WindowBounds,
screenshot_width: u32,
screenshot_height: u32,
) -> Vec<Value> {
if bounds.width <= 0.0 || bounds.height <= 0.0 {
return Vec::new();
}
let scale_x = screenshot_width as f64 / bounds.width;
let scale_y = screenshot_height as f64 / bounds.height;
+ let window_left = bounds.x;
+ let window_top = bounds.y;
+ let window_right = bounds.x + bounds.width;
+ let window_bottom = bounds.y + bounds.height;
nodes
.iter()
.filter_map(|node| {
let idx = node.element_index?;
let [x, y, width, height] = node.screen_rect?;
+ let left = x.max(window_left);
+ let top = y.max(window_top);
+ let right = (x + width).min(window_right);
+ let bottom = (y + height).min(window_bottom);
+
+ if right <= left || bottom <= top {
+ return None;
+ }
+
Some(serde_json::json!({
"element_index": idx,
- "x": ((x - bounds.x) * scale_x).round() as i64,
- "y": ((y - bounds.y) * scale_y).round() as i64,
- "width": (width * scale_x).round() as i64,
- "height": (height * scale_y).round() as i64,
+ "x": ((left - window_left) * scale_x).round() as i64,
+ "y": ((top - window_top) * scale_y).round() as i64,
+ "width": ((right - left) * scale_x).round() as i64,
+ "height": ((bottom - top) * scale_y).round() as i64,
}))
})
.collect()
}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@libs/cua-driver/rust/crates/platform-macos/src/tools/get_window_state.rs`
around lines 77 - 90, The node serialization in get_window_state (the
nodes.iter() filter_map using node.screen_rect, bounds, scale_x, scale_y) must
skip nodes that do not intersect the captured window and clip those that
partially overlap: compute the intersection between node.screen_rect
[x,y,width,height] and bounds, skip if intersection width or height <= 0,
otherwise use the intersection rect (left = max(x,bounds.x), top =
max(y,bounds.y), inter_w = min(x+width, bounds.x+bounds.width)-left, inter_h =
min(y+height, bounds.y+bounds.height)-top) and then convert to window-local
pixels using ((left - bounds.x) * scale_x).round() etc for x/y and (inter_w *
scale_x).round() etc for width/height before serializing; update the filter_map
around node.element_index and node.screen_rect accordingly.
|
closing as superseded by #1961 |
2 participants
Refs #1564
Summary
get_window_state.elements[]bounds in window-local screenshot pixelstree_markdownunchanged and skip nodes without a stableelement_indexor geometryVerification
DYLD_FALLBACK_LIBRARY_PATH=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift-5.5/macosx cargo test -p platform-macos --lib get_window_state::tests::build_structured_elements -- --nocaptureDYLD_FALLBACK_LIBRARY_PATH=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift-5.5/macosx cargo test -p platform-macos --lib -- --nocaptureDYLD_FALLBACK_LIBRARY_PATH=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift-5.5/macosx cargo check -p platform-macosrustfmt --edition 2021 --check crates/platform-macos/src/ax/cache.rs crates/platform-macos/src/ax/tree.rs crates/platform-macos/src/tools/get_window_state.rsgit diff --checkSummary by CodeRabbit
New Features
Tests