fix: cursor clicks on input text fields #763
Conversation
WalkthroughAdds element-relative click handling for INPUT/TEXTAREA: the renderer emits relative coordinates, and the server clicks at element-local offsets using bounding boxes. On successful input/textarea clicks, the server skips DOM snapshot generation. Non-input clicks and general behavior remain unchanged; SELECT handling is unaffected in this branch. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant U as User
participant R as DOMBrowserRenderer
participant S as Server inputHandlers
participant P as Page/Browser
U->>R: Click in iframe
alt Target is INPUT/TEXTAREA
R->>S: dom:click(target=INPUT/TEXTAREA, coords=relative)
S->>P: locate element by selector/handle
alt boundingBox available
S->>P: mouse.click(bbox.x + rel.x, bbox.y + rel.y)
Note right of S: Skip DOM snapshot (early return)
else Fallback
S->>P: page.click(selector)
Note over S,P: Warning logged on error path
end
else Other elements (non-SELECT)
R->>S: dom:click(coords=absolute)
S->>P: page.click(selector or coords)
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (3)
src/components/recorder/DOMBrowserRenderer.tsx (1)
575-589: Element-relative clicks for inputs/textareas look good; clamp and fallback for robustnessGreat fix. To harden against fractional/negative offsets and degenerate rects, clamp to the element box and fallback to absolute coords if bbox is 0×0.
- const element = target as HTMLElement; - const elementRect = element.getBoundingClientRect(); - const relativeX = iframeX - elementRect.left; - const relativeY = iframeY - elementRect.top; + const element = target as HTMLElement; + const { left, top, width, height } = element.getBoundingClientRect(); + let relativeX = iframeX - left; + let relativeY = iframeY - top; + if (width > 0 && height > 0) { + relativeX = Math.max(0, Math.min(relativeX, width - 1)); + relativeY = Math.max(0, Math.min(relativeY, height - 1)); + } else { + // Fallback to iframe coordinates if the rect is not usable + socket.emit("dom:click", { + selector, + url: snapshot.baseUrl, + userId: user?.id || "unknown", + elementInfo, + coordinates: { x: iframeX, y: iframeY }, + isSPA: false, + }); + return; + }server/src/browser-management/inputHandlers.ts (2)
639-661: Prefer elementHandle.click({ position }) to honor element-local offsets and auto-scrollCurrent approach works, but using ElementHandle.click with a position lets Playwright handle scrolling/occlusion and avoids manual page.mouse math. Optionally clamp when bbox is available.
- try { - const elementHandle = await page.$(selector); - if (elementHandle) { - const boundingBox = await elementHandle.boundingBox(); - if (boundingBox) { - await page.mouse.click( - boundingBox.x + coordinates.x, - boundingBox.y + coordinates.y - ); - } else { - await page.click(selector); - } - } else { - await page.click(selector); - } - } catch (error: any) { + try { + const elementHandle = await page.$(selector); + if (elementHandle) { + const bbox = await elementHandle.boundingBox(); + const pos = bbox + ? { + x: Math.max(0, Math.min(coordinates.x, Math.max(0, bbox.width - 1))), + y: Math.max(0, Math.min(coordinates.y, Math.max(0, bbox.height - 1))), + } + : { x: Math.max(0, coordinates.x), y: Math.max(0, coordinates.y) }; + await elementHandle.click({ position: pos }); + } else { + await page.click(selector); + } + } catch (error: any) { logger.log("warn", `Failed to click at coordinates: ${error.message}`); await page.click(selector); }
709-716: Align type to include isSPA in onDOMClickActionThe renderer can send isSPA; include it in this type for consistency and editor tooling.
data: { selector: string; url: string; userId: string; elementInfo?: any; coordinates?: { x: number; y: number }; + isSPA?: boolean; },
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (2)
server/src/browser-management/inputHandlers.ts(1 hunks)src/components/recorder/DOMBrowserRenderer.tsx(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
server/src/browser-management/inputHandlers.ts (1)
maxun-core/src/browserSide/scraper.js (1)
selector(28-28)
🔇 Additional comments (1)
server/src/browser-management/inputHandlers.ts (1)
668-672: Early-return to skip DOM snapshot for inputs is sensibleThis avoids snapshot thrash while typing. Ensure a later, debounced snapshot (e.g., after key idle) exists so the UI eventually reflects caret/value changes.
2 participants
What this PR does?
Fixes the issue wherein the user cursor clicks between characters were not being registered when performing keyboard actions. This was leading to wrong data being input.
2025-09-03.00-28-46.mov
Summary by CodeRabbit
New Features
Bug Fixes
Performance