[computer-server] Fix coordinate scaling initialization in MCP screenshot tool

[synacktraa]

Related #898 (Issue 2)

Fixes a critical bug where coordinates from grounding models were never scaled back to actual device resolution, causing clicks to land at completely wrong locations.

The Problem:

• MCP server auto-resizes screenshots (e.g., 1440x3040 → 606x1280) before sending to grounding models
• Grounding models predict coordinates in the resized space (e.g., (414, 1069))
• These coordinates were sent directly to the device without scaling
• Clicks landed ~1575 pixels away from intended targets

The Fix:

• Call _configure_scaling() after auto-resize to initialize scale factors
• Coordinates are now properly scaled: (414, 1069) → (983, 2538)
• Clicks land at correct locations

Changes:

• computer_server/mcp_server.py: Added 4 lines after auto-resize logic to initialize coordinate scaling

---

Visual Demonstration

Original Screenshot (1440x3040)

Android home screen with Chrome and Messages icons at the bottom

---

Reproduction & Testing

Run this script to visualize the bug and fix:

demo_coordinate_fix.py (click to expand)

#!/usr/bin/env python3
# /// script
# dependencies = ["pillow"]
# ///
"""Demonstrates the coordinate scaling fix."""

from PIL import Image, ImageDraw, ImageFont

# MCP auto-resize logic
def resize_image(img, max_dim=1280):
    w, h = img.size
    if max(w, h) > max_dim:
        scale = max_dim / max(w, h)
        return img.resize((int(w * scale), int(h * scale)), Image.LANCZOS)
    return img

# Draw marker on image
def mark_point(img, x, y, label, color):
    draw = ImageDraw.Draw(img)
    # Crosshair
    s = 50
    draw.line([(x-s, y), (x+s, y)], fill=color, width=6)
    draw.line([(x, y-s), (x, y+s)], fill=color, width=6)
    draw.ellipse([(x-10, y-10), (x+10, y+10)], fill=color)
    # Label
    try:
        font = ImageFont.truetype("/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf", 36)
    except:
        font = ImageFont.load_default()
    bbox = draw.textbbox((0, 0), label, font=font)
    tw = bbox[2] - bbox[0]
    lx, ly = x + 70, y - 70
    draw.rectangle([lx-12, ly-12, lx+tw+12, ly+bbox[3]-bbox[1]+12],
                   fill=(255,255,0), outline=(0,0,0), width=4)
    draw.text((lx, ly), label, fill=(0,0,0), font=font)
    return img

# Load and process
original = Image.open("android_screenshot.png")
orig_w, orig_h = original.size
resized = resize_image(original.copy())
res_w, res_h = resized.size

# Chrome icon coordinates (from grounding model on resized image)
chrome_x, chrome_y = 414, 1069

# Calculate scale factors (what _configure_scaling() would do)
scale_x = orig_w / res_w
scale_y = orig_h / res_h

# WITHOUT FIX: coordinates sent directly (bug)
bug_x, bug_y = chrome_x, chrome_y

# WITH FIX: coordinates scaled to device space
fixed_x, fixed_y = int(chrome_x * scale_x), int(chrome_y * scale_y)

# Generate 3 images
mark_point(resized.copy(), chrome_x, chrome_y,
           "Model Predicts\n(414, 1069)", (0,150,255)).save("1_model_prediction.png")

mark_point(original.copy(), bug_x, bug_y,
           "Without Fix\n(414, 1069)\nWRONG", (255,50,50)).save("2_without_fix.png")

mark_point(original.copy(), fixed_x, fixed_y,
           f"With Fix\n({fixed_x}, {fixed_y})\nCORRECT", (50,255,50)).save("3_with_fix.png")

print(f"Original: {orig_w}x{orig_h} → Resized: {res_w}x{res_h}")
print(f"Scale: {scale_x:.2f}x, {scale_y:.2f}x")
print(f"Without fix: ({bug_x}, {bug_y}) - {((fixed_x-bug_x)**2+(fixed_y-bug_y)**2)**0.5:.0f}px off")
print(f"With fix: ({fixed_x}, {fixed_y}) - correct!")
print("Generated: 1_model_prediction.png, 2_without_fix.png, 3_with_fix.png")

Run with:

uv run demo_coordinate_fix.py

Output:

Original: 1440x3040 → Resized: 606x1280
Scale: 2.38x, 2.38x
Without fix: (414, 1069) - 1575px off
With fix: (983, 2538) - correct!
Generated: 1_model_prediction.png, 2_without_fix.png, 3_with_fix.png

---

Results

Coordinate Scaling Comparison

1. Model Prediction (Resized 606x1280) Grounding model sees resized screenshot and predicts Chrome icon at (414, 1069)

2. Without Fix (Bug) (Original 1440x3040) Coordinates sent directly to device: (414, 1069) ❌ Click lands 1575px away from target

3. With Fix (Original 1440x3040) Coordinates scaled to device: (983, 2538) ✅ Click lands on Chrome icon

---

Technical Details

Before (Bug):

# Screenshot auto-resized but scaling never initialized
img = img.resize((new_width, new_height), PILImage.Resampling.LANCZOS)
# _scale_x and _scale_y remain at default 1.0
# Coordinates are NOT scaled when clicking

After (Fixed):

# Screenshot auto-resized AND scaling initialized
img = img.resize((new_width, new_height), PILImage.Resampling.LANCZOS)

# Initialize coordinate scaling if not already configured
if not _target_width:
    _configure_scaling(target_width=new_width, target_height=new_height)
# Now _scale_x = 2.38, _scale_y = 2.38
# Coordinates are properly scaled: (414, 1069) * 2.38 = (983, 2538)

Impact:

• Fixes grounding model clicks on all platforms (Android, desktop, tablets)
• Most severe on high-resolution mobile devices (1440x3040, 1080x2340, etc.)
• Existing coordinate scaling infrastructure now actually works

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Review	Updated (UTC)
docs	Ready	Preview, Comment	Jan 29, 2026 2:23pm

[sentry]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!