Add interactions when clicking on/in virtual screen PIPs

[waydabber]

Discussed in #1546

I often found myself accidentally trying to manipulate the contents of a PIP with my cursor. So, could a PIP capture my pointer (like some virtualisation software does) and move my cursor pointer into the dummy display?

NB: #1 Escaping the cursor from the PIP trap would need considering and all this could break some UI conventions if not done neatly. You do have the coordinates for this bi-directional translation between PIP and its host display. But there would be unexpected situations where traditionally you would expect moving out of the right of PIP 2 would enter into the left of PIP 3, but instead the cursor would pop out of PIP 2 onto the host display to then could push back into PIP 3.

But the traditional display arrangement perhaps doesn't really make sense when dealing with PIPs of Dummy displays. The above, sounds nicer at first thought to me. Perhaps a click would be needed to push the cursor into the dummy? But this seems unnecessary if the transitions happen smoothly for both push (in) and pop (out).

Normal cursor behaviour would continue if there was no PIP for a Dummy, but there could be a trap if there is no route for the cursor past a PIP of dummy 2 into a dummy 3 without a PIP. Argh. In that case the cursor should not pop back to the host display, but instead slide as normal into (the invisible) dummy 3. Programming is fun remember!

---

I think warping the mouse over with a PIP window is problematic and could be possibly disorienting so I am leaning towards not doing that (or maybe do it with some modifier key pressed while doing a long press with the mouse so it is absolutely intentional?).

[PuzzledUser]

Just for completeness, referencing a comment from: #1546 (reply in thread)

I've seen it work very nicely for virtualisation: https://www.techwalla.com/articles/how-to-move-a-mouse-out-of-virtual-pc If you pull off the cursor teleportation illusion it could be less confusing.

I've watched users lose their pointer off the primary display in a dummy display and have no idea how to find it again. It's quite a fun game actually! Keeps the mouse pointer mileage up! 😂

[waydabber]

Yes, this is something I definetly want to do - I plan to work on all PIP/Stream tagged enhancements in one big "PIP/Stream update" focused release (or series of releases) as I want to do the transition to the newer API that should serve this feature in v2.x.

[devnoname120]

It seems like https://github.com/Stengo/DeskPad implements this feature

[waydabber]

Right. Sorry about not progressing faster with newer pip/streaming related features - was planning to focus on these specifically in one of the upcoming updates. DeskPad is a valid alternative indeed.

[devnoname120]

@waydabber No worries, you're doing incredible work on BetterDisplay and I'm really grateful for this amazing tool!

[PuzzledUser]

Right. Sorry about not progressing faster with newer pip/streaming related features - was planning to focus on these specifically in one of the upcoming updates. DeskPad is a valid alternative indeed.

When you're ready to explore this, there's some great thinking shared on cursor teleportation and resolutions at Stengo/DeskPad#16

[waydabber]

Right. I have some experience transporting the mouse (some early versions of BetterDisplay actually did this in a somewhat different context but removed the feature saying it needs a bit polish and reintroduction which just never happened). :) There are multiple methods but the one I should use is an outdated one that (at least when I tried with Sonoma) did not request additional accessibility permissions - https://developer.apple.com/documentation/coregraphics/1455258-cgdisplaymovecursortopoint or https://developer.apple.com/documentation/coregraphics/1456387-cgwarpmousecursorposition. Although I am not sure how long Apple plans to keep this working like this.

[waydabber]

This is a duplicate of #612 - let's merge the two (continue there).