Web: Mobile Safari support

[rebornix]

This is a meta issue for our Safari (on iOS/iPadOS) support (issue list) and this can be used to track the limitations and gaps we see today on mobile / touch screens.

updated Oct 2020.

To enable users to use the Web UI in Safari on iOS for some basic work, we need to fix the following blocking issues

Blockers

• ✋ Missing wheel event iPad OS trackpad scroll / mouse wheel support  #106232 (upstream https://bugs.webkit.org/show_bug.cgi?id=210071 )
• Blockers on Safari desktop Web: Safari on macOS testing #108637
• 🏃 Workbench size inconsistent with visual viewport Workbench miss relayout after viewport size changes #108267
• 🏃 Port forwarding blocked https://github.com/MicrosoftDocs/vscodespaces/issues/728
  • No authentication flow as it can't open a new window
  • can not debug extensions
• UI affordance fallback for cursor: styles Safari/iPadOS: UI affordance for non default cursor style #108832
• Drag & Drop views/panels with mouse events https://dvcs.w3.org/hg/d4e/raw-file/tip/mouse-event-test.html

Bugs

• Touch/pointers not working (Settings Editor dropdown, view header, etc) https://github.com/microsoft/vscode/issues?q=is%3Aopen+label%3Atouch%2Fpointer+label%3Aios
• MISC https://github.com/microsoft/vscode/issues?q=is%3Aopen+label%3Aios+label%3Abug+-label%3Atouch%2Fpointer

---

With above items finished, users can use the Web UI for basic tasks if they us an external keyboard and mouse. However if you are not using an external keyboard with Escape and Function keys, or a mouse/pointer device, you can easily find the Web UI difficult to use and you can't be as productive as on the desktop. Major blockers are:

• Escape. Escape key is the major key for hiding UI elements in the workbench. For native elements, iOS may know how to map Ctrl+[ to behave the same as Escape, however for our keybinding service, we always check keyboard event, which doesn't understand Ctrl+[ at all.
  • To mimic the native smart keyboard experience, we can build a Keymap extension, which remaps Ctrl+[ to commands which used to be triggered by Escape.
• Function keys. Without Fn keys, quite a lot of our core features are not highly accessible: Command Palette (F1), Rename (F2), Next Match (F3), Next Reference (F4), Debug (F5, F9, F10, F11), Next Problem (F8), Go to Definition/Impl (F12), etc.
• Hover is missing. Definition/Implementation hovers in editor, View title bar actions, List View quick actions, etc rely on Hover events triggered from mouse move. They are hidden by default and only visible and actionable when hover happens (or sometimes even worse, hover with keyboard shortcuts). On a touch screen, these features are not accessible.
• Debug
  • window.open only works if it's triggered from user click and synchronous. For debugging VS Code extension in the Web, it's silently broken as the window will not be opened without any error message.
• Most UI elements are small. This is exacerbated by even the smallest mouse cursor being a rather large circle.
  • Button/Checkbox/ActionbarItem are 10-20 px wide. It requires users to be very precise while acting on them.
  • Default list view height is small and not customizable.
  • Editor gutter consists of glyph area (we put breakpoints on it), line number and margin (we put folding icon on it) and users can easily mis-click/touch anyone of them.
  • Sashes are thin and no indicator for clicking or dragging

[rebornix]

Move this item to backlog as there are blockers that we want to track.

[kieferrm]

The following comes from a short test-run with @rebornix of GH Codespaces in Safari on iPad. Some of the issues are caused by the VS Code WebUI being embedded in an iframe. This will change in the near future as we’ll move out of iFrame.

IFrame issues

1. The workbench is too big; the workbench resizes according to the size of the container which currently is the iframe, however once we are out of the iframe it seems we won't be able to get the minimized browser chrome or we size the workbench by the amount the minimized browser chrome is smaller than the regular one.
2. When switching apps and coming back to the browser tab that contains the codespace then the editor appears as having focus but does not receive any keyboard events.

Other issues

3. When switching apps and coming back to the browser tab that contains the codespace, the codespace needs to reconnect.
4. Scrolling inside the editor or any view neither works vertically nor horizontally, all scroll events are going to the browser.
5. When having my sidebar on the left, I cannot grap the editor scrollbar on the right because it's covered by the native scrollbar.
6. When having my sidebar on the right, trying to grab the scrollbar thumb with the mouse is hard. When I place the mouse in the middle of the scrollbar width then I grab the sash. The mouse pointer needs to be only halfway over the scrollbar.
7. When working with the magic keyboard, every time the editor has focus the screen keyboard affordance comes up. Is there any way to avoid this from happening?
8. We're using the native menus for dropdowns such as Terminal or Debugging.
9. Only native Copy&Paste works which loses formatting information (and is non-trivial visually “read”). I also have to unbind the keyboard shortcuts to make that work with the keyboard.

Polish

10. We should think about mimicking the native controls. I.e. the mouse pointer disappears in favor of a highlighted element.

[kieferrm]

@grorg you mentioned on twitter you’d be interested in helping to improve the safari experience. 5, 6, 7, 9 seem that they could benefit from your help.

[rebornix]

@grorg the first time when we tested iPadOS 13.4 we found that scrolling with trackpad doesn't work and we thought it's because we didn't handle the events correctly. Turns out to be that it's a bug in webkit (tracked as https://bugs.webkit.org/show_bug.cgi?id=210071 ) and currently with our virtual rendering, we can't workaround this issue. It would be great if this issue can be tackled, thanks in advance!

[mjbvz]

Hi @grorg, if you are still looking for webkit bugs effecting VS Code, this one came up recently: https://bugs.webkit.org/show_bug.cgi?id=215589 (it may be a duplicate)

This bug prevents out extension detail pages, which use webviews with script disabled, from showing in safari. I tried to work around it last iteration but ended up causing a regression. I'm going to try another approach this iteration but it would be great it the solution we use today worked in Safari as well

[VincentDondain]

Hi,

The workbench is too big; the workbench resizes according to the size of the container which currently is the iframe, however once we are out of the iframe it seems we won't be able to get the minimized browser chrome or we size the workbench by the amount the minimized browser chrome is smaller than the regular one.

@rebornix, @kieferrm What is the ETA on moving away from using an iFrame? (edit: it looks like deiframing already shipped no?)
To clarify, is your comment affecting the following?

Render workbench in visual viewport (to avoid being covered by on screen keyboard, or assistive editor)

I am still seeing the virtual keyboard overlaying the cursor on iPadOS.

Also, is this meta issue the place to track adding support for iPadOS 13.4 pointer interactions?

[rebornix]

I am still seeing the virtual keyboard overlaying the cursor on iPadOS.

An embedder (e.g., codespace) needs to listen to viewport size change and then resize the vscode workbench.

Also, is this meta issue the place to track adding support for iPadOS 13.4

@VincentDondain can you help elaborate a bit on this one? I thought it's for UIKit but not Web elements.

[VincentDondain]

@rebornix ok thanks for the info I'll check if we're resizing the workbench properly on our end (;

For the pointer interactions: I thought I read they added special features to WebKit/Safari but I think they're only saying use PointerEvents which we hopefully already do.

For reference: https://webkit.org/blog/10247/new-webkit-features-in-safari-13-1/

[alefragnani]

Just to update that issues 4 and 5, commented in #85254 (comment) remains not working on iPadOS 14.

In my tests, I’m using a regular bluetooth mouse, instead of a Trackpad.

[VincentDondain]

@alefragnani I believe the trackpad issue is being tracked here: #106232

[alefragnani]

@VincentDondain It appears to be. The only problem/misinformation that I see about that issue, it only describes about trackpad, not mouse wheel. The WebKit issue, on the other hand, talks about both.

I would ask you to update that issue, correctly describing it handles both scenarios.

Thank you

[kieferrm]

@rebornix pls see also https://github.com/github/codespaces/issues/1345

[k8n]

The upstream WebKit issue now appears fixed.