change(web): now utilizes 'inputMode="none"' on supported touch devices 📴

[jahorton]

Addresses #3030.

This will remove our old "touch alias element" workaround for touch devices; said workaround predates widespread support inputMode attribute, hence why it's stuck around for so long. Note current compatibility charts: https://developer.mozilla.org/en-US/docs/Web/HTML/Global_attributes/inputmode#browser_compatibility

That said, we're not 100% sure which version of Safari is actually needed for proper compatibility: iOS 12.2 Safari doesn't interpret "none" in the manner we expect. Current iOS versions definitely do, though.

Note that this will cause a break in behavior for older devices than those indicated above. It's not the "best" thing, but we're doing this for a number of reasons - this allows the Keyman Engine for Web to handle input on touch-based devices much better than before.

• Selection of text in KMW-enabled inputs is now supported.
• "Flash of unstyled content" issues that arose when rotating affected devices and/or swapping between keyboards with significantly different styling are now eliminated. (Well, at least the ones that KMW's touch-aliasing was causing.)
  • Since we had resorted to "fighting the browser" to prevent auto-display of a device's system keyboard to ensure our keyboards were properly displayed, browsers were unable to provide typical integrations with forms, alternate forms of input like accessibility and/or dictation services, etc where touch-alias elements were used.
• Scrolling complications are eliminated.
• This removes a significant hurdle toward rich text editing.
• Newly-handled: contentEditable elements - see 95706dc
• Newly-handled: non-design-mode iframes (and their contents) - see feaa353
  • This one required removal of numerous prevention checks and a touch of extra support in related touch-event handlers.

Note that design-mode iframes (which are used for some forms of rich-text editing) are still off-limits due to some internal architectural issues for related event-handler setup. I imagine that it would be possible to address, but the changes there would be too significant to "fold in" to this PR. (Such changes would deserve their own focused PR for review.)

---

While this does drop a feature used by notably old browsers, based on the notes above, we've determined that continuing to maintain the feature would come at too high of an ongoing cost to the majority of users. While it might be possible to instead add some sort of auto-detection to selectively enable touch-aliasing when needed for older devices, adding that would further increase complexity and the maintenance load involved, especially for parts managing the element types listed above. Note the scale of the changeset - there's been quite a significant amount of code devoted to maintaining the status quo on this.

User Testing

This is a very notable change to KMW as used in web browsers, rather than in the mobile apps. As such, user testing will be focused on such environments.

Test Environments

• GROUP_ANDROID: Use any Android 8.0+ mobile device (real or emulated)
• GROUP_IOS: Use any iOS 13.0+ mobile device (real or emulated)
  • Important note: if using emulated iOS, make sure that Simulator is set to display the device's keyboard.
    • Hardware keyboard shortcut: ⇧ + ⌘ + K. Use that once or twice within a standard iOS app (i.e., not Keyman or a KMW page) until the keyboard shows, then perform the tests.

Use the KeymanWeb Test Home link below for all user tests!

Test Specs

• TEST_KEYBOARD_DISPLAY: Using the "Test unminified Keymanweb" test page, verify that only one keyboard is ever displayed at a time.

  1. Select the text box at the top ("Type here") as if to enter text. KMW's OSK should appear.
  2. Select the next text box ("or here") as if to enter text. KMW's OSK should remain visible.
  3. Select one of the small text boxes next to an "Add" button. KMW's OSK should disappear and the system keyboard should appear.
  4. Reselect the text box at the top ("Type here"). The system keyboard should disappear and KMW's OSK should appear.
  5. Tap a part of the page that does not receive text input. KMW's OSK should disappear.
  6. Select the second text box ("or here"). KMW's OSK should reappear.

  Note: it is possible that the OSK may be partially obscured by parts of the browser at the bottom of the web page - especially on iOS/Safari.

  • Do not fail this test if the full OSK would have been visible if the browser's UI were not covering the unseen parts.

    Passing screenshot

    

    Note how the address bar is covering up part of the spacebar. Recent versions of iOS/Safari have... interesting behaviors, shall we say, that make it very difficult to 100% prevent this.

  • Do fail this text if the OSK would have been partially offscreen anyway, or if there is empty space below it that is not filled by browser UI (that is, if part of the webpage itself is visible underneath it).

• TEST_CONTENT_EDITABLE: Using the "Tests the new Attachment/Enablement API functionality" test page link, verify that interactions with content-editable elements feel natural on touch devices.

  1. Ensure that you are "currently testing under the auto attachment mode." (First sentence of the page's second paragraph.) Otherwise, click the "Auto" link that should be present.

  2. Click the "Create editable DIV" button once - the last button from the set of five.

  3. Scroll down the page - you should see a bordered area containing the text "Edit me!" underneath the contenteditable Elements header. An example screenshot (for phones):

     

  4. Scroll the page so that the "Edit me!" section is displayed just above the bottom of the page's border within the browser / at the bottom of your device's screen.

  5. Touch the "Edit me!" text as if you wished to type there.

     • And here the magic happens.
     • The OSK should appear when you tap the "Edit me!" text.
     • Secondly, the page should automatically scroll to keep the "Edit me!" text visible. It will likely be visible just above the top of the OSK.
     • Fail this user test if either statement above does not occur for you when testing.
       • Note: during development, a "slow tap" sometimes caused the OSK to flash visible briefly then disappear, likely because of interactions with the OSK being displayed underneath the active touch event. Quick taps avoided this.

  6. Swap to the Lao keyboard and type a letter or two. Verify that the keyboard operates normally.

• TEST_STANDARD_IFRAME: Using the "Tests the new Attachment/Enablement API functionality" test page link, verify that interactions with "normal" IFrame elements operates as expected.

  1. Ensure that you are "currently testing under the auto attachment mode." (First sentence of the page's second paragraph.) Otherwise, click the "Auto" link that should be present.
  2. Click the "Create IFrame" button once - the third button from the set of five.
  3. Scroll the page so that the text input area (showing a gray "This is only a test!") is displayed just above the bottom of the page's border within the browser / at the bottom of your device's screen.
  4. Touch the text input area as if you wished to type there.
     • And here the magic happens.
     • The OSK should appear when you tap it.
     • Secondly, the page should automatically scroll to keep the text input area visible; it will likely be visible just above the top of the OSK.
     • Fail this user test if either statement above does not occur for you when testing.
       • Note: during development, a "slow tap" sometimes caused the OSK to flash visible briefly then disappear, likely because of interactions with the OSK being displayed underneath the active touch event. Quick taps avoided this.
  5. Swap to the Lao keyboard and type a letter or two. Verify that the keyboard operates normally.

• TEST_INPUT_FORMATTING: Using the "Test unminified Keymanweb" test page, verify that font formatting for the active keyboard is properly applied.

  1. Click on the input area ("or here") to begin input.
  2. Select the first available keyboard for English, then type "the".
  3. Select the keyboard for Khmer. Input formatting should shift in the following ways:
     • The input area should become taller.
     • The font should change significantly.
  4. Click the text area ("type here"). Its formatting may also change.
  5. Select the keyboard for Norwegian. The formatting for the in the input area should shift again.
  6. For comparison, select the first available keyboard for English. the should become narrower.

• TEST_DRAG_SELECT: Using the "Test unminified Keymanweb" test page, verify that the OSK is not hidden when selecting text on mobile devices.

  1. Click on the input area ("or here") to begin input.
  2. Select the first available keyboard for English, then type "there is a".
  3. Select the is part of the test you just typed. Verify that the OSK did not disappear during or after selection of that part of your input.

There's a strong argument to be made for more unit tests, though that may be best addressed via Web regression testing at the end of the chain.

Ideas for more:

• external keyboard use with mobile devices should still work (and still not show the default system keyboard)

Other notes:

Known bug:

• _DISABLED_TEST_CONTEXT_AND_PREDICTIONS: Using the "Predictive Text: robust testing" test page, verify that interactions between moving the caret and text predictions works correctly.

  1. Click on the text area and type trials and.
  2. Place the caret after the t of trials. The suggestions for and should disappear and be replaced by predictions based on t.
  3. Select one of the resulting suggestions - it should continue from t to become the full word, with text to the right remaining after the whitespace added by the suggestion.

This (disabled) test will fail because standard elements do not reset predictive context when the caret is relocated. In fact... there's a fair shot they may not properly be updating context at all. Either way, this matches current behavior on master.

[keymanapp-test-bot]

User Test Results

Test specification and instructions

• ✅ GROUP_ANDROID: Use any Android 8.0+ mobile device (real or emulated)

  5 tests PASSED

  • ✅ TEST_KEYBOARD_DISPLAY (PASSED): Tested this in 'Test unminified Keymanweb' test page in the Android Mobile device (Ver 11) and the OSK is working as expected. However, after re-selecting the text box at the top ("Type here", the OSK shrink its from actual size on the page. (notes)
  • ✅ TEST_CONTENT_EDITABLE (PASSED): Tested this in "Tests the new Attachment/Enablement API functionality" test page link in my Android Mobile device and noticed that the Auto attachment mode appear by default. Able to see the OSK after touching the 'Edit me!' text. Able to type some letters in Lao after swaping the keyboard from English. (notes)
  • ✅ TEST_STANDARD_IFRAME (PASSED): Tested this in "Tests the new Attachment/Enablement API functionality" test page, and OSK appears after I touching the 'text input' area. Also, the page is automatically scrolling to keep the text input are visible. Seems to be working as expected. (notes)
  • ✅ TEST_INPUT_FORMATTING (PASSED): Testd this in "Test unminified Keymanweb' test page and verified that the font formatting for the active keyboard is properly applied. ie., when using the Khmer Keyboard the font format is entirely different from the English Font format. The Input text area is also taller in size. When compare to Norwegian font format the English font format is become narrower. Seems to be working as expected. (notes)
  • ✅ TEST_DRAG_SELECT (PASSED): Tested this in API 26 / Android 8.0 emulator and verified that the OSK is still appear during or after selecting the part of the input. (notes)

• ✅ GROUP_IOS: Use any iOS 13.0+ mobile device (real or emulated)

  5 tests PASSED

  • ✅ TEST_KEYBOARD_DISPLAY (PASSED): Tested this in the "Test unminified Keymanweb" test page in iOS 13.7 / iPhone Simulator and the OSK appears on the screen when I select "Type here" / "or here" text boxes. And it does not appear If I tap a part of the page. The system keyboard appear if I select the text boxes near the "Add" button. (notes)
  • ✅ TEST_CONTENT_EDITABLE (PASSED): Tested this in "Tests the new Attachment/Enablement API functionality" test page and here is my observations: Clicking the on the "Edit me!" text shows OSK appear on the View. However, there is a space appearing at the right side of the OSK on the screen. (notes)
  • ✅ TEST_STANDARD_IFRAME (PASSED): Tested this as per the instructions and it is working as expected. (notes)
  • ✅ TEST_INPUT_FORMATTING (PASSED): Tested this in "Test unminified Keymanweb" test page and it is working as expected. (notes)
  • ✅ TEST_DRAG_SELECT (PASSED): Tested this as per the instructions and verified that the OSK is still appear during or after selecting the part of the input. (notes)

Test Artifacts

• Android
  • Keyman for Android apk
  • FirstVoices Keyboards for Android apk
  • KMSample1 apk
  • KMSample2 apk
• Developer
  • Keyman Developer
  • Compiler Regression Tests
  • kmcomp.zip
• iOS
  • Keyman for iOS (simulator image)
  • FirstVoices Keyboards for iOS (simulator image)
  • TestFlight internal PR build version - 16.0.77 (0.7343.7066)
• Keyboards
  • Test Keyboards
• Web
  • KeymanWeb Test Home
• Windows
  • Keyman for Windows
  • FirstVoices Keyboards for Windows

[jahorton]

I've found some errors after detaching page elements on touch, so that needs addressing before merge.

Later edit: and, resolved.

[jahorton]

Found a couple more bits to remove and fixed an issue with touch-based text selection within attached inputs causing the OSK to hide.

[jahorton]

GROUP_IOS: Use any iOS 12.4+ mobile device (real or emulated)

* **TEST_KEYBOARD_DISPLAY (FAILED):** Before I started the test, I pressed ⇧ + ⌘ + K in the iPhone7 simulator and followed the steps as it mentioned in the Keyboard Display Test. After clicking the Add button in the 'Test Unminified Keymanweb' test page, the system keyboard still appears above the Keyman Keyboard.
   <img alt="" width="300" src="https://user-images.githubusercontent.com/19683143/193248885-6098673f-e047-4a9f-bcce-3df7c02c02bc.png">

* **TEST_CONTENT_EDITABLE (FAILED):** Here, I noticed that two Keyboards are appearing after clicking the Edit me! text. Also, long pressing the globe key shows only one keyboard (EuroLatin (SIL) in the Keyman menu.
   <img alt="" width="300" src="https://user-images.githubusercontent.com/19683143/193248899-d38a2ca0-b35b-4ce4-b8e9-5b91d0dfc7e3.png">

* **TEST_STANDARD_IFRAME (FAILED):** Not able to swap to Lao keyboard since it is showing only EuroLatin (SIL) keyboard after long pressing the globe key.
   <img alt="" width="300" src="https://user-images.githubusercontent.com/19683143/193248920-01c248fe-3bab-4e0b-8145-4a289992d233.png">

* **TEST_INPUT_FORMATTING (FAILED):** Not able to swap to Lao keyboard since it is showing only EuroLatin (SIL) keyboard after long pressing the globe key.
   <img alt="" width="300" src="https://user-images.githubusercontent.com/19683143/193248958-29b39c16-a7d5-40d0-a517-e519a542631d.png">

Which version of iOS simulator did you use for this test, @bharanidharanj? We've noticed that we don't actually know the minimum version this should work for, so if it's 12.4... well, that's useful info, at least.

Still waiting on a response to this, @bharanidharanj.

[jahorton]

Finally bothered trying to check the Safari version requirement via the Xcode's iOS Simulator.

In iOS 12.4: 🟥

In iOS 13.2: ✅

Here, there's even a period where neither is shown (as the KMW OSK loads). It's clearly an "either-or" situation, unlike with 12.4.

I'd need to go downloading extra runtimes to provide a more precise investigation, but I get the feeling this means that the breaking point is iOS 13.

---

Also, I'm not sure if it's a Simulator-only thing, but "connecting" a hardware keyboard instantly disables the "software" (visual) keyboard on iOS. This prevented the odd Android-Chrome scenario mentioned in one of the comment threads - the issue where typing a physical keystroke force-displayed the OSK, regardless of inputMode setting.

[jahorton]

One download later... suspicion confirmed:

[jahorton]

Now that we have a better idea what the minimum iOS version needed is...

@keymanapp-test-bot retest GROUP_IOS TEST_KEYBOARD_DISPLAY TEST_CONTENT_EDITABLE TEST_STANDARD_IFRAME TEST_INPUT_FORMATTING

[bharanidharanj]

GROUP_IOS: Use any iOS 13.0+ mobile device (real or emulated)

• TEST_KEYBOARD_DISPLAY (PASSED): Tested this in the "Test unminified Keymanweb" test page in iOS 13.7 / iPhone Simulator and the OSK appears on the screen when I select "Type here" / "or here" text boxes. And it does not appear If I tap a part of the page. The system keyboard appear if I select the text boxes near the "Add" button.

  

• TEST_CONTENT_EDITABLE (PASSED): Tested this in "Tests the new Attachment/Enablement API functionality" test page and here is my observations: Clicking the on the "Edit me!" text shows OSK appear on the View. However, there is a space appearing at the right side of the OSK on the screen.

  

• TEST_STANDARD_IFRAME (PASSED): Tested this as per the instructions and it is working as expected.

  

• TEST_INPUT_FORMATTING (PASSED): Tested this in "Test unminified Keymanweb" test page and it is working as expected.

..English

..Khmer and Norwegian

• TEST_DRAG_SELECT (PASSED): Tested this as per the instructions and verified that the OSK is still appear during or after selecting the part of the input.

  

[bharanidharanj]

GROUP_ANDROID: Use any Android 8.0+ mobile device (real or emulated)

• TEST_DRAG_SELECT (PASSED): Tested this in API 26 / Android 8.0 emulator and verified that the OSK is still appear during or after selecting the part of the input.

  

[keyman-server]

Changes in this pull request will be available for download in Keyman version 16.0.82-alpha