spec(web,developer): multitap and flick for touch keyboards 🐵

[mcdurdin]

This is part of a broader specification for Caps Lock support on touch devices. Development of this touches multiple components; each component will be implemented in a separate PR, referencing this issue.

This feature has emerged from the caps layer feature and is now treated separately for implementation.

Related features

• spec: Caps Lock layer for touch layouts #3620: spec: Caps Lock layer for touch layouts
• spec: Start of text/sentence selects shift layer #3621: spec: Start of text/sentence selects shift layer
• spec: Suggestions respect capitalization #3720: spec: Suggestions respect capitalization
• feat(android): Add arrow keys to the keyboard #5790: feat(android): Add arrow keys to the keyboard
• feat: Consider long-press default key (i.e. when user long-presses but doesn't swipe) as dev-configurable 🐵 #9416
• feat(web): Extend target box on Longpress #1025

Introduction

A caps layer, if present, will be accessible by double-tapping the Shift key. We won't add a longpress to Shift because in future we might want to use this for multitouch approach (e.g. hold Shift, press X with another finger to give a capital X and return to default). (Also, double-tap is standard Caps-lock UX)

The double-tap will be defined by a new attribute for keys, called multiTap. This will be an array similar to the subkeys array. Although the key cap won't be initially meaningful, we will store it in case we want to use it for visual feedback in a future version (e.g. a brief popup).

• (See also LDML multitap property.)
• We will ignore multitap on globe button, just as we ignore longpress.

C5029.1 File Formats: .keyman-touch-layout changes

Changes to the file format and the JSON schema are required.

1. Required: Add a multiTap property to the key object. This will be an array in the same format as the existing sk longpress property.

We will not require specific key codes for multitap. A recommended pattern would be T_2X_A, T_3X_A for a double tap and a triple tap on an 'A' key.

2. Optional: Add flick object to the key object. This will be an object with cardinal directions n,e,s,w and intercardinal directions ne,se,sw,nw as properties of the object. Each of those properties will be a key in the same format as the keys in the sk array. (Note: LDML has a flick property so we may need to implement this in order to complete the LDML implementation.)

Backward compatibility

Touch layout files that contain these features will be backwardly compatible with earlier versions of Keyman, with a graceful degradation of functionality.

• The multiTap property will be ignored by older versions of KeymanWeb.

C5029.2 Changes to Developer IDE

• Required: The touch layout editor needs to surface the multiTap property as a new section similar to the longpress interface.
• Required: The touch layout editor needs to visually flag the presence of multiTap properties in its keyboard view. (For longpress this is currently a slash on top right of the key; something similar but differentiated is needed for multitap).
• Optional: The touch layout editor needs to surface the flick property again as a new section similar to the longpress interface.
• Optional: The touch layout editor needs to visually flag the presence of flick properties in its keyboard view.

C5029.3 KeymanWeb enhancements

• Required: KeymanWeb needs to support the multiTap property.

Caps Lock Interaction: double-tapping Shift will switch to the caps layer.

• Question: What do we do when the first tap in a multitap changes the layer? How do we link the second tap to the first one?
  • Could we ‘cache’ the multitap key?
    • Implementation, loosely: if touch is within the multitap time window, and location is within the touch-bounds of the multitap source key:
    • Automatically use the source key & continue the multitap
    • Reset the time window’s start, allowing it to be further continued.

The time between the two taps should be less than 300ms to be treated as a double tap. If the time between taps is longer than that, the second tap should trigger a normal key interaction. Any other interaction after the first tap will cancel the double-tap state.

Subsequent taps in a multi-tap sequence will never trigger a longpress or a flick.

• Required: KeymanWeb will need to use the transcriptions feature to roll back previous outputs in the multitap sequence. Layer changes will not be rolled back, just output.

• Optional: We will need an API in web to change the default double-tap time, accessible from Android, iOS and web apps.

• Optional: KeymanWeb needs to support the flick property and gesture.

  • If a keyboard defines flicks, then we may need to disable the model of sliding across keys to a choose a different key that we currently support, for that keyboard.
  • Flicks may be shown in the OSK as a small character on the appropriate edge or corner of the key?
  • May need an API to enable/disable flicks per user preference.

C5029.4: Android and iOS changes

• Optional: Multitap speed may need to be user-configurable
  • Similar pains to Windows double-clicking, which is a well-known accessibility issue.
  • Note existing issue on longpress speed. feat(android/app): Feature Request: customize long-press delay time #877, [iOS] Feature Request: Customize long-press delay time #878
  • But let's see if we can use system accessibility values for this.
    • [DW] Android only provides a longpress value, not any sort of double-tap.
  • Q: Do browsers provide accessibility values?
• Optional: Add a setting to disable flicks for accessibility reasons.

Related Issues and discussions

• Touch Layouts need to support permanent vs transitory layer selection #246
• https://community.software.sil.org/t/how-to-return-from-shift-automatically-and-how-to-caps-lock-by-double-tapping/3469/3
• https://community.software.sil.org/t/problem-with-caps-lock-output/3718
• Original design document
• RFC
• 15.0 planning document

[mcdurdin]

Most of the KeymanWeb-side infrastructure for multi-tap is implemented in #5989, as a generalized solution with a specialized case for Shift -> Caps in 15.0.

Developer-side infrastructure is needed, plus a more detailed discussion of how to handle multitap and rollback (while we could use the transcriptions feature, it is not entirely clear to me that this is the most appropriate way to solve this):

• Required: KeymanWeb will need to use the transcriptions feature to roll back previous outputs in the multitap sequence. Layer changes will not be rolled back, just output.

[MayuraVerma]

Is it possible to enable double tap gesture for all keys in version 15?
there is no need for developer GUI, we can edit the source in textpad, as long as keyman can compile it.
developer gui can wait.
double tap will make a huge difference
I can finally release the keyboard