fix editing with IMEs

[ianstormtaylor]

Do you want to request a feature or report a bug?

Bug.

What's the current behavior?

Right now Slate doesn't work well with IME's, for different reasons. There are a few different issues we are tracking that we hope will make this better.

• don't re-render editor during compositions #2067 — we added a hack to re-render the placeholder when a composition starts, but it's re-rendering the entire editor, which we think has an issue with IME's. This would be the first thing to fix.

• fix editing with "soft keyboards" (eg. Android, IMEs) #2062 — second, @thesunny has been awesome enough to do a bunch of research to document the exact order of composition events on Android devices with IME-based keyboards. Although fixing this issue would likely guarantee IME support, it's also possible that there's a smaller amount of work that can get IME support without full Android support first. (Just implementing desktop Chrome's behaviors.)

There might be other things we're not aware of. But until these are solved, IME's will have issues. If you are experience with IME's and want to help, that would be amazing! I'm going to be referring all IME-related bugs here in the meantime, to focus efforts.

[ericedem]

I was doing some research on handling IMEs, and found these articles helpful:

Background in what IMEs are and how they work:
https://developer.mozilla.org/en-US/docs/Mozilla/IME_handling_guide

Some potential solutions for handling IME events:
https://www.stum.de/2016/06/24/handling-ime-events-in-javascript/

[CodePlayer]

This issue is also bothering me, I hope to have a better solution as soon as possible.

[thesunny]

FYI, Android support published to Slate @ 0.44.12 four days ago.

There are still some bugs but a huge improvement over existing support.

[froatsnook]

@thesunny Just wanted to report that I updated to the latest slate and it's working SO much better in Android. Thank you for fixing this!

[thesunny]

I'm working on fixing the remaining Android bugs now.

In order to be more efficient, please submit bugs and help with testing over in this issue:

Fix all showstopper bugs in Slate on Android (Help us by reporting and testing bugs)
#2726

[akashdeep-sarin]

Is there any plan for fixing the Japanese Language bugs...??
The issue seems to exist till now and is replicable in this example

[akashdeep-sarin]

Hi @thesunny @ianstormtaylor @milkman4
I have seen this PR solves this issue to a great extent... can you guys can have a look...??

[knubie]

@ianstormtaylor I believe each type of IME should probably receive its own issue. A lot of problems overlap, but some are unique to each, and require special "emulation" to make them work with Slate.

I've been spending a lot of time looking into / fixing IME related issues over the past few weeks and here are some findings / thoughts.

1. Safari seems to handle all IME events perfectly fine because they support native "beforeInput" events with rich information about the composition that can simply be passed down to Slate. This PR Fix composition editing in Safari #3040 fixes an issue in which some of those events weren't being utilized.
2. Korean / Hangul input functions slightly differently from Japanese in that it doesn't have "selection" events. For example with Korean, hitting enter or ↓ functions exactly as it would while not composing. While with Japanese, those keys have special behavior during the composition. Hence this piece of code https://github.com/ianstormtaylor/slate/blob/master/packages/slate-react/src/plugins/dom/before.js#L399 is not applicable to Korean.
3. For both Japanese and Korean, the cursor stays at the front of the composition instead of at the end. This can probably be fixed with emulation for Korean, but not sure about Japanese, since we lack data about the composition as it's happening.
4. MacOS has a type of composition that (AFAIK) is unique to it in which you long press a character, which brings up a selection menu containing different accented version of that character. I have a pull request here: Fix MacOS long-press accented char bug #3041 that emulates this behavior for non-safari browsers.
5. De-focusing or changing the selection during a composition does not insert the composition text into the editor data. This causes the DOM to be out of sync with the Editor. I have a fix for this but haven't opened a PR yet. It uses the "emulation" strategy of trying to figure out when this scenario occurs and inserts the text.
6. When composing a Korean character with more than one jamo (vowel/consonant character), if the user backspaces, the last backspace will not delete the last jamo, but will instead end the composition. (Haven't looked into why this happens yet)

This is not an exhaustive list, and I'm not sure the best way to divide and tackle all of these issues from a process or code POV. One large PR? How do we organize this in the code without a bunch of special-case emulation code scattered around? Should we make a plugin just for composition?

[knubie]

Here's some more specific information about the particular composition problems in Slate:

You can use this tool to explore the different events: https://w3c.github.io/uievents/tools/key-event-viewer.html

MacOS Opt + letter accented composition

• Safari

  • Safari handles this pretty well with the following events:
    • <compositionstart>
    • <beforeinput {type: 'insertCompositionText', data: '˜'}>
    • <beforeinput {type: 'deleteCompositionText'}>
    • <beforeinput {type: 'insertFromComposition', data: 'ñ'}
    • <compositionend>
  • Currently this is broken in Slate, as slate does not handle insertFromComposition in the dom/AfterPlugin.

• Chrome

  • Chrome also handles this, but with a different set of events.
    • <compositionstart>
    • <beforeinput {type: 'insertCompositionText', data: '˜'}>
    • <beforeinput {type: 'insertCompositionText', data: 'ñ'}>
    • <compositionend>
  • Works fine in Slate

MacOS long-press accented character selection. More details here

• Safari
  • Again Safari handles this pretty well with the following events:
    • <beforeinput {type: 'insertText', data: 'n'}>
    • <beforeinput {type: 'insertReplacementText', data: 'ñ'}>
  • Works in Slate.
• Chrome
  • Chrome incorrectly handles this with the following events.
    • <beforeinput {type: 'insertText', data: 'n'}>
    • <beforeinput {type: 'insertText', data: 'ñ'}>
  • Notice how it fires insertText twice. This causes Slate to insert nñ instead of just ñ.
  • I've filed a bug report in Chromium here.

Hangul input

• Take the following character as a test case: 한(gks)↵
• Safari
  • Handled with the following events.
    • <compositionstart>
    • <beforeinput {type: 'insertCompositionText', data: 'ㅎ'}>
    • <beforeinput {type: 'insertCompositionText', data: '하'}>
    • <beforeinput {type: 'insertCompositionText', data: '한'}>
    • <beforeinput {type: 'insertCompositionText', data: '한'}> (on enter)
    • <beforeinput {type: 'deleteCompositionText'>
    • <beforeinput {type: 'insertFromComposition', data: '한'}
    • <compositionend>
    • <keydown 'Enter'>
  • This is broken in Slate for two reasons:
    • It seems that after compositionstart is fired, an onSelect hook is called which somehow triggers a compositionend event. So that the characters never compose.
    • Slate does not listen for insertFromComposition anyway.
• Chrome
  • Handled a little differently in Chrome:
    • <compositionstart>
    • <beforeinput {type: 'insertCompositionText', data: 'ㅎ'}>
    • <beforeinput {type: 'insertCompositionText', data: '하'}>
    • <beforeinput {type: 'insertCompositionText', data: '한'}>
    • <beforeinput {type: 'insertCompositionText', data: '한'}> (on enter)
    • <compositionend>
    • <keydown 'Enter'>
    • <beforeinput {type: 'insertLineBreak', data: 'null'}>
  • In both cases the default behavior of hitting enter to insert a new line while composing does not work in Slate, because slate preventDefault on enter while composing here. Presumably because most other IMEs (like Japanese) uses enter to select composition text.

[undeadfrost]

Using Chinese pinyin input method character loss and cursor position problems still exist

[ianstormtaylor]

Fixed by #3093.

[dawsbot]

We have this IME issue in a massive production app that's stuck on slate v0.47.8 and slate-react v0.22.9.

Is there any hope for us to fix this bug without doing the massive rewrite to the JSON-version of slate? We're locked into the immutable version for now

[thesunny]

@dawsbot Just so you are aware, there is no Android support in the latest Slate version. Not sure how well other IME's are supported.

[icarusysuper]

Is there some roadmap about IME?
@ianstormtaylor @thesunny

[thesunny]

From what I understand, IME is something that should be working in the current version but without Android support. Android support is significantly different from supporting other browsers.

Ian has stated that his use case does not include Android support and therefore it is something the community will have to build. I built a version of Android support about a year ago which I contributed but it did not make it through the upgrade to the latest version which was a rewrite.

I'm considering a paid version of Android support here #3573.

[dawsbot]

Perhaps I'm describing something different here. I'm not using Android, the bug we have in all browsers is the final character in Korean is removed from text inputs. This sounds like what @undeadfrost mentioned above.

Should I create a new GitHub issue for this? Is there hope for a fix on this bug in a pre-next version @ianstormtaylor?