Switching to Chinese/Korean IMEs produce Latin/QWERTY output until switching windows

[rlue]

I'm running into a strange problem where I can use xkbswitch to switch IMEs, but for certain non-Latin input methods, input to the current window comes out as US English until you switch to another window.

Specifically, I've tested on a handful of languages (Arabic, Spanish, English, Japanese-Hiragana, Japanese-Katakana, Korean, Simplified Chinese-Pinyin, Traditional Chinese-Zhuyin), and it only seems to be happening in Korean and Chinese (both Pinyin and Zhuyin) — both East Asian IMEs with alternate modes for English input.

Observations:

• It's not limited to the GUI terminal.

  I thought it might just be limited to the terminal at first, but then tested by running

  sleep 3 && xkbswitch -s 1 
  

  switching over to Firefox, waiting for the IME to change, and then starting to type. The same thing happens.

• It's always US English output, even if you're switching from a non-Latin language. (Possibly because US English is the default on my system.)

  If switching from Arabic to Chinese, US English characters still come out, not Arabic or Chinese. Usually, it's normal (single-width) characters, but sometimes (still haven't figured out why/when), it's double-width characters. Originally, this led me to believe that the unexpected US English input I was observing was a sub-mode of the IME that xkbswitch had switched to, but then I noticed the following:

  Both Korean and Chinese IMEs offer some mechanism for inputting US English characters without having to switch back to the English IME (in Zhuyin, caps lock will force all input to Latin/QWERTY; in Korean, holding down <Option> does the same). The behavior described in this issue is not a result of triggering these mechanisms: when this problem begins, <Option-a> outputs ‘å’, which is ordinarily not possible in Zhuyin.

• xkbswitch appears to preserve some IME statefulness that is not preserved by the standard system keyboard shortcut for switching IMEs.

  This one's kind of hard to explain, so I'll format it as a series of shell commands with comments:

  $ xkbswitch -s 1     # switches to Chinese
  $ asdf               # at this point, input comes out in Latin/QWERTY
  
  # but if I switch to Firefox, the Chinese IME begins working correctly
  # and if I switch back to the terminal...
  
  $ ㄅㄆㄇㄈ            # Chinese continues to work
  $ xkbswitch -s 0     # at that point, I can switch back to English (pulling up this command via readline history)
  $ xkbswitch -s 1     # then back to Chinese
  $ ㄅㄆㄇㄈ            # and it works _without having to switch windows_
  
  # but then if I switch back to English with the system keyboard shortcut (CMD-Space)
  
  $ xkbswitch -s 1     # and switch back to Chinese again
  $ asdf               # then we're back where we started...
  

Thoughts?

[myshov]

@rlue hello!

Thank you for bug report!

I haven't used xkbswitch for layout switching some time, so I was unaware about this issue.
And it seems that something went wrong after updating of macOS, because it doesn't work for me either now. There is only changing of indicator but without actual changing of layout.

The statefullness is probably result of an activation of the option “Automatically switch to a document’s input source” in Settings -> Keyboard -> Input Sources.

By the way which version of macOS do you use?
Did you compile utility by yourself or did you get compiled binary from my repo?
Do you have some warnings/errors in macOS "console" utility when you use xkbswitch?

[rlue]

By the way which version of macOS do you use? Did you compile utility by yourself or did you get compiled binary from my repo?

Ah, yes. I'm on 10.11 El Capitan. (I haven't had the guts to upgrade to Sierra yet — I'm on a Hackintosh, and I'm pretty sure something's going to break.)

I used the binary straight from the repo.

---

I actually found the same issue in a similar utility (input-source-switcher), so I filed a bug there too. The dev (@vovkasm) says he thinks he's figured it out — maybe you guys could coordinate? or even join forces? (Your utilities do pretty much exactly the same things, apart from their CLI syntax.)

[vovkasm]

The dev (@vovkasm) says he thinks he's figured it out

Actually not (I was think that this issue can be fixed by run actual NSRunLoop to allow HIToolbox to manage its cooperation with other system processes). I converted code to use run loop, but unfortunately issue remain. I have some thoughts...

Documentation (note that this is all old and deprecated apis) from TextInputServices.h:

When an input method or mode is the selected input source, TSM will by default use the most-recently-used ASCII-capable keyboard layout to translate key events* (this keyboard layout is also the one that will appear in Keyboard Viewer); an input source for this keyboard layout is returned by TISCopyCurrentASCIICapableKeyboardLayout. If a different keyboard layout should be used for a particular input method or mode, then when that input method/mode is activated it should call TISSetInputMethodKeyboardLayoutOverride to specify the desired keyboard layout.

So this is documented behavior. And I think that by call TISSetInputMethodKeyboardLayoutOverride it is possible to switch to normal keyboard layout (but still not tested it). My current problem that I can't find method to find proper input source id for this "normal" layout. Experimentally I found it for method com.apple.inputmethod.TCIM.Zhuyin it is com.apple.keylayout.ZhuyinBopomofo, but I cant figure out how to programmatically get this id (or set of candidates) from original input source (source that user want to switch to).

[myshov]

An update from me.
I've just figured out that utility doesn't work only under tmux. If I run it in regular terminal it does work. @rlue maybe do you use some terminal multiplexer (tmux or GNU Screen) as in my case?

[rlue]

Hey @myshov, I do use tmux, but the behavior I've described in this issue continues to occur in the bare terminal.

[rlue]

FYI, I'm using xkbswitch in a vim plugin, and it displays the same behavior when invoked from GUI vim, so it's definitely not a tmux issue.

[laishulu]

emacs-mac don't have such a problem

https://bitbucket.org/mituharu/emacs-mac/src/26c8fd9920db9d34ae8f78bceaec714230824dac/src/macfns.c?at=master#macfns.c

[laishulu]

kawa has a workaround

https://github.com/utatti/kawa/blob/master/kawa/TISInputSourceExtension.swift