Generate layout-agnostic events when holding Ctrl

[maximbaz]

xev shows a very interesting behavior, when holding Ctrl it always generates latin key events even if the currently active layout is Russian (which only has cyrillic letters).

---

Click to see the output

English layout, press "d", xev sees it as "d":

KeyPress event, serial 28, synthetic NO, window 0x2c00001,
    root 0x1cf, subw 0x0, time 287760, (-807,-100), root:(1140,0),
    state 0x10, keycode 40 (keysym 0x64, d), same_screen YES,
    XLookupString gives 1 bytes: (64) "d"
    XmbLookupString gives 1 bytes: (64) "d"
    XFilterEvent returns: False

KeyRelease event, serial 28, synthetic NO, window 0x2c00001,
    root 0x1cf, subw 0x0, time 287884, (-807,-100), root:(1140,0),
    state 0x10, keycode 40 (keysym 0x64, d), same_screen YES,
    XLookupString gives 1 bytes: (64) "d"
    XFilterEvent returns: False

Russian layout, press "d", xev sees it as cyrillic "в":

KeyPress event, serial 28, synthetic NO, window 0x2c00001,
    root 0x1cf, subw 0x0, time 507942, (-490,157), root:(1457,257),
    state 0x2010, keycode 40 (keysym 0x6d7, Cyrillic_ve), same_screen YES,
    XLookupString gives 2 bytes: (d0 b2) "в"
    XmbLookupString gives 2 bytes: (d0 b2) "в"
    XFilterEvent returns: False

KeyRelease event, serial 28, synthetic NO, window 0x2c00001,
    root 0x1cf, subw 0x0, time 508090, (-490,157), root:(1457,257),
    state 0x2010, keycode 40 (keysym 0x6d7, Cyrillic_ve), same_screen YES,
    XLookupString gives 2 bytes: (d0 b2) "в"
    XFilterEvent returns: False

English layout, press "Ctrl+d", xev sees it as "Ctrl+d":

KeyPress event, serial 28, synthetic NO, window 0x2c00001,
    root 0x1cf, subw 0x0, time 309258, (-1349,-37), root:(598,63),
    state 0x10, keycode 37 (keysym 0xffe3, Control_L), same_screen YES,
    XLookupString gives 0 bytes: 
    XmbLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyPress event, serial 28, synthetic NO, window 0x2c00001,
    root 0x1cf, subw 0x0, time 310271, (-1349,-37), root:(598,63),
    state 0x14, keycode 40 (keysym 0x64, d), same_screen YES,
    XLookupString gives 1 bytes: (04) ""
    XmbLookupString gives 1 bytes: (04) ""
    XFilterEvent returns: False

KeyRelease event, serial 28, synthetic NO, window 0x2c00001,
    root 0x1cf, subw 0x0, time 310441, (-1349,-37), root:(598,63),
    state 0x14, keycode 40 (keysym 0x64, d), same_screen YES,
    XLookupString gives 1 bytes: (04) ""
    XFilterEvent returns: False

KeyRelease event, serial 28, synthetic NO, window 0x2c00001,
    root 0x1cf, subw 0x0, time 310973, (-1349,-37), root:(598,63),
    state 0x14, keycode 37 (keysym 0xffe3, Control_L), same_screen YES,
    XLookupString gives 0 bytes: 
    XFilterEvent returns: False

Russian layout, press "Ctrl+d", xev sees it as latin "Ctrl+d" and not cyrillic "Ctrl+в":

KeyPress event, serial 28, synthetic NO, window 0x2c00001,
    root 0x1cf, subw 0x0, time 538564, (-342,490), root:(1605,590),
    state 0x2010, keycode 37 (keysym 0xffe3, Control_L), same_screen YES,
    XLookupString gives 0 bytes: 
    XmbLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyPress event, serial 28, synthetic NO, window 0x2c00001,
    root 0x1cf, subw 0x0, time 540366, (-342,490), root:(1605,590),
    state 0x2014, keycode 40 (keysym 0x64, d), same_screen YES,
    XLookupString gives 1 bytes: (04) ""
    XmbLookupString gives 1 bytes: (04) ""
    XFilterEvent returns: False

KeyRelease event, serial 28, synthetic NO, window 0x2c00001,
    root 0x1cf, subw 0x0, time 540546, (-342,490), root:(1605,590),
    state 0x2014, keycode 40 (keysym 0x64, d), same_screen YES,
    XLookupString gives 1 bytes: (04) ""
    XFilterEvent returns: False

KeyRelease event, serial 28, synthetic NO, window 0x2c00001,
    root 0x1cf, subw 0x0, time 540946, (-342,490), root:(1605,590),
    state 0x2014, keycode 37 (keysym 0xffe3, Control_L), same_screen YES,
    XLookupString gives 0 bytes: 
    XFilterEvent returns: False

---

I know this might sound like an xkcd comic, but please bear with me, this is truly very useful, because applications can have keyboard shortcuts and they work regardless of the currently active layout. For example, in terminal Ctrl+C, Ctrl+D, Ctrl+Shift+C do not work unless you configure an English keyboard layout on your computer.

I was told xev uses xlib to get this behavior, and it looks like libxkbcommon doesn't follow this behavior.

---

Here's what libxkbcommon emits when I press Ctrl+d in Russian layout:

scancode: 0x25 release: 0 clean_sym: Control_L composed_sym: Control_L mods: numlock glfw_key: LEFT CONTROL
scancode: 0x28 release: 0 clean_sym: Cyrillic_ve composed_sym: Cyrillic_ve mods: ctrl+numlock glfw_key: UNKNOWN
scancode: 0x28 release: 1 clean_sym: Cyrillic_ve mods: ctrl+numlock glfw_key: UNKNOWN
scancode: 0x25 release: 1 clean_sym: Control_L mods: ctrl+numlock glfw_key: LEFT CONTROL

---

This was originally reported for kitty terminal, the author suggested this to be fixed in libxkbcommon instead.

[bluetech]

xev uses the XLookupString() function from Xlib. The keysym and string it displays are the one returned by XLookupString().

The behavior you describe is implemented in Xlib by a hidden library control called XkbLC_ControlFallback (enabled by default).

xkbcommon implements this behavior, but not for keysyms, only the strings, i.e. xkb_state_key_get_one_sym is not affected, xkb_state_key_get_utf8() is affected. Adding it for strings was necessary for backward compatibility. No one asked for it to be added for keysyms as well.

I am inclined to not support this. In general, adding these special cases just makes it harder for an application to get things working in the hand, since it has to handle a bunch of obscure edge-cases. This solution is not general; for example, in the terminal example it might (arguably) fix the problem with Ctrl-a but it won't fix it for Alt-a. So I think the application can handle this on its own, if it wants to.

BTW, the switching-language-breaks-shortcuts problem is well known, and is handled by both big toolkits (GTK and Qt) at least. A terminal has its own input pipeline though.

[maximbaz]

Thanks for the answer, @kovidgoyal what do you think about this argument for implementing the fix in the kitty application itself?

[kovidgoyal]

This really doesn't seem like something that every application should have to workaround. And there really is no good way to workaround it. You want ctrl+X in the russian layout to generate Ctrl+. Somebody else might want it to generate Arabic Aleph. This kind of thing belongs in the layout definition,or if that is not possible, then via XCompose rules. Applications should not be second guessing what keys the OS wants them to generate, that way lies chaos. It means that it is not possible for users to configure their keyboards in one cetral place, instead they have to somehow get every different application to do what they want, that is insanity.

[felselva]

From the link that you provided:

# Ctrl+C in English layout
scancode: 0x25 release: 0 clean_sym: Control_L composed_sym: Control_L mods: numlock glfw_key: LEFT CONTROL
scancode: 0x36 release: 0 clean_sym: c composed_sym: c mods: ctrl+numlock glfw_key: C
scancode: 0x36 release: 1 clean_sym: c mods: ctrl+numlock glfw_key: C
scancode: 0x25 release: 1 clean_sym: Control_L mods: ctrl+numlock glfw_key: LEFT CONTROL

# Ctrl+C in Russian layout
scancode: 0x25 release: 0 clean_sym: Control_L composed_sym: Control_L mods: numlock glfw_key: LEFT CONTROL
scancode: 0x36 release: 0 clean_sym: Cyrillic_es composed_sym: Cyrillic_es mods: ctrl+numlock glfw_key: UNKNOWN
scancode: 0x36 release: 1 clean_sym: Cyrillic_es mods: ctrl+numlock glfw_key: UNKNOWN
scancode: 0x25 release: 1 clean_sym: Control_L mods: ctrl+numlock glfw_key: LEFT CONTROL

Shouldn't the application be using the key code scancode to test shortcuts, instead?

[maximbaz]

For me as a user this would be ideal, e.g. I already use such approach in my i3wm key bindings and I'm happy that my shortcuts consistently work regardless of currently active layout (in other words, I configure Ctrl+36 instead of Ctrl+C). However @kovidgoyal said this would make it impossible to use compose keys to enter non-english text — I confess I still don't understand the reason 100%, so I'll let him describe the details if necessary.

[felselva]

For me as a user this would be ideal, e.g. I already use such approach in my i3wm key bindings and I'm happy that my shortcuts consistently work regardless of currently active layout (in other words, I configure Ctrl+36 instead of Ctrl+C)

Yes, that's what every other application does. GTK and SDL applications for example. Keysyms or the composed string shouldn't be used for testing shortcuts.

[bluetech]

This really doesn't seem like something that every application should have to workaround.

Shortcuts are not a concept in XKB. Shortcuts are an application-level concept. Different applications want different things (for the same user). To use your example, Application A wants ctrl+X in the russian layout to generate Ctrl+. Application B might want it to generate Arabic Aleph

Yes, that's what every other application does. GTK and SDL applications for example.

I don't know about SDL, but GTK and Qt use keysyms.

Keysyms or the composed string shouldn't be used for testing shortcuts.

Using keycodes for shortcuts does not work well in general. For example, "Ctrl-S" is the universally accepted shortcut for "Save"; if you configure your qwerty keyboard with a dvorak layout, it should still be "Ctrl-S" (here I mean the key that generates "S" in the dvorak layout).

[kovidgoyal]

Indeed, it is to support people with layouts like dvorak that kitty uses keysyms not keycodes. To me it is logical that if you press the key X and get S output as text, you should be able to press the key Ctrl+X and get the shortcut Ctrl+S not Ctrl+X

As I said before, it is not possible for an application to figure out what the user wants. It is up to the user to configure their keyboard layout to generate the logical keypresses they intend, with and without modifiers. No other approach can possibly hope to be consistent or logical.

[felselva]

Using keycodes for shortcuts does not work well in general. For example, "Ctrl-S" is the universally accepted shortcut for "Save"; if you configure your qwerty keyboard with a dvorak layout, it should still be "Ctrl-S" (here I mean the key that generates "S" in the dvorak layout).

Sorry, I meant scancode there.

[maximbaz]

Scancodes are different according to kitty's debug mode:

# QWERTY, pressing "c" (bottom row, third key from the left)
scancode: 0x36 release: 0 clean_sym: c composed_sym: c text: c mods: numlock glfw_key: C
scancode: 0x36 release: 1 clean_sym: c mods: numlock glfw_key: C

# Dvorak, pressing "c" (top row, the key right under "8" and "9")
scancode: 0x1f release: 0 clean_sym: c composed_sym: c text: c mods: numlock glfw_key: C
scancode: 0x1f release: 1 clean_sym: c mods: numlock glfw_key: C

Maybe the solution is to make application support both bindings with keysym and scancode? That's what i3wm does for example, it has commands bindsym and bindcode. In other words, if I could have configured Ctrl+40 to do what Ctrl+D does, I would be happy.

I also tried to make use of XCompose as suggested, it's very difficult to find how to properly configure it, but trial and error helped achieve the following ~/.XCompose:

include "%L"

! Ctrl <Cyrillic_ve>: "\033D" d
! Ctrl <Cyrillic_es>: "\033C" c

It does send Ctrl+C and Ctrl+D to the application when active layout is Russian, but (1) I can't figure out how to make a binding for Ctrl+Shift+C, i.e. what escape code corresponds to Shift, and (2) this magic with escape code "\033" feels a bit specific to terminal apps only, isn't it?

[kovidgoyal]

I suspect I was wrong about xcompose, the fundamental problem is that xcompose is designed to output text, not key presses, so there is no way to represent a mapping from <modifier>+key -> modifier+another_key in it (at least as far as I can tell from reading it's absolutely pathetic documentation). Which makes it useless for your purpose.

That means you have to rely on the XKB layout, and that is, as far as I can tell an undocumented rat's nest. I dont see in principle why XKB is not able to map modifier+key to modifier+another key but I cannot find out how to do it, without spending indecent amounts of time on it, at least. Maybe somebody with more experience with XKB can point you in the right direction.

As for binding via keycode instead of keysym -- it is an ugly hack to a problem that is correctly solved at the system level. And it means that in order to bind the shortcuts, users have to now figure out what the scancode for their keys is. That is really unacceptable. Not to mention fragile. As far as I know there is no guarantee that scancodes are portable across systems, so tomorrow if you change your computer/kernel version/graphics stack/keyboard model all your shortcuts will stop working, and I will have to deal with the bug reports.

[maximbaz]

This was solved on application level without introducing the initially suggested hack, the application now makes use of xkb fallback keyboard layout. Thank you for your input, closing the ticket.

[bluetech]

That's not a bad way to do it.

Another way to do it is, instead of looking at a fallback keymap, look at a fallback "ascii layout" in the same keymap. By "ascii layout" I mean a layout which generates English-like Latin keysyms. I believe that for this purpose, modern desktop environments ensure that there is always at least one "ascii layout", i.e. they add a hidden one if not present.

[kovidgoyal]

How would I I do that? It's not clear to me form the libxkb API docs. And would doing it modify the xkb_state or xkb_compose_state?

[bluetech]

You can use a "scratch" xkb_state (like qt does), or you can query the keymap directly if you don't need some info like consumed modifiers. Here is some old code I dug up which shows the idea:

static xkb_keysym_t
get_ascii(struct xkb_state *state, xkb_keycode_t keycode)
{
    struct xkb_keymap *keymap;
    xkb_layout_index_t num_layouts;
    xkb_layout_index_t layout;
    xkb_level_index_t level;
    const xkb_keysym_t *syms;
    int num_syms;

    keymap = xkb_state_get_keymap(state);
    num_layouts = xkb_keymap_num_layouts_for_key(keymap, keycode);

    for (layout = 0; layout < num_layouts; layout++) {
        level = xkb_state_key_get_level(state, keycode, layout);
        num_syms = xkb_keymap_key_get_syms_by_level(keymap, keycode,
                                                    layout, level, &syms);
        if (num_syms != 1)
            continue;

        if (syms[0] > 0 && xkb_keysym_to_utf32(syms[0]) < 128)
            return syms[0];
    }

    return XKB_KEY_NoSymbol;
}

[kovidgoyal]

Interesting, thanks. However, I'm not sure how reliable that would be. A lot of my users use alternative desktop environments, such as i3 or other window managers/lightweight desktops. If as you say, it is the responsibility of the desktop environment to ensure there is always an ascii layout, it might not be the case that all of them do that. On the other hand, I dont know how often the default keyboard layout would match the keyboard the user has connected either. I wish there was some "official" way to do this.

The advantage with using the default keyboard layout is that it can be easily configured by the user. Hidden layouts automagically inserted by desktop environments, not so much. Incidentally how would the desktop environment know what the physical layout of the keyboard is, to insert such an ascii layout?

[bluetech]

I agree you can't be completely sure that an ascii layout is present. In my experience (which might be biased), most users do have one. And for terminal users, not having one would be very hard. But again, I agree.

Incidentally how would the desktop environment know what the physical layout of the keyboard is, to insert such an ascii layout?

I don't know how they do it, but I'm willing to bet what's added is always the simple us layout.