Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Press Key issue with combined keys #67

Open
Mavnus04 opened this issue May 2, 2022 · 30 comments
Open

Press Key issue with combined keys #67

Mavnus04 opened this issue May 2, 2022 · 30 comments
Assignees
Labels
bug Something isn't working

Comments

@Mavnus04
Copy link

Mavnus04 commented May 2, 2022

I am having an issue pressing several keys on my application.

I can send a single key press with t'~' without an issue. But when I try to send ALT+SHIFT+- or ALT+SHIFT+=, it seems to respond only to the keys - or =. Yes, this key combo normally works on the application.
As you can see in my example, I tried sending it as t'' or s'' or assign it as a string, then sending it.

Show the info bar on the unity window 
    [Tags]  unity  
    Log to console  ...
    ${gamename} =  Set Variable  mygame
    Attach application by name  ${gamename}
    FlaUILibrary.Focus   /Window[@Name='${gamename}']
    
    FlaUILibrary.Press Key  t'~'
    Sleep  3s
    FlaUILibrary.Press Key  t'~'
    Sleep  1s
    
    # 'Half Time Scale': 'Alt+Shift+(minus)',
    FlaUILibrary.Press Key  t'ALT+SHIFT+-'
    Sleep  1s
    FlaUILibrary.Press Key  s'ALT+SHIFT+-'
    Sleep  1s
    ${halfTime} =  Convert To String  t'ALT+SHIFT+-'
    FlaUILibrary.Press Key  ${halfTime}
    Sleep  1s
    ${halfTime2} =  Convert To String  s'ALT+SHIFT+-'
    FlaUILibrary.Press Key  ${halfTime2}
    Sleep  3s
    
    #'Double Time Scale': 'Alt+Shift+(equals)',
    FlaUILibrary.Press Key  t'ALT+SHIFT+='
    Sleep  1s
    FlaUILibrary.Press Key  s'ALT+SHIFT+='
    Sleep  1s
    ${doubleTime} =  Convert To String  t'ALT+SHIFT+='
    FlaUILibrary.Press Key  ${doubleTime}
    Sleep  1s
    ${doubleTime2} =  Convert To String  s'ALT+SHIFT+='
    FlaUILibrary.Press Key  ${doubleTime2}

Current version:

C:\Windows\system32>pip show robotframework-flaui
Name: robotframework-flaui
Version: 1.7.3

Related...
I don't understand the difference between the t'' and s'', can you explain more with an example?
Finally, can you clear up this statement? If identifier set try to attach to given element if operation was successfully old element will be reattached automatically. When does the attachment occur and to which element.

@Mavnus04
Copy link
Author

Mavnus04 commented May 2, 2022

I also have combo commands for ALT+SHIFT+BACK and ALT+SHIFT+0 (The '0' key on the top of the alphanumeric keyboard.), but these don't work either.

@Nepitwin
Copy link
Member

Nepitwin commented May 4, 2022

Code looks quite good that it should work. I will investigate on Keyboard module where the problem comes.

I think it would be a good advice to extend keyboard tests by a gui element which send the response from entered input to a input field to verify if keyboard bindings work as expected.

@Mavnus04
Copy link
Author

@Nepitwin Any update?
Or could you at least answer my other questions for now?

Related...
I don't understand the difference between the t'' and s'', can you explain more with an example?
Finally, can you clear up this statement? If identifier set try to attach to given element if operation was successfully old element will be reattached automatically. When does the attachment occur and to which element.

@Nepitwin
Copy link
Member

@Mavnus04 oh i don't see your question.

t'' syntax is used for a text usage and s'' by syntax

For example if you want to input into a text field you should use t"Your Text Here"

By a shortcut you use s"CTRL+A" this type of can be mixxed up by a list

For example

${EXP_VALUE_OVERRIDE_INPUT_TEXT} = Override
@{KEYBOARD_INPUT_OVERRIDE_TEXT} s'CTRL+A' t'${EXP_VALUE_OVERRIDE_INPUT_TEXT}'

@Nepitwin
Copy link
Member

What i can also see from docs

    The following keys are supported for usage as a part of key_combination:
    | LBUTTON     |     Left mouse button                           |
    | RBUTTON     |     Right mouse button                          |
    | CANCEL      |     Control-break processing                    |
    | MBUTTON     |     Middle mouse button (three-button mouse)    |
    | XBUTTON1    |     Windows 2000/XP: X1 mouse button            |
    | XBUTTON2    |     Windows 2000/XP: X2 mouse button            |
    | BACK        |     BACKSPACE key                               |
    | TAB         |     TAB key                                     |
    | CLEAR       |     CLEAR key                                   |
    | ENTER       |     ENTER key                                   |
    | SHIFT       |     SHIFT key                                   |
    | CTRL        |     CTRL key                                    |
    | ALT         |     ALT key                                     |
    | CAPITAL     |     CAPITAL key                                 |
    | PAUSE       |     PAUSE key                                   |
    | ESCAPE      |     ESC key                                     |
    | ESC         |     ESC key                                     |
    | SPACE       |     Blank space key                             |
    | NEXT        |     Next key                                    |
    | END         |     END key                                     |
    | HOME        |     HOME key                                    |
    | LEFT        |     LEFT ARROW key                              |
    | RIGHT       |     RIGHT ARROW key                             |
    | UP          |     UP ARROW key                                |
    | DOWN        |     DOWN ARROW key                              |
    | SELECT      |     SELECT key                                  |
    | PRINT       |     PRINT key                                   |
    | EXECUTE     |     EXEC key                                    |
    | INSERT      |     INS key                                     |
    | DELETE      |     DEL key                                     |
    | HELP        |     HELP key                                    |
    | 0 - 9       |                                                 |
    | A - Z       |                                                 |
    | F1 - F12    |                                                 |
    | LWIN        |     Left Windows key                            |
    | RWIN        |     Right Windows key                           |
    | APPS        |                                                 |
    | SLEEP       |                                                 |
    | MULTIPLY    |     '*' key                                     |
    | ADD         |     '+' key                                     |
    | SEPARATOR   |                                                 |
    | SUBTRACT    |                                                 |
    | DECIMAL     |                                                 |
    | DIVIDE      |                                                 |

You want to create a shortcut so s is correct for my opinion.

So the input from your sample code should work i will investigate your example until next week.

@Mavnus04
Copy link
Author

@Nepitwin
One more question related to the documentation while we wait for your investigation:
Can you clear up this statement? If identifier set try to attach to given element if operation was successfully old element will be reattached automatically. When does the attachment occur and to which element.

@Nepitwin
Copy link
Member

This documentation is outdated. The only thing that the Keyword is doing that Press Key can be used by XPATH optional to focus the element before a keyboard execution is triggered.

So it's quite similar like your code

Focus
Press Keys

Is equivalent to

Press Keys

I will now extend the test gui by a keyboard testing view. Which displays the keyboard inputs and extend the testcases by multiple keyboard bindings then i should find easily the issue.

@Nepitwin Nepitwin self-assigned this May 18, 2022
Nepitwin added a commit that referenced this issue May 18, 2022
Keyboard input example implemented
Nepitwin added a commit that referenced this issue May 18, 2022
Keyboard input example extended
Keyboard robot file adjusted
Unecessary documentation removed
Nepitwin added a commit that referenced this issue May 19, 2022
Keyboard input example extended
Keyboard robot file adjusted
Unecessary documentation removed
@Nepitwin
Copy link
Member

Ah now i see what the issue is. You try to use the VirtuaKeyShort with simple mapping. But what i can see is that +,- and equals is not supported from current Key mapping to flaui

Keys = {
    "LBUTTON": VirtualKeyShort.LBUTTON,  # Left mouse button
    "RBUTTON": VirtualKeyShort.RBUTTON,  # Right mouse button
    "CANCEL": VirtualKeyShort.CANCEL,  # Control-break processing
    "MBUTTON": VirtualKeyShort.MBUTTON,  # Middle mouse button (three-button mouse)
    "XBUTTON1": VirtualKeyShort.XBUTTON1,  # Windows 2000/XP: X1 mouse button
    "XBUTTON2": VirtualKeyShort.XBUTTON2,  # Windows 2000/XP: X2 mouse button
    "BACK": VirtualKeyShort.BACK,  # BACKSPACE key
    "TAB": VirtualKeyShort.TAB,  # TAB key
    "CLEAR": VirtualKeyShort.CLEAR,  # CLEAR key
    "ENTER": VirtualKeyShort.ENTER,  # ENTER key
    "SHIFT": VirtualKeyShort.SHIFT,  # SHIFT key
    "CONTROL": VirtualKeyShort.CONTROL,  # CTRL key
    "CTRL": VirtualKeyShort.CONTROL,
    "ALT": VirtualKeyShort.ALT,
    "CAPITAL": VirtualKeyShort.CAPITAL,
    "PAUSE": VirtualKeyShort.PAUSE,
    "ESCAPE": VirtualKeyShort.ESCAPE,
    "ESC": VirtualKeyShort.ESCAPE,
    "CONVERT": VirtualKeyShort.CONVERT,  # IME convert
    "SPACE": VirtualKeyShort.SPACE,
    "PRIOR": VirtualKeyShort.PRIOR,  # PAGE UP key
    "NEXT": VirtualKeyShort.NEXT,  # PAGE DOWN key
    "END": VirtualKeyShort.END,
    "HOME": VirtualKeyShort.HOME,
    "LEFT": VirtualKeyShort.LEFT,  # LEFT ARROW key
    "RIGHT": VirtualKeyShort.RIGHT,  # RIGHT ARROW key
    "UP": VirtualKeyShort.UP,  # UP ARROW key
    "DOWN": VirtualKeyShort.DOWN,  # DOWN ARROW key
    "SELECT": VirtualKeyShort.SELECT,
    "PRINT": VirtualKeyShort.PRINT,
    "EXECUTE": VirtualKeyShort.EXECUTE,
    "SNAPSHOT": VirtualKeyShort.SNAPSHOT,  # PRINT SCREEN key
    "INSERT": VirtualKeyShort.INSERT,
    "DELETE": VirtualKeyShort.DELETE,
    "HELP": VirtualKeyShort.HELP,
    "0": VirtualKeyShort.KEY_0,
    "1": VirtualKeyShort.KEY_1,
    "2": VirtualKeyShort.KEY_2,
    "3": VirtualKeyShort.KEY_3,
    "4": VirtualKeyShort.KEY_4,
    "5": VirtualKeyShort.KEY_5,
    "6": VirtualKeyShort.KEY_6,
    "7": VirtualKeyShort.KEY_7,
    "8": VirtualKeyShort.KEY_8,
    "9": VirtualKeyShort.KEY_9,
    "A": VirtualKeyShort.KEY_A,
    "B": VirtualKeyShort.KEY_B,
    "C": VirtualKeyShort.KEY_C,
    "D": VirtualKeyShort.KEY_D,
    "E": VirtualKeyShort.KEY_E,
    "F": VirtualKeyShort.KEY_F,
    "G": VirtualKeyShort.KEY_G,
    "H": VirtualKeyShort.KEY_H,
    "I": VirtualKeyShort.KEY_I,
    "J": VirtualKeyShort.KEY_J,
    "K": VirtualKeyShort.KEY_K,
    "L": VirtualKeyShort.KEY_L,
    "M": VirtualKeyShort.KEY_M,
    "N": VirtualKeyShort.KEY_N,
    "O": VirtualKeyShort.KEY_O,
    "P": VirtualKeyShort.KEY_P,
    "Q": VirtualKeyShort.KEY_Q,
    "R": VirtualKeyShort.KEY_R,
    "S": VirtualKeyShort.KEY_S,
    "T": VirtualKeyShort.KEY_T,
    "U": VirtualKeyShort.KEY_U,
    "V": VirtualKeyShort.KEY_V,
    "W": VirtualKeyShort.KEY_W,
    "X": VirtualKeyShort.KEY_X,
    "Y": VirtualKeyShort.KEY_Y,
    "Z": VirtualKeyShort.KEY_Z,
    "LWIN": VirtualKeyShort.LWIN,
    "RWIN": VirtualKeyShort.RWIN,
    "APPS": VirtualKeyShort.APPS,
    "SLEEP": VirtualKeyShort.SLEEP,
    "MULTIPLY": VirtualKeyShort.MULTIPLY,  # '*'
    "ADD": VirtualKeyShort.ADD,  # '+'
    "SEPARATOR": VirtualKeyShort.SEPARATOR,
    "SUBTRACT": VirtualKeyShort.SUBTRACT,
    "DECIMAL": VirtualKeyShort.DECIMAL,
    "DIVIDE": VirtualKeyShort.DIVIDE,
    "F1": VirtualKeyShort.F1,
    "F2": VirtualKeyShort.F2,
    "F3": VirtualKeyShort.F3,
    "F4": VirtualKeyShort.F4,
    "F5": VirtualKeyShort.F5,
    "F6": VirtualKeyShort.F6,
    "F7": VirtualKeyShort.F7,
    "F8": VirtualKeyShort.F8,
    "F9": VirtualKeyShort.F9,
    "F10": VirtualKeyShort.F10,
    "F11": VirtualKeyShort.F11,
    "F12": VirtualKeyShort.F12
}

I will extend the missing VirtualKeyShort bindings.

Afterwards you can use the KeyShorts by corresponding bindings

FlaUILibrary.Press Key t'ALT+SHIFT+SUBTRACT' <-- Should already work
FlaUILibrary.Press Key t'ALT+SHIFT+ADD' <-- Should already work
FlaUILibrary.Press Key t'ALT+SHIFT+EQUALS' <-- I don't find a corresponding from it

@Nepitwin
Copy link
Member

Nepitwin commented May 19, 2022

If you want to test it simple checkout keyboard branch this contains the new application test by keyboard view.

Here you can see input which are received by keyboard or flaui.

Two ways to test :

Window events are triggered for key up and down event
Input field can be used for keyboard input tests

image

@Mavnus04
Copy link
Author

Mavnus04 commented May 23, 2022

@Nepitwin I got the Keyboard branch my system and built.
When I hit Alt on my keyboard, I get System
While I hold Alt, and try to hit other keys, nothing else shows up in the 'Keyboard * Events'.

If I hold Shift first, then press other keys, they come up. Again, no more keys appear after Alt, if I do Shift then Alt.
image

Shift + - + Alt
image

As you can see, I am getting OemMinus for - key.

@Mavnus04
Copy link
Author

Forgot to mention, that when I try ALT+SHIFT+SUBTRACT or ALT+SHIFT+ADD, it still doesn't do what it is supposed to.

@Nepitwin
Copy link
Member

From sample application the intresting part is that WPF is sending System Event if you first execute ALT keyboard.
If you first try to use STRG+ALT it displays the correct behvior.

If you execute first ALT by Down Event you got system. Your source code i think will handle to wait for ALT event :) So it does not recognize it. The question is why system key event is triggered.

@Nepitwin
Copy link
Member

@Mavnus04
Copy link
Author

Tried putting Shift before Alt by manually pressing keys and the application responded properly. Shift+Alt+- & Shift+Alt+=
However, when I put it in the code, it still does NOT work.
Pressing D as text is a sanity check for me that the application is receiving commands.

Show the info bar on the unity window 
    [Tags]  unity  
    Log to console  ...
    ${gamename} =  Set Variable  mygame
    Set Up Unity Game For Key Presses  ${gamename}

    FlaUILibrary.Focus  /Window[@Name='${gamename}']

    Log To Console  texts...
    FlaUILibrary.Press Key  t'D'
    Sleep  2s
    FlaUILibrary.Press Key  t'D'
    Sleep  2s

    Log To Console  shortcuts...
    FlaUILibrary.Press Key  s'SHIFT+ALT+SUBTRACT'
    Sleep  2s
    FlaUILibrary.Press Key  s'SHIFT+ALT+ADD'
    Sleep  2s

@Nepitwin
Copy link
Member

@Mavnus04

Oh cool you test by a unity based application.

Is their any kind of keyboard interaction be triggered by flaui?

For example you set a simple input by keyboard or an ESC to open the main menu?

If yes sounds good so far. If not i have to investigate which keyboard event is be triggered from flaui.

So it could be related for the tripple usage binding or probably that any keyboard events are not triggered by unity because the simulation from the keyboard is not handle correctly by windows :).

@Mavnus04
Copy link
Author

@Nepitwin

Yes, it is a game created with Unity.

In my last example, I can send the character D and that works just fine, the game responds as it should. It is my sanity check that something is going through from FlaUI

But for some reason, combining those key presses will not go through. Shift, Alt, [Other Key], Doesn't seem to matter the order.

@Nepitwin
Copy link
Member

@Mavnus04

Do you have a small source code sample application for it like the wpf example?

I could then analyze triggered events from unity and flaui to identify the issue probably.

@Mavnus04
Copy link
Author

Sorry, I can't share anything from my company, but I could try making a simple Unity application which accepts and responds to some simple key presses.
Give me a week or so since I have a lot of other tasks that I have to attend to.

@Nepitwin
Copy link
Member

@Mavnus04

I have also thought of such an example application :).

No problem we have all work to do :)

Nepitwin added a commit that referenced this issue Jun 1, 2022
Keyboard input example extended
Keyboard robot file adjusted
Unecessary documentation removed
@Nepitwin Nepitwin added the bug Something isn't working label Jun 1, 2022
@Mavnus04
Copy link
Author

Mavnus04 commented Jun 2, 2022

@Nepitwin
Here is the unity application I created.
Pressing certain keys will cause the background color to change.

W = White
Y = Yellow
Alt + Shift +

  • Backspace = Blue
  • Equals = Red
  • Minus = Green
  • Alpha0 = Grey

Application.zip

Manually pressing the keys, I have no issue changing the color.
When I use FlaUI, only the single character key presses seem to work.

Change color of unity application
    [Tags]  unityapp
    Log to console  ...

    ${gamename} =  Set Variable  Keypress
    Attach application by name  ${gamename}
    FlaUILibrary.Focus  /Window[@Name='${gamename}']

    Log To Console  texts...
    FlaUILibrary.Press Key  t'Y'
    Sleep  2s
    FlaUILibrary.Press Key  t'W'
    Sleep  2s

    Log To Console  shortcuts...
    FlaUILibrary.Press Key  s'SHIFT+ALT+BACK'
    Sleep  2s
    FlaUILibrary.Press Key  s'SHIFT+ALT+ADD'
    Sleep  2s
    FlaUILibrary.Press Key  s'SHIFT+ALT+SUBTRACT'
    Sleep  2s

The source code is 55MB, so I am not sure how to share that with you, if you need it? GitHub said the upload limit is 25MB

@Nepitwin
Copy link
Member

Nepitwin commented Jun 2, 2022

@Mavnus04

Thanks a lot i will investigate what happens on flaui side now by this sample application and hopefully find the issue 👍

@Nepitwin
Copy link
Member

Nepitwin commented Jun 8, 2022

What i can see by my tests that an array is created with correct converted syntax

https://github.com/FlaUI/FlaUI/blob/58b5954349afe2343aeb16ea55b19962b28cd3e1/src/FlaUI.Core/WindowsAPI/Constants.cs#L934

SHIFT (0x10) + ALT (0x12) + BACKSPACE (0x8)
[16, 18, 8]

Type simualtenously method is called which shoud execute all these keyboard bindings

https://github.com/FlaUI/FlaUI/blob/6841da31e13d74f8ad2128e6a48cfa00c9e3501c/src/FlaUI.Core/Input/Keyboard.cs#L110

It's quite strange
i will try to implement a direct test in C# usage without wrapper libary to avoid possible data type converting issues. If the C# solution the issue is not on FlaUI it's on wrapper data type converting by robotframework-flaui wrapper which has to be identified why commands are not working like expected.

Nepitwin pushed a commit that referenced this issue Jun 8, 2022
Sample implementation without example project for analyze purpose.
@Nepitwin
Copy link
Member

Nepitwin commented Jun 11, 2022

@Mavnus04

Okay i found some issues related to FlaUI by Alt Keyboard usage.

FlaUI/FlaUI#295
FlaUI/FlaUI#320

Also the C# sample code have the same behavior like robotframework usage here.

using FlaUI.UIA3;
using FlaUI.Core.Input;
using FlaUI.Core.WindowsAPI;

var app = FlaUI.Core.Application.Launch("F:\\Unity\\Keypress.exe");
using (var automation = new UIA3Automation())
{
	var window = app.GetMainWindow(automation);
	Thread.Sleep(6000);
	Keyboard.TypeSimultaneously(VirtualKeyShort.KEY_Y);
	Keyboard.TypeSimultaneously(VirtualKeyShort.SHIFT, VirtualKeyShort.ALT, VirtualKeyShort.BACK);
	Keyboard.TypeSimultaneously(VirtualKeyShort.SHIFT, VirtualKeyShort.LCONTROL, VirtualKeyShort.BACK);
	Keyboard.TypeSimultaneously(VirtualKeyShort.SHIFT, VirtualKeyShort.CONTROL, VirtualKeyShort.BACK);

	Keyboard.Type(VirtualKeyShort.SHIFT);
	Keyboard.Type(VirtualKeyShort.CONTROL);
	Keyboard.Type(VirtualKeyShort.BACK);
}

I will retry FlaUI by a master build from current source code if it's working then i will update the flaui wrapper to the latest master version. But it seems a known issue on flaui that the combinations usage by ALT does not work like expectly.

@Nepitwin
Copy link
Member

From my point of view it's a keyboard issue on flaui. So the general idea was is to replace the keyword module by something different by keyword modules like

Pyinput - https://nitratine.net/blog/post/simulate-keypresses-in-python/
Keyboard - https://pypi.org/project/keyboard/
Pyautogui - https://github.com/asweigart/pyautogui

But after some additional analyze i also found https://stackoverflow.com/questions/13564851/how-to-generate-keyboard-events

Which cause issues on keyboard handling by ui automation especially for games.

So in general my plan is now to implement a plugin system like selenium already has.

https://github.com/robotframework/SeleniumLibrary/blob/master/docs/extending/extending.rst#Plugins

This interface could be used for use cases like yours to implement workaround keywords by handling seperatly on your use case. I will implement as soon as possible to make a new release and support this feature. So afterwards this ui automation library can be used and extended by anyone with this plugin system for all specific acceptance criterias.

@Mavnus04
Copy link
Author

Nice investigation!
Looking forward to the new version. Thank you very much.

@Mavnus04
Copy link
Author

@Nepitwin How is it going? Wondering if you could provide an update?
Thanks.

@Nepitwin
Copy link
Member

@Mavnus04

Currently i'm busy at work to do a quick fix for it and i think about the plugin mechanis if this behavior should fix it. I'm not sure. But you can already try a second solution to write a simple Keyword by your own in python code.

I think you know that this possiblity is always supported by robotframework.

Simple write a Python file and define a custom keyword which probably contains a sub call by another keyboard library which you include and use for your use case.

It would not solve the issue on flaui side but for thie one case your e2e test would work :).

@Mavnus04
Copy link
Author

I already have something that can send all these key commands for my testing purposes. My hope was to replace it with FlaUI since it supports so much more.

Please let me know once you have a fix in, thanks.

Nepitwin pushed a commit that referenced this issue Aug 21, 2022
Sample implementation without example project for analyze purpose.
@Mavnus04
Copy link
Author

@Nepitwin Hey, checking for an update?

@Nepitwin
Copy link
Member

Currently no news about an upcoming change how to fix your issue by unity. The only thing what i found is that flaui is sending by SendInput controls keyboard interactions. https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-sendinput

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants