Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support of keyboard and mouse #436

Merged
merged 2 commits into from
Aug 2, 2024
Merged

Conversation

sergystepanov
Copy link
Collaborator

@sergystepanov sergystepanov commented Nov 22, 2023

This PR adds mouse and keyboard controls for aplicable Libretro cores.
As simple as it may sound, the main idea is to capture browser Keyboard and Mouse API events and then feed them in a digestible form to the Libretro frontend. Unfortunately, there are several problems related to the original design choices of cloud-game.
Initially, it was assumed that each player would have just one controller device occupying a single controller port in terms of the underlying Libretro API. This way, it would be pretty easy to distinguish controller events of each player, knowing that you can safely forward player control events to their dedicated port without the fear of interfering with other players. However, that was quite a wrong assumption, thinking that one player occupies just one device port. In reality, many cores allow the use of several different controller devices, in theory mapped to an unlimited number of ports (in practice, only up to 9 ports can be used), or even multiplex multiple devices (merge controller events) on a single port, as in the case of the DOSBox Pure core.

Keyboard in Libretro

To use a keyboard with Libretro, two callback functions from the core are required:

  1. Keyboard Callback Function: RETRO_ENVIRONMENT_SET_KEYBOARD_CALLBACK

    • Set when the core pushes out the environment variable with the same name.
    • This callback should be called once a user presses or releases a keyboard key.
    • It's important to note that the keyboard controls in Libretro do not have a distinct port argument in the callback function. This limitation makes it impossible to separate player keyboards in cores like DosBox, even though DosBox polls the state of pressed keys for each port separately in the polling callback.
    • The callback function has the following parameters:
      • bool down: Indicates if the key is pressed.
      • unsigned keycode: Internal Libretro representation of the pressed key.
      • uint32_t character: Ignored.
      • uint16_t key_modifiers: Special key modifiers such as shift, alt, ctrl, meta, num/caps/scroll-locks.
  2. Device Callback Function: RETRO_DEVICE_KEYBOARD

    • It is polled after each emulation tick in the core_input_state polling function.
    • Each key that was previously pressed by the keyboard callback function must be left in the pressed state or released (by setting the device callback function result to 1 or 0).

Mouse in Libretro

Mouse controls in Libretro are polled similarly to the keyboard, using the core_input_state function with the device name RETRO_DEVICE_MOUSE. The behavior of the callback depends on the id parameter:

  • Button States:

    • When the id parameter corresponds to a mouse button, the callback returns the state of that button (pressed or released).
    • In this PR, only the states of the Left, Right, and Middle buttons are implemented.
  • Cursor Position:

    • When the id parameter corresponds to the cursor position, the callback returns the position of the mouse cursor.
    • It should be noted that in the case of DOSBox, the X and Y positions of the mouse are not absolute, but rather the relative deltas of X and Y coordinates between two emulation ticks.

Scale cursor speed to native resolution

Since raw pointer movement is not supported in all browsers or platforms, when you stream a display which size is different from a display on which you use mouse, the mouse movement (DPI) should be approximated accordingly.
Used algorithm takes coordinates (movement delta) from the virtual display (cursor over the browser video element) and scales them to the default DosBox VGA display size of 640x480.
Because the scale ratio value is a floating point number and we can't use subpixel coordinates on the server side, we need to take into account accumulated floating point errors when rounding coordinates to the destination screen (Libretro framebuffer). See an example.

@sergystepanov sergystepanov linked an issue Nov 22, 2023 that may be closed by this pull request
@sergystepanov sergystepanov force-pushed the feature/keyboard-n-mouse branch 4 times, most recently from d84a7a8 to 36a4ca0 Compare November 26, 2023 19:39
@sergystepanov sergystepanov force-pushed the feature/keyboard-n-mouse branch 4 times, most recently from 79530fb to eac770e Compare February 22, 2024 16:16
@sergystepanov sergystepanov force-pushed the feature/keyboard-n-mouse branch 4 times, most recently from 48dcab5 to d78ba6f Compare March 2, 2024 13:50
@sergystepanov sergystepanov force-pushed the feature/keyboard-n-mouse branch 7 times, most recently from ffc1ffa to 08393ae Compare March 8, 2024 15:47
@sergystepanov sergystepanov force-pushed the feature/keyboard-n-mouse branch 10 times, most recently from d15f04d to 75bce44 Compare March 21, 2024 21:18
@sergystepanov sergystepanov force-pushed the feature/keyboard-n-mouse branch from e2b13ce to 99394a0 Compare April 13, 2024 15:19
@sergystepanov sergystepanov force-pushed the feature/keyboard-n-mouse branch 5 times, most recently from 0c51a66 to 3b65dd8 Compare April 25, 2024 08:22
@sergystepanov sergystepanov force-pushed the feature/keyboard-n-mouse branch from 8c64b3d to f42c3c6 Compare May 7, 2024 20:34
@sergystepanov sergystepanov force-pushed the feature/keyboard-n-mouse branch 2 times, most recently from 0f1a45d to 3f24e38 Compare May 27, 2024 14:45
@guozanhua218

This comment was marked as off-topic.

@sergystepanov sergystepanov force-pushed the feature/keyboard-n-mouse branch 4 times, most recently from 0171719 to dad17e9 Compare July 9, 2024 21:37
@sergystepanov sergystepanov force-pushed the feature/keyboard-n-mouse branch 2 times, most recently from 286d75e to 043e4b1 Compare July 16, 2024 08:18
@guozanhua

This comment was marked as off-topic.

@sergystepanov

This comment was marked as off-topic.

@guozanhua218

This comment was marked as off-topic.

@guozanhua218

This comment was marked as off-topic.

@sergystepanov

This comment was marked as off-topic.

@guozanhua

This comment was marked as off-topic.

@sergystepanov

This comment was marked as off-topic.

@sergystepanov sergystepanov force-pushed the feature/keyboard-n-mouse branch from 9b35182 to 81c8dd1 Compare July 26, 2024 14:46
Keyboard and mouse controls will now work if you use the kbMouseSupport parameter in the config for Libretro cores. Be aware that capturing mouse and keyboard controls properly is only possible in fullscreen mode.

Note: In the case of DOSBox, a virtual filesystem handler is not yet implemented, thus each game state will be shared between all rooms (DOS game instances) of CloudRetro.
@sergystepanov sergystepanov force-pushed the feature/keyboard-n-mouse branch from c5c77f0 to ec17f3e Compare August 2, 2024 07:57
@sergystepanov sergystepanov merged commit d855e56 into master Aug 2, 2024
4 checks passed
@sergystepanov sergystepanov deleted the feature/keyboard-n-mouse branch August 2, 2024 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

dosbox_pure_libretro.so keyboard mapping
3 participants