Add virtual display feature #5370
Conversation
|
Hi @rom1v, binary file cannot be downloaded |
|
@huynhtanloc2612 Thanks, fixed. |
|
@rom1v |
|
tested this on mac(m1pro) - works! Steps I followed -
|
|
Thank you for your contribution. Test platform: Mac(M2)/Meizu 21 Test way: git fetch origin pull/5370/head:pr-5370
git checkout pr-5370
meson setup x --buildtype=release --strip -Db_lto=true
ninja -Cx # DO NOT RUN AS ROOT
./run x --new-display=1920x1080I mainly tested the following scenarios:
Due to this flag VIRTUAL_DISPLAY_FLAG_SHOULD_SHOW_SYSTEM_DECORATIONSOn the Meizu device I tested, the Launcher will start on this virtual display. This is not an issue, but rather an unexpected benefit. Here is a screenshot. 
By the way, I have ported scrcpy to Android. Once I complete the remaining work, I invite you to experience this software. Thank you very much. |
|
A small issue: IME will not appear on the virtual display unless the following setting is applied: This is likely because the Additionally, on some devices, the launcher displays incorrectly on non-default screens. |
Launcher I tested that works: Lawnchair (playstore, github). Another instance of the launcher can be opened on secondary displays without additional flags. Good enough for quickly launching apps for now scrcpy-vd-lawnchair.mp4 |
|
@rom1v thank you for working on this feature! I'm getting permission issue on android 12. @yume-chan's proof-of-concept works on this device |
|
We probably need to remove this flag for Android < 13 (according to c7b04a6): diff --git a/server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java b/server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java
index 0622998ed..9440199ce 100644
--- a/server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java
+++ b/server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java
@@ -85,7 +85,6 @@ public class NewDisplayCapture extends SurfaceCapture {
| VIRTUAL_DISPLAY_FLAG_ROTATES_WITH_CONTENT
| VIRTUAL_DISPLAY_FLAG_DESTROY_CONTENT_ON_REMOVAL
| VIRTUAL_DISPLAY_FLAG_SHOULD_SHOW_SYSTEM_DECORATIONS
- | VIRTUAL_DISPLAY_FLAG_TRUSTED
| VIRTUAL_DISPLAY_FLAG_OWN_DISPLAY_GROUP
| VIRTUAL_DISPLAY_FLAG_ALWAYS_UNLOCKED
| VIRTUAL_DISPLAY_FLAG_TOUCH_FEEDBACK_DISABLED |
|
|
@eiyooooo The flag exists, but apparently in Android 12, shell does not have permission to create a virtual display with this flag enabled. |
Yes, at least Android 13 is required to use the |
If you don't pass it, what happens? |
Based on my tests, the flags |
On the S20+ (Android 13), after multiple attempts to create virtual displays using |
|
I added features to list and start apps. I updated the main post, and also published a binary for this new version. Please test and give feedback 😉 |
rom1v
force-pushed
the
virtual_display
branch
from
51fff66 to
11eca93
Compare
After digging deeper, I found that Android 12L still encounters the following issue: References:
Therefore, I think this part of the code: should be updated to: if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU) { |
rom1v
force-pushed
the
virtual_display
branch
from
11eca93 to
e828667
Compare
|
@eiyooooo OK.
Indeed, it has been merged in Android 13: # in aosp/framework_base
git tag --contains 990e3429636382175ca8e7bf04df054f48fbd130I first added readable Android version constants (already merged into Then I set the TRUSTED and other flags only since Android 13: e05be0b |
@rom1v By the way, Android 15 has been released, so you'll need to add a new Android version constant. 😄 public static final int API_35_ANDROID_15 = Build.VERSION_CODES.VANILLA_ICE_CREAM; |
rom1v
force-pushed
the
virtual_display
branch
from
e828667 to
b771db3
Compare
|
Given that loading the app names may take time (several seconds), by default, I reworked so that For convenience, however, it is still possible to explicitly select an app by its name (but with some delay) by adding a Like in the previous version, a scrcpy --new-display --start-app=+org.mozilla.firefox # by package name
scrcpy --new-display --start-app=+?firefox # by name |
|
I'm getting a issue, when the phone goes to lock screen i lose control(keyboard and left&right buttons, if a home button is pressed it is work, so just input actions related to lock screen works.) of virtual display. |
|
@mauriciofelippe On Samsung phones, clicks stop working on virtual display when the device goes to sleep apparently: #5557 (comment) |
Move the code related to screen size and rotation/fold to ScreenCapture. For now, keep the ScreenInfo instance in the Device class to communicate with the Controller, but it will be removed by further commits. PR Genymobile#5370 <Genymobile#5370>
An event is posted on registration to signal the initial state. This had no impact when the listener was registered from Device (before it was moved to ScreenCapture), because this first initial event was already triggered when ScreenCapture started listening. But now, it causes the first encoding to be reset immediately. To avoid that, ignore the first event. Refs <https://android.googlesource.com/platform/frameworks/base.git/+/refs/tags/android-15.0.0_r3/services/core/java/com/android/server/policy/DisplayFoldController.java#138> PR Genymobile#5370 <Genymobile#5370>
Avoid to call capture.getSize() (provided by the SurfaceCapture implementation) twice. PR Genymobile#5370 <Genymobile#5370>
Do not initialize variables when they are not used. PR Genymobile#5370 <Genymobile#5370>
Do not use an unnecessary intermediate Rect object. PR Genymobile#5370 <Genymobile#5370>
Add a function called before each capture starts (before getSize() is called). This allows to compute the ScreenInfo instance once exactly when needed. PR Genymobile#5370 <Genymobile#5370>
Remove from Device the functions using an implicit displayId. Move them to Controller, which knows best which displayId it must use. This will allow to properly dispatch events either to the origin display or to the virtual display created for mirroring. PR Genymobile#5370 <Genymobile#5370>
Continue to declutter the global Device. PR Genymobile#5370 <Genymobile#5370>
Extract the function that converts coordinates from video space to display space into a separate component. It only requires the specific data it uses and does not need a full ScreenInfo object (although it can be created from a ScreenInfo instance). PR Genymobile#5370 <Genymobile#5370>
When a new capture starts, send a new PositionMapper to the Controller without using the global Device as an intermediate. Now all Device methods are static. PR Genymobile#5370 <Genymobile#5370>
Mouse and touch events must be sent to the virtual display id (used for mirroring), other events (like key events) must be sent to the original display id. Fixes Genymobile#4598 <Genymobile#4598> Fixes Genymobile#5137 <Genymobile#5137> PR Genymobile#5370 <Genymobile#5370> Co-authored-by: nightmare <[email protected]>
It will be useful to automatically set an appropriate DPI for new virtual displays. PR Genymobile#5370 <Genymobile#5370>
Add a feature to create a new (separate) virtual display instead of
mirroring the device screen:
scrcpy --new-display=1920x1080
scrcpy --new-display=1920x1080/420 # force 420 dpi
scrcpy --new-display # use the main display size and density
scrcpy --new-display -m1920 # scaled to fit a max size of 1920
scrcpy --new-display=/240 # use the main display size and 240 dpi
Fixes Genymobile#1887 <Genymobile#1887>
PR Genymobile#5370 <Genymobile#5370>
Co-authored-by: Simon Chan <[email protected]>
Co-authored-by: anirudhb <[email protected]>
Fail explicitly if a new virtual display is requested on an Android version lower than 10. PR Genymobile#5370 <Genymobile#5370>
Add an option to list all apps installed on the device:
scrcpy --list-apps
PR Genymobile#5370 <Genymobile#5370>
Add a command line option --start-app=name to start an Android app by
its package name. For example:
scrcpy --start-app=org.mozilla.firefox
The app will be started on the correct target display:
scrcpy --new-display=1920x1080 --start-app=org.videolan.vlc
PR Genymobile#5370 <Genymobile#5370>
Co-authored-by: Simon Chan <[email protected]>
The previous commit introduced:
scrcpy --start-app=name
By adding a '+' prefix, the app is stopped beforehand:
scrcpy --start-app=+name
This may be useful to start a fresh app on a new virtual display:
scrcpy --new-display --start-app=+org.mozilla.firefox
PR Genymobile#5370 <Genymobile#5370>
By adding the '?' prefix, the app is searched by its name instead of its
package name (retrieving app names on the device may take some time):
scrcpy --start-app=?firefox
An app matches if its label starts with the given name,
case-insensitive.
If '+' is also passed to force-stop the app before starting, then the
prefixes must be in that order:
scrcpy --start-app=+?firefox
PR Genymobile#5370 <Genymobile#5370>
Listen to display size changes and rotate the virtual display accordingly. Note: use `git show -b` to Show this commit ignoring whitespace changes. Fixes Genymobile#5428 <Genymobile#5428> Refs Genymobile#5370 <Genymobile#5370> PR Genymobile#5455 <Genymobile#5455>
Allow capturing virtual displays at a lower resolution using -m/--max-size. In the original implementation in Genymobile#5370, the virtual display size was necessarily the same as the capture size. The --max-size value was only allowed to determine the virtual display size when no explicit size was provided. Since the dpi was scaled down accordingly, it is often better to create a virtual display at the target capture size directly. However, not everything is rendered according to the virtual display DPI. For example, a page in Firefox is rendered too big on small virtual displays. Thus, it makes sense to be able create a virtual display at a given size, and capture it at a lower resolution with --max-size. This is now possible using OpenGL filters. Therefore, change the behavior of --max-size for virtual displays: - --max-size does not impact --new-display without size argument anymore (the virtual display size is the main display size); - it is used to limit the capture size (whether an explicit size is provided or not). This new behavior is consistent with main display capture. Refs Genymobile#5370 comment <Genymobile#5370 (comment)> Refs Genymobile#5370 <Genymobile#5370> PR Genymobile#5506 <Genymobile#5506>
When mirroring a secondary display, touch and scroll events must be sent to the mirroring virtual display id (with coordinates relative to the virtual display size), rather than to the original display (with coordinates relative to the original display size). This behavior, introduced by d193967, was also applied for the main display for consistency. However, it causes some UI elements to become unclickable. To minimize inconveniences, restore the previous behavior when mirroring the main display: send all events to the original display id (0) with coordinates relative to the original display size. Fixes Genymobile#5545 <Genymobile#5545> Fixes Genymobile#5605 <Genymobile#5605> Fixes Genymobile#5616 <Genymobile#5616> Refs Genymobile#4598 <Genymobile#4598> Refs Genymobile#5137 <Genymobile#5137> Refs Genymobile#5370 <Genymobile#5370> PR Genymobile#5614 <Genymobile#5614>
This PR adds the possibility to start a new virtual display instead of mirroring the screen.
Implementation
There are 3 steps:
Refactors
The (7) firsts commits refactor to move code from
DevicetoControllerandScreenCapture, to prepare the next commits.Events handling
The commit
Inject display-related events to virtual displayaims to fix #4598 and #5137 (based on the work by @mengyanshou in #5137 and #5214), also necessary for making virtual display events work.Basically, for event injection, there are two display ids:
--display-id=…, default to 0 for the main display)(In case the ScreenCapture uses the "SurfaceControl API", then both ids are equals, but this is an implementation detail.)
In order to make events work correctly in all cases:
Virtual display
Then, based on the discussions in #1887 and the work by @yume-chan and @anirudhb:
The commit
Add virtual display featureimplements a first version of virtual display support:The size is fixed (not resizable).
On a Pixel 8, the virtual display contains a menu, so it is possible to open an app easily.
On other devices, it may be empty. In that case, you can start your app manually. Find the new display id is scrcpy logs (let's say it's 42), and start an app manually:
To start an app manually:
TODO
Start app
The feature really needs a new command to start an app, for example:
# not implemented, just an idea scrcpy --new-display --start-app=firefoxI might need to rework how arguments are passed to the server (to start an app "My gallery" for example), because currently it does not support spaces or other special characters (see bec3321).
I don't know if the expected behavior is to just have a virtual display and open any app (like on Pixel 8, where there is a "launcher"), or expose something like a "single app" feature.
It probably depends on the device (I don't know if it's possible to limit a single app on Pixel 8 for example, or to navigate to other apps on device without a launcher). What do you think? (also ref discussion #1887 (comment) @4nric)
Detect failure
If I run
scrcpy --new-displayon a Nexus 5 with Android 6, it does not fail, it just waits indefinitely for a frame. How to detect that "it works"? Should we only use checks based on the Android version?Draft v2
Start app
Here is a draft v2, with new options to list and start apps (edit: changed by #5370 (comment)):
There is a performance issue on some devices to retrieve the app names (but not package names), because
loadLabel()must be called for each app). As a consequence,--list-appsmay take several seconds, and selecting an application by name may also take time. However, starting via package name is quick. Any solution welcome.