feat: Implement internal audio/video recording for devices >= Android Q

[sirmordred]

…Android.Q

REQUIRED STEPS to activate screenrecord feature:

• adb shell pm grant io.appium.settings android.permission.RECORD_AUDIO
• adb shell appops set io.appium.settings PROJECT_MEDIA allow

TO START SCREENRECORD:

• adb shell am start -n "io.appium.settings/io.appium.settings.Settings" -a io.appium.settings.recording.ACTION_START --es filename abc.mp4

TO STOP SCREENRECORD:

• adb shell am start -n "io.appium.settings/io.appium.settings.Settings" -a io.appium.settings.recording.ACTION_STOP

TO PULL RECORDED MP4 FILE:

• adb pull /storage/emulated/0/Android/data/io.appium.settings/files/abc.mp4 abc.mp4

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: sirmordred / name: Oğuzhan Yiğit (6366d21, dc868e5, 54594d7, e457e35, 6a203ac, b2f96c7, 031b55b, 396e861, 80afc5f, cb83b09, ce4e1ae, b951e4e, 2230b3d, deff3af, 42c5d34, dc8dc27, 9177d64, 1f0d2a0, f6624c5, f878b49, deae0f7, 4ff72ed, a109618, d1ae86a, cd56a26, d229c00, 0c1907e, afa8fbc, a49f98e, ee1bb4e, 404754c, f0b2628, 2c8878a, 991c57e, f618ac9, 203ef2d, 4a0e029, 92eb6c0, c279b9e, 1fafbbe, d646cff, a61a988, 1326e79, cf3aa47, bc9ba91)

[sirmordred]

cc @KazuCocoa @mykola-mokhnach

[sirmordred]

addressing appium/appium#13177

[sirmordred]

It works flawlessly as far as I tested it on my android device with latest appium, note: io.appium.settings apk which includes this commit must be built and installed and above 3 adb commands need to be issued first, here is sample test script including recording https://gist.github.com/sirmordred/4c505ddce96f7513bfbe49d38ef041be

[KazuCocoa]

Thank you.
Is there any reason why you did not add this feature in https://github.com/appium/appium-uiautomator2-server , for example, as a mobile: command? https://github.com/appium/appium-uiautomator2-driver
(kind of a basic question as my out of curiosity)

[updated]

ah, this was for the below comment in the linked issue?

Which means one can simply install any sound recording app, control it using UIA2 to start/stop recording process and be happy about it %)

[KazuCocoa]

Should this be start activity? I wonder if this feature can be via broadcast. (I haven't dug in the recording feature, so I might miss something)

[sirmordred]

i tried the initiating recording with broadcast but since i had to trigger startActivityForResult() method supplied with mediaprojection intent object (see https://github.com/sirmordred/io.appium.settings/blob/master/app/src/main/java/io/appium/settings/Settings.java#L108) i had to use Settings activity

[sirmordred]

but i will try to dig more on this

[mykola-mokhnach]

AFAIK app sound would only be recorded if the particular app allows that. Also, does the appops command below suppress the confirmation dialog that Media projection usually shows before starting the recording?

[sirmordred]

thank you so much for your review and feedbacks, i'll address them asap i hope, im doing this in my free time and i'll be on vacation for a while, after that i will continue to work on it and try to address your questions etc. thanks

[sirmordred]

okay while i was testing recording in different emulators (even if it worked perfect on real devices, Huawei Android 10 and Samsung Android 12) it crashed on emulators (media encoder was crashing while configuring it with given width/height resolutions) and i realized that this is because media codecs and its capabilities as well as device display resolutions vary device-to-device very much and the core problem is media encoder/decoder (encoder in our case) is not GUARANTEED to encode/decode in given any resolutions (width/height)

in the past, we were giving raw display width/height to media encoder https://github.com/sirmordred/io.appium.settings/blob/203ef2dd68231f69d368a3a64a1909f614f3aaa8/app/src/main/java/io/appium/settings/recorder/RecorderThread.java#L372 of course we were clamping it, but clamp was not not working reliably, even if we clamp width/height against media encoder's capabilities, it was resulting values like 1080x1823 (some devices have notch or virtual back/home button fields some not) and media encoder was crashing with these type of values,

so i deep dive into media encoder and its capabilities, i saw that media encoder can support up to 5 different resolution modes, i got that assumption from CTS tests, see https://github.com/aosp/cts/blob/08e720f8ee9a1170d995b03b748528098f22e101/tests/tests/media/src/android/media/cts/VideoEncoderTest.java#L1346 and above, since all released android phones MUST pass CTS tests, we can safely assume that media encoder can support these resolution modes, so i added these resolution modes and helper functions to detect supported max resolution on-device, and media encoder is now works without crashing as far as i tested it on both real-devices and emulators 👍 i also added user-selectable-resolution-mode argument to making it dynamic (didnt test different resolutions yet, only tested max supported resolution on device, but will test it ASAP 👍 EDIT: i tried and it works great #88 (comment) )

other changes:

• i was giving display density as hardcoded while configuring VirtualDisplay sirmordred@4a0e029#diff-423e8dd14708c0aa3b8df6ff5c75cd8a780ea48625ac50a32a194b167cfe1259R138 so i changed it to dynamic since we can get value from device directly
• I added additional control that checks if media projection permission is granted or not (user typed adb shell appops project_media allow command or not) sirmordred@203ef2d to make recording more solid (tested it on different phones and works great) 👍

[sirmordred]

Voilaa 💯 i tested different resolution modes and it works perfect 👍 and resulting output videos are really smaller in size (as expected) we now support different resolutions 👍

[mykola-mokhnach]

AFAIK the AVC video encoder used here supports pretty wide set of resolutions. One of main requirements to these resolutions is, though, that both width and height must be divisible by two. You could also use https://stackoverflow.com/questions/44902334/how-to-get-max-resolution-of-mediacodec-before-configuring to check if the particular width, height and frame rate are supported by the selected codec instead of hardcoding the limits. Usually one would only provide the scale value to keep the original aspect ratio rather than both width and height

[mykola-mokhnach]

In iOS, for example, we set the scale value to 2, which means the video resolution is by default half of the original device resolution

[sirmordred]

AFAIK the AVC video encoder used here supports pretty wide set of resolutions. One of main requirements to these resolutions is, though, that both width and height must be divisible by two. You could also use https://stackoverflow.com/questions/44902334/how-to-get-max-resolution-of-mediacodec-before-configuring to check if the particular width, height and frame rate are supported by the selected codec instead of hardcoding the limits. Usually one would only provide the scale value to keep the original aspect ratio rather than both width and height

i thought the same but in android's AVC/H264 codec, i saw and experienced supported resolutions are not really that much wide, for example try the following code on any high-end phone or low-end phone (it doesn't matter)

try {
    String videoMime = MediaFormat.MIMETYPE_VIDEO_AVC;

    MediaCodec videoEncoder = MediaCodec.createEncoderByType(videoMime);

    MediaCodecInfo.VideoCapabilities videoEncoderCapabilities = videoEncoder.getCodecInfo().
            getCapabilitiesForType(videoMime).getVideoCapabilities();

    boolean a1 = videoEncoderCapabilities.isSizeSupported(176, 144);
    boolean a2 = videoEncoderCapabilities.isSizeSupported(320, 240);
    boolean a3 = videoEncoderCapabilities.isSizeSupported(720, 480);
    boolean a4 = videoEncoderCapabilities.isSizeSupported(1280, 720);
    boolean a5 = videoEncoderCapabilities.isSizeSupported(1920, 1080);
    boolean a6 = videoEncoderCapabilities.isSizeSupported(2560, 1440);


    Log.e("OGIX", a1 ? "yes1" : "no");
    Log.e("OGIX", a2 ? "yes2" : "no");
    Log.e("OGIX", a3 ? "yes3" : "no");
    Log.e("OGIX", a4 ? "yes4" : "no");
    Log.e("OGIX", a5 ? "yes5" : "no");
    Log.e("OGIX", a6 ? "yes6" : "no");
} catch (Exception e) {
    e.printStackTrace();
}

it will print "yes" for all resolutions from 176x144 to 1920x1080 range but it will print "no" for the last one -native 2K resolution 2560x1440- even though it is divisible by 2, so we are much likely limited to these resolutions i guess

[sirmordred]

Okay i have done everything in my mind/which i noted 👍 also tried on the following devices and i can confirm its working good without problem 👍

• Huawei P40 Lite running Android 10
• Samsung A53 running Android 12
• Xiaomi MI5 running Android 12 (customized ROM, LineageOS)
• [Emulator] Pixel C Tablet running Android 11
• [Emulator] Pixel 2 Phone running Android 11
  Nearly all devices have also different resolutions and dpi as well as display sizes

EDIT: lastly i tried it on Pixel 2 Phone Emulator running upcoming Android 13, and it worked fine too without problem 👍

[mykola-mokhnach]

usually when we are talking about priorities then lower value means higher priority. Wouldn't the current notation be confusing?

[mykola-mokhnach]

maybe it even makes more sence to provide priority as a string (e.g. high/normal/low)

[sirmordred]

good idea 👍 done, i changed it now to string (high/normal/low) sirmordred@a61a988

[mykola-mokhnach]

I would rather prefer to explicitly provide resolution string

[sirmordred]

you mean we should enforce resolution arg as Mandatory ? or passing resolution arg as string (full_hd, hd, 480p etc.) instead of integer?

[mykola-mokhnach]

pass a string instead of integer, e.g. 640x480 or 1920x1080

[sirmordred]

okay done sirmordred@bc9ba91 reason i couldn't use switch-case in here sirmordred@bc9ba91#diff-f3d89bcfbd433c431567a1fc5306d1e60589ae7abfc5add09b4e0bbcb87cbe66R256 is that it doesn't support android.util.Size object 👍 also shortened argument name resolution_mode to resolution 👍

[mykola-mokhnach]

Just an idea for the future:
I assume the main issue with runtime rotation is the fact we cannot change the video resolution dynamically in runtime. So it makes sence to setup the maximum video resolution as (max(W, H) x max(W, H)), which could potentially fit both orientations. Of course, this should be optional.

[sirmordred]

you mean values like 1920x1920 or 4096x4096 right? yeah that is good idea and can potentially fit both orientations as you said 👍 noted, will try to sum up and write as comment all future-directions after the PR 👍

[mykola-mokhnach]

Ideas for the future:
It would be nice to have FPS value configurable

[sirmordred]

+1 yeah, noted, will try to sum up and write as comment all future-directions after the PR 👍

[mykola-mokhnach]

had some minor comments though

[sirmordred]

thank you so much for the review @mykola-mokhnach 👍

after resolving last feedback #88 (comment) and receive LGTM from you 👍 i'll squash all commits into single one (not doing this for now because of information loss of commits) and sign EasyCLA 👍

[sirmordred]

Okay i squashed all changes into single commit but individual commit history can be seen through here (i opened another branch) https://github.com/sirmordred/io.appium.settings/tree/recording_feature and also signed EasyCLA and after all changes, i double-checked that it runs without problem on my real test phones 👍

[sirmordred]

FUTURE DIRECTIONS:

• Add user selectable argument/flag to support the following modes separately A) Audio-only recording B) Video-only recording C) Audio/Video Mixed recording(currently supported)
• Add user selectable FPS argument/flag to dynamically control frame rate
• Add real orientation support with runtime rotation change handled(by manipulating/rotating frames via GL/MatrixTransformer utils) (Side note: setup the maximum video resolution as (max(W, H) x max(W, H)), which could potentially fit both portrait and landscape orientation modes)
• Add and implement START/STOP/PULL commands into appium-android-driver https://github.com/appium/appium-android-driver to fold and shorten current start/stop commands and make life easier for appium users :)
• Change activity-initiated-recording adb shell am start xxx into broadcast-initiated-recording adb shell am broadcast xxx to eliminate current activity open/close requirement (this is currently impossible AFAIK because result intent data of MediaProjectionManager.createScreenCaptureIntent() must be retrieved through onActivityResult and only-activity-classes can define, initiate and handle onActivityResult)
• Try to support larger resolutions than Full HD 1920x1080 (current maximum supported resolution)

[github-actions]

🎉 This PR is included in version 4.1.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀