Refactor, update, and move around testing/scenario_app/README.md

testing/scenario_app/README.md

@@ -1,85 +1,43 @@
 # Scenario App
 
-This folder contains e2e integration tests for the engine in conjunction with a
-fake dart:ui framework running in JIT or AOT.
+[![GitHub Issues or Pull Requests by label](https://img.shields.io/github/issues/flutter/flutter/e%3A%20scenario-app)](https://github.com/flutter/flutter/issues?q=is%3Aopen+is%3Aissue+label%3A%22e%3A+scenario-app%22)
 
-It intentionally has no dependencies on the Flutter framework or tooling, such
-that it should be buildable as a presubmit or postsubmit to the engine even in
-the face of changes to Dart or dart:ui that require upstream changes in the
-Flutter tooling.
+This package simulates a Flutter app that uses the engine (`dart:ui`) only,
+in conjunction with Android and iOS-specific embedding code that simulates the
+use of the engine in a real app (such as plugins and platform views).
 
-## Adding a New Scenario
+The [`run_android_tests.sh`](run_android_tests.sh) and
+[`run_ios_tests.sh`](run_ios_tests.sh) are then used to run the tests on a
+connected device or emulator.
 
-Create a new subclass of [Scenario](https://github.com/flutter/engine/blob/5d9509ae056b04c30295df27f201f31af9777842/testing/scenario_app/lib/src/scenario.dart#L9)
-and add it to the map in [scenarios.dart](https://github.com/flutter/engine/blob/db4d423ad9c6dad373618712690acd06b0a385fd/testing/scenario_app/lib/src/scenarios.dart#L22).
-For an example, see [animated_color_square.dart](https://github.com/flutter/engine/blob/5d9509ae056b04c30295df27f201f31af9777842/testing/scenario_app/lib/src/animated_color_square.dart#L15),
-which draws a continuously animating colored square that bounces off the sides
-of the viewport.
+See also:
 
-Then set the scenario from the Android or iOS app by calling "set_scenario" on
-platform channel.
+- [`bin/`](bin/), the entry point for running Android integration tests.
+- [`lib/`](lib/), the Dart code and instrumentation for the scenario app.
+- [`ios/`](ios/), the iOS-side native code and tests.
+- [`android/`](android/), the Android-side native code and tests.
 
-## Running for iOS
+## Running a smoke test on Firebase TestLab
 
-Build the `ios_debug_sim_unopt` engine variant, and run
+To run the smoke test on Firebase TestLab test, build `android_profile_arm64`,
+and run [`./ci/firebase_testlab.py`](../../ci/firebase_testlab.py), or pass
+`--variant` to run a different configuration.
 
 ```sh
-./run_ios_tests.sh
-```
-
-in your shell.
-
-To run or debug in Xcode, open the xcodeproj file located in
-`<engine_out_dir>/ios_debug_sim_unopt/scenario_app/Scenarios/Scenarios.xcodeproj`.
-
-### iOS Platform View Tests
-
-For PlatformView tests on iOS, you'll also have to edit the dictionaries in
-[AppDelegate.m](https://github.com/flutter/engine/blob/5d9509ae056b04c30295df27f201f31af9777842/testing/scenario_app/ios/Scenarios/Scenarios/AppDelegate.m#L29) and [GoldenTestManager.m](https://github.com/flutter/engine/blob/db4d423ad9c6dad373618712690acd06b0a385fd/testing/scenario_app/ios/Scenarios/ScenariosUITests/GoldenTestManager.m#L25) so that the correct golden image can be found.  Also, you'll have to add a [GoldenPlatformViewTests](https://github.com/flutter/engine/blob/5d9509ae056b04c30295df27f201f31af9777842/testing/scenario_app/ios/Scenarios/ScenariosUITests/GoldenPlatformViewTests.h#L18) in [PlatformViewUITests.m](https://github.com/flutter/engine/blob/af2ffc02b72af2a89242ca3c89e18269b1584ce5/testing/scenario_app/ios/Scenarios/ScenariosUITests/PlatformViewUITests.m).
-
-If `PlatformViewRotation` is failing, make sure Simulator app Device > Rotate Device Automatically
-is selected, or run:
-
-```bash
-defaults write com.apple.iphonesimulator RotateWindowWhenSignaledByGuest -int 1
+# From the root of the engine repository
+$ ./ci/firebase_testlab.py --variant android_debug_arm64
 ```
 
-### Generating Golden Images on iOS
-
-Screenshots are saved as
-[XCTAttachment](https://developer.apple.com/documentation/xctest/activities_and_attachments/adding_attachments_to_tests_and_activities?language=objc)'s.
-If you look at the output from running the tests you'll find a path in the form:
-`/Users/$USER/Library/Developer/Xcode/DerivedData/Scenarios-$HASH`.
-Inside that directory you'll find
-`./Build/Products/Debug-iphonesimulator/ScenariosUITests-Runner.app/PlugIns/ScenariosUITests.xctest/` which is where all the images that were
-compared against golden reside.
-
-## Running for Android
+> [!NOTE]
+> These instructions were not verified at the time of writing/refactoring.
 
-### Integration tests
-
-For emulators running on a x64 host, build `android_debug_unopt_x64` using
-`./tools/gn --android --unoptimized --goma --android-cpu=x64`.
-
-Then, launch the emulator, and run `./testing/scenario_app/run_android_tests.sh android_debug_unopt_x64`.
-
-If you wish to build a different engine variant, make sure to pass that variant to the script `run_android_tests.sh`.
-
-If you make a change to the source code, you would need to rebuild the same engine variant.
-
-### Smoke test on FTL
-
-To run the smoke test on Firebase TestLab test, build `android_profile_arm64`, and run
-`./flutter/ci/firebase_testlab.py`. If you wish to test a different variant, e.g.
-debug arm64, pass `--variant android_debug_arm64`.
-
-### Updating Gradle dependencies
-
-If a Gradle dependency is updated, lockfiles must be regenerated.
+## Adding a New Scenario
 
-To generate new lockfiles, run:
+Create a new subclass of [Scenario](lib/src/scenario.dart) and add it to the map
+in [scenarios.dart](lib/src/scenarios.dart). For an example, see
+[animated_color_square.dart](lib/src/animated_color_square.dart), which draws a
+continuously animating colored square that bounces off the sides of the
+viewport.
 
-```bash
-cd android/app
-../../../../../third_party/gradle/bin/gradle generateLockfiles
-```
+Then set the scenario from the Android or iOS app by calling `set_scenario` on
+platform channel `driver`.

testing/scenario_app/android/README.md

@@ -0,0 +1,42 @@
+# Scenario App: Android Tests
+
+As mentioned in the [top-level README](../README.md), this directory contains
+the Android-specific native code and tests for the [scenario app](../lib). To
+run the tests, you will need to build the engine with the appropriate
+configuration.
+
+For example, `android_debug_unopt` or `android_debug_unopt_arm64` was built,
+run:
+
+```sh
+# From the root of the engine repository
+$ ./testing/run_android_tests.sh android_debug_unopt
+
+# Or, for arm64
+$ ./testing/run_android_tests.sh android_debug_unopt_arm64
+```
+
+## CI Configuration
+
+See [`ci/builders/linux_android_emulator.json`](../../../ci/builders/linux_android_emulator.json)
+, and grep for `run_android_tests.sh`.
+
+The following matrix of configurations is tested on the CI:
+
+| API Version | Graphics Backend    | Skia Gold                                                        | Rationale                                                  |
+| ----------- | ------------------- | ---------------------------------------------------------------- | ---------------------------------------------------------- |
+| 28          | Skia                | [Android 28 + Skia][skia-gold-skia-28]                           | Older Android devices (without `ImageReader`) on Skia.     |
+| 28          | Impeller (OpenGLES) | [Android 28 + Impeller OpenGLES][skia-gold-impeller-opengles-28] | Older Android devices (without `ImageReader`) on Impeller. |
+| 34          | Skia                | [Android 34 + Skia][skia-gold-skia-34]                           | Newer Android devices on Skia.                             |
+| 34          | Impeller (OpenGLES) | [Android 34 + Impeller OpenGLES][skia-gold-impeller-opengles-34] | Newer Android devices on Impeller with OpenGLES.           |
+| 34          | Impeller (Vulkan)   | [Android 34 + Impeller Vulkan][skia-gold-impeller-vulkan-34]     | Newer Android devices on Impeller.                         |
+
+[skia-gold-skia-28]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D28%26GraphicsBackend%3Dskia&negative=true&positive=true
+[skia-gold-impeller-opengles-28]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D28%26GraphicsBackend%3Dimpeller-opengles&negative=true&positive=true
+[skia-gold-skia-34]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dskia&negative=true&positive=true
+[skia-gold-impeller-opengles-34]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dimpeller-opengles&negative=true&positive=true
+[skia-gold-impeller-vulkan-34]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dimpeller-vulkan&negative=true&positive=true
+
+## Updating Gradle dependencies
+
+See [Updating the Embedding Dependencies](../../../tools/cipd/android_embedding_bundle/README.md).

testing/scenario_app/bin/README.md

@@ -0,0 +1,40 @@
+# `android_integration_tests` runner
+
+This directory contains code specific to running Android integration tests.
+
+The tests are uploaded and run on the device using `adb`, and screenshots are
+captured and compared using Skia Gold (if available, for example on CI).
+
+## Usage
+
+```sh
+dart bin/android_integration_tests.dart \
+  --adb ../third_party/android_tools/sdk/platform-tools/adb \
+  --out-dir ../out/android_debug_unopt_arm64
+```
+
+## Debugging
+
+When debugging, you can use the `--smoke-test` argument to run a single test
+by class name, which can be useful to verify the setup.
+
+For example, to run the `EngineLaunchE2ETest` test:
+
+```sh
+dart bin/android_integration_tests.dart \
+  --adb ../third_party/android_tools/sdk/platform-tools/adb \
+  --out-dir ../out/android_debug_unopt_arm64 \
+  --smoke-test dev.flutter.scenarios.EngineLaunchE2ETest
+```
+
+## Additional arguments
+
+- `--use-skia-gold`: Use Skia Gold to compare screenshots. Defaults to true
+  when running on CI, and false otherwise (i.e. when running locally). If
+  set to true, [isSkiaGoldClientAvailable] must be true.
+
+- `--enable-impeller`: Enable Impeller for the Android app. Defaults to
+  false, which means that the app will use Skia as the graphics backend.
+
+- `--impeller-backend`: The Impeller backend to use for the Android app.
+  Defaults to 'vulkan'. Only used when `--enable-impeller` is set to true.

testing/scenario_app/bin/android_integration_tests.dart

@@ -16,8 +16,7 @@ import 'utils/logs.dart';
 import 'utils/process_manager_extension.dart';
 import 'utils/screenshot_transformer.dart';
 
-const int tcpPort = 3001;
-
+// If you update the arguments, update the documentation in the README.md file.
 void main(List<String> args) async {
   final ArgParser parser = ArgParser()
     ..addOption(
@@ -82,6 +81,8 @@ void main(List<String> args) async {
   );
 }
 
+const int _tcpPort = 3001;
+
 enum _ImpellerBackend {
   vulkan,
   opengles;
@@ -137,7 +138,7 @@ Future<void> _run({
   late  ServerSocket server;
   final List<Future<void>> pendingComparisons = <Future<void>>[];
   await step('Starting server...', () async {
-    server = await ServerSocket.bind(InternetAddress.anyIPv4, tcpPort);
+    server = await ServerSocket.bind(InternetAddress.anyIPv4, _tcpPort);
     stdout.writeln('listening on host ${server.address.address}:${server.port}');
     server.listen((Socket client) {
       stdout.writeln('client connected ${client.remoteAddress.address}:${client.remotePort}');
@@ -237,7 +238,7 @@ Future<void> _run({
     });
 
     await step('Reverse port...', () async {
-      final int exitCode = await pm.runAndForward(<String>[adb.path, 'reverse', 'tcp:3000', 'tcp:$tcpPort']);
+      final int exitCode = await pm.runAndForward(<String>[adb.path, 'reverse', 'tcp:3000', 'tcp:$_tcpPort']);
       if (exitCode != 0) {
         panic(<String>['could not forward port']);
       }

testing/scenario_app/ios/README.md

@@ -0,0 +1,55 @@
+# Scenario App: iOS Tests
+
+As mentioned in the [top-level README](../README.md), this directory contains
+the iOS-specific native code and tests for the [scenario app](../lib). To run
+the tests, you will need to build the engine with the appropriate configuration.
+
+For example, after building `ios_debug_sim_unopt` (to run on Intel Macs) or `ios_debug_sim_unopt_arm64` (to run on ARM Macs),
+run:
+
+```sh
+# From the root of the engine repository
+$ ./testing/run_ios_tests.sh ios_debug_sim_unopt
+```
+or 
+```sh
+# From the root of the engine repository
+$ ./testing/run_ios_tests.sh ios_debug_sim_unopt_arm64
+```
+
+To run or debug in Xcode, open the xcodeproj file located in
+`<engine_out_dir>/ios_debug_sim_unopt/scenario_app/Scenarios/Scenarios.xcodeproj`.
+
+## CI Configuration
+
+See [`ci/builders/mac_unopt.json`](../../../../ci/builders/mac_unopt.json), and
+grep for `run_ios_tests.sh`.
+
+## iOS Platform View Tests
+
+For PlatformView tests on iOS, edit the dictionaries in
+[AppDelegate.m](Scenarios/Scenarios/AppDelegate.m) and
+[GoldenTestManager.m](Scenarios/ScenariosUITests/GoldenTestManager.m) so that
+the correct golden image can be found. Also, add a
+[GoldenPlatformViewTests](Scenarios/ScenariosUITests/GoldenPlatformViewTests.h)
+in [PlatformViewUITests.m](Scenarios/ScenariosUITests/PlatformViewUITests.m).
+
+If `PlatformViewRotation` is failing, make sure
+`Simulator app Device > Rotate Device Automatically` is selected, or run:
+
+```bash
+defaults write com.apple.iphonesimulator RotateWindowWhenSignaledByGuest -int 1
+```
+
+## Generating Golden Images on iOS
+
+Screenshots are saved as
+[XCTAttachment](https://developer.apple.com/documentation/xctest/activities_and_attachments/adding_attachments_to_tests_and_activities?language=objc)'s.
+
+A path in the form of
+`/Users/$USER/Library/Developer/Xcode/DerivedData/Scenarios-$HASH` will be
+printed to the console.
+
+Inside that directory there is a directory
+`./Build/Products/Debug-iphonesimulator/ScenariosUITests-Runner.app/PlugIns/ScenariosUITests.xctest/`
+which is where all the images that were compared against golden reside.