Multiple unexpected exceptions after updating to Patrol 2.3.2 #1874

Arkadiusz-futurehome · 2023-11-07T14:24:37Z

Steps to reproduce

Example Integration Test

// ignore_for_file: prefer-static-class, file_names

import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:futurehome_app/generated/l10n.dart';
import 'package:patrol/patrol.dart';
import 'package:pubdev_registry/fh_ui_new/fh_ui_new.dart';

import '../../_src_utils.dart';

/// KEYS
const ValueKey<String> emailLoginFieldKey = ValueKey<String>('emailLoginFieldKey');
const ValueKey<String> passwordLoginFieldKey = ValueKey<String>('passwordLoginFieldKey');

void main() {
  patrolTest(
    'Log in with correct username and incorrect & empty password',
    nativeAutomation: true,
    (PatrolIntegrationTester $) async {
      /// PREPARE
      await TestHelper.startApp($);

      /// TRIGGER AND TEST
      await _invalidLogin($);

      await Future<void>.delayed(const Duration(seconds: 3));
    },
  );
}

Future<void> _invalidLogin(PatrolIntegrationTester $) async {
  await $(emailLoginFieldKey).enterText(Credentials.email);
  await $.waitUntilVisible(
    $(Credentials.email),
    timeout: const Duration(seconds: 10),
  );

  await $(FhButton).tap();
  await $.waitUntilVisible(
    $(S.current.passwordValidatorEmptyError),
    timeout: const Duration(seconds: 10),
  );

  expect(find.text(S.current.passwordValidatorEmptyError), findsOneWidget);

  await $(emailLoginFieldKey).enterText(Credentials.email);
  await $.waitUntilVisible(
    $(Credentials.email),
    timeout: const Duration(seconds: 10),
  );

  await $(passwordLoginFieldKey).enterText(
    FakeCredentials.invalidPassword,
  );
  await $.waitUntilVisible(
    $(FakeCredentials.invalidPassword),
    timeout: const Duration(seconds: 10),
  );

  await $(FhButton).tap();
  await $.waitUntilVisible(
    $(S.current.authLoginError),
    timeout: const Duration(seconds: 10),
  );

  expect(find.text(S.current.authLoginError), findsOneWidget);

  await $.waitUntilVisible(
    $(S.current.authLoginBtn),
    timeout: const Duration(seconds: 10),
  );
}

Before Patrol update this test normally finished with success. But after updating Patrol to the newest version there is an error output:

The following message was thrown:
Multiple exceptions (2) were detected during the running of the
current test, and at least one was unexpected.

Actual results

After updating Patrol to version 2.3.2, we are encountering an issue where multiple exceptions are being thrown during the execution of our tests. The error message states:

The following message was thrown:
Multiple exceptions (2) were detected during the running of the
current test, and at least one was unexpected.

This behavior is unexpected and has started occurring since the update to the latest version of Patrol. The test suite was working without any problems before the update, so it seems to be a regression introduced in the new version. This issue is affecting our testing workflow and needs to be addressed to ensure the reliability of our tests.

Please let me know if you need any more information or if I can provide additional details to help diagnose and resolve this problem.

Logs

    : 
        : > Task :app:connectedDebugAndroidTest
        : Starting 1 tests on Pixel_3a(AVD) - 14
        : 
        : no.futurehome.futurehome_app.MainActivityTest > runDartTest[sdu_integration_tests.src.smoke_tests.01_log_in_tests.01_correct_username_invalid_password_patrol_test Log in with correct username and incorrect & empty password][Pixel_3a(AVD) - 14] FAILED 
        :       java.lang.AssertionError: Dart test failed: sdu_integration_tests.src.smoke_tests.01_log_in_tests.01_correct_username_invalid_password_patrol_test Log in with correct username and incorrect & empty password
        :       ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞═════════════════
        : Tests on Pixel_3a(AVD) - 14 failed: There was 1 failure(s).
        lis 07, 2023 2:46:52 PM com.android.tools.utp.plugins.host.additionaltestoutput.AndroidAdditionalTestOutputPlugin deviceShellAndCheckSuccess
        WARNING: Shell command failed (1): ls "/sdcard/Android/media/no.futurehome.futurehome_app/additional_test_output"
        ls: /sdcard/Android/media/no.futurehome.futurehome_app/additional_test_output: No such file or directory
        
        lis 07, 2023 2:46:52 PM com.android.tools.utp.plugins.host.additionaltestoutput.AndroidAdditionalTestOutputPlugin afterAll
        WARNING: Failed to retrieve additional test outputs from device.
        com.android.ddmlib.SyncException: Remote object doesn't exist!
                at com.android.ddmlib.SyncService.pullFile(SyncService.java:341)
                at com.android.ddmlib.internal.DeviceImpl.pullFile(DeviceImpl.java:1083)
                at com.android.tools.utp.plugins.deviceprovider.ddmlib.DdmlibAndroidDevice.pullFile(DdmlibAndroidDevice.kt)
                at com.android.tools.utp.plugins.deviceprovider.ddmlib.DdmlibAndroidDeviceController.pull(DdmlibAndroidDeviceController.kt:254)
                at com.google.testing.platform.executor.DeviceControllerProxy$pull$2.invoke(DeviceControllerProxy.kt:82)
                at com.google.testing.platform.executor.DeviceControllerProxy$pull$2.invoke(DeviceControllerProxy.kt:82)
                at com.google.testing.platform.core.telemetry.common.noop.NoopDiagnosticsScope.recordEvent(NoopDiagnosticsScope.kt:35)
                at com.google.testing.platform.core.telemetry.TelemetryKt.recordEvent(Telemetry.kt:66)
                at com.google.testing.platform.executor.DeviceControllerProxy.pull(DeviceControllerProxy.kt:79)
                at com.android.tools.utp.plugins.host.additionaltestoutput.AndroidAdditionalTestOutputPlugin.copyAdditionalTestOutputsFromDeviceToHost(AndroidAdditionalTestOutputPlugin.kt:222)
                at com.android.tools.utp.plugins.host.additionaltestoutput.AndroidAdditionalTestOutputPlugin.afterAll(AndroidAdditionalTestOutputPlugin.kt:206)
                at com.google.testing.platform.plugin.PluginLifecycle$onAfterAll$1$2$1.invoke(PluginLifecycle.kt:180)
                at com.google.testing.platform.plugin.PluginLifecycle$onAfterAll$1$2$1.invoke(PluginLifecycle.kt:179)
                at com.google.testing.platform.plugin.PluginLifecycleKt$invokeOrThrow$1.invoke(PluginLifecycle.kt:213)
                at com.google.testing.platform.plugin.PluginLifecycleKt$invokeOrThrow$1.invoke(PluginLifecycle.kt:212)
                at com.google.testing.platform.core.telemetry.common.noop.NoopDiagnosticsScope.recordEvent(NoopDiagnosticsScope.kt:35)
                at com.google.testing.platform.core.telemetry.TelemetryKt.recordEvent(Telemetry.kt:105)
                at com.google.testing.platform.core.telemetry.TelemetryKt.recordEvent$default(Telemetry.kt:98)
                at com.google.testing.platform.plugin.PluginLifecycleKt.invokeOrThrow(PluginLifecycle.kt:212)
                at com.google.testing.platform.plugin.PluginLifecycleKt.invokeOrThrow$default(PluginLifecycle.kt:205)
                at com.google.testing.platform.plugin.PluginLifecycle$onAfterAll$1.invoke(PluginLifecycle.kt:179)
                at com.google.testing.platform.plugin.PluginLifecycle$onAfterAll$1.invoke(PluginLifecycle.kt:173)
                at com.google.testing.platform.core.telemetry.common.noop.NoopDiagnosticsScope.recordEvent(NoopDiagnosticsScope.kt:35)
                at com.google.testing.platform.core.telemetry.TelemetryKt.recordEvent(Telemetry.kt:105)
                at com.google.testing.platform.core.telemetry.TelemetryKt.recordEvent$default(Telemetry.kt:98)
                at com.google.testing.platform.plugin.PluginLifecycle.onAfterAll(PluginLifecycle.kt:173)
                at com.google.testing.platform.executor.SingleDeviceExecutor$execute$3.invoke(SingleDeviceExecutor.kt:83)
                at com.google.testing.platform.executor.SingleDeviceExecutor$execute$3.invoke(SingleDeviceExecutor.kt:82)
                at com.google.testing.platform.result.TestResultListenerManager.afterTestSuite(TestResultListenerManager.kt:140)
                at com.google.testing.platform.runtime.android.driver.AndroidInstrumentationDriver.runTestSuite(AndroidInstrumentationDriver.kt:305)
                at com.google.testing.platform.runtime.android.driver.AndroidInstrumentationDriver.run(AndroidInstrumentationDriver.kt:180)
                at com.google.testing.platform.driver.TestDriverProxy$run$1.invoke(TestDriverProxy.kt:110)
                at com.google.testing.platform.driver.TestDriverProxy$run$1.invoke(TestDriverProxy.kt:109)
                at com.google.testing.platform.core.telemetry.common.noop.NoopDiagnosticsScope.recordEvent(NoopDiagnosticsScope.kt:35)
                at com.google.testing.platform.core.telemetry.TelemetryKt.recordEvent(Telemetry.kt:105)
                at com.google.testing.platform.core.telemetry.TelemetryKt.recordEvent$default(Telemetry.kt:98)
                at com.google.testing.platform.driver.TestDriverProxy.run(TestDriverProxy.kt:109)
                at com.google.testing.platform.executor.SingleDeviceExecutor$execute$5.invoke(SingleDeviceExecutor.kt:87)
                at com.google.testing.platform.executor.SingleDeviceExecutor$execute$5.invoke(SingleDeviceExecutor.kt:87)
                at com.google.testing.platform.executor.SingleDeviceExecutor.runUnlessCancelled(SingleDeviceExecutor.kt:105)
                at com.google.testing.platform.executor.SingleDeviceExecutor.execute(SingleDeviceExecutor.kt:87)
                at com.google.testing.platform.RunnerImpl.run(RunnerImpl.kt:108)
                at com.google.testing.platform.server.strategy.NonInteractiveServerStrategy$run$4.invoke(NonInteractiveServerStrategy.kt:80)
                at com.google.testing.platform.server.strategy.NonInteractiveServerStrategy$run$4.invoke(NonInteractiveServerStrategy.kt:79)
                at com.google.testing.platform.core.telemetry.common.noop.NoopDiagnosticsScope.recordEvent(NoopDiagnosticsScope.kt:35)
                at com.google.testing.platform.core.telemetry.TelemetryKt.recordEvent(Telemetry.kt:66)
                at com.google.testing.platform.server.strategy.NonInteractiveServerStrategy.run(NonInteractiveServerStrategy.kt:79)
                at com.google.testing.platform.main.MainKt$main$4.invokeSuspend(Main.kt:67)
                at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
                at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
                at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
                at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85)
                at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
                at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
                at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
                at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
                at com.google.testing.platform.main.MainKt.main(Main.kt:66)
                at com.google.testing.platform.main.MainKt.main$default(Main.kt:34)
                at com.google.testing.platform.main.MainKt.main(Main.kt)
                at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
                at java.base/java.lang.reflect.Method.invoke(Method.java:578)
                at com.google.testing.platform.launcher.Launcher.main(Launcher.java:149)
        
        lis 07, 2023 2:46:52 PM com.google.testing.platform.RunnerImpl$Companion summarizeAndLogTestResult
        SEVERE: Execute no.futurehome.futurehome_app.MainActivityTest.runDartTest[sdu_integration_tests.src.smoke_tests.01_log_in_tests.01_correct_username_invalid_password_patrol_test Log in with correct username and incorrect & empty password]: FAILED
        java.lang.AssertionError: java.lang.AssertionError: Dart test failed: sdu_integration_tests.src.smoke_tests.01_log_in_tests.01_correct_username_invalid_password_patrol_test Log in with correct username and incorrect & empty password
        ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞═════════════════
        The following message was thrown:
        Multiple exceptions (2) were detected during the running of the
        current test, and at least one was unexpected.
        ═════════════════════════════════════════════════════════════════
        
        at pl.leancode.patrol.PatrolJUnitRunner.runDartTest(PatrolJUnitRunner.java:135)
        at no.futurehome.futurehome_app.MainActivityTest.runDartTest(MainActivityTest.java:29)
        java.lang.AssertionError: Dart test failed: sdu_integration_tests.src.smoke_tests.01_log_in_tests.01_correct_username_invalid_password_patrol_test Log in with correct username and incorrect & empty password
        ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞═════════════════
        The following message was thrown:
        Multiple exceptions (2) were detected during the running of the
        current test, and at least one was unexpected.
        ═════════════════════════════════════════════════════════════════
        
        at pl.leancode.patrol.PatrolJUnitRunner.runDartTest(PatrolJUnitRunner.java:135)
        at no.futurehome.futurehome_app.MainActivityTest.runDartTest(MainActivityTest.java:29)
        
        
        FAILURE: Build failed with an exception.
        
        * What went wrong:
        Execution failed for task ':app:connectedDebugAndroidTest'.
        See the report at: file:///Users/amika/Documents/smarthome-app/build/app/reports/androidTests/connected/index.html
        
        * Try:
        > Run with --stacktrace option to get the stack trace.
        > Run with --info or --debug option to get more log output.
        > Run with --scan to get full insights.
        
        * Get more help at https://help.gradle.org
        
        BUILD FAILED in 34s
        : 
        : Test results saved as file:/Users/amika/Documents/smarthome-app/build/app/outputs/androidTest-results/connected/Pixel_3a(AVD)%20-%2014/test-result.pb. Inspect these results in Android Studio by selecting Run > Import Tests From File from the menu bar and importing test-result.pb.
        : 
        : > Task :app:connectedDebugAndroidTest FAILED
        : 
        : Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
        : 
        : You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
        : 
        : See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
        : 543 actionable tasks: 7 executed, 536 up-to-date
✗ Failed to execute tests of apk with entrypoint test_bundle.dart on emulator-5554 (Gradle test execution failed with code 1) (35.0s)
Exception: Gradle test execution failed with code 1
#0      AndroidTestBackend.execute.<anonymous closure> (package:patrol_cli/src/android/android_test_backend.dart:140:9)
<asynchronous suspension>
#1      DisposeScope.run (package:dispose_scope/src/dispose_scope.dart:46:7)
<asynchronous suspension>
#2      AndroidTestBackend.execute (package:patrol_cli/src/android/android_test_backend.dart:106:5)
<asynchronous suspension>
#3      TestCommand._execute (package:patrol_cli/src/commands/test.dart:277:7)
<asynchronous suspension>
#4      TestCommand.run (package:patrol_cli/src/commands/test.dart:169:23)
<asynchronous suspension>
#5      CommandRunner.runCommand (package:args/command_runner.dart:212:13)
<asynchronous suspension>
#6      PatrolCommandRunner.runCommand (package:patrol_cli/src/runner/patrol_command_runner.dart:321:18)
<asynchronous suspension>
#7      PatrolCommandRunner.run (package:patrol_cli/src/runner/patrol_command_runner.dart:265:18)
<asynchronous suspension>
#8      patrolCommandRunner (package:patrol_cli/src/runner/patrol_command_runner.dart:77:20)
<asynchronous suspension>
#9      main (file:///Users/amika/.pub-cache/hosted/pub.dev/patrol_cli-2.2.2/bin/main.dart:6:20)
<asynchronous suspension>

See the logs above to learn what happened. Also consider running with --verbose. If the logs still aren't useful, then it's a bug - please report it.
Uninstalling no.futurehome.futurehome_app from sdk gphone64 arm64
Uninstalling no.futurehome.futurehome_app.test from sdk gphone64 arm64

Patrol version

patrol: ^2.3.2

Patrol Doctor output

Patrol CLI version: 2.2.2
Program adb found in /opt/homebrew/bin/adb
Env var $ANDROID_HOME set to /Users/amika/Library/Android/sdk
Program xcodebuild found in /usr/bin/xcodebuild
Program ideviceinstaller found in /opt/homebrew/bin/ideviceinstaller

Flutter Doctor output

[✓] Flutter (Channel stable, 3.10.5, on macOS 14.1 23B74 darwin-arm64 (Rosetta), locale pl-PL)
    • Flutter version 3.10.5 on channel stable at /Users/amika/fvm/versions/3.10.5
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 796c8ef792 (5 months ago), 2023-06-13 15:51:02 -0700
    • Engine revision 45f6e00911
    • Dart version 3.0.5
    • DevTools version 2.23.1

[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at /Users/amika/Library/Android/sdk
    • Platform android-34, build-tools 34.0.0
    • ANDROID_HOME = /Users/amika/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b829.9-10027231)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 15.0)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 15A240d
    • CocoaPods version 1.13.0

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2022.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b829.9-10027231)

[✓] IntelliJ IDEA Ultimate Edition (version 2023.2.4)
    • IntelliJ at /Applications/IntelliJ IDEA.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] Connected device (4 available)
    • M2007J20CG (mobile)         • 1025aa50      • android-arm64  • Android 12 (API 31)
    • sdk gphone64 arm64 (mobile) • emulator-5554 • android-arm64  • Android 14 (API 34) (emulator)
    • macOS (desktop)             • macos         • darwin-arm64   • macOS 14.1 23B74 darwin-arm64 (Rosetta)
    • Chrome (web)                • chrome        • web-javascript • Google Chrome 119.0.6045.105

[✓] Network resources
    • All expected network resources are available.

The text was updated successfully, but these errors were encountered:

bartekpacia · 2023-11-07T21:26:01Z

Hi 👋🏻 thanks for using Patrol and raising this issue. Sorry for the problems!

First of all – this "multiple exceptions were thrown" problem is tracked in #951. We should definitely do a better job when it comes to reporting such test failures.

Regarding the main problem – it indeed sounds like a regression. But without more info I can't say anything. Your "reproduction steps" aren't really reproduction steps - I can't copy&paste and have a working sample of the bug.

Please let me know if you need any more information or if I can provide additional details to help diagnose and resolve this problem.

The best thing you can do is provide a minimal, reproducible example of this problem. Otherwise there's little chance we'll be able to pinpoint the cause and fix the issue.

mateuszwojtczak · 2023-11-17T08:56:18Z

@Arkadiusz-futurehome please provide a reproducible sample. Closing this issue for now.

github-actions · 2023-11-24T09:02:22Z

This issue has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar problem, please file a new issue. Make sure to follow the template and provide all the information necessary to reproduce the issue.

mateuszwojtczak closed this as completed Nov 17, 2023

github-actions bot locked as resolved and limited conversation to collaborators Nov 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiple unexpected exceptions after updating to Patrol 2.3.2 #1874

Multiple unexpected exceptions after updating to Patrol 2.3.2 #1874

Arkadiusz-futurehome commented Nov 7, 2023

bartekpacia commented Nov 7, 2023 •

edited

Loading

mateuszwojtczak commented Nov 17, 2023

github-actions bot commented Nov 24, 2023

Multiple unexpected exceptions after updating to Patrol 2.3.2 #1874

Multiple unexpected exceptions after updating to Patrol 2.3.2 #1874

Comments

Arkadiusz-futurehome commented Nov 7, 2023

Steps to reproduce

Actual results

Logs

Patrol version

Patrol Doctor output

Flutter Doctor output

bartekpacia commented Nov 7, 2023 • edited Loading

mateuszwojtczak commented Nov 17, 2023

github-actions bot commented Nov 24, 2023

bartekpacia commented Nov 7, 2023 •

edited

Loading