[BUG][2.8.0]Observability 2.8.0.0 failed to install due to jarhell with guava

[peterzhuamazon]

Hi,

% ./opensearch-plugin install file://$PWD/obs.zip
-> Installing file:///home/zhujiaxi/2.8.0/opensearch-2.8.0-SNAPSHOT-0422/bin/obs.zip
-> Downloading file:///home/zhujiaxi/2.8.0/opensearch-2.8.0-SNAPSHOT-0422/bin/obs.zip
[=================================================] 100%
-> Failed installing file:///home/zhujiaxi/2.8.0/opensearch-2.8.0-SNAPSHOT-0422/bin/obs.zip
-> Rolling back file:///home/zhujiaxi/2.8.0/opensearch-2.8.0-SNAPSHOT-0422/bin/obs.zip
-> Rolled back file:///home/zhujiaxi/2.8.0/opensearch-2.8.0-SNAPSHOT-0422/bin/obs.zip
Exception in thread "main" java.lang.IllegalStateException: failed to load plugin opensearch-observability due to jar hell
	at org.opensearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:681)
	at org.opensearch.plugins.InstallPluginCommand.jarHellCheck(InstallPluginCommand.java:862)
	at org.opensearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:830)
	at org.opensearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:875)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:275)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:249)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.Command.main(Command.java:101)
	at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)
Caused by: java.lang.IllegalStateException: jar hell!
class: com.google.common.annotations.Beta
jar1: /local/home/zhujiaxi/2.8.0/opensearch-2.8.0-SNAPSHOT-0422/plugins/.installing-16749857429110592339/guava-31.0.1-jre.jar
jar2: /local/home/zhujiaxi/2.8.0/opensearch-2.8.0-SNAPSHOT-0422/plugins/opensearch-job-scheduler/guava-31.0.1-jre.jar
	at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316)
	at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215)
	at org.opensearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:667)
	... 11 more

Seems like caused by guava dep here:
https://github.com/opensearch-project/observability/blob/2.x/build.gradle#L197

This dep has been removed in 3.0.0/main it seems.

This is not caught in observability ci because the ci only run gradle build and does not test the plugin installation.
https://github.com/opensearch-project/observability/blob/2.x/.github/workflows/opensearch-observability-test-and-build-workflow.yml#L30

Suggest adding these into your ci:
https://github.com/opensearch-project/ml-commons/blob/main/.github/workflows/CI-workflow.yml#L52-L60C52

Thanks.

[joshuali925]

./gradlew build will run integTest, which installs job-scheduler and assembled observability artifacts to opensearch.

❯ ./gradlew build
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.6.1
  OS Info               : Linux 5.4.241-160.348.amzn2int.x86_64 (amd64)
  JDK Version           : 17 (Eclipse Temurin JDK)
  JAVA_HOME             : /local/home/user/.asdf/installs/java/adoptopenjdk-17.0.7+7
  Random Testing Seed   : 89E8BF48F2683F10
  In FIPS 140 mode      : false
=======================================

> Task :compileTestKotlin
w: /local/home/user/projects/os-2.0/observability/src/test/kotlin/org/opensearch/observability/bwc/TABackwardCompatibilityIT.kt: (19, 8): This class shouldn't be used in Kotlin. Use kotlin.collections.List or kotlin.collections.MutableList instead.
w: /local/home/user/projects/os-2.0/observability/src/test/kotlin/org/opensearch/observability/bwc/TABackwardCompatibilityIT.kt: (20, 8): This class shouldn't be used in Kotlin. Use kotlin.collections.Map or kotlin.collections.MutableMap instead.
w: /local/home/user/projects/os-2.0/observability/src/test/kotlin/org/opensearch/observability/bwc/TABackwardCompatibilityIT.kt: (70, 50): Unchecked cast: Any? to Map<String, Map<String, Any>>
w: /local/home/user/projects/os-2.0/observability/src/test/kotlin/org/opensearch/observability/bwc/TABackwardCompatibilityIT.kt: (70, 53): This class shouldn't be used in Kotlin. Use kotlin.collections.Map or kotlin.collections.MutableMap instead.
w: /local/home/user/projects/os-2.0/observability/src/test/kotlin/org/opensearch/observability/bwc/TABackwardCompatibilityIT.kt: (70, 65): This class shouldn't be used in Kotlin. Use kotlin.collections.Map or kotlin.collections.MutableMap instead.
w: /local/home/user/projects/os-2.0/observability/src/test/kotlin/org/opensearch/observability/bwc/TABackwardCompatibilityIT.kt: (72, 47): Unchecked cast: Any! to List<Map<String, Any>>
w: /local/home/user/projects/os-2.0/observability/src/test/kotlin/org/opensearch/observability/bwc/TABackwardCompatibilityIT.kt: (72, 50): This class shouldn't be used in Kotlin. Use kotlin.collections.List or kotlin.collections.MutableList instead.
w: /local/home/user/projects/os-2.0/observability/src/test/kotlin/org/opensearch/observability/bwc/TABackwardCompatibilityIT.kt: (72, 55): This class shouldn't be used in Kotlin. Use kotlin.collections.Map or kotlin.collections.MutableMap instead.
w: /local/home/user/projects/os-2.0/observability/src/test/kotlin/org/opensearch/observability/bwc/TABackwardCompatibilityIT.kt: (88, 13): Unreachable code

> Task :integTest FAILED
Exec output and error:
| Output for ./bin/opensearch-plugin:warning: no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release
| -> Installing file:/local/home/user/.gradle/caches/modules-2/files-2.1/org.opensearch.plugin/opensearch-job-scheduler/2.8.0.0-SNAPSHOT/e9cdb3fc8a76b15f432ec1bd8bf1cf36b97bfdb9/opensearch-job-scheduler-2.8.0.0-SNAPSHOT.zip
| -> Downloading file:/local/home/user/.gradle/caches/modules-2/files-2.1/org.opensearch.plugin/opensearch-job-scheduler/2.8.0.0-SNAPSHOT/e9cdb3fc8a76b15f432ec1bd8bf1cf36b97bfdb9/opensearch-job-scheduler-2.8.0.0-SNAPSHOT.zip
| -> Installed opensearch-job-scheduler with folder name opensearch-job-scheduler
| -> Installing file:/local/home/user/projects/os-2.0/observability/build/distributions/opensearch-observability-2.8.0.0-SNAPSHOT.zip
| -> Downloading file:/local/home/user/projects/os-2.0/observability/build/distributions/opensearch-observability-2.8.0.0-SNAPSHOT.zip
| -> Failed installing file:/local/home/user/projects/os-2.0/observability/build/distributions/opensearch-observability-2.8.0.0-SNAPSHOT.zip
| -> Rolling back opensearch-job-scheduler
| -> Rolled back opensearch-job-scheduler
| -> Rolling back file:/local/home/user/projects/os-2.0/observability/build/distributions/opensearch-observability-2.8.0.0-SNAPSHOT.zip
| -> Rolled back file:/local/home/user/projects/os-2.0/observability/build/distributions/opensearch-observability-2.8.0.0-SNAPSHOT.zip
| Exception in thread "main" java.lang.IllegalStateException: failed to load plugin opensearch-observability due to jar hell
|       at org.opensearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:681)
|       at org.opensearch.plugins.InstallPluginCommand.jarHellCheck(InstallPluginCommand.java:862)
|       at org.opensearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:830)
|       at org.opensearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:875)
|       at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:276)
|       at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:250)
|       at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
|       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
|       at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
|       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
|       at org.opensearch.cli.Command.main(Command.java:101)
|       at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)
| Caused by: java.lang.IllegalStateException: jar hell!
| class: com.google.common.annotations.Beta
| jar1: /local/home/user/projects/os-2.0/observability/build/testclusters/integTest-0/distro/2.8.0-INTEG_TEST/plugins/opensearch-job-scheduler/guava-31.0.1-jre.jar
| jar2: /local/home/user/projects/os-2.0/observability/build/testclusters/integTest-0/distro/2.8.0-INTEG_TEST/plugins/.installing-11928477191048430528/guava-31.0.1-jre.jar
|       at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316)
|       at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215)
|       at org.opensearch.plugins.PluginsService.checkBundleJarHell(PluginsService.java:667)
|       ... 11 more

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':integTest'.
> Process 'command './bin/opensearch-plugin'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.6.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 24s
23 actionable tasks: 14 executed, 9 up-to-date

I'm able to reproduce the issue with ./gradlew build. If CI was passing before, maybe it happened before opensearch introduced conflicting jars?

[joshuali925]

CI couldn't catch this because it was passed a month ago #1505
The breaking dependency was introduced yesterday opensearch-project/OpenSearch#7568

[peterzhuamazon]

Hi @joshuali925 I synced this up with @derek-ho last week.
It seems like a missing backport.

Thanks.

[peterzhuamazon]

Seems resolved, closing.