[POC] [Security Manager Replacement] GraalVM sandboxing

[reta]

Description

Use GraalVM capability to spin off a separate JVM to host the sandboxed component. With this model, it becomes possible to:

• run OpenSearch core on any JVM that GraalVM supports (without SM)
• use older JDK versions (up to 23) with SM enabled for non-trusted components

The POC does a bare minimum work to host the ShiroIdentityPlugin in the separated JVM (21.0.5+11-Ubuntu-1ubuntu124.10) that is running under SecurityManager:

OpenSearch home: /home/opensearch-3.0.0-SNAPSHOT
Host JVM version: 22.0.2+9-jvmci-b01
Polyglot JVM version: 21.0.5+11-Ubuntu-1ubuntu124.10
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/home/opensearch-3.0.0-SNAPSHOT/lib/opensearch-3.0.0-SNAPSHOT.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
Security Manager? org.opensearch.secure_sm.SecureSM@7b303608
[2024-12-16T15:00:18,828][WARN ][stderr                   ] [] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[2024-12-16T15:00:18,864][WARN ][stderr                   ] [] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2024-12-16T15:00:18,875][WARN ][stderr                   ] [] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Shiro Plugin? org.opensearch.identity.shiro.ShiroIdentityPlugin@5ddd3150
Exception in thread "main" access denied ("java.net.SocketPermission" "localhost:0" "listen,resolve")
	at <java> checkPermission(Ljava/security/Permission;)V(Unknown)
	at <java> checkPermission(Ljava/security/Permission;)V(java/security/AccessController.java:1071:0)
	at <java> checkPermission(Ljava/security/Permission;)V(java/lang/SecurityManager.java:411:0)
	at <java> checkListen(I)V(java/lang/SecurityManager.java:985:0)
	at <java> bind(Ljava/net/SocketAddress;I)V(java/net/ServerSocket.java:387:0)
	at <java> <init>(IILjava/net/InetAddress;)V(java/net/ServerSocket.java:278:0)
	at <java> <init>(I)V(java/net/ServerSocket.java:171:0)
	at <java> getSocket()Ljava/net/ServerSocket;(org/opensearch/identity/shiro/ShiroIdentityPlugin.java:144:0)
	at <java> RootNode for interop message: 'invokeMember'.(Unknown)
	at org.graalvm.polyglot.Value.invokeMember(Value.java:1021)
	at org.opensearch.espresso.sandbox.Sandbox.loadPlugin(Sandbox.java:86)
	at org.opensearch.espresso.sandbox.Sandbox.main(Sandbox.java:22)

With host to guest communication:

OpenSearch home: /home/andriy.redko/Development/opensearch-3.0.0-SNAPSHOT
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::objectFieldOffset has been called by WARNING: Please consider reporting this to the maintainers of class com.oracle.truffle.runtime.hotspot.HotSpotTruffleRuntime
WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release
Host JVM version: 24+27-jvmci-b01
Polyglot JVM version: 21.0.5+11-Ubuntu-1ubuntu124.10
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
Security Manager? org.opensearch.secure_sm.SecureSM@5bf86398
[2025-01-09T14:55:23,722][WARN ][stderr                   ] [] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[2025-01-09T14:55:23,798][WARN ][stderr                   ] [] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2025-01-09T14:55:23,821][WARN ][stderr                   ] [] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Shiro Plugin? org.opensearch.identity.shiro.ShiroIdentityPlugin@59c530b5
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
[2025-01-09T14:55:24,619][INFO ][stdout                   ] [] Client? org.opensearch.client.node.NodeClient@35c66ad
[2025-01-09T14:55:24,652][INFO ][stdout                   ] [] AdminClient? org.opensearch.client.support.AbstractClient$Admin@250988e5
[2025-01-09T14:55:26,498][INFO ][stdout                   ] [] Cluster? ClusterStateResponse{clusterState=cluster uuid: _na_ [committed: false]
[2025-01-09T14:55:26,502][INFO ][stdout                   ] [] version: 0
[2025-01-09T14:55:26,505][INFO ][stdout                   ] [] state uuid: d61W3KGJSs-NrZUPHY5eqA
[2025-01-09T14:55:26,508][INFO ][stdout                   ] [] from_diff: false
[2025-01-09T14:55:26,511][INFO ][stdout                   ] [] meta data version: 0
[2025-01-09T14:55:26,514][INFO ][stdout                   ] []    coordination_metadata:
[2025-01-09T14:55:26,517][INFO ][stdout                   ] []       term: 0
[2025-01-09T14:55:26,521][INFO ][stdout                   ] []       last_committed_config: VotingConfiguration{}
[2025-01-09T14:55:26,524][INFO ][stdout                   ] []       last_accepted_config: VotingConfiguration{}
[2025-01-09T14:55:26,527][INFO ][stdout                   ] []       voting tombstones: []
[2025-01-09T14:55:26,531][INFO ][stdout                   ] [] metadata customs:
[2025-01-09T14:55:26,534][INFO ][stdout                   ] []    index-graveyard: IndexGraveyard[[]]
[2025-01-09T14:55:26,536][INFO ][stdout                   ] [] nodes: 
[2025-01-09T14:55:26,538][INFO ][stdout                   ] [] routing_table (version 0):
[2025-01-09T14:55:26,540][INFO ][stdout                   ] [] routing_nodes:
[2025-01-09T14:55:26,542][INFO ][stdout                   ] [] ---- unassigned
[2025-01-09T14:55:26,543][INFO ][stdout                   ] [] }
Exception in thread "main" access denied ("java.net.SocketPermission" "localhost:0" "listen,resolve")
	at <java> java.security.AccessControlContext.checkPermission(java/security/AccessControlContext.java:488:0)
	at <java> java.security.AccessController.checkPermission(java/security/AccessController.java:1071:0)
	at <java> java.lang.SecurityManager.checkPermission(java/lang/SecurityManager.java:411:0)
	at <java> java.lang.SecurityManager.checkListen(java/lang/SecurityManager.java:985:0)
	at <java> java.net.ServerSocket.bind(java/net/ServerSocket.java:387:0)
	at <java> java.net.ServerSocket.<init>(java/net/ServerSocket.java:278:0)
	at <java> java.net.ServerSocket.<init>(java/net/ServerSocket.java:171:0)
	at <java> org.opensearch.identity.shiro.ShiroIdentityPlugin.getSocket(org/opensearch/identity/shiro/ShiroIdentityPlugin.java:154:0)
	at <java> RootNode for interop message: 'invokeMember'.(Unknown)
	at org.graalvm.polyglot.Value.invokeMember(Value.java:1159)
	at com.opensearch.graalvm.polyglot.PolyglotRunner.loadPlugin(PolyglotRunner.java:106)
	at com.opensearch.graalvm.polyglot.PolyglotRunner.main(PolyglotRunner.java:37)

Related Issues

Closes #16861

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for a8e52f3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 6201d8c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[kumargu]

Thank-you again for putting this up.

Apart from the debugging pain, which i hope would be one-time while setting up a plugin); I don't see a reason why we would not include this as an alternative for SM. Let's see what others feel about it.

This would look much better, allowing plugins to move to JDK-24 with a real look-and-feel of a plugin sandbox env when Graal fully addresses oracle/graal#10239

[reta]

Apart from the debugging pain, which i hope would be one-time while setting up a plugin); I don't see a reason why we would not include this as an alternative for SM. Let's see what others feel about it.

I have included it here #16861 but to reiterate, the most difficult issue with such a model is communication between host and spawned JVM/context: it seems not being possible to wire up the services from the host JVM. I will spend more time to explore the limitations here.

[github-actions]

❌ Gradle check result for ed8cd55: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[kumargu]

Apart from the debugging pain, which i hope would be one-time while setting up a plugin); I don't see a reason why we would not include this as an alternative for SM. Let's see what others feel about it.

I have included it here #16861 but to reiterate, the most difficult issue with such a model is communication between host and spawned JVM/context: it seems not being possible to wire up the services from the host JVM. I will spend more time to explore the limitations here.

Asking to keep myself upto-date -- I guess you have figured it out usingjava.PolyglotInterfaceMappings?. Just that its broken in 24.1.1?

[reta]

Asking to keep myself upto-date -- I guess you have figured it out usingjava.PolyglotInterfaceMappings?. Just that its broken in 24.1.1?

Yes, sadly it is. And AFAIK, the earliest release with the fix is in March (as per Slack thread response), this is a blocker for us at the moment since we cannot support host <-> guest exchange :(

[kumargu]

(posting for visibility)
we requested GraalVM to-try-make the fix available via a patch in their upcoming Jan-25 release.

[github-actions]

❌ Gradle check result for c3e5143: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?