Arrow Flight Server bootstrap logic #16962

rishabhmaurya · 2025-01-06T20:21:38Z

Description

This PR adds server bootstrap logic for Arrow Flight. This is in continuation to #16691

Flight server bootstrap logic and integration with Auxiliary Transport.
new plugin for StreamManager implementation behind feature flag with default behaviour as disabled.
support for SslContext in Flight server and client.
update in SecureTransportSettingsProvider to build SslContext outside of security plugin.
ClientManager for creating a pool of flight clients for data nodes only.
custom event loop group and thread pool for server and client channel.
Default netty server and allocator configuration defined in ServerConfig.
2 new gradle projects - arrow-memory-shaded and flight-core-shaded to share dependencies to avoid polluting server classpath.
Flight info API to get the publish and bound address for each of the node.
Integration test for FlightServer client handshake.
~~StreamManager wrapper for integration with task API.~~ moved to next PR (Base Flight producer integration with Flight Server #17066)
~~integration of StreamManager with server module.~~ moved to next PR (Base Flight producer integration with Flight Server #17066)

TODOs:

Security plugin changes to expose parameters to build SslContext using SecureTransportSettingsProvider [FEATURE] Support additional params in SecureTransportSettingsProvider to enable building of SslContext outside of security plugin security#5011
Handle certificate hot reload [Feature Request] Handle certificate hot reload for Flight server #16965
Tuning up default values for configurations defined in ServerConfig - [Feature Request] Tune up server config for Flight server #16966

I will handle both of them as part of follow up PRs.

Related Issues

Resolves #16963

#16679

Check List

Functionality includes testing.
~~- [ ] API changes companion pull request created, if applicable.~~
~~- [ ] Public documentation issue/PR created, if applicable.~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-01-06T20:27:28Z

❌ Gradle check result for 53ad56f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-01-06T20:40:34Z

❌ Gradle check result for f360c80: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-01-06T21:05:59Z

❌ Gradle check result for b38f302: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

reta · 2025-01-06T22:02:40Z

@rishabhmaurya I know that this is work in progress but please take a look at [1] where the notion of auxiliary transports (and plugins implementing ones, like gRPC), has been introduced. We may (and very likely - should) consider Apache Arrow Flight as one of such transports, it should be optional plugin and not part of the core server. Thank you

[1] #16534

rishabhmaurya · 2025-01-06T22:18:06Z

@reta thanks for the pointer, I will update the PR and try to use aux transport.

github-actions · 2025-01-06T23:49:55Z

❌ Gradle check result for 60f82e9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-01-07T00:46:52Z

❌ Gradle check result for f9623e9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-01-07T01:11:48Z

❌ Gradle check result for 93df0aa: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-01-07T01:30:31Z

❌ Gradle check result for 63bc0f4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-01-07T02:15:30Z

❌ Gradle check result for d364c1c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

rishabhmaurya · 2025-01-07T19:15:17Z

@reta do you know how can i address the gradle check failure related to ApiAnnotationProcessorTests ? I know i have messed something up but unable to troubleshoot.

java.lang.AssertionError: 
Expected: an instance of org.opensearch.common.annotation.processor.CompilerSupport$Failure
     but: <org.opensearch.common.annotation.processor.CompilerSupport$Success@1b7367d4> is a org.opensearch.common.annotation.processor.CompilerSupport$Success
	at __randomizedtesting.SeedInfo.seed([188B34A9BBC5FD49:D29516146CC3C7B0]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:964)
	at org.junit.Assert.assertThat(Assert.java:930)
	at org.opensearch.common.annotation.processor.ApiAnnotationProcessorTests.testPublicApiMethodReturnNotAnnotatedWildcardGenerics(ApiAnnotationProcessorTests.java:310)

ref: https://build.ci.opensearch.org/job/gradle-check/51929/testReport/junit/org.opensearch.common.annotation.processor/ApiAnnotationProcessorTests/testPublicApiMethodReturnNotAnnotatedWildcardGenerics/

reta · 2025-01-07T19:51:09Z

@reta do you know how can i address the gradle check failure related to ApiAnnotationProcessorTests ? I know i have messed something up but unable to troubleshoot.

@rishabhmaurya It is caused by https://github.com/opensearch-project/OpenSearch/pull/16962/files#diff-756d8dcb1dfd9f8fa9bf974880af0b001cef12541eb17857ff30d4aa922bc05eR200 (change in server/src/main/resources/org/opensearch/bootstrap/security.policy

rishabhmaurya · 2025-01-07T19:55:59Z

@reta ah, thanks. How did you find that out as I couldn't find any error message in the logs

reta · 2025-01-07T20:01:07Z

@reta ah, thanks. How did you find that out as I couldn't find any error message in the logs

@rishabhmaurya needed some debugging, afair some tests were required some more security permissions but those could be ignored (since tests are focused on diagnostics)

reta · 2025-01-28T19:06:56Z

libs/arrow-spi/build.gradle

-  api "com.fasterxml.jackson.core:jackson-annotations:${versions.jackson}"
-
-  implementation "commons-codec:commons-codec:${versions.commonscodec}"
+  api project(path:':libs:opensearch-arrow-memory-shaded', configuration: 'shadow')


So these are the only dependencies that SPI layer needs:

dependencies { api project(':libs:opensearch-core') api "org.apache.arrow:arrow-memory-core:${versions.arrow}" api "org.apache.arrow:arrow-vector:${versions.arrow}" }

Right. But at runtime, there are other transitive dependencies needed like netty-buffer, jackson-annotations etc which are part of arrow-memory-netty-shaded as shaded dependencies.
So if we have to use vector in server module, then these runtime dependencies should be present on its runtime classpath.

So if we have to use vector in server module, then these runtime dependencies should be present on its runtime classpath.

We will not use vector in server module, the server has to be sufficient with SPI layer at the moment (since everything else is in plugin)

reta · 2025-01-28T19:12:08Z

plugins/arrow-flight-rpc/build.gradle

+
+dependencies {
+  implementation project(':libs:opensearch-arrow-spi')
+  implementation project(path:':libs:opensearch-flight-core-shaded', configuration: 'shadow')


The plugin could bundle all these dependencies directly, right? There is no need to shade as far as we don't have netty / jackson JarHell (that we shouldn't)

plugin has a different classloader than server, so even if plugin adds them to their classloader, it won't work for server at runtime.

* new plugin for StreamManager implementation * integration with server module * support for SslContext in Flight server and client * ClientManager for creating a pool of flight clients for data nodes * custom event loop group and thread pool for server and client channel Signed-off-by: Rishabh Maurya <[email protected]>

Signed-off-by: Rishabh Maurya <[email protected]>

…on was causing client close failure Signed-off-by: Rishabh Maurya <[email protected]>

Signed-off-by: Rishabh Maurya <[email protected]>

github-actions · 2025-01-28T21:44:30Z

❌ Gradle check result for 126f8f0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Rishabh Maurya <[email protected]>

github-actions · 2025-01-28T21:52:31Z

❌ Gradle check result for afea399: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions bot added enhancement Enhancement or improvement to existing feature or request Search:Performance labels Jan 6, 2025

rishabhmaurya self-assigned this Jan 6, 2025

rishabhmaurya force-pushed the stream-bootstrap branch from 53ad56f to f360c80 Compare January 6, 2025 20:29

rishabhmaurya force-pushed the stream-bootstrap branch 2 times, most recently from 83b63d3 to b38f302 Compare January 6, 2025 20:52

This was referenced Jan 6, 2025

[Feature Request] Handle certificate hot reload for Flight server #16965

Open

[Feature Request] Tune up server config for Flight server #16966

Open

rishabhmaurya mentioned this pull request Jan 6, 2025

[FEATURE] Support additional params in SecureTransportSettingsProvider to enable building of SslContext outside of security plugin opensearch-project/security#5011

Open

rishabhmaurya force-pushed the stream-bootstrap branch 2 times, most recently from 4cae87e to 60f82e9 Compare January 6, 2025 23:42

rishabhmaurya force-pushed the stream-bootstrap branch from 60f82e9 to f9623e9 Compare January 7, 2025 00:41

rishabhmaurya force-pushed the stream-bootstrap branch 2 times, most recently from 4601334 to 93df0aa Compare January 7, 2025 01:03

rishabhmaurya force-pushed the stream-bootstrap branch from 93df0aa to 63bc0f4 Compare January 7, 2025 01:18

rishabhmaurya force-pushed the stream-bootstrap branch from 63bc0f4 to d364c1c Compare January 7, 2025 02:04

rishabhmaurya removed the v2.19.0 Issues and PRs related to version 2.19.0 label Jan 28, 2025

reta reviewed Jan 28, 2025

View reviewed changes

rishabhmaurya added 21 commits January 28, 2025 12:29

interim changes - integration with Auxiliary Transport

e2803cf

Signed-off-by: Rishabh Maurya <[email protected]>

changes to use grpc-netty-shaded

3f89e7b

Signed-off-by: Rishabh Maurya <[email protected]>

Update javadoc

267fd1e

Signed-off-by: Rishabh Maurya <[email protected]>

fix the shaded dependencies

f7fca07

Signed-off-by: Rishabh Maurya <[email protected]>

Move arrow-flight-rpc from module to plugin

4a71105

Signed-off-by: Rishabh Maurya <[email protected]>

remove unnecessary imports

b9f37db

Signed-off-by: Rishabh Maurya <[email protected]>

rebase fixes

c35d955

Signed-off-by: Rishabh Maurya <[email protected]>

Fix permissions and other runtime issues

67f7176

Signed-off-by: Rishabh Maurya <[email protected]>

Remove StreamManagerWrapper and Node.java changes from PR

0a271b6

Signed-off-by: Rishabh Maurya <[email protected]>

Fix permissions for test

a04a2cc

Signed-off-by: Rishabh Maurya <[email protected]>

remove testGetFlightClientLocationExecutionError as thread interrupti…

7ae5215

…on was causing client close failure Signed-off-by: Rishabh Maurya <[email protected]>

Fix the issue with single node ClientManager

71a4185

Signed-off-by: Rishabh Maurya <[email protected]>

Fix flight server integ test on unix machine

22784c3

Signed-off-by: Rishabh Maurya <[email protected]>

suppress JSM removal warning

e467a52

Signed-off-by: Rishabh Maurya <[email protected]>

Fix security policy and FlightClientManagerTests

bbfae70

Signed-off-by: Rishabh Maurya <[email protected]>

remove StreamManagerWrapper from the PR

7b50831

Signed-off-by: Rishabh Maurya <[email protected]>

Set multi-release in manifest while shadowing arrow-memory-shaded

28a402b

Signed-off-by: Rishabh Maurya <[email protected]>

Disable jacocoReport for shaded projects

5a06b31

Signed-off-by: Rishabh Maurya <[email protected]>

Remove multi version classes from arrow-memory-shaded

0b16e48

Signed-off-by: Rishabh Maurya <[email protected]>

Address the PR comments

f70e749

Signed-off-by: Rishabh Maurya <[email protected]>

rishabhmaurya force-pushed the stream-bootstrap branch from b22ff8f to f70e749 Compare January 28, 2025 21:31

rishabhmaurya requested a review from peternied as a code owner January 28, 2025 21:31

Move the arrow-memory-shaded and flight within flight-rpc plugin

126f8f0

Signed-off-by: Rishabh Maurya <[email protected]>

Move the arrow-memory-shaded and flight within flight-rpc plugin

afea399

Signed-off-by: Rishabh Maurya <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Arrow Flight Server bootstrap logic #16962

Arrow Flight Server bootstrap logic #16962

rishabhmaurya commented Jan 6, 2025 •

edited

Loading

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

reta commented Jan 6, 2025 •

edited

Loading

rishabhmaurya commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 7, 2025

github-actions bot commented Jan 7, 2025

github-actions bot commented Jan 7, 2025

github-actions bot commented Jan 7, 2025

rishabhmaurya commented Jan 7, 2025 •

edited

Loading

reta commented Jan 7, 2025 •

edited

Loading

rishabhmaurya commented Jan 7, 2025

reta commented Jan 7, 2025

reta Jan 28, 2025 •

edited

Loading

rishabhmaurya Jan 28, 2025 •

edited

Loading

reta Jan 28, 2025

reta Jan 28, 2025

rishabhmaurya Jan 28, 2025

github-actions bot commented Jan 28, 2025

github-actions bot commented Jan 28, 2025

Arrow Flight Server bootstrap logic #16962

Are you sure you want to change the base?

Arrow Flight Server bootstrap logic #16962

Conversation

rishabhmaurya commented Jan 6, 2025 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

reta commented Jan 6, 2025 • edited Loading

rishabhmaurya commented Jan 6, 2025

github-actions bot commented Jan 6, 2025

github-actions bot commented Jan 7, 2025

github-actions bot commented Jan 7, 2025

github-actions bot commented Jan 7, 2025

github-actions bot commented Jan 7, 2025

rishabhmaurya commented Jan 7, 2025 • edited Loading

reta commented Jan 7, 2025 • edited Loading

rishabhmaurya commented Jan 7, 2025

reta commented Jan 7, 2025

reta Jan 28, 2025 • edited Loading

Choose a reason for hiding this comment

rishabhmaurya Jan 28, 2025 • edited Loading

Choose a reason for hiding this comment

reta Jan 28, 2025

Choose a reason for hiding this comment

reta Jan 28, 2025

Choose a reason for hiding this comment

rishabhmaurya Jan 28, 2025

Choose a reason for hiding this comment

github-actions bot commented Jan 28, 2025

github-actions bot commented Jan 28, 2025

rishabhmaurya commented Jan 6, 2025 •

edited

Loading

reta commented Jan 6, 2025 •

edited

Loading

rishabhmaurya commented Jan 7, 2025 •

edited

Loading

reta commented Jan 7, 2025 •

edited

Loading

reta Jan 28, 2025 •

edited

Loading

rishabhmaurya Jan 28, 2025 •

edited

Loading