Skip to content

Allow external override of the sandbox native library path - #21592

Merged
sandeshkr419 merged 4 commits into
opensearch-project:mainfrom
bowenlan-amzn:experiment/native-lib-override
May 11, 2026
Merged

Allow external override of the sandbox native library path#21592
sandeshkr419 merged 4 commits into
opensearch-project:mainfrom
bowenlan-amzn:experiment/native-lib-override

Conversation

@bowenlan-amzn

@bowenlan-amzn bowenlan-amzn commented May 10, 2026

Copy link
Copy Markdown
Member

Description

Add OPENSEARCH_NATIVE_LIB env var (or -PnativeLibOverride Gradle project property) so sandbox tests can reuse a prebuilt libopensearch_native.{dylib,so,dll} from another worktree, a blessed shared copy, or a CI-provided binary.

When set:

  • buildRustLibrary is skipped via onlyIf
  • ext.nativeLibPath resolves to the override path

All 14 sandbox/plugins build.gradle sites already read project(':sandbox:libs:dataformat-native').ext.nativeLibPath, so the override propagates to every consumer without per-plugin changes. ./gradlew run is unaffected—it already accepts -Dnative.lib.path via tests.jvm.argline and does not depend on buildRustLibrary.

Without the env var / property, behavior is unchanged: buildRustLibrary runs (or is UP-TO-DATE) and rust/target/release/libopensearch_native.dylib is used.

Usage

# Skip the Rust build entirely, use a prebuilt .dylib
OPENSEARCH_NATIVE_LIB=~/.cache/opensearch-sandbox/native/current.dylib \
    ./gradlew check -p sandbox -Dsandbox.enabled=true

# Or via Gradle property:
./gradlew check -p sandbox -Dsandbox.enabled=true \
    -PnativeLibOverride=~/.cache/opensearch-sandbox/native/current.dylib
Example: bless a shared .dylib from one worktree for all others
#!/usr/bin/env bash
set -euo pipefail

# Build the native lib on your "main" worktree, then copy it to a shared
# location so other worktrees can skip the 20-min Rust rebuild.

WORKTREE="${1:-$HOME/code/opensearch/OpenSearch}"
DEST_DIR="$HOME/.cache/opensearch-sandbox/native"

cd "$WORKTREE"
./gradlew :sandbox:libs:dataformat-native:buildRustLibrary -Dsandbox.enabled=true

mkdir -p "$DEST_DIR"
cp sandbox/libs/dataformat-native/rust/target/release/libopensearch_native.dylib \
   "$DEST_DIR/current.dylib"

echo "Blessed from $(git rev-parse --short HEAD) at $(date -u +%Y-%m-%dT%H:%M:%SZ)"

Then in any other worktree:

OPENSEARCH_NATIVE_LIB=~/.cache/opensearch-sandbox/native/current.dylib \
    ./gradlew check -p sandbox -Dsandbox.enabled=true

Check List

  • New functionality includes testing
  • New functionality has been documented
  • API changes companion pull request - N/A
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certification of Origin and signing off your commits, please check here.

@github-actions

Copy link
Copy Markdown
Contributor

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit 449a89f.

PathLineSeverityDescription
sandbox/libs/dataformat-native/build.gradle94mediumIntroduces an unverified native library injection point via the OPENSEARCH_NATIVE_LIB environment variable or -PnativeLibOverride Gradle property. When set, the trusted Rust build step is bypassed entirely and the provided binary is loaded directly — with no hash, signature, or path-prefix check. Because tests already run with --enable-native-access=ALL-UNNAMED, a malicious actor who can influence CI/CD environment variables (e.g., through a compromised workflow or a poisoned secrets store) could substitute an arbitrary native binary that executes unrestricted native code. The stated purpose (sharing prebuilt artifacts across worktrees) is plausible, but the lack of any integrity verification makes this a meaningful supply-chain injection risk in automated pipelines.

The table above displays the top 10 most important findings.

Total: 1 | Critical: 0 | High: 0 | Medium: 1 | Low: 0


Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.


⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.

@bowenlan-amzn bowenlan-amzn added the skip-diff-analyzer Maintainer to skip code-diff-analyzer check, after reviewing issues in AI analysis. label May 10, 2026
@bowenlan-amzn
bowenlan-amzn force-pushed the experiment/native-lib-override branch from 449a89f to 2938bf7 Compare May 10, 2026 19:25
@github-actions

Copy link
Copy Markdown
Contributor

❌ Gradle check result for 2938bf7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Add OPENSEARCH_NATIVE_LIB env var (or -PnativeLibOverride project property) so
sandbox tests can reuse a prebuilt libopensearch_native.{dylib,so,dll} from
another worktree, a blessed shared copy, or a CI-provided binary. When set,
buildRustLibrary is skipped via onlyIf and ext.nativeLibPath resolves to the
override path.

All 14 sandbox/plugins build.gradle sites already read
project(':sandbox:libs:dataformat-native').ext.nativeLibPath, so the override
propagates to every consumer without per-plugin changes. ./gradlew run is
unaffected — it already accepts -Dnative.lib.path via tests.jvm.argline and
does not depend on buildRustLibrary.

Without the env var / property, behavior is unchanged: buildRustLibrary runs
(or is UP-TO-DATE) and rust/target/release/libopensearch_native.dylib is used.

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>
@bowenlan-amzn
bowenlan-amzn force-pushed the experiment/native-lib-override branch from 2938bf7 to ad17ed9 Compare May 10, 2026 22:59
@bowenlan-amzn
bowenlan-amzn marked this pull request as ready for review May 10, 2026 22:59
@bowenlan-amzn
bowenlan-amzn requested a review from a team as a code owner May 10, 2026 22:59
@github-actions

Copy link
Copy Markdown
Contributor

❕ Gradle check result for ad17ed9: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@codecov

codecov Bot commented May 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.52%. Comparing base (36809cc) to head (ad17ed9).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main   #21592      +/-   ##
============================================
+ Coverage     73.50%   73.52%   +0.02%     
+ Complexity    74644    74623      -21     
============================================
  Files          5980     5980              
  Lines        338777   338777              
  Branches      48848    48848              
============================================
+ Hits         249011   249081      +70     
+ Misses        69946    69795     -151     
- Partials      19820    19901      +81     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

Copy link
Copy Markdown
Contributor

❌ Gradle check result for 6462775: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions

Copy link
Copy Markdown
Contributor

❌ Gradle check result for b3dc2cb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions

Copy link
Copy Markdown
Contributor

❌ Gradle check result for 6f5b1a0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@sandeshkr419
sandeshkr419 merged commit a02606f into opensearch-project:main May 11, 2026
32 of 48 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-diff-analyzer Maintainer to skip code-diff-analyzer check, after reviewing issues in AI analysis.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants