Allow plugins to copy folders into their config dir during installation

[cwperks]

Description

This PR relaxes the plugin installation CLI to allow plugins to copy directories to their config folder. Currently, plugins can only copy files from src/main/config to their config folder underneath config/{plugin-name} after running the plugin installation script.

The main motivation for this is the security-analytics plugin which bundles yaml files corresponding to sigma rules which they currently bundle as classpath resources in the jar.

The security analytics plugin bundles these so that they are readable at runtime. Since files within a plugin's config folder are also available at runtime, its desirable to move these out of the jar and into the config folder.

Without relaxing the plugin installation script, its not possible to keep the folder structure that the security-analytics repo already has. I propose relaxing the plugin installation script to allow folders in the plugin's config directory.

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 7f65b0c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 0f8d0c4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 5f45ed2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[cwperks]

@andrross any concerns with relaxing this restriction? FYI the reason I opened this PR is to facilitate the change in opensearch-project/security-analytics#1580.

I'm not very familiar with the SAP (security-analytics plugin) repo, but its being flagged by anti-virus software scanners bc of all the yaml files included in the jar. That PR moves them to the config folder as configuration files. That being said, there are 2k of them so idk why SAP (security-analytics plugin) doesn't get them from some registry somewhere and not have them checked into the repo.

[github-actions]

❌ Gradle check result for 5f45ed2: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 5f45ed2: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

❌ Patch coverage is 78.94737% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.80%. Comparing base (eb28e77) to head (456fd82).
⚠️ Report is 18 commits behind head on main.

Files with missing lines	Patch %	Lines
...nsearch/tools/cli/plugin/InstallPluginCommand.java	78.94%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #19343      +/-   ##
============================================
- Coverage     72.81%   72.80%   -0.02%     
+ Complexity    69801    69795       -6     
============================================
  Files          5674     5674              
  Lines        320850   320863      +13     
  Branches      46383    46386       +3     
============================================
- Hits         233638   233615      -23     
- Misses        68264    68344      +80     
+ Partials      18948    18904      -44     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

❌ Gradle check result for 456fd82: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 456fd82: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 456fd82: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 456fd82: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[opensearch-trigger-bot]

The backport to 2.19 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.19 2.19
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.19
# Create a new branch
git switch --create backport/backport-19343-to-2.19
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 bb9a7d4486dde017cab986310f8c70e3573f4b72
# Push it to GitHub
git push --set-upstream origin backport/backport-19343-to-2.19
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.19

Then, create a pull request where the base branch is 2.19 and the compare/head branch is backport/backport-19343-to-2.19.