[kbn/es serverless] Allow ES serverless resources (ex. `users`, `users_roles`, etc) to be overwritten via `yarn es serverless` command by paul-tavares · Pull Request #167087 · elastic/kibana

paul-tavares · 2023-09-22T19:20:19Z

Summary

`@kbn/es` package

Introduces --resources option to the es serverless command
- Allows for serverless_resources to be overwritten

Security Solution Plugin

Added customized users, users_roles and roles.yml files that includes:
- Updated roles.yml file (sync'ed with project controller version)
- Updated users / users_roles file that includes one one per security project role (name the same as the role)
New CLI script - node x-pack/plugins/security_solution/scripts/endpoint/start_es_serverless_with_security_users.js - that will start ES in serverless mode and inject the customized --resources into the command
- Any existing option that can be passed to es or es serverless can still continue to be passed to this new script. I will pass them along.

➡️ Run example: Override users and users_roles with customized version

yarn es serverless --resources=./relative/path/to/users --resources=/absolute/path/to/users_roles

➡️ yarn es serverless --help

usage: es serverless [<args>]

 Run Serverless Elasticsearch through Docker

 Options:

  --tag               Image tag of ES serverless to run from docker.elastic.co/elasticsearch-ci/elasticsearch-serverless
  --image             Full path of ES serverless image to run, has precedence over tag. [default: docker.elastic.co/kibana-ci/elasticsearch-serverless:latest-verified]
  --background        Start ES serverless without attaching to the first node's logs
  --basePath          Path to the directory where the ES cluster will store data
  --clean             Remove existing file system object store before running
  --kill              Kill running ES serverless nodes if detected on startup
  --port              The port to bind to on 127.0.0.1 [default: 9200]
  --ssl               Enable HTTP SSL on the ES cluster
  --skipTeardown      If this process exits, leave the ES cluster running in the background
  --waitForReady      Wait for the ES cluster to be ready to serve requests
+ --resources         Overrides resources under ES 'config/' directory, which are by default
+                     mounted from 'packages/kbn-es/src/serverless_resources/users'. Value should
+                     be a valid file path (relative or absolute). This option can be used multiple
+                     times if needing to override multiple files. The following files can be overwritten:
+                     operator_users.yml | role_mapping.yml | roles.yml | service_tokens | users | users_roles

  -E                  Additional key=value settings to pass to ES
  -F                  Absolute paths for files to mount into containers

Examples:

  es serverless --tag git-fec36430fba2-x86_64 # loads docker.elastic.co/elasticsearch-ci/elasticsearch-serverless:git-fec36430fba2-x86_64
  es serverless --image docker.elastic.co/kibana-ci/elasticsearch-serverless:latest-verified

➡️ Start ES with Security Roles/Users

node x-pack/plugins/security_solution/scripts/endpoint/start_es_serverless_with_security_users.js --clean --kill -E xpack.security.authc.api_key.enabled=true -E http.host=0.0.0.0

Checklist

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios

…m-users-roles' into task/olm-serverless-es-file-realm-users-roles

…-es-file-realm-users-roles # Conflicts: # packages/kbn-es/src/cli_commands/serverless.ts

…umes()`

Ikuni17

Thanks for doing this Paul! The code changes LGTM. I will submit a proper review after it is out of draft and I test locally.

...ty_solution/scripts/endpoint/common/roles_users/serverless/es_serverless_resources/README.md

elasticmachine · 2023-09-26T17:14:26Z

Pinging @elastic/security-defend-workflows (Team:Defend Workflows)

…-es-file-realm-users-roles

Ikuni17

Tested OPs changes locally. LGTM, nice work.

packages/kbn-es/src/utils/docker.ts

…-es-file-realm-users-roles

patrykkopycinski

Thank you!

tomsonpl

Looks good to me, great job 👍

…-es-file-realm-users-roles # Conflicts: # x-pack/plugins/security_solution/public/common/components/markdown_editor/plugins/index.ts

kibana-ci · 2023-09-27T18:11:02Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 10243af

Failed CI Steps

Jest Integration Tests #1

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @paul-tavares

…oles to cypress e2e test setup (#167446) ## Summary Goal of this PR is to re-enable the serverless tests that require the login credentials for users that have the pre-defined roles from serverless assigned to them. ### `@kbn/test` changes - Added support for `esServerlessOptions` to FTR config. Currently allows for `resources` to be defined - `resources` overrides were introduced [in this PR](#167087) - new FTR option will allow for testing serverless with a set of users/roles that are specific to the project type ### Security Solution Plugin - Added `esServerlessOptions` to the Defend Workflows cypress configurations - Un-skips all serverless specific tests (now that we have support for users/roles that are specific to the Security project) - Changed the default username for cypress `login()` task to be `endpoint_operations_analyst` - Note that the previously used `endpoint_operations_analyst` role was also updated to match the definition used for serverless. - Added new common `fleet_server_services` cli module with reusable methods for working with fleet server, including generic `startFleetServer()` method - New CLI script: `node x-pack/plugins/security_solution/scripts/endpoint/start_fleet_server.js` - Starts a fleet server locally (via Docker) and connects it to the Kibana - Supports running fleet server locally for serverless as well

add soc_manager to ES serverless resources

bf8ea61

paul-tavares self-assigned this Sep 22, 2023

paul-tavares changed the title ~~[POC] Add users/roles for serverless for dev~~ [kbn/es serverless] Allow users, user_roles and roles.yml to be overwritten without having to change source code Sep 25, 2023

paul-tavares mentioned this pull request Sep 25, 2023

[kbn/es serverless] Allow overriding config files #167092

Closed

paul-tavares added 11 commits September 25, 2023 16:23

Update serverless CLI command to support --resources argument

246d64c

add list of security solution ES serverless resources

bbe498a

Merge remote-tracking branch 'origin/task/olm-serverless-es-file-real…

7a241f6

…m-users-roles' into task/olm-serverless-es-file-realm-users-roles

add security solution script to start ES in serverless with users/roles

2e49a09

Fix resources file paths

b816b1c

fix es serverless script

1d971cd

Merge remote-tracking branch 'upstream/main' into task/olm-serverless…

ec6debf

…-es-file-realm-users-roles # Conflicts: # packages/kbn-es/src/cli_commands/serverless.ts

additional improvements to the es start script

c27b914

Updates to the README file for security solution es serverless resources

31d00c5

updated README file

0ccbb7b

add tests for to validate use of --resources in `setupServerlessVol…

3822817

…umes()`

paul-tavares added release_note:skip Skip the PR/issue when compiling release notes Team:Defend Workflows “EDR Workflows” sub-team of Security Solution v8.11.0 labels Sep 26, 2023

add available users for login to script output

64be34c

Ikuni17 reviewed Sep 26, 2023

View reviewed changes

...ty_solution/scripts/endpoint/common/roles_users/serverless/es_serverless_resources/README.md Outdated Show resolved Hide resolved

better format of users available on kbn serverless

8feff00

paul-tavares marked this pull request as ready for review September 26, 2023 17:14

paul-tavares requested review from a team as code owners September 26, 2023 17:14

paul-tavares requested review from dasansol92 and parkiino September 26, 2023 17:14

paul-tavares added 3 commits September 26, 2023 13:15

remove --teardown from example

fb04041

Merge remote-tracking branch 'upstream/main' into task/olm-serverless…

34e9b85

…-es-file-realm-users-roles

remove commented out code

b5d7109

Ikuni17 approved these changes Sep 26, 2023

View reviewed changes

packages/kbn-es/src/utils/docker.ts Show resolved Hide resolved

paul-tavares added 4 commits September 26, 2023 15:35

add log when resource is overridden

5d50a93

Merge remote-tracking branch 'upstream/main' into task/olm-serverless…

3a7fadd

…-es-file-realm-users-roles

Correct duplicate word

37cfd8f

Fix type issue in markdown editor

3fa9282

paul-tavares requested a review from a team as a code owner September 26, 2023 20:45

patrykkopycinski approved these changes Sep 26, 2023

View reviewed changes

tomsonpl approved these changes Sep 27, 2023

View reviewed changes

Merge remote-tracking branch 'upstream/main' into task/olm-serverless…

cca86c9

…-es-file-realm-users-roles # Conflicts: # x-pack/plugins/security_solution/public/common/components/markdown_editor/plugins/index.ts

paul-tavares enabled auto-merge (squash) September 27, 2023 12:54

Merge branch 'main' into task/olm-serverless-es-file-realm-users-roles

10243af

paul-tavares merged commit 45a28d4 into elastic:main Sep 27, 2023

kibanamachine added the backport:skip This PR does not require backporting label Sep 27, 2023

paul-tavares deleted the task/olm-serverless-es-file-realm-users-roles branch September 27, 2023 19:10

paul-tavares mentioned this pull request Sep 29, 2023

[Security Solution][Endpoint] Add serverless Security project users/roles to cypress e2e test setup #167446

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[kbn/es serverless] Allow ES serverless resources (ex. `users`, `users_roles`, etc) to be overwritten via `yarn es serverless` command#167087

[kbn/es serverless] Allow ES serverless resources (ex. `users`, `users_roles`, etc) to be overwritten via `yarn es serverless` command#167087
paul-tavares merged 23 commits intoelastic:mainfrom
paul-tavares:task/olm-serverless-es-file-realm-users-roles

paul-tavares commented Sep 22, 2023 •

edited

Loading

Uh oh!

Ikuni17 left a comment

Uh oh!

Uh oh!

elasticmachine commented Sep 26, 2023

Uh oh!

Ikuni17 left a comment •

edited

Loading

Uh oh!

Uh oh!

patrykkopycinski left a comment

Uh oh!

tomsonpl left a comment

Uh oh!

kibana-ci commented Sep 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

paul-tavares commented Sep 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

@kbn/es package

Security Solution Plugin

Checklist

Uh oh!

Ikuni17 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticmachine commented Sep 26, 2023

Uh oh!

Ikuni17 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

patrykkopycinski left a comment

Choose a reason for hiding this comment

Uh oh!

tomsonpl left a comment

Choose a reason for hiding this comment

Uh oh!

kibana-ci commented Sep 27, 2023

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

paul-tavares commented Sep 22, 2023 •

edited

Loading

`@kbn/es` package

Ikuni17 left a comment •

edited

Loading