[Security Solution][Endpoint] Add serverless Security project users/roles to cypress e2e test setup by paul-tavares · Pull Request #167446 · elastic/kibana

paul-tavares · 2023-09-27T19:35:50Z

Summary

Goal of this PR is to re-enable the serverless tests that require the login credentials for users that have the pre-defined roles from serverless assigned to them.

`@kbn/test` changes

Added support for esServerlessOptions to FTR config. Currently allows for resources to be defined
- resources overrides were introduced in this PR
- new FTR option will allow for testing serverless with a set of users/roles that are specific to the project type

Security Solution Plugin

Added esServerlessOptions to the Defend Workflows cypress configurations
Un-skips all serverless specific tests (now that we have support for users/roles that are specific to the Security project)
Changed the default username for cypress login() task to be endpoint_operations_analyst
- Note that the previously used endpoint_operations_analyst role was also updated to match the definition used for serverless.
Added new common fleet_server_services cli module with reusable methods for working with fleet server, including generic startFleetServer() method
New CLI script: node x-pack/plugins/security_solution/scripts/endpoint/start_fleet_server.js
- Starts a fleet server locally (via Docker) and connects it to the Kibana
- Supports running fleet server locally for serverless as well

Serverless specific tests now passing

Options available with new start_fleet_server.js cli script:

node ./x-pack/plugins/security_solution/scripts/endpoint/start_fleet_server.js --help

  node x-pack/plugins/security_solution/scripts/endpoint/start_fleet_server.js

  Start fleet-server locally and connect it to Kibana/ES

  Options:
    --version           Optional. The Agent version to be used when installing fleet server.
                        Default: uses the same version as the stack (kibana). Version
                        can also be from 'SNAPSHOT'.
                        NOTE: this value will be specifically set to 'latest' when ran against
                        kibana in serverless mode.
                        Examples: 8.6.0, 8.7.0-SNAPSHOT
    --policy            Optional. The UUID of the agent policy that should be used to enroll
                        fleet-server with Kibana/ES (Default: uses existing (if found) or
                        creates a new one)
    --force             Optional. If true, then fleet-server will be started and connected to
                        kibana even if one seems to already be configured.
    --port              Optional. The port number where fleet-server will listen for requests.
                        (Default: 8220)
    --username          Optional. User name to be used for auth against elasticsearch and
                        kibana (Default: elastic).
    --password          Optional. Password associated with the username (Default: changeme)
    --kibanaUrl         Optional. The url to Kibana (Default: http://127.0.0.1:5601)
    --elasticUrl        Optional. The url to Elasticsearch (Default: http://127.0.0.1:9200)
    --verbose, -v      Log verbosely
    --debug            Log debug messages (less than verbose)
    --quiet            Only log errors
    --silent           Don't log anything
    --help             Show this message

Output of the start_fleet_server.js script

start_fleet_server.js
 info Starting Fleet Server and connecting it to Kibana
   │ info Verifying Docker is installed.
       │ info Docker version 20.10.22, build 3a2c30b
   │ info Starting a new fleet server using Docker (version: 8.11.0-SNAPSHOT)
       │ info Kibana running in serverless mode.
       │        - will install/run standalone Fleet Server
       │        - version adjusted to [latest] from [8.11.0-SNAPSHOT]
       │ info Checking status of elastic Docker network.
           │ info Using existing network.
       │ info Fleet server started
       │ info Updating Fleet with new fleet server host: https://192.168.1.87:8220
           │ info No update needed. Fleet server host URL already defined in fleet settings.
       │ info Checking if Fleet output for Elasticsearch needs to be updated
       │ info Waiting for server to register with Elasticsearch
   │ info Done. Fleet server up and running
 info 

      Container Name: dev-fleet-server.8220
      Container Id:   ba131b3dd3a2ceb7a95e40b54bde7b8498318136f6665aa076b6716760ef1906

      View running output:  docker attach ---sig-proxy=false dev-fleet-server.8220
      Shell access:         docker exec -it dev-fleet-server.8220 /bin/bash
      Kill container:       docker kill ba131b3dd3a2ceb7a95e40b54bde7b8498318136f6665aa076b6716760ef1906

…erless

…e-serverless-tests

…inition

…e-serverless-tests # Conflicts: # x-pack/plugins/security_solution/scripts/endpoint/common/roles_users/serverless/es_serverless_resources/roles.yml

…e-serverless-tests

paul-tavares · 2023-10-03T00:46:34Z

x-pack/plugins/security_solution/scripts/endpoint/common/fleet_server/fleet_server_services.ts

@@ -0,0 +1,673 @@
+/*


FYI: I will have a follow up PR to refactor the existing endpint_agent_runner utility to use this module (will also look for import's for private modules of endpoint_agent_runner). For now, most of the code here is a duplicate.

elasticmachine · 2023-10-03T13:10:23Z

Pinging @elastic/security-defend-workflows (Team:Defend Workflows)

…e-serverless-tests

tomsonpl

Hey, I've gone through the code, left a few suggestions and questions (just to help me understand, thanks in advance 👍 ).
All in all, great job as usual, thanks for doing this 👍

tomsonpl · 2023-10-03T13:50:42Z

x-pack/plugins/security_solution/scripts/endpoint/fleet_server/index.ts

+export const cli = async () => {
+  return run(
+    async (cliContext: RunContext) => {
+      const username = cliContext.flags.username as string;


Why do we need to cast this like that?

username is not set as a string in .flags

ref:

kibana/packages/kbn-dev-cli-runner/src/flags.ts

Line 23 in 7f82102

[key: string]: undefined | boolean | string | string[];

tomsonpl · 2023-10-03T13:51:54Z

x-pack/plugins/security_solution/scripts/endpoint/common/stack_services.ts

-    const isServerlessEs = (await tmpEsClient.info()).version.build_flavor === 'serverless';
+    await waitForKibana(tmpKbnClient);
+
+    // const isServerlessEs = (await tmpEsClient.info()).version.build_flavor === 'serverless';


tomsonpl · 2023-10-03T13:54:15Z

x-pack/plugins/security_solution/scripts/endpoint/common/fleet_server/fleet_server_services.ts

+      if (isServerless) {
+        log.info(`Waiting for server to register with Elasticsearch`);
+
+        await waitForFleetServerToRegisterWithElasticsearch(kbnClient, hostname, 120000);


would it be worth to add a log.verbose after this? as with hostToEnroll below?

There is nothing here to log. The one below logs the Agent record in log.verbose()

tomsonpl · 2023-10-03T13:56:38Z

x-pack/plugins/security_solution/scripts/endpoint/common/fleet_server/fleet_server_services.ts

+  kbnClient: KbnClient;
+  logger: ToolingLog;
+  policyId: string;
+  serviceToken: string;


maybe worth to specify this as string | undefined , passing an empty string may be confusing. what do you think?

same with policyId I guess ?

These can't be undefined for managed fleet server. I'll make them optional and will add runtime validation.

Yes, what I meant is it's either a string for manager fs or really undefined right?
But what you just did is also great, so thanks!

tomsonpl · 2023-10-03T13:59:49Z

x-pack/plugins/security_solution/scripts/endpoint/common/fleet_server/fleet_server_services.ts

+export const startFleetServer = async ({
+  kbnClient,
+  logger,
+  policy,


this policy, coming from cliConfig.flags.policy - is this supposed to be supported only if used as a CLI and not CI (eg. cypress?). Which means that if provided, we do not call getOrCreateFleetServerAgentPolicyId ?

correct - if policy (Fleet agent policy UUID) is provided, then we don't create one automatically if one does not yet exist. It could be used from CI or CLI - it does not matter. It just provides the consumer the ability to define the specific agent policy ID to use in starting the server

tomsonpl · 2023-10-03T14:02:49Z

x-pack/plugins/security_solution/scripts/endpoint/common/fleet_server/fleet_server_services.ts

+  - version adjusted to [latest] from [${agentVersion}]`);
+
+      agentVersion = 'latest';
+      await maybeCreateDockerNetwork(log);


Is there any case where we need to additionally create the network (in serverless mode)? I understand that network is being already created with dockerized ES.

Not sure 🤷 . I think there are some additional discussions in @patrykkopycinski 's PR about creating additional networks for Agent (hosts).

I just wanted to make it consistent to make sure all docker containers are running on the same docker network

tomsonpl · 2023-10-03T14:04:15Z

x-pack/plugins/security_solution/scripts/endpoint/common/fleet_server/fleet_server_services.ts

+    const fleetServerUrl = `https://${localhostRealIp}:${port}`;
+    const esURL = new URL(await getFleetElasticsearchOutputHost(kbnClient));
+    const containerName = `dev-fleet-server.${port}`;
+    const hostname = `dev-fleet-server.${port}.${Math.random().toString(32).substring(2, 6)}`;


Is this necessary?

Yes. If we don't, the host name in fleet is duplicated when we run it multiple times locally. This ensure that the host name is unique. It was also causing an issue when calling waitForHostToEnroll()

tomsonpl · 2023-10-03T14:05:35Z

x-pack/plugins/security_solution/scripts/endpoint/common/fleet_server/fleet_server_services.ts

+
+    // Only fetch/create a fleet-server policy
+    if (!policyId && !isServerless) {
+      policyId = await getOrCreateFleetServerAgentPolicyId(kbnClient, logger);


I think we can set policyId the same way as serviceToken below, it might get more readable, what do you think?

tomsonpl · 2023-10-03T14:08:10Z

x-pack/plugins/security_solution/scripts/endpoint/common/fleet_server/fleet_server_services.ts

+              await kbnClient
+                .request<GetOneOutputResponse>({
+                  method: 'PUT',
+                  headers: { 'elastic-api-version': '2023-10-31' },


I see you use API_VERSIONS.public.v1 is it possible to use it also here?

That const never made sense to me and I think will lead to issues. I was never really sure why a const was used. Version number will diverge at the api level (is my understanding) not the release level across all APIs

tomsonpl · 2023-10-03T14:08:43Z

x-pack/plugins/security_solution/scripts/endpoint/common/fleet_services.ts

-      },
-      query: {
-        perPage: 100,
+        'elastic-api-version': '2023-10-31',


…e-serverless-tests

jbudz

packages/kbn-test

…e-serverless-tests # Conflicts: # packages/kbn-test/src/es/test_es_cluster.ts # packages/kbn-test/src/functional_test_runner/lib/config/schema.ts # packages/kbn-test/src/functional_tests/lib/run_elasticsearch.ts # x-pack/plugins/security_solution/scripts/endpoint/common/stack_services.ts # x-pack/test/defend_workflows_cypress/serverless_config.ts

… path

dasansol92

LGTM! Thanks!

dasansol92 · 2023-10-04T15:15:58Z

x-pack/plugins/security_solution/scripts/endpoint/common/fleet_services.ts

+      method: 'GET',
+      path: PACKAGE_POLICY_API_ROUTES.LIST_PATTERN,
+      headers: {
+        'elastic-api-version': '2023-10-31',


Should we use the API_VERSIONS.public.v1 from fleet plugin?

See my prior comment to @tomsonpl 's (same comment). I never really understood why we are using const's for API version

ashokaditya

Thanks for the changes. 🔥 I've not tested this locally but will take it for a spin later.

ashokaditya · 2023-10-04T15:31:50Z

...plugins/security_solution/scripts/endpoint/common/roles_users/endpoint_operations_analyst.ts

+            '.lists*',
+            '.items*',


I believe these should be .lists-* and .items-*

I'm going to leave it as is, since it matches whats defined in Project Controller (cc/ @dhurley14 )

ashokaditya · 2023-10-04T15:33:07Z

...plugins/security_solution/scripts/endpoint/common/roles_users/endpoint_operations_analyst.ts

+        {
+          names: [
+            'names:',
+            '.alerts-security*',


This should be .alerts-security.alerts-*

same comment as above

kibana-ci · 2023-10-04T16:56:34Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 5693c7c

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #1 / dashboard app - group 6 dashboard snapshots compare controls snapshot in dark mode
[job] [logs] Security Solution Cypress Tests #9 / Ransomware Detection Alerts Ransomware in Timelines Renders ransomware entries in timelines table Renders ransomware entries in timelines table

Metrics [docs]

✅ unchanged

History

💔 Build #165361 failed ea37cb3
💔 Build #165342 failed ecacb30
💔 Build #165319 failed f61e534
💚 Build #165123 succeeded 4b21538
💔 Build #165072 failed 0010825
💔 Build #165036 failed 09c5dab

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @paul-tavares

patrykkopycinski

Thank you @paul-tavares 🙇

dmlemeshko

kbn-test changes LGTM

paul-tavares added 4 commits September 27, 2023 15:12

add esServerlessOptions to the FTR config

3a184aa

Add esServerlessOptions to the endpoint FTR configurations for serv…

d605d93

…erless

Un-skip servereless tests for roles/pli's

9cc175d

Merge remote-tracking branch 'upstream/main' into task/olm-7614-enabl…

b9f3cd5

…e-serverless-tests

paul-tavares added release_note:skip Skip the PR/issue when compiling release notes Team:Defend Workflows “EDR Workflows” sub-team of Security Solution v8.11.0 labels Sep 27, 2023

paul-tavares self-assigned this Sep 27, 2023

paul-tavares added 11 commits September 27, 2023 15:38

Change default login username to endpoint_operations_analyst

4b1a27e

add missing system_indices_superuser role

fe94d41

new start_fleet_server script

7338703

Improve logging in start fleet server cli tool

26460f6

Update endpoint_operations_analyst role to match the serverless def…

dd2f28e

…inition

Merge remote-tracking branch 'upstream/main' into task/olm-7614-enabl…

42f68fe

…e-serverless-tests # Conflicts: # x-pack/plugins/security_solution/scripts/endpoint/common/roles_users/serverless/es_serverless_resources/roles.yml

actually check if fleet server is running by calling its status api

7f63092

add support for serverless (fleet server standalone)

f9c52f6

adjust username on rbac ESS test

0f393e9

Merge remote-tracking branch 'upstream/main' into task/olm-7614-enabl…

5d3c4a3

…e-serverless-tests

Minor updates

c71fcc8

patrykkopycinski mentioned this pull request Oct 2, 2023

Setup E2E against Serverless ES, Kibana, Fleet server standalone and Elastic endpoint agent in VM #167720

Merged

paul-tavares added 7 commits October 2, 2023 10:02

Merge remote-tracking branch 'upstream/main' into task/olm-7614-enabl…

198ee95

…e-serverless-tests

add comment to refactor test file

fe8b3b5

Update fetchKibanaStatus() and waitForKibana() to use kbnClient

b764816

Merge remote-tracking branch 'upstream/main' into task/olm-7614-enabl…

e4b83d8

…e-serverless-tests

increase timeout for respons actions api checks

e348eb5

skip test failures

ba82263

Merge remote-tracking branch 'upstream/main' into task/olm-7614-enabl…

f961432

…e-serverless-tests

paul-tavares commented Oct 3, 2023

View reviewed changes

paul-tavares marked this pull request as ready for review October 3, 2023 13:10

paul-tavares requested a review from a team as a code owner October 3, 2023 13:10

add issue number to skip tests

d4da51a

paul-tavares requested a review from patrykkopycinski October 3, 2023 13:12

Merge remote-tracking branch 'upstream/main' into task/olm-7614-enabl…

a10e34b

…e-serverless-tests

tomsonpl approved these changes Oct 3, 2023

View reviewed changes

paul-tavares and others added 5 commits October 3, 2023 11:21

add check/validation to policyId and serviceToken

b5bfd11

Merge remote-tracking branch 'upstream/main' into task/olm-7614-enabl…

2c8a0dd

…e-serverless-tests

adjust code from code review

2ba060b

Merge branch 'main' into task/olm-7614-enable-serverless-tests

09c5dab

Merge branch 'main' into task/olm-7614-enable-serverless-tests

0010825

jbudz approved these changes Oct 3, 2023

View reviewed changes

paul-tavares and others added 3 commits October 3, 2023 17:12

Merge branch 'main' into task/olm-7614-enable-serverless-tests

4b21538

rename common method back to getLocalhostRealIp() and adjust import…

ecacb30

… path

paul-tavares requested review from MadameSheema, banderror, maximpn and oatkiller as code owners October 4, 2023 15:04

dasansol92 approved these changes Oct 4, 2023

View reviewed changes

fix serverless config for Cy

ea37cb3

ashokaditya approved these changes Oct 4, 2023

View reviewed changes

fix config again

5693c7c

paul-tavares enabled auto-merge (squash) October 4, 2023 16:28

patrykkopycinski approved these changes Oct 4, 2023

View reviewed changes

dmlemeshko approved these changes Oct 4, 2023

View reviewed changes

paul-tavares merged commit 990963f into elastic:main Oct 4, 2023

kibanamachine added the backport:skip This PR does not require backporting label Oct 4, 2023

paul-tavares deleted the task/olm-7614-enable-serverless-tests branch October 4, 2023 19:02

banderror mentioned this pull request Oct 5, 2023

[Security Solution] Support testing with different roles in Serverless Cypress and API integration tests #164451

Open

Conversation

paul-tavares commented Sep 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

@kbn/test changes

Security Solution Plugin

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Oct 3, 2023

Uh oh!

tomsonpl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paul-tavares Oct 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jbudz left a comment

Choose a reason for hiding this comment

Uh oh!

dasansol92 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ashokaditya left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kibana-ci commented Oct 4, 2023

paul-tavares commented Sep 27, 2023 •

edited

Loading

`@kbn/test` changes

paul-tavares Oct 3, 2023 •

edited

Loading