Skip to content

Introduce an index_mode param into the elastic/security Rally track#640

Merged
salvatore-campagna merged 4 commits intoelastic:masterfrom
salvatore-campagna:feature/elastic-security-index-mode-param
Aug 21, 2024
Merged

Introduce an index_mode param into the elastic/security Rally track#640
salvatore-campagna merged 4 commits intoelastic:masterfrom
salvatore-campagna:feature/elastic-security-index-mode-param

Conversation

@salvatore-campagna
Copy link
Contributor

@salvatore-campagna salvatore-campagna commented Aug 20, 2024

The index_mode parameter will be used to run Rally benchmarks comparing
indexing using standard and logsdb mode for the elastic/security track.

Enabling LogsDB is done by means of a component template which is added and
later used if the index_mode is provided. In case it is missing no index mode
will be used which will default to standard.

@salvatore-campagna
Copy link
Contributor Author

salvatore-campagna commented Aug 21, 2024

Double-checked that logsdb is used on one of the indices at the end of a benchmark

{
  ".ds-filebeat-default-2024.08.21-000001" : {
    "settings" : {
      "index" : {
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        },
        "refresh_interval" : "5s",
        "hidden" : "true",
        "provided_name" : ".ds-filebeat-default-2024.08.21-000001",
        "query" : {
          "default_field" : [
            "tags",
            "agent.ephemeral_id",
            *** REDACTED ***
            "zookeeper.audit.user",
            "fields.*"
          ]
        },
        "creation_date" : "1724230227468",
        "number_of_replicas" : "1",
        "uuid" : "JrIhT_vCTwWjlWDxINdpoQ",
        "version" : {
          "created" : "8512000"
        },
        "lifecycle" : {
          "name" : "security"
        },
        "mode" : "logsdb",
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_hot"
            }
          }
        },
        "number_of_shards" : "1",
        "max_docvalue_fields_search" : "200",
        "default_pipeline" : "filebeat-7.16.0-system-syslog-pipeline"
      }
    }
  }
}

Copy link
Member

@martijnvg martijnvg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - I left a comment about security adoption, doesn't block this PR. I think the name pattern / not using builtin stack logging template results in logsdb never to be enabled.

".fleet_globals-1",
".fleet_agent_id_verification-1"
".fleet_agent_id_verification-1",
"track-shared-logsdb-mode"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This are just my observations for security and logsdb:

  • The index pattern doesn't match with logs-*, the pattern used here is: .logs-endpoint.action.responses-*
  • The built in logging stack templates are not used here, which mean logsdb will never be enabled automatically.

Copy link
Member

@gareth-ellis gareth-ellis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - remember this will need backporting to 8.15 too

@salvatore-campagna salvatore-campagna merged commit 7051baa into elastic:master Aug 21, 2024
salvatore-campagna added a commit to salvatore-campagna/rally-tracks that referenced this pull request Aug 21, 2024
…ck (elastic#640)

The index_mode parameter will be used to run Rally benchmarks comparing
indexing using standard and logsdb mode for the elastic/security track.

Enabling LogsDB is done by means of a component template which is added and
later used if the index_mode is provided. In case it is missing no index mode
will be used which will default to standard.
salvatore-campagna added a commit that referenced this pull request Aug 21, 2024
…ck (#640) (#641)

The index_mode parameter will be used to run Rally benchmarks comparing
indexing using standard and logsdb mode for the elastic/security track.

Enabling LogsDB is done by means of a component template which is added and
later used if the index_mode is provided. In case it is missing no index mode
will be used which will default to standard.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants