Closed
Conversation
…o elastic/logs track (elastic#622) (elastic#625) Backporting elastic#622 to the 8.15 branch. Otherwise rally nightly and esbench will not pick the change up. The version of Elasticsearch main is 8.15.0-SNAPSHOT and therefor rally nightly / esbench will use the rally track's 8.15 branch. These search are based from searches that are executed by search/discovery search challenge that fetch top documents. The queries are without query and fetch 100, 500 and 1000 documents. The source is fetches using the field fetch feature, like in the searches that execute as part of search/discovery workflow. The problem is that we see combined latency and service time for search/discovery challenge and not for each search that is executed as part of this challenge (it is composite operation). By adding these searches we can get latency and service time of searches that specifically fetch _source. This information is useful for the logsdb effort. Synthetic source makes fetching _source more expensive, but currently we can't introspect at a closer level what the impact is (since search/discovery search challenge report latency / service time for multiple operations).
Update integrations for elastic/logs to 8.13.3
…ck (elastic#640) (elastic#641) The index_mode parameter will be used to run Rally benchmarks comparing indexing using standard and logsdb mode for the elastic/security track. Enabling LogsDB is done by means of a component template which is added and later used if the index_mode is provided. In case it is missing no index mode will be used which will default to standard.
…lastic#643) The fleet component template is used when we try to delete it. Here we introduce a parameter that allows us to skip deletion of the component template. The default value is false, which means normally we attempt to delete it. Setting it explicitly to true we avoid deleting it. This prevents errors happening if we try to delete it and it is in use.
Serverless deployments miss ILM. As a result component templates should not use the lifecycle setting. Here we introduce a setting which allows us to exclude the lifecycle setting either using `lifecycle` parameter or a `build_flavor` parameter. This mimics what we do already for the elastic/logs track.
Backport to 8.15: - Keep array source in logsdb mode (655)
* Update to use JDK 21 for build
* Paramaterise timeout * Update README.md --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
…tic#664) From ES v8.14 the default index type for dense_vectors is int8_hnsw. This modifies our rally tracks to refect it.
`copy_to` is used to copy from `kubernetes.event.message` to `message`. Now it is supported in Elasticsearch 8.15 and we can benchmark the security track including it. We also remove a parameter which was used to run a modified workflow, which was using `kubernetes.event.message` instead of `message`.
This PR changes the security track so that we can enable LogsDB in index templates. Note that the failure store is only available in serverless so we gate its usage excluding it in case the deployment is not serverless. For LogsDB testing we rely on Kibana to install all other component/composable templates. This is to make sure we need limited changes to the Rally track. While testing this new configuration we discovered that installation of (component) templates done by Kibana is Serverless only happens when a user interacts with it. This means (component) templates are not installed and the `elastic/security` track execution fails as a result of using (component) templates that do not exist.
This back ports elastic#672 to 8.15 branch. * `enable_logsdb` (default: false) Determines whether the logsdb index mode gets used. If set then index sorting is configured to only use `@timestamp` field and the `source_enabled` parameter will have no effect. * `force_merge_max_num_segments` (default: unset): An integer specifying the max amount of segments the force-merge operation should use.
…astic#679) If the `host.name` field does not exists, indices created as backing indices of a data stream are injected with empty values of `host.name`. Sorting on `host.name` and `@timestamp` results in sorting just on `@timestamp`. Looking at some mappings I see a `host.hostname` exists. Also a cardinality aggregation results in hundreds of distinct values which suggests the filed is not empty. We would like to test using a meaningful combination of fields to sort on. Ideally we expect better benchmark results despite being possible that other, more effective, combinations of fields might exist. We are interested, anyway, in changes over time **given a valid set of fields to sort on**. (cherry picked from commit 0ca00a0)
(cherry picked from commit 3ae3304) Co-authored-by: Gareth Ellis <gareth.ellis@elastic.co>
…tic#682) (elastic#684) This PR introduces a new track parameter, `synthetic_source_keep` which is used to control the behaviour of synthetic source for all field types. It can have values `none`, `arrays` or `all` (`all` not usable when set at index level). See elastic/elasticsearch#112706 to understand the effect of each value. Later on we will use this to change the behaviour in our nightlies and run benchmarks on both `elastic/logs` and `elastic/security` using value `arrays`.
…lastic#683) (elastic#685) Backporting elastic#683 to 8.15 branch. The addition of the index.mapping.synthetic_source_keep to tsdb is new. To http_logs is not and before the index.mapping.synthetic_source_keep setting was hard coded to arrays. I will open a separate PR that adds the source_keep track param to nightly configs. Having the source_keep makes comparing benchmark results between the different source keep options easier.
(cherry picked from commit 4493616) Co-authored-by: Grzegorz Banasiak <grzegorz.banasiak@elastic.co>
Skips Fleet component templates in elastic/logs when running with serverless (cherry picked from commit 4cd9d4a) Co-authored-by: Grzegorz Banasiak <grzegorz.banasiak@elastic.co>
host.hostname has cardinality 100 while host.id has cardinality 50. This happen because in the dataset there is a host.if per each couple ho hostnames, like a single host.id and for each of them two hostnames like 'dustin.windows' and 'dustin.linux'. This is probably an artifact of the data generation script. Lower cardinality fields might: * reduce sorting overhead due to less comparisons * improve compression due to more data clustering together This change should at least allow us if there is any benefit in choosing a lower cardinality field. (cherry picked from commit e2ca95e) Co-authored-by: Salvatore Campagna <93581129+salvatore-campagna@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Add support for source mode to elastic/logs, elastic/security and http_logs. Backport of elastic#692 to 8.15 branch. (cherry picked from commit ae63824)
* Add recall and NDCG operations in msmarco-v2-vector (elastic#610) This change adds an operation called knn-recall that computes the following metrics: * Recall * NDCG * Avg number of nodes visited during search Given the size of the corpus, the true top N values used for recall operations have been approximated offline for each query as follows: ``` { "knn": { "field": "emb", "query_vector": query['emb'], "k": 10000, "num_candidates": 10000 }, "rescore": { "window_size": 10000, "query": { "query_weight": 0, "rescore_query": { "script_score": { "query": { "match_all": {} }, "script": { "source": "double value = dotProduct(params.query_vector, 'emb'); return sigmoid(1, Math.E, -value);", "params": { "query_vector": vec } } } } } } } ``` This means that the computed recall is measured against the system's best possible approximate neighbor run rather than the actual top N. For the relevance metrics, the `qrels.tsv` file contains annotations for all the queries listed in `queries.json`. This file is generated from the original training data available at [ir_datasets/msmarco_passage_v2](https://ir-datasets.com/msmarco-passage-v2.html#msmarco-passage-v2/train). (cherry picked from commit b6f3535) * Exclude msmarco from IT tests (elastic#708) --------- Co-authored-by: Jim Ferenczi <jim.ferenczi@elastic.co>
(cherry picked from commit 17cc202)
(cherry picked from commit 23f6712) Co-authored-by: Nik Everett <nik9000@gmail.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
(cherry picked from commit 68fb7d4)
Pinning action to a full length commit SHA [see](https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions#using-third-party-actions) (cherry picked from commit 50e9ddb) # Conflicts: # .github/workflows/backport.yml Co-authored-by: Paul McCann <paul.mccann@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This reverts commit 6532f10.
…#795) (elastic#810) AD (anomaly detection) and DFA (data frame analytics) ML endpoints are no longer available in general purpose ES projects in QA, so this is switching IT tests to vector-optimized projects. This is a temporary workaround. Ultimately we need to differentiate serverless project type depending on Rally track. (cherry picked from commit 530092b) Co-authored-by: Grzegorz Banasiak <grzegorz.banasiak@elastic.co>
(cherry picked from commit b3a16c2) Co-authored-by: Grzegorz Banasiak <grzegorz.banasiak@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
…tic/logs track (elastic#716) (elastic#717) (cherry picked from commit 76072a1) Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com> Co-authored-by: Oleksandr Kolomiiets <olkolomiiets@gmail.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
(cherry picked from commit 7250bb3) Co-authored-by: Grzegorz Banasiak <grzegorz.banasiak@elastic.co>
(cherry picked from commit 571300c) Co-authored-by: Grzegorz Banasiak <grzegorz.banasiak@elastic.co>
(cherry picked from commit 581478b) # Conflicts: # catalog-info.yaml
(cherry picked from commit 37a20a8) Co-authored-by: Grzegorz Banasiak <grzegorz.banasiak@elastic.co>
The parse tool requires the compressed file to be passed to it. (cherry picked from commit 3f72278) Co-authored-by: Jesse Bacon <dread.roberts@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
index_modeparam into theelastic/securityRally track (Introduce anindex_modeparam into theelastic/securityRally track elastic/rally-tracks#640) (Introduce anindex_modeparam into theelastic/securityRally track elastic/rally-tracks#641)copy_to(Synthetic source does not still supportcopy_toelastic/rally-tracks#653)elastic/security(Include ability to force merge segments inelastic/securityelastic/rally-tracks#657) (Include ability to force merge segments inelastic/securityelastic/rally-tracks#659)copy_tointoelastic/security(Re-enablecopy_tointoelastic/securityelastic/rally-tracks#667) (Re-enablecopy_tointoelastic/security(#667) elastic/rally-tracks#669)logsdbindex mode in security track (Enablelogsdbindex mode in security track elastic/rally-tracks#670) (Enablelogsdbindex mode in security track (#670) elastic/rally-tracks#674)host.nameis empty we need to usehost.hostname(host.nameis empty we need to usehost.hostnameelastic/rally-tracks#678) ([8.15]host.nameis empty we need to usehost.hostname(#678) elastic/rally-tracks#679)synthetic_source_keepparameter (Include a variable to controlsynthetic_source_keepparameter elastic/rally-tracks#682) (Backport synthetic source keep param elastic/rally-tracks#684)host.idfor sorting security data elastic/rally-tracks#687) ([8.15] host.id has lower cardinality (#687) elastic/rally-tracks#689)