Closed
Conversation
Update integrations for elastic/logs to 8.13.3
This change adds an operation called knn-recall that computes the following metrics:
* Recall
* NDCG
* Avg number of nodes visited during search
Given the size of the corpus, the true top N values used for recall operations have been approximated offline for each query as follows:
```
{
"knn": {
"field": "emb",
"query_vector": query['emb'],
"k": 10000,
"num_candidates": 10000
},
"rescore": {
"window_size": 10000,
"query": {
"query_weight": 0,
"rescore_query": {
"script_score": {
"query": {
"match_all": {}
},
"script": {
"source": "double value = dotProduct(params.query_vector, 'emb'); return sigmoid(1, Math.E, -value);",
"params": {
"query_vector": vec
}
}
}
}
}
}
}
```
This means that the computed recall is measured against the system's best possible approximate neighbor run rather than the actual top N.
For the relevance metrics, the `qrels.tsv` file contains annotations for all the queries listed in `queries.json`. This file is generated from the original training data available at [ir_datasets/msmarco_passage_v2](https://ir-datasets.com/msmarco-passage-v2.html#msmarco-passage-v2/train).
* Make exclude conditional on not serverless * Typo
The default is now host.name AND @timestamp
…ic/logs track (elastic#622) These search are based from searches that are executed by search/discovery search challenge that fetch top documents. The queries are without query and fetch 100, 500 and 1000 documents. The source is fetches using the field fetch feature, like in the searches that execute as part of search/discovery workflow. The problem is that we see combined latency and service time for search/discovery challenge and not for each search that is executed as part of this challenge (it is composite operation). By adding these searches we can get latency and service time of searches that specifically fetch _source. This information is useful for the logsdb effort. Synthetic source makes fetching _source more expensive, but currently we can't introspect at a closer level what the impact is (since search/discovery search challenge report latency / service time for multiple operations).
…ic/logs track (elastic#623) Backporting elastic#622 to the 8.15 branch. Otherwise rally nightly and esbench will not pick the change up. The version of Elasticsearch main is 8.15.0-SNAPSHOT and therefor rally nightly / esbench will use the rally track's 8.15 branch. These search are based from searches that are executed by search/discovery search challenge that fetch top documents. The queries are without query and fetch 100, 500 and 1000 documents. The source is fetches using the field fetch feature, like in the searches that execute as part of search/discovery workflow. The problem is that we see combined latency and service time for search/discovery challenge and not for each search that is executed as part of this challenge (it is composite operation). By adding these searches we can get latency and service time of searches that specifically fetch _source. This information is useful for the logsdb effort. Synthetic source makes fetching _source more expensive, but currently we can't introspect at a closer level what the impact is (since search/discovery search challenge report latency / service time for multiple operations).
Most of these are optimized in main, except two, the ESQL versions of sorting by distance, which we've set to one iteration for now to save on benchmarking time before we do performance optimizations on those.
* Add query rule tests to the wikipedia track * Update rule query to 8.15.0 syntax * Add pinned * Revert back to 8.15 syntax after local testing is completed
…weighted_tokens (elastic#633) * Update msmarco-passage-ranking track to use sparse_vector instead of weighted_tokens * Update field type to sparse_vector
* Add exclude to query ruleset benchmarks * Linting * Make pinned ID selection random * Update wikipedia/track.py Co-authored-by: Quentin Pradet <quentin.pradet@gmail.com> --------- Co-authored-by: Quentin Pradet <quentin.pradet@gmail.com>
…ck (elastic#640) The index_mode parameter will be used to run Rally benchmarks comparing indexing using standard and logsdb mode for the elastic/security track. Enabling LogsDB is done by means of a component template which is added and later used if the index_mode is provided. In case it is missing no index mode will be used which will default to standard.
* Add a retriever test with optional rerank * Linting * Fix retriever query * Fix retriever * Linting
The fleet component template is used when we try to delete it. Here we introduce a parameter that allows us to skip deletion of the component template. The default value is false, which means normally we attempt to delete it. Setting it explicitly to true we avoid deleting it. This prevents errors happening if we try to delete it and it is in use.
Serverless deployments miss ILM. As a result component templates should not use the lifecycle setting. Here we introduce a setting which allows us to exclude the lifecycle setting either using `lifecycle` parameter or a `build_flavor` parameter. This mimics what we do already for the elastic/logs track.
This causes the elastic/security track to fail execution when index_mode is set to logsdb. This is happening because LogsDB uses synthetic source which, in turn, does not support copy_to. Supporting copy_to is expected to come in Elasticsearch 8.16. In the meanwhile we just exclude the copy_to setting from the mapping so to avoid triggering the error.
Test this change in nightlies, to decide whether to enable it by default. Related to elastic/elasticsearch#112354
* Update to use JDK 21 for build
* Paramaterise timeout * Update README.md
From ES v8.14 the default index type for dense_vectors is int8_hnsw. This modifies our rally tracks to refect it.
`copy_to` is used to copy from `kubernetes.event.message` to `message`. Now it is supported in Elasticsearch 8.15 and we can benchmark the security track including it. We also remove a parameter which was used to run a modified workflow, which was using `kubernetes.event.message` instead of `message`.
This PR changes the security track so that we can enable LogsDB in index templates. Note that the failure store is only available in serverless so we gate its usage excluding it in case the deployment is not serverless. For LogsDB testing we rely on Kibana to install all other component/composable templates. This is to make sure we need limited changes to the Rally track. While testing this new configuration we discovered that installation of (component) templates done by Kibana is Serverless only happens when a user interacts with it. This means (component) templates are not installed and the `elastic/security` track execution fails as a result of using (component) templates that do not exist.
* `enable_logsdb` (default: false) Determines whether the logsdb index mode gets used. If set then index sorting is configured to only use `@timestamp` field and the `source_enabled` parameter will have no effect. * `force_merge_max_num_segments` (default: unset): An integer specifying the max amount of segments the force-merge operation should use.
If the `host.name` field does not exists, indices created as backing indices of a data stream are injected with empty values of `host.name`. Sorting on `host.name` and `@timestamp` results in sorting just on `@timestamp`. Looking at some mappings I see a `host.hostname` exists. Also a cardinality aggregation results in hundreds of distinct values which suggests the filed is not empty. We would like to test using a meaningful combination of fields to sort on. Ideally we expect better benchmark results despite being possible that other, more effective, combinations of fields might exist. We are interested, anyway, in changes over time **given a valid set of fields to sort on**.
…and (elastic#801) Also add mapping track parameter which controls whether all fields are mapped or whether almost all fields are unmapped. This allows for benchmarking elastic/logs in an unmapped context with experimental INSIST_🐔 esql command.
This unblocks queries using `pragma` when used with non-snapshot Elasticsearch builds.
* streams challenge * dummy change * add to test_logs * review comments * fix * fix 2 * clean up * remove newlines * fix another thing * control replicas * fix and cleanup * delete accidentally comitted files * reset properly * add reroute back * Update elastic/logs/pipelines/logs@stream.processing.json Co-authored-by: Grzegorz Banasiak <grzegorz.banasiak@elastic.co> * fixes * remove pipeline --------- Co-authored-by: Grzegorz Banasiak <grzegorz.banasiak@elastic.co>
Make sure the `logs` stream works with the `mapped = "unmapped"` parameter. This isn't the case right now because all other data streams are captured by the default `logs-*-*` mappings, but `logs` is not. The "unmapped" parameter requires tests with dedicated ES cluster because it operates on a different set of templates, so it cannot clean templates created by earlier races. For this reason, elastic/logs tests are not split in 2 modules.
…RoundTo transformation (elastic#824) * add more date_histogram esql queries with different date range and intervals
Sort patterned-text elastic/logs track by host.name, message.template_id, timestamp.
* Adding request-timeout for recall metrics for so_vector * adjusting request timeout logic
Now that is #131907 merged, remove all the source exclusion from our tracks since this is handled by default.
This change adds a parameter called `paragraph_size` tha determines how many random vectors are indexed per document. When the value is > 1, the track switches to the nested field automatically.
NickDris
added a commit
that referenced
this pull request
Dec 10, 2025
NickDris
added a commit
that referenced
this pull request
Dec 11, 2025
NickDris
added a commit
that referenced
this pull request
Dec 11, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
index_modeparam into theelastic/securityRally track (Introduce anindex_modeparam into theelastic/securityRally track elastic/rally-tracks#640)copy_to(Synthetic source does not still supportcopy_toelastic/rally-tracks#652)elastic/security(Include ability to force merge segments inelastic/securityelastic/rally-tracks#657)copy_tointoelastic/security(Re-enablecopy_tointoelastic/securityelastic/rally-tracks#667)logsdbindex mode in security track (Enablelogsdbindex mode in security track elastic/rally-tracks#670)host.nameis empty we need to usehost.hostname(host.nameis empty we need to usehost.hostnameelastic/rally-tracks#678)synthetic_source_keepparameter (Include a variable to controlsynthetic_source_keepparameter elastic/rally-tracks#682)host.idfor sorting security data elastic/rally-tracks#687)use_synthetic_source_recoverytrack parameter (Introduce a newuse_synthetic_source_recoverytrack parameter elastic/rally-tracks#713)tsdbtrack compatible with Elasticsearch 7.17 (Make thetsdbtrack compatible with Elasticsearch 7.17 elastic/rally-tracks#743)