Add date histogram with filter, and filter and metrics #700

asimmahmood1 · 2025-10-08T17:37:11Z

Description

to test out skiplist based date histogram
skiplist is enabled by default http_logs has @timestamp field: Enable skip_list for @timestamp field or index sort field by default OpenSearch#19480
TODO: add index.sort.field option on @timestamp

Issues Resolved

Closes [https://github.com/opensearch-project/OpenSearch/issues/19123]

Testing

New functionality includes testing

[Describe how this change was tested]

See

opensearch-project/OpenSearch#19509 (comment)

Backport to Branches:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@timestamp

* to test out skiplist based date histogram * skiplist is enabled by default http_logs has @timestamp field: opensearch-project/OpenSearch#19480 * TODO: add index.sort.field option on @timestamp Signed-off-by: Asim Mahmood <[email protected]>

@timestamp

* to test out skiplist based date histogram * skiplist is enabled by default http_logs has @timestamp field: opensearch-project/OpenSearch#19480 * TODO: add index.sort.field option on @timestamp Signed-off-by: Asim Mahmood <[email protected]> (cherry picked from commit bf2c36e) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

asimmahmood1 · 2025-10-08T20:29:11Z

[ec2-user@ip-172-31-61-197 ~]$ opensearch-benchmark execute-test --pipeline=benchmark-only --workload=http_logs --target-hosts=localhost:9200 --kill-running-processes --workload-param "max_num_segments:10,ingest_percentage:20,warmup_iterations:10,iterations:20" --include-tasks="hourly_agg,hourly_agg_with_filter,hourly_agg_with_filter_and_metrics" --user-tag asimmahm:3.3-rc2-http_logs-skiplist_sorted_baseline

/ __ ____ ___ ____ / / ____ / / / __ ) ____ / / ____ ___ ____ / /
/ / / / __ / _ / __ \ / _ / __ / ___/ ___/ __ \ / __ / _ \/ __ \/ ___/ __ \/ __ / __ `/ / ///
/ // / // / / / / // / / /_/ / / / // / / / / // / / / / / // / / / / / / / / // / / / ,<
_/ ./_// //____/_/_,/_/ _// // //_// //_// /// // //_,// //||
/_/

[INFO] [Test Execution ID]: fff1d3f3-2ba6-4033-87f9-b2955288ed7a
[WARNING] Local changes in [/home/ec2-user/.osb/benchmarks/workloads/default] prevent workloads update from remote. Please commit your changes.
^C[WARNING] Local changes in [/home/ec2-user/.osb/benchmarks/workloads/default] prevent workloads update from remote. Please commit your changes.
[WARNING] Local changes in [/home/ec2-user/.osb/benchmarks/workloads/default] prevent workloads update from remote. Please commit your changes.
^C[ERROR] Cannot execute-test. .

Getting further help:

Check the log files in /home/ec2-user/.osb/logs for errors.
Read the documentation at https://opensearch.org/docs.
Ask a question on the forum at https://forum.opensearch.org/.
Raise an issue at https://github.com/opensearch-project/OpenSearch-Benchmark/issues and include the log files in /home/ec2-user/.osb/logs.

[INFO] FAILURE (took 2 seconds)

[ec2-user@ip-172-31-61-197 ~]$ opensearch-benchmark execute-test --pipeline=benchmark-only --workload=http_logs --target-hosts=localhost:9200 --kill-running-processes --workload-param "max_num_segments:10,ingest_percentage:20,warmup_iterations:10,iterations:20" --include-tasks="hourly_agg,hourly_agg_with_filter,hourly_agg_with_filter_and_metrics" --user-tag asimmahm:3.3-rc2-http_logs-skiplist_sorted_baseline

/ __ ____ ___ ____ / / ____ / / / __ ) ____ / / ____ ___ ____ / /
/ / / / __ / _ / __ \ / _ / __ / ___/ ___/ __ \ / __ / _ \/ __ \/ ___/ __ \/ __ / __ `/ / ///
/ // / // / / / / // / / /_/ / / / // / / / / // / / / / / // / / / / / / / / // / / / ,<
_/ ./_// //____/_/_,/_/ _// // //_// //_// /// // //_,// //||
/_/

[INFO] [Test Execution ID]: 481f61d2-e8f4-41cf-9c08-79d8285942af
[WARNING] Local changes in [/home/ec2-user/.osb/benchmarks/workloads/default] prevent workloads update from remote. Please commit your changes.
[WARNING] Local changes in [/home/ec2-user/.osb/benchmarks/workloads/default] prevent workloads update from remote. Please commit your changes.
[WARNING] Local changes in [/home/ec2-user/.osb/benchmarks/workloads/default] prevent workloads update from remote. Please commit your changes.
[INFO] Executing test with workload [http_logs], test_procedure [append-no-conflicts] and provision_config_instance ['external'] with version [3.3.0-SNAPSHOT].

[WARNING] merges_total_time is 122139 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] indexing_total_time is 1826612 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] refresh_total_time is 448790 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] flush_total_time is 105991 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
Running hourly_agg [100% done]
Running hourly_agg_with_filter [100% done]
Running hourly_agg_with_filter_and_metrics [100% done]

_______             __   _____

/ () __ / / / /_ ______
/ /_ / / / `/ / / / / / _
/ / / / / / / // / / / / // // / / / /
// /// //_,// /__/_/_// ___/

Metric	Task	Value	Unit
Cumulative indexing time of primary shards		30.4435	min
Min cumulative indexing time across primary shards		0	min
Median cumulative indexing time across primary shards		0.534992	min
Max cumulative indexing time across primary shards		2.48933	min
Cumulative indexing throttle time of primary shards		0	min
Min cumulative indexing throttle time across primary shards		0	min
Median cumulative indexing throttle time across primary shards		0	min
Max cumulative indexing throttle time across primary shards		0	min
Cumulative merge time of primary shards		2.03565	min
Cumulative merge count of primary shards		26
Min cumulative merge time across primary shards		0	min
Median cumulative merge time across primary shards		0.0483833	min
Max cumulative merge time across primary shards		0.138283	min
Cumulative merge throttle time of primary shards		0	min
Min cumulative merge throttle time across primary shards		0	min
Median cumulative merge throttle time across primary shards		0	min
Max cumulative merge throttle time across primary shards		0	min
Cumulative refresh time of primary shards		7.47983	min
Cumulative refresh count of primary shards		383
Min cumulative refresh time across primary shards		0	min
Median cumulative refresh time across primary shards		0.155075	min
Max cumulative refresh time across primary shards		0.5468	min
Cumulative flush time of primary shards		1.76652	min
Cumulative flush count of primary shards		55
Min cumulative flush time across primary shards		0	min
Median cumulative flush time across primary shards		0.000525	min
Max cumulative flush time across primary shards		0.354567	min
Total Young Gen GC time		0	s
Total Young Gen GC count		0
Total Old Gen GC time		0	s
Total Old Gen GC count		0
Store size		3.28917	GB
Translog size		2.04891e-06	GB
Heap used for segments		0	MB
Heap used for doc values		0	MB
Heap used for terms		0	MB
Heap used for norms		0	MB
Heap used for points		0	MB
Heap used for stored fields		0	MB
Segment count		25
Min Throughput	hourly_agg	0.21	ops/s
Mean Throughput	hourly_agg	0.21	ops/s
Median Throughput	hourly_agg	0.21	ops/s
Max Throughput	hourly_agg	0.22	ops/s
50th percentile latency	hourly_agg	42.8195	ms
90th percentile latency	hourly_agg	46.2467	ms
100th percentile latency	hourly_agg	50.6097	ms
50th percentile service time	hourly_agg	39.8683	ms
90th percentile service time	hourly_agg	41.6636	ms
100th percentile service time	hourly_agg	43.4268	ms
error rate	hourly_agg	0	%
Min Throughput	hourly_agg_with_filter	0.21	ops/s
Mean Throughput	hourly_agg_with_filter	0.21	ops/s
Median Throughput	hourly_agg_with_filter	0.21	ops/s
Max Throughput	hourly_agg_with_filter	0.22	ops/s
50th percentile latency	hourly_agg_with_filter	249.561	ms
90th percentile latency	hourly_agg_with_filter	253.833	ms
100th percentile latency	hourly_agg_with_filter	256.109	ms
50th percentile service time	hourly_agg_with_filter	247.171	ms
90th percentile service time	hourly_agg_with_filter	251.315	ms
100th percentile service time	hourly_agg_with_filter	253.597	ms
error rate	hourly_agg_with_filter	0	%
Min Throughput	hourly_agg_with_filter_and_metrics	0.21	ops/s
Mean Throughput	hourly_agg_with_filter_and_metrics	0.21	ops/s
Median Throughput	hourly_agg_with_filter_and_metrics	0.21	ops/s
Max Throughput	hourly_agg_with_filter_and_metrics	0.22	ops/s
50th percentile latency	hourly_agg_with_filter_and_metrics	1126.16	ms
90th percentile latency	hourly_agg_with_filter_and_metrics	1143.5	ms
100th percentile latency	hourly_agg_with_filter_and_metrics	1148.94	ms
50th percentile service time	hourly_agg_with_filter_and_metrics	1123	ms
90th percentile service time	hourly_agg_with_filter_and_metrics	1138.64	ms
100th percentile service time	hourly_agg_with_filter_and_metrics	1146.07	ms
error rate	hourly_agg_with_filter_and_metrics	0	%

[INFO] SUCCESS (took 460 seconds)

[ec2-user@ip-172-31-61-197 ~]$

@timestamp

* to test out skiplist based date histogram * skiplist is enabled by default http_logs has @timestamp field: opensearch-project/OpenSearch#19480 * TODO: add index.sort.field option on @timestamp (cherry picked from commit bf2c36e) Signed-off-by: Asim Mahmood <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

asimmahmood1 requested review from IanHoang, OVI3D0, VijayanB, beaioun, gkamat and rishabh6788 as code owners October 8, 2025 17:37

asimmahmood1 self-assigned this Oct 8, 2025

asimmahmood1 added the backport 3 Backport to the "3" branch label Oct 8, 2025

asimmahmood1 added this to Performance Roadmap Oct 8, 2025

github-project-automation bot moved this to Todo in Performance Roadmap Oct 8, 2025

asimmahmood1 moved this from Todo to In Progress in Performance Roadmap Oct 8, 2025

asimmahmood1 moved this from In Progress to In-Review in Performance Roadmap Oct 8, 2025

asimmahmood1 mentioned this pull request Oct 8, 2025

Add nyc_taxis date_histogram_calendar_interval_with_filter operation #697

Closed

6 tasks

rishabh6788 approved these changes Oct 8, 2025

View reviewed changes

github-project-automation bot moved this from In-Review to In Progress in Performance Roadmap Oct 8, 2025

rishabh6788 merged commit bf2c36e into opensearch-project:main Oct 8, 2025
4 checks passed

github-project-automation bot moved this from In Progress to Done in Performance Roadmap Oct 8, 2025

opensearch-trigger-bot bot mentioned this pull request Oct 8, 2025

[Backport 3] Add date histogram with filter, and filter and metrics #701

Merged

asimmahmood1 mentioned this pull request Oct 8, 2025

Skiplist - date histogram sub aggregation performance change opensearch-project/OpenSearch#19509

Closed

3 tasks

asimmahmood1 deleted the httplogs_datehisto branch October 21, 2025 17:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add date histogram with filter, and filter and metrics #700

Add date histogram with filter, and filter and metrics #700

Uh oh!

asimmahmood1 commented Oct 8, 2025 •

edited

Loading

Uh oh!

Uh oh!

asimmahmood1 commented Oct 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Add date histogram with filter, and filter and metrics #700

Add date histogram with filter, and filter and metrics #700

Uh oh!

Conversation

asimmahmood1 commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues Resolved

Testing

Backport to Branches:

Uh oh!

Uh oh!

asimmahmood1 commented Oct 8, 2025

[INFO] FAILURE (took 2 seconds)

/ () ____ / / / /_____ ________ / /_ / / __ / __ `/ / __ / / __ / / _ / __/ / / / / / // / / / / // // / / / __/ // /// //_,// /__/_/_// ___/

[INFO] SUCCESS (took 460 seconds)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

asimmahmood1 commented Oct 8, 2025 •

edited

Loading

/ () __ / / / /_ ______
/ /_ / / / `/ / / / / / _
/ / / / / / / // / / / / // // / / / /
// /// //_,// /__/_/_// ___/