Skip to content

Conversation

@asimmahmood1
Copy link
Contributor

Description

  • test skiplist optimization
  • added both date_histogram_calendar_interval and date_histogram_calendar_interval_with_filter to test procedures so it shows up in default run
  • manually tested

Issues Resolved

Updates [https://github.com/opensearch-project/OpenSearch/issues/17964]

Testing

  • New functionality includes testing

[Describe how this change was tested]

Tested manually:

 opensearch-benchmark execute-test --pipeline=benchmark-only --workload=nyc_taxis --target-hosts=localhost:9200 --kill-running-processes   --workload-param "max_num_segments:10,ingest_percentage:20"  --include-tasks="date_histogram_calendar_interval_with_filter"  --user-tag asimmahm:3.3-nyc-dateskiplist-candidate
Metric Task Baseline Contender %Diff Diff Unit
Min Throughput date_histogram_calendar_interval 1.50556 1.50476 -0.05% -0.0008 ops/s
Mean Throughput date_histogram_calendar_interval 1.50916 1.50775 -0.09% -0.00141 ops/s
Median Throughput date_histogram_calendar_interval 1.50834 1.50707 -0.08% -0.00127 ops/s
Max Throughput date_histogram_calendar_interval 1.51644 1.51374 -0.18% -0.0027 ops/s
50th percentile latency date_histogram_calendar_interval 8.55135 171.146 +1901.40% 🔴 162.595 ms
90th percentile latency date_histogram_calendar_interval 9.55957 177.176 +1753.39% 🔴 167.616 ms
99th percentile latency date_histogram_calendar_interval 10.5288 210.3 +1897.38% 🔴 199.771 ms
100th percentile latency date_histogram_calendar_interval 12.3942 225.214 +1717.09% 🔴 212.82 ms
50th percentile service time date_histogram_calendar_interval 7.07599 169.712 +2298.42% 🔴 162.636 ms
90th percentile service time date_histogram_calendar_interval 8.06119 176.087 +2084.38% 🔴 168.026 ms
99th percentile service time date_histogram_calendar_interval 8.97942 208.5 +2221.97% 🔴 199.52 ms
100th percentile service time date_histogram_calendar_interval 10.5131 223.478 +2025.71% 🔴 212.964 ms
error rate date_histogram_calendar_interval 0 0 0.00% 0 %
Min Throughput date_histogram_calendar_interval_with_filter 1.48533 1.50975 1.64% 0.02441 ops/s
Mean Throughput date_histogram_calendar_interval_with_filter 1.49176 1.51611 1.63% 0.02435 ops/s
Median Throughput date_histogram_calendar_interval_with_filter 1.49247 1.51467 1.49% 0.0222 ops/s
Max Throughput date_histogram_calendar_interval_with_filter 1.49498 1.52903 2.28% 0.03405 ops/s
50th percentile latency date_histogram_calendar_interval_with_filter 33.6987 9.22881 -72.61% 🟢 -24.4699 ms
90th percentile latency date_histogram_calendar_interval_with_filter 38.0945 10.5454 -72.32% 🟢 -27.5491 ms
99th percentile latency date_histogram_calendar_interval_with_filter 38.9266 12.8582 -66.97% 🟢 -26.0685 ms
100th percentile latency date_histogram_calendar_interval_with_filter 39.0018 13.3591 -65.75% 🟢 -25.6427 ms
50th percentile service time date_histogram_calendar_interval_with_filter 32.2213 7.70275 -76.09% 🟢 -24.5186 ms
90th percentile service time date_histogram_calendar_interval_with_filter 36.5675 8.91834 -75.61% 🟢 -27.6492 ms
99th percentile service time date_histogram_calendar_interval_with_filter 37.1727 10.9156 -70.64% 🟢 -26.2571 ms
100th percentile service time date_histogram_calendar_interval_with_filter 37.7295 11.7682 -68.81% 🟢 -25.9612 ms
error rate date_histogram_calendar_interval_with_filter 0 0 0.00% 0 %

Backport to Branches:

  • 6
  • 7
  • 1
  • 2
  • 3

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…to test skiplist optimization

* added both date_histogram_calendar_interval and date_histogram_calendar_interval_with_filter to test procedures so it shows up in default run
* manually tested

Signed-off-by: Asim Mahmood <[email protected]>
@asimmahmood1 asimmahmood1 added v2.0.0 Related to OpenSearch 2.0 and removed v2.0.0 Related to OpenSearch 2.0 labels Sep 23, 2025
@asimmahmood1 asimmahmood1 moved this from Todo to In-Review in Performance Roadmap Sep 23, 2025
@asimmahmood1 asimmahmood1 self-assigned this Sep 23, 2025
@asimmahmood1 asimmahmood1 added the backport 3 Backport to the "3" branch label Oct 2, 2025
@asimmahmood1
Copy link
Contributor Author

Since nyc_taxis uses dropoff_time as primary key, will need to explictly enable skiplist. Instead http_logs uses @timestamp field which has skiplist enabled by default.
Closing this in favor of #700 since nyc_taxis

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport 3 Backport to the "3" branch

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant