Skip to content

Aggregate counter downsampling preserves resets #143381

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gmarouli merged 38 commits into from
Mar 13, 2026
Merged

Aggregate counter downsampling preserves resets #143381

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
3b0b865
Add test to check for downsampled rate accuracy
gmarouli Mar 2, 2026
9adf17a
Implement downsampling reset counter documents
gmarouli Mar 2, 2026
f606d95
Update unit tests
gmarouli Mar 2, 2026
4401898
Update yaml tests
gmarouli Mar 2, 2026
a6cf2b1
Remove assertion about downsampled index being forcemerged
gmarouli Mar 2, 2026
84a3ab6
Do not fetch timestamps if we have not counters to aggregate
gmarouli Mar 2, 2026
4b0b8f1
Polishing
gmarouli Mar 2, 2026
d763b16
Fix tests
gmarouli Mar 2, 2026
59616c8
Update docs/changelog/143381.yaml
gmarouli Mar 2, 2026
e43823b
Merge branch 'main' into downsampling-counters-with-resets
gmarouli Mar 3, 2026
ac83a08
Polishing
gmarouli Mar 3, 2026
2b80aca
Use rate instead of max_rate in DownsampleRateIT
gmarouli Mar 3, 2026
b876d05
Improve comments on the lastBucketValue
gmarouli Mar 3, 2026
9c86ad8
Apply suggestions from code review
gmarouli Mar 3, 2026
d1e5651
Improve the AggregateCounter
gmarouli Mar 3, 2026
d3a7e90
Use list of tuples instead of nested map
gmarouli Mar 3, 2026
4f2f6ff
Add issue on the TODO comment
gmarouli Mar 3, 2026
3a2ddfb
Add randomised test for a single TSID
gmarouli Mar 4, 2026
f0ea13b
Merge branch 'main' into downsampling-counters-with-resets
gmarouli Mar 5, 2026
049eee3
Extend test cases
gmarouli Mar 5, 2026
5e9fd39
Merge branch 'main' into downsampling-counters-with-resets
gmarouli Mar 6, 2026
5eb52f0
Add randomised rate test with multiple time buckets and tsids
gmarouli Mar 6, 2026
9fe9c7f
fix format
gmarouli Mar 6, 2026
a599170
Merge branch 'main' into downsampling-counters-with-resets
gmarouli Mar 9, 2026
0b6809c
Fix doc count to not count reset documents multiple times
gmarouli Mar 9, 2026
4cd449b
Merge branch 'main' into downsampling-counters-with-resets
gmarouli Mar 9, 2026
37a41f7
Uncomment after fix
gmarouli Mar 9, 2026
7000d71
Decrease flakiness
gmarouli Mar 9, 2026
b525321
Merge branch 'main' into downsampling-counters-with-resets
gmarouli Mar 10, 2026
5696feb
Polishing
gmarouli Mar 10, 2026
1c84501
Use ArrayDeque instead of stack because we do not need any thread safety
gmarouli Mar 10, 2026
ebc6d5f
Merge branch 'main' into downsampling-counters-with-resets
gmarouli Mar 11, 2026
81a08bc
Merge branch 'main' into downsampling-counters-with-resets
gmarouli Mar 11, 2026
b5710ab
Create primitive arrays for field producers in one loop
gmarouli Mar 12, 2026
aaa5073
Merge branch 'main' into downsampling-counters-with-resets
gmarouli Mar 12, 2026
66d7969
Fix assertion
gmarouli Mar 12, 2026
4851045
Merge branch 'main' into downsampling-counters-with-resets
gmarouli Mar 12, 2026
c4a1d9c
Merge branch 'main' into downsampling-counters-with-resets
gmarouli Mar 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/changelog/143381.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
area: Downsampling
issues:
- 136178
pr: 143381
summary: Aggregate counter downsampling preserves resets
type: enhancement
2 changes: 1 addition & 1 deletion x-pack/plugin/downsample/qa/mixed-cluster/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ dependencies {

restResources {
restApi {
include '_common', 'bulk', 'cluster', 'indices', 'search', 'ingest.put_pipeline', 'ingest.delete_pipeline'
include '_common', 'bulk', 'cluster', 'indices', 'search', 'ingest.put_pipeline', 'ingest.delete_pipeline', 'capabilities'
}
}

Expand Down
95 changes: 70 additions & 25 deletions ...le/qa/mixed-cluster/src/yamlRestTest/resources/rest-api-spec/test/downsample/10_basic.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,9 +83,12 @@ setup:
---
"Downsample index":
- requires:
cluster_features: ["gte_v8.10.0"]
reason: "Downsampling executed using persistent task framework from version 8.10"
test_runner_features: allowed_warnings
capabilities:
- method: POST
path: /{index}/_downsample/{target_index}
capabilities: [ "downsampling.store_reset_counters" ]
test_runner_features: [ capabilities, allowed_warnings ]
reason: Storing counter resets when downsampling was added in 9.4

- do:
allowed_warnings:
Expand All @@ -105,30 +108,72 @@ setup:
body:
sort: [ "_tsid", "@timestamp" ]

- length: { hits.hits: 4 }
- match: { hits.hits.0._source._doc_count: 2 }
# Verify dimensions & time
- match: { hits.hits.0._source.metricset: pod }
- match: { hits.hits.0._source.k8s.pod.uid: df3145b3-0563-4d3b-a0f7-897eb2876ea9 }
- length: { hits.hits: 7 }

# Downsampled doc
- match: { hits.hits.0._source._doc_count: 1 }
- match: { hits.hits.0._source.@timestamp: 2021-04-28T18:00:00.000Z }
# Dimensions
- match: { hits.hits.0._source.k8s\.pod\.uid: df3145b3-0563-4d3b-a0f7-897eb2876ea9 }
- match: { hits.hits.0._source.metricset: pod }
# Metrics
- match: { hits.hits.0._source.k8s\.pod\.multi-counter: 7 }
- match: { hits.hits.0._source.k8s\.pod\.scaled-counter: 7.0 }
- match: { hits.hits.0._source.k8s\.pod\.multi-gauge.min: 100 }
- match: { hits.hits.0._source.k8s\.pod\.multi-gauge.max: 102 }
- match: { hits.hits.0._source.k8s\.pod\.multi-gauge.sum: 607 }
- match: { hits.hits.0._source.k8s\.pod\.multi-gauge.value_count: 6 }
- match: { hits.hits.0._source.k8s\.pod\.scaled-gauge.min: 100.0 }
- match: { hits.hits.0._source.k8s\.pod\.scaled-gauge.max: 101.0 }
- match: { hits.hits.0._source.k8s\.pod\.scaled-gauge.sum: 201.0 }
- match: { hits.hits.0._source.k8s\.pod\.scaled-gauge.value_count: 2 }
- match: { hits.hits.0._source.k8s\.pod\.network\.tx.min: 1434521831 }
- match: { hits.hits.0._source.k8s\.pod\.network\.tx.max: 1434577921 }
- match: { hits.hits.0._source.k8s\.pod\.network\.tx.value_count: 2 }
# Labels
- match: { hits.hits.0._source.k8s\.pod\.ip: "10.10.55.56" }
- match: { hits.hits.0._source.k8s\.pod\.created_at: "2021-04-28T19:43:00.000Z" }
- match: { hits.hits.0._source.k8s\.pod\.number_of_containers: 1 }
- match: { hits.hits.0._source.k8s\.pod\.tags: ["backend", "test", "us-west2"] }
- match: { hits.hits.0._source.k8s\.pod\.values: [1, 1, 2] }
- is_false: hits.hits.0._source.k8s\.pod\.running

# Doc with counter resets
- is_false: hits.hits.1._source._doc_count
- match: { hits.hits.1._source.@timestamp: 2021-04-28T18:50:23.142Z }
# Dimensions
- match: { hits.hits.1._source.k8s\.pod\.uid: df3145b3-0563-4d3b-a0f7-897eb2876ea9 }
- match: { hits.hits.1._source.metricset: pod }
# Metrics
- match: { hits.hits.1._source.k8s\.pod\.multi-counter: 0 }
- match: { hits.hits.1._source.k8s\.pod\.scaled-counter: 0.0 }
# Only dimensions and counters that have been reset are in this doc
- is_false: hits.hits.1._source.k8s\.pod\.multi-gauge

# Next downsampled doc
- match: { hits.hits.2._source._doc_count: 1 }
- match: { hits.hits.2._source.@timestamp: 2021-04-28T19:00:00.000Z }
# Dimensions
- match: { hits.hits.2._source.k8s\.pod\.uid: df3145b3-0563-4d3b-a0f7-897eb2876ea9 }
- match: { hits.hits.2._source.metricset: pod }
# Metrics
- match: { hits.hits.2._source.k8s\.pod\.multi-counter: 1000 }
- match: { hits.hits.2._source.k8s\.pod\.scaled-counter: 1000.0 }
- match: { hits.hits.2._source.k8s\.pod\.multi-gauge.min: 95 }

# Verify metrics
- match: { hits.hits.0._source.k8s.pod.multi-gauge.min: 100 }
- match: { hits.hits.0._source.k8s.pod.multi-gauge.max: 102 }
- match: { hits.hits.0._source.k8s.pod.multi-gauge.sum: 607 }
- match: { hits.hits.0._source.k8s.pod.multi-gauge.value_count: 6 }
- match: { hits.hits.0._source.k8s.pod.multi-counter: 0 }
- match: { hits.hits.0._source.k8s.pod.network.tx.min: 1434521831 }
- match: { hits.hits.0._source.k8s.pod.network.tx.max: 1434577921 }
- match: { hits.hits.0._source.k8s.pod.network.tx.value_count: 2 }
- match: { hits.hits.0._source.k8s.pod.ip: "10.10.55.56" }
- match: { hits.hits.0._source.k8s.pod.created_at: "2021-04-28T19:43:00.000Z" }
- match: { hits.hits.0._source.k8s.pod.number_of_containers: 1 }
- match: { hits.hits.0._source.k8s.pod.tags: [ "backend", "test", "us-west2" ] }
- match: { hits.hits.0._source.k8s.pod.values: [ 1, 1, 2 ] }
- is_false: hits.hits.0._source.k8s.pod.running
# Doc with counter resets
- is_false: hits.hits.3._source._doc_count
- match: { hits.hits.3._source.@timestamp: 2021-04-28T19:51:03.142Z }
# Dimensions
- match: { hits.hits.3._source.k8s\.pod\.uid: df3145b3-0563-4d3b-a0f7-897eb2876ea9 }
- match: { hits.hits.3._source.metricset: pod }
# Metrics
- match: { hits.hits.3._source.k8s\.pod\.multi-counter: 76 }
# Only dimensions and counters that have been reset are in this doc
- is_false: hits.hits.3._source.k8s\.pod\.scaled-counter
- is_false: hits.hits.3._source.k8s\.pod\.multi-gauge

# Assert rollup index settings
# Assert downsample index settings
- do:
indices.get_settings:
index: test-downsample
Expand All @@ -138,7 +183,7 @@ setup:
- match: { test-downsample.settings.index.time_series.start_time: 2021-04-28T00:00:00Z }
- match: { test-downsample.settings.index.routing_path: [ "metricset", "k8s.pod.uid"] }

# Assert rollup index mapping
# Assert downsample index mapping
- do:
indices.get_mapping:
index: test-downsample
Expand Down
68 changes: 38 additions & 30 deletions .../rest/src/yamlRestTest/resources/rest-api-spec/test/downsample-with-security/10_basic.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -293,9 +293,12 @@ setup:
---
"Downsample index":
- requires:
cluster_features: ["gte_v8.13.0"]
reason: _tsid hashing introduced in 8.13
test_runner_features: allowed_warnings
capabilities:
- method: POST
path: /{index}/_downsample/{target_index}
capabilities: [ "downsampling.store_reset_counters" ]
test_runner_features: [ capabilities, allowed_warnings ]
reason: Storing counter resets when downsampling was added in 9.4

- do:
allowed_warnings:
Expand All @@ -315,26 +318,40 @@ setup:
body:
sort: [ "_tsid", "@timestamp" ]

- length: { hits.hits: 4 }
- match: { hits.hits.0._source._doc_count: 2 }
- match: { hits.hits.0._source.k8s.pod.uid: df3145b3-0563-4d3b-a0f7-897eb2876ea9 }
- match: { hits.hits.0._source.metricset: pod }
- match: { hits.hits.0._source.@timestamp: "2021-04-28T18:00:00.000Z" }
- match: { hits.hits.0._source.k8s.pod.multi-counter: 0 }
- match: { hits.hits.0._source.k8s.pod.multi-gauge.min: 100.0 }
- match: { hits.hits.0._source.k8s.pod.multi-gauge.max: 102.0 }
- match: { hits.hits.0._source.k8s.pod.multi-gauge.sum: 607.0 }
- match: { hits.hits.0._source.k8s.pod.multi-gauge.value_count: 6 }
- match: { hits.hits.0._source.k8s.pod.network.tx.min: 1434521831 }
- match: { hits.hits.0._source.k8s.pod.network.tx.max: 1434577921 }
- match: { hits.hits.0._source.k8s.pod.network.tx.value_count: 2 }
- match: { hits.hits.0._source.k8s.pod.ip: "10.10.55.56" }
- match: { hits.hits.0._source.k8s.pod.created_at: "2021-04-28T19:43:00.000Z" }
- match: { hits.hits.0._source.k8s.pod.number_of_containers: 1 }
- match: { hits.hits.0._source.k8s.pod.tags: [ "backend", "test", "us-west2" ] }
- match: { hits.hits.0._source.k8s.pod.values: [ 1, 1, 2 ] }
- length: { hits.hits: 7 }

# Downsampled doc
- match: { hits.hits.0._source._doc_count: 1 }
- match: { hits.hits.0._source.@timestamp: 2021-04-28T18:00:00.000Z }
# Dimensions
- match: { hits.hits.0._source.k8s.pod.uid: df3145b3-0563-4d3b-a0f7-897eb2876ea9 }
- match: { hits.hits.0._source.metricset: pod }
# Metrics
- match: { hits.hits.0._source.k8s.pod.multi-counter: 7 }
- match: { hits.hits.0._source.k8s.pod.multi-gauge.min: 100 }
- match: { hits.hits.0._source.k8s.pod.multi-gauge.max: 102 }
- match: { hits.hits.0._source.k8s.pod.multi-gauge.sum: 607 }
- match: { hits.hits.0._source.k8s.pod.multi-gauge.value_count: 6 }
- match: { hits.hits.0._source.k8s.pod.network.tx.min: 1434521831 }
- match: { hits.hits.0._source.k8s.pod.network.tx.max: 1434577921 }
- match: { hits.hits.0._source.k8s.pod.network.tx.value_count: 2 }
# Labels
- match: { hits.hits.0._source.k8s.pod.ip: "10.10.55.56" }
- match: { hits.hits.0._source.k8s.pod.created_at: "2021-04-28T19:43:00.000Z" }
- match: { hits.hits.0._source.k8s.pod.number_of_containers: 1 }
- match: { hits.hits.0._source.k8s.pod.tags: ["backend", "test", "us-west2"] }
- match: { hits.hits.0._source.k8s.pod.values: [1, 1, 2] }
- is_false: hits.hits.0._source.k8s.pod.running

# Doc with counter resets
- is_false: hits.hits.1._source._doc_count
- match: { hits.hits.1._source.@timestamp: 2021-04-28T18:50:23.142Z }
# Dimensions
- match: { hits.hits.1._source.k8s.pod.uid: df3145b3-0563-4d3b-a0f7-897eb2876ea9 }
- match: { hits.hits.1._source.metricset: pod }
# Metrics
- match: { hits.hits.1._source.k8s.pod.multi-counter: 0 }

# Assert downsample index settings
- do:
indices.get_settings:
Expand Down Expand Up @@ -369,12 +386,3 @@ setup:
- do:
indices.get:
index: test

# Assert downsample index has been force merged
- do:
indices.segments:
index: test-downsample

- match: { _shards.total: 1}
- match: { indices.test-downsample.shards.0.0.num_committed_segments: 1}
- match: { indices.test-downsample.shards.0.0.num_search_segments: 1}
Loading
Loading