Skip to content

*: Using native histograms for grpc middleware metrics #7393

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
May 28, 2024
Merged

*: Using native histograms for grpc middleware metrics #7393

merged 3 commits into from
May 28, 2024

Conversation

@pedro-stanaka
Copy link
Contributor

@pedro-stanaka pedro-stanaka commented May 28, 2024 •
edited
Loading

Since we updated the middleware library, we can now use native histograms to keep track of latencies in grpc calls.
This is a semi-breaking change if people enabled native histogram collection on their Prometheus monitoring Thanos instances.

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

  • Update grpc middleware library to latest version
  • Using native histograms for grpc middleware metrics (both server and client side).

Verification

image

@pedro-stanaka pedro-stanaka force-pushed the feat/native-histo-middleware branch 2 times, most recently from 87f64bd to 45b7b84 Compare May 28, 2024 07:15
@pedro-stanaka pedro-stanaka marked this pull request as ready for review May 28, 2024 07:15
@pedro-stanaka
*: Using native histograms for grpc middleware metrics
2e0c120 
Since we updated the middleware library, we can now use native histograms to keep track of latencies in grpc calls.
This is a semi-breaking change if people enabled native histogram collection on their Prometheus monitoring Thanos instances.

Signed-off-by: Pedro Tanaka <[email protected]>

adding change log

Signed-off-by: Pedro Tanaka <[email protected]>
@pedro-stanaka pedro-stanaka force-pushed the feat/native-histo-middleware branch from 45b7b84 to 2e0c120 Compare May 28, 2024 07:16
fpetkovski
fpetkovski previously approved these changes May 28, 2024
View reviewed changes
@fpetkovski
Copy link
Contributor

fpetkovski commented May 28, 2024

Looks like the docs failure is legit.

@pedro-stanaka
removing empty space;
bd17754 
Signed-off-by: Pedro Tanaka <[email protected]>
@MichaHoffmann
Copy link
Contributor

MichaHoffmann commented May 28, 2024

Can we still scrape classic histograms?

@pedro-stanaka
Copy link
Contributor Author

pedro-stanaka commented May 28, 2024 •
edited
Loading

Can we still scrape classic histograms?

Yes, if someone either did not enable native histogram feature flag or has scrape_classic_histograms: true for the scrape job, the classic histogram will still be scraped.

My local running instance scrape endpoint:

grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="0.001"} 1
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="0.01"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="0.1"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="0.3"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="0.6"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="1"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="3"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="6"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="9"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="20"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="30"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="60"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="90"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="120"} 2
grpc_server_handling_seconds_bucket{grpc_method="Query",grpc_service="thanos.Query",grpc_type="server_stream",le="+Inf"} 2

MichaHoffmann
MichaHoffmann previously approved these changes May 28, 2024
View reviewed changes
Copy link
Contributor

@MichaHoffmann MichaHoffmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@MichaHoffmann
Copy link
Contributor

MichaHoffmann commented May 28, 2024

If this breaking for some people then we should add the breaking emoji to the changelog probably

fpetkovski
fpetkovski previously approved these changes May 28, 2024
View reviewed changes
Copy link
Contributor

@fpetkovski fpetkovski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Native histograms are opt-in, so there should be no breaking changes.

@fpetkovski fpetkovski enabled auto-merge May 28, 2024 08:42
auto-merge was automatically disabled May 28, 2024 08:43

Head branch was pushed to by a user without write access

@pedro-stanaka pedro-stanaka dismissed stale reviews from fpetkovski and MichaHoffmann via 77afbda May 28, 2024 08:43
@MichaHoffmann
Copy link
Contributor

MichaHoffmann commented May 28, 2024

Native histograms are opt-in, so there should be no breaking changes.

Was just noting because the PR description mentioned that its semi-breaking

MichaHoffmann
MichaHoffmann approved these changes May 28, 2024
View reviewed changes
Copy link
Contributor

@MichaHoffmann MichaHoffmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@MichaHoffmann MichaHoffmann enabled auto-merge (squash) May 28, 2024 08:50
@MichaHoffmann MichaHoffmann merged commit dfa7dd5 into thanos-io:main May 28, 2024
@BrewTestBot BrewTestBot mentioned this pull request Jul 31, 2024
Merged
hczhu-db pushed a commit to databricks/thanos that referenced this pull request Aug 22, 2024
@pedro-stanaka @hczhu-db
*: Using native histograms for grpc middleware metrics (thanos-io#7393)
b2c9883 
* *: Using native histograms for grpc middleware metrics

Since we updated the middleware library, we can now use native histograms to keep track of latencies in grpc calls.
This is a semi-breaking change if people enabled native histogram collection on their Prometheus monitoring Thanos instances.

Signed-off-by: Pedro Tanaka <[email protected]>

adding change log

Signed-off-by: Pedro Tanaka <[email protected]>

* removing empty space;

Signed-off-by: Pedro Tanaka <[email protected]>

* Put full disclaimer in changelog

Signed-off-by: Pedro Tanaka <[email protected]>

---------

Signed-off-by: Pedro Tanaka <[email protected]>
hczhu-db pushed a commit to databricks/thanos that referenced this pull request Aug 22, 2024
@pedro-stanaka @hczhu-db
*: Using native histograms for grpc middleware metrics (thanos-io#7393)
5a3bc00 
* *: Using native histograms for grpc middleware metrics

Since we updated the middleware library, we can now use native histograms to keep track of latencies in grpc calls.
This is a semi-breaking change if people enabled native histogram collection on their Prometheus monitoring Thanos instances.

Signed-off-by: Pedro Tanaka <[email protected]>

adding change log

Signed-off-by: Pedro Tanaka <[email protected]>

* removing empty space;

Signed-off-by: Pedro Tanaka <[email protected]>

* Put full disclaimer in changelog

Signed-off-by: Pedro Tanaka <[email protected]>

---------

Signed-off-by: Pedro Tanaka <[email protected]>
jnyi pushed a commit to jnyi/thanos that referenced this pull request Oct 17, 2024
@pedro-stanaka @jnyi
*: Using native histograms for grpc middleware metrics (thanos-io#7393)
db34dcf 
* *: Using native histograms for grpc middleware metrics

Since we updated the middleware library, we can now use native histograms to keep track of latencies in grpc calls.
This is a semi-breaking change if people enabled native histogram collection on their Prometheus monitoring Thanos instances.

Signed-off-by: Pedro Tanaka <[email protected]>

adding change log

Signed-off-by: Pedro Tanaka <[email protected]>

* removing empty space;

Signed-off-by: Pedro Tanaka <[email protected]>

* Put full disclaimer in changelog

Signed-off-by: Pedro Tanaka <[email protected]>

---------

Signed-off-by: Pedro Tanaka <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MichaHoffmann MichaHoffmann MichaHoffmann approved these changes

@fpetkovski fpetkovski fpetkovski left review comments

Assignees

No one assigned

Labels

size/M

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@pedro-stanaka @fpetkovski @MichaHoffmann