Skip to content
This repository was archived by the owner on Jun 26, 2024. It is now read-only.

Conversation

@Harnoor-se7en
Copy link
Contributor

@Harnoor-se7en Harnoor-se7en commented Apr 7, 2022

Description

Currently, there are no metrics to track the number of failed requests in the data query layer. Adding an error counter to track requests failing in gateway service. The metric is exposed on /metrics endpoint.

For query-service - https://github.com/hypertrace/gateway-service/pull/128

@Harnoor-se7en Harnoor-se7en requested a review from a team April 7, 2022 12:22
@Harnoor-se7en
Copy link
Contributor Author

adding some further commits based on the feedback will reopen the PR later.

@Harnoor-se7en Harnoor-se7en reopened this Apr 8, 2022
@codecov
Copy link

codecov bot commented Apr 19, 2022

Codecov Report

Merging #128 (1562d94) into main (2856a11) will decrease coverage by 0.30%.
The diff coverage is 25.80%.

@@             Coverage Diff              @@
##               main     #128      +/-   ##
============================================
- Coverage     79.56%   79.26%   -0.31%     
- Complexity     1215     1216       +1     
============================================
  Files           113      113              
  Lines          5472     5503      +31     
  Branches        446      446              
============================================
+ Hits           4354     4362       +8     
- Misses          920      943      +23     
  Partials        198      198              
Flag Coverage Δ
unit 79.26% <25.80%> (-0.31%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...hypertrace/gateway/service/GatewayServiceImpl.java 18.89% <25.80%> (+1.15%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2856a11...1562d94. Read the comment docs.


private Counter serviceResponseErrorCounter;
private Counter serviceResponseSuccessCounter;
private static final String ERROR_COUNTER_NAME = "hypertrace.gateway.response.errors";
Copy link
Contributor

@kotharironak kotharironak Apr 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we have single metrics instead of two with status for fail/success? This way, you can check the call rate too.
e.g
hypertrace_gateway_requests_status {"code"="fail"}
hypertrace_gateway_requests_status {"code"="success"}

Any suggestions code or status or error?

hypertrace_gateway_requests_status {"error"="true"}
hypertrace_gateway_requests_status {"error"="false"}

Secondly, shall we have the same naming convention across services? I see that here - hypertrace/query-service#138 - we have a different convention.

So, should we go with :
hypertrace_gateway_service_requests_status ?

And in PR - hypertrace/query-service#138,
hypertrace.query.service.request.status?

Copy link
Contributor Author

@Harnoor-se7en Harnoor-se7en Apr 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we have single metrics instead of two with status for fail/success?

Yes makes sense.

Below looks good:

hypertrace.gateway.service.requests.status {"error"="true"}
hypertrace.gateway.service.requests.status {"error"="false"}

shall we have the same naming convention across services?

Yes my bad, keeping the same naming convention across both the services like - hypertrace.<servicename>.service.requests.status

Pushing the above changes.


private void initMetrics() {
serviceResponseErrorCounter =
PlatformMetricsRegistry.registerCounter(ERROR_COUNTER_NAME, ImmutableMap.of());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we go with the above metrics - #128 (comment), you may have to register a counter with

serviceResponseErrorCounter = PlatformMetricsRegistry.registerCounter("hypertrace.gateway.requests.status", ImmutableMap.of("code", "fail"))

And,

serviceResponseSuccessCounter =
        PlatformMetricsRegistry.registerCounter("hypertrace.gateway.requests.status", ImmutableMap.of("code", "success"))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Making the change.

@kotharironak
Copy link
Contributor

@laxman-traceable @ravisingal Do you have any suggestions on metrics?

Copy link
Contributor

@kotharironak kotharironak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

private static final String REQUEST_TIMEOUT_CONFIG_KEY = "request.timeout";
private static final int DEFAULT_REQUEST_TIMEOUT_MILLIS = 10000;

private Counter serviceResponseErrorCounter;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:
serviceResponseErrorCounter -> requestStatusErrorCounter
serviceResponseSuccessCounter -> requestStatusSuccessCounter

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@kotharironak kotharironak merged commit 9d710f8 into hypertrace:main Apr 22, 2022
@github-actions
Copy link

Unit Test Results

  60 files  ±0    60 suites  ±0   7s ⏱️ -2s
337 tests ±0  337 ✔️ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 9d710f8. ± Comparison against base commit 2856a11.

Harnoor-se7en added a commit to razorpay/hypertrace-gateway-service that referenced this pull request May 17, 2022
…rvice (hypertrace#128)

* added error count of failed requests from gateway service to query service

* added success metric in the gateway service

* added success metric in gateway service

* spotless apply fix for the github actions

* adressing the PR comments 1

* spotlessApply fix

* changed the metric tags

* adressing the PR comments 2
Harnoor-se7en added a commit to razorpay/hypertrace-gateway-service that referenced this pull request Jun 20, 2022
…rvice (hypertrace#128)

* added error count of failed requests from gateway service to query service

* added success metric in the gateway service

* added success metric in gateway service

* spotless apply fix for the github actions

* adressing the PR comments 1

* spotlessApply fix

* changed the metric tags

* adressing the PR comments 2

cherry-picking gateway service error counts
Harnoor-se7en added a commit to razorpay/hypertrace-gateway-service that referenced this pull request Jun 20, 2022
…rvice (hypertrace#128)

* added error count of failed requests from gateway service to query service

* added success metric in the gateway service

* added success metric in gateway service

* spotless apply fix for the github actions

* adressing the PR comments 1

* spotlessApply fix

* changed the metric tags

* adressing the PR comments 2
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants