[CapMan visibility] emit a warning when a query gets throttled by the capacity management system #73442
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔍 Existing Issues For ReviewYour pull request is modifying functions with the following pre-existing issues: 📄 File: src/sentry/utils/snuba.py
Did you find this useful? React with a 👍 or 👎 |
github-actions
bot
added
the
Scope: Backend
volokluev
reviewed
Jun 27, 2024
volokluev
reviewed
Jun 27, 2024
xurui-c
force-pushed
the
rachel/throttleWarningMetric
branch
from
b2357b7
to
9d2d3a5
Compare
xurui-c
force-pushed
the
rachel/throttleWarningMetric
branch
from
9d2d3a5
to
4affcc4
Compare
xurui-c
force-pushed
the
rachel/throttleWarningMetric
branch
from
4affcc4
to
8801af4
Compare
xurui-c
force-pushed
the
rachel/throttleWarningMetric
branch
from
8801af4
to
6b9f5f9
Compare
xurui-c
force-pushed
the
rachel/throttleWarningMetric
branch
from
6b9f5f9
to
40dd852
Compare
volokluev
approved these changes
Jun 28, 2024
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #73442 +/- ##
========================================
Coverage 78.01% 78.01%
========================================
Files 6634 6637 +3
Lines 296698 296856 +158
Branches 51095 51120 +25
========================================
+ Hits 231455 231601 +146
- Misses 58860 58867 +7
- Partials 6383 6388 +5
|
nhsiehgit
pushed a commit
that referenced
this pull request
Jul 1, 2024
Snuba throttles queries in the magnitude of 10k-20k. A [previous PR](#73442) emits a warning to Sentry and logs to GCP for every throttled query, and thus has the potential to overwhelm Sentry/GCP. This PR removes the warnings and logs in case we have an incident. Co-authored-by: Rachel Chen <[email protected]>
xurui-c
added a commit
that referenced
this pull request
Jul 3, 2024
…c) (#73709) Allocation policies are our mechanism for doing traffic management for Snuba queries. Currently, the result of applying multiple allocation policies in the internal API is simply accept/reject/throttle. In a #73442, we emit a Sentry warning and a GCP log for every query throttled by Snuba's capacity management system. However, we quickly realized [a large volume of queries were throttled](https://sentry.sentry.io/issues/5550501231/?project=1&query=&referrer=issue-stream&statsPeriod=24h&stream_index=5). To address this, we sample the throttled queries such that we will only emit a Sentry warning and a GCP log for only 1% of all throttled queries, across all referrers and policies. Co-authored-by: Rachel Chen <[email protected]>
xurui-c
changed the title
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://getsentry.atlassian.net/browse/SNS-2799
Allocation policies are our mechanism for doing traffic management for Snuba queries. Currently, the result of applying allocation policies in the internal API is simply accept/reject/throttle, and Snuba sends back a payload to Sentry that contains metadata about those policy decisions. In the case of a throttled query, we want to emit a warning to GCP as well as a warning to Sentry Issues