Emit Sentry warning and GCP log for sampled throttled queries

[xurui-c]

Allocation policies are our mechanism for doing traffic management for Snuba queries. Currently, the result of applying multiple allocation policies in the internal API is simply accept/reject/throttle.

In a previous PR, we emit a Sentry warning and a GCP log for every query throttled by Snuba's capacity management system. However, we quickly realized a large volume of queries were throttled.

To address this, we sample the throttled queries such that we will only emit a Sentry warning and a GCP log for every SAMPLING_FREQUENCYth throttled query for a given referrer. For example, if SAMPLING_FREQUENCY = 100, then we will emit the warning and log for every 100th throttled query for the same referrer.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.03%. Comparing base (d724281) to head (b5279a6).

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #73620      +/-   ##
==========================================
- Coverage   78.03%   78.03%   -0.01%     
==========================================
  Files        6640     6640              
  Lines      297033   297040       +7     
  Branches    51152    51153       +1     
==========================================
+ Hits       231797   231800       +3     
- Misses      58961    58965       +4     
  Partials     6275     6275              

Files	Coverage Δ
src/sentry/utils/snuba.py	89.75% <100.00%> (+0.10%)	⬆️

... and 2 files with indirect coverage changes

[xurui-c]

close in favor of #73709