-
Notifications
You must be signed in to change notification settings - Fork 322
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore: autofix corrupted job-status stats #4224
Conversation
Important Auto Review SkippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the To trigger a single review, invoke the Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on X ? TipsChat with CodeRabbit Bot (
|
|
||
func (r *Stats) fixCorrupted() { | ||
if r.corrupted() { | ||
r.In = r.Out + r.Failed |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the reason behind this corruption? Why are we programatically fixing this corruption?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This doesn't happen regularly, however there are some pretty rare occasions where server bugs can cause this kind of corruption (example).
The idea here is to "fix" this corruption virtually for the consuming service to resume without other teams being alerted and fire an alert on our side so that we can further investigate. The alert would be setup on the new warning log.
3f6e02a
to
88084d9
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #4224 +/- ##
==========================================
+ Coverage 73.79% 73.81% +0.01%
==========================================
Files 388 388
Lines 54800 54834 +34
==========================================
+ Hits 40440 40476 +36
+ Misses 12038 12034 -4
- Partials 2322 2324 +2 ☔ View full report in Codecov by Sentry. |
88084d9
to
14b3216
Compare
Description
Whenever corrupted job-status stats are encountered (in < out), gateway fixes them before responding and logs a warning.
Linear Ticket
resolves PIPE-617
Security