Skip to content

Commit

Permalink
fix: change .inc calls to .increment (#8000)
Browse files Browse the repository at this point in the history
We are observing incorrect data in Prometheus, which is consistently
non-reproducible. After a restart, the issue does not occur, but if the
pods run for an extended period, they seem to enter a strange state
where the counters become entangled and start sharing arbitrary values
that are added to the counters.

For example, the `feature_lifecycle_stage_entered` counter gets an
arbitrary value, such as 12, added when `inc()` is called. The
`exceedsLimitErrorCounter` shows the same behavior, and the code
implementation is identical.

We also tested some existing `increase()` counters, and they do not
suffer from this issue.

All calls to `counter.labels(labels).inc(`) will be replaced by
`counter.increment()` to try to mitigate the issue.
  • Loading branch information
sjaanus authored Aug 28, 2024
1 parent bed4f66 commit 7ad686e
Showing 1 changed file with 8 additions and 10 deletions.
18 changes: 8 additions & 10 deletions src/lib/metrics.ts
Original file line number Diff line number Diff line change
Expand Up @@ -428,9 +428,9 @@ export default class MetricsMonitor {
eventBus.on(
events.STAGE_ENTERED,
(entered: { stage: string; feature: string }) => {
featureLifecycleStageEnteredCounter
.labels({ stage: entered.stage })
.inc();
featureLifecycleStageEnteredCounter.increment({
stage: entered.stage,
});
},
);

Expand All @@ -440,9 +440,7 @@ export default class MetricsMonitor {
resource,
limit,
}: { resource: string; limit: number }) => {
exceedsLimitErrorCounter
.labels({ resource, limit })
.inc();
exceedsLimitErrorCounter.increment({ resource, limit });
},
);

Expand Down Expand Up @@ -862,16 +860,16 @@ export default class MetricsMonitor {
});
});
eventStore.on(PROJECT_CREATED, () => {
projectActionsCounter.labels({ action: PROJECT_CREATED }).inc();
projectActionsCounter.increment({ action: PROJECT_CREATED });
});
eventStore.on(PROJECT_ARCHIVED, () => {
projectActionsCounter.labels({ action: PROJECT_ARCHIVED }).inc();
projectActionsCounter.increment({ action: PROJECT_ARCHIVED });
});
eventStore.on(PROJECT_REVIVED, () => {
projectActionsCounter.labels({ action: PROJECT_REVIVED }).inc();
projectActionsCounter.increment({ action: PROJECT_REVIVED });
});
eventStore.on(PROJECT_DELETED, () => {
projectActionsCounter.labels({ action: PROJECT_DELETED }).inc();
projectActionsCounter.increment({ action: PROJECT_DELETED });
});

const logger = config.getLogger('metrics.ts');
Expand Down

0 comments on commit 7ad686e

Please sign in to comment.