#3666 - Queue Monitoring - Enable Prometheus Metrics #4012
Conversation
andrewsignori-aot
changed the title
sources/packages/backend/apps/queue-consumers/src/queues-metrics.module.module.ts
Show resolved
Hide resolved
|
Backend Unit Tests Coverage Report
|
E2E Workflow Workers Coverage Report
|
E2E Queue Consumers Coverage Report
|
E2E SIMS API Coverage Report
|
| if (!this.monitoredQueueProviders) { | ||
| const queues = await this.queueService.queueConfigurationModel(); | ||
| this.monitoredQueueProviders = queues | ||
| .filter((queueModel) => queueModel.isActive) |
There was a problem hiding this comment.
Are we not ignoring the FT queues here?
There was a problem hiding this comment.
The full-time toggle is not checked here. If the queues are not executing they will not produce any event.
The events will be associated but never triggered. I do not see any harm in it. Please let me know if further discussion would be required on this.
There was a problem hiding this comment.
Not required and I am good based on what we discussed.
| */ | ||
| setGlobalMetricsConfigurations(): void { | ||
| register.setDefaultLabels({ app: DEFAULT_METRICS_APP_LABEL }); | ||
| collectDefaultMetrics({ labels: { app: DEFAULT_METRICS_APP_LABEL } }); |
There was a problem hiding this comment.
for collect default metrics, is it not necesscary to set the register. Is it because we are using default registry? let me know if I am missing something.
collectDefaultMetrics({
labels: { app: DEFAULT_METRICS_APP_LABEL, register },
});There was a problem hiding this comment.
Yes, there is no need to set the register unless using a custom one.
https://github.com/siimon/prom-client?tab=readme-ov-file#default-metrics
There was a problem hiding this comment.
Great Work @andrewsignori-aot 👍 Thank you for the PR walkthrough.
andrewsignori-aot
had a problem deploying
to
DEV
GitHub Actions
Failure
/metricsendpoint onqueue-consumersto expose Prometheus metrics following BC Gov docs.active,completed,failed,delayed, and 'waiting' jobs. This represents the same status observed in the Bull Board. The garage metrics rely on Redis queries and will always get the most updated values from Redis.metricsendpoint is invoked. This metric is not needed to achieve queue monitoring but seems a great addition and useful data to support future analysis.queueName,queueEvent, andqueueTypeto allow querying in Sysdig.Sysdig POCs
The Sysdig configurations are not final and were created to support the validation of the code in this PR but should not be considered final or part of the PR evaluation.
Alerts generated for a queue with a failed job.
Same alerts are configured to be sent using an email channel.
Sample Dashboard
The Queues Overview dashboard has some examples of data but should not be considered final or part of the PR evaluation.
Note: the sysdig users and roles were updated and added to this PR, and it is already deployed to both tools environments. If time allows, further effort can be made to enhance the current process but any action beyond the user list update is not part of this PR.