Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GCP pubsub scaler not working in projects with shared metrics scopes #5256

Closed
DP19 opened this issue Dec 5, 2023 · 0 comments · Fixed by #5258
Closed

GCP pubsub scaler not working in projects with shared metrics scopes #5256

DP19 opened this issue Dec 5, 2023 · 0 comments · Fixed by #5258
Labels
bug Something isn't working

Comments

@DP19
Copy link
Contributor

DP19 commented Dec 5, 2023

Report

We have a dev project that has workloads and is also the scoping project for other environments. Due to this, the metric client will sometimes send back the backlog size of one of the monitored projects instead of the project itself

Expected Behavior

I'd expect the backlog size to be for the project the pub/sub subscription is in

Actual Behavior

data from the metrics client is not filtering on the project itself so it will give back num_undelivered_messages for multiple projects in this metric scoped project. This leads to the keda operator to not increase scale from 0 since it sees no messages in the backlog

Steps to Reproduce the Problem

  1. setup a gcp project that contains a topic/subscription
  2. setup a secondary project with the same topic/subscription
  3. scope project 2 metrics to project 1
  4. run keda in a cluster in project 1, create a scaled object to scale on the subscription created in step 1/2
  5. add messages to step 1's topic
  6. see behavor where it may sometimes not see the backlog if there is no backlog in project 2

Logs from KEDA operator

2023-12-05T20:04:52Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "project1", "scaledObject.Name": "testing", "scaler": "pubsubScaler", "metricName": "s0-gcp-ps-projects-project1-subscriptions-subscription1", "metrics": [{"metricName":"s0-gcp-ps-projects-project1-subscriptions-subscription1","metricLabels":null,"timestamp":"2023-12-05T20:04:52Z","value":"0"}], "activity": false, "scalerError": null}

screen shot at the time where project1 had 7k backlog with the other projects having 0
Screenshot 2023-12-05 at 4 16 42 PM

KEDA Version

2.12.0

Kubernetes Version

1.27

Platform

Google Cloud

Scaler Details

GCP pubsub scaler

Anything else?

It looks like this should be easily fixed by adding in the project Id to the filter here:
https://github.com/kedacore/keda/blob/main/pkg/scalers/gcp_pubsub_scaler.go#L195

Since we have this info already we should be able to extend the filter to include resource.labels.project_id while preserving current functionality. It looks redundant in most cases but fixes this issue we're seeing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant