Add a scaler based on number of objects in a GCP Storage bucket #2648

RamCohen · 2022-02-16T11:08:38Z

Add a scaler based on number of objects in a GCP Storage bucket

Fixes #2628

Signed-off-by: Ram Cohen [email protected]

tomkerkhove · 2022-02-16T12:48:43Z

Would you mind opening a PR for the docs as well please?

pkg/scalers/gcp_common.go

pkg/scalers/gcp_storage_scaler.go

zroubalik · 2022-02-16T13:41:20Z

/run-e2e gcp*
Update: You can check the progres here

zroubalik · 2022-02-16T13:44:59Z

@RamCohen That e2e with fake GCS is neat, though we have just get GCP account for KEDA related testing, so it might be great to rewrite the tests and point them against actual GCP storage? WDYT?

RamCohen · 2022-02-16T14:39:58Z

Where can I see the credentials to use for the GCP account ?

zroubalik · 2022-02-16T16:54:49Z

Where can I see the credentials to use for the GCP account ?

We usually have the credentials stored as GH Secrets and exposed via ENV variables, so the apporach would be to modify the test to expect those env variables. For example you can take a look on this PR for an inspiration: #2643

We haven't added the GCP credentials to GH yet, @tomkerkhove could you please assist with that? Thanks!

tomkerkhove · 2022-02-17T12:56:41Z

I haven't used GCP before but if there are any docs on what I should follow then I can set it up.

RamCohen · 2022-02-20T15:03:38Z

You need to create a service account in the 'IAM & Admin' panel, add a key to it by generating a new key and save the downloaded json file as the secret
You'll also need to add roles to that service account depending on what it should be allowed to do.
You can consider using separate accounts per scaler in order to have each one with just the roles needed for that scaler particular test

RamCohen · 2022-02-20T15:30:18Z

See also instructions here: https://github.com/kedacore/sample-go-gcppubsub#readme

tomkerkhove · 2022-02-22T07:06:30Z

You need to create a service account in the 'IAM & Admin' panel, add a key to it by generating a new key and save the downloaded json file as the secret You'll also need to add roles to that service account depending on what it should be allowed to do. You can consider using separate accounts per scaler in order to have each one with just the roles needed for that scaler particular test

Sure, but what permissions do you need? :) Is Monitoring Viewer enough?

Relates to #2648 Signed-off-by: Tom Kerkhove <[email protected]>

tomkerkhove · 2022-02-22T07:12:21Z

Auth is added through #2662, but I did not add any other resources.

What would you need in GCP from us?

RamCohen · 2022-02-22T07:39:28Z

For pubsub e2e I need pubsub admin as I create and delete a topic on each test I we'll use the same topic which will be pre-created, pubsub editor should be enough The topic will need to have a message retention policy of 1 day to automatically clean up For GCS I need Storage admin as I create and delete a bucket each time If the test will use the same bucket we can have object admin permissions instead, but we'll need to make sure the bucket is empty before each test

…

On Tue, Feb 22, 2022 at 9:12 AM Tom Kerkhove ***@***.***> wrote: Auth is added through #2662 <#2662>, but I did not add any other resources. What would you need in GCP from us? — Reply to this email directly, view it on GitHub <#2648 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AATJ7GUAENPIV74F6XVEJ6TU4MZOBANCNFSM5ORIVOZA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

zroubalik · 2022-02-28T09:03:09Z

For pubsub e2e I need pubsub admin as I create and delete a topic on each test I we'll use the same topic which will be pre-created, pubsub editor should be enough The topic will need to have a message retention policy of 1 day to automatically clean up For GCS I need Storage admin as I create and delete a bucket each time If the test will use the same bucket we can have object admin permissions instead, but we'll need to make sure the bucket is empty before each test
…
On Tue, Feb 22, 2022 at 9:12 AM Tom Kerkhove @.> wrote: Auth is added through #2662 <#2662>, but I did not add any other resources. What would you need in GCP from us? — Reply to this email directly, view it on GitHub <#2648 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATJ7GUAENPIV74F6XVEJ6TU4MZOBANCNFSM5ORIVOZA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.Message ID: @.>

Yeah, we should create all needed resources directly from the test via API. We should just point credentials and all should be done, @tomkerkhove we should add admin rights to the credentials.

pkg/scalers/gcp_storage_scaler.go

zroubalik · 2022-02-28T18:27:45Z

Also, could you please update Changelog?

RamCohen · 2022-03-01T10:37:16Z

I also need the GCP project ID

zroubalik · 2022-03-01T10:39:01Z

Could you please rebase this PR to contain only relevant changes?

zroubalik · 2022-03-01T10:40:55Z

I also need the GCP project ID

Do you need anything else than this? https://github.com/kedacore/keda/pull/2662/files

RamCohen · 2022-03-01T10:46:12Z

Yes, the GCP project ID

tomkerkhove · 2022-03-01T15:13:39Z

Yes, the GCP project ID

That's cncf-keda-testing

zroubalik · 2022-03-01T15:24:34Z

Yes, the GCP project ID

That's cncf-keda-testing

@tomkerkhove would be better to store this in GitHub as ENV as well.

RamCohen · 2022-03-01T15:29:29Z

The project id value can be extracted from the gcp key: JSON.parse(process.env['GCP_SP_KEY']).project_id

…

On Tue, Mar 1, 2022 at 5:24 PM Zbynek Roubalik ***@***.***> wrote: Yes, the GCP project ID That's cncf-keda-testing @tomkerkhove <https://github.com/tomkerkhove> would be better to store this in GitHub as ENV as well. — Reply to this email directly, view it on GitHub <#2648 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AATJ7GUSF252BTZZQOL3JDDU5YZDZANCNFSM5ORIVOZA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

zroubalik · 2022-03-01T15:32:07Z

The project id value can be extracted from the gcp key: JSON.parse(process.env['GCP_SP_KEY']).project_id

@RamCohen cool, so are you all set with this PR? If so, could you please resolve the conflict and we can run the test and do final reivew and merge. Thanks!

tomkerkhove · 2022-03-01T16:29:54Z

Yes, the GCP project ID

That's cncf-keda-testing

@tomkerkhove Tom Kerkhove FTE would be better to store this in GitHub as ENV as well.

Yes, but not as a secret - No? It can be just a variable in the workflow.

Fixes kedacore#2628 Signed-off-by: Ram Cohen <[email protected]>

Signed-off-by: Ram Cohen <[email protected]>

Also remove now unnecessary 'endpoint' property as we are now using real GCS Signed-off-by: Ram Cohen <[email protected]>

Signed-off-by: Ram Cohen <[email protected]>

zroubalik · 2022-03-03T08:03:18Z

/run-e2e gcp-storage.test.*
Update: You can check the progres here

zroubalik · 2022-03-03T08:58:43Z

/run-e2e gcp-storage.test.* Update: You can check the progres here

The test fail with:

 ✔ creating the gcp-sdk pod should work.. (570ms)
error: unable to upgrade connection: container not found ("gcp-sdk-container")
error: unable to upgrade connection: container not found ("gcp-sdk-container")
Creating gs://keda-test-storage-bucket/...
You are attempting to perform an operation that requires a project id, with none configured. Please re-run gsutil config and make sure to follow the instructions for finding and entering your default project id.
command terminated with exit code ***
  ✖ initializing the gcp-sdk pod should work.. Setting GCP authentication on gcp-sdk should work..
BucketNotFoundException: 404 gs://keda-test-storage-bucket bucket does not exist.
command terminated with exit code ***
BucketNotFoundException: 404 gs://keda-test-storage-bucket bucket does not exist.

Signed-off-by: Ram Cohen <[email protected]>

JorTurFer

LGTM!
Thanks ❤️ ❤️ ❤️
Only 1 small nit: Could you update the changelog to

- **General:** Introduce new GCP Storage Scaler ([#2628](https://github.com/kedacore/keda/issues/2628))

zroubalik · 2022-03-04T08:18:38Z

/run-e2e gcp-storage.test.*
Update: You can check the progres here

Signed-off-by: Ram Cohen <[email protected]>

RamCohen · 2022-03-05T14:28:43Z

/run-e2e gcp-storage.test.* Update: You can check the progres here

Test has error:
Creating gs://keda-test-storage-bucket/...
AccessDeniedException: 403 The billing account for the owning project is disabled in state absent

JorTurFer · 2022-03-11T23:26:47Z

/run-e2e gcp-storage.test.*
Update: You can check the progres here

JorTurFer · 2022-03-12T08:41:47Z

It seems that the problem was also the credentials

Signed-off-by: Ram Cohen <[email protected]>

JorTurFer · 2022-03-12T11:15:13Z

/run-e2e gcp-storage.test.*
Update: You can check the progres here

Signed-off-by: Ram Cohen [email protected]

zroubalik · 2022-03-14T06:59:46Z

/run-e2e gcp-storage.test.*
Update: You can check the progres here

zroubalik

LGTM, thanks a lot @RamCohen!

…core#2648) Signed-off-by: Ram Cohen <[email protected]> Co-authored-by: Zbynek Roubalik <[email protected]> Signed-off-by: Ram Cohen <[email protected]>

RamCohen requested a review from a team as a code owner February 16, 2022 11:08

zroubalik reviewed Feb 16, 2022

View reviewed changes

pkg/scalers/gcp_common.go Show resolved Hide resolved

pkg/scalers/gcp_storage_scaler.go Outdated Show resolved Hide resolved

pkg/scalers/gcp_storage_scaler.go Outdated Show resolved Hide resolved

pkg/scalers/gcp_storage_scaler.go Show resolved Hide resolved

RamCohen mentioned this pull request Feb 20, 2022

Provide e2e test for all scalers and secret providers (Help Wanted) #2357

Open

35 tasks

tomkerkhove added a commit that referenced this pull request Feb 22, 2022

feat: Provide e2e variables for GCP auth

edee249

Relates to #2648 Signed-off-by: Tom Kerkhove <[email protected]>

tomkerkhove added a commit that referenced this pull request Feb 22, 2022

feat: Provide e2e variables for GCP auth

a022e91

Relates to #2648 Signed-off-by: Tom Kerkhove <[email protected]>

tomkerkhove added a commit that referenced this pull request Feb 22, 2022

feat: Provide e2e variables for GCP auth

114f05f

Relates to #2648 Signed-off-by: Tom Kerkhove <[email protected]>

zroubalik requested changes Feb 28, 2022

View reviewed changes

pkg/scalers/gcp_storage_scaler.go Outdated Show resolved Hide resolved

pkg/scalers/gcp_storage_scaler.go Outdated Show resolved Hide resolved

RamCohen force-pushed the gcs branch from d4516bd to 1d44417 Compare March 1, 2022 10:48

RamCohen added 12 commits March 3, 2022 10:02

Add a scaler based on number of GCP Storage bucket objects

3ace51f

Fixes kedacore#2628 Signed-off-by: Ram Cohen <[email protected]>

Added e2e test to gcp_storage scaler

241830f

Signed-off-by: Ram Cohen <[email protected]>

Fix static styling issues

b2617ff

Signed-off-by: Ram Cohen <[email protected]>

Update gcs-storage scaler test

1b0b636

Signed-off-by: Ram Cohen <[email protected]>

Review fixes

0c9d868

Signed-off-by: Ram Cohen <[email protected]>

Update google storage related packages

33618e1

Signed-off-by: Ram Cohen <[email protected]>

Precalculate metric name

de42e6a

Signed-off-by: Ram Cohen <[email protected]>

Updated changelog

46057da

Signed-off-by: Ram Cohen <[email protected]>

Remove option to specify an alternate metric name

ce54e9d

Signed-off-by: Ram Cohen <[email protected]>

Make e2e test work with the test GCP project

681f16c

Also remove now unnecessary 'endpoint' property as we are now using real GCS Signed-off-by: Ram Cohen <[email protected]>

Use methods from helpers utilities

f93b5cd

Signed-off-by: Ram Cohen <[email protected]>

Fix GCP authentication in test

8e434b2

Signed-off-by: Ram Cohen <[email protected]>

RamCohen force-pushed the gcs branch from 1436acc to 8e434b2 Compare March 3, 2022 08:02

Wait for gcp-sdk container to be ready in test

7ee6ab8

Signed-off-by: Ram Cohen <[email protected]>

JorTurFer approved these changes Mar 4, 2022

View reviewed changes

Update changelog

67fe599

Signed-off-by: Ram Cohen <[email protected]>

Merge remote-tracking branch 'origin' into gcs

8733dec

Signed-off-by: Ram Cohen <[email protected]>

Merge remote-tracking branch 'origin' into gcs

9d8addc

Signed-off-by: Ram Cohen [email protected]

Merge branch 'main' into gcs

6acdef5

zroubalik approved these changes Mar 14, 2022

View reviewed changes

zroubalik merged commit b8913a1 into kedacore:main Mar 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a scaler based on number of objects in a GCP Storage bucket #2648

Add a scaler based on number of objects in a GCP Storage bucket #2648

RamCohen commented Feb 16, 2022 •

edited by zroubalik

Loading

tomkerkhove commented Feb 16, 2022

zroubalik commented Feb 16, 2022 •

edited by github-actions bot

Loading

zroubalik commented Feb 16, 2022

RamCohen commented Feb 16, 2022

zroubalik commented Feb 16, 2022 •

edited

Loading

tomkerkhove commented Feb 17, 2022

RamCohen commented Feb 20, 2022

RamCohen commented Feb 20, 2022

tomkerkhove commented Feb 22, 2022

tomkerkhove commented Feb 22, 2022

RamCohen commented Feb 22, 2022 via email

zroubalik commented Feb 28, 2022

zroubalik commented Feb 28, 2022

RamCohen commented Mar 1, 2022

zroubalik commented Mar 1, 2022

zroubalik commented Mar 1, 2022

RamCohen commented Mar 1, 2022

tomkerkhove commented Mar 1, 2022

zroubalik commented Mar 1, 2022

RamCohen commented Mar 1, 2022 via email

zroubalik commented Mar 1, 2022 •

edited

Loading

tomkerkhove commented Mar 1, 2022

zroubalik commented Mar 3, 2022 •

edited by github-actions bot

Loading

zroubalik commented Mar 3, 2022

JorTurFer left a comment •

edited

Loading

zroubalik commented Mar 4, 2022 •

edited by github-actions bot

Loading

RamCohen commented Mar 5, 2022

JorTurFer commented Mar 11, 2022 •

edited by github-actions bot

Loading

JorTurFer commented Mar 12, 2022 •

edited

Loading

JorTurFer commented Mar 12, 2022 •

edited by github-actions bot

Loading

zroubalik commented Mar 14, 2022 •

edited by github-actions bot

Loading

zroubalik left a comment

Add a scaler based on number of objects in a GCP Storage bucket #2648

Add a scaler based on number of objects in a GCP Storage bucket #2648

Conversation

RamCohen commented Feb 16, 2022 • edited by zroubalik Loading

tomkerkhove commented Feb 16, 2022

zroubalik commented Feb 16, 2022 • edited by github-actions bot Loading

zroubalik commented Feb 16, 2022

RamCohen commented Feb 16, 2022

zroubalik commented Feb 16, 2022 • edited Loading

tomkerkhove commented Feb 17, 2022

RamCohen commented Feb 20, 2022

RamCohen commented Feb 20, 2022

tomkerkhove commented Feb 22, 2022

tomkerkhove commented Feb 22, 2022

RamCohen commented Feb 22, 2022 via email

zroubalik commented Feb 28, 2022

zroubalik commented Feb 28, 2022

RamCohen commented Mar 1, 2022

zroubalik commented Mar 1, 2022

zroubalik commented Mar 1, 2022

RamCohen commented Mar 1, 2022

tomkerkhove commented Mar 1, 2022

zroubalik commented Mar 1, 2022

RamCohen commented Mar 1, 2022 via email

zroubalik commented Mar 1, 2022 • edited Loading

tomkerkhove commented Mar 1, 2022

zroubalik commented Mar 3, 2022 • edited by github-actions bot Loading

zroubalik commented Mar 3, 2022

JorTurFer left a comment • edited Loading

Choose a reason for hiding this comment

zroubalik commented Mar 4, 2022 • edited by github-actions bot Loading

RamCohen commented Mar 5, 2022

JorTurFer commented Mar 11, 2022 • edited by github-actions bot Loading

JorTurFer commented Mar 12, 2022 • edited Loading

JorTurFer commented Mar 12, 2022 • edited by github-actions bot Loading

zroubalik commented Mar 14, 2022 • edited by github-actions bot Loading

zroubalik left a comment

Choose a reason for hiding this comment

RamCohen commented Feb 16, 2022 •

edited by zroubalik

Loading

zroubalik commented Feb 16, 2022 •

edited by github-actions bot

Loading

zroubalik commented Feb 16, 2022 •

edited

Loading

zroubalik commented Mar 1, 2022 •

edited

Loading

zroubalik commented Mar 3, 2022 •

edited by github-actions bot

Loading

JorTurFer left a comment •

edited

Loading

zroubalik commented Mar 4, 2022 •

edited by github-actions bot

Loading

JorTurFer commented Mar 11, 2022 •

edited by github-actions bot

Loading

JorTurFer commented Mar 12, 2022 •

edited

Loading

JorTurFer commented Mar 12, 2022 •

edited by github-actions bot

Loading

zroubalik commented Mar 14, 2022 •

edited by github-actions bot

Loading