Using Hashicorp Vault secrets with TriggerAuthentication results in unable to convert Vault Data value error #2645

chaunceyt · 2022-02-15T19:09:31Z

Report

Configuring a TriggerAuthentication object to use hashiCorpVault to get the values for various parameters on a ScaledObject. The scale_resolvers returns unable to convert Vault Data value

Expected Behavior

The result of the queryKey to be used for the new-relic scaler

Actual Behavior

The following error related to the new-relic scaler

keda-operator-ddd8757f-9bnnv keda-operator 1.6449390228145654e+09	ERROR	scalehandler	Error trying to convert Data secret vaule	{"type": "ScaledObject", "namespace": "keda-test", "name": "newrelic-ta-scaledobject", "error": "unable to convert Vault Data value"}

Steps to Reproduce the Problem

Create kind cluster kind create cluster --name keda-test
kubectl create namespace keda
helm install keda kedacore/keda --namespace keda
Setup vault server: vault server -dev -dev-root-token-id="root" -dev-listen-address=0.0.0.0:8200 >> /dev/null &
export VAULT_ADDR='http://0.0.0.0:8200'
vault login root
export VAULT_SA_NAME=$(kubectl get sa keda-operator -n keda --output jsonpath="{.secrets[*]['name']}")
export SA_JWT_TOKEN=$(kubectl get secret -n keda $VAULT_SA_NAME --output 'go-template={{ .data.token }}' | base64 --decode)
export SA_CA_CRT=$(kubectl config view --raw --minify --flatten --output 'jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)
export K8S_HOST=$(kubectl config view --raw --minify --flatten --output 'jsonpath={.clusters[].cluster.server}')
vault auth enable kubernetes
Write vault config

vault write auth/kubernetes/config \
        token_reviewer_jwt="$SA_JWT_TOKEN" \
        kubernetes_host="$K8S_HOST" \
        kubernetes_ca_cert="$SA_CA_CRT" \
        issuer="https://kubernetes.default.svc.cluster.local"

vault write auth/kubernetes/role/keda \
        bound_service_account_names=keda-operator \
        bound_service_account_namespaces=keda \
        policies=keda \
        ttl=24h

Create the TriggerAuthentication object triggerauthentication.yaml

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-trigger-auth-vault
spec:
  hashiCorpVault:
    address: http://192.168.0.4:8200
    authentication: kubernetes
    role: keda
    mount: kubernetes
    credential:
      serviceAccount: /var/run/secrets/kubernetes.io/serviceaccount/token
    secrets:
    - parameter: queryKey
      key: keda-nr-key
      path: /kv-v1/keda/secret

Create ScaledObject scaledobject.yaml

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: newrelic-ta-scaledobject
spec:
  scaleTargetRef:
    name: deployment-nr-ta
  minReplicaCount: 1
  maxReplicaCount: 50
  cooldownPeriod: 5
  idleReplicaCount: 0
  triggers:
  - type: new-relic
    metadata:
      account: '1234567'
      region: "US"
      noDataError: "true"
      nrql: "SELECT latest(allocatablePods) from K8sNodeSample WHERE clusterName = 'cluster-name'"
      threshold: '10'
    authenticationRef:
     name: keda-trigger-auth-vault

Create deployment object deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-nr-ta
spec:
  replicas: 0
  selector:
    matchLabels:
      app: deployment-nr-ta
  template:
    metadata:
      labels:
        app: deployment-nr-ta
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            name: deployment-nr-ta
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            name: deployment-nr-ta
      terminationGracePeriodSeconds: 0
      containers:
        - name: pause-deployment-nr-ta
          image: public.ecr.aws/eks-distro/kubernetes/pause:3.2
          resources:
            requests:
              cpu: 250m
              memory: 250m

Create namespace kubectl create ns keda-test
Apply deployment kubectl apply -f deployment.yaml -n keda-test
Apply TriggerAuthentication object kubectl apply -f triggerauthentication.yaml -n keda-test
Apply ScaledObject kubectl apply -f scaledobject.yaml -n keda-test
Review the logs using stern or kubectl logs

Logs from KEDA operator

keda-operator-ddd8757f-9bnnv keda-operator 1.6449390227989428e+09	INFO	controller.scaledobject	Creating a new HPA	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "newrelic-ta-scaledobject", "namespace": "keda-test", "HPA.Namespace": "keda-test", "HPA.Name": "keda-hpa-newrelic-ta-scaledobject"}
keda-operator-ddd8757f-9bnnv keda-operator 1.6449390228145654e+09	ERROR	scalehandler	Error trying to convert Data secret vaule	{"type": "ScaledObject", "namespace": "keda-test", "name": "newrelic-ta-scaledobject", "error": "unable to convert Vault Data value"}

KEDA Version

2.6.0

Kubernetes Version

v1.21.1

Platform

Other

Scaler Details

new-relic and rabbitmq

Anything else?

Create Vault v1 secret vault secrets enable -path="kv-v1" -description="Test V1" kv
Added NR key vault kv put kv-v1/keda/secret keda-nr-key=NRAK-12345678901234
Also tested using a v2 and got same error.

The text was updated successfully, but these errors were encountered:

zroubalik · 2022-02-15T19:28:46Z

I am not Haschicorp Vault expert, but seems like it hasn't been able to properly parse secret.Data:

keda/pkg/scaling/resolver/scale_resolvers.go

Lines 450 to 454 in 6e8e195

    
           if v2Data, ok := data["data"].(map[string]interface{}); ok { 
        
           	if value, ok := v2Data[key]; ok { 
        
           		if s, ok := value.(string); ok { 
        
           			return s 
        
           		}

zroubalik · 2022-02-15T19:36:35Z

@chapurlatn @nisan270390 would you mind looking at this? Thanks

chaunceyt · 2022-02-16T13:59:07Z

Some additional debugging.

Added this above pkg/scaling/resolver/scale_resolvers.go#L205

	fmt.Printf("Secret: %v\n", secret)
	fmt.Printf("Secret.Data: %v\n", secret.Data)
	logger.Error(fmt.Errorf("type: %T data: %v", secret.Data, secret.Data), "Error trying to convert Data secret vaule")

Results in log output.

keda-operator-6f5f99d65d-wrfqj keda-operator 2022-02-16T08:43:49.405594182-05:00 1.645019029405501e+09	INFO	controller.scaledobject	Creating a new HPA	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "newrelic-ta-scaledobject", "namespace": "keda-test", "HPA.Namespace": "keda-test", "HPA.Name": "keda-hpa-newrelic-ta-scaledobject"}
keda-operator-6f5f99d65d-wrfqj keda-operator 2022-02-16T08:43:49.425098794-05:00 Secret: &{de787917-da41-557e-69ff-c4fd0a088d13  2764800 false map[keda-nr-key:NRAK-12345678901234] [] <nil> <nil>}
keda-operator-6f5f99d65d-wrfqj keda-operator 2022-02-16T08:43:49.425329399-05:00 Secret.Data: map[keda-nr-key:NRAK-12345678901234]
keda-operator-6f5f99d65d-wrfqj keda-operator 2022-02-16T08:43:49.425101094-05:00 1.64501902942483e+09	ERROR	scalehandler	Error trying to convert Data secret vaule	{"type": "ScaledObject", "namespace": "keda-test", "name": "newrelic-ta-scaledobject", "error": "type: map[string]interface {} data: map[keda-nr-key:NRAK-12345678901234]"}
keda-operator-6f5f99d65d-wrfqj keda-operator 2022-02-16T08:43:49.425391368-05:00 github.com/kedacore/keda/v2/pkg/scaling/resolver.ResolveAuthRefAndPodIdentity
keda-operator-6f5f99d65d-wrfqj keda-operator 2022-02-16T08:43:49.425394336-05:00 	/workspace/pkg/scaling/resolver/scale_resolvers.go:135

Added debug code above this line pkg/scaling/resolver/scale_resolvers.go#L450

Results in log output.

keda-operator-6f5f99d65d-wrfqj keda-operator 2022-02-16T08:43:49.425552463-05:00 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
keda-operator-6f5f99d65d-wrfqj keda-operator 2022-02-16T08:43:49.425579420-05:00 1.645019029425019e+09	ERROR	scalehandler	Error trying to convert Data secret vaule	{"type": "ScaledObject", "namespace": "keda-test", "name": "newrelic-ta-scaledobject", "error": "type: map[string]interface {} data: map[keda-nr-key:NRAK-12345678901234]"}
keda-operator-6f5f99d65d-wrfqj keda-operator 2022-02-16T08:43:49.425586058-05:00 github.com/kedacore/keda/v2/pkg/scaling/resolver.resolveAuthRef

Thanks.

zroubalik · 2022-02-16T17:42:08Z

Hm for some reason the Secret.Data is map[keda-nr-key:NRAK-12345678901234] while the code expects map[data:A_NEW_MAP_WITH_KEY]

keda/pkg/scaling/resolver/scale_resolvers.go

Line 450 in 18428b2

if v2Data, ok := data["data"].(map[string]interface{}); ok {

The data key is missing there, it is missing additional map[string]interface inside the secret.Data 🤷‍♂️

zroubalik · 2022-02-16T17:51:17Z

We bumped github.com/hashicorp/vault/api v1.3.0 -> v1.3.1 in the last release, but I don't think that it caused some changes in the way how vault secrets are being resolved here:

keda/pkg/scaling/resolver/hashicorpvault_handler.go

Line 170 in 18428b2

return vh.client.Logical().Read(path)

By chance could you please try some older KEDA versions (2.5/2.4) so we can be sure that it is not a regression ?

chaunceyt · 2022-02-16T22:00:30Z

Test results Got the same error using the rabbitmq scaler

2.5.0

keda-operator-69cd6c66bd-g6gd8 keda-operator 2022-02-16T16:46:32.978589823-05:00 2022-02-16T21:46:32.978Z	INFO	controller.scaledobject	Initializing Scaling logic according to ScaledObject Specification	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "prometheus-scaledobject", "namespace": "keda-demo"}
keda-operator-69cd6c66bd-g6gd8 keda-operator 2022-02-16T16:46:32.988624874-05:00 2022-02-16T21:46:32.988Z	ERROR	scalehandler	Error trying to convert Data secret vaule	{"type": "ScaledObject", "namespace": "keda-test", "name": "my-application", "error": "unable to convert Vault Data value"}

2.4.0

keda-operator-7df5bb89cc-4qmdp keda-operator 2022-02-16T16:58:22.048777722-05:00 2022-02-16T21:58:22.048Z	INFO	controllers.ScaledObject	Reconciling ScaledObject	{"ScaledObject.Namespace": "keda-test", "ScaledObject.Name": "my-application"}
keda-operator-7df5bb89cc-4qmdp keda-operator 2022-02-16T16:58:22.069202014-05:00 2022-02-16T21:58:22.068Z	ERROR	scalehandler	Error trying to convert Data secret vaule	{"type": "ScaledObject", "namespace": "keda-test", "name": "my-application", "error": "unable to convert Vault Data value"}
keda-operator-7df5bb89cc-4qmdp keda-operator 2022-02-16T16:58:22.069231515-05:00 github.com/kedacore/keda/v2/pkg/scaling/resolver.resolveAuthRef

chaunceyt · 2022-02-16T22:38:52Z

After some additional investigation.

It seems this code here expects one to have created the vault secret using the vault go sdk.

Using this code the secret added using vault kv put kv-v1/keda/test keda-nr-key=NRAK-12345678901234 failed to be resolved.

package main

import (
	"fmt"
	"net/http"
	"time"

	"github.com/hashicorp/vault/api"
)

var httpClient = &http.Client{
	Timeout: 10 * time.Second,
}

func main() {
	token := "root"
	vaultAddr := "http://192.168.0.9:8200"

	client, err := api.NewClient(&api.Config{Address: vaultAddr, HttpClient: httpClient})
	if err != nil {
		panic(err)
	}
	client.SetToken(token)

	//writing the data
	inputData := map[string]interface{}{
		"data": map[string]interface{}{
			"keda-nr-key": "NRAK-12345678901234",
		},
	}
	output, err := client.Logical().Write("kv-v1/keda/newrelic", inputData)
	fmt.Println(output)
	if err != nil {
		panic(err)
	}

	data, err := client.Logical().Read("kv-v1/keda/newrelic")
	key := "keda-nr-key"
	if err != nil {
		panic(err)
	}

	value := resolveVaultSecret(data.Data, key)
	fmt.Println(value)

	// vault kv put kv-v1/keda/test keda-nr-key=NRAK-12345678901234
	dataTest, dataTestErr := client.Logical().Read("kv-v1/keda/test")
	dataTestKey := "keda-nr-key"
	if dataTestErr != nil {
		panic(dataTestErr)
	}

	fmt.Println("data.Data", dataTest.Data)

	fmt.Printf("Type: %T values: %v\n", dataTest.Data, dataTest.Data)
	dataTestValue := resolveVaultSecret(dataTest.Data, dataTestKey)
	fmt.Println(dataTestValue)

}

func resolveVaultSecret(data map[string]interface{}, key string) string {
	if v2Data, ok := data["data"].(map[string]interface{}); ok {
		if value, ok := v2Data[key]; ok {
			if s, ok := value.(string); ok {
				return s
			}
		} else {
			fmt.Println(fmt.Errorf("key '%s' not found", key), "Error trying to get key from Vault secret")
			return ""
		}
	}

	fmt.Println(fmt.Errorf("unable to convert Vault Data value"), "Error trying to convert Data secret vaule")
	return ""
}

Output executing the above code

 $ go run main.go
<nil>
NRAK-12345678901234
data.Data map[keda-nr-key:NRAK-12345678901234]
Type: map[string]interface {} values: map[keda-nr-key:NRAK-12345678901234]
unable to convert Vault Data value Error trying to convert Data secret vaule

chaunceyt · 2022-02-16T23:15:35Z

Sorry left out what the data looks like in vault

vault kv get kv-v1/keda/newrelic
==== Data ====
Key     Value
---     -----
data    map[keda-nr-key:NRAK-12345678901234]

Using vault cli

 vault kv get kv-v1/keda/test
======= Data =======
Key            Value
---            -----
keda-nr-key    NRAK-12345678901234

zroubalik · 2022-02-17T08:33:45Z

@chaunceyt oh that's really awful 🤦‍♂️ Great investigation!

Could you please open a PR with a fix? To support both approaches based on what is in the key & value? To check what is the key and what is the value (whether a map or value) etc.

racdev · 2022-02-17T12:22:01Z

@chaunceyt Isn't that just because when you added the secret via the vault go sdk you explicitly added a "data" wrapper around the actual secret, whereas via the CLI you didn't (and it doesn't). The resolving code is then expecting that "data" wrapper - which was artificially added via the go sdk write.

Wouldn't you have identical results if you had just written the secret (rather than wrap in a "data" map), i.e.

       //writing the data
	inputData := map[string]interface{}{
		"keda-nr-key": "NRAK-12345678901234",
	}
	output, err := client.Logical().Write("kv-v1/keda/newrelic", inputData)
	fmt.Println(output)
	if err != nil {
		panic(err)
	}

And then remove the "data" lookup from the secret resolving code:

func resolveVaultSecret(logger logr.Logger, data map[string]interface{}, key string) string {
		if value, ok := data[key]; ok {
			if s, ok := value.(string); ok {
				return s
			}
		} else {
			logger.Error(fmt.Errorf("key '%s' not found", key), "Error trying to get key from Vault secret")
			return ""
		}
}

Unless I am misreading it all

chaunceyt · 2022-02-17T13:57:30Z

After some additional investigation. This issue is centered around v1 secrets. The current code resolves a v2 secret as expected.

I was able to get v1 working using this patch.

diff --git a/pkg/scaling/resolver/scale_resolvers.go b/pkg/scaling/resolver/scale_resolvers.go
index c30a32c..c504ad1 100644
--- a/pkg/scaling/resolver/scale_resolvers.go
+++ b/pkg/scaling/resolver/scale_resolvers.go
@@ -456,8 +456,12 @@ func resolveVaultSecret(logger logr.Logger, data map[string]interface{}, key str
                        logger.Error(fmt.Errorf("key '%s' not found", key), "Error trying to get key from Vault secret")
                        return ""
                }
+       } else if vData, ok := data[key]; ok {
+               if s, ok := vData.(string); ok {
+                       return s
+               }
        }

-       logger.Error(fmt.Errorf("unable to convert Vault Data value"), "Error trying to convert Data secret vaule")
+       logger.Error(fmt.Errorf("unable to convert Vault Data value"), "Error trying to convert Data secret value")
        return ""
 }

zroubalik · 2022-02-17T15:00:46Z

I am glad it has been resolved.
@chaunceyt would you mind opening an PR with the fix to support v1?

zroubalik · 2022-03-04T19:22:13Z

@chaunceyt any update on the PR?

chaunceyt · 2022-03-11T01:19:15Z

@zroubalik Sorry for the delay got busy on a different project, I should have the PR in a couple of days.

Signed-off-by: Chauncey Thorn <[email protected]>

stale · 2022-05-10T01:26:10Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

stale · 2022-07-09T17:19:31Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

stale · 2022-07-16T17:42:52Z

This issue has been automatically closed due to inactivity.

srinivascnetric · 2023-02-15T12:26:26Z

Hi Guys , I have done the changes in the keda/pkg/scaling/resolver/scale_resolvers.go

} else if v1Data, ok := data[key]; ok {
	if s, ok := v1Data.(string); ok {
		return s
	}
}

But still no luck . I am using enterprise vault with kv1 . Could some help here ?

Error:

1.6764615272404535e+09 INFO controller.scaledobject Creating a new HPA {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "kafka-scaledobject", "namespace": "carbon-health-ns", "HPA.Namespace": "carbon-health-ns", "HPA.Name": "keda-hpa-kafka-scaledobject"}
1.6764615286955714e+09 ERROR scalehandler Error trying to convert Data secret vaule {"type": "ScaledObject", "namespace": "carbon-health-ns", "name": "kafka-scaledobject", "error": "unable to convert Vault Data value"}
github.com/kedacore/keda/v2/pkg/scaling/resolver.resolveAuthRef
/workspace/pkg/scaling/resolver/scale_resolvers.go:205
github.com/kedacore/keda/v2/pkg/scaling/resolver.ResolveAuthRefAndPodIdentity
/workspace/pkg/scaling/resolver/scale_resolvers.go:135
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers.func1
/workspace/pkg/scaling/scale_handler.go:318
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
/workspace/pkg/scaling/scale_handler.go:326
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
/workspace/pkg/scaling/scale_handler.go:194
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
/workspace/controllers/keda/hpa.go:163
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
/workspace/controllers/keda/hpa.go:63
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).createAndDeployNewHPA
/workspace/controllers/keda/hpa.go:46
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
/workspace/controllers/keda/scaledobject_controller.go:361
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).r

chaunceyt added the bug Something isn't working label Feb 15, 2022

chaunceyt added a commit to chaunceyt/keda that referenced this issue Mar 17, 2022

fix: Added support for vault kv-v1 (kedacore#2645)

78284e1

Signed-off-by: Chauncey Thorn <[email protected]>

chaunceyt mentioned this issue Mar 17, 2022

fix: Added support for vault kv-v1 (#2645) #2772

Closed

5 tasks

stale bot added the stale All issues that are marked as stale due to inactivity label May 10, 2022

JorTurFer removed the stale All issues that are marked as stale due to inactivity label May 10, 2022

stale bot added the stale All issues that are marked as stale due to inactivity label Jul 9, 2022

stale bot closed this as completed Jul 16, 2022

zroubalik removed the stale All issues that are marked as stale due to inactivity label Jul 18, 2022

zroubalik reopened this Jul 18, 2022

JorTurFer added the stale-bot-ignore All issues that should not be automatically closed by our stale bot label Jul 29, 2022

zroubalik closed this as completed Aug 3, 2022

dangerousplay mentioned this issue Jan 21, 2023

Add support for Hashicorp Vault KV engine V1 #4152

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Using Hashicorp Vault secrets with TriggerAuthentication results in unable to convert Vault Data value error #2645

Using Hashicorp Vault secrets with TriggerAuthentication results in unable to convert Vault Data value error #2645

chaunceyt commented Feb 15, 2022 •

edited

Loading

zroubalik commented Feb 15, 2022 •

edited

Loading

zroubalik commented Feb 15, 2022

chaunceyt commented Feb 16, 2022 •

edited

Loading

zroubalik commented Feb 16, 2022 •

edited

Loading

zroubalik commented Feb 16, 2022 •

edited

Loading

chaunceyt commented Feb 16, 2022

chaunceyt commented Feb 16, 2022

chaunceyt commented Feb 16, 2022

zroubalik commented Feb 17, 2022 •

edited

Loading

racdev commented Feb 17, 2022

chaunceyt commented Feb 17, 2022

zroubalik commented Feb 17, 2022

zroubalik commented Mar 4, 2022

chaunceyt commented Mar 11, 2022

stale bot commented May 10, 2022

stale bot commented Jul 9, 2022

stale bot commented Jul 16, 2022

srinivascnetric commented Feb 15, 2023 •

edited

Loading

Using Hashicorp Vault secrets with TriggerAuthentication results in unable to convert Vault Data value error #2645

Using Hashicorp Vault secrets with TriggerAuthentication results in unable to convert Vault Data value error #2645

Comments

chaunceyt commented Feb 15, 2022 • edited Loading

Report

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Logs from KEDA operator

KEDA Version

Kubernetes Version

Platform

Scaler Details

Anything else?

zroubalik commented Feb 15, 2022 • edited Loading

zroubalik commented Feb 15, 2022

chaunceyt commented Feb 16, 2022 • edited Loading

zroubalik commented Feb 16, 2022 • edited Loading

zroubalik commented Feb 16, 2022 • edited Loading

chaunceyt commented Feb 16, 2022

chaunceyt commented Feb 16, 2022

chaunceyt commented Feb 16, 2022

zroubalik commented Feb 17, 2022 • edited Loading

racdev commented Feb 17, 2022

chaunceyt commented Feb 17, 2022

zroubalik commented Feb 17, 2022

zroubalik commented Mar 4, 2022

chaunceyt commented Mar 11, 2022

stale bot commented May 10, 2022

stale bot commented Jul 9, 2022

stale bot commented Jul 16, 2022

srinivascnetric commented Feb 15, 2023 • edited Loading

chaunceyt commented Feb 15, 2022 •

edited

Loading

zroubalik commented Feb 15, 2022 •

edited

Loading

chaunceyt commented Feb 16, 2022 •

edited

Loading

zroubalik commented Feb 16, 2022 •

edited

Loading

zroubalik commented Feb 16, 2022 •

edited

Loading

zroubalik commented Feb 17, 2022 •

edited

Loading

srinivascnetric commented Feb 15, 2023 •

edited

Loading