Can my trigger (Scaler) warn me that my metadata is wrong?

[robin-wayve]

I'm a first time KEDA user, and was trying to configure a basic Scaling Job to consume one message at a time from an Azure Service Bus queue.

After reading Scaling Jobs I wound up configuring my triggers[].metadata wrong after I saw this (I guess I confused Azure Storage Queue with Service Bus):

Target Average Value: The number of messages that will be consumed on a job. It is defined on the scaler side. e.g. queueLength on Azure Storage Queue scaler.

I should have read the Azure Service Bus Scaler documentation more closely which says:

messageCount - Amount of active messages in your Azure Service Bus queue or topic to scale on.

Could something have warned me that queueLength was the wrong metadata to be using on this trigger and saved me some confusion?

[tomkerkhove]

Is that something we can do in Go @zroubalik ?

I have this in Promitor where it gives suggestions but not sure about Go.

[Ritikaa96]

@zroubalik @tomkerkhove @robin-wayve
I guess we have a similar approach in knative and k8s too? it might need to have a pre-declared template for each scaler. So that the wrong declaration metadata can be warned about.
It might be relative to this :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      **label**:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

when we apply this declaration it throws error:

error validating data: ValidationError(Deployment.spec.template.metadata): unknown field "label" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta; if you choose to ignore these errors, turn validation off with --validate=false

Going through same approach will be helpful and convenient .

[arschles]

@robin-wayve @tomkerkhove @zroubalik it looks like there are some scalers that do validation manually. for example, the AWS cloudwatch scaler validates some of the trigger metadata. in Go, something like this could be done in a few different ways, including by using reflection (similar to how the JSON/YAML/TOML decoders do it, as well as envconfig) or generating writing validation functions automatically from a schema or struct (which kubebuilder does something similar when it generates CRD definitions).

I would personally do this with reflection, and am happy to submit a proof of concept in a PR if you all would like.

[zroubalik]

@arschles I'd say that all scalers validate manually the input fields (or at least the ones that are expected to be specified there). I think that the original ask is too print a warning/error if user specifies unknown/wrong field.

It is like that because Metadata is map[string]string and I think that the problem to solve is whether we would like to move from that approach to struct{} with a proper fields and what not.

[zroubalik]

@arschles the reflection approach seems interesting though. I we don't want to move the Metadata to struct{} we would anyway need to handle the config parsing a better way, with some unified approach. I don't like the current approach, where the parsing is done for each scaler separately.

[arschles]

@zroubalik there are a few ideas for ways to validate, without moving all the data into structs:

1. build a func ValidateMetadata(mdData map[string]string, validation interface{}) error to validate each scaler metadata. the validation parameter in that function would be a struct with tags to define data types and other rules, etc... - similar to how encoding/json does things. you wouldn't store data in that struct, however, it just would hold rules. for example:

type cloudWatchSchema struct {
    Namespace string `keda:"namespace"`
    MetricName string `keda:"metricName"`
    // so on
    DimensionName string `keda:"dimensionName" matchesLen:"dimensionValue"`
    DimensionValue string `keda:"dimensionValue" matchesLen:"dimensionName"`
}

1. build a DSL in Go code to define schemas for metadata rules. this would be more verbose than the reflection-based approach, but compile-time checked and less "magic". example:

cloudWatchSchema := schema.New(cloudWatchMetadata).Require("namespace", "metricName", "dimensionName", "dimensionValue", "targetMetricValue" /* etc... */).Validate([]string{"dimensionName", "dimensionValue"}, func(vals []string) error {
    dName := vals[0]
    dVal := vals[1]
    if len(dName) != len(dVal) {
        return fmt.Errorf(fmt.Errorf("dimensionName and dimensionValue are not matching in size"))
    }
})

1. write validation rules in some markup language (e.g. JSONSchema, YAML) for each scaler's metadata, load them into memory, and then apply them to each incoming ScaledObject. example in YAML:

requiredFields: ["namespace", "metricName", "dimensionName", "dimensionValue", "targetMetricValue"] # more fields omitted
validations:
- name: dimensionName
  validation:
    matchesLen: dimensionValue
- name: dimensionValue
  validation:
     matchesLen: dimensionName

WDYT?