Metrics Scaler: Improve Error Handling on not-ok response

[alexkunde]

Proposal

What's even going on?

The Metrics API Scaler is pretty quit on non-200 API responses. (See this link for code)

At the moment we are using this for around 2000 scale targets and whenever something on API side goes wrong its a real hassle to even find out what the query was, when this is your response:

2021-11-23T11:59:33.926Z    ERROR    metrics_api_scaler    Error when checking metric value: api returned 403    {"error": "api returned 403"} 
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
    /workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
    /workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
    /workspace/pkg/scaling/scale_handler.go:145

Suggestion

Integrate the requested API endpoint and an optional response body into the scale error to get decent info on why the query didn't go through.

Possible new log

2021-11-23T11:59:33.926Z    ERROR    metrics_api_scaler    Error when checking metric value for /api/v1/files/on/server: api returned 403    {"error": "403 forbidden - cannot change into directory"} 
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledObjectActive
    /workspace/pkg/scaling/scale_handler.go:228
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
    /workspace/pkg/scaling/scale_handler.go:211
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
    /workspace/pkg/scaling/scale_handler.go:145

Use-Case

• Improve debugging of big scaled systems
• No need to have access to logs of metric endpoint to debug

Anything else?

No response

[zroubalik]

This seems like a fair request, are you willing to contribut this?

[alexkunde]

I'd be happy to contribute, I'll look into the process and will link a PR

[ahmelsayed]

Also while you're at it, this defer

keda/pkg/scalers/metrics_api_scaler.go

Line 219 in cbe929b

defer r.Body.Close()

should move above that if r.StatusCode != http.StatusOK

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[zroubalik]

@schoki040 what is the status of this please?

[alexkunde]

I was not yet able to make any changes on this. This improvement is nonetheless still dear to my heart. Please dont close.

[fira42073]

Hey. I created PR for this issue. Please review - #2739