Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segmentation violation #3811

Closed
tomasgiden opened this issue Nov 3, 2022 · 0 comments · Fixed by #3812
Closed

Segmentation violation #3811

tomasgiden opened this issue Nov 3, 2022 · 0 comments · Fixed by #3812
Assignees
@tomasgiden
Labels
bug Something isn't working

Comments

@tomasgiden
Copy link
Contributor

Report

Starting up, Keda Operator has a segmentation violation in logger. Presumably because Azure Blob Scaler is missing to store the initialized logger on line 70 in azure_blob_scaler.go. I will contribute a PR and link to this issue.

Expected Behavior

I expect Keda Operator to not have a segmentation violation. If my scaler credentials are bad, I expect Keda to handle it gracefully.

Actual Behavior

Keda crashes cyclically.

Steps to Reproduce the Problem

  1. Add a ScaledJob with azure blob scaling with something wrong in the credentials to blob storage (I presume)
  2. Sit back and watch keda operator crash

Logs from KEDA operator

LogEntry
" /workspace/pkg/scaling/scale_handler.go:108 +0x465"
created by github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).HandleScalableObject
" /workspace/pkg/scaling/scale_handler.go:149 +0x31c"
"github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop(0xc000863650, {0x3cae988, 0xc0011982c0}, 0xc0004e9900, {0x34ee760, 0xc0011a4240}, {0x3c8b6e8, 0xc000e7ff68})"
" /workspace/pkg/scaling/scale_handler.go:286 +0x317"
"github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers(0xc000863650, {0x3cae988, 0xc0011982c0}, {0x34ee760, 0xc0011a4240}, {0x3c8b6e8, 0xc000e7ff68})"
" /workspace/pkg/scaling/cache/scalers_cache.go:125 +0xda"
"github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledJobActive(0xc00004dd10, {0x3cae988, 0xc0011982c0}, 0xc0011a4240)"
" /workspace/pkg/scaling/cache/scalers_cache.go:263 +0x332"
"github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).getScaledJobMetrics(0xc0002dc690, {0x3cae988, 0xc0011982c0}, 0xc0011a4240)"
" /workspace/pkg/scalers/azure_blob_scaler.go:193 +0xa5"
"github.com/kedacore/keda/v2/pkg/scalers.(*azureBlobScaler).IsActive(0xc0011a21e0, {0x3cae988, 0xc0011982c0})"
" /go/pkg/mod/github.com/go-logr/[email protected]/logr.go:279 +0x80"
"github.com/go-logr/logr.Logger.Error({{0x0, 0x0}, 0x35a828d}, {0x3c59720, 0xc000412dc0}, {0x35a49bb, 0x6}, {0x0, 0x0, 0x0})"
goroutine 350 [running]:

[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x585a40]
panic: runtime error: invalid memory address or nil pointer dereference
"2022-11-03T17:37:27Z INFO Initializing Scaling logic according to ScaledJob Specification {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""scaledJob"": {""name"":""reconstruction"",""namespace"":""app-env-dtu""}, ""namespace"": ""app-env-dtu"", ""name"": ""reconstruction"", ""reconcileID"": ""386dff46-d3f6-4e03-99bc-a2d86ca1a35d""}"
"2022-11-03T17:37:27Z INFO RolloutStrategy: gradual, Not deleting jobs owned by the previous version of the scaleJob {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""scaledJob"": {""name"":""reconstruction"",""namespace"":""app-env-dtu""}, ""namespace"": ""app-env-dtu"", ""name"": ""reconstruction"", ""reconcileID"": ""386dff46-d3f6-4e03-99bc-a2d86ca1a35d""}"
"2022-11-03T17:37:27Z INFO Reconciling ScaledJob {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""scaledJob"": {""name"":""reconstruction"",""namespace"":""app-env-dtu""}, ""namespace"": ""app-env-dtu"", ""name"": ""reconstruction"", ""reconcileID"": ""386dff46-d3f6-4e03-99bc-a2d86ca1a35d""}"
"2022-11-03T17:37:27Z INFO Initializing Scaling logic according to ScaledJob Specification {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""scaledJob"": {""name"":""preprocessing"",""namespace"":""app-env-dtu""}, ""namespace"": ""app-env-dtu"", ""name"": ""preprocessing"", ""reconcileID"": ""6e92f8cf-d4c8-43f2-848a-5060ef1dbeae""}"
"2022-11-03T17:37:27Z INFO RolloutStrategy: gradual, Not deleting jobs owned by the previous version of the scaleJob {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""scaledJob"": {""name"":""preprocessing"",""namespace"":""app-env-dtu""}, ""namespace"": ""app-env-dtu"", ""name"": ""preprocessing"", ""reconcileID"": ""6e92f8cf-d4c8-43f2-848a-5060ef1dbeae""}"
"2022-11-03T17:37:27Z INFO Reconciling ScaledJob {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""scaledJob"": {""name"":""preprocessing"",""namespace"":""app-env-dtu""}, ""namespace"": ""app-env-dtu"", ""name"": ""preprocessing"", ""reconcileID"": ""6e92f8cf-d4c8-43f2-848a-5060ef1dbeae""}"
"2022-11-03T17:37:27Z INFO RolloutStrategy: gradual, Not deleting jobs owned by the previous version of the scaleJob {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""scaledJob"": {""name"":""big-reconstruction"",""namespace"":""app-env-dtu""}, ""namespace"": ""app-env-dtu"", ""name"": ""big-reconstruction"", ""reconcileID"": ""ba3f1142-a3bd-4c3d-a1c7-a07b1e103dc9""}"
"2022-11-03T17:37:27Z INFO Reconciling ScaledJob {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""scaledJob"": {""name"":""big-reconstruction"",""namespace"":""app-env-dtu""}, ""namespace"": ""app-env-dtu"", ""name"": ""big-reconstruction"", ""reconcileID"": ""ba3f1142-a3bd-4c3d-a1c7-a07b1e103dc9""}"
"2022-11-03T17:37:27Z INFO Starting workers {""controller"": ""triggerauthentication"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""TriggerAuthentication"", ""worker count"": 1}"
"2022-11-03T17:37:27Z INFO Starting workers {""controller"": ""scaledobject"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledObject"", ""worker count"": 5}"
"2022-11-03T17:37:27Z INFO Starting workers {""controller"": ""clustertriggerauthentication"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ClusterTriggerAuthentication"", ""worker count"": 1}"
"2022-11-03T17:37:27Z INFO Starting workers {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""worker count"": 1}"
"2022-11-03T17:37:27Z INFO KubeAPIWarningLogger autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+  use autoscaling/v2 HorizontalPodAutoscaler"
"2022-11-03T17:37:27Z INFO Starting Controller {""controller"": ""clustertriggerauthentication"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ClusterTriggerAuthentication""}"
"2022-11-03T17:37:27Z INFO Starting EventSource {""controller"": ""clustertriggerauthentication"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ClusterTriggerAuthentication"", ""source"": ""kind source: *v1alpha1.ClusterTriggerAuthentication""}"
"2022-11-03T17:37:27Z INFO Starting Controller {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob""}"
"2022-11-03T17:37:27Z INFO Starting EventSource {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""source"": ""kind source: *v1alpha1.ScaledJob""}"
"2022-11-03T17:37:27Z INFO Starting Controller {""controller"": ""triggerauthentication"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""TriggerAuthentication""}"
"2022-11-03T17:37:27Z INFO Starting EventSource {""controller"": ""triggerauthentication"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""TriggerAuthentication"", ""source"": ""kind source: *v1alpha1.TriggerAuthentication""}"
"2022-11-03T17:37:27Z INFO Starting Controller {""controller"": ""scaledobject"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledObject""}"
"2022-11-03T17:37:27Z INFO Starting EventSource {""controller"": ""scaledobject"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledObject"", ""source"": ""kind source: *v2beta2.HorizontalPodAutoscaler""}"
"2022-11-03T17:37:27Z INFO Starting EventSource {""controller"": ""scaledobject"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledObject"", ""source"": ""kind source: *v1alpha1.ScaledObject""}"
I1103 17:37:27.219554       1 leaderelection.go:258] successfully acquired lease keda/operator.keda.sh
"2022-11-03T17:37:27Z INFO Initializing Scaling logic according to ScaledJob Specification {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""scaledJob"": {""name"":""big-reconstruction"",""namespace"":""app-env-dtu""}, ""namespace"": ""app-env-dtu"", ""name"": ""big-reconstruction"", ""reconcileID"": ""ba3f1142-a3bd-4c3d-a1c7-a07b1e103dc9""}"
"2022-11-03T17:37:11Z INFO Starting server {""path"": ""/metrics"", ""kind"": ""metrics"", ""addr"": ""[::]:8080""}"
"2022-11-03T17:37:11Z INFO Starting server {""kind"": ""health probe"", ""addr"": ""[::]:8081""}"
I1103 17:37:11.962951       1 leaderelection.go:248] attempting to acquire leader lease keda/operator.keda.sh...
2022-11-03T17:37:11Z INFO setup Go OS/Arch: linux/amd64
2022-11-03T17:37:11Z INFO setup Go Version: go1.17.13
2022-11-03T17:37:11Z INFO setup Git Commit: a4a118201214e7abdeebad72cbe337b9856f8191
2022-11-03T17:37:11Z INFO setup KEDA Version: 2.8.0
2022-11-03T17:37:11Z INFO setup Starting manager
"2022-11-03T17:37:11Z INFO setup Running on Kubernetes 1.24 {""version"": ""v1.24.6""}"
"2022-11-03T17:37:11Z INFO controller-runtime.metrics Metrics server is starting to listen {""addr"": "":8080""}"```


### KEDA Version

2.8.0

### Kubernetes Version

1.24

### Platform

Microsoft Azure

### Scaler Details

Azube Blob Scaler

### Anything else?

_No response_
@tomasgiden tomasgiden added the bug Something isn't working label Nov 3, 2022
@tomkerkhove tomkerkhove moved this to Proposed in Roadmap - KEDA Core Nov 3, 2022
@tomasgiden tomasgiden mentioned this issue Nov 3, 2022
Merged
3 tasks
@JorTurFer JorTurFer moved this from Proposed to In Review in Roadmap - KEDA Core Nov 3, 2022
Repository owner moved this from In Review to Ready To Ship in Roadmap - KEDA Core Nov 4, 2022
@lkishalmi lkishalmi mentioned this issue Dec 2, 2022
Merged
2 tasks
@JorTurFer JorTurFer mentioned this issue Dec 2, 2022
Closed
@JorTurFer JorTurFer moved this from Ready To Ship to Done in Roadmap - KEDA Core Mar 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tomasgiden tomasgiden

Labels
bug Something isn't working
Projects
Roadmap - KEDA Core
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(azure blob scaler): initialize logger #3811 tomasgiden/keda
1 participant
@tomasgiden