Event Hub Scaler: Remove or replace usages of Event Hub offsets #5600

windy1 · 2024-03-13T21:51:25Z

This PR removes or replaces uses of Event Hub offsets, as this is no longer a supported metric to use for determining consumer locations within a partition.

Checklist

When introducing a new scaler, I agree with the scaling governance policy
I have verified that my change is according to the deprecations & breaking changes policy
Tests have been added
Changelog has been updated and is aligned with our changelog requirements
A PR is opened to update our Helm chart (repo) (if applicable, ie. when deployment manifests are modified)
A PR is opened to update the documentation on (repo) (if applicable)
Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #5574

Relates to Azure/azure-sdk-for-net#42409

windy1 · 2024-03-13T21:54:50Z

@JorTurFer Hi there, I've marked this PR as WIP because I've had some trouble testing my changes. I was able to get a debugger working with the operator and confirm that the totalUnprocessedEventCount is within expected ranges but I have been unsuccessful getting it to work with a local Kubernetes cluster I set up.

My current environment is debugging the operator in a devcontainer which seems to successfully connect to the cluster, but it looks like the metrics server isn't able to connect to it, so I am unable to verify if the scaling is working correctly.

Here's some logs from the metric server:

E0313 21:46:41.764411       1 timeout.go:142] post-timeout activity - time-elapsed: 236.679959ms, GET "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/s0-azure-eventhub-local" result: runtime error: invalid memory address or nil pointer dereference
goroutine 645 [running]:
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1()
	/workspace/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:110 +0x9c
panic({0x20a9300?, 0x4177940?})
	/usr/local/go/src/runtime/panic.go:914 +0x218
sigs.k8s.io/custom-metrics-apiserver/pkg/registry/external_metrics.(*REST).List(0x40008859b0, {0x2a081f0, 0x40006eb470}, 0x0?)
	/workspace/vendor/sigs.k8s.io/custom-metrics-apiserver/pkg/registry/external_metrics/reststorage.go:92 +0xf0
k8s.io/apiserver/pkg/endpoints/handlers.ListResource.func1({0x29f92d0?, 0x4001577440}, 0x40011e7400)
	/workspace/vendor/k8s.io/apiserver/pkg/endpoints/handlers/get.go:278 +0xc04
sigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/installer.(*EMHandlers).registerResourceHandlers.restfulListResource.func1(0x4001577420, 0x40006cb3b0)
	/workspace/vendor/sigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/installer/installer.go:274 +0x6c
sigs.k8s.io/custom-metrics-apiserver/pkg/apiserver/installer.(*EMHandlers).registerResourceHandlers.InstrumentRouteFunc.func2(0x4001577420, 0x40006cb3b0)
	/workspace/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:571 +0x160
github.com/emicklei/go-restful/v3.(*Container).dispatch(0x400137ee10, {0x29f92d0, 0x40015771e0}, 0x40011e7400)
	/workspace/vendor/github.com/emicklei/go-restful/v3/container.go:299 +0x470
github.com/emicklei/go-restful/v3.(*Container).Dispatch(...)
	/workspace/vendor/github.com/emicklei/go-restful/v3/container.go:204
k8s.io/apiserver/pkg/server.director.ServeHTTP({{0x254bd40?, 0x23daba0?}, 0x400137ee10?, 0x40003b6ee0?}, {0x29f92d0, 0x40015771e0}, 0x40011e7400)
	/workspace/vendor/k8s.io/apiserver/pkg/server/handler.go:146 +0x4a8
k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func21({0x29f92d0?, 0x40015771e0}, 0x40011e7400)
	/workspace/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:110 +0x134
net/http.HandlerFunc.ServeHTTP(0x2a081f0?, {0x29f92d0?, 0x40015771e0?}, 0x4?)
	/usr/local/go/src/net/http/server.go:2136 +0x38
k8s.io/apiserver/pkg/endpoints/filters.withAuthorization.func1({0x29f92d0, 0x40015771e0}, 0x40011e7400)
	/workspace/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:78 +0x45c
net/http.HandlerFunc.ServeHTTP(0xa43f3b26b?, {0x29f92d0?, 0x40015771e0?}, 0x88ef90?)
	/usr/local/go/src/net/http/server.go:2136 +0x38
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x29f92d0, 0x40015771e0}, 0x40011e7400)
	/workspace/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:84 +0x154
net/http.HandlerFunc.ServeHTTP(0x4191b80?, {0x29f92d0?, 0x40015771e0?}, 0x4?)
	/usr/local/go/src/net/http/server.go:2136 +0x38
k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1({0x29f92d0, 0x40015771e0}, 0x40011e7400)
	/workspace/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:196 +0x228
net/http.HandlerFunc.ServeHTTP(0x4000dd3178?, {0x29f92d0?, 0x40015771e0?}, 0x1754e04?)
	/usr/local/go/src/net/http/server.go:2136 +0x38
k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func23({0x29f92d0?, 0x40015771e0}, 0x40011e7400)
	/workspace/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:110 +0x134
net/http.HandlerFunc.ServeHTTP(0x0?, {0x29f92d0?, 0x40015771e0?}, 0x4001546b28?)
	/usr/local/go/src/net/http/server.go:2136 +0x38
k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.WithImpersonation.func4({0x29f92d0, 0x40015771e0}, 0x40011e7400)
	/workspace/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x154
net/http.HandlerFunc.ServeHTTP(0x4000dd3828?, {0x29f92d0?, 0x40015771e0?}, 0xb29c0?)
	/usr/local/go/src/net/http/server.go:2136 +0x38
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x29f92d0, 0x40015771e0}, 0x40011e7400)
	/workspace/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:84 +0x154
net/http.HandlerFunc.ServeHTTP(0x4000dd38a8?, {0x29f92d0?, 0x40015771e0?}, 0x1753764?)
	/usr/local/go/src/net/http/server.go:2136 +0x38
k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func24({0x29f92d0?, 0x40015771e0}, 0x40011e7400)
	/workspace/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:110 +0x134
net/http.HandlerFunc.ServeHTTP(0x4000dd3a58?, {0x29f92d0?, 0x40015771e0?}, 0xb29c0?)
	/usr/local/go/src/net/http/server.go:2136 +0x38
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x29f92d0, 0x40015771e0}, 0x40011e7400)
	/workspace/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:84 +0x154
net/http.HandlerFunc.ServeHTTP(0x4000dd3ad8?, {0x29f92d0?, 0x40015771e0?}, 0x17528f4?)
	/usr/local/go/src/net/http/server.go:2136 +0x38
k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain.TrackCompleted.trackCompleted.func26({0x29f92d0?, 0x40015771e0}, 0x40011e7400)
	/workspace/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:110 +0x134
net/http.HandlerFunc.ServeHTTP(0x2a081f0?, {0x29f92d0?, 0x40015771e0?}, 0x29cabc0?)
	/usr/local/go/src/net/http/server.go:2136 +0x38
k8s.io/apiserver/pkg/endpoints/filters.withAuthentication.func1({0x29f92d0, 0x40015771e0}, 0x40011e7400)
	/workspace/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:120 +0x614
net/http.HandlerFunc.ServeHTTP(0x2a081f0?, {0x29f92d0?, 0x40015771e0?}, 0x29d0c58?)
	/usr/local/go/src/net/http/server.go:2136 +0x38
k8s.io/apiserver/pkg/endpoints/filterlatency.trackStarted.func1({0x29f92d0, 0x40015771e0}, 0x40011e7200)
	/workspace/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:94 +0x2f4
net/http.HandlerFunc.ServeHTTP(0x4001101f60?, {0x29f92d0?, 0x40015771e0?}, 0x81f14?)
	/usr/local/go/src/net/http/server.go:2136 +0x38
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1()
	/workspace/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:115 +0x68
created by k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP in goroutine 644
	/workspace/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:101 +0x180

Any guidance you are able to provide would be appreciated! Thank you.

Also, I've just noticed that last_enqueued_sequence_number is present in the RPC response for HubPartitionRuntimeInformation, but it's not loaded into the struct by the Azure lib. Not sure what to do about this either as I don't know what other information we could use here.

zroubalik · 2024-03-27T16:11:22Z

Any updates on this?

windy1 · 2024-03-30T17:31:56Z

I believe this should be ready to go now.

pkg/scalers/azure/azure_eventhub_checkpoint.go

JorTurFer · 2024-03-30T23:11:49Z