Using Azure Event Hubs scaler results in "unable to download file from blob storage" error

[abhirockzz]

After creating the scaled object for Azure Event Hub along with a Event Hub consumer app in form of a Deployment, results in the following error:

time="2019-10-13T17:39:33Z" level=error msg="error getting metric for scaler : unable to get checkpoint from storage: unable to download file from blob storage: -> github.com/kedacore/keda/vendor/github.com/Azure/azure-storage-blob-go/azblob.NewResponseError, /go/src/github.com/kedacore/keda/vendor/github.com/Azure/azure-storage-blob-go/azblob/zz_generated_response_error.go:29\n===== RESPONSE ERROR (ServiceCode=ContainerNotFound) =====\nDescription=The specified container does not exist.\nRequestId:9c06b5bc-c01e-0071-21ed-81daf2000000\nTime:2019-10-13T17:39:33.5143470Z, Details: \n Code: ContainerNotFound\n GET https://<STORAGE_ACC_NAME>.blob.core.windows.net/azure-webjobs-eventhub/<EVENTHUB_NAMESPACE>.servicebus.windows.net/<EVENTHUB_NAME>/$Default/0?timeout=61\n Authorization: REDACTED\n User-Agent: [Azure-Storage/0.6 (go1.12.9; linux)]\n X-Ms-Client-Request-Id: [e05f5fd6-4052-45d4-717e-5252f51054aa]\n X-Ms-Date: [Sun, 13 Oct 2019 17:39:32 GMT]\n X-Ms-Version: [2018-11-09]\n --------------------------------------------------------------------------------\n RESPONSE Status: 404 The specified container does not exist.\n Content-Length: [225]\n Content-Type: [application/xml]\n Date: [Sun, 13 Oct 2019 17:39:32 GMT]\n Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]\n X-Ms-Error-Code: [ContainerNotFound]\n X-Ms-Request-Id: [9c06b5bc-c01e-0071-21ed-81daf2000000]\n X-Ms-Version: [2018-11-09]\n\n\n"
I1013 17:39:33.592524 1 trace.go:76] Trace[1768462654]: "List /apis/external.metrics.k8s.io/v1beta1/namespaces/default/unprocessedeventthreshold" (started: 2019-10-13 17:39:32.78777752 +0000 UTC m=+20633.475899632) (total time: 804.722558ms):
Trace[1768462654]: [804.644958ms] [804.577258ms] Listing from storage done

Notice the URI - GET https://<STORAGE_ACC_NAME>.blob.core.windows.net/azure-webjobs-eventhub/<EVENTHUB_NAMESPACE>.servicebus.windows.net/<EVENTHUB_NAME>/$Default/0?timeout=61

It's as per this logic I guess https://github.com/kedacore/keda/blob/master/pkg/scalers/azure_eventhub.go#L88

But I am not sure why such a format is being used along with a hardcoded azure-webjobs-eventhub and the event hub info. I obviously don't have the historical context behind this so pls let me know if I am missing something.

FYI, I ran a sanity producer-consumer test (without KEDA) and here is the URI for the checkpointed Blob https://<STORAGE_ACC_NAME>.blob.core.windows.net/<STORAGE_ACC_NAME>/$Default/0

$Default is the consumer name and 0 is the partition ID

[abhirockzz]

Can this be confirmed as a valid issue? If yes, I would be happy to dig in further

[abhirockzz]

@tomkerkhove / @ahmelsayed / @Aarthisk / @jeffhollan - just wanted to check if you have any insights? Happy to work on a PR once I have some clarification

[jeffhollan]

I see what this code path is trying to do. Assuming you provide a storage account connection string in the metadata as described in the sample:
https://github.com/kedacore/keda/blob/master/spec/triggers/azure-event-hub.md

It will create a blob container to manage the checkpoints. Does your metadata not include the connection string secret reference?

[abhirockzz]

@jeffhollan yes, I do have the right metadata (as env variable) for blob storage.

Just to clarify again, the problem right now is that blob storage URL which KEDA is using (https://github.com/kedacore/keda/blob/master/pkg/scalers/azure_eventhub.go#L88) is not as per the format which is actually created by the Event Hubs Java client (https://<STORAGE_ACC_NAME>.blob.core.windows.net/<STORAGE_ACC_NAME>/$Default/0) and this is where the issue pops up - https://github.com/kedacore/keda/blob/master/pkg/scalers/azure_eventhub.go#L100

Clearly, there is a mismatch here

Not sure why this format is being used by KEDA (is this specific to Functions)? https://<STORAGE_ACC_NAME>.blob.core.windows.net/azure-webjobs-eventhub/<EVENTHUB_NAMESPACE>.servicebus.windows.net/<EVENTHUB_NAME>/$Default/0?timeout=61

[ahmelsayed]

Closed in #517

[lenardchristopher]

Is keda supposed to create the "azure-webjobs-eventhub" in Azure Blob Storage? I get the same ContainerNotFound error, but the connection string looks fine.

Edit: I looked through the code and Keda isn't creating the containers. Is it on the developers to create the container in blob?

[leonids2005]

I have the same issue and I think the documentation should be updated to mention this.
I am using Azure Function written in C# and scaling does not work until you create azure-webjobs-eventhub container.