[CosmosDb] Avoid swallowing exceptions when retrieving votes

[alrod]

Proposed Improvement
We want to emit error logs to the customer’s Application Insights whenever there are connectivity issues to the resource.

Current Behavior
Exceptions are swallowed in the extension code and logged only in internal logs.

Planned Change

• The change will allow connectivity exceptions to bubble up to the WebJobs SDK, so they can be logged in the standard WebJobs format.
• All extensions will follow the same logging format for consistency.
• In case of an exception, the trigger will be ignored for the current vote integration.

References
-https://github.com/Azure/azure-webjobs-sdk/blob/dev/src/Microsoft.Azure.WebJobs.Host/Scale/ScaleManager.cs#L195
-https://github.com/Azure/azure-webjobs-sdk/blob/dev/src/Microsoft.Azure.WebJobs.Host/Scale/ScaleManager.cs#L121

Additional Notes

• The ScaleManager class in WebJobs SDK is used for both regular scale and runtime-driven scale.
• We are also adding voting logs in the PR to standardize how the current vote is logged. These logs will also be provided to the customer.
• A bunch of exceptions that were previously not emitted in Scale Controller internally (client exceptions) have been removed. Exceptions are swallowed only in scaling code; the listener code already logs them. Need to follow with CosmosDb team if we need to filter out some transient CosmosDb exceptions.

[mathewc]

From the links you shared Alexey, we have upstream handling already for any exceptions coming from this code on the scale monitor path. My primary concern with this was whether letting these through was going to start failing other scenarios that are currently passing (i.e. regression). I don't know who else may be using these APIs.

[mathewc]

I don't understand - why won't these error messages written to the supplied ILogger go to the customer's App Insights or other log streams? The logger should already be configured to do this.

[jviau]

Changing this exception logic is a high risk of a breaking change. While this class is internal, the exception could now bubble up into public API and customer code depending on how it is used.

Unless we can verify 100% this will never impact a customer scenario I don't think we can take it. And I don't mean just specific functions scenario, I mean any customer using any public API in this package, functions or not.

[alrod]

Closing in favor - #975