Refresh expired/rotated AWS credentials without emitting a scaler failure

[JacobHenner]

Proposal

In #2573, support for AWS temporary credentials (using session tokens) was introduced.

Assuming Secrets are kept up-to-date with valid session tokens, scalers using temporary credentials will error once after token expiration. The scaler cache for the corresponding ScaledObject will be cleared, the scaler will be rebuilt using the updated temporary credentials, and the scaler will resume operation.

This approach has a potentially undesired side effect - an HPA Scaler Error will be recorded in the metrics each time a temporary credential expires, even if the replacement scaler reloads valid credentials and succeeds immediately following the expiration failure. Since users might be relying on scaler error metrics to indicate unexpected failures, these newly reported failures might be confusing or misleading.

To resolve this (assuming the AWS libraries in use can propagate back credential expiration errors), we could:

• Create a new type of metric for creds expiration - but I think that'd be similarly confusing (one event at expiration time is expected, > 1 event at expiration time is bad and indicates actual scaler failure).
• Only report a failure for credential expiration errors if there are >= 2 consecutive failures
  • We might want to consider a similar approach for static credentials too, as they can also be rotated (generally with a much lower frequency).
• Build an independent credential refresh mechanism - e.g. Watch created HPA and TriggerAuthentication resources #511, TriggerAuthentication Secret is not reloaded when Secret changes #563
• (something else?)

Use-Case

No response

Anything else?

No response

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[JacobHenner]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

Not stale

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.