Add wait time after scaling up/down operations to ensure logs are uploaded to CW

[EddyMM]

Description of changes

• When scaling up or down, we run assertions on the scaling metrics after a fixed amount of time
• If any of the assertions fail, the test fails and the cluster is deleted as part of the test teardown.
• The CW logs are preserved but they may not have captured some relevant info about the compute nodes that were launched e.g. nodes with bootstrap errors.
• The changes in this PR add a wait time before we run assertions on the collected scaling metrics to ensure we can review logs linked to the assertions in case of a failure

Tests

• Ran test with a manually injected assertion failure to ensure we receive logs that are at least up to the timestamp of the latest scaling metric used in the assertion

References

• N/A

Checklist

• Make sure you are pointing to the right branch.
• If you're creating a patch for a branch other than develop add the branch name as prefix in the PR title (e.g. [release-3.6]).
• Check all commits' messages are clear, describing what and why vs how.
• Make sure to have added unit tests or integration tests to cover the new/modified code.
• Check if documentation is impacted by this change.

Please review the guidelines for contributing and Pull Request Instructions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (c3f620c) 90.13% compared to head (b33f0a4) 90.13%.
Report is 2 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #5876   +/-   ##
========================================
  Coverage    90.13%   90.13%           
========================================
  Files          180      180           
  Lines        15725    15725           
========================================
  Hits         14173    14173           
  Misses        1552     1552           

Flag	Coverage Δ
unittests	90.13% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.