Fix race condition when multiple nodes reconcile S3 snapshots

[brandond]

Proposed Changes

Fix race condition when multiple nodes reconcile S3 snapshots

Don't delete s3 etcdsnapshotfiles if they are missing from s3 but less than a minute old, its possible the other node just finished uploading it and the object key has not yet become visible.

This is most easily reproduced when an on-demand etcd snapshot is requested via Rancher, as it triggers simultaneous snapshots on all etcd nodes, and they will all try to reconcile s3 objects against the etcdsnapshotfile list at the same time despite not necessarily having a consistent view of the bucket.

Types of Changes

bugfix

Verification

See linked issue

Testing

Linked Issues

• S3 snapshots missing from etcdsnapshotfile list / k3s-etcd-snapshots configmap when multiple nodes save at the same time #10989
• [BUG] etcd snapshot count reported in rancher is incorrect when taking manual snapshot with s3 enabled on rke2 clusters, but is correct in s3 rancher/rancher#47339

User-Facing Change

Further Comments

[codecov]

Codecov Report

Attention: Patch coverage is 28.00000% with 18 lines in your changes missing coverage. Please review.

Project coverage is 43.78%. Comparing base (0942e6a) to head (12020bc).
Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/etcd/snapshot.go	13.33%	13 Missing ⚠️
pkg/etcd/s3/s3.go	50.00%	2 Missing and 3 partials ⚠️

❗ There is a different number of reports uploaded between BASE (0942e6a) and HEAD (12020bc). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (0942e6a)	HEAD (12020bc)
e2etests	7	6

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10979      +/-   ##
==========================================
- Coverage   49.95%   43.78%   -6.18%     
==========================================
  Files         178      178              
  Lines       14801    14812      +11     
==========================================
- Hits         7394     6485     -909     
- Misses       6056     7126    +1070     
+ Partials     1351     1201     -150     

Flag	Coverage Δ
e2etests	36.12% <0.00%> (-10.05%)	⬇️
inttests	36.73% <8.00%> (-0.03%)	⬇️
unittests	13.52% <20.00%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.