Fix snapshot reconcile retry

[brandond]

Proposed Changes

Fix snapshot reconcile retry

• Fix excessive retry on snapshot configmap reconcile
• Retry startup snapshot reconcile.
  The reconcile may run before the kubelet has created the node object; retry until it succeeds

Types of Changes

bugfix

Verification

See linked issues

Testing

Linked Issues

• etcd snapshot ConfigMap controller excessive sync requeuing #9241

User-Facing Change

Further Comments

[codecov]

Codecov Report

Attention: 2543 lines in your changes are missing coverage. Please review.

Comparison is base (ce6158f) 49.07% compared to head (765d7b0) 40.56%.
Report is 184 commits behind head on master.

Files	Patch %	Lines
pkg/etcd/snapshot.go	35.15%	439 Missing and 72 partials ⚠️
pkg/etcd/s3.go	0.00%	233 Missing ⚠️
pkg/etcd/snapshot_controller.go	0.00%	216 Missing ⚠️
pkg/etcd/etcd.go	19.65%	168 Missing and 16 partials ⚠️
pkg/spegel/spegel.go	2.98%	130 Missing ⚠️
pkg/agent/containerd/containerd.go	0.00%	116 Missing ⚠️
pkg/spegel/bootstrap.go	1.81%	108 Missing ⚠️
pkg/apis/k3s.cattle.io/v1/zz_generated_deepcopy.go	17.35%	93 Missing and 7 partials ⚠️
pkg/server/secrets-encrypt.go	0.00%	84 Missing ⚠️
...d/controllers/k3s.cattle.io/v1/etcdsnapshotfile.go	0.00%	75 Missing ⚠️
... and 51 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9318      +/-   ##
==========================================
- Coverage   49.07%   40.56%   -8.51%     
==========================================
  Files         154      154              
  Lines       16416    16558     +142     
==========================================
- Hits         8056     6717    -1339     
- Misses       7090     8695    +1605     
+ Partials     1270     1146     -124     

Flag	Coverage Δ
e2etests	?
inttests	37.73% <25.67%> (-4.56%)	⬇️
unittests	14.52% <6.84%> (-0.06%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.