[release-23.0] BuiltinBackupEngine: Retry file close and fail backup when we cannot (#18848)

[vitess-bot]

Description

This is a backport of #18848

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

• Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

• Apply the release notes (needs details) label if users need to know about this change.
• New features should be documented.
• There should be some code comments as to why things are implemented the way they are.
• There should be a comment at the top of each new or modified test to explain what the test does.

New flags

• Is this flag really necessary?
• Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
• vtctl command output order should be stable and awk-able.

[codecov]

Codecov Report

❌ Patch coverage is 69.64286% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.70%. Comparing base (58344e1) to head (e7507b1).
⚠️ Report is 15 commits behind head on release-23.0.

Files with missing lines	Patch %	Lines
go/vt/mysqlctl/builtinbackupengine.go	73.58%	14 Missing ⚠️
go/vt/mysqlctl/gcsbackupstorage/gcs.go	0.00%	3 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##           release-23.0   #18862   +/-   ##
=============================================
  Coverage         69.70%   69.70%           
=============================================
  Files              1605     1606    +1     
  Lines            214449   214514   +65     
=============================================
+ Hits             149475   149529   +54     
- Misses            64974    64985   +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.