[CoreML] Add retry logic to database/key-value store

[metascroy]

This PR introduces changes to the database and key-value store to retry if there are certain kinds of failures.

I've confirmed it can recover and re-create the DB if the DB is deleted during predictions.

The "std::cout" statements will be removed before landing.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15170

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures, 2 Cancelled Jobs

As of commit ac16839 with merge base 3ccb6ab ():

NEW FAILURES - The following jobs have failed:

• trunk / test-coreml-delegate / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 127
• trunk / test-llama-runner-mac (fp32, coreml) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-mac (fp32, mps) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-mac (fp32, xnnpack+custom+quantize_kv) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

CANCELLED JOBS - The following jobs were cancelled. Please retry:

• trunk / test-models-macos-coreml (w2l) / macos-job (gh)
  ##[error]The operation was canceled.
• trunk / test-models-macos-cpu (w2l, xnnpack-quantization-delegation) / macos-job (gh)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[meta-codesync]

@metascroy has imported this pull request. If you are a Meta employee, you can view this in D84748624.

[metascroy]

@cymbalrush can you have a look?

[cymbalrush]

@metascroy do we know for sure if the database file is being deleted?
All reads and writes to the database happen on the sync queue (ETCoreMLAssetManager.mm), so access should be fully serialized. An SQLite transaction shouldn’t fail under serialized access.

If the database file is being deleted, that’s concerning — the OS will not automatically remove files from the Application Support directory. If the application itself is deleting it, can we exclude this file from that process?

Also, could we log the SQLite error code to confirm what’s actually failing? is it failing when the app is backgrounded?