Create files without O_NONBLOCK flag

[kskalski]

Problem

We do not currently perform writes with ASYNC flag, so they are first considered in non-blocking context (directly in the requesting thread, not in a worker) and re-submitted in the worker when they cannot be serviced immediately.

On older kernels (5.15) such re-submission is however broken in the case of file being open with O_NONBLOCK mode, but request needing to be done in blocking fashion (e.g. on a file-system not supporting non-blocking execution natively).

In other words when IO needs to block, but it is submitted on a file required to not block, kernel just returns error, requiring the user to submit IO to the kernel worker, i.e. with ASYNC flag (on newer kernel this path is handled automatically). See also discussion in #8053 (comment)

Also, O_NONBLOCK flag without O_DIRECT is actually an unusual / not supported mode, with io_uring we shouldn't use it at all.

Summary of Changes

Don't create files in non-blocking mode, which relieves constraints on how IO is executed by the kernel and prevents EAGAIN errors to be surfaced in io_uring completions on older kernels.

Performance is not affected (leaning positive, but that is within noise):

• this PR: snapshot untar took 93.7s, snapshot untar took 97.8s, snapshot untar took 94.9s
• master: snapshot untar took 97.5s, snapshot untar took 97.7s

Fixes #8036

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.0%. Comparing base (b6c922c) to head (80d80eb).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #8161   +/-   ##
=======================================
  Coverage    83.0%    83.0%           
=======================================
  Files         826      826           
  Lines      362561   362561           
=======================================
  Hits       301086   301086           
  Misses      61475    61475           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mergify]

Backports to the beta branch are to be avoided unless absolutely necessary for fixing bugs, security issues, and perf regressions. Changes intended for backport should be structured such that a minimum effective diff can be committed separately from any refactoring, plumbing, cleanup, etc that are not strictly necessary to achieve the goal. Any of the latter should go only into master and ride the normal stabilization schedule. Exceptions include CI/metrics changes, CLI improvements and documentation updates on a case by case basis.

[brooksprumo]