Enable concurrent write to both stage files and object storage to get 5GB/s write throughput

[polyrabbit]

Currently, the writeback mode works by writing the block to the staging files first, and then uploading them to the object storage. So the throughput of the writeback mode is limited by the speed of disk. And the max write concurrency is buffer-size / block-size which could be overwhelming for some disks.

By limiting concurrent writes to staged files and spill others to object backend, we can have concurrent uploads to two backends, and theoretically the write throughput equals to sum of the two (see following experiment).

By setting --max-stage-write=10, we got a write throughput of 5GB/s, almost double the speed of original write. And it lasts for a long time, as long as we have enough space for stage files

[jiefenghuang]

This modification may impact the performance in small file scenarios, which are the main focus of write-back optimization.

[polyrabbit]

Then, we expanded write-back scenarios :)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 63.38%. Comparing base (f553dcf) to head (3f64165).
Report is 5 commits behind head on main.

❗ Current head 3f64165 differs from pull request most recent head 3e36f8a. Consider uploading reports for the commit 3e36f8a to get more accurate results

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #4743       +/-   ##
===========================================
+ Coverage   25.01%   63.38%   +38.37%     
===========================================
  Files          25       24        -1     
  Lines       16011    15906      -105     
===========================================
+ Hits         4005    10082     +6077     
+ Misses      11518     4435     -7083     
- Partials      488     1389      +901     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jiefenghuang]

Then, we expanded write-back scenarios :)

ok, but I think if the write performance of large files is not satisfactory, it's a matter to consider for object storage. Allocating the cost of increasing disk cache on this side towards improving object storage bandwidth might be a more direct approach.

[polyrabbit]

I don't get it. Why this change increases disk cache? it only rearranges the upload order, trying to take full advantage of both storage backend.

Isn't it better if we can improve performance by optimizing software instead of increasing hardware?