feat: support disabled, unified, and separated runtime

[MrCroxx]

What's changed and what's your intention?

Please explain IN DETAIL what the changes are in this PR and why they are needed. :D

Support disabled, unified, and separated runtime to reduce unnecessary thread synchronization overhead.

Checklist

• I have written the necessary rustdoc comments
• I have added the necessary unit tests and integration tests
• I have passed make all (or make fast instead if the old tests are not modified) in my local environment.

Related issues or PRs (optional)

#651

[codecov]

Codecov Report

Attention: Patch coverage is 51.74419% with 83 lines in your changes missing coverage. Please review.

Project coverage is 82.66%. Comparing base (a3e0da8) to head (0efcc47).
Report is 1 commits behind head on main.

Files	Patch %	Lines
foyer-storage/src/store.rs	53.65%	57 Missing ⚠️
foyer-bench/src/main.rs	23.33%	23 Missing ⚠️
foyer-storage/src/large/generic.rs	86.66%	2 Missing ⚠️
foyer-storage/src/storage/noop.rs	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #652      +/-   ##
==========================================
+ Coverage   82.64%   82.66%   +0.01%     
==========================================
  Files          66       66              
  Lines        8068     8115      +47     
==========================================
+ Hits         6668     6708      +40     
- Misses       1400     1407       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[MrCroxx]

Benchmark

unified (baseline)

cargo build --release --features jemalloc && PATH=/home/mrcroxx/.local/bin:$PATH RUST_BACKTRACE=1 RUST_LOG=info flamegraph -- ./target/release/foyer-bench --file /dev/nvme2n1 --mem 64 --disk 1048576 --region-size 64 --get-range 10000 --flushers 4 --reclaimers 4 --time 30 --writers 256 --w-rate 4 --admission-rate-limit 500 --readers 32 --r-rate 32 --metrics --warm-up 5 --runtime unified

Output:

Total:
disk total iops: 12908.6
disk total throughput: 1.2 GiB/s
disk read iops: 11364.3
disk read throughput: 754.7 MiB/s
disk write iops: 1544.3
disk write throughput: 499.4 MiB/s
insert iops: 14050.1/s
insert throughput: 878.2 MiB/s
insert lat p50: 12us
insert lat p90: 29us
insert lat p99: 106us
insert lat p999: 425us
insert lat p9999: 1151us
insert lat p99999: 1863us
insert lat pmax: 2175us
get iops: 34916.5/s
get miss: 59.78% 
get throughput: 877.7 MiB/s
get hit lat p50: 261us
get hit lat p90: 583us
get hit lat p99: 2207us
get hit lat p999: 2927us
get hit lat p9999: 5183us
get hit lat p99999: 6111us
get hit lat pmax: 6239us
get miss lat p50: 19us
get miss lat p90: 65us
get miss lat p99: 531us
get miss lat p999: 1207us
get miss lat p9999: 1655us
get miss lat p99999: 2007us
get miss lat pmax: 2127us

separated (with worker threads control)

cargo build --release --features jemalloc && PATH=/home/mrcroxx/.local/bin:$PATH RUST_BACKTRACE=1 RUST_LOG=info flamegraph -- ./target/release/foyer-bench --file /dev/nvme2n1 --mem 64 --disk 1048576 --region-size 64 --get-range 10000 --flushers 4 --reclaimers 4 --time 30 --writers 256 --w-rate 4 --admission-rate-limit 500 --readers 32 --r-rate 32 --metrics --warm-up 5 --runtime separated --user-runtime-worker-threads 8 --read-runtime-worker-threads 8 --write-runtime-worker-threads 4

Output:

Total:
disk total iops: 12647.9
disk total throughput: 1.2 GiB/s
disk read iops: 11359.7
disk read throughput: 754.4 MiB/s
disk write iops: 1288.3
disk write throughput: 500.0 MiB/s
insert iops: 14049.9/s
insert throughput: 878.2 MiB/s
insert lat p50: 5us
insert lat p90: 9us
insert lat p99: 15us
insert lat p999: 26us
insert lat p9999: 56us
insert lat p99999: 317us
insert lat pmax: 659us
get iops: 35750.9/s
get miss: 60.72% 
get throughput: 877.7 MiB/s
get hit lat p50: 245us
get hit lat p90: 451us
get hit lat p99: 2047us
get hit lat p999: 2767us
get hit lat p9999: 4703us
get hit lat p99999: 6015us
get hit lat pmax: 6047us
get miss lat p50: 18us
get miss lat p90: 33us
get miss lat p99: 100us
get miss lat p999: 285us
get miss lat p9999: 345us
get miss lat p99999: 391us
get miss lat pmax: 453us

[MrCroxx]

flamegraph

Configuration is listed above.

unified

separated