refactor: a more friendly dynamic_dispatch bench #332

xiaguan · 2024-04-12T12:07:34Z

What's changed and what's your intention?

Just a little bit enhancement for the bench.

Checklist

I have written the necessary rustdoc comments
I have added the necessary unit tests and integration tests
I have passed make all (or make fast instead if the old tests are not modified) in my local environment.

Related issues or PRs (optional)

Signed-off-by: MrCroxx <[email protected]>

* feat: add foyer bench, reorg workspace Signed-off-by: MrCroxx <[email protected]> * sort cargo file Signed-off-by: MrCroxx <[email protected]> * make fmt and clippy happy Signed-off-by: MrCroxx <[email protected]> * fix bug Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

Signed-off-by: MrCroxx <[email protected]>

* ci: add asan test Signed-off-by: MrCroxx <[email protected]> * fix Cargo.toml and CI Signed-off-by: MrCroxx <[email protected]> * regen ci Signed-off-by: MrCroxx <[email protected]> * rename CI step, test asan fail Signed-off-by: MrCroxx <[email protected]> * regen CI Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

* chore: add license checker Signed-off-by: MrCroxx <[email protected]> * fix license checker config Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

…rs#15) Signed-off-by: MrCroxx <[email protected]>

Signed-off-by: MrCroxx <[email protected]>

* feat: introduce instrusive indexers and collections Signed-off-by: MrCroxx <[email protected]>

* feat: introduce FTL-like storage engine Signed-off-by: MrCroxx <[email protected]> * update ci Signed-off-by: MrCroxx <[email protected]> * sort cargo file Signed-off-by: MrCroxx <[email protected]> * fix memory leak Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

Signed-off-by: MrCroxx <[email protected]>

* feat: enable direct i/o on linux target - enable direct i/o on linux target - refine flusher and reclaimer Signed-off-by: MrCroxx <[email protected]> * fix unit test Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

…s#30) * chore: remove unused old storage engien and other components Signed-off-by: MrCroxx <[email protected]> * update ci Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

…-rs#31) Signed-off-by: MrCroxx <[email protected]>

* feat: impl storage recovery Signed-off-by: MrCroxx <[email protected]> * chore Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

* feat: add segment fifo eviction policy Signed-off-by: MrCroxx <[email protected]> * export fifo fs store Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

* feat: export mods Signed-off-by: MrCroxx <[email protected]> * make cargo sort happy Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

Signed-off-by: MrCroxx <[email protected]>

This reverts commit a8a56cc.

Signed-off-by: MrCroxx <[email protected]>

* fix: deadlock fix deadlocks: 1. exclusive lock was not `Send` but unsafe impled. 2. submit flush task await blocks `set_region_evictable` 3. no new reclamation task created after `set_region_evctable` if all write process are waiting Signed-off-by: MrCroxx <[email protected]> * update ci Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

Signed-off-by: MrCroxx <[email protected]>

* fix: gracefullly shutdown runners Use `Store::shutdown_runners()` to gracefully shutdown runners.` Signed-off-by: MrCroxx <[email protected]> * log warn if send failed instead of panic Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

Signed-off-by: MrCroxx <[email protected]>

- fix region advance - fix recovery slice destroy - seal last dirty region when shutdown Signed-off-by: MrCroxx <[email protected]>

* feat: introduce prometheus metrics Signed-off-by: MrCroxx <[email protected]> * calc metrics Signed-off-by: MrCroxx <[email protected]> * fix registry metrics Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

Signed-off-by: MrCroxx <[email protected]>

* feat: add eviction alg hit ratio bench * refactor: use enum cache instead of generic cache The enum cache is more easy to use since it only has two generic parameters. The generic cache is more like an internal interface. Signed-off-by: xiaguan <[email protected]> * chore : sort foyer_memory's dependencies Signed-off-by: xiaguan <[email protected]> * chore: use hakari to gen better dependencies Signed-off-by: xiaguan <[email protected]> --------- Signed-off-by: xiaguan <[email protected]>

* chore: update license Signed-off-by: MrCroxx <[email protected]> * chore: fix license checker Signed-off-by: MrCroxx <[email protected]> * chore: update license header Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

* Add S3Fifo eviction for memory * fix: refine s3fifo, fix some bugs Signed-off-by: MrCroxx <[email protected]> * fix: fix license Signed-off-by: MrCroxx <[email protected]> * refactor: expose s3fifo, fix hakari, add s3fifo fuzzy test Signed-off-by: MrCroxx <[email protected]> * bench: add s3fifo to hit ratio bench Signed-off-by: MrCroxx <[email protected]> * test: add s3fifo uts Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]> Co-authored-by: MrCroxx <[email protected]>

…#307) * chore: bump foyer-memory to 0.1.3 Signed-off-by: MrCroxx <[email protected]> * chore: bump foyer-intrusive to 0.3.1 Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

Signed-off-by: MrCroxx <[email protected]>

…er-rs#314) Signed-off-by: MrCroxx <[email protected]>

…rs#315) Signed-off-by: MrCroxx <[email protected]>

* refactor: remove usage of unstable features Signed-off-by: MrCroxx <[email protected]> * refactor: some more Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

…er-rs#317) * refactor: use crate allocator_api2 to make foyer build on stable Signed-off-by: MrCroxx <[email protected]> * fix: use stable pipeline for ci tests, use nightly for sanitizer Signed-off-by: MrCroxx <[email protected]> * fix: update ci prefix key Signed-off-by: MrCroxx <[email protected]> * fix: try fix ci Signed-off-by: MrCroxx <[email protected]> * fix: update ci nightly version Signed-off-by: MrCroxx <[email protected]> * chore: try resolve github issue Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

* test: run CI on both linux and macos Signed-off-by: MrCroxx <[email protected]> * fix: fix build on macos Signed-off-by: MrCroxx <[email protected]> * fix: try fix ci run on targets Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

* doc: update README and add a simple example Signed-off-by: MrCroxx <[email protected]> * chore: update Cargo.toml Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

Signed-off-by: MrCroxx <[email protected]>

* chore: fix ci coverage Signed-off-by: MrCroxx <[email protected]> * chore: fix check Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

foyer-memory/benches/bench_dynamic_dispatch.rs

MrCroxx · 2024-04-12T12:37:25Z

foyer-memory/benches/bench_dynamic_dispatch.rs

+    // Warmup
+    for _ in 0..WARMUP_LOOPS {
+        let s = rand_string(thread_rng().gen_range(0..100));
+        let _ = (t.f2)(&s);
+    }


Any reason to add a warmup loop for this case? 👀

When the iteration number equals 10,000, there is significant performance variability on my machine.

With count the random generation or not? I think the variability may come from that. 🤔

Signed-off-by: MrCroxx <[email protected]>

Signed-off-by: xiaguan <[email protected]>

MrCroxx added 30 commits May 24, 2023 15:31

feat: introduce read only file store (foyer-rs#9)

24cccaf

Signed-off-by: MrCroxx <[email protected]>

fix: add simple test for read only file store and fix bugs (foyer-rs#10)

7a2467c

Signed-off-by: MrCroxx <[email protected]>

fix: fix bug with bench (foyer-rs#12)

3e9cc5f

Signed-off-by: MrCroxx <[email protected]>

chore: add license checker (foyer-rs#14)

5c09e60

* chore: add license checker Signed-off-by: MrCroxx <[email protected]> * fix license checker config Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

chore: wrapper cache type, reorg utils, add metrics framework (foyer-…

e38ad6a

…rs#15) Signed-off-by: MrCroxx <[email protected]>

fix: fix read only store recovery, impl random drop (foyer-rs#16)

f44aec1

Signed-off-by: MrCroxx <[email protected]>

feat: add metrics (foyer-rs#17)

8e80c6c

Signed-off-by: MrCroxx <[email protected]>

feat: add log support (foyer-rs#19)

79610aa

Signed-off-by: MrCroxx <[email protected]>

feat: add write stall config for read only file store (foyer-rs#20)

23e2612

Signed-off-by: MrCroxx <[email protected]>

feat: introduce instrusive indexers and collections (foyer-rs#21)

506f3ef

* feat: introduce instrusive indexers and collections Signed-off-by: MrCroxx <[email protected]>

chore: remove unused deps (foyer-rs#28)

64b72b4

Signed-off-by: MrCroxx <[email protected]>

feat: foyer storage bench support flexible writers and readers (foyer…

85d8341

…-rs#31) Signed-off-by: MrCroxx <[email protected]>

feat: impl storage recovery (foyer-rs#32)

7c81272

* feat: impl storage recovery Signed-off-by: MrCroxx <[email protected]> * chore Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

feat: introduce segment fifo eviction policy (foyer-rs#35)

ad3f125

* feat: add segment fifo eviction policy Signed-off-by: MrCroxx <[email protected]> * export fifo fs store Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

feat: export mods (foyer-rs#37)

98a0af7

* feat: export mods Signed-off-by: MrCroxx <[email protected]> * make cargo sort happy Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

fix: export extern crate (foyer-rs#38)

a8a56cc

Signed-off-by: MrCroxx <[email protected]>

Revert "fix: export extern crate (foyer-rs#38)" (foyer-rs#39)

09bf57a

This reverts commit a8a56cc.

fix: disable O_DIRECT on non-linux targets (foyer-rs#40)

691054a

Signed-off-by: MrCroxx <[email protected]>

chore: ignore Cargo.lock and bump toolchain (foyer-rs#46)

7fc8542

Signed-off-by: MrCroxx <[email protected]>

chore: export storage config (foyer-rs#50)

1018bca

Signed-off-by: MrCroxx <[email protected]>

fix: AdmitAll & ReinsertNone no longer requires KV default (foyer-rs#51)

011a401

Signed-off-by: MrCroxx <[email protected]>

fix: region advance & recovery & seal (foyer-rs#53)

e61eb5d

- fix region advance - fix recovery slice destroy - seal last dirty region when shutdown Signed-off-by: MrCroxx <[email protected]>

MrCroxx and others added 24 commits March 14, 2024 10:55

refactor: impl Clone for config for simplification (foyer-rs#293)

4db04a2

Signed-off-by: MrCroxx <[email protected]>

chore: bump foyer-memory to 0.1.1 (foyer-rs#294)

fabed44

Signed-off-by: MrCroxx <[email protected]>

refactor: export DefaultCacheEventListener (foyer-rs#296)

050a3d0

Signed-off-by: MrCroxx <[email protected]>

chore: bump foyer-memory version (foyer-rs#297)

524b645

Signed-off-by: MrCroxx <[email protected]>

chore: bump foyer-storage to 0.5.1 (foyer-rs#309)

5899ee4

Signed-off-by: MrCroxx <[email protected]>

fix: fix state() for s3fifo (foyer-rs#312)

b50813b

Signed-off-by: MrCroxx <[email protected]>

chore: bump foyer-memory to 0.1.4 (foyer-rs#313)

5a410c8

Signed-off-by: MrCroxx <[email protected]>

refactor: use generic type for cursor instead of associated type (foy…

823a879

…er-rs#314) Signed-off-by: MrCroxx <[email protected]>

refactor: remove unstable feature let_chains and lint_reasons (foyer-…

3aa689d

…rs#315) Signed-off-by: MrCroxx <[email protected]>

refactor: remove usage of unstable features (foyer-rs#316)

2693669

* refactor: remove usage of unstable features Signed-off-by: MrCroxx <[email protected]> * refactor: some more Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

doc: update README and add a simple example (foyer-rs#319)

475aee0

* doc: update README and add a simple example Signed-off-by: MrCroxx <[email protected]> * chore: update Cargo.toml Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

chore: add msrv for foyer, update bager in readme (foyer-rs#321)

2abe3df

Signed-off-by: MrCroxx <[email protected]>

fix: do not truncate when open wal (foyer-rs#322)

cb2616e

Signed-off-by: MrCroxx <[email protected]>

chore: bump foyer to 0.7.0 (foyer-rs#323)

48c3990

Signed-off-by: MrCroxx <[email protected]>

refactor: loose query key bound for in-memory cache (foyer-rs#324)

cc64fc9

Signed-off-by: MrCroxx <[email protected]>

feat: impl contains for in-memory cache (foyer-rs#325)

e47c090

Signed-off-by: MrCroxx <[email protected]>

perf: add dynamic dispatch bench (foyer-rs#326)

818e3f9

Signed-off-by: MrCroxx <[email protected]>

chore: fix ci coverage (foyer-rs#331)

3fda8cc

* chore: fix ci coverage Signed-off-by: MrCroxx <[email protected]> * chore: fix check Signed-off-by: MrCroxx <[email protected]> --------- Signed-off-by: MrCroxx <[email protected]>

MrCroxx reviewed Apr 12, 2024

View reviewed changes

refactor: refine generic handle type (foyer-rs#330)

6b820fa

Signed-off-by: MrCroxx <[email protected]>

xiaguan force-pushed the refactor_dynamic_bench branch from eecb910 to c247346 Compare April 12, 2024 13:22

refactor: a more friendly dynamic_dispatch bench

41b6e07

Signed-off-by: xiaguan <[email protected]>

xiaguan force-pushed the refactor_dynamic_bench branch from c247346 to 41b6e07 Compare April 12, 2024 13:32

MrCroxx force-pushed the main branch from 04d5ac3 to a2b91e3 Compare April 17, 2024 01:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: a more friendly dynamic_dispatch bench #332

refactor: a more friendly dynamic_dispatch bench #332

xiaguan commented Apr 12, 2024

MrCroxx Apr 12, 2024

xiaguan Apr 12, 2024

MrCroxx Apr 12, 2024

xiaguan Apr 12, 2024

refactor: a more friendly dynamic_dispatch bench #332

Are you sure you want to change the base?

refactor: a more friendly dynamic_dispatch bench #332

Conversation

xiaguan commented Apr 12, 2024

What's changed and what's your intention?

Checklist

Related issues or PRs (optional)

MrCroxx Apr 12, 2024

Choose a reason for hiding this comment

xiaguan Apr 12, 2024

Choose a reason for hiding this comment

MrCroxx Apr 12, 2024

Choose a reason for hiding this comment

xiaguan Apr 12, 2024

Choose a reason for hiding this comment