Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: simple performance bench for foyer-memory #336

Open
wants to merge 260 commits into
base: main
Choose a base branch
from
Open

feat: simple performance bench for foyer-memory #336

wants to merge 260 commits into from

Conversation

xiaguan
Copy link
Contributor

@xiaguan xiaguan commented Apr 14, 2024

What's changed and what's your intention?

Please explain IN DETAIL what the changes are in this PR and why they are needed. :D

As title

Checklist

  • I have written the necessary rustdoc comments
  • I have added the necessary unit tests and integration tests
  • I have passed make all (or make fast instead if the old tests are not modified) in my local environment.

Related issues or PRs (optional)

MrCroxx added 30 commits May 25, 2023 11:23
@MrCroxx
fix: add simple test for read only file store and fix bugs (foyer-rs#10)
7a2467c 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
feat: add foyer bench, reorg workspace (foyer-rs#11)
d6a7a28 
* feat: add foyer bench, reorg workspace

Signed-off-by: MrCroxx <[email protected]>

* sort cargo file

Signed-off-by: MrCroxx <[email protected]>

* make fmt and clippy happy

Signed-off-by: MrCroxx <[email protected]>

* fix bug

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
fix: fix bug with bench (foyer-rs#12)
3e9cc5f 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
ci: add asan test (foyer-rs#13)
4d3a8e5 
* ci: add asan test

Signed-off-by: MrCroxx <[email protected]>

* fix Cargo.toml and CI

Signed-off-by: MrCroxx <[email protected]>

* regen ci

Signed-off-by: MrCroxx <[email protected]>

* rename CI step, test asan fail

Signed-off-by: MrCroxx <[email protected]>

* regen CI

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
chore: add license checker (foyer-rs#14)
5c09e60 
* chore: add license checker

Signed-off-by: MrCroxx <[email protected]>

* fix license checker config

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
chore: wrapper cache type, reorg utils, add metrics framework (foyer-…
e38ad6a 
…rs#15)

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
fix: fix read only store recovery, impl random drop (foyer-rs#16)
f44aec1 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
feat: add metrics (foyer-rs#17)
8e80c6c 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
feat: add log support (foyer-rs#19)
79610aa 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
feat: add write stall config for read only file store (foyer-rs#20)
23e2612 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
feat: introduce instrusive indexers and collections (foyer-rs#21)
506f3ef 
* feat: introduce instrusive indexers and collections

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
feat: introduce FTL-like storage engine (foyer-rs#22)
e9714c5 
* feat: introduce FTL-like storage engine

Signed-off-by: MrCroxx <[email protected]>

* update ci

Signed-off-by: MrCroxx <[email protected]>

* sort cargo file

Signed-off-by: MrCroxx <[email protected]>

* fix memory leak

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
chore: remove unused deps (foyer-rs#28)
64b72b4 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
feat: enable direct i/o on linux target (foyer-rs#29)
8388de6 
* feat: enable direct i/o on linux target

- enable direct i/o on linux target
- refine flusher and reclaimer

Signed-off-by: MrCroxx <[email protected]>

* fix unit test

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
chore: remove unused old storage engien and other components (foyer-r…
7a5624e 
…s#30)

* chore: remove unused old storage engien and other components

Signed-off-by: MrCroxx <[email protected]>

* update ci

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
feat: foyer storage bench support flexible writers and readers (foyer…
85d8341 
…-rs#31)

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
feat: impl storage recovery (foyer-rs#32)
7c81272 
* feat: impl storage recovery

Signed-off-by: MrCroxx <[email protected]>

* chore

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
feat: introduce segment fifo eviction policy (foyer-rs#35)
ad3f125 
* feat: add segment fifo eviction policy

Signed-off-by: MrCroxx <[email protected]>

* export fifo fs store

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
feat: export mods (foyer-rs#37)
98a0af7 
* feat: export mods

Signed-off-by: MrCroxx <[email protected]>

* make cargo sort happy

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
fix: export extern crate (foyer-rs#38)
a8a56cc 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
Revert "fix: export extern crate (foyer-rs#38)" (foyer-rs#39)
09bf57a 
This reverts commit a8a56cc.
@MrCroxx
fix: disable O_DIRECT on non-linux targets (foyer-rs#40)
691054a 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
fix: deadlock (foyer-rs#43)
f067d1c 
* fix: deadlock

fix deadlocks:

1. exclusive lock was not `Send` but unsafe impled.
2. submit flush task await blocks `set_region_evictable`
3. no new reclamation task created after `set_region_evctable` if all
   write process are waiting

Signed-off-by: MrCroxx <[email protected]>

* update ci

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
chore: ignore Cargo.lock and bump toolchain (foyer-rs#46)
7fc8542 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
fix: gracefullly shutdown runners (foyer-rs#47)
8979084 
* fix: gracefullly shutdown runners

Use `Store::shutdown_runners()` to gracefully shutdown runners.`

Signed-off-by: MrCroxx <[email protected]>

* log warn if send failed instead of panic

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
chore: export storage config (foyer-rs#50)
1018bca 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
fix: AdmitAll & ReinsertNone no longer requires KV default (foyer-rs#51)
011a401 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
fix: region advance & recovery & seal (foyer-rs#53)
e61eb5d 
- fix region advance
- fix recovery slice destroy
- seal last dirty region when shutdown

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
feat: introduce prometheus metrics (foyer-rs#54)
f9548e6 
* feat: introduce prometheus metrics

Signed-off-by: MrCroxx <[email protected]>

* calc metrics

Signed-off-by: MrCroxx <[email protected]>

* fix registry metrics

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
fix: recovery test (foyer-rs#55)
3c0a0ca 
Signed-off-by: MrCroxx <[email protected]>
MrCroxx and others added 29 commits March 12, 2024 18:41
@MrCroxx
fix: fix build with trace future (foyer-rs#291)
12eff65 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
chore: bump bench tools version (foyer-rs#292)
bd3fc1a 
* chore: bump bench tools version

Signed-off-by: MrCroxx <[email protected]>

* chore: update change log, restore unreleased version

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
refactor: impl Clone for config for simplification (foyer-rs#293)
4db04a2 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
chore: bump foyer-memory to 0.1.1 (foyer-rs#294)
fabed44 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
refactor: export DefaultCacheEventListener (foyer-rs#296)
050a3d0 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
chore: bump foyer-memory version (foyer-rs#297)
524b645 
Signed-off-by: MrCroxx <[email protected]>
@xiaguan
feat: add eviction alg hit ratio bench (foyer-rs#299)
e76a499 
* feat: add eviction alg hit ratio bench

* refactor: use enum cache instead of generic cache

The enum cache is more easy to use since it only has two generic parameters. The generic cache is more like an internal interface.

Signed-off-by: xiaguan <[email protected]>

* chore : sort foyer_memory's dependencies

Signed-off-by: xiaguan <[email protected]>

* chore: use hakari to gen better dependencies

Signed-off-by: xiaguan <[email protected]>

---------

Signed-off-by: xiaguan <[email protected]>
@MrCroxx
chore: update license (foyer-rs#295)
6bea810 
* chore: update license

Signed-off-by: MrCroxx <[email protected]>

* chore: fix license checker

Signed-off-by: MrCroxx <[email protected]>

* chore: update license header

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
@Ishiihara @MrCroxx
feat: add S3Fifo eviction for memory (foyer-rs#303)
41eebba 
* Add S3Fifo eviction for memory

* fix: refine s3fifo, fix some bugs

Signed-off-by: MrCroxx <[email protected]>

* fix: fix license

Signed-off-by: MrCroxx <[email protected]>

* refactor: expose s3fifo, fix hakari, add s3fifo fuzzy test

Signed-off-by: MrCroxx <[email protected]>

* bench: add s3fifo to hit ratio bench

Signed-off-by: MrCroxx <[email protected]>

* test: add s3fifo uts

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
Co-authored-by: MrCroxx <[email protected]>
@MrCroxx
chore: bump foyer-intrusive to 0.3.1, foyer-memory to 0.1.3 (foyer-rs…
cd612f6 
…#307)

* chore: bump foyer-memory to 0.1.3

Signed-off-by: MrCroxx <[email protected]>

* chore: bump foyer-intrusive to 0.3.1

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
chore: bump foyer-storage to 0.5.1 (foyer-rs#309)
5899ee4 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
fix: fix state() for s3fifo (foyer-rs#312)
b50813b 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
chore: bump foyer-memory to 0.1.4 (foyer-rs#313)
5a410c8 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
refactor: use generic type for cursor instead of associated type (foy…
823a879 
…er-rs#314)

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
refactor: remove unstable feature let_chains and lint_reasons (foyer-…
3aa689d 
…rs#315)

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
refactor: remove usage of unstable features (foyer-rs#316)
2693669 
* refactor: remove usage of unstable features

Signed-off-by: MrCroxx <[email protected]>

* refactor: some more

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
refactor: use crate allocator_api2 to make foyer build on stable (foy…
e423dc4 
…er-rs#317)

* refactor: use crate allocator_api2 to make foyer build on stable

Signed-off-by: MrCroxx <[email protected]>

* fix: use stable pipeline for ci tests, use nightly for sanitizer

Signed-off-by: MrCroxx <[email protected]>

* fix: update ci prefix key

Signed-off-by: MrCroxx <[email protected]>

* fix: try fix ci

Signed-off-by: MrCroxx <[email protected]>

* fix: update ci nightly version

Signed-off-by: MrCroxx <[email protected]>

* chore: try resolve github issue

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
test: run CI on both linux and macos (foyer-rs#318)
3e8f334 
* test: run CI on both linux and macos

Signed-off-by: MrCroxx <[email protected]>

* fix: fix build on macos

Signed-off-by: MrCroxx <[email protected]>

* fix: try fix ci run on targets

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
doc: update README and add a simple example (foyer-rs#319)
475aee0 
* doc: update README and add a simple example

Signed-off-by: MrCroxx <[email protected]>

* chore: update Cargo.toml

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
chore: add msrv for foyer, update bager in readme (foyer-rs#321)
2abe3df 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
fix: do not truncate when open wal (foyer-rs#322)
cb2616e 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
chore: bump foyer to 0.7.0 (foyer-rs#323)
48c3990 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
refactor: loose query key bound for in-memory cache (foyer-rs#324)
cc64fc9 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
feat: impl contains for in-memory cache (foyer-rs#325)
e47c090 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
perf: add dynamic dispatch bench (foyer-rs#326)
818e3f9 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
chore: fix ci coverage (foyer-rs#331)
3fda8cc 
* chore: fix ci coverage

Signed-off-by: MrCroxx <[email protected]>

* chore: fix check

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
refactor: refine generic handle type (foyer-rs#330)
6b820fa 
Signed-off-by: MrCroxx <[email protected]>
@MrCroxx
refactor: loose bound of in-memory eviction related traits (foyer-rs#334
f590f09 
)

* refactor: remove Handle bound from Eviction

Signed-off-by: MrCroxx <[email protected]>

* refactor: loose bound of in-memory eviction related traits

Signed-off-by: MrCroxx <[email protected]>

---------

Signed-off-by: MrCroxx <[email protected]>
@xiaguan
feat: performance becnh for foyer-memory
1c7381c 
Signed-off-by: xiaguan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
foyer - Development Roadmap
Status: No status
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@xiaguan @MrCroxx @wenym1 @xxchan @TennyZhuang @Ishiihara