Skip to content

I/O statistic for key-value databases#294

Merged
arkpar merged 17 commits into
masterfrom
nv-io-stats
Jan 2, 2020
Merged

I/O statistic for key-value databases#294
arkpar merged 17 commits into
masterfrom
nv-io-stats

Conversation

@NikVolf
Copy link
Copy Markdown
Contributor

@NikVolf NikVolf commented Dec 26, 2019 •
edited
Loading

This provides basic i/o stats api for key-value databases.

User can:

  1. Query overall stats. They persist during all database instance lifetime.
  2. Query intermediate stats since the last querying.

@arkpar
Copy link
Copy Markdown
Member

arkpar commented Dec 26, 2019

Would be great to have byte stats as well. Number of bytes written to the database before any amplification. I.e. combined key and value size.

@arkpar
Copy link
Copy Markdown
Member

arkpar commented Dec 26, 2019

Is there a way to get stats without clearing them? I'd expect stats to be accumulating over all of the run time.

@NikVolf
Copy link
Copy Markdown
Contributor Author

NikVolf commented Dec 26, 2019

Is there a way to get stats without clearing them? I'd expect stats to be accumulating over all of the run time.

I'll add another function for it

@NikVolf
Copy link
Copy Markdown
Contributor Author

NikVolf commented Dec 26, 2019

@arkpar addressed!

arkpar
arkpar reviewed Dec 30, 2019
View reviewed changes
Comment thread kvdb-rocksdb/src/lib.rs Outdated
arkpar
arkpar reviewed Dec 30, 2019
View reviewed changes
Comment thread kvdb-rocksdb/src/lib.rs Outdated
arkpar
arkpar reviewed Dec 30, 2019
View reviewed changes
Comment thread kvdb/src/io_stats.rs Outdated
arkpar
arkpar reviewed Dec 30, 2019
View reviewed changes
Comment thread kvdb/src/io_stats.rs
dvdplm
dvdplm reviewed Dec 30, 2019
View reviewed changes
Comment thread kvdb-rocksdb/src/lib.rs Outdated
Comment thread kvdb-rocksdb/src/lib.rs Outdated
Comment thread kvdb-rocksdb/src/lib.rs Outdated
Comment thread kvdb-rocksdb/src/lib.rs Outdated
Comment thread kvdb-rocksdb/src/lib.rs Outdated
@NikVolf
Copy link
Copy Markdown
Contributor Author

NikVolf commented Dec 30, 2019

@dvdplm @arkpar everything addressed!

ordian
ordian reviewed Dec 31, 2019
View reviewed changes
Comment thread kvdb-rocksdb/src/lib.rs
Comment thread kvdb-rocksdb/src/lib.rs Outdated
Comment thread kvdb/src/io_stats.rs
@NikVolf NikVolf changed the title I/O statistic for key-value databases [wip] I/O statistic for key-value databases Jan 1, 2020
@NikVolf NikVolf changed the title [wip] I/O statistic for key-value databases I/O statistic for key-value databases Jan 1, 2020
@NikVolf
Copy link
Copy Markdown
Contributor Author

NikVolf commented Jan 1, 2020

Hey guys, I decided to refactor a bit.

The main change is now overall statistics is always kept, so if user cares only about overall stats, he will always have it, no matter if any intermediate queries were performed.

Pls re-review @arkpar @dvdplm @ordian

ordian
ordian approved these changes Jan 1, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@ordian ordian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

ordian
ordian reviewed Jan 1, 2020
View reviewed changes
Comment thread kvdb/src/lib.rs Outdated
Comment thread kvdb-rocksdb/src/stats.rs Outdated
Comment thread kvdb-rocksdb/src/stats.rs Outdated
@arkpar arkpar merged commit fe98963 into master Jan 2, 2020
@arkpar arkpar deleted the nv-io-stats branch January 2, 2020 11:10
dvdplm
dvdplm reviewed Jan 2, 2020
View reviewed changes
Comment thread kvdb-rocksdb/src/lib.rs
.map(|r| r.map(|v| v.to_vec()))
.map_err(other_io_err),
None => {
let aquired_val = cfs
Copy link
Copy Markdown
Contributor

@dvdplm dvdplm Jan 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo here: "acquire"

dvdplm added a commit that referenced this pull request Jan 14, 2020
@dvdplm
Merge branch 'master' into dp/chore/secp256k1-from-upstream
48a2ef6 
* master: (27 commits)
  update parity-util-mem (#309)
  Update features and feature dependencies (#307)
  Use proper memory queries to rocksdb (#308)
  Draft version updates and changelog (#299)
  Use custom error type for `from_hex` (#305)
  Fix typo. (#303)
  kvdb: remove KeyValueDBHandler (#304)
  extract common kvdb tests into a crate (#301)
  Expose to_hex and from_hex from impl-serde (#302)
  Add a top level function to parity-util-mem (#298)
  I/O statistic for key-value databases (#294)
  use for_tuples (#300)
  Add memory extensions for LRUCache, hashbrown (#293)
  Add memory stats for kvdb-s (#292)
  Provide #[derive(MallocSizeOf)] that is actually working (#291)
  Move and extend impls for locking primitives (#290)
  Update uint README (#288)
  Update README.md (#287)
  [ci]: remove feature flags in virtual workspace (#289)
  Prepare releases (#286)
  ...
@ordian ordian mentioned this pull request Feb 3, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@arkpar arkpar arkpar approved these changes

+2 more reviewers

@dvdplm dvdplm dvdplm left review comments

@ordian ordian ordian approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

A8-looksgood

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@NikVolf @arkpar @dvdplm @ordian