State metrics possible changes

[cheme]

This is a small PR where I tried to implement my comments on #4562. CC\ @NikVolf
Currently I am not very happy with the way things are registered for the state machine overlay writes, but I am not very sure it is that usefull to collect.
I feel like it is interesting as it can give an idea of the difference between the writes the runtime ask and the actual writes that propagates to rocksdb.
While writing this, I also ask myself if using directly prometheus atomic u64 could not be more straightforward.
So I am putting the PR in InProgress state for now, @pscott I think you are also touching this code.

[bkchr]

@NikVolf @cheme what are we waiting for?

[NikVolf]

@bkchr this is marked "in progress"?

[bkchr]

There was no progress in 18 days. 🤷‍♂️

[bkchr]

So I wanted to ask what the status is.

[cheme]

Oh sorry, I was waiting for feedback on wether we want those indicators or not, and then I lost trace of the PR.
This may need more tests (real use testing).
@pscott , is this a blocker? I can close it if it is.

[NikVolf]

I would eventually want to move all this stuff to prometheus for production code (but have some performance concerns).

But then we will have issue with measuring this for benchmarks

Perhaps we need to make some abstraction which can either push metrics to prometheus immediately or collect in some local struct and then consumed by benchmarks.

Anyway, this should be ok for now, the above are further considerations

[bkchr]

Sounds like you want to create an issue @NikVolf? :D

[pscott]

@cheme I don't think this PR is a blocker for my PR. :)