Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: block queue metrics and stalled submission alarm #1649
feat: block queue metrics and stalled submission alarm #1649
Changes from 39 commits
b057f25
f637985
5c771dd
b8a8c81
17c76c4
a55a7e5
ec310b3
fb97563
244f608
782c3e3
de16c8e
c5da3ea
c4e437f
b87a712
03b983b
8773021
0f127a8
04a5754
d1594d3
0916252
44c5ebc
1771b48
14c6f37
d4079fc
047bd98
d8d3bf3
69fe633
2a20535
79b2b22
e452b96
7ce931d
91f4d4b
bd4b2c1
aaacf5c
5faa0a2
365d029
0a3ae70
d6543e5
d676020
02a541f
1fb5f4d
b0f8748
File filter
Filter by extension
Conversations
Jump to
There are no files selected for viewing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
kinda curious, all OMG.Bus subscriber need to implement listener for all topics? Otherwise why we need to implement a noreply?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's sharing the same
"blocks"
topic in:ok = OMG.Bus.subscribe({:child_chain, "blocks"}, link: true)
.We can make the topic more granular but it opens a different can of worms, i.e. setting a new convention, to include in this PR 😢
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so essentially what you want to figure out is:
the difference between blocks mined minus blocks formed, correct?
If yes, why are you storing pending blocks?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mainly to track
{blknum, root_chain_height}
. There was a lot of false alarms when I tracked just themined - formed
because of the large swing in block time. And it's a large swing all the time, not just once in a while, so the alarm is barely useful with an alarm every few minutes.Some supporting info. Below is rootchain blocktime over a single day. You can see the spikes throughout the day, multiple times each hour:
And in the diagram below, if we use the average blocktime of 12.5 - 15s as usual, only 6% (369/6401) of all blocks in a day will fall in the range.
So in my trial it was impossible to stick to
mined - formed
, because over a single daymined - formed >= 4
could be from 10 seconds to 188 seconds. So tracking the chch blknum against the first root chain height it was submitted at was the most reliable way to track stalled submissions as far as I tried.Source data: https://docs.google.com/spreadsheets/d/1wzniBoO2nonRo83im0VRcaSPj_pr-ZZXKV-yiOOFVOo/edit