Skip to content

statement-store: add prometheus metrics for observability#10976

Merged
DenzelPenzel merged 8 commits intomasterfrom
denzelpenzel/statement-store-metrics
Feb 19, 2026
Merged

statement-store: add prometheus metrics for observability#10976
DenzelPenzel merged 8 commits intomasterfrom
denzelpenzel/statement-store-metrics

Conversation

@DenzelPenzel
Copy link
Copy Markdown
Contributor

@DenzelPenzel DenzelPenzel commented Feb 3, 2026

Description

Implemented #10944

Changes:

  • Add storage state metrics (statement count, byte usage, capacity utilization)
  • Add network/gossip metrics (connected peers, received statements)

@DenzelPenzel DenzelPenzel added the R0-no-crate-publish-required The change does not require any crates to be re-published. label Feb 3, 2026
@DenzelPenzel DenzelPenzel marked this pull request as draft February 3, 2026 22:20
@DenzelPenzel DenzelPenzel force-pushed the denzelpenzel/statement-store-metrics branch from fae1ef3 to ba4faa7 Compare February 3, 2026 22:25
@DenzelPenzel DenzelPenzel changed the title [WIP] statement-store: add prometheus metrics for observability statement-store: add prometheus metrics for observability Feb 4, 2026
@P1sar P1sar requested review from AndreiEres and alexggh February 4, 2026 15:06
alexggh
alexggh reviewed Feb 5, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@alexggh alexggh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work, it is a good start.

I would also add some histograms for how long .submit takes and notify_matching_filters.

Can you also build the dashboards that show us this metrics ?

Comment thread substrate/client/network/statement/src/lib.rs Outdated
Comment thread substrate/client/network/statement/src/lib.rs Outdated
Comment thread substrate/client/statement-store/src/lib.rs Outdated
@DenzelPenzel
Copy link
Copy Markdown
Contributor Author

DenzelPenzel commented Feb 6, 2026 •
edited
Loading

I would also add some histograms for how long .submit takes and notify_matching_filters.
Can you also build the dashboards that show us this metrics ?

  1. Added three new metrics for measurement.
    submit_duration_seconds, validation_duration_seconds, db_write_duration_seconds

  2. Dashboards are ready as well.

  3. Added metrics docs https://github.com/paritytech/polkadot-sdk/blob/393808be29db066321dba550e9eb23d259938da1/substrate/client/statement-store/Metrics.md
    cc @P1sar

@DenzelPenzel DenzelPenzel force-pushed the denzelpenzel/statement-store-metrics branch from a7cffaf to 108b1e5 Compare February 6, 2026 19:10
@DenzelPenzel DenzelPenzel force-pushed the denzelpenzel/statement-store-metrics branch from 16e117f to 34f42d6 Compare February 9, 2026 09:39
@DenzelPenzel DenzelPenzel marked this pull request as ready for review February 9, 2026 09:41
@DenzelPenzel
statement-store: clean up non-priority metrics from dashboards
2046fb8 
Removed low-priority and relative metrics from Grafana dashboards to focus on high-value signals per USE/RED methodology.

Removed from statement-store dashboard:
- Capacity Utilization (relative metrics)
- Statements Over Time (informational)
- Accounts Over Time (informational)
- Cumulative Totals section (lifetime counters)

Removed from network dashboard:
- Network Cumulative Totals (informational)
- Propagation Chunk Rate (optimization detail)
- Average Chunk Size (relative metric)
- Known Statements Received (lower priority)
- Network Inefficiencies detail
- Pending Validations Over Time (duplicate)
@DenzelPenzel DenzelPenzel force-pushed the denzelpenzel/statement-store-metrics branch 2 times, most recently from 393808b to d06d9c0 Compare February 10, 2026 22:09
AndreiEres
AndreiEres reviewed Feb 11, 2026
View reviewed changes
Comment thread substrate/client/network/statement/src/lib.rs
@DenzelPenzel DenzelPenzel force-pushed the denzelpenzel/statement-store-metrics branch from d06d9c0 to 45b6185 Compare February 11, 2026 14:02
@DenzelPenzel DenzelPenzel requested a review from alexggh February 11, 2026 21:46
@DenzelPenzel DenzelPenzel force-pushed the denzelpenzel/statement-store-metrics branch from 45b6185 to 54cf034 Compare February 13, 2026 12:14
alexggh
alexggh reviewed Feb 13, 2026
View reviewed changes
Comment thread substrate/client/network/statement/src/lib.rs Outdated
Comment thread substrate/client/network/statement/src/lib.rs Outdated
Comment thread substrate/client/network/statement/src/lib.rs Outdated
Comment thread substrate/client/network/statement/src/lib.rs Outdated
Comment thread substrate/client/statement-store/src/lib.rs Outdated
Comment thread substrate/client/statement-store/src/lib.rs Outdated
@DenzelPenzel DenzelPenzel force-pushed the denzelpenzel/statement-store-metrics branch 2 times, most recently from a6b3348 to 641a3f6 Compare February 13, 2026 16:42
@DenzelPenzel DenzelPenzel force-pushed the denzelpenzel/statement-store-metrics branch from 7592ca5 to 73c165f Compare February 13, 2026 22:46
alexggh
alexggh reviewed Feb 16, 2026
View reviewed changes
Comment thread substrate/client/network/statement/benches/statement_network.rs Outdated
@DenzelPenzel DenzelPenzel force-pushed the denzelpenzel/statement-store-metrics branch from 73c165f to b6f36a9 Compare February 16, 2026 15:57
@DenzelPenzel DenzelPenzel force-pushed the denzelpenzel/statement-store-metrics branch 6 times, most recently from 800883e to e406e3e Compare February 16, 2026 16:27
@DenzelPenzel DenzelPenzel force-pushed the denzelpenzel/statement-store-metrics branch from e406e3e to 16c8342 Compare February 16, 2026 16:41
alexggh
alexggh approved these changes Feb 17, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@alexggh alexggh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Comment thread substrate/client/statement-store/src/lib.rs
AndreiEres
AndreiEres approved these changes Feb 19, 2026
View reviewed changes
Comment thread substrate/client/statement-store/Metrics.md
@@ -0,0 +1,348 @@
# Statement Store & Network Dashboards
Copy link
Copy Markdown
Contributor

@AndreiEres AndreiEres Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO very soon it's going to become outdated. Better to add the parts that matter to the docs of the metrics in the code

Comment thread substrate/client/network/statement/src/lib.rs Outdated
Comment thread substrate/client/statement-store/src/lib.rs Outdated
Comment thread substrate/client/statement-store/src/lib.rs Outdated
Comment thread substrate/client/statement-store/src/lib.rs Outdated
@DenzelPenzel DenzelPenzel added this pull request to the merge queue Feb 19, 2026
Merged via the queue into master with commit 13aca01 Feb 19, 2026
242 of 246 checks passed
@DenzelPenzel DenzelPenzel deleted the denzelpenzel/statement-store-metrics branch February 19, 2026 15:11
@DenzelPenzel DenzelPenzel added the A4-backport-stable2603 Pull request must be backported to the stable2603 release branch label Mar 3, 2026
@paritytech-release-backport-bot
Copy link
Copy Markdown

paritytech-release-backport-bot Bot commented Mar 3, 2026

Created backport PR for stable2603:

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-10976-to-stable2603
git worktree add --checkout .worktree/backport-10976-to-stable2603 backport-10976-to-stable2603
cd .worktree/backport-10976-to-stable2603
git reset --hard HEAD^
git cherry-pick -x 13aca011ecc3184fcaa853526169381104b4c4e3
git push --force-with-lease

@paritytech-release-backport-bot paritytech-release-backport-bot Bot mentioned this pull request Mar 3, 2026
Merged
DenzelPenzel added a commit that referenced this pull request Mar 3, 2026
@DenzelPenzel
statement-store: add prometheus metrics for observability (#10976)
db34eec 
Implemented #10944

Changes:
- Add storage state metrics (statement count, byte usage, capacity
utilization)
- Add network/gossip metrics (connected peers, received statements)

(cherry picked from commit 13aca01)
DenzelPenzel added a commit that referenced this pull request Mar 5, 2026
@paritytech-release-backport-bot @DenzelPenzel
[stable2603] Backport #10976 (#11238)
928ed64 
Backport #10976 into `stable2603` from DenzelPenzel.

See the
[documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md)
on how to use this bot.

<!--
  # To be used by other automation, do not modify:
  original-pr-number: #${pull_number}
-->

Co-authored-by: DenzelPenzel <15388928+DenzelPenzel@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@AndreiEres AndreiEres AndreiEres approved these changes

@alexggh alexggh alexggh approved these changes

Assignees

No one assigned

Labels

A4-backport-stable2603 Pull request must be backported to the stable2603 release branch R0-no-crate-publish-required The change does not require any crates to be re-published.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@DenzelPenzel @AndreiEres @alexggh