Skip to content

chore(sequencer)!: add metrics#1248

Merged
Fraser999 merged 1 commit intomainfrom
fraser/add-check-tx-metrics
Jul 9, 2024
Merged

chore(sequencer)!: add metrics#1248
Fraser999 merged 1 commit intomainfrom
fraser/add-check-tx-metrics

Conversation

@Fraser999
Copy link
Contributor

@Fraser999 Fraser999 commented Jul 8, 2024 •
edited
Loading

Summary

This adds further metrics to the sequencer.

Background

This should help diagnose block production slowdown when the sequencer is stress-tested.

Changes

  • Added metrics (see below for list).
  • Enabled cnidarium metrics.

Note that all histograms are still rendered as Prometheus summaries for now. I have an open PR which will make it simple to provide buckets for histograms, after which they will be rendered as true histograms.

Testing

Testing will likewise be relatively simple once #1192 is merged. In the meantime, I ran the smoke test using the code in this PR and manually checked the new metrics are available and appear sane.

Metrics

  • Added astria_sequencer_check_tx_duration_seconds histograms with the following labels:
    • length check and parse raw tx
    • stateless check
    • nonce check
    • chain id check
    • balance check
    • check for removal
    • insert to app mempool
  • Added astria_sequencer_actions_per_transaction_in_mempool histogram
  • Added astria_sequencer_transaction_in_mempool_size_bytes histogram
  • Added astria_sequencer_transactions_in_mempool_total gauge
  • Enabled cnidarium_get_raw_duration_seconds histogram
  • Enabled cnidarium_nonverifiable_get_raw_duration_seconds histogram

Related Issues

Closes #1247.

@Fraser999 Fraser999 requested a review from a team as a code owner July 8, 2024 20:51
@Fraser999 Fraser999 requested a review from SuperFluffy July 8, 2024 20:51
@github-actions github-actions bot added the sequencer pertaining to the astria-sequencer crate label Jul 8, 2024
noot
noot reviewed Jul 9, 2024
View reviewed changes
crates/astria-sequencer/src/metrics.rs
Comment on lines +332 to +343
pub(crate) fn record_actions_per_transaction_in_mempool(&self, count: usize) {
// allow: precision loss is unlikely (values too small) but also unimportant in histograms.
#[allow(clippy::cast_precision_loss)]
self.actions_per_transaction_in_mempool.record(count as f64);
}

pub(crate) fn record_transaction_in_mempool_size_bytes(&self, count: usize) {
// allow: precision loss is unlikely (values too small) but also unimportant in histograms.
#[allow(clippy::cast_precision_loss)]
self.transaction_in_mempool_size_bytes.record(count as f64);
}

Copy link
Contributor

@noot noot Jul 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these are set every time a tx is inserted (and is per-tx) right? would it be more/less useful to also have a mempool total?

Copy link
Contributor Author

@Fraser999 Fraser999 Jul 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was wondering about that too. Thing is, it's relatively cheap to get the number of txs in the mempool (call to lock the RwLock and then a call to len()), but it could be expensive to track either of these (e.g. lock then iterate all txs).

If we do want this, it's probably best to track this inside the mempool. We'd want the tx size to be passed in via insert to avoid having to convert the signed tx back into PB form to reserialize it.

I'll merge this PR as is, and if anyone thinks we should add these metrics, I can do a follow-up PR.

@Fraser999 Fraser999 added this pull request to the merge queue Jul 9, 2024
Merged via the queue into main with commit 0fe870c Jul 9, 2024
@Fraser999 Fraser999 deleted the fraser/add-check-tx-metrics branch July 9, 2024 15:45
@Fraser999 Fraser999 mentioned this pull request Jul 9, 2024
Closed
steezeburger added a commit that referenced this pull request Jul 15, 2024
@steezeburger
Merge branch 'main' into fix/hermes-chart
5a3c8e7 
* main:
  feat(cli): add cmd to collect withdrawal events and submit as actions (#1261)
  fix(core, bridge, sequencer)!: dismabiguate return addresses (#1266)
  fix(withdrawer): support withdrawer address that differs from bridge address   (#1262)
  (core, sequencer)!: generate serde traits impls for all protocol protobufs (#1260)
  fix(charts): add resources for sequencer/cometbft (#1254)
  chore(sequencer)!: add metrics (#1248)
  fix(sequencer-utils): fixes issue in `parse_blob` tests (#1243)
  feat(core, proto)!: make bridge unlock memo string (#1244)
  fix(conductor): don't panic during panic (#1252)
  feat(core)!: lowerCamelCase for protobuf json mapping (#1250)
  refactor(bridge-withdrawer)!: refactor startup to a separate subtask and remove balance check from startup (#1190)
  fix: rollup archive node configurations (#1249)
  refactor(core, bridge-withdrawer)!: move bridge-unlock memo to core (#1245)
  fix(sequencer)!: store native asset ibc->trace mapping in init_chain (#1242)
steezeburger added a commit that referenced this pull request Jul 19, 2024
@steezeburger
Merge branch 'main' into joroshiba/breakup-rollup
ec89454 
* main: (24 commits)
  chore: update `bytes` and `ics23` crates (#1279)
  fix(sequencer): improve and fix instrumentation (#1255)
  feature(charts): hermes chart fixes, bech32 updates, ibc bridge test (#1130)
  chore(cli): remove unused rollup cli code (#1275)
  chore(test): use a temporary file to not pollute the workspace (#1269)
  chore(sequencer): add mempool benchmarks (#1238)
  fix(bridge-withdrawer)!: fix nonce handling (#1215)
  feat(cli, bridge-withdrawer)!: share code between cli and service (#1270)
  feat(cli): add cmd to collect withdrawal events and submit as actions (#1261)
  fix(core, bridge, sequencer)!: dismabiguate return addresses (#1266)
  fix(withdrawer): support withdrawer address that differs from bridge address   (#1262)
  (core, sequencer)!: generate serde traits impls for all protocol protobufs (#1260)
  fix(charts): add resources for sequencer/cometbft (#1254)
  chore(sequencer)!: add metrics (#1248)
  fix(sequencer-utils): fixes issue in `parse_blob` tests (#1243)
  feat(core, proto)!: make bridge unlock memo string (#1244)
  fix(conductor): don't panic during panic (#1252)
  feat(core)!: lowerCamelCase for protobuf json mapping (#1250)
  refactor(bridge-withdrawer)!: refactor startup to a separate subtask and remove balance check from startup (#1190)
  fix: rollup archive node configurations (#1249)
  ...
bharath-123 pushed a commit that referenced this pull request Jul 25, 2024
@Fraser999 @bharath-123
chore(sequencer)!: add metrics (#1248)
049e100 
## Summary
This adds further metrics to the sequencer.

## Background
This should help diagnose block production slowdown when the sequencer
is stress-tested.

## Changes
- Added metrics (see below for list).
- Enabled `cnidarium` metrics.

Note that all histograms are still rendered as Prometheus summaries for
now. I have [an open PR](#1192)
which will make it simple to provide buckets for histograms, after which
they will be rendered as true histograms.

## Testing
Testing will likewise be relatively simple once #1192 is merged. In the
meantime, I ran the smoke test using the code in this PR and manually
checked the new metrics are available and appear sane.

## Metrics
- Added `astria_sequencer_check_tx_duration_seconds` histograms with the
following labels:
    - `length check and parse raw tx`
    - `stateless check`
    - `nonce check`
    - `chain id check`
    - `balance check`
    - `check for removal`
    - `insert to app mempool`
- Added `astria_sequencer_actions_per_transaction_in_mempool` histogram
- Added `astria_sequencer_transaction_in_mempool_size_bytes` histogram
- Added `astria_sequencer_transactions_in_mempool_total` gauge
- Enabled `cnidarium_get_raw_duration_seconds` histogram
- Enabled `cnidarium_nonverifiable_get_raw_duration_seconds` histogram

## Related Issues
Closes #1247.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@SuperFluffy SuperFluffy Awaiting requested review from SuperFluffy SuperFluffy is a code owner automatically assigned from astriaorg/rust-reviewers

1 more reviewer

@noot noot noot approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

sequencer pertaining to the astria-sequencer crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Provide new sequencer metrics related to check_tx.

2 participants

@Fraser999 @noot