feat: add transactions.mempool.lockedTransactions and chainlocks.blockHeight stats #6237

PastaPastaPasta · 2024-08-28T16:07:24Z

Issue being fixed or feature implemented

Add basic lockedTransactions count and chain locked block height to stats

What was done?

How Has This Been Tested?

Will be tested once this has a docker image

Breaking Changes

Checklist:

Go over all the following points, and put an x in all the boxes that apply.

I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have added or updated relevant unit/integration/functional/e2e tests
I have made corresponding changes to the documentation
I have assigned this pull request to a milestone (for repository code-owners and collaborators only)

DashCoreAutoGuix · 2024-08-28T17:16:05Z

Guix Automation has began to build this PR tagged as v21.1.0-devpr6237.ab3598b1. A new comment will be made when the image is pushed.

DashCoreAutoGuix · 2024-08-28T18:01:32Z

Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v21.1.0-devpr6237.ab3598b1. The image should be on dockerhub soon.

DashCoreAutoGuix · 2024-08-29T00:04:04Z

Guix Automation has began to build this PR tagged as v21.1.0-devpr6237.763e4fcc. A new comment will be made when the image is pushed.

DashCoreAutoGuix · 2024-08-29T00:47:51Z

Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v21.1.0-devpr6237.763e4fcc. The image should be on dockerhub soon.

DashCoreAutoGuix · 2024-08-29T00:48:04Z

Guix Automation has began to build this PR tagged as v21.1.0-devpr6237.63241c25. A new comment will be made when the image is pushed.

DashCoreAutoGuix · 2024-08-29T01:32:11Z

Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v21.1.0-devpr6237.63241c25. The image should be on dockerhub soon.

src/validation.cpp

DashCoreAutoGuix · 2024-09-10T02:56:04Z

Guix Automation has began to build this PR tagged as v21.1.0-devpr6237.2f26f788. A new comment will be made when the image is pushed.

PastaPastaPasta · 2024-09-10T02:56:11Z

Applied

src/llmq/instantsend.cpp

DashCoreAutoGuix · 2024-09-10T04:11:23Z

Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v21.1.0-devpr6237.2f26f788. The image should be on dockerhub soon.

UdjinM6 · 2024-09-10T13:46:49Z

pls see 7aa0a48 + clang format is complaining

4faf35f chore: add `stats` as a pull request header scope (Kittywhiskers Van Gogh) Pull request description: ## Additional Information Would allow tagging #5167, #6267 and #6237 as `feat(stats)`. ## Breaking Changes None. ## Checklist: - [x] I have performed a self-review of my own code **(note: N/A)** - [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)** - [x] I have added or updated relevant unit/integration/functional/e2e tests **(note: N/A)** - [x] I have made corresponding changes to the documentation **(note: N/A)** - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_ **(note: N/A)** ACKs for top commit: PastaPastaPasta: utACK 4faf35f UdjinM6: utACK 4faf35f thephez: utACK 4faf35f Tree-SHA512: 25cc28792351852b9e920d980b8814d93274bc53d22ce63fb1a5bf32821b10902d22b384a408e1c4a7b97239e53e235c45ac008d0f82e1afe5d6071b392acb47

github-actions · 2024-09-11T16:24:19Z

This pull request has conflicts, please rebase.

knst · 2024-11-25T14:34:07Z

src/llmq/instantsend.h

    std::unordered_set<uint256, StaticSaltedHasher> pendingRetryTxs GUARDED_BY(cs_pendingRetry);

+    mutable Mutex cs_timingsTxSeen;
+    std::unordered_map<uint256, int64_t, StaticSaltedHasher> timingsTxSeen GUARDED_BY(cs_timingsTxSeen);


it seems as this map can indefinitely grow and eat ~~all~~ significant amount of RAM

resolved in d5dbbdb

knst · 2024-11-25T14:38:20Z

src/llmq/instantsend.cpp

+        if (auto it = timingsTxSeen.find(tx->GetHash()); it == timingsTxSeen.end()) {
+            timingsTxSeen[tx->GetHash()] = GetTimeMillis();
+        }


let's avoid double lookup on map here:

Suggested change

if (auto it = timingsTxSeen.find(tx->GetHash()); it == timingsTxSeen.end()) {

timingsTxSeen[tx->GetHash()] = GetTimeMillis();

}

timingsTxSeen.try_emplace(tx->GetHash(), GetTimeMillis());

emplace() and try_emplace() are similar, but difference in performance - one is expecting success by default, other is expecting failure by default

proof that it works:

$ cat a.cpp #include <iostream> #include <unordered_map> using namespace std; int main() { unordered_map<int, int> m; if (m.try_emplace(2, 3).second) cout << "OK" << endl; else cout << "FAIL" << endl; if (m.try_emplace(3, 4).second) cout << "OK" << endl; else cout << "FAIL" << endl; if (m.try_emplace(2, 5).second) cout << "OK" << endl; else cout << "FAIL" << endl; for (auto i : m) { cout << i.first << ' ' << i.second << endl; } } $ ./a OK OK FAIL 3 4 2 3

DashCoreAutoGuix · 2024-12-07T20:32:06Z

Guix Automation has began to build this PR tagged as v22.1.0-devpr6237.3e524228. A new comment will be made when the image is pushed.

DashCoreAutoGuix · 2024-12-07T20:34:54Z

Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v22.1.0-devpr6237.3e524228. The image should be on dockerhub soon.

DashCoreAutoGuix · 2024-12-07T20:46:04Z

Guix Automation has began to build this PR tagged as v22.1.0-devpr6237.35745503. A new comment will be made when the image is pushed.

DashCoreAutoGuix · 2024-12-07T21:34:42Z

Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v22.1.0-devpr6237.35745503. The image should be on dockerhub soon.

DashCoreAutoGuix · 2024-12-17T17:49:39Z

Guix Automation has failed due to the HEAD commit not being signed by an authorized core-team member. Please rebase and sign or push a new empty signed commit to allow Guix build to happen.

…kHeight stats

…ead of `ConnectBlock`

DashCoreAutoGuix · 2025-03-02T06:00:08Z

Guix Automation has began to build this PR tagged as v23.0.0-devpr6237.ebbc1dde. A new comment will be made when the image is pushed.

coderabbitai

Actionable comments posted: 0

🔭 Outside diff range comments (1)

src/evo/deterministicmns.cpp (1)

1-1: ⚠️ Potential issue

Fix Clang format issues.

The pipeline reports Clang format differences. Please run clang-format to fix formatting issues before merging.

🧰 Tools

🪛 GitHub Actions: Clang Diff Format Check

[error] 1-1: Clang format differences found. Please run 'clang-format' to fix formatting issues.

🧹 Nitpick comments (1)

src/evo/deterministicmns.cpp (1)

653-662: Consider optimizing duplicate call to GetValidWeightedMNsCount()

There's a redundant call to newList.GetValidWeightedMNsCount() on lines 655 and 657. If this method is computationally expensive (involves looping through the masternode list), consider storing the result in a local variable to avoid calculating it twice.

if (::g_stats_client->active()) {
    ::g_stats_client->gauge("masternodes.count", newList.GetAllMNsCount());
-   ::g_stats_client->gauge("masternodes.weighted_count", newList.GetValidWeightedMNsCount());
+   const auto validWeightedCount = newList.GetValidWeightedMNsCount();
+   ::g_stats_client->gauge("masternodes.weighted_count", validWeightedCount);
    ::g_stats_client->gauge("masternodes.enabled", newList.GetValidMNsCount());
-   ::g_stats_client->gauge("masternodes.weighted_enabled", newList.GetValidWeightedMNsCount());
+   ::g_stats_client->gauge("masternodes.weighted_enabled", validWeightedCount);
    ::g_stats_client->gauge("masternodes.evo.count", newList.GetAllEvoCount());
    ::g_stats_client->gauge("masternodes.evo.enabled", newList.GetValidEvoCount());
    ::g_stats_client->gauge("masternodes.mn.count", newList.GetAllMNsCount() - newList.GetAllEvoCount());
    ::g_stats_client->gauge("masternodes.mn.enabled", newList.GetValidMNsCount() - newList.GetValidEvoCount());
}

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 7bbbc15 and ebbc1dd.

📒 Files selected for processing (5)

src/evo/deterministicmns.cpp (2 hunks)
src/init.cpp (2 hunks)
src/llmq/chainlocks.cpp (2 hunks)
src/llmq/instantsend.cpp (3 hunks)
src/llmq/instantsend.h (1 hunks)

🚧 Files skipped from review as they are similar to previous changes (4)

src/llmq/chainlocks.cpp
src/llmq/instantsend.h
src/init.cpp
src/llmq/instantsend.cpp

🧰 Additional context used

🪛 GitHub Actions: Clang Diff Format Check

src/evo/deterministicmns.cpp

[error] 1-1: Clang format differences found. Please run 'clang-format' to fix formatting issues.

⏰ Context from checks skipped due to timeout of 90000ms (8)

GitHub Check: linux64_ubsan-build / Build source
GitHub Check: linux64-build / Build source
GitHub Check: linux64_fuzz-build / Build source
GitHub Check: linux64_sqlite-build / Build source
GitHub Check: linux64_multiprocess-build / Build source
GitHub Check: linux64_tsan-build / Build source
GitHub Check: linux64_nowallet-build / Build source
GitHub Check: x86_64-w64-mingw32 / Build depends

🔇 Additional comments (2)

src/evo/deterministicmns.cpp (2)

24-24: LGTM - Added required header for statistics integration.

The inclusion of the stats client header is appropriate for implementing the new masternode statistics reporting functionality.

653-662: Include statistics tracking only when client is active - good implementation.

The statistics collection is properly wrapped in a check for ::g_stats_client->active(), which aligns with UdjinM6's previous comment about avoiding unnecessary computation during Initial Block Download (IBD).

The implementation correctly reports various masternode metrics:

Total and weighted masternode counts

Enabled masternode counts (regular and weighted)

EvoNode counts (total and enabled)

Regular masternode counts (total and enabled)

This addresses the PR objective of adding masternode statistics.

DashCoreAutoGuix · 2025-03-02T06:07:41Z

Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v23.0.0-devpr6237.ebbc1dde. The image should be on dockerhub soon.

DashCoreAutoGuix · 2025-03-02T06:14:07Z

Guix Automation has began to build this PR tagged as v23.0.0-devpr6237.38d78924. A new comment will be made when the image is pushed.

DashCoreAutoGuix · 2025-03-02T07:21:42Z

Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v23.0.0-devpr6237.38d78924. The image should be on dockerhub soon.

…tional on either stats or logger being active

DashCoreAutoGuix · 2025-03-10T23:54:07Z

Guix Automation has began to build this PR tagged as v23.0.0-devpr6237.d5dbbdb8. A new comment will be made when the image is pushed.

coderabbitai

Actionable comments posted: 0

🧹 Nitpick comments (1)

src/llmq/instantsend.cpp (1)

1-1654: Fix Clang format issues.
The pipeline failure indicates formatting discrepancies. Please run clang-format (or the project’s equivalent formatting check) on this file to ensure compliance and unblock the CI pipeline.

🧰 Tools

🪛 Cppcheck (2.10-2)

[error] 623-623: syntax error

(syntaxError)

🪛 GitHub Actions: Clang Diff Format Check

[error] 1-1: Clang format differences found. Please run 'clang-format' to fix formatting issues.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d5dbbdb and abcba59.

📒 Files selected for processing (1)

src/llmq/instantsend.cpp (4 hunks)

🧰 Additional context used

🪛 GitHub Actions: Clang Diff Format Check

src/llmq/instantsend.cpp

[error] 1-1: Clang format differences found. Please run 'clang-format' to fix formatting issues.

⏰ Context from checks skipped due to timeout of 90000ms (6)

GitHub Check: x86_64-w64-mingw32 / Build depends
GitHub Check: x86_64-apple-darwin / Build depends
GitHub Check: x86_64-pc-linux-gnu_nowallet / Build depends
GitHub Check: x86_64-pc-linux-gnu_multiprocess / Build depends
GitHub Check: x86_64-pc-linux-gnu / Build depends
GitHub Check: arm-linux-gnueabihf / Build depends

🔇 Additional comments (4)

src/llmq/instantsend.cpp (4)

26-26: Include is properly placed.
The addition of <stats/client.h> is consistent with the newly introduced functionality that leverages the g_stats_client instance. No issues found.

745-747: Helper function for logging conditions looks good.
This succinct function clearly indicates when IS lock timing stats should be recorded. Implementation is straightforward and aligns with usage patterns in the file.

782-798: Concurrent timing capture logic is sound.
Locking cs_timingsTxSeen prevents data races when measuring the time difference between the transaction's arrival and the IS lock reception. The code correctly removes the entry from timingsTxSeen to avoid repeated measurements.

1193-1198: Accurate timestamp assignment for first-seen TX.
Utilizing LOCK(cs_timingsTxSeen) ensures thread safety. try_emplace properly avoids overwriting recorded timestamps if the TX is already tracked, which meets the requirement to measure only the first arrival time.

coderabbitai

Actionable comments posted: 0

🧹 Nitpick comments (1)

src/validation.cpp (1)

2963-2966: Consider extracting signature operation counting to a helper function

The signature operation counting logic is duplicated between DisconnectTip and ConnectTip (lines 3090-3093). Consider extracting this to a helper function to avoid duplication and ensure consistent behavior.

+// Helper function to count signature operations in a block
+unsigned int GetBlockSigOpCount(const CBlock& block) {
+    unsigned int nSigOps = 0;
+    for (const auto& tx : block.vtx) {
+        nSigOps += GetLegacySigOpCount(*tx);
+    }
+    return nSigOps;
+}

 bool CChainState::DisconnectTip(BlockValidationState& state, DisconnectedBlockTransactions* disconnectpool)
 {
     // Existing code...
-    unsigned int nSigOps = 0;
-    for (const auto& tx : block.vtx) {
-        nSigOps += GetLegacySigOpCount(*tx);
-    }
+    unsigned int nSigOps = GetBlockSigOpCount(block);
     // Rest of the function...
 }

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between abcba59 and c3b1b3e.

📒 Files selected for processing (1)

src/validation.cpp (3 hunks)

⏰ Context from checks skipped due to timeout of 90000ms (1)

GitHub Check: check_merge

🔇 Additional comments (3)

src/validation.cpp (3)

2899-2901: LGTM: Added timing measurement for DisconnectTip

This is a good addition that will help with performance monitoring by tracking the execution time of DisconnectTip.

2960-2972: Good addition of block metrics tracking to DisconnectTip

This code adds proper instrumentation to track and report key metrics about disconnected blocks:

Block execution time

Block size

Chain height

Block version

Transaction count

Signature operations count

All metrics are properly reported via the stats client.

3090-3099: Good parallel implementation of block metrics in ConnectTip

This correctly implements the same metrics tracking for ConnectTip as was added to DisconnectTip, ensuring consistent reporting of block statistics in both directions (connecting and disconnecting).

DashCoreAutoGuix · 2025-03-11T00:51:17Z

Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v23.0.0-devpr6237.d5dbbdb8. The image should be on dockerhub soon.

DashCoreAutoGuix · 2025-03-11T00:52:08Z

Guix Automation has began to build this PR tagged as v23.0.0-devpr6237.c3b1b3e1. A new comment will be made when the image is pushed.

DashCoreAutoGuix · 2025-03-11T01:47:10Z

Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v23.0.0-devpr6237.c3b1b3e1. The image should be on dockerhub soon.

knst

LGTM c3b1b3e

knst · 2025-03-11T07:17:26Z

src/validation.cpp

    AssertLockHeld(cs_main);
    if (m_mempool) AssertLockHeld(m_mempool->cs);

+    int64_t nTime1 = GetTimeMicros();


Consider using here SteadyClock::now()? It returns time in time units, not int64_t.

GetTimeMicros is going to be removed by bitcoin#27233

It's better to use GetTimeMicros here for consistency sake and replace it with SteadyClock::now() in the same report it's replaced for other Dash-specific timers and timers inherited from bitcoin imo.

UdjinM6 · 2025-03-13T16:58:11Z

c3b1b3e LGTM, might want to apply clang-format suggestions though

kwvg

utACK c3b1b3e

dismissing "requesting changes" due to subsequent LGTM review

PastaPastaPasta added this to the 21.2 milestone Aug 28, 2024

PastaPastaPasta force-pushed the stats-add-basic-islock-clsig branch from 8f47ceb to ab3598b Compare August 28, 2024 17:15

PastaPastaPasta added the guix-build label Aug 28, 2024

UdjinM6 reviewed Aug 31, 2024

View reviewed changes

src/validation.cpp Outdated Show resolved Hide resolved

PastaPastaPasta removed the guix-build label Sep 10, 2024

PastaPastaPasta commented Sep 10, 2024

View reviewed changes

src/llmq/instantsend.cpp Outdated Show resolved Hide resolved

kwvg mentioned this pull request Sep 10, 2024

chore: add stats as a pull request header type and scope #6262

Merged

5 tasks

github-actions bot added the needs rebase label Sep 11, 2024

UdjinM6 removed this from the 21.2 milestone Oct 29, 2024

knst reviewed Nov 25, 2024

View reviewed changes

PastaPastaPasta force-pushed the stats-add-basic-islock-clsig branch from 2f26f78 to 3e52422 Compare December 7, 2024 20:30

github-actions bot removed the needs rebase label Dec 7, 2024

PastaPastaPasta added the guix-build label Dec 7, 2024

PastaPastaPasta requested a review from UdjinM6 December 17, 2024 04:38

UdjinM6 added this to the 22.1 milestone Dec 18, 2024

PastaPastaPasta and others added 7 commits March 1, 2025 23:58

feat: add transactions.mempool.lockedTransactions and chainlocks.bloc…

021e499

…kHeight stats

fix: add 1 to tip.Height so output matches RPCs such as getblockcount

5b1af45

feat: implement masternode counts in stats

b794170

feat: add islock timing statistics

18b509d

fix: update blocks.tip.* stats in ConnectTip/DisconnectTip inst…

0e4f46d

…ead of `ConnectBlock`

fixup: compilation errors

83d9f73

fixup: avoid expensive evaluations in stats code

ebbc1dd

PastaPastaPasta force-pushed the stats-add-basic-islock-clsig branch from 7bbbc15 to ebbc1dd Compare March 2, 2025 05:58

coderabbitai bot reviewed Mar 2, 2025

View reviewed changes

fix: include instantsend

38d7892

feat: also log the islock time, and erase the received at time; condi…

d5dbbdb

…tional on either stats or logger being active

fix: use try_emplace

abcba59

coderabbitai bot reviewed Mar 10, 2025

View reviewed changes

fixup: drop blocks.tip stats from ConnectBlock

c3b1b3e

coderabbitai bot reviewed Mar 11, 2025

View reviewed changes

knst approved these changes Mar 11, 2025

View reviewed changes

kwvg approved these changes Apr 1, 2025

View reviewed changes

PastaPastaPasta merged commit e8ddec0 into dashpay:develop Apr 1, 2025
31 of 35 checks passed

feat: add transactions.mempool.lockedTransactions and chainlocks.blockHeight stats #6237

feat: add transactions.mempool.lockedTransactions and chainlocks.blockHeight stats #6237

Uh oh!

Conversation

PastaPastaPasta commented Aug 28, 2024

Issue being fixed or feature implemented

What was done?

How Has This Been Tested?

Breaking Changes

Checklist:

Uh oh!

DashCoreAutoGuix commented Aug 28, 2024

Uh oh!

DashCoreAutoGuix commented Aug 28, 2024

Uh oh!

DashCoreAutoGuix commented Aug 29, 2024

Uh oh!

DashCoreAutoGuix commented Aug 29, 2024

Uh oh!

DashCoreAutoGuix commented Aug 29, 2024

Uh oh!

DashCoreAutoGuix commented Aug 29, 2024

Uh oh!

Uh oh!

DashCoreAutoGuix commented Sep 10, 2024

Uh oh!

PastaPastaPasta commented Sep 10, 2024

Uh oh!

Uh oh!

DashCoreAutoGuix commented Sep 10, 2024

Uh oh!

UdjinM6 commented Sep 10, 2024

Uh oh!

github-actions bot commented Sep 11, 2024

Uh oh!

knst Nov 25, 2024

Choose a reason for hiding this comment

Uh oh!

PastaPastaPasta Mar 10, 2025

Choose a reason for hiding this comment

Uh oh!

knst Nov 25, 2024

Choose a reason for hiding this comment

Uh oh!

knst Nov 25, 2024

Choose a reason for hiding this comment

Uh oh!

PastaPastaPasta Mar 10, 2025

Choose a reason for hiding this comment

Uh oh!

DashCoreAutoGuix commented Dec 7, 2024

Uh oh!

DashCoreAutoGuix commented Dec 7, 2024

Uh oh!

DashCoreAutoGuix commented Dec 7, 2024

Uh oh!

DashCoreAutoGuix commented Dec 7, 2024

Uh oh!

DashCoreAutoGuix commented Dec 17, 2024

Uh oh!

DashCoreAutoGuix commented Mar 2, 2025

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

DashCoreAutoGuix commented Mar 2, 2025

Uh oh!

DashCoreAutoGuix commented Mar 2, 2025

Uh oh!

DashCoreAutoGuix commented Mar 2, 2025

Uh oh!

DashCoreAutoGuix commented Mar 10, 2025

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

DashCoreAutoGuix commented Mar 11, 2025

knst Mar 11, 2025 •

edited

Loading