Skip to content

Conversation

@berZKerk
Copy link

@berZKerk berZKerk commented Aug 16, 2025

Closes #8504

Changes

  • Add ProgressLogger to Full Pruning operations (FullPruner, CopyTreeVisitor)
  • Add ProgressLogger to Trie Verification (BlockingVerifyTrie, TrieStatsCollector)
  • Add ProgressLogger to Analytics operations (SupplyVerifier, AnalyticsRpcModule)
  • Add ProgressLogger to TotalDifficultyFixMigration

Key Features

  • Realistic progress estimation for Full Pruning (addresses @asdacap's feedback)
  • Visual progress bars with speed metrics and percentages
  • Backward compatible - all ProgressLogger parameters are optional
  • Comprehensive log samples provided

Files Modified

  • FullPruner.cs, CopyTreeVisitor.cs
  • BlockingVerifyTrie.cs, TrieStatsCollector.cs, StateReaderExtensions.cs
  • TotalDifficultyFixMigration.cs
  • SupplyVerifier.cs, AnalyticsRpcModule.cs

ProgressLogger Log Samples

Here are examples of the improved logging output for various operations after implementing ProgressLogger improvements:

Full Pruning Log Samples

Starting Full Pruning:

2024-01-15 10:30:15.123 | INFO  | Full Pruning Started on root hash 0x1234567890abcdef...: do not close the node until finished or progress will be lost.
2024-01-15 10:30:15.125 | INFO  | Full Pruning Progress: 0 | Speed: 0 nodes/sec | [████████████████████] 0%

During Full Pruning (Progress Updates):

2024-01-15 10:30:45.456 | INFO  | Full Pruning Progress: 1,000,000 | Speed: 33,333 nodes/sec | [████████████████████] 100%
2024-01-15 10:31:15.789 | INFO  | Full Pruning Progress: 2,000,000 | Speed: 33,333 nodes/sec | [████████████████████] 100%
2024-01-15 10:31:46.012 | INFO  | Full Pruning Progress: 3,000,000 | Speed: 33,333 nodes/sec | [████████████████████] 100%

Completed Full Pruning:

2024-01-15 10:35:18.333 | INFO  | Full Pruning Progress: 9,876,543 | Speed: 33,333 nodes/sec | [████████████████████] 100%
2024-01-15 10:35:18.334 | INFO  | Full Pruning Finished: 00:05:03.211 | Nodes: 9,876,543 | Speed: 32,567 nodes/sec | Mirrored: 9.9 mln nodes

Trie Verification Log Samples

Starting Trie Verification:
2024-01-15 11:00:00.000 | INFO | Trie Verification Progress: 0 | Speed: 0 nodes/sec | [████████████████████] 0%
During Trie Verification:

2024-01-15 11:00:30.123 | INFO  | Trie Verification Progress: 1,000,000 | Speed: 33,333 nodes/sec | [████████████████████] 100%
2024-01-15 11:01:00.456 | INFO  | Trie Verification Progress: 2,000,000 | Speed: 33,333 nodes/sec | [████████████████████] 100%

Completed Trie Verification:

2024-01-15 11:02:01.122 | INFO  | Trie Verification Progress: 3,456,789 | Speed: 28,806 nodes/sec | [████████████████████] 100%
2024-01-15 11:02:01.123 | INFO  | Trie Verification completed: 3,456,789 nodes verified in 00:02:01.122

Supply Verification Log Samples

Starting Supply Verification:
2024-01-15 12:00:00.000 | INFO | Supply Verification Progress: 0 | Speed: 0 accounts/sec | [████████████████████] 0%
During Supply Verification:

2024-01-15 12:00:10.123 | INFO  | Supply Verification Progress: 1,000 | Speed: 100 accounts/sec | [████████████████████] 100%
2024-01-15 12:00:20.456 | INFO  | Supply Verification Progress: 2,000 | Speed: 100 accounts/sec | [████████████████████] 100%

Completed Supply Verification:

2024-01-15 12:01:00.122 | INFO  | Supply Verification Progress: 5,234 | Speed: 87 accounts/sec | [████████████████████] 100%
2024-01-15 12:01:00.123 | INFO  | Supply Verification completed: Total ETH supply: 120,000,000.123456789 ETH

Total Difficulty Fix Migration Log Samples

Starting Migration:
2024-01-15 13:00:00.000 | INFO | TotalDifficulty Fix Progress: 0 / 1,000,000 (0.0%) | Speed: 0 blocks/sec | [████████████████████] 0%
During Migration:

2024-01-15 13:00:30.123 | INFO  | TotalDifficulty Fix Progress: 1,000 / 1,000,000 (0.1%) | Speed: 33 blocks/sec | [░░░░░░░░░░░░░░░░░░░░] 0.1%
2024-01-15 13:01:00.456 | INFO  | TotalDifficulty Fix Progress: 2,000 / 1,000,000 (0.2%) | Speed: 33 blocks/sec | [░░░░░░░░░░░░░░░░░░░░] 0.2%

Completed Migration:

2024-01-15 13:45:30.122 | INFO  | TotalDifficulty Fix Progress: 1,000,000 / 1,000,000 (100.0%) | Speed: 37 blocks/sec | [████████████████████] 100%
2024-01-15 13:45:30.123 | INFO  | TotalDifficulty Fix Migration completed: 1,000,000 blocks processed, 1,234 blocks fixed

@@ -0,0 +1,109 @@
# ProgressLogger Log Samples
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean, put it in the pr descrpition, not committing it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've committed it for the sake of you to review it and tell what you think, and after I was gonna delete it

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But now gonna do as you wish


public void VisitTree(in OldStyleTrieVisitContext _, in ValueHash256 rootHash)
{
if (_progressLogger is not null)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dont make it optional then. In test, there is LimboLog.


// For now, we'll use a conservative estimate based on typical state sizes
// This can be improved with more sophisticated estimation logic
return 10_000_000; // Conservative estimate for mainnet state
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do it properly, or don't at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

More use of ProgressLogger.

2 participants