Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Access] Add registerDB pruning module #6397

Open
wants to merge 47 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
5584146
Added new cli flags to access and observer nodes
UlyanaAndrukhiv Aug 15, 2024
fd5fa65
Updated last commit
UlyanaAndrukhiv Aug 15, 2024
8aab5d9
Added skeleton of register db pruner module
UlyanaAndrukhiv Aug 16, 2024
08274d1
Added missed logic for checkPrune implementation for register db pruning
UlyanaAndrukhiv Aug 20, 2024
61d3be2
Added pruner integration to AN and ON
UlyanaAndrukhiv Aug 20, 2024
8910083
Added prune up to height implementation
UlyanaAndrukhiv Aug 21, 2024
b863b61
Updated iteration over the register keys, moved update first height t…
UlyanaAndrukhiv Aug 22, 2024
fc3c44c
Updated collecting keys to remove
UlyanaAndrukhiv Aug 22, 2024
64e9c9b
Added tests for register pruner
UlyanaAndrukhiv Aug 23, 2024
177651c
Added basic metrics for pruner
Guitarheroua Aug 23, 2024
33fe4a1
Merge branch 'UlyanaAndrukhiv/6068-registerDB-pruning-module' of gith…
Guitarheroua Aug 23, 2024
2d847ba
Added aditional metrics
Guitarheroua Aug 26, 2024
1018ba2
Added missing metric
Guitarheroua Aug 26, 2024
0e8cc4c
Added throttle delay to prevent excessive system load during pruning
UlyanaAndrukhiv Aug 26, 2024
afe9e56
Merge branch 'UlyanaAndrukhiv/6068-registerDB-pruning-module' of gith…
UlyanaAndrukhiv Aug 26, 2024
fec4ba2
Merge branch 'master' into UlyanaAndrukhiv/6068-registerDB-pruning-mo…
UlyanaAndrukhiv Aug 26, 2024
b9247b6
Updated checkPrune implementation
UlyanaAndrukhiv Aug 26, 2024
97a046d
Merge branch 'UlyanaAndrukhiv/6068-registerDB-pruning-module' of gith…
UlyanaAndrukhiv Aug 26, 2024
0e7f9e5
Merge branch 'master' into UlyanaAndrukhiv/6068-registerDB-pruning-mo…
UlyanaAndrukhiv Aug 27, 2024
8080a15
Merge branch 'master' into UlyanaAndrukhiv/6068-registerDB-pruning-mo…
UlyanaAndrukhiv Aug 29, 2024
b57bc4e
Added first batch of suggested changes according to comments
UlyanaAndrukhiv Aug 29, 2024
3ca7883
Removed ComponentManager
UlyanaAndrukhiv Aug 29, 2024
1ff10bc
Simplified prune throttle delay implementation and updated error hand…
UlyanaAndrukhiv Aug 29, 2024
25e1356
Removed unnecessary checks for metrics and updated tests
UlyanaAndrukhiv Aug 29, 2024
66e810b
Removed extra scope according to comments
UlyanaAndrukhiv Aug 29, 2024
0aada71
Fixed check for pruning
UlyanaAndrukhiv Aug 29, 2024
f039d15
Updated godoc
UlyanaAndrukhiv Aug 29, 2024
b6c8040
Added more comments
UlyanaAndrukhiv Aug 29, 2024
db82c62
Updated comments for register priner tests
UlyanaAndrukhiv Aug 29, 2024
dc984cf
Added functional test for error handling for register pruner, added m…
UlyanaAndrukhiv Aug 30, 2024
84a8f9a
Merged with master
UlyanaAndrukhiv Aug 30, 2024
bf15f92
Merge branch 'master' into UlyanaAndrukhiv/6068-registerDB-pruning-mo…
Guitarheroua Sep 3, 2024
919f03b
Merge branch 'master' into UlyanaAndrukhiv/6068-registerDB-pruning-mo…
UlyanaAndrukhiv Sep 9, 2024
9ed6ca8
Added basic integration test functionality
Guitarheroua Sep 10, 2024
2cb1264
Merge branch 'UlyanaAndrukhiv/6068-registerDB-pruning-module' of gith…
Guitarheroua Sep 10, 2024
f040c83
Merge branch 'master' into UlyanaAndrukhiv/6068-registerDB-pruning-mo…
UlyanaAndrukhiv Sep 12, 2024
fc4d6cb
Merged with master
UlyanaAndrukhiv Sep 13, 2024
87fe8fe
Merge branch 'UlyanaAndrukhiv/6068-registerDB-pruning-module' of gith…
Guitarheroua Sep 19, 2024
d70106a
Merge branch 'master' into UlyanaAndrukhiv/6068-registerDB-pruning-mo…
Guitarheroua Sep 19, 2024
f276544
Merge branch 'UlyanaAndrukhiv/6068-registerDB-pruning-module' of gith…
Guitarheroua Sep 19, 2024
7ffc1a4
db path fix
Guitarheroua Sep 23, 2024
66418af
Merge branch 'master' into UlyanaAndrukhiv/6068-registerDB-pruning-mo…
Guitarheroua Sep 24, 2024
ebcc13f
Merge branch 'UlyanaAndrukhiv/6068-registerDB-pruning-module' of gith…
Guitarheroua Sep 24, 2024
10cb2f2
Fixed crashes in tests
Guitarheroua Sep 26, 2024
23f1e36
Merge branch 'master' of github.com:The-K-R-O-K/flow-go into UlyanaAn…
Guitarheroua Oct 9, 2024
d251445
Merge branch 'master' into UlyanaAndrukhiv/6068-registerDB-pruning-mo…
Guitarheroua Oct 17, 2024
5833735
Merge branch 'master' into UlyanaAndrukhiv/6068-registerDB-pruning-mo…
Guitarheroua Oct 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 81 additions & 22 deletions cmd/access/node_builder/access_node_builder.go
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ import (
"github.com/onflow/flow-go/module/execution"
"github.com/onflow/flow-go/module/executiondatasync/execution_data"
execdatacache "github.com/onflow/flow-go/module/executiondatasync/execution_data/cache"
"github.com/onflow/flow-go/module/executiondatasync/pruner"
edpruner "github.com/onflow/flow-go/module/executiondatasync/pruner"
edstorage "github.com/onflow/flow-go/module/executiondatasync/storage"
"github.com/onflow/flow-go/module/executiondatasync/tracker"
finalizer "github.com/onflow/flow-go/module/finalizer/consensus"
Expand Down Expand Up @@ -179,6 +179,9 @@ type AccessNodeConfig struct {
versionControlEnabled bool
storeTxResultErrorMessages bool
stopControlEnabled bool
registerDBPruningEnabled bool
registerDBPruneTickerInterval time.Duration
registerDBPruneThrottleDelay time.Duration
registerDBPruneThreshold uint64
}

Expand Down Expand Up @@ -271,8 +274,8 @@ func DefaultAccessNodeConfig() *AccessNodeConfig {
executionDataIndexingEnabled: false,
executionDataDBMode: execution_data.ExecutionDataDBModeBadger.String(),
executionDataPrunerHeightRangeTarget: 0,
executionDataPrunerThreshold: pruner.DefaultThreshold,
executionDataPruningInterval: pruner.DefaultPruningInterval,
executionDataPrunerThreshold: edpruner.DefaultThreshold,
executionDataPruningInterval: edpruner.DefaultPruningInterval,
registersDBPath: filepath.Join(homedir, ".flow", "execution_state"),
checkpointFile: cmd.NotSet,
scriptExecutorConfig: query.NewDefaultConfig(),
Expand All @@ -285,7 +288,10 @@ func DefaultAccessNodeConfig() *AccessNodeConfig {
versionControlEnabled: true,
storeTxResultErrorMessages: false,
stopControlEnabled: false,
registerDBPruneThreshold: pruner.DefaultThreshold,
registerDBPruningEnabled: false,
registerDBPruneTickerInterval: pstorage.DefaultPruneTickerInterval,
registerDBPruneThrottleDelay: pstorage.DefaultPruneThrottleDelay,
registerDBPruneThreshold: pstorage.DefaultPruneThreshold,
}
}

Expand All @@ -308,8 +314,9 @@ type FlowAccessNodeBuilder struct {
BlocksToMarkExecuted *stdmap.Times
BlockTransactions *stdmap.IdentifierMap
TransactionMetrics *metrics.TransactionCollector
TransactionValidationMetrics *metrics.TransactionValidationCollector
RestMetrics *metrics.RestCollector
RegisterDBPrunerMetrics *metrics.RegisterDBPrunerCollector
TransactionValidationMetrics *metrics.TransactionValidationCollector
AccessMetrics module.AccessMetrics
PingMetrics module.PingMetrics
Committee hotstuff.DynamicCommittee
Expand All @@ -332,11 +339,13 @@ type FlowAccessNodeBuilder struct {
TxResultsIndex *index.TransactionResultsIndex
IndexerDependencies *cmd.DependencyList
collectionExecutedMetric module.CollectionExecutedMetric
ExecutionDataPruner *pruner.Pruner
ExecutionDataPruner *edpruner.Pruner
ExecutionDatastoreManager edstorage.DatastoreManager
ExecutionDataTracker tracker.Storage
VersionControl *version.VersionControl
StopControl *stop.StopControl
RegisterDB *pebble.DB
RegisterDBPrunerDependencies *cmd.DependencyList

// The sync engine participants provider is the libp2p peer store for the access node
// which is not available until after the network has started.
Expand Down Expand Up @@ -558,6 +567,10 @@ func (builder *FlowAccessNodeBuilder) BuildExecutionSyncComponents() *FlowAccess
requesterDependable := module.NewProxiedReadyDoneAware()
builder.IndexerDependencies.Add(requesterDependable)

// setup dependency chain to ensure register db pruner starts after the indexer
indexerDependable := module.NewProxiedReadyDoneAware()
builder.RegisterDBPrunerDependencies.Add(indexerDependable)

executionDataPrunerEnabled := builder.executionDataPrunerHeightRangeTarget != 0

builder.
Expand Down Expand Up @@ -787,16 +800,16 @@ func (builder *FlowAccessNodeBuilder) BuildExecutionSyncComponents() *FlowAccess
}

var err error
builder.ExecutionDataPruner, err = pruner.NewPruner(
builder.ExecutionDataPruner, err = edpruner.NewPruner(
node.Logger,
prunerMetrics,
builder.ExecutionDataTracker,
pruner.WithPruneCallback(func(ctx context.Context) error {
edpruner.WithPruneCallback(func(ctx context.Context) error {
return builder.ExecutionDatastoreManager.CollectGarbage(ctx)
}),
pruner.WithHeightRangeTarget(builder.executionDataPrunerHeightRangeTarget),
pruner.WithThreshold(builder.executionDataPrunerThreshold),
pruner.WithPruningInterval(builder.executionDataPruningInterval),
edpruner.WithHeightRangeTarget(builder.executionDataPrunerHeightRangeTarget),
edpruner.WithThreshold(builder.executionDataPrunerThreshold),
edpruner.WithPruningInterval(builder.executionDataPruningInterval),
)
if err != nil {
return nil, fmt.Errorf("failed to create execution data pruner: %w", err)
Expand Down Expand Up @@ -861,16 +874,17 @@ func (builder *FlowAccessNodeBuilder) BuildExecutionSyncComponents() *FlowAccess
// Note: using a DependableComponent here to ensure that the indexer does not block
// other components from starting while bootstrapping the register db since it may
// take hours to complete.

pdb, err := pstorage.OpenRegisterPebbleDB(builder.registersDBPath)
var err error
builder.RegisterDB, err = pstorage.OpenRegisterPebbleDB(builder.registersDBPath)
builder.Logger.Warn().Msg(fmt.Sprintf("!!!!!!!!!! builder.registersDBPath: %s", builder.registersDBPath))
if err != nil {
return nil, fmt.Errorf("could not open registers db: %w", err)
}
builder.ShutdownFunc(func() error {
return pdb.Close()
return builder.RegisterDB.Close()
})

bootstrapped, err := pstorage.IsBootstrapped(pdb)
bootstrapped, err := pstorage.IsBootstrapped(builder.RegisterDB)
if err != nil {
return nil, fmt.Errorf("could not check if registers db is bootstrapped: %w", err)
}
Expand Down Expand Up @@ -902,7 +916,7 @@ func (builder *FlowAccessNodeBuilder) BuildExecutionSyncComponents() *FlowAccess
}

rootHash := ledger.RootHash(builder.RootSeal.FinalState)
bootstrap, err := pstorage.NewRegisterBootstrap(pdb, checkpointFile, checkpointHeight, rootHash, builder.Logger)
bootstrap, err := pstorage.NewRegisterBootstrap(builder.RegisterDB, checkpointFile, checkpointHeight, rootHash, builder.Logger)
if err != nil {
return nil, fmt.Errorf("could not create registers bootstrap: %w", err)
}
Expand All @@ -915,7 +929,7 @@ func (builder *FlowAccessNodeBuilder) BuildExecutionSyncComponents() *FlowAccess
}
}

registers, err := pstorage.NewRegisters(pdb, builder.registerDBPruneThreshold)
registers, err := pstorage.NewRegisters(builder.RegisterDB, builder.registerDBPruneThreshold)
if err != nil {
return nil, fmt.Errorf("could not create registers storage: %w", err)
}
Expand Down Expand Up @@ -1013,8 +1027,31 @@ func (builder *FlowAccessNodeBuilder) BuildExecutionSyncComponents() *FlowAccess
builder.StopControl.RegisterHeightRecorder(builder.ExecutionIndexer)
}

// add indexer into ReadyDoneAware dependency passed to pruner. This allows the register db pruner
// to wait for the indexer to be ready before starting.
indexerDependable.Init(builder.ExecutionIndexer)

return builder.ExecutionIndexer, nil
}, builder.IndexerDependencies)
}, builder.IndexerDependencies).
DependableComponent("register db pruner", func(node *cmd.NodeConfig) (module.ReadyDoneAware, error) {
if !builder.registerDBPruningEnabled {
return &module.NoopReadyDoneAware{}, nil
}

registerDBPruner, err := pstorage.NewRegisterPruner(
node.Logger,
builder.RegisterDB,
pstorage.WithPrunerMetrics(builder.RegisterDBPrunerMetrics),
//pstorage.WithPruneThreshold(builder.registerDBPruneThreshold),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WithPruneThreshold is temporarily commented out and will be re-enabled once PR #6345 is merged.

pstorage.WithPruneThrottleDelay(builder.registerDBPruneThrottleDelay),
pstorage.WithPruneTickerInterval(builder.registerDBPruneTickerInterval),
)
if err != nil {
return nil, fmt.Errorf("failed to create register db pruner: %w", err)
}

return registerDBPruner, nil
}, builder.RegisterDBPrunerDependencies)
}

if builder.stateStreamConf.ListenAddr != "" {
Expand Down Expand Up @@ -1141,10 +1178,11 @@ func FlowAccessNode(nodeBuilder *cmd.FlowNodeBuilder) *FlowAccessNodeBuilder {
dist := consensuspubsub.NewFollowerDistributor()
dist.AddProposalViolationConsumer(notifications.NewSlashingViolationsConsumer(nodeBuilder.Logger))
return &FlowAccessNodeBuilder{
AccessNodeConfig: DefaultAccessNodeConfig(),
FlowNodeBuilder: nodeBuilder,
FollowerDistributor: dist,
IndexerDependencies: cmd.NewDependencyList(),
AccessNodeConfig: DefaultAccessNodeConfig(),
FlowNodeBuilder: nodeBuilder,
FollowerDistributor: dist,
IndexerDependencies: cmd.NewDependencyList(),
RegisterDBPrunerDependencies: cmd.NewDependencyList(),
}
}

Expand Down Expand Up @@ -1324,6 +1362,8 @@ func (builder *FlowAccessNodeBuilder) extraFlags() {
"execution-data-db",
defaultConfig.executionDataDBMode,
"[experimental] the DB type for execution datastore. One of [badger, pebble]")

// Execution data pruner
flags.Uint64Var(&builder.executionDataPrunerHeightRangeTarget,
"execution-data-height-range-target",
defaultConfig.executionDataPrunerHeightRangeTarget,
Expand All @@ -1337,6 +1377,20 @@ func (builder *FlowAccessNodeBuilder) extraFlags() {
defaultConfig.executionDataPruningInterval,
"duration after which the pruner tries to prune execution data. The default value is 10 minutes")

// RegisterDB pruning
flags.BoolVar(&builder.registerDBPruningEnabled,
"registerdb-pruning-enabled",
defaultConfig.registerDBPruningEnabled,
"whether to enable the pruning for register db")
flags.DurationVar(&builder.registerDBPruneThrottleDelay,
"registerdb-prune-throttle-delay",
defaultConfig.registerDBPruneThrottleDelay,
"delay for controlling a pause between batches of registers inspected and pruned")
flags.DurationVar(&builder.registerDBPruneTickerInterval,
"registerdb-prune-ticker-interval",
defaultConfig.registerDBPruneTickerInterval,
"duration after which the pruner tries to prune data. The default value is 10 minutes")

// Execution State Streaming API
flags.Uint32Var(&builder.stateStreamConf.ExecutionDataCacheSize, "execution-data-cache-size", defaultConfig.stateStreamConf.ExecutionDataCacheSize, "block execution data cache size")
flags.Uint32Var(&builder.stateStreamConf.MaxGlobalStreams, "state-stream-global-max-streams", defaultConfig.stateStreamConf.MaxGlobalStreams, "global maximum number of concurrent streams")
Expand Down Expand Up @@ -1721,12 +1775,17 @@ func (builder *FlowAccessNodeBuilder) Build() (cmd.Node, error) {
builder.RestMetrics = m
return nil
}).
Module("register db metrics", func(node *cmd.NodeConfig) error {
builder.RegisterDBPrunerMetrics = metrics.NewRegisterDBPrunerCollector()
return nil
}).
Module("access metrics", func(node *cmd.NodeConfig) error {
builder.AccessMetrics = metrics.NewAccessCollector(
metrics.WithTransactionMetrics(builder.TransactionMetrics),
metrics.WithTransactionValidationMetrics(builder.TransactionValidationMetrics),
metrics.WithBackendScriptsMetrics(builder.TransactionMetrics),
metrics.WithRestMetrics(builder.RestMetrics),
metrics.WithRegisterDBPrunerMetrics(builder.RegisterDBPrunerMetrics),
)
return nil
}).
Expand Down
Loading
Loading