Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DB] DB Pruner refactoring and add transaction pruner #212

Merged
merged 1 commit into from
Mar 17, 2022
Merged

[DB] DB Pruner refactoring and add transaction pruner #212

merged 1 commit into from
Mar 17, 2022

Conversation

sitalkedia
Copy link
Contributor

Motivation

This is the PR for supporting ledger history pruning. See #103 for more details. Following changes have been added.

Refactoring the existing pruner, so that we can add support for more pruner in next diff.
Implementation of transaction pruner, which prunes all transaction-related data from DB before a certain version

Test Plan

Added UTs for testing the pruner.

@netlify
Copy link

netlify bot commented Mar 16, 2022 •
edited
Loading

✔️ Deploy Preview for aptos-developer-docs ready!

🔨 Explore the source changes: 7e627c4

🔍 Inspect the deploy log: https://app.netlify.com/sites/aptos-developer-docs/deploys/6232caf3a4e919000920f8a7

😎 Browse the preview: https://deploy-preview-212--aptos-developer-docs.netlify.app

@sitalkedia sitalkedia changed the title [DB] DB pruner [DB] DB Pruner refactoring and add transaction pruner Mar 16, 2022
@sitalkedia sitalkedia requested a review from msmouse March 16, 2022 00:47
@sitalkedia sitalkedia mentioned this pull request Mar 16, 2022
Closed
msmouse
msmouse reviewed Mar 16, 2022
View reviewed changes
Copy link
Contributor

@msmouse msmouse left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice.

And we need to fix

aptos-core/storage/aptosdb/src/lib.rs

Line 808 in 1521a7e

fn get_first_txn_version(&self) -> Result<Option<Version>> {

(link was wrong on the issue, sorry)

cc @JoshLind

config/src/config/storage_config.rs Outdated Show resolved Hide resolved
storage/aptosdb/src/metrics.rs Outdated Show resolved Hide resolved
storage/aptosdb/src/pruner/mod.rs Outdated Show resolved Hide resolved
storage/aptosdb/src/pruner/mod.rs Outdated Show resolved Hide resolved
storage/aptosdb/src/pruner/db_pruner.rs Show resolved Hide resolved
storage/aptosdb/src/pruner/state_store/mod.rs Show resolved Hide resolved
storage/aptosdb/src/pruner/mod.rs Outdated Show resolved Hide resolved
storage/aptosdb/src/pruner/worker.rs Show resolved Hide resolved
storage/aptosdb/src/pruner/mod.rs Outdated Show resolved Hide resolved
storage/aptosdb/src/pruner/mod.rs Outdated Show resolved Hide resolved
@msmouse msmouse requested a review from JoshLind March 16, 2022 06:42
msmouse
msmouse reviewed Mar 16, 2022
View reviewed changes
Copy link
Contributor

@msmouse msmouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please give it another round. Many things can be separate follow up PRs, just comment on those you want to delay to the next PR. :)

storage/aptosdb/src/pruner/transaction_store/test.rs Outdated Show resolved Hide resolved
storage/aptosdb/src/pruner/transaction_store/test.rs Outdated Show resolved Hide resolved
storage/aptosdb/src/pruner/transaction_store/test.rs Show resolved Hide resolved
storage/aptosdb/src/pruner/transaction_store/test.rs Show resolved Hide resolved
storage/aptosdb/src/pruner/transaction_store/test.rs Outdated Show resolved Hide resolved
@sitalkedia
Copy link
Contributor Author

@msmouse - Addressed majority of the review comments - PTAL.

msmouse
msmouse reviewed Mar 16, 2022
View reviewed changes
storage/aptosdb/src/pruner/worker.rs Show resolved Hide resolved
storage/aptosdb/src/pruner/worker.rs Show resolved Hide resolved
storage/aptosdb/src/pruner/mod.rs Outdated Show resolved Hide resolved
storage/aptosdb/src/pruner/mod.rs Outdated Show resolved Hide resolved
@sitalkedia
Copy link
Contributor Author

/canary

aptos-bot pushed a commit that referenced this pull request Mar 17, 2022
@sitalkedia @aptos-bot
[DB] DB pruner
4618d73 
Closes: #212
@aptos-bot
Copy link
Contributor

Forge run: https://circleci.com/gh/aptos-labs/aptos-core/4501
Forge Test Result: all up : 2032 TPS, 2153 ms latency, 3650 ms p99 latency,no expired txns

@aptos-bot
Copy link
Contributor

☀️ Canary successful

@sitalkedia
Copy link
Contributor Author

/land

@aptos-bot
Copy link
Contributor

Forge run: https://circleci.com/gh/aptos-labs/aptos-core/4535
Forge Test Result: all up : 1997 TPS, 2189 ms latency, 3650 ms p99 latency,no expired txns

@aptos-bot aptos-bot merged commit 7e627c4 into aptos-labs:main Mar 17, 2022
@sitalkedia sitalkedia mentioned this pull request Mar 18, 2022
Closed
@sitalkedia sitalkedia mentioned this pull request Mar 18, 2022
Merged
@sitalkedia sitalkedia linked an issue Mar 22, 2022 that may be closed by this pull request
[Feature Request] Ledger history pruning #103
Closed
@sitalkedia sitalkedia mentioned this pull request May 24, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@msmouse msmouse msmouse approved these changes

@JoshLind JoshLind Awaiting requested review from JoshLind JoshLind is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature Request] Ledger history pruning
3 participants
@sitalkedia @aptos-bot @msmouse