Add --X-trie-log subcommand #6303

gfukushima · 2023-12-15T04:10:01Z

PR description

This is a more direct approach to the previous attempt in #6188.
Tests showed that the previous approach was taking too long (3 hours) to prune a 3.5 months old (keeping the lastest 512 only).
The approach this PR takes is faster since it resets the column family and only retain the necessary trie logs (saved in files in batches of 20_000 trie logs).
Drawback of this approach is that we do store the trie logs in disk (in files) before dropping the column. Realistic speaking there's little to no reason to actually keep many more trie logs than the default(512).
In any case, I've made some adjustments to the code to allow this to go as far as a user might like.

Ran tests on nodes retaining 512 / 20_000 / 250_000 / 1M trie logs, all instances had approx 1.8M trie logs in the db when node was stopped for prunning.

Below some of the logs condensed

Real use case scenario - 512 trie logs retained (1 second)

2023-12-22 05:40:22.409+00:00 | main | INFO  | TrieLogHelper | Saving trie logs to retain in file (batch 1)...
2023-12-22 05:40:22.416+00:00 | main | INFO  | TrieLogHelper | Obtaining trielogs from db, this may take a few minutes...
2023-12-22 05:40:22.692+00:00 | main | INFO  | TrieLogHelper | Clear trie logs...
2023-12-22 05:40:23.131+00:00 | main | INFO  | TrieLogHelper | Restoring trie logs retained from batch 1...
2023-12-22 05:40:23.446+00:00 | main | INFO  | TrieLogHelper | Deleting files...
2023-12-22 05:40:23.452+00:00 | main | INFO  | TrieLogHelper | Prune ran successfully. Enjoy some disk space back! 🚀

a more extreme case where for some reason someone has been running an archieve and decides to keep only the latest 1M trie logs - 1M trie logs retained (approx 2 hours)

2024-01-03 01:48:18.042+00:00 | main | INFO  | BonsaiWorldStateKeyValueStorage | Bonsai flat db mode found PARTIAL
2024-01-03 01:48:18.806+00:00 | main | INFO  | TrieLogHelper | Saving trie logs to retain in file (batch 1)...
2024-01-03 01:48:18.809+00:00 | main | INFO  | TrieLogHelper | Obtaining trielogs from db, this may take a few minutes...
2024-01-03 01:48:34.970+00:00 | main | INFO  | TrieLogHelper | Saving trie logs to retain in file (batch 2)...

2024-01-03 03:42:11.016+00:00 | main | INFO  | TrieLogHelper | Deleting files...
2024-01-03 03:42:11.029+00:00 | main | INFO  | TrieLogHelper | Prune ran successfully. Enjoy some disk space back! 🚀

Fixed Issue(s)

Part of #5390

github-actions · 2023-12-15T04:10:17Z

I thought about documentation and added the doc-change-required label to this PR if updates are required.
I thought about the changelog and included a changelog update if required.
If my PR includes database changes (e.g. KeyValueSegmentIdentifier) I have thought about compatibility and performed forwards and backwards compatibility tests

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/RocksDbUsageHelper.java

jframe · 2023-12-15T05:26:30Z

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java

+
+  private static IdentityHashMap<byte[], byte[]> readTrieLogsFromFile() throws IOException {
+
+    File file = new File(trieLogFile);


Wold be better to use File.createTempFile so it's created in the temp directory and we won't have any write permissions issues

We need the file to exist after we finish the run if for some reason we fail to reload the trielogs into the db

jframe · 2023-12-15T05:27:12Z

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java

+  }
+
+  private static void deleteTrieLogFile() {
+    File file = new File(trieLogFile);


When we create the file we could use the File.deleteOnExit so we don't need to worry about manually deleting this file

jframe · 2023-12-15T05:29:36Z

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java

+                .ifPresent(trieLog -> trieLogsToRetain.put(hash.toArrayUnsafe(), trieLog));
+          });
+      out.println("Saving trielogs to retain in file...");
+      saveTrieLogsInFile(trieLogsToRetain);


Not sure how the fail-safe is being used. Expected to see that if we got an error clearing and inserting entries we ready from the file or allow specifying an existing backup file to restore from.

Added a comment and instructions that will be logged for the user. Basically the idea here is, we retain the logs in a file in case something goes wrong.
In that case the user can re-run the subcommand and we will read the trielogs from the file since the column family might have been dropped already.

jframe · 2023-12-15T05:46:57Z

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java

+      saveTrieLogsInFile(trieLogsToRetain);
+    } else {
+      // try to read the triLogs from the flatfile
+      trieLogsToRetain = readTrieLogsFromFile();


Don't understand how reading from the file helps in this case. We didn't find all the hashes but also haven't created the file at this point either unless we are relying running the command again to read the previous state

I think this is a failsafe in for a subsequent execution where something went terribly wrong during the truncation and restore-from-file process.

what would be nice, but not idiomatic, is if we had a few "are you sure?" or "do you want to?" prompts that could be bypassed by -y similar to ubuntu apt commands

jframe · 2023-12-15T05:52:43Z

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java

+        (key, value) -> {
+          updater.getTrieLogStorageTransaction().put(key, value);
+        });
+    updater.getTrieLogStorageTransaction().commit();


If this fails then do have a way of restoring the trielog from the backup?

Signed-off-by: Simon Dudley <[email protected]> Signed-off-by: Gabriel Fukushima <[email protected]>

Signed-off-by: Gabriel Fukushima <[email protected]>

…tIdentifier.reset() Signed-off-by: Gabriel Fukushima <[email protected]>

Signed-off-by: Gabriel Fukushima <[email protected]>

…emory Signed-off-by: Gabriel Fukushima <[email protected]>

garyschulte · 2023-12-15T19:01:17Z

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/RocksDbUsageHelper.java

@@ -34,17 +34,17 @@ static void printUsageForColumnFamily(
      final RocksDB rocksdb, final ColumnFamilyHandle cfHandle, final PrintWriter out)
      throws RocksDBException, NumberFormatException {
    final String size = rocksdb.getProperty(cfHandle, "rocksdb.estimate-live-data-size");
+    final String numberOfKeys = rocksdb.getProperty(cfHandle, "rocksdb.estimate-num-keys");


changes here seem fine but unrelated. Every time I look at this class name though I think it is a CLI Usage helper, like the output that gets printed when one asks for help or gets command usage wrong. completely unrelated, but if we are going to make changes in this pr ... :)

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java

garyschulte · 2023-12-15T19:13:20Z

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java

+  private static IdentityHashMap<byte[], byte[]> readTrieLogsFromFile() throws IOException {
+
+    File file = new File(trieLogFile);
+    IdentityHashMap<byte[], byte[]> trieLogs = new IdentityHashMap<>();


trielogs can vary in size, and the retention count is user configurable. We could easily blow up the heap with bad settings. We might want to add some sanity check on the file size here, and perhaps also in trielog streaming/save to file.

Agree, users could eventually set the layersToRetain to a high number and this could blow up. Is there real scenario where we should handle a high number having in mind that rolling back too many layer is not something feasible in bonsai (afaik)?

Trielogs can be the basis of a specific worldstate for debug_ and trace_ endpoints. But really if that were the case I imagine the node would be an archive node. I think streaming to and from disk would likely just be more bulletproof.

Not blocking feedback, just seems a robust mechanism

I've changed this so we process it in batches now so it should be a safer mechanism.

garyschulte

Looks good. I think it wouldn't be too difficult to stream trielogs to and from the file directly to prevent OOM and heap issues.

Signed-off-by: Gabriel Fukushima <[email protected]>

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java

matkt · 2023-12-20T08:48:37Z

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java

+      LOG.info("Restoring trie logs retained from batch {}...", batchNumber);
+      recreateTrieLogs(rootWorldStateStorage, batchNumber, batchFileNameBase);
+    } catch (IOException | ClassNotFoundException e) {
+      LOG.error("Error recreating trie logs from batch {}: {}", batchNumber, e.getMessage());


What should we do at this point? Do we have a way to restart from an existing restore file?

I'm going to change this to throw a new Runtime exception so we abort the process here, meaning if you re-run the command it's gonna detect the files exist and skip the save to file part but it will still reset the db and try to restore from file.

Signed-off-by: Gabriel Fukushima <[email protected]>

...reum/core/src/main/java/org/hyperledger/besu/ethereum/trie/bonsai/trielog/TrieLogPruner.java

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java

Signed-off-by: Gabriel Fukushima <[email protected]>

…trie-log-subcommand-2

* mark deleted slot during clear storage step (hyperledger#6305) Signed-off-by: Karim Taam <[email protected]> Co-authored-by: garyschulte <[email protected]> * made directory structure of tests match source; fixed one typo (hyperledger#6337) Signed-off-by: Sally MacFarlane <[email protected]> * migrate controller tests to junit 5 (hyperledger#6338) Signed-off-by: Sally MacFarlane <[email protected]> * add new forkids for testnets, update forkid test to Junit5, no longer need named network specific trusted setups (hyperledger#6322) Signed-off-by: jflo <[email protected]> * Fix trielog shipping issue during self destruct (hyperledger#6340) * fix trielog shipping issue Signed-off-by: Karim Taam <[email protected]> * bump gradle properties version and adjust changelog to match release (hyperledger#6347) Signed-off-by: garyschulte <[email protected]> * finalized cancun spec (hyperledger#6351) * finalized cancun spec Signed-off-by: jflo <[email protected]> * finalized cancun spec Signed-off-by: jflo <[email protected]> --------- Signed-off-by: jflo <[email protected]> * Optimize RocksDB WAL file (hyperledger#6328) Signed-off-by: Fabio Di Fabio <[email protected]> * Make RPC reason settable, pass execution failure reason in RPC error message (hyperledger#6343) * Make RPC reason settable, pass execution failure reason in RPC error message Signed-off-by: Matthew Whitehead <[email protected]> * Update unit tests Signed-off-by: Matthew Whitehead <[email protected]> * Update tests Signed-off-by: Matthew Whitehead <[email protected]> * Update change log Signed-off-by: Matthew Whitehead <[email protected]> * Update integration tests Signed-off-by: Matthew Whitehead <[email protected]> --------- Signed-off-by: Matthew Whitehead <[email protected]> * TestWatcher junit5 (hyperledger#6339) * TestWatcher junit5 * add test class and method name to context * moved the testwatcher junit5 function to a new junit5 superclass * one qbft test to junit5 superclass Signed-off-by: Sally MacFarlane <[email protected]> --------- Signed-off-by: Sally MacFarlane <[email protected]> Co-authored-by: Stefan Pingel <[email protected]> * Migrate BFT tests to junit 5 (hyperledger#6350) * bft tests to junit 5 * base class for pki extend AcceptanceTestBaseJunit5 * try/catch in case of empty optionals * fixed parameterization method Signed-off-by: Sally MacFarlane <[email protected]> --------- Signed-off-by: Sally MacFarlane <[email protected]> * fixing on selfdestruct (hyperledger#6359) Signed-off-by: Karim Taam <[email protected]> * migrate clique tests fully to junit5 (hyperledger#6362) * migrate clique tests fully to junit5 Signed-off-by: Sally MacFarlane <[email protected]> --------- Signed-off-by: Sally MacFarlane <[email protected]> * fixed link to logging docs (hyperledger#6366) Signed-off-by: Sally MacFarlane <[email protected]> * Move logging to RunnerBuilder (hyperledger#6367) Signed-off-by: Gabriel-Trintinalia <[email protected]> * Use synchronized call to access the chain head block in `eth_estimateGas` (hyperledger#6345) * Use synchronized call to access the chain head block in estimateGas() Signed-off-by: Matthew Whitehead <[email protected]> * Add error log entries when throwing internal error from estimateGas() Signed-off-by: Matthew Whitehead <[email protected]> * Update unit tests Signed-off-by: Matthew Whitehead <[email protected]> * Update changelog Signed-off-by: Matthew Whitehead <[email protected]> --------- Signed-off-by: Matthew Whitehead <[email protected]> * Add --X-trie-log subcommand (hyperledger#6303) * Add x-trie-log subcommand for one-off trie log backlog prune Signed-off-by: Simon Dudley <[email protected]> Signed-off-by: Gabriel Fukushima <[email protected]> --------- Signed-off-by: Simon Dudley <[email protected]> Signed-off-by: Gabriel Fukushima <[email protected]> Co-authored-by: Simon Dudley <[email protected]> * fix typos (hyperledger#6368) Signed-off-by: vuittont60 <[email protected]> * Added alias --sync-min-peers for --fast-sync-min-peers (hyperledger#6372) * sync-min-peers as an alias * added unit tests Signed-off-by: Sally MacFarlane <[email protected]> --------- Signed-off-by: Sally MacFarlane <[email protected]> * Fix: Fallback to getName when canonicalName is null in BlockHeaderValidator DEBUG log (hyperledger#6332) * fallback to simple name when canonical name is null * use getName instead of getSimpleName to include the package name Signed-off-by: Manoj P R <[email protected]> --------- Signed-off-by: Manoj P R <[email protected]> Co-authored-by: Sally MacFarlane <[email protected]> * fix: use UID 1000 for besu user (hyperledger#6358) (hyperledger#6360) The openjdk-latest Docker image is using UID 1001 for besu, because its base image ubuntu:23.10 now contains a default "ubuntu" user with UID 1000. (This UID change causes the besu user with UID 1001 to not have access to files created for past versions with UID 1000.) We now remove the default ubuntu user and explicitly use UID 1000 when creating the besu user. Signed-off-by: Hal Blackburn <[email protected]> * Ignore generated files when running the spdx license check (hyperledger#6379) Signed-off-by: Meredith Baxter <[email protected]> * full sync - don't fail startup if sync-min-peers specified (hyperledger#6373) Signed-off-by: Sally MacFarlane <[email protected]> * Copy also computed fields, when doing a Transaction detached copy (hyperledger#6329) Signed-off-by: Fabio Di Fabio <[email protected]> * Disable txpool when not in sync (hyperledger#6302) Signed-off-by: Fabio Di Fabio <[email protected]> * Bump to nex release snapshot 24.1.1 (hyperledger#6383) * release next snapshot 24.1.1 Signed-off-by: garyschulte <[email protected]> * move recent changelog items to 24.1.1-SNAPSHOT Signed-off-by: garyschulte <[email protected]> --------- Signed-off-by: garyschulte <[email protected]> * Correct Tangerine Whistle definition in Fluent EVM APIs. (hyperledger#6382) The fluent API incorrectly added the code size limit in Tangerine Whistle instead of first adding it in Spurious Dragon. Signed-off-by: Danno Ferrin <[email protected]> * [MINOR] Fix pki tests condition check on mac (hyperledger#6387) Signed-off-by: Gabriel-Trintinalia <[email protected]> * Upgrade dependencies (hyperledger#6377) * Bump com.github.oshi:oshi-core to 6.4.10 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump com.github.tomakehurst to org.wiremock 3.3.1 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump com.google.auto.service:auto-service to 1.1.1 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump com.google.dagger group to 2.50 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump com.graphql-java:graphql-java to 21.3 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump com.splunk.logging:splunk-library-javalogging to 1.11.8 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump com.squareup.okhttp3:okhttp to 4.12.0 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump commons-io:commons-io to 2.15.1 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump dnsjava:dnsjava to 3.5.3 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump info.picocli group to 4.7.5 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump io.grpc group to 1.60.1 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump io.kubernetes:client-java to 18.0.1 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump io.netty group to 4.1.104.Final Signed-off-by: Fabio Di Fabio <[email protected]> * Bump net.java.dev.jna:jna to 5.14.0 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump org.apache.commons:commons-compress to 1.25.0 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump org.apache.commons:commons-lang3 to 3.14.0 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump org.apache.commons:commons-text to 1.11.0 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump org.apache.logging.log4j group to 2.22.1 Signed-off-by: Fabio Di Fabio <[email protected]> * Redorder io.tmio group Signed-off-by: Fabio Di Fabio <[email protected]> * Bump org.assertj:assertj-core to 3.25.1 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump org.bouncycastle group to 1.77 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump org.fusesource.jansi:jansi to 2.4.1 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump org.immutables group 2.10.0 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump org.java-websocket:Java-WebSocket to 1.5.5 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump org.jetbrains.kotlin:kotlin-stdlib to 1.9.22 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump org.junit.jupiter group to 5.10.1 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump org.jupnp group to 2.7.1 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump org.rocksdb:rocksdbjni to 8.9.1 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump org.slf4j group to 2.0.10 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump org.springframework.security:spring-security-crypto to 6.2.1 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump org.testcontainers:testcontainers to 1.19.3 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump org.web3j group to 4.10.3 Signed-off-by: Fabio Di Fabio <[email protected]> * Bump org.xerial.snappy:snappy-java to 1.1.10.5 Signed-off-by: Fabio Di Fabio <[email protected]> * Regenerate gradle verification metadata Signed-off-by: Fabio Di Fabio <[email protected]> * Update commons-codec:commons-codec to 1.16.0 Signed-off-by: Fabio Di Fabio <[email protected]> * Update org.junit.vintage:junit-vintage-engine to 5.10.1 Signed-off-by: Fabio Di Fabio <[email protected]> * Update CHANGELOG Signed-off-by: Fabio Di Fabio <[email protected]> --------- Signed-off-by: Fabio Di Fabio <[email protected]> * add a fallback for docker detection on Mac (hyperledger#6356) Signed-off-by: garyschulte <[email protected]> * Fix test flackyness of acceptanceTestsPermissioning (hyperledger#6384) Signed-off-by: Fabio Di Fabio <[email protected]> Signed-off-by: Sally MacFarlane <[email protected]> Co-authored-by: Sally MacFarlane <[email protected]> * Upgrade `com.fasterxml.jackson` dependencies (hyperledger#6378) Signed-off-by: Fabio Di Fabio <[email protected]> * Use mining beneficiary from protocol spec in TraceServiceImpl (hyperledger#6390) * use mining beneficiary from protocol spec Signed-off-by: Daniel Lehrner <[email protected]> --------- Signed-off-by: Daniel Lehrner <[email protected]> Co-authored-by: Sally MacFarlane <[email protected]> * Update verification metadata and allowed licenses for Linea-Besu --------- Signed-off-by: Karim Taam <[email protected]> Signed-off-by: Sally MacFarlane <[email protected]> Signed-off-by: jflo <[email protected]> Signed-off-by: garyschulte <[email protected]> Signed-off-by: Fabio Di Fabio <[email protected]> Signed-off-by: Matthew Whitehead <[email protected]> Signed-off-by: Gabriel-Trintinalia <[email protected]> Signed-off-by: Simon Dudley <[email protected]> Signed-off-by: Gabriel Fukushima <[email protected]> Signed-off-by: vuittont60 <[email protected]> Signed-off-by: Manoj P R <[email protected]> Signed-off-by: Hal Blackburn <[email protected]> Signed-off-by: Meredith Baxter <[email protected]> Signed-off-by: Danno Ferrin <[email protected]> Signed-off-by: Daniel Lehrner <[email protected]> Co-authored-by: Karim TAAM <[email protected]> Co-authored-by: garyschulte <[email protected]> Co-authored-by: Sally MacFarlane <[email protected]> Co-authored-by: Justin Florentine <[email protected]> Co-authored-by: Fabio Di Fabio <[email protected]> Co-authored-by: Matt Whitehead <[email protected]> Co-authored-by: Stefan Pingel <[email protected]> Co-authored-by: Gabriel-Trintinalia <[email protected]> Co-authored-by: Gabriel Fukushima <[email protected]> Co-authored-by: Simon Dudley <[email protected]> Co-authored-by: vuittont60 <[email protected]> Co-authored-by: Manoj P R <[email protected]> Co-authored-by: Hal Blackburn <[email protected]> Co-authored-by: mbaxter <[email protected]> Co-authored-by: Danno Ferrin <[email protected]> Co-authored-by: daniellehrner <[email protected]>

gfukushima changed the title ~~X trie log subcommand~~ Add --X-trie-log subcommand Dec 15, 2023

gfukushima force-pushed the x-trie-log-subcommand-2 branch 2 times, most recently from 20960ce to 9f149a5 Compare December 15, 2023 05:53

jframe reviewed Dec 15, 2023

View reviewed changes

siladu and others added 8 commits December 15, 2023 16:00

Add x-trie-log subcommand for one-off backlog prune

16c0a49

Signed-off-by: Simon Dudley <[email protected]> Signed-off-by: Gabriel Fukushima <[email protected]>

long -> int

7dd4928

Signed-off-by: Simon Dudley <[email protected]> Signed-off-by: Gabriel Fukushima <[email protected]>

Removed banned method

bf2b098

Signed-off-by: Gabriel Fukushima <[email protected]>

Preload process stream in parallel

e67ae51

Signed-off-by: Gabriel Fukushima <[email protected]>

Drop unwanted trielogs and keep reatain layers only

9b4e0c9

Signed-off-by: Gabriel Fukushima <[email protected]>

Add output to user and cleanup refactor

0b9fe83

Signed-off-by: Gabriel Fukushima <[email protected]>

small tweak to display cf that had reference dropped by RocksDbSegmen…

426848e

…tIdentifier.reset() Signed-off-by: Gabriel Fukushima <[email protected]>

spotless

7401b59

Signed-off-by: Gabriel Fukushima <[email protected]>

gfukushima force-pushed the x-trie-log-subcommand-2 branch from 9f149a5 to 7401b59 Compare December 15, 2023 06:00

gfukushima added 4 commits December 15, 2023 16:27

Fix classes that changed package

1b7fb72

Signed-off-by: Gabriel Fukushima <[email protected]>

spotless

11e6b05

Signed-off-by: Gabriel Fukushima <[email protected]>

Code review

f2d01e2

Signed-off-by: Gabriel Fukushima <[email protected]>

Only clear DB when we have the exact amount of trie logs we want in m…

04f1aaa

…emory Signed-off-by: Gabriel Fukushima <[email protected]>

non-fungible-nelson requested a review from matkt December 15, 2023 14:37

garyschulte reviewed Dec 15, 2023

View reviewed changes

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java Outdated Show resolved Hide resolved

garyschulte reviewed Dec 15, 2023

View reviewed changes

gfukushima added 3 commits December 18, 2023 13:02

Trielogs stream to and from file to avoid possibly OOM

2f01c5a

Signed-off-by: Gabriel Fukushima <[email protected]>

Process trie logs in chunks to avoid OOM

56e4c8e

Signed-off-by: Gabriel Fukushima <[email protected]>

save and read in batches to handle edge cases

78561b0

Signed-off-by: Gabriel Fukushima <[email protected]>

gfukushima added the TeamGroot GH issues worked on by Groot Team label Dec 19, 2023

gfukushima added 2 commits December 20, 2023 14:42

save and read files to/from database dir

42c72cf

Signed-off-by: Gabriel Fukushima <[email protected]>

Merge branch 'main' into x-trie-log-subcommand-2

9961fc2

gfukushima marked this pull request as ready for review December 20, 2023 06:31

gfukushima requested a review from garyschulte December 20, 2023 06:31

matkt reviewed Dec 20, 2023

View reviewed changes

gfukushima added 3 commits December 21, 2023 15:20

add unit tests and PR review fixes

9389540

Signed-off-by: Gabriel Fukushima <[email protected]>

Merge branch 'main' into x-trie-log-subcommand-2

e3d4fbc

spdx

c7144fe

Signed-off-by: Gabriel Fukushima <[email protected]>

gfukushima mentioned this pull request Dec 21, 2023

Removal of old trie logs specified by historical block limit #5390

Closed

21 tasks

gfukushima added 2 commits December 21, 2023 20:52

Fix unit tests directory creation and deletion

20b0ba5

Signed-off-by: Gabriel Fukushima <[email protected]>

Merge branch 'main' into x-trie-log-subcommand-2

e7d175c

jframe reviewed Jan 4, 2024

View reviewed changes

gfukushima added 3 commits January 4, 2024 16:20

PR review

b214bf2

Signed-off-by: Gabriel Fukushima <[email protected]>

Merge branch 'main' into x-trie-log-subcommand-2

aa75348

Merge remote-tracking branch 'origin/x-trie-log-subcommand-2' into x-…

2bc0732

…trie-log-subcommand-2

gfukushima mentioned this pull request Jan 8, 2024

Import export trie log #6363

Merged

Merge branch 'main' into x-trie-log-subcommand-2

deec021

jframe approved these changes Jan 8, 2024

View reviewed changes

Merge branch 'main' into x-trie-log-subcommand-2

5d7b68e

gfukushima merged commit e51e042 into hyperledger:main Jan 8, 2024
18 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add --X-trie-log subcommand #6303

Add --X-trie-log subcommand #6303

gfukushima commented Dec 15, 2023 •

edited

Loading

github-actions bot commented Dec 15, 2023

jframe Dec 15, 2023

gfukushima Dec 15, 2023

jframe Dec 15, 2023

jframe Dec 15, 2023

gfukushima Dec 15, 2023

jframe Dec 15, 2023

garyschulte Dec 15, 2023

jframe Dec 15, 2023

garyschulte Dec 15, 2023

garyschulte Dec 15, 2023

gfukushima Dec 17, 2023 •

edited

Loading

garyschulte Dec 17, 2023

gfukushima Jan 3, 2024

garyschulte left a comment

matkt Dec 20, 2023

gfukushima Dec 20, 2023


		private static IdentityHashMap<byte[], byte[]> readTrieLogsFromFile() throws IOException {

		File file = new File(trieLogFile);

Add --X-trie-log subcommand #6303

Add --X-trie-log subcommand #6303

Conversation

gfukushima commented Dec 15, 2023 • edited Loading

PR description

Fixed Issue(s)

github-actions bot commented Dec 15, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gfukushima Dec 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

garyschulte left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gfukushima commented Dec 15, 2023 •

edited

Loading

gfukushima Dec 17, 2023 •

edited

Loading