[BESU-169] cache logs bloom filters automatically.#367
Conversation
Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
…-indexing Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
…-indexing Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
…-indexing Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
…-indexing Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
…-indexing Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
…-indexing Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
3a6861a to
8a3c988
Compare
…-indexing Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
…-indexing Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
…-indexing Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
shemnon
left a comment
There was a problem hiding this comment.
So it should be 'caching' not 'indexing' everywhere. And 'auto log bloom cache' to align from the existing CLI verbiage. Log is singular in this case.
| params.add("--key-value-storage"); | ||
| params.add("rocksdb"); | ||
|
|
||
| params.add("--auto-logs-bloom-indexing-enabled"); |
There was a problem hiding this comment.
| params.add("--auto-logs-bloom-indexing-enabled"); | |
| params.add("--auto-log-bloom-caching-enabled"); |
| .map(EnodeURL::fromString) | ||
| .collect(Collectors.toList())) | ||
| .besuPluginContext(new BesuPluginContextImpl()) | ||
| .autoLogsBloomIndexing(false) |
There was a problem hiding this comment.
| .autoLogsBloomIndexing(false) | |
| .autoLogBloomCaching(false) |
| public void cacheLogsBloomForBlockHeader(final BlockHeader blockHeader) { | ||
| try { | ||
| final long blockNumber = blockHeader.getNumber(); | ||
| LOG.info("Caching logs bloom for block {}.", "0x" + Long.toHexString(blockNumber)); |
There was a problem hiding this comment.
This should be debug - it happens way too often.
| } | ||
|
|
||
| public IndexingStatus indexAll() { | ||
| return generateLogBloomCache(0, Long.MAX_VALUE); |
There was a problem hiding this comment.
This can result in indexing running twice... perhaps ensurePreviousSegmentsArePresent(blockchain.getChainHeadBlockNumber());?
There was a problem hiding this comment.
Yes, good suggestion. Done.
| long currentSegment = (blockNumber / BLOCKS_PER_BLOOM_CACHE) - 1; | ||
| while (currentSegment > 0) { | ||
| try { | ||
| if (!isCachePresentForSegment(currentSegment)) { |
There was a problem hiding this comment.
I think we should track what segments we have made in a list or tree map. Check the map before going to disk.
| } | ||
|
|
||
| private void ensurePreviousSegmentsArePresent(final long blockNumber) { | ||
| scheduler.scheduleFutureTask( |
There was a problem hiding this comment.
We should check indexing status before submitting the work.
shemnon
left a comment
There was a problem hiding this comment.
Renaming from indexing to caching should also extend to the mistakes of the previous author (me!). TransactionLogsIndexer -> TransactionLogsCacher and IndexingStatus -> CachingStatus
shemnon
left a comment
There was a problem hiding this comment.
On third thought, save the renaming for a follow on PR and focus on the other comments I left for implementation details.
…-indexing Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
* First iteration. Draft PR. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * fix SPDX header Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Use block broadcaster to index log bloom. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Remove useless toString method Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * spotless apply Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * cacheLogsBloomForBlockHeader Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * spotless apply Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * ensurePreviousSegmentsArePresent Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Added CLI flag to enable / disable automatic logs bloom indexing. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Create cache directory and cache file if not exist. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Fix acceptance test Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Write cache for block only if block is new canonical head. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Handling of chain reorg. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * fix Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * sportless apply Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Address PR comments. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Remove unused constant. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * spotless apply Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> (cherry picked from commit 6677362) Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* Plugin error stack traces (besu-eth#369) Because of how the Log4J2 api works exception stack traces were not being printed. Update to use the explicit "throwable" overloaded methods. Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com> * VM Trace fixes (besu-eth#372) * correct refund addresses * correct returned memory from static precompiled calls. * update integration test * precompiles sometimes get plain old CALLs Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com> * Validate private transaction before sending to enclave (besu-eth#356) Signed-off-by: Jason Frame <jasonwframe@gmail.com> * Trace API fixes (besu-eth#377) - Correct Reporting of reverts in nested call - correct reporting and handling of value transfer in nested calls - correct handling of precompiles via DELEGATECALL & CALLCODE - Addition of precompiled contract gas costs - Re-work handling of storage writes - Initial handling of gas refunds - fix bug in DELEGATECALL tests, we don't need gas in the stack * this has a cascading effect on balances in diff tests - rework depth detection in flat trace - two new tests blocks Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com> * [BOUNTY-2] Add NAT Docker Support (besu-eth#368) * add docker detection Signed-off-by: Karim TAAM <karim.t2am@gmail.com> * add port mapping detection Signed-off-by: Karim TAAM <karim.t2am@gmail.com> * add tests and refactor ip detection Signed-off-by: Karim TAAM <karim.t2am@gmail.com> * clean RunnerBuilder Signed-off-by: Karim TAAM <karim.t2am@gmail.com> * clean useless modification Signed-off-by: Karim TAAM <karim.t2am@gmail.com> * spotless Signed-off-by: Karim TAAM <karim.t2am@gmail.com> * resolve tests issues Signed-off-by: Karim TAAM <karim.t2am@gmail.com> * streamline auto detection Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com> Co-authored-by: Abdelhamid Bakhta <45264458+abdelhamidbakhta@users.noreply.github.com> Co-authored-by: Ratan Rai Sur <ratan.r.sur@gmail.com> * [PIE-1798] Priv RPC acceptance tests with stub enclave. (besu-eth#330) * [PIE-1798] Added some Privacy RPC ATs with a stub enclave. Signed-off-by: Mark Terry <mark.terry@consensys.net> * More specific task metrics names (besu-eth#389) A prior refactoring had accidentally removed the specific task names from the metrics labels. Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com> * More trace fixes (besu-eth#386) * pop flat trace context when handling halts * Better detection of precompiled and non-executed contracts * correct from address when calling in init code * fix some exotic nesting cases * correct from field for init code calls at depth >1 * correct cost on a non-call * changelog and notes Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com> * adding the plugin-api javadoc jar at the root level (besu-eth#378) Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net> * BESU-56: remove erroneous links to errorprone checks (besu-eth#385) Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com> * Changelog entry for multi-tenancy feature (besu-eth#394) Signed-off-by: Edward Evans <edward.joshua.evans@gmail.com> Signed-off-by: Jason Frame <jasonwframe@gmail.com> * 1.5 RC Changelog (besu-eth#395) * 1.5 RC changelog additions Signed-off-by: Sally MacFarlane <sally.macfarlane@consensys.net> * Fixed link (besu-eth#393) Signed-off-by: Madeline <madeline.murray@consensys.net> Co-authored-by: Edward <edjojob@gmail.com> * docker changelog (besu-eth#391) * docker changelog Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com> * address comments Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com> Co-authored-by: MadelineMurray <43356962+MadelineMurray@users.noreply.github.com> * Added known bug to changelog (besu-eth#388) * Added known bug Signed-off-by: Madeline <madeline.murray@consensys.net> * Added another known bug Signed-off-by: Madeline <madeline.murray@consensys.net> * updating version to 1.4.0-rc1 (besu-eth#397) Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net> * updating verion to 1.4.1-snapshot (besu-eth#398) Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net> * [BESU-169] cache logs bloom filters automatically. (besu-eth#367) * First iteration. Draft PR. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * fix SPDX header Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Use block broadcaster to index log bloom. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Remove useless toString method Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * spotless apply Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * cacheLogsBloomForBlockHeader Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * spotless apply Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * ensurePreviousSegmentsArePresent Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Added CLI flag to enable / disable automatic logs bloom indexing. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Create cache directory and cache file if not exist. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Fix acceptance test Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Write cache for block only if block is new canonical head. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Handling of chain reorg. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * fix Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * sportless apply Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Address PR comments. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Remove unused constant. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * spotless apply Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * [BESU-25] Use Devp2p Ping packets at v5 (besu-eth#392) Broadcast that we support snappy compression Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com> * Rename logs bloom indexer to log bloom cache to match CLI flag. (besu-eth#401) Changes class names, variables, and CLI flags as needed. Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com> * [PIE-1798] Fail cases for multitenancy ATs (besu-eth#400) Signed-off-by: Mark Terry <mark.terry@consensys.net> * Private state update metadata and migration (besu-eth#404) (backport from release-1.4) Private state update metadata and migration Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net> * [PIE-2303] Automatic log bloom caching - Remove usage of pending file. (besu-eth#407) * Don't use pending file. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Don't use pending file. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Create a custom error when plugin is not found. (besu-eth#409) Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Rename method (besu-eth#412) * rename the method isPersistingState to isPersistingPrivateState because that is what it is used for Signed-off-by: Stefan Pingel <stefan.pingel@consensys.net> * rename the method isPersistingState to isPersistingPrivateState because that is what it is used for Signed-off-by: Stefan Pingel <stefan.pingel@consensys.net> * rename the method isPersistingState to isPersistingPrivateState because that is what it is used for Signed-off-by: Stefan Pingel <stefan.pingel@consensys.net> * [BOUNTY-4] Add NAT Kubernetes Support (besu-eth#410) * add kubernetes support Signed-off-by: Karim TAAM <karim.t2am@gmail.com> * fix review issues Signed-off-by: Karim TAAM <karim.t2am@gmail.com> * LogBloomCache - make sure the current segment is filled (besu-eth#411) Make sure we cache the current cache segment with all of the data from the beginning of the segment. Use a flip file approach since it will be a partial file until done. Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com> Co-authored-by: Abdelhamid Bakhta <45264458+abdelhamidbakhta@users.noreply.github.com> * Reduce recaching in Transaction Log Bloom Filter Cache (besu-eth#415) Do a cursory cache check at start up (file is present and correct size) instead of re-generating the cache at startup. Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com> * fix order of nat detector (besu-eth#414) Signed-off-by: Karim TAAM <karim.t2am@gmail.com> * Update SLOAD_GAS cost to 200 in Aztlan Gas Calculator (#23) (besu-eth#382) * Update SLOAD_GAS cost to 200 in Aztlan Gas Calculator Change SLOAD_GAS cost in Aztlan Gas Calculator from 800 to 200 and update functions that use SLOAD_GAS. Signed-off-by: edwardmack <ed@edwardmack.com> * Update SLOAD_GAS cost to 200 in Aztlan Gas Calculator Change SLOAD_GAS cost in Aztlan Gas Calculator from 800 to 200 and update functions that use SLOAD_GAS. Signed-off-by: edwardmack <ed@edwardmack.com> Signed-off-by: Edward Mack <ed@edwardmack.com> * remove overrides removed overrides of calculateStorageCost and calculateStorageRefundAmount in AztlanGasCalculator because these were causing aztlan fork not to sync with kotti testnet. Signed-off-by: Edward Mack <ed@edwardmack.com> * merge Signed-off-by: Edward Mack <ed@edwardmack.com> * Remove unused ExecutorService init/termination (besu-eth#419) Signed-off-by: Horacio Mijail Anton Quiles <hmijail@gmail.com> Co-authored-by: CJ Hare <CjHare@users.noreply.github.com> * Implement Eth/64 (besu-eth#425) Wire in the fork identifier into the status messages as Eth64. Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com> * Adds priv_getcode (besu-eth#428) * Adds priv_getcode Signed-off-by: Joshua Richardson <joshua@richardson.tech> * BESU-146 - check if success and return errorResponse otherwise (besu-eth#424) Signed-off-by: Anthony Buckle <anthonybuckle@gmail.com> Co-authored-by: CJ Hare <CjHare@users.noreply.github.com> * fixed typos (besu-eth#429) Signed-off-by: Sally MacFarlane <sally.macfarlane@consensys.net> Co-authored-by: CJ Hare <CjHare@users.noreply.github.com> * roll back on ALL-CAPS.md files for TSC proposal (besu-eth#376) Signed-off-by: Felipe Faraggi <felipefaraggi@gmail.com> * updating the changelog with 1.4.0 details (besu-eth#431) Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net> * [BESU-194] Remove max pivot block resets during fast sync (besu-eth#427) * remove max pivot block resets during fast sync * increase max retry number and fix test * change logs in the handleFailure method * change logs related to suspicious number of retries Signed-off-by: Karim TAAM <karim.t2am@gmail.com> Co-authored-by: Danno Ferrin <danno.ferrin@shemnon.com> Co-authored-by: Jason Frame <jasonwframe@gmail.com> Co-authored-by: Karim T. <karim.t2am@gmail.com> Co-authored-by: Abdelhamid Bakhta <45264458+abdelhamidbakhta@users.noreply.github.com> Co-authored-by: Ratan Rai Sur <ratan.r.sur@gmail.com> Co-authored-by: mark-terry <36909937+mark-terry@users.noreply.github.com> Co-authored-by: Joshua Fernandes <joshua.fernandes@consensys.net> Co-authored-by: Antoine Toulme <atoulme@users.noreply.github.com> Co-authored-by: Sally MacFarlane <sally.macfarlane@consensys.net> Co-authored-by: MadelineMurray <43356962+MadelineMurray@users.noreply.github.com> Co-authored-by: Edward <edjojob@gmail.com> Co-authored-by: Lucas Saldanha <lucascrsaldanha@gmail.com> Co-authored-by: pinges <16143240+pinges@users.noreply.github.com> Co-authored-by: Horacio Mijail Antón Quiles <4139546+hmijail@users.noreply.github.com> Co-authored-by: CJ Hare <CjHare@users.noreply.github.com> Co-authored-by: anthonybuckle <anthonybuckle@gmail.com> Co-authored-by: Felipe Faraggi <felipefaraggi@gmail.com>
PR description
Log index is generated either by starting Besu using the
generate-log-bloom-cacheCLI subcommand or by calling theadmin_generatelogbloomcacheRPC method.However, it seems necessary to make this call or run this subcommand every once a while to regenerate log index cache.
A workaround is to have a cron task sending an RPC request to
admin_generatelogbloomcache, but then the return result have to be stored to be used as parameters for the next call.It then would be great to have Besu able to update the index cache by itself on a configurable schedule (every once x seconds, or every once x blocks, or automatically by letting Besu decide the best moment to update) and with an automatic management of the range to index (without the need of the two params the admin_generatelogbloomcache requires).
Fixed Issue(s)
https://jira.hyperledger.org/browse/BESU-169
Documentation changes required
--auto-logs-bloom-indexing-enabledCLI option: Enable Automatic logs bloom indexingTODO
Implement a hook to cache missing logs bloom at startup DONEImplement a hook to cache missing logs bloom on new block propagated. iterate over all previous segments to see if they are missing, trigger indexing if some are missing. DONESigned-off-by: Abdelhamid Bakhta abdelhamid.bakhta@consensys.net