feat: compute receipts root #111

ricomateo · 2024-07-02T18:18:42Z

Closes #106

**Motivation** Replace evm code in eftests with an implementation that uses ethrex types **Description** * Implement `execute_tx` using revm * Move `evm` crate into `core` crate * Move `Transaction` into its own module * Implement multiple getters for `Transaction` fields * Implement address recovery from signed transactions * Implement conversions between ef tests types and ethrex types * Add one more ef test  Closes #21

the value must be rlp(receipt) instead of tx_type || rlp(receipt) (if tx_type != 0)

the previous way in which the receipts trie values were computed was the right way

crates/core/src/types/block.rs

MegaRedHand

LGTM

**Motivation**  **Description**   Closes #issue_number

**Motivation** Obtain world state from account info stored in db **Description** * Implement storage root computation * Implement word state  Closes #44

**Motivation** Add support for EIP2930 transaction **Description** Implements EIP2930Transaction + required methods & traits   Closes None, but will enable #135 to also close #24

This PR is based on #138 (as they modify the same code), please merge it first **Motivation** Add support for EIP4844 transactions  **Description** Add `EIP4844` transactions + needed trait and methods and integrate it into existing transaction-related code   Closes #26 (evm already performs validations before executing so adding the transaction itself will sufice)

**Motivation** Add error handling to `execute_tx` and remove unwrap  **Description** Add `EvmError` Map revm's `EVMError` to `EvmError` Add error handling to `execute_tx`   Closes None

**Motivation**  **Description**   Closes #125 --------- Co-authored-by: Tomás Grüner <[email protected]>

**Motivation** Being able to store and fetch blocks from the db via `Store` api **Description** * Replace `Transaction::encode` with `Transaction::encode_with_type` logic (Reasoning: We cannot decode the transactions without knowing their type with the current behaviour, making it impossible to implement `RLPDecode` for `Transaction`) * Implement `RLPDecode` for `Transaction` (Using the logic that was previoulsy used in `EncodedTransaction::decode` (payload module)) * Implement `RLPDecode` for `Withdrawal`, `BlockHeader` and `BlockBody` * Add the folloewng methods to `Store` and `StoreEngine`: `add_block_header`, `add_block_body`, `get_block_header`, `get_block_body`, and implement them for `InMemory` and `Libmdbx` engine types   Closes None, but is needed for #145

**Motivation** Being able to serialize blocks matching the output required by the rpc spec  **Description** *Add struct `BlockSerializable` which can be serialized to match the specifications of the rpc. It can contain either the full transactions or just their hashes. *Implement serialization for `BlockSerializable`, `BlockHeader`, `BlockBody`, `Withdrawal` and `Transaction`. Other changes: *Replace `Bloom` with `ethereum_types::Bloom` *Rename some fields in `Eip1559Transaction` to match other transaction types' fields   Closes None, but is needed in order to implement #31

**Motivation** We currently have a 1 hour ttl for our rocksdb WAL for no discernible reason, and I think it's better to leave it to rocksdb to configure until we find a reason to change it. This 1 hour ttl is the main reason why the database size for our nodes drops massively an hour after snap sync finishes (in mainnet this drop is of more than 100 GB) **Description**   Closes #issue_number

**Motivation**  We are deploying releases without the ability to test them before. **Description**  The new workflow is: - All new versions should be tagged as RC first (e.g., `v6.0.0-rc.1`). - The CI will compile binaries and docker images, and will create a pre-release in Github. - New RC can be made and the process will repeat. - Once the binaries were confirmed to work, the pre-release needs to be edited, changing the name and creating a new tag without the "subversion" (e.g., `v6.0.0`). Also, the release should be set to `latest` now. - The CI will upload the previous compiled docker images with new tags `latest` and final version. Changelog was improved too. Now it takes commits since the first common parent of the previous version (not RC)

**Motivation**  **Description**   Closes #issue_number

This PR replaces the hash function used for the diff layers hash map to improve performance **Motivation**  **Description**   Closes #issue_number --------- Co-authored-by: Javier Chatruc <[email protected]>

**Motivation**  Committer is not always waking up after `committer_wake_up_ms` but `commit_time_ms` **Description**  Change the delay time to wake up again  --------- Co-authored-by: Tomás Grüner <[email protected]>

**Motivation**  Recently changes turned off ANSI colors when TTY is not detected. This is sometimes annoying (e.g., with `systemd`). **Description**  Add a `--log.color` flag (following `--log.level` format) with options `auto` (current and default behaviour), `always` and `never`  --------- Co-authored-by: Copilot <[email protected]>

**Motivation**  Avoid depending on unknown third-party actions **Description**  Create a custom free-disk action that removes Android SDK, .NET and Haskell. Other tools can be added later if more space is required

**Motivation** This PR moves the trie portion of the `apply_updates` function to a background thread, improving block execution times. This background thread receives messages through a channel to apply new trie updates and does two things: - First, it updates the in-memory diff layers and notifies the process that sent the message (i.e. the block production thread) so it can continue with block execution. - Second, it performs the logic of persisting the bottom-most diff layer to disk. To do this, we first pause the ongoing thread in charge of generating the snapshots (i.e. FlatKeyValue) through a message, then persist the diff layer to disk, then notify again the thread to continue snapshot generation. **Description**   Closes #issue_number --------- Co-authored-by: Mario Rugiero <[email protected]> Co-authored-by: Javier Chatruc <[email protected]> Co-authored-by: Javier Rodríguez Chatruc <[email protected]>

**Motivation** Enable hive p2p test BlobViolations This test behaves by first sending a `NewPooledTransactionHashes` message, and then responding to our node's `GetPooledTransaction` with a `NewPooledTransactions` message that contains transactions with sizes & types that don't match the ones advertised in the first message. As we were not keeping track of the received `NewPooledTransactionHashes` messages, we were not able to validate the incoming transactions. This PR fixes this problem and re-enables the test  **Description** * Track incoming `NewPooledTransactionHashes` * Enable hive p2p `BlobViolations` test   Part of #4941

During `init_l1`, we instanciate a `NodeRecord` named `local_node_record` that is used in a few places. A deep copy is passed to the RPC API via a call to `init_rpc`. A shallow copy (`Arc::clone`) is passed to the P2P layer via a call to `init_network`. Then the original instance is returned to the caller, which passes it on to `shutdown_process` at the end of the program lifetime. The following was observed by inspecting the code: - Only one of `NodeRecord`'s methods mutates it, and it's never called (deleted without errors); - The only other method receiving a mutable reference doesn't mutate it, as shown by the fact removing the `mut` caused no errors either; - `P2PContext` doesn't actually use `local_node_record`, it was deleted without issues; All of this means we can simply share a regular instance and refer to it by regular reference, avoinding any locking or indirection. The sequence number is currently only updated on startup, by initializing with an already incremented seqno. No perf improvements are expected, but the code is slightly cleaner without it. **Motivation**  **Description**   Closes #issue_number

…5051) Extract the block execution for payloads to an OS thread, make all its callees regular blocking functions, and removes the reliance on Tokio. Caveats: 1. The worker is in the RPC, mostly because the Blockchain is not Clone; 2. The L2 is broken from the removal of async, fee config mostly becomes a mess; 3. The channel is provided by Tokio because the RPC is still async and shouldn't block waiting. **Motivation**  **Description**   Closes #issue_number --------- Co-authored-by: Tomás Grüner <[email protected]> Co-authored-by: Javier Chatruc <[email protected]>

**Motivation**  `Url` is the correct type for the `EthClient` **Description**  Replace all `String` uses with `Url` when used for `EthClient`

) **Motivation**  Warning logs raise quick attention **Description**  Logs were left from a debugging session and are useless

) **Motivation**  Remove pending transactions from a list is very expensive as it has to shift and pop `n` elements. **Description**  Use a new variable to store the current first element index. Instead of removing elements from the list, increment the variable. These changes reduce the amount of gas used in a `verifyBatch` transaction. The gas used is no more variable by the amount of privileged transactions in the batch. Currently, 200 priv. txs costs around 1 MGas. With this, only around 100 kGas  --------- Co-authored-by: Copilot <[email protected]>

**Motivation** We shouldn't have useless clones, and it would be nice to have a warning for them if they appear. **Description** - Added a new clippy lint for redundant clones in the linter rules - Fixed redundant clones found by clippy - Added the rule to use the linter of the workspace for all cargo toml without linter rules **Notes** We have to manually add to use the workspace for each cargo toml Documentation for the new clippy lint: https://rust-lang.github.io/rust-clippy/master/index.html#redundant_clone **Motivation**  **Description**   Closes #issue_number --------- Co-authored-by: Tomás Grüner <[email protected]> Co-authored-by: Javier Rodríguez Chatruc <[email protected]>

**Motivation**  We use `ethrex --dev` as L1 in L2 tests and it'd be useful to test in an Osaka network. **Description**  Use `GetPayloadV5` in dev block producer

) **Motivation** In hive tests we often receive an fcu with either the genesis or last imported block as head to notify the node that it is already synced. As this is not usual in a real-case scenario, we just trigger a snap sync to the advertised head. This PR solves this by first checking if the fcu head is already part of our canonic chain before triggering a sync  **Description** * When handling a forkchoice in snap sync mode, check that the head is not already canonical before triggering a sync, if it is already canonical, apply the forkchoice   Closes #4846

**Motivation**  **Description**   Closes #issue_number --------- Co-authored-by: Pablo Deymonnaz <[email protected]>

chain_config is a matter of initialization with mutable stores, while latest_block_header is replaced by an ArcSwap to allow lock-free swapping with new values. **Motivation**  **Description**   Closes #issue_number --------- Co-authored-by: Lucas Fiegl <[email protected]> Co-authored-by: Javier Rodríguez Chatruc <[email protected]> Co-authored-by: Javier Chatruc <[email protected]>

…ed (#5144) **Motivation** There were compilation warnings while compiling ethrex-crypto with no features. **Description**   Closes #issue_number

**Motivation** The L2 CI is not running on some PRs it should.  **Description** The problem is that the required checks `Integration Test` and `Lint` have the same name for both L1 and L2. So if one of them passes, then the other is not needed to merge. This can be a problem in the case that the L1 CI finishes before the L2 one starts, in that case the L2 CI will not run. The solution is the following: - Add `Integration Test L2` and `Lint L2` to the branch protection rules - Rename the corresponding jobs on the L2 workflow. With only those changes, we will always require the L2 workflows to run in order to merge, since we want to be able only run the L1 CI in PRs that don't touch the L2, we need a way to skip it. Note: The current way of not running a workflow doesn't work, since skipping a complete workflow with `paths:` makes the jobs abscent, what we need is for the jobs to be marked as skipped. To do this, [dorny/paths-filter@v3](https://github.com/dorny/paths-filter) action is used, which performs the same action as the `paths:` option, but as a job itself. Then with the result of this job we can add ifs to the rest of the jobs to mark if they should be run. Examples with a fork of ethrex with the branch protection already active (For Integration Test L2) - [PR with changes only in L1](gianbelinche#4): It ran L1 checks and skipped L2 ones. - [PR with changes only in L2](gianbelinche#3): It ran L2 checks and skipped L1 ones. - [PR with both L1 and L2 changes](gianbelinche#5): It ran both L1 and L2 checks. You can also check that when the PRs were merged to main, all checks ran even though the PRs skipped the checks. https://github.com/gianbelinche/ethrex/commits/main/

**Motivation** We want to parallelize VM execution and merkelization. **Description**   Closes #issue_number --------- Co-authored-by: Mario Rugiero <[email protected]> Co-authored-by: Lucas Fiegl <[email protected]> Co-authored-by: Javier Rodríguez Chatruc <[email protected]> Co-authored-by: Javier Chatruc <[email protected]>

**Motivation**  We are saving the latest `Store` as the "current" one. This means the next time a batch is constructed, it might use a latter checkpoint than the needed. This can occur if the batch is sealed with less blocks than the available (e.g., if there's no more available space). **Description**  Save a checkpoint from the batch execution instead

**Motivation** Checkpoints are never deleted  **Description** Delete checkpoints once we verify the corresponding batch   --------- Co-authored-by: Manuel Iñaki Bilbao <[email protected]> Co-authored-by: Ivan Litteri <[email protected]>

**Motivation**  Reapply #4814 **Description**   --------- Co-authored-by: Tomás Paradelo <[email protected]> Co-authored-by: Tomás Paradelo <[email protected]> Co-authored-by: Gianbelinche <[email protected]> Co-authored-by: Estéfano Bargas <[email protected]> Co-authored-by: ilitteri <[email protected]> Co-authored-by: Ivan Litteri <[email protected]> Co-authored-by: Copilot <[email protected]>

**Motivation**  OZ's upgrade tools require the variables to be set in initializers and not in constructor/variable declaration. This should not be such a problem as it will always be initialized as 0, but with this change we avoid the tool error **Description**

**Motivation** Bloom filter was set in a small capacity. **Description**   Closes #issue_number --------- Co-authored-by: Javier Rodríguez Chatruc <[email protected]> Co-authored-by: Javier Chatruc <[email protected]>

**Motivation** When encoding non-legacy transactions we need to encode the payload (txType || rlp(Transaction) as a bytes object. In order to do so we copy the payload to a `Bytes` object and then encode it in rlp, this is not needed as we can just encode the payload as bytes by invoking the implementation for [u8] directly **Description** * Avoid using `Bytes::copy_from_slice` when encoding transactions

MegaRedHand and others added 10 commits July 1, 2024 13:03

feat: compute transactions root for a block

a685ff2

fix: was encoding LegacyTransaction incorrectly

611bc86

fix: RLP encode index

49f0374

Merge branch 'main' into compute-transactions-root

7080893

Merge branch 'main' into compute-transactions-root

5f9e6eb

refactor: add Transaction::encode_with_type

a81897f

refactor: add TxType enum for encoding tx type

b966cf4

refactor: add TxKind struct

2c59b6d

Merge branch 'main' into compute-transactions-root

830ecac

add receipts root computation

67f1db0

ricomateo added 4 commits July 2, 2024 17:07

fix: the receipt trie value was being computed wrong

3f2dac6

the value must be rlp(receipt) instead of tx_type || rlp(receipt) (if tx_type != 0)

fix: compute receipts trie value correctly

8e23699

the previous way in which the receipts trie values were computed was the right way

remove unnecessary constructor

3b32d29

merge main

ff376e1

ricomateo marked this pull request as ready for review July 2, 2024 20:59

ricomateo requested a review from a team as a code owner July 2, 2024 20:59

MegaRedHand reviewed Jul 2, 2024

View reviewed changes

crates/core/src/types/block.rs Outdated Show resolved Hide resolved

fix: move compute_receipts_root out of BlockBody impl

f02abf1

MegaRedHand approved these changes Jul 3, 2024

View reviewed changes

MegaRedHand merged commit b3a16bf into main Jul 3, 2024

MegaRedHand deleted the compute-receipts-root branch July 3, 2024 13:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: compute receipts root #111

feat: compute receipts root #111

Uh oh!

ricomateo commented Jul 2, 2024

Uh oh!

Uh oh!

MegaRedHand left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat: compute receipts root #111

feat: compute receipts root #111

Uh oh!

Conversation

ricomateo commented Jul 2, 2024

Uh oh!

Uh oh!

MegaRedHand left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants