Merged
Conversation
op-geth operates two TBC instances, a lightweight (header-only) instance which it uses to track Bitcoin consensus as known by the Hemi protocol based on L2 blocks, and a full instance which synchronizes with the P2P network and is used to index the Bitcoin chain. Hemi creates "Bitcoin Attributes Deposited" transactions which communicate new Bitcoin headers to be "known" by the Hemi protocol. The protocol uses these headers to maintain a complete, lightweight view of Bitcoin consensus that is synchronized across all Hemi nodes regardless of their view of Bitcoin's P2P network. When Hemi's lightweight view is advanced by new information as part of the Bitcoin derivation process, Hemi's embedded full node proceeds to index along the canonical tip of the lightweight view, always staying 2 blocks behind the known tip to prevent a data withholding attack against Hemi's state transition function. If the full node does not have the correct full blocks to advance its indexers to the delayed tip behind the lightweight view, it waits for these blocks to become available. The state of the full BTC node indexers must be identical across all Hemi nodes at a given L2 height so that hVM precompile calls are determinstic, otherwise nodes would calculate EVM state transitions incorrectly and cause a state divergence. Bitcoin Attributes Deposited transactions can only be created by the Sequencer - similar to other System transactions ([Ethereum] Attributes Deposited and PoP Payout). For now, a default starting Bitcoin testnet header is configured and will be used by default if not overridden, but the hVM Phase 0 activation height MUST be overridden. This update also provides bug-fixes for existing precompiles which were activated on Hemi Testnet prior to this update which makes hVM state deterministic across the Hemi network.
Fixed hVM activation time when not immediately after genesis
…rather than only when needed for EVM state transition calculation Fixed bug getting header when updateHvmHeaderConsensus is called during insertChain before header is written to disk Fixed off-by-one-error in ancestor search Added additional logging
…fter state transition, additional state transitions will still move TBC to correct point before validation to ensure consistency Temporarily only include one BTC header in each BTC Attributes Deposited block pending further TBC optimizations to increase responsiveness, not a consensus rule Fixed initial sync blockheight check More logging
Simplify TBC Full Node progression logic Update default Bitcoin starting height Fix logging
Fix misc. logging
Misc logging fixes
Remove misc logging
* check for fresh tbc directory and brute force index * bugfix * get best again * squashme * squashme * try syncing indexers * squashme * More Logging --------- Co-authored-by: Maxwell Sanchez <max@bloq.com>
.github/workflows/docker.yml
Outdated
| on: | ||
| push: | ||
| branches: [ "hemi" ] | ||
| # branches: [ "hemi" ] |
Author
There was a problem hiding this comment.
revert this before merge; it's commented out so we can deploy and test
ClaytonNorthey92
requested review from
AL-CT,
jcvernaleo,
joshuasing,
marcopeereboom and
max-sanchez
max-sanchez
approved these changes
Feb 5, 2025
dhidalgX
approved these changes
Feb 5, 2025
ClaytonNorthey92
added a commit
that referenced
this pull request
Apr 1, 2025
* hVM Phase 0: op-geth operates two TBC instances, a lightweight (header-only) instance which it uses to track Bitcoin consensus as known by the Hemi protocol based on L2 blocks, and a full instance which synchronizes with the P2P network and is used to index the Bitcoin chain. Hemi creates "Bitcoin Attributes Deposited" transactions which communicate new Bitcoin headers to be "known" by the Hemi protocol. The protocol uses these headers to maintain a complete, lightweight view of Bitcoin consensus that is synchronized across all Hemi nodes regardless of their view of Bitcoin's P2P network. When Hemi's lightweight view is advanced by new information as part of the Bitcoin derivation process, Hemi's embedded full node proceeds to index along the canonical tip of the lightweight view, always staying 2 blocks behind the known tip to prevent a data withholding attack against Hemi's state transition function. If the full node does not have the correct full blocks to advance its indexers to the delayed tip behind the lightweight view, it waits for these blocks to become available. The state of the full BTC node indexers must be identical across all Hemi nodes at a given L2 height so that hVM precompile calls are determinstic, otherwise nodes would calculate EVM state transitions incorrectly and cause a state divergence. Bitcoin Attributes Deposited transactions can only be created by the Sequencer - similar to other System transactions ([Ethereum] Attributes Deposited and PoP Payout). For now, a default starting Bitcoin testnet header is configured and will be used by default if not overridden, but the hVM Phase 0 activation height MUST be overridden. This update also provides bug-fixes for existing precompiles which were activated on Hemi Testnet prior to this update which makes hVM state deterministic across the Hemi network. * Update heminetwork tag to v0.2.7-hom (feature branch for header only mode) * Update heminetwork dependency * Update heminetwork dependency * Fix compatibility with heminetwork changes #1 * Updated marshalling for BtcAttributesDepositedTxType Fixed hVM activation time when not immediately after genesis * Logging fixes, flip sanity check to be correct * Update heminetwork version * Update heminetwork version * Optimize TBC full node progression to happen on each tip advancement rather than only when needed for EVM state transition calculation Fixed bug getting header when updateHvmHeaderConsensus is called during insertChain before header is written to disk Fixed off-by-one-error in ancestor search Added additional logging * Bump heminetwork version * Temporarily disabled restoring full-node indexers to previous state after state transition, additional state transitions will still move TBC to correct point before validation to ensure consistency Temporarily only include one BTC header in each BTC Attributes Deposited block pending further TBC optimizations to increase responsiveness, not a consensus rule Fixed initial sync blockheight check More logging * Update to work with upstream TBC changes Simplify TBC Full Node progression logic Update default Bitcoin starting height Fix logging * Changed updateHvmHeaderConsensus to use updateFullTBCToLightweight method * Update to use max-hom5 branch of heminetwork after rebase onto RawDB work * Bump heminetwork version Fix misc. logging * Set External Header Mode TBC's Block and Block Header caches to zero * Add temporary logging and attempt bug fix * Fix concurrent map modification bug Misc logging fixes * Logging fix * More logging * Update heminetwork version to add temp logging * Fix bug * Fix bug, and enable complete header-only node reset on invalid startup state * Move TBC header node and configuration earlier * Refactor some logic to init instead of setup for hVM header-only node * Add additional logging around clearing data directory * Fix bug when doing TBC recovery * Temp logging * More temp logging * More temp logging * Do not generate Bitcoin Attributes Deposited transaction unless creating a new block that is within the last hour of wall time. * Temporarily disable output spent check for TxID Bump Bitcoin Attributes Deposited transaction to handle 3 headers at a time * Temporarily always return spent=false for hVM Tx by TxID * Do not create BTC Attributes Deposited transactions with BTC headers that TBC full node does not have complete data for yet During testnet3 difficulty bomb, lag indexer 100 blocks behind tip to prevent block unavailability due to easy reorgs * Update default hVM genesis header information * Update hVM Genesis header * More logging * More logging * Update hVM genesis header * Update heminetwork version for TBC * More logging, fix for genesis activation * Do not let effective hVM lag start prior to the beginning of a difficulty bomb, and require two low-difficulty blocks in a row for difficulty bomb check. * Update hVM Genesis Header * Fix header truncation when full blocks are not available * Update heminetwork dependency to max-hom9 branch commit 102f8a9 * Revert heminetwork dependency to max-hom7 branch with header height fix backported * Update hVM Genesis Header * Update hVM Genesis Header * Edge case on lookback during difficulty bomb right after hVM Phase 0 activation * Add ability to restart TBC if expected full blocks are not available for long enough * Temporary testing code for TBC full node restart logic * Sleep after TBC full node restart * Sleep after TBC full node restart * Sleep after shutting down TBC and again after starting back up * Increase time between TBC restarts when full block not found * Update hVM Phase 0 starting block * Update hVM Phase 0 starting block * Update hVM Phase 0 starting block and update heminetwork dependency to max-hom10 branch * Update hVM Phase 0 starting block * Fix pointer use to adhere to upstream heminetwork changes, and update to use separated TxById and BlockHashByTxId methods as appropriate * Remove fjl/memsize as per ethereum-optimism@dc04347 for go1.23 compatibility. * Disable TBC mempool for header-only mode TBC instance * Update maximum BTC headers in BTC Attr Dep tx temporarily * Update with heminetwork TBC indexer performance improvement Update hVM genesis header * Update with 2nd heminetwork TBC indexer performance improvement * Update hVM genesis header * Update TBC restart logic to not reset missing block counter when blocks are available to cursor * Updated heminetwork to max-hom12 branch for TBC improvements * Updated heminetwork to max-hom13 branch for TBC improvements * Handle updated error for FullBlockAvailable * Update hVM Genesis Header * Attempt missing block refetch over P2P * Attempt missing block refetch over P2P * Update hVM header * Update hVM header and update to max-hom16 * Update heminetwork to pending external header mode PR * Update hVM phase 0 start block * Added new input and output by txid hVM endpoints Updated TBC dependency preparing for TBC code freeze Re-enabled block refetch on error Updated misc. logging * Update hVM Genesis Block * Update error handling for invalid light/full TBC state transitions Better startup TBC fresh sync logic Fixed logging Improved TBC full block availability checks to return missing blocks upstream Predictive fetching of missing BTC full blocks that will be needed in future Reviewed log.Crit instances and converted to recovery modes where applicable Improved codepaths for updating hVM header-only or header+full during various chain reorg logic Minor tweaks to existing hVM precompiles Added new hVM precompiles for BTC inputs/outputs directly as well as witness data * Update docker.yml * Update docker.yml * Update docker.yml * Update Dockerfile Update dockerfile to use 1.23 Golang image * Update Dockerfile fixed hash * Update hVM lag logic for difficulty bomb in testnet3 to avoid long delays on false positives * Adding logging * Adding logging * Max of 3 BTC headers per block from Sequencer Fixed witness contract address * Do not undo hVM progression when processing child to current tip * More logging * Longer time delay when starting TBC * Fix indexer mismatch handling to not call utxo and tx indexer separately * Add wait for TBC ready check * Fix TBC check with updated Running() function * Update hVM indexing tip lag during difficulty bomb to be less aggressive * Return SYNCING instead of INVALID to op-node when a forkchoice update requests progression to a tip which fails a state transition due to TBC full node missing a full block or header. Correctly identify missing TBC full node header in TBCBlocksAvailableToHeader * Change noisy logging to trace * Added ability for op-geth to regenerate lightweight TBC state on startup. * Do not try to prefetch full BTC blocks when regenerating hVM state * Additional logging * Additional logging * Update Sequencer to add up to 4 BTC headers per hVM block. * Bump BTC headers * BTC block back-channel over geth P2P * Logging * Logging * Set blockchain in peer correctly * Fix message IDs * Logging * Logging * Logging * Fixed test request * Fixed test request * Remove RLP encode-decode for BTC block byte bodies * Attempt to add P2P BTC block from geth P2P if TBC query for determining full block fails with error * Rework serialization * Rework serialization * Rework serialization * Rework serialization * Different test block, logging * Serialization * Logging * Testing * Testing * Testing * Logging * Logging * Logging * Fix * Different serialization * Remove p2pserver from blockchain.go Remove misc logging * Cleanup * Insert block headers for full missing blocks fetched from op-geth P2P * Update hVM header-only TBC node config * Don't CRIT on error updating full node * Insert missing block headers into TBC full node for P2P fetching * Store missing headers during progression for P2P geth fetching * Walk backwards in lightweight header view to find multiple missing headers * Fix * ensure parent's pointers are set before deferencing * Clayton/hvmphase0 (#16) * check for fresh tbc directory and brute force index * bugfix * get best again * squashme * squashme * try syncing indexers * squashme * More Logging --------- Co-authored-by: Maxwell Sanchez <max@bloq.com> * re-enabled only build on main --------- Co-authored-by: Max Sanchez <max@bloq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
These are changes that we have been testing for HVM Phase 0.