Release 23.1.1 rc1 2

CHANGELOG.md

@@ -1,29 +1,17 @@
 # Changelog
 
-## 23.1.0
-Besu 23.1.0 is a recommended update for Mainnet users. Thank you all for your patience as we crafted this quarterly release.
-
-This is a rather large release with some breaking changes, so please be sure to read these notes carefully before you upgrade any Besu instances. We are including a move to Java 17 LTS. To build and run Besu, please make sure you have Java 17 on the host machine. Additionally, there are a host of spec compliance changes that change existing formats, so please check the specific RPC updates. Lastly, this release formalizes a deprecation notice for GoQuorum privacy modes and IBFT1.0 in Besu. These will be removed in the 23.4 series, unless otherwise stated.
-
-From the improvements and fixes side, we have a host of execution performance improvements and fixes for defects with bonsai storage. We have also included an error detection and auto-heal capability for nodes that encounter state issues. This should keep nodes online and validating that may have previously required a resync.
-
-One final note. 23.1.0 is not a Shanghai ready release. If you intend to test Besu on the long-lived testnets like Zhejiang, please follow the instructions here. We will have more to share on our official Shanghai releases soon.
+## 23.1.0-RC2
 
 ### Breaking Changes
+- Add a new CLI option to limit the number of requests in a single RPC batch request. Default=1 [#4965](https://github.com/hyperledger/besu/pull/4965)
 - Change JsonRpc http service to return the error -32602 (Invalid params) with a 200 http status code
 - Besu requires minimum Java 17 and up to build and run [#3320](https://github.com/hyperledger/besu/issues/3320)
-- PKCS11 with nss module (PKCS11 based HSM can be used in DevP2P TLS and QBFT PKI) does not work with RSA keys
-in Java 17. SoftHSM is tested manually and working. (Other PKCS11 HSM are not tested). The relevant unit and acceptance
+- PKCS11 with nss module (PKCS11 based HSM can be used in DevP2P TLS and QBFT PKI) does not work with RSA keys 
+in Java 17. SoftHSM is tested manually and working. (Other PKCS11 HSM are not tested). The relevant unit and acceptance 
 tests are updated to use EC private keys instead of RSA keys.
 - Change eth_feeHistory parameter `blockCount` to accept hexadecimal string (was accepting plain integer) [#5047](https://github.com/hyperledger/besu/pull/5047)
-- Default configurations for the deprecated Ropsten, Kiln, Shandong, and Astor networks have been removed from the CLI network list. These networks can currently be accessed but will require a user-provided genesis configuration. [#4869](https://github.com/hyperledger/besu/pull/4869)
-- GoQuorum-compatible privacy is deprecated and will be removed in 23.4
-- IBFT 1.0 is deprecated and will be removed in 23.4
-- Optimize SSTORE Operation execution time (memoize current and original value) [#4836](https://github.com/hyperledger/besu/pull/4836)
 
 ### Additions and Improvements
-- Default rpc batch request to 1000 [#5104](https://github.com/hyperledger/besu/pull/5104)
-- Add a new CLI option to limit the number of requests in a single RPC batch request. [#4965](https://github.com/hyperledger/besu/pull/4965)
 - Support for new DATAHASH opcode as part of EIP-4844 [#4823](https://github.com/hyperledger/besu/issues/4823)
 - Send only hash announcement for blob transaction type [#4940](https://github.com/hyperledger/besu/pull/4940)
 - Add `excess_data_gas` field to block header [#4958](https://github.com/hyperledger/besu/pull/4958)
@@ -32,28 +20,18 @@ tests are updated to use EC private keys instead of RSA keys.
 - Improve get account performance by using the world state updater cache [#4897](https://github.com/hyperledger/besu/pull/4897)
 - Add new KZG precompile and option to override the trusted setup being used [#4822](https://github.com/hyperledger/besu/issues/4822)
 - Add implementation for eth_createAccessList RPC method [#4942](https://github.com/hyperledger/besu/pull/4942)
+- Add implementation for engine_exchangeCapabilities [#4997](https://github.com/hyperledger/besu/pull/4997)
+- Add implementation for engine_getPayloadBodiesByRangeV1 and engine_getPayloadBodiesByHashV1 [#4980](https://github.com/hyperledger/besu/pull/4980)
 - Updated reference tests to v11.3 [#4996](https://github.com/hyperledger/besu/pull/4996)
 - Add DebugGetRawBlock and DebugGetRawHeader RPC methods [#5011](https://github.com/hyperledger/besu/pull/5011)
 - Besu requires minimum Java 17 and up to build and run [#3320](https://github.com/hyperledger/besu/issues/3320)
 - Add worldstate auto-heal mechanism [#5059](https://github.com/hyperledger/besu/pull/5059)
 - Support for EIP-4895 - Withdrawals for Shanghai fork
-- Improve SLOAD and SSTORE performance by caching empty slots [#4874](https://github.com/hyperledger/besu/pull/4874)
-- RPC methods that lookup block by hash will now return an error response if no block found [#4582](https://github.com/hyperledger/besu/pull/4582)
-- Added support for `safe` and `finalized` strings for the RPC methods using defaultBlock parameter [#4902](https://github.com/hyperledger/besu/pull/4902)
-- Added post-execution state logging option to EVM Tool [#4709](https://github.com/hyperledger/besu/pull/4709)
-- Add access list to Transaction Call Object [#4802](https://github.com/hyperledger/besu/issues/4801)
-- Add timestamp fork support, including shanghaiTime and cancunTime forks [#4743](https://github.com/hyperledger/besu/pull/4743)
-- Optimization:  Memoize transaction size and hash at the same time [#4812](https://github.com/hyperledger/besu/pull/4812)
-- Add chain data pruning feature with three experimental CLI options: `--Xchain-pruning-enabled`, `--Xchain-pruning-blocks-retained` and `--Xchain-pruning-frequency` [#4686](https://github.com
-/hyperledger/besu/pull/4686)
-  - Note that chain pruning is hidden and disabled by default. Once you choose to enable chain pruning, a new column family will be added to the db and you cannot roll back to a previous versi
-on of Besu.
+- If a PoS block creation repetition takes less than a configurable duration, then waits before next repetition [#5048](https://github.com/hyperledger/besu/pull/5048)
 
 ### Bug Fixes
 - Mitigation fix for stale bonsai code storage leading to log rolling issues on contract recreates [#4906](https://github.com/hyperledger/besu/pull/4906)
 - Ensure latest cached layered worldstate is subscribed to storage, fix problem with RPC calls using 'latest' [#5076](https://github.com/hyperledger/besu/pull/5076)
-- Fix for segmentation faults on worldstate truncation, snap-sync starts [#4786](https://github.com/hyperledger/besu/pull/4786)
-- Fix for worldstate mismatch on failed forkchoiceUpdate [#4862](https://github.com/hyperledger/besu/pull/4862)
 
 ## 23.1.0-RC1
 

besu/src/main/java/org/hyperledger/besu/cli/BesuCommand.java

@@ -1891,6 +1891,13 @@ private void validateMiningParams() {
       throw new ParameterException(
           this.commandLine, "--Xpos-block-creation-max-time must be positive and ≤ 12000");
     }
+
+    if (unstableMiningOptions.getPosBlockCreationRepetitionMinDuration() <= 0
+        || unstableMiningOptions.getPosBlockCreationRepetitionMinDuration() > 2000) {
+      throw new ParameterException(
+          this.commandLine,
+          "--Xpos-block-creation-repetition-min-duration must be positive and ≤ 2000");
+    }
   }
 
   /**
@@ -2271,6 +2278,8 @@ public BesuControllerBuilder getControllerBuilder() {
                 .powJobTimeToLive(unstableMiningOptions.getPowJobTimeToLive())
                 .maxOmmerDepth(unstableMiningOptions.getMaxOmmersDepth())
                 .posBlockCreationMaxTime(unstableMiningOptions.getPosBlockCreationMaxTime())
+                .posBlockCreationRepetitionMinDuration(
+                    unstableMiningOptions.getPosBlockCreationRepetitionMinDuration())
                 .build())
         .transactionPoolConfiguration(buildTransactionPoolConfiguration())
         .nodeKey(new NodeKey(securityModule()))
@@ -3510,7 +3519,7 @@ private void validatePostMergeCheckpointBlockRequirements() {
     if (synchronizerConfiguration.isCheckpointPostMergeEnabled()) {
       if (!checkpointConfigOptions.isValid()) {
         throw new InvalidConfigurationException(
-            "Near head checkpoint sync requires a checkpoint block configured in the genesis file");
+            "PoS checkpoint sync requires a checkpoint block configured in the genesis file");
       }
       terminalTotalDifficulty.ifPresentOrElse(
           ttd -> {
@@ -3519,18 +3528,18 @@ private void validatePostMergeCheckpointBlockRequirements() {
                     .equals(UInt256.ZERO)
                 && ttd.equals(UInt256.ZERO)) {
               throw new InvalidConfigurationException(
-                  "Post Merge checkpoint sync can't be used with TTD = 0 and checkpoint totalDifficulty = 0");
+                  "PoS checkpoint sync can't be used with TTD = 0 and checkpoint totalDifficulty = 0");
             }
             if (UInt256.fromHexString(
                     genesisOptions.getCheckpointOptions().getTotalDifficulty().get())
-                .lessOrEqualThan(ttd)) {
+                .lessThan(ttd)) {
               throw new InvalidConfigurationException(
-                  "Near head checkpoint sync requires a block with total difficulty greater than the TTD");
+                  "PoS checkpoint sync requires a block with total difficulty greater or equal than the TTD");
             }
           },
           () -> {
             throw new InvalidConfigurationException(
-                "Near head checkpoint sync requires TTD in the genesis file");
+                "PoS checkpoint sync requires TTD in the genesis file");
           });
     }
   }

besu/src/main/java/org/hyperledger/besu/cli/options/unstable/MiningOptions.java

@@ -16,6 +16,7 @@
 
 import static org.hyperledger.besu.ethereum.core.MiningParameters.DEFAULT_MAX_OMMERS_DEPTH;
 import static org.hyperledger.besu.ethereum.core.MiningParameters.DEFAULT_POS_BLOCK_CREATION_MAX_TIME;
+import static org.hyperledger.besu.ethereum.core.MiningParameters.DEFAULT_POS_BLOCK_CREATION_REPETITION_MIN_DURATION;
 import static org.hyperledger.besu.ethereum.core.MiningParameters.DEFAULT_POW_JOB_TTL;
 import static org.hyperledger.besu.ethereum.core.MiningParameters.DEFAULT_REMOTE_SEALERS_LIMIT;
 import static org.hyperledger.besu.ethereum.core.MiningParameters.DEFAULT_REMOTE_SEALERS_TTL;
@@ -67,6 +68,15 @@ public class MiningOptions {
           "Specifies the maximum time, in milliseconds, a PoS block creation jobs is allowed to run. Must be positive and ≤ 12000 (default: ${DEFAULT-VALUE} milliseconds)")
   private final Long posBlockCreationMaxTime = DEFAULT_POS_BLOCK_CREATION_MAX_TIME;
 
+  @CommandLine.Option(
+      hidden = true,
+      names = {"--Xpos-block-creation-repetition-min-duration"},
+      description =
+          "If a PoS block creation repetition takes less than this duration, in milliseconds,"
+              + " then it waits before next repetition. Must be positive and ≤ 2000 (default: ${DEFAULT-VALUE} milliseconds)")
+  private final Long posBlockCreationRepetitionMinDuration =
+      DEFAULT_POS_BLOCK_CREATION_REPETITION_MIN_DURATION;
+
   /**
    * Create mining options.
    *
@@ -129,4 +139,13 @@ public int getMaxOmmersDepth() {
   public Long getPosBlockCreationMaxTime() {
     return posBlockCreationMaxTime;
   }
+
+  /**
+   * Gets pos block creation repetition min duration.
+   *
+   * @return the pos block creation repetition min duration.
+   */
+  public Long getPosBlockCreationRepetitionMinDuration() {
+    return posBlockCreationRepetitionMinDuration;
+  }
 }

besu/src/main/java/org/hyperledger/besu/controller/BesuControllerBuilder.java

@@ -75,6 +75,7 @@
 import org.hyperledger.besu.ethereum.eth.transactions.TransactionPoolConfiguration;
 import org.hyperledger.besu.ethereum.eth.transactions.TransactionPoolFactory;
 import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
+import org.hyperledger.besu.ethereum.mainnet.ProtocolSpec;
 import org.hyperledger.besu.ethereum.p2p.config.SubProtocolConfiguration;
 import org.hyperledger.besu.ethereum.storage.StorageProvider;
 import org.hyperledger.besu.ethereum.storage.keyvalue.KeyValueSegmentIdentifier;
@@ -548,9 +549,12 @@ public BesuController build() {
       }
     }
     final int maxMessageSize = ethereumWireProtocolConfiguration.getMaxMessageSize();
+    final Supplier<ProtocolSpec> currentProtocolSpecSupplier =
+        () -> protocolSchedule.getByBlockHeader(blockchain.getChainHeadHeader());
     final EthPeers ethPeers =
         new EthPeers(
             getSupportedProtocol(),
+            currentProtocolSpecSupplier,
             clock,
             metricsSystem,
             maxPeers,

besu/src/main/java/org/hyperledger/besu/controller/MergeBesuControllerBuilder.java

@@ -14,6 +14,7 @@
  */
 package org.hyperledger.besu.controller;
 
+import org.hyperledger.besu.config.GenesisConfigOptions;
 import org.hyperledger.besu.consensus.merge.MergeContext;
 import org.hyperledger.besu.consensus.merge.MergeProtocolSchedule;
 import org.hyperledger.besu.consensus.merge.PostMergeContext;
@@ -186,19 +187,24 @@ protected MergeContext createConsensusContext(
       final WorldStateArchive worldStateArchive,
       final ProtocolSchedule protocolSchedule) {
 
-    OptionalLong terminalBlockNumber = configOptionsSupplier.get().getTerminalBlockNumber();
-    Optional<Hash> terminalBlockHash = configOptionsSupplier.get().getTerminalBlockHash();
+    final GenesisConfigOptions genesisConfigOptions = configOptionsSupplier.get();
+    final OptionalLong terminalBlockNumber = genesisConfigOptions.getTerminalBlockNumber();
+    final Optional<Hash> terminalBlockHash = genesisConfigOptions.getTerminalBlockHash();
+    final boolean isPostMergeAtGenesis =
+        genesisConfigOptions.getTerminalTotalDifficulty().isPresent()
+            && genesisConfigOptions.getTerminalTotalDifficulty().get().isZero()
+            && blockchain.getGenesisBlockHeader().getDifficulty().isZero();
 
     final MergeContext mergeContext =
         PostMergeContext.get()
             .setSyncState(syncState.get())
             .setTerminalTotalDifficulty(
-                configOptionsSupplier
-                    .get()
+                genesisConfigOptions
                     .getTerminalTotalDifficulty()
                     .map(Difficulty::of)
                     .orElse(Difficulty.ZERO))
-            .setCheckpointPostMergeSync(syncConfig.isCheckpointPostMergeEnabled());
+            .setCheckpointPostMergeSync(syncConfig.isCheckpointPostMergeEnabled())
+            .setPostMergeAtGenesis(isPostMergeAtGenesis);
 
     blockchain
         .getFinalized()

besu/src/test/java/org/hyperledger/besu/ForkIdsNetworkConfigTest.java

@@ -24,14 +24,13 @@
 import org.hyperledger.besu.cli.config.NetworkName;
 import org.hyperledger.besu.config.GenesisConfigFile;
 import org.hyperledger.besu.config.GenesisConfigOptions;
+import org.hyperledger.besu.consensus.merge.TransitionProtocolSchedule;
 import org.hyperledger.besu.ethereum.chain.Blockchain;
 import org.hyperledger.besu.ethereum.chain.GenesisState;
 import org.hyperledger.besu.ethereum.core.BlockHeader;
 import org.hyperledger.besu.ethereum.forkid.ForkId;
 import org.hyperledger.besu.ethereum.forkid.ForkIdManager;
-import org.hyperledger.besu.ethereum.mainnet.MainnetProtocolSchedule;
 import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
-import org.hyperledger.besu.evm.internal.EvmConfiguration;
 
 import java.util.Collection;
 import java.util.List;
@@ -73,8 +72,9 @@ public static Collection<Object[]> parameters() {
             NetworkName.SEPOLIA,
             List.of(
                 new ForkId(Bytes.ofUnsignedInt(0xfe3366e7L), 1735371L),
-                new ForkId(Bytes.ofUnsignedInt(0xb96cbd13L), 0L),
-                new ForkId(Bytes.ofUnsignedInt(0xb96cbd13L), 0L))
+                new ForkId(Bytes.ofUnsignedInt(0xb96cbd13L), 1677557088L),
+                new ForkId(Bytes.ofUnsignedInt(0xf7f9bc08L), 0L),
+                new ForkId(Bytes.ofUnsignedInt(0xf7f9bc08L), 0L))
           },
           new Object[] {
             NetworkName.RINKEBY,
@@ -168,8 +168,7 @@ public void testForkId() {
       final GenesisConfigFile genesisConfigFile =
           GenesisConfigFile.fromConfig(EthNetworkConfig.jsonConfig(chainName));
       final GenesisConfigOptions configOptions = genesisConfigFile.getConfigOptions();
-      final ProtocolSchedule schedule =
-          MainnetProtocolSchedule.fromConfig(configOptions, EvmConfiguration.DEFAULT);
+      final ProtocolSchedule schedule = TransitionProtocolSchedule.fromConfig(configOptions);
       final GenesisState genesisState = GenesisState.fromConfig(genesisConfigFile, schedule);
       final Blockchain mockBlockchain = mock(Blockchain.class);
       final BlockHeader mockBlockHeader = mock(BlockHeader.class);
@@ -179,6 +178,7 @@ public void testForkId() {
       final AtomicLong blockNumber = new AtomicLong();
       when(mockBlockchain.getChainHeadHeader()).thenReturn(mockBlockHeader);
       when(mockBlockHeader.getNumber()).thenAnswer(o -> blockNumber.get());
+      when(mockBlockHeader.getTimestamp()).thenAnswer(o -> blockNumber.get());
 
       final ForkIdManager forkIdManager =
           new ForkIdManager(
@@ -187,7 +187,7 @@ public void testForkId() {
               genesisConfigFile.getForkTimestamps(),
               false);
 
-      final var actualForkIds =
+      final List<ForkId> actualForkIds =
           Streams.concat(schedule.streamMilestoneBlocks(), Stream.of(Long.MAX_VALUE))
               .map(
                   block -> {

besu/src/test/java/org/hyperledger/besu/cli/BesuCommandTest.java

@@ -5518,7 +5518,7 @@ public void checkpointPostMergeShouldFailWhenGenesisHasNoTTD() throws IOExceptio
 
     assertThat(commandOutput.toString(UTF_8)).isEmpty();
     assertThat(commandErrorOutput.toString(UTF_8))
-        .contains("Near head checkpoint sync requires TTD in the genesis file");
+        .contains("PoS checkpoint sync requires TTD in the genesis file");
   }
 
   @Test
@@ -5529,7 +5529,7 @@ public void checkpointPostMergeShouldFailWhenGenesisUsesCheckpointFromPreMerge()
     assertThat(commandOutput.toString(UTF_8)).isEmpty();
     assertThat(commandErrorOutput.toString(UTF_8))
         .contains(
-            "Near head checkpoint sync requires a block with total difficulty greater than the TTD");
+            "PoS checkpoint sync requires a block with total difficulty greater or equal than the TTD");
   }
 
   @Test
@@ -5559,6 +5559,25 @@ public void checkpointPostMergeWithPostMergeBlockSucceeds() throws IOException {
     assertThat(commandErrorOutput.toString(UTF_8)).isEmpty();
   }
 
+  @Test
+  public void checkpointPostMergeWithPostMergeBlockTDEqualsTTDSucceeds() throws IOException {
+    final String configText =
+        Resources.toString(
+            Resources.getResource("valid_pos_checkpoint_pos_TD_equals_TTD.json"),
+            StandardCharsets.UTF_8);
+    final Path genesisFile = createFakeGenesisFile(new JsonObject(configText));
+
+    parseCommand(
+        "--genesis-file",
+        genesisFile.toString(),
+        "--sync-mode",
+        "X_CHECKPOINT",
+        "--Xcheckpoint-post-merge-enabled");
+
+    assertThat(commandOutput.toString(UTF_8)).isEmpty();
+    assertThat(commandErrorOutput.toString(UTF_8)).isEmpty();
+  }
+
   @Test
   public void checkpointMergeAtGenesisWithGenesisBlockDifficultyZeroFails() throws IOException {
     final String configText =
@@ -5577,6 +5596,6 @@ public void checkpointMergeAtGenesisWithGenesisBlockDifficultyZeroFails() throws
     assertThat(commandOutput.toString(UTF_8)).isEmpty();
     assertThat(commandErrorOutput.toString(UTF_8))
         .contains(
-            "Post Merge checkpoint sync can't be used with TTD = 0 and checkpoint totalDifficulty = 0");
+            "PoS checkpoint sync can't be used with TTD = 0 and checkpoint totalDifficulty = 0");
   }
 }