Possible (to be confirmed) Consensus (bank hash) mismatch with v1.9.8

[steviez]

Problem

One of our nodes was running v1.9.8 and encountered the following:

solana_core::cluster_slot_state_verifier] Cluster duplicate confirmed slot 119324256
with hash AegBYCpY5ET8nmWKLWEGsWdFZWJMRHMZi6TrDeTQ2bzM,
but our version has hash B9zJTYie64wnexRt9se7d7c7X5JZDa9fLwRCCfaXYahm

Work in progress to confirm / reproduce / track down the issue

[jstarry]

EDIT: Think I figured out the reason this time...

On epoch boundaries we check if any new features have been activated and, if so, apply changes to builtin programs. On testnet, there weren't any new feature activation at the latest epoch boundary. But in #23233, builtin addition/removal transitions were combined such that if the removal feature id was not activated, the builtin would be re-added. This behavior differs from v1.9.6 because that version doesn't try to re-add feature-gated builtins at all. When the built-in for ed25519 is re-added, the account becomes native loader owned and the "ed25519_program" string is stored in its data buffer. This results in a hash mismatch because on testnet, the ed25519 program id is a system owned account with an empty data buffer.

[github-actions]

This issue has been automatically locked since there has not been any activity in past 7 days after it was closed. Please open a new issue for related bugs.