Fix - `FailedVerification` and `Closed` tombstones by Lichtso · Pull Request #419 · anza-xyz/agave

Lichtso · 2024-03-25T17:40:13Z

Problems

TransactionBatchProcessor::load_program_with_pubkey() returns LoadedProgramType::FailedVerification even for accounts which never got to the verifier (because they are empty or buffers for example) which should be marked as LoadedProgramType::Closed instead. LoadedProgramType::FailedVerification should only be used if there is a chance a program gets valid again in the recompilation phase when the feature set and environment changes.
In the tests we currently check that loading non existent accounts results in LoadedProgramType::Closed entries. This however can not happen in integration because we only attempt to load accounts which have a loader as a owner, thus these account must exist. The situation is similar to when we tested LoadedProgramType::FailedVerification existing in the global cache (which can't happen either) and it led to confusion and wasted efforts.

Summary of Changes

Allow the transition from LoadedProgramType::Closed to loaded programs in the same slot.
Replace ProgramAccountLoadResult::AccountNotFound by None
Have TransactionBatchProcessor::load_program_accounts() only return ProgramAccountLoadResult::InvalidAccountData in other error cases
Have TransactionBatchProcessor::load_program_with_pubkey() return LoadedProgramType::Closed when TransactionBatchProcessor::load_program_accounts() returned ProgramAccountLoadResult::InvalidAccountData
Panic in replenish_program_cache() when attempting to load a nonexistent account, and add a test for that
Allowed ProgramCache::assign_program() transition from LoadedProgramType::Closed to loaded programs in the same slot (reverted in Fix - ProgramCache::assign_program() Closed => Loaded in same slot #673).

test_bpf_loader_upgradeable_deploy_with_max_len():

Replace mock_process_instruction() by a proper message send
Tests the invocation of an empty account before a program is deployed there
Tests the invocation of a buffer account before it is used to deploy a program

mergify · 2024-03-25T17:40:51Z

Backports to the beta branch are to be avoided unless absolutely necessary for fixing bugs, security issues, and perf regressions. Changes intended for backport should be structured such that a minimum effective diff can be committed separately from any refactoring, plumbing, cleanup, etc that are not strictly necessary to achieve the goal. Any of the latter should go only into master and ride the normal stabilization schedule. Exceptions include CI/metrics changes, CLI improvements and documentation updates on a case by case basis.

t-nelson · 2024-04-02T20:53:36Z

is there something hold review here? landing this is blocking 1.18 to mb

alessandrod · 2024-04-03T12:22:46Z

is there something hold review here? landing this is blocking 1.18 to mb

I did a live (on slack) review of this with Alex last week. I'll do another round tomorrow.

alessandrod

Thanks! This is so much easier to reason about with the Option return.

I just left a few nits otherwise looks good. The major thing is we don't test the new transitions from Closed => XXX - is this what you said you want to do in a follow PR that removes TestLoaded? And if so, we're backporting that too to 1.18 right? I'm not too comfortable adding new transitions without tests.

alessandrod · 2024-04-04T10:23:56Z

        bank.new_program_cache_for_tx_batch_for_slot(bank.slot() + DELAY_VISIBILITY_SLOT_OFFSET);
    for key in cached_account_keys {
-        loaded_programs.replenish(key, bank.load_program(&key, false, bank.epoch()));
+        loaded_programs.replenish(key, bank.load_program(&key, false, bank.epoch()).unwrap());


nit: this should be expect() instead of unwrap

alessandrod · 2024-04-04T10:28:47Z

+    /// Tombstone for programs which currently do not pass the verifier but could if the feature set changed.
    FailedVerification(ProgramRuntimeEnvironment),
-    /// Tombstone for programs which were explicitly undeployed / closed.
+    /// Tombstone for accounts which are not programs but might still be owned by a loader.


I think this comment now is a bit confusing. The variant is still conceptually mostly about closed
programs, but also - temporarily - used for things like buffer programs. Something like:

/// Tombstone for programs that were explicitly closed. It's also used for accounts belonging to program loaders, that don't actually contain program code (for example buffer accounts for BpfLoaderUpgradeable programs).

temporarily? You know there is nothing more permanent than a "temporary" solution.

alessandrod · 2024-04-04T10:32:38Z

                let existing = slot_versions.get_mut(index).unwrap();
                match (&existing.program, &entry.program) {
                    (LoadedProgramType::Builtin(_), LoadedProgramType::Builtin(_))
+                    | (LoadedProgramType::Closed, LoadedProgramType::LegacyV0(_))


I find all other transitions pretty straightforward, but this Closed => xxx
is not obvious. It would be good to add a comment that explains when it happens
(on deploy) and why

alessandrod · 2024-04-04T10:36:12Z

    }

+    #[test_case(
+        LoadedProgramType::Closed,


Can you add a FIXME here that says we need to test the missing Closed => Legacy cases?

CI does not like FIXME or TODO

worst CI ever

alessandrod · 2024-04-04T11:00:51Z

@@ -386,82 +385,77 @@ impl<FG: ForkGraph> TransactionBatchProcessor<FG> {
        pubkey: &Pubkey,


The rustdocs for this method are 100% off? Could you update to say something like

Loads the program with the given pubkey. If the account doesn't exist it returns `None`. If the account does exist, it must be a program account (belong to one of the program loaders). Returns Some(InvalidAccountData) if the program account is Closed, contains invalid data or any of the ancillary program accounts contain invalid data.

alessandrod · 2024-04-04T11:01:40Z

+    ) -> Option<ProgramAccountLoadResult> {
+        let program_account = callbacks.get_account_shared_data(pubkey)?;

        debug_assert!(solana_bpf_loader_program::check_loader_id(


From a standalone SVM API perspective, I'm undecided on whether it makes sense to
keep this or just return None

alessandrod · 2024-04-04T11:07:27Z

-
-        let loaded_program = LoadedProgram::new_tombstone(0, LoadedProgramType::Closed);
-        assert_eq!(result, Arc::new(loaded_program));
+        assert!(result.is_none());


The diff in this test 😍

alessandrod

Approved, tests for Closed => XX to come in followup PR

…osed

…ram_with_pubkey().

codecov-commenter · 2024-04-05T10:35:14Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.8%. Comparing base (562254e) to head (fd971ad).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #419   +/-   ##
=======================================
  Coverage    81.8%    81.8%           
=======================================
  Files         851      851           
  Lines      230155   230165   +10     
=======================================
+ Hits       188443   188492   +49     
+ Misses      41712    41673   -39

* Only the verifier can cause FailedVerification, everything else is Closed * Removes the environments parameter from load_program_accounts(). * cargo fmt * Simplify invocation of deployed program * Attempt to invoke a program before it is deployed * Attempt to invoke a buffer before it is used in a deployment * Escalates Option return value of load_program_accounts() to load_program_with_pubkey(). * Review feedback (cherry picked from commit 55c05c5) # Conflicts: # ledger-tool/src/program.rs # program-runtime/src/loaded_programs.rs # runtime/src/bank.rs # runtime/src/bank/tests.rs # svm/src/transaction_processor.rs

* Only the verifier can cause FailedVerification, everything else is Closed * Removes the environments parameter from load_program_accounts(). * cargo fmt * Simplify invocation of deployed program * Attempt to invoke a program before it is deployed * Attempt to invoke a buffer before it is used in a deployment * Escalates Option return value of load_program_accounts() to load_program_with_pubkey(). * Review feedback (cherry picked from commit 55c05c5)

…#419) (#605) Fix - `FailedVerification` and `Closed` tombstones (#419) * Only the verifier can cause FailedVerification, everything else is Closed * Removes the environments parameter from load_program_accounts(). * cargo fmt * Simplify invocation of deployed program * Attempt to invoke a program before it is deployed * Attempt to invoke a buffer before it is used in a deployment * Escalates Option return value of load_program_accounts() to load_program_with_pubkey(). * Review feedback (cherry picked from commit 55c05c5) Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>

…anza-xyz#419) (anza-xyz#605) Fix - `FailedVerification` and `Closed` tombstones (anza-xyz#419) * Only the verifier can cause FailedVerification, everything else is Closed * Removes the environments parameter from load_program_accounts(). * cargo fmt * Simplify invocation of deployed program * Attempt to invoke a program before it is deployed * Attempt to invoke a buffer before it is used in a deployment * Escalates Option return value of load_program_accounts() to load_program_with_pubkey(). * Review feedback (cherry picked from commit 55c05c5) Co-authored-by: Alexander Meißner <AlexanderMeissner@gmx.net>

only reroute if relayer connected (anza-xyz#123) feat: add client tls config (anza-xyz#121) remove extra val (anza-xyz#129) fix clippy (anza-xyz#130) copy all binaries to docker-output (anza-xyz#131) Ledger tool halts at slot passed to create-snapshot (anza-xyz#118) update program submodule (anza-xyz#133) quick fix for tips and clearing old bundles (anza-xyz#135) update submodule to new program (anza-xyz#136) Improve stake-meta-generator usability (anza-xyz#134) pinning submodule head (anza-xyz#140) Use BundleAccountLocker when handling tip txs (anza-xyz#147) Add metrics for relayer + block engine proxy (anza-xyz#149) Build claim-mev in docker (anza-xyz#141) Rework bundle receiving and add metrics (anza-xyz#152) (anza-xyz#154) update submodule + dev files (anza-xyz#158) Deterministically find tip amounts, add meta to stake info, and cleanup pubkey/strings in MEV tips (anza-xyz#159) update jito-programs submodule (anza-xyz#160) Separate MEV tip related workflow (anza-xyz#161) Add block builder fee protos (anza-xyz#162) fix jito programs (anza-xyz#163) update submodule so autosnapshot exits out of ledger tool early (anza-xyz#164) Pipe through block builder fee (anza-xyz#167) pull in new snapshot code (anza-xyz#171) block builder bug (anza-xyz#172) Pull in new slack autosnapshot submodule (anza-xyz#174) sort stake meta json and use int math (anza-xyz#176) add accountsdb conn submod (anza-xyz#169) Update tip distribution parameters (anza-xyz#177) new submodules (anza-xyz#180) Add buildkite link for jito CI (anza-xyz#183) Fixed broken links to repositories (anza-xyz#184) Changed from ssh to https transfer for clone Seg/update submods (anza-xyz#187) fix tests (anza-xyz#190) rm geyser submod (anza-xyz#192) rm dangling geyser references (anza-xyz#193) fix syntax err (anza-xyz#195) use deterministic req ids in batch calls (anza-xyz#199) update jito-programs revert cargo update Cargo lock update with path fix fix cargo update autosnapshot with block lookback (anza-xyz#201) [JIT-460] When claiming mev tips, skip accounts that won't have min rent exempt amount after claiming (anza-xyz#203) Add logging for sol balance desired (anza-xyz#205) * add logging * add logging * update msg * tweak vars update submodule (anza-xyz#204) use efficient data structures when calling batch_simulate_bundles (anza-xyz#206) [JIT-504] Add low balance check in uploading merkle roots (anza-xyz#209) add config to simulate on top of working bank (anza-xyz#211) rm frozen bank check simulate_bundle rpc bugfixes (anza-xyz#214) rm frozen bank check in simulate_bundle rpc method [JIT-519] Store ClaimStatus address in merkle-root-json (anza-xyz#210) * add files * switch to include bump update submodule (anza-xyz#217) add amount filter (anza-xyz#218) update autosnapshot (anza-xyz#222) Print TX error in Bundles (anza-xyz#223) add new args to support single relayer and block-engine endpoints (anza-xyz#224) point to new jito-programs submod and invoke updated init tda instruction (anza-xyz#228) fix clippy errors (anza-xyz#230) fix validator start scripts (anza-xyz#232) Point README to gitbook (anza-xyz#237) use packaged cargo bin to build (anza-xyz#239) Add validator identity pubkey to StakeMeta (anza-xyz#226) The vote account associated with a validator is not a permanent link, so log the validator identity as well. bugfix: conditionally compile with debug flags (anza-xyz#240) Seg/tip distributor master (anza-xyz#242) * validate tree nodes * fix unit tests * pr feedback * bump jito-programs submod Simplify bootstrapping (anza-xyz#241) * startup without precompile * update spacing * use release mode * spacing fix validation rm validation skip Account for block builder fee when generating excess tip balance (anza-xyz#247) Improve docker caching delay constructing claim mev txs (anza-xyz#253) fix stake meta tests from bb fee (anza-xyz#254) fix tests Buffer bundles that exceed cost model (anza-xyz#225) * buffer bundles that exceed cost model clear qos failed bundles buffer if not leader soon (anza-xyz#260) update Cargo.lock to correct solana versions in jito-programs submodule (anza-xyz#265) fix simulate_bundle client and better error handling (anza-xyz#267) update submod (anza-xyz#272) Preallocate Bundle Cost (anza-xyz#238) fix Dockerfile (anza-xyz#278) Fix Tests (anza-xyz#279) Fix Tests (anza-xyz#281) * fix tests update jito-programs submod (anza-xyz#282) add reclaim rent workflow (anza-xyz#283) update jito-programs submod fix clippy errs rm wrong assertion and swap out file write fn call (anza-xyz#292) Remove security.md (anza-xyz#293) demote frequent relayer_stage-stream_error to warn (anza-xyz#275) account for case where TDA exists but not allocated (anza-xyz#295) implement better retries for tip-distributor workflows (anza-xyz#297) limit number of concurrent rpc calls (anza-xyz#298) Discard Empty Packet Batches (anza-xyz#299) Identity Hotswap (anza-xyz#290) small fixes (anza-xyz#305) Set backend config from admin rpc (anza-xyz#304) Admin Shred Receiver Change (anza-xyz#306) Seg/rm bundle UUID (anza-xyz#309) Fix github workflow to recursively clone (anza-xyz#327) Add recursive checkout for downstream-project-spl.yaml (anza-xyz#341) Use cluster info functions for tpu (anza-xyz#345) Use git rev-parse for git sha Remove blacklisted tx from message_hash_to_transaction (anza-xyz#374) Updates bootstrap and start scripts needed for local dev. (anza-xyz#384) Remove Deprecated Cli Args (anza-xyz#387) Master Rebase improve simulate_bundle errors and response (anza-xyz#404) derive Clone on accountoverrides (anza-xyz#416) Add upsert to AccountOverrides (anza-xyz#419) update jito-programs (anza-xyz#430) [JIT-1661] Faster Autosnapshot (anza-xyz#436) Reverts simulate_transaction result calls to upstream (anza-xyz#446) Don't unlock accounts in TransactionBatches used during simulation (anza-xyz#449) first pass at wiring up jito-plugin (anza-xyz#428) [JIT-1713] Fix bundle's blockspace preallocation (anza-xyz#489) [JIT-1708] Fix TOC TOU condition for relayer and block engine config (anza-xyz#491) [JIT-1710] - Optimize Bundle Consumer Checks (anza-xyz#490) Add Blockhash Metrics to Bundle Committer (anza-xyz#500) add priority fee ix to mev-claim (anza-xyz#520) Update Autosnapshot (anza-xyz#548) Run MEV claims + reclaiming rent-exempt amounts in parallel. (anza-xyz#582) Update CI (anza-xyz#584) - Add recursive submodule checkouts. - Re-add solana-secondary step Add more release fixes (anza-xyz#585) Fix more release urls (anza-xyz#588) [JIT-1812] Fix blocking mutexs (anza-xyz#495) [JIT-1711] Compare the unprocessed transaction storage BundleStorage against a constant instead of VecDeque::capacity() (anza-xyz#587) Automatically rebase Jito-Solana on a periodic basis. Send message on slack during any failures or success. Fix periodic rebase anza-xyz#594 Fixes the following bugs in the periodic rebase: Sends multiple messages on failure instead of one Cancels entire job if one branch fails Ignore buildkite curl errors for rebasing and try to keep curling until job times out (anza-xyz#597) Sleep longer waiting for buildkite to start (anza-xyz#598) correctly initialize account overrides (anza-xyz#595) Fix: Ensure set contact info to UDP port instead of QUIC (anza-xyz#603) Add fast replay branch to daily rebase (anza-xyz#607) take a snapshot of all bundle accounts before sim (anza-xyz#13) (anza-xyz#615) update jito-programs submodule Add 2.0 to daily rebase (anza-xyz#626) Export agave binaries during docker build (anza-xyz#627) Buffer bundles that exceed processing time and make the allowed processing time longer (anza-xyz#611) Publish releases to S3 and GCS (anza-xyz#633) Rebase from different repos (anza-xyz#637) Point SECURITY.md to immunefi (anza-xyz#671) Loosen requirements on tip accounts touchable in BankingStage (anza-xyz#683) Separate out broadcast + retransmit shredstream (anza-xyz#703) Add packet flag for staked node (anza-xyz#705) Add auto-rebase to v2.1 (anza-xyz#739) Fix release github (anza-xyz#745) Move block_cost_limit tracking to BankingStage in preparation for SIMD-0207 (anza-xyz#753) Add precompile checks in BundleStage (anza-xyz#787) Add auto-rebase to v2.2 (anza-xyz#818) Add better error handling around missing transaction signatures for bundle id generation (anza-xyz#860) Remove unwrap from authentication (anza-xyz#861) Revert Jito-Solana WorkingBankEntry changes (anza-xyz#873) BP anza-xyz#885: Add libclang to Dockerfile (anza-xyz#886) Remove the tip distributor code (anza-xyz#888) Rebase: Update anchor to not use deprecated crates

only reroute if relayer connected (anza-xyz#123) feat: add client tls config (anza-xyz#121) remove extra val (anza-xyz#129) fix clippy (anza-xyz#130) copy all binaries to docker-output (anza-xyz#131) Ledger tool halts at slot passed to create-snapshot (anza-xyz#118) update program submodule (anza-xyz#133) quick fix for tips and clearing old bundles (anza-xyz#135) update submodule to new program (anza-xyz#136) Improve stake-meta-generator usability (anza-xyz#134) pinning submodule head (anza-xyz#140) Use BundleAccountLocker when handling tip txs (anza-xyz#147) Add metrics for relayer + block engine proxy (anza-xyz#149) Build claim-mev in docker (anza-xyz#141) Rework bundle receiving and add metrics (anza-xyz#152) (anza-xyz#154) update submodule + dev files (anza-xyz#158) Deterministically find tip amounts, add meta to stake info, and cleanup pubkey/strings in MEV tips (anza-xyz#159) update jito-programs submodule (anza-xyz#160) Separate MEV tip related workflow (anza-xyz#161) Add block builder fee protos (anza-xyz#162) fix jito programs (anza-xyz#163) update submodule so autosnapshot exits out of ledger tool early (anza-xyz#164) Pipe through block builder fee (anza-xyz#167) pull in new snapshot code (anza-xyz#171) block builder bug (anza-xyz#172) Pull in new slack autosnapshot submodule (anza-xyz#174) sort stake meta json and use int math (anza-xyz#176) add accountsdb conn submod (anza-xyz#169) Update tip distribution parameters (anza-xyz#177) new submodules (anza-xyz#180) Add buildkite link for jito CI (anza-xyz#183) Fixed broken links to repositories (anza-xyz#184) Changed from ssh to https transfer for clone Seg/update submods (anza-xyz#187) fix tests (anza-xyz#190) rm geyser submod (anza-xyz#192) rm dangling geyser references (anza-xyz#193) fix syntax err (anza-xyz#195) use deterministic req ids in batch calls (anza-xyz#199) update jito-programs revert cargo update Cargo lock update with path fix fix cargo update autosnapshot with block lookback (anza-xyz#201) [JIT-460] When claiming mev tips, skip accounts that won't have min rent exempt amount after claiming (anza-xyz#203) Add logging for sol balance desired (anza-xyz#205) * add logging * add logging * update msg * tweak vars update submodule (anza-xyz#204) use efficient data structures when calling batch_simulate_bundles (anza-xyz#206) [JIT-504] Add low balance check in uploading merkle roots (anza-xyz#209) add config to simulate on top of working bank (anza-xyz#211) rm frozen bank check simulate_bundle rpc bugfixes (anza-xyz#214) rm frozen bank check in simulate_bundle rpc method [JIT-519] Store ClaimStatus address in merkle-root-json (anza-xyz#210) * add files * switch to include bump update submodule (anza-xyz#217) add amount filter (anza-xyz#218) update autosnapshot (anza-xyz#222) Print TX error in Bundles (anza-xyz#223) add new args to support single relayer and block-engine endpoints (anza-xyz#224) point to new jito-programs submod and invoke updated init tda instruction (anza-xyz#228) fix clippy errors (anza-xyz#230) fix validator start scripts (anza-xyz#232) Point README to gitbook (anza-xyz#237) use packaged cargo bin to build (anza-xyz#239) Add validator identity pubkey to StakeMeta (anza-xyz#226) The vote account associated with a validator is not a permanent link, so log the validator identity as well. bugfix: conditionally compile with debug flags (anza-xyz#240) Seg/tip distributor master (anza-xyz#242) * validate tree nodes * fix unit tests * pr feedback * bump jito-programs submod Simplify bootstrapping (anza-xyz#241) * startup without precompile * update spacing * use release mode * spacing fix validation rm validation skip Account for block builder fee when generating excess tip balance (anza-xyz#247) Improve docker caching delay constructing claim mev txs (anza-xyz#253) fix stake meta tests from bb fee (anza-xyz#254) fix tests Buffer bundles that exceed cost model (anza-xyz#225) * buffer bundles that exceed cost model clear qos failed bundles buffer if not leader soon (anza-xyz#260) update Cargo.lock to correct solana versions in jito-programs submodule (anza-xyz#265) fix simulate_bundle client and better error handling (anza-xyz#267) update submod (anza-xyz#272) Preallocate Bundle Cost (anza-xyz#238) fix Dockerfile (anza-xyz#278) Fix Tests (anza-xyz#279) Fix Tests (anza-xyz#281) * fix tests update jito-programs submod (anza-xyz#282) add reclaim rent workflow (anza-xyz#283) update jito-programs submod fix clippy errs rm wrong assertion and swap out file write fn call (anza-xyz#292) Remove security.md (anza-xyz#293) demote frequent relayer_stage-stream_error to warn (anza-xyz#275) account for case where TDA exists but not allocated (anza-xyz#295) implement better retries for tip-distributor workflows (anza-xyz#297) limit number of concurrent rpc calls (anza-xyz#298) Discard Empty Packet Batches (anza-xyz#299) Identity Hotswap (anza-xyz#290) small fixes (anza-xyz#305) Set backend config from admin rpc (anza-xyz#304) Admin Shred Receiver Change (anza-xyz#306) Seg/rm bundle UUID (anza-xyz#309) Fix github workflow to recursively clone (anza-xyz#327) Add recursive checkout for downstream-project-spl.yaml (anza-xyz#341) Use cluster info functions for tpu (anza-xyz#345) Use git rev-parse for git sha Remove blacklisted tx from message_hash_to_transaction (anza-xyz#374) Updates bootstrap and start scripts needed for local dev. (anza-xyz#384) Remove Deprecated Cli Args (anza-xyz#387) Master Rebase improve simulate_bundle errors and response (anza-xyz#404) derive Clone on accountoverrides (anza-xyz#416) Add upsert to AccountOverrides (anza-xyz#419) update jito-programs (anza-xyz#430) [JIT-1661] Faster Autosnapshot (anza-xyz#436) Reverts simulate_transaction result calls to upstream (anza-xyz#446) Don't unlock accounts in TransactionBatches used during simulation (anza-xyz#449) first pass at wiring up jito-plugin (anza-xyz#428) [JIT-1713] Fix bundle's blockspace preallocation (anza-xyz#489) [JIT-1708] Fix TOC TOU condition for relayer and block engine config (anza-xyz#491) [JIT-1710] - Optimize Bundle Consumer Checks (anza-xyz#490) Add Blockhash Metrics to Bundle Committer (anza-xyz#500) add priority fee ix to mev-claim (anza-xyz#520) Update Autosnapshot (anza-xyz#548) Run MEV claims + reclaiming rent-exempt amounts in parallel. (anza-xyz#582) Update CI (anza-xyz#584) - Add recursive submodule checkouts. - Re-add solana-secondary step Add more release fixes (anza-xyz#585) Fix more release urls (anza-xyz#588) [JIT-1812] Fix blocking mutexs (anza-xyz#495) [JIT-1711] Compare the unprocessed transaction storage BundleStorage against a constant instead of VecDeque::capacity() (anza-xyz#587) Automatically rebase Jito-Solana on a periodic basis. Send message on slack during any failures or success. Fix periodic rebase anza-xyz#594 Fixes the following bugs in the periodic rebase: Sends multiple messages on failure instead of one Cancels entire job if one branch fails Ignore buildkite curl errors for rebasing and try to keep curling until job times out (anza-xyz#597) Sleep longer waiting for buildkite to start (anza-xyz#598) correctly initialize account overrides (anza-xyz#595) Fix: Ensure set contact info to UDP port instead of QUIC (anza-xyz#603) Add fast replay branch to daily rebase (anza-xyz#607) take a snapshot of all bundle accounts before sim (anza-xyz#13) (anza-xyz#615) update jito-programs submodule Add 2.0 to daily rebase (anza-xyz#626) Export agave binaries during docker build (anza-xyz#627) Buffer bundles that exceed processing time and make the allowed processing time longer (anza-xyz#611) Publish releases to S3 and GCS (anza-xyz#633) Rebase from different repos (anza-xyz#637) Point SECURITY.md to immunefi (anza-xyz#671) Loosen requirements on tip accounts touchable in BankingStage (anza-xyz#683) Separate out broadcast + retransmit shredstream (anza-xyz#703) Add packet flag for staked node (anza-xyz#705) Add auto-rebase to v2.1 (anza-xyz#739) Fix release github (anza-xyz#745) Move block_cost_limit tracking to BankingStage in preparation for SIMD-0207 (anza-xyz#753) Add precompile checks in BundleStage (anza-xyz#787) Add auto-rebase to v2.2 (anza-xyz#818) Add better error handling around missing transaction signatures for bundle id generation (anza-xyz#860) Remove unwrap from authentication (anza-xyz#861) Revert Jito-Solana WorkingBankEntry changes (anza-xyz#873) Add libclang to Dockerfile (anza-xyz#885) Remove the tip distributor code (anza-xyz#888) Rebase: Update anchor to not use deprecated crates Add TLS webpki roots back in (anza-xyz#933) Remove trusted relayer packets (anza-xyz#952) Fix shred retransmit (anza-xyz#954) [v3.0] Automatically use optimal Block Engine region (anza-xyz#983) (Backport: anza-xyz#995) Disable autoconfig (anza-xyz#996)

It seems simpler to have two distinct generator templates: one for attacks that want to use the maximum possible number of accounts in a transaction, and another one that uses the maximum possible number of accounts of the maximum size. These two templates have constraints that are quite different, and it simplifies both the implementation and the interfaces of the templates. SQUASH: This change should be incorporated into the following changes: Brennan <brennan.watt@anza.xyz> Tue Sep 3 21:28:31 2024 -0700 constrain max accounts, shrink batch size (anza-xyz#419) Brennan <brennan.watt@anza.xyz> Thu Apr 18 18:10:43 2024 -0700 read max accounts attack (anza-xyz#327) Illia Bobyr <illia.bobyr@solana.com> Mon Oct 23 12:21:02 2023 -0700 Read and write lock attack (anza-xyz#196)

Original names were not specific enough. "Max" in the name could refer to the transaction size, account size, number of accounts or, possibly, even more other things. And we already have a `max_accounts_tx` template that indeed is about putting as many accounts into a transaction - it is confusing that `read_max_accounts` and `max_accounts_tx` are unrelated. FIXUP: This change should be absorbed into the changes that touch `read_max_accounts.rs` and `write_max_accounts.rs`: ``` ❯ git log --format=%s agave/master..upstream/master -- core/src/banking_stage/adversary/transaction_generators/{read,write}_max_accounts.rs SQUASH Split max_size_accounts_tx out of max_accounts_tx (anza-xyz#454) constrain max accounts, shrink batch size (anza-xyz#419) write max accounts test (anza-xyz#328) read max accounts attack (anza-xyz#327) Read and write lock attack (anza-xyz#196) ```

only reroute if relayer connected (anza-xyz#123) feat: add client tls config (anza-xyz#121) remove extra val (anza-xyz#129) fix clippy (anza-xyz#130) copy all binaries to docker-output (anza-xyz#131) Ledger tool halts at slot passed to create-snapshot (anza-xyz#118) update program submodule (anza-xyz#133) quick fix for tips and clearing old bundles (anza-xyz#135) update submodule to new program (anza-xyz#136) Improve stake-meta-generator usability (anza-xyz#134) pinning submodule head (anza-xyz#140) Use BundleAccountLocker when handling tip txs (anza-xyz#147) Add metrics for relayer + block engine proxy (anza-xyz#149) Build claim-mev in docker (anza-xyz#141) Rework bundle receiving and add metrics (anza-xyz#152) (anza-xyz#154) update submodule + dev files (anza-xyz#158) Deterministically find tip amounts, add meta to stake info, and cleanup pubkey/strings in MEV tips (anza-xyz#159) update jito-programs submodule (anza-xyz#160) Separate MEV tip related workflow (anza-xyz#161) Add block builder fee protos (anza-xyz#162) fix jito programs (anza-xyz#163) update submodule so autosnapshot exits out of ledger tool early (anza-xyz#164) Pipe through block builder fee (anza-xyz#167) pull in new snapshot code (anza-xyz#171) block builder bug (anza-xyz#172) Pull in new slack autosnapshot submodule (anza-xyz#174) sort stake meta json and use int math (anza-xyz#176) add accountsdb conn submod (anza-xyz#169) Update tip distribution parameters (anza-xyz#177) new submodules (anza-xyz#180) Add buildkite link for jito CI (anza-xyz#183) Fixed broken links to repositories (anza-xyz#184) Changed from ssh to https transfer for clone Seg/update submods (anza-xyz#187) fix tests (anza-xyz#190) rm geyser submod (anza-xyz#192) rm dangling geyser references (anza-xyz#193) fix syntax err (anza-xyz#195) use deterministic req ids in batch calls (anza-xyz#199) update jito-programs revert cargo update Cargo lock update with path fix fix cargo update autosnapshot with block lookback (anza-xyz#201) [JIT-460] When claiming mev tips, skip accounts that won't have min rent exempt amount after claiming (anza-xyz#203) Add logging for sol balance desired (anza-xyz#205) * add logging * add logging * update msg * tweak vars update submodule (anza-xyz#204) use efficient data structures when calling batch_simulate_bundles (anza-xyz#206) [JIT-504] Add low balance check in uploading merkle roots (anza-xyz#209) add config to simulate on top of working bank (anza-xyz#211) rm frozen bank check simulate_bundle rpc bugfixes (anza-xyz#214) rm frozen bank check in simulate_bundle rpc method [JIT-519] Store ClaimStatus address in merkle-root-json (anza-xyz#210) * add files * switch to include bump update submodule (anza-xyz#217) add amount filter (anza-xyz#218) update autosnapshot (anza-xyz#222) Print TX error in Bundles (anza-xyz#223) add new args to support single relayer and block-engine endpoints (anza-xyz#224) point to new jito-programs submod and invoke updated init tda instruction (anza-xyz#228) fix clippy errors (anza-xyz#230) fix validator start scripts (anza-xyz#232) Point README to gitbook (anza-xyz#237) use packaged cargo bin to build (anza-xyz#239) Add validator identity pubkey to StakeMeta (anza-xyz#226) The vote account associated with a validator is not a permanent link, so log the validator identity as well. bugfix: conditionally compile with debug flags (anza-xyz#240) Seg/tip distributor master (anza-xyz#242) * validate tree nodes * fix unit tests * pr feedback * bump jito-programs submod Simplify bootstrapping (anza-xyz#241) * startup without precompile * update spacing * use release mode * spacing fix validation rm validation skip Account for block builder fee when generating excess tip balance (anza-xyz#247) Improve docker caching delay constructing claim mev txs (anza-xyz#253) fix stake meta tests from bb fee (anza-xyz#254) fix tests Buffer bundles that exceed cost model (anza-xyz#225) * buffer bundles that exceed cost model clear qos failed bundles buffer if not leader soon (anza-xyz#260) update Cargo.lock to correct solana versions in jito-programs submodule (anza-xyz#265) fix simulate_bundle client and better error handling (anza-xyz#267) update submod (anza-xyz#272) Preallocate Bundle Cost (anza-xyz#238) fix Dockerfile (anza-xyz#278) Fix Tests (anza-xyz#279) Fix Tests (anza-xyz#281) * fix tests update jito-programs submod (anza-xyz#282) add reclaim rent workflow (anza-xyz#283) update jito-programs submod fix clippy errs rm wrong assertion and swap out file write fn call (anza-xyz#292) Remove security.md (anza-xyz#293) demote frequent relayer_stage-stream_error to warn (anza-xyz#275) account for case where TDA exists but not allocated (anza-xyz#295) implement better retries for tip-distributor workflows (anza-xyz#297) limit number of concurrent rpc calls (anza-xyz#298) Discard Empty Packet Batches (anza-xyz#299) Identity Hotswap (anza-xyz#290) small fixes (anza-xyz#305) Set backend config from admin rpc (anza-xyz#304) Admin Shred Receiver Change (anza-xyz#306) Seg/rm bundle UUID (anza-xyz#309) Fix github workflow to recursively clone (anza-xyz#327) Add recursive checkout for downstream-project-spl.yaml (anza-xyz#341) Use cluster info functions for tpu (anza-xyz#345) Use git rev-parse for git sha Remove blacklisted tx from message_hash_to_transaction (anza-xyz#374) Updates bootstrap and start scripts needed for local dev. (anza-xyz#384) Remove Deprecated Cli Args (anza-xyz#387) Master Rebase improve simulate_bundle errors and response (anza-xyz#404) derive Clone on accountoverrides (anza-xyz#416) Add upsert to AccountOverrides (anza-xyz#419) update jito-programs (anza-xyz#430) [JIT-1661] Faster Autosnapshot (anza-xyz#436) Reverts simulate_transaction result calls to upstream (anza-xyz#446) Don't unlock accounts in TransactionBatches used during simulation (anza-xyz#449) first pass at wiring up jito-plugin (anza-xyz#428) [JIT-1713] Fix bundle's blockspace preallocation (anza-xyz#489) [JIT-1708] Fix TOC TOU condition for relayer and block engine config (anza-xyz#491) [JIT-1710] - Optimize Bundle Consumer Checks (anza-xyz#490) Add Blockhash Metrics to Bundle Committer (anza-xyz#500) add priority fee ix to mev-claim (anza-xyz#520) Update Autosnapshot (anza-xyz#548) Run MEV claims + reclaiming rent-exempt amounts in parallel. (anza-xyz#582) Update CI (anza-xyz#584) - Add recursive submodule checkouts. - Re-add solana-secondary step Add more release fixes (anza-xyz#585) Fix more release urls (anza-xyz#588) [JIT-1812] Fix blocking mutexs (anza-xyz#495) [JIT-1711] Compare the unprocessed transaction storage BundleStorage against a constant instead of VecDeque::capacity() (anza-xyz#587) Automatically rebase Jito-Solana on a periodic basis. Send message on slack during any failures or success. Fix periodic rebase anza-xyz#594 Fixes the following bugs in the periodic rebase: Sends multiple messages on failure instead of one Cancels entire job if one branch fails Ignore buildkite curl errors for rebasing and try to keep curling until job times out (anza-xyz#597) Sleep longer waiting for buildkite to start (anza-xyz#598) correctly initialize account overrides (anza-xyz#595) Fix: Ensure set contact info to UDP port instead of QUIC (anza-xyz#603) Add fast replay branch to daily rebase (anza-xyz#607) take a snapshot of all bundle accounts before sim (anza-xyz#13) (anza-xyz#615) update jito-programs submodule Add 2.0 to daily rebase (anza-xyz#626) Export agave binaries during docker build (anza-xyz#627) Buffer bundles that exceed processing time and make the allowed processing time longer (anza-xyz#611) Publish releases to S3 and GCS (anza-xyz#633) Rebase from different repos (anza-xyz#637) Point SECURITY.md to immunefi (anza-xyz#671) Loosen requirements on tip accounts touchable in BankingStage (anza-xyz#683) Separate out broadcast + retransmit shredstream (anza-xyz#703) Add packet flag for staked node (anza-xyz#705) Add auto-rebase to v2.1 (anza-xyz#739) Fix release github (anza-xyz#745) Move block_cost_limit tracking to BankingStage in preparation for SIMD-0207 (anza-xyz#753) Add precompile checks in BundleStage (anza-xyz#787) Add auto-rebase to v2.2 (anza-xyz#818) Add better error handling around missing transaction signatures for bundle id generation (anza-xyz#860) Remove unwrap from authentication (anza-xyz#861) Revert Jito-Solana WorkingBankEntry changes (anza-xyz#873) Add libclang to Dockerfile (anza-xyz#885) Remove the tip distributor code (anza-xyz#888) Rebase: Update anchor to not use deprecated crates Add TLS webpki roots back in (anza-xyz#933) Remove trusted relayer packets (anza-xyz#952) Fix shred retransmit (anza-xyz#954) [v3.0] Automatically use optimal Block Engine region (anza-xyz#983) (Backport: anza-xyz#995) Disable autoconfig (anza-xyz#996)

only reroute if relayer connected (anza-xyz#123) feat: add client tls config (anza-xyz#121) remove extra val (anza-xyz#129) fix clippy (anza-xyz#130) copy all binaries to docker-output (anza-xyz#131) Ledger tool halts at slot passed to create-snapshot (anza-xyz#118) update program submodule (anza-xyz#133) quick fix for tips and clearing old bundles (anza-xyz#135) update submodule to new program (anza-xyz#136) Improve stake-meta-generator usability (anza-xyz#134) pinning submodule head (anza-xyz#140) Use BundleAccountLocker when handling tip txs (anza-xyz#147) Add metrics for relayer + block engine proxy (anza-xyz#149) Build claim-mev in docker (anza-xyz#141) Rework bundle receiving and add metrics (anza-xyz#152) (anza-xyz#154) update submodule + dev files (anza-xyz#158) Deterministically find tip amounts, add meta to stake info, and cleanup pubkey/strings in MEV tips (anza-xyz#159) update jito-programs submodule (anza-xyz#160) Separate MEV tip related workflow (anza-xyz#161) Add block builder fee protos (anza-xyz#162) fix jito programs (anza-xyz#163) update submodule so autosnapshot exits out of ledger tool early (anza-xyz#164) Pipe through block builder fee (anza-xyz#167) pull in new snapshot code (anza-xyz#171) block builder bug (anza-xyz#172) Pull in new slack autosnapshot submodule (anza-xyz#174) sort stake meta json and use int math (anza-xyz#176) add accountsdb conn submod (anza-xyz#169) Update tip distribution parameters (anza-xyz#177) new submodules (anza-xyz#180) Add buildkite link for jito CI (anza-xyz#183) Fixed broken links to repositories (anza-xyz#184) Changed from ssh to https transfer for clone Seg/update submods (anza-xyz#187) fix tests (anza-xyz#190) rm geyser submod (anza-xyz#192) rm dangling geyser references (anza-xyz#193) fix syntax err (anza-xyz#195) use deterministic req ids in batch calls (anza-xyz#199) update jito-programs revert cargo update Cargo lock update with path fix fix cargo update autosnapshot with block lookback (anza-xyz#201) [JIT-460] When claiming mev tips, skip accounts that won't have min rent exempt amount after claiming (anza-xyz#203) Add logging for sol balance desired (anza-xyz#205) * add logging * add logging * update msg * tweak vars update submodule (anza-xyz#204) use efficient data structures when calling batch_simulate_bundles (anza-xyz#206) [JIT-504] Add low balance check in uploading merkle roots (anza-xyz#209) add config to simulate on top of working bank (anza-xyz#211) rm frozen bank check simulate_bundle rpc bugfixes (anza-xyz#214) rm frozen bank check in simulate_bundle rpc method [JIT-519] Store ClaimStatus address in merkle-root-json (anza-xyz#210) * add files * switch to include bump update submodule (anza-xyz#217) add amount filter (anza-xyz#218) update autosnapshot (anza-xyz#222) Print TX error in Bundles (anza-xyz#223) add new args to support single relayer and block-engine endpoints (anza-xyz#224) point to new jito-programs submod and invoke updated init tda instruction (anza-xyz#228) fix clippy errors (anza-xyz#230) fix validator start scripts (anza-xyz#232) Point README to gitbook (anza-xyz#237) use packaged cargo bin to build (anza-xyz#239) Add validator identity pubkey to StakeMeta (anza-xyz#226) The vote account associated with a validator is not a permanent link, so log the validator identity as well. bugfix: conditionally compile with debug flags (anza-xyz#240) Seg/tip distributor master (anza-xyz#242) * validate tree nodes * fix unit tests * pr feedback * bump jito-programs submod Simplify bootstrapping (anza-xyz#241) * startup without precompile * update spacing * use release mode * spacing fix validation rm validation skip Account for block builder fee when generating excess tip balance (anza-xyz#247) Improve docker caching delay constructing claim mev txs (anza-xyz#253) fix stake meta tests from bb fee (anza-xyz#254) fix tests Buffer bundles that exceed cost model (anza-xyz#225) * buffer bundles that exceed cost model clear qos failed bundles buffer if not leader soon (anza-xyz#260) update Cargo.lock to correct solana versions in jito-programs submodule (anza-xyz#265) fix simulate_bundle client and better error handling (anza-xyz#267) update submod (anza-xyz#272) Preallocate Bundle Cost (anza-xyz#238) fix Dockerfile (anza-xyz#278) Fix Tests (anza-xyz#279) Fix Tests (anza-xyz#281) * fix tests update jito-programs submod (anza-xyz#282) add reclaim rent workflow (anza-xyz#283) update jito-programs submod fix clippy errs rm wrong assertion and swap out file write fn call (anza-xyz#292) Remove security.md (anza-xyz#293) demote frequent relayer_stage-stream_error to warn (anza-xyz#275) account for case where TDA exists but not allocated (anza-xyz#295) implement better retries for tip-distributor workflows (anza-xyz#297) limit number of concurrent rpc calls (anza-xyz#298) Discard Empty Packet Batches (anza-xyz#299) Identity Hotswap (anza-xyz#290) small fixes (anza-xyz#305) Set backend config from admin rpc (anza-xyz#304) Admin Shred Receiver Change (anza-xyz#306) Seg/rm bundle UUID (anza-xyz#309) Fix github workflow to recursively clone (anza-xyz#327) Add recursive checkout for downstream-project-spl.yaml (anza-xyz#341) Use cluster info functions for tpu (anza-xyz#345) Use git rev-parse for git sha Remove blacklisted tx from message_hash_to_transaction (anza-xyz#374) Updates bootstrap and start scripts needed for local dev. (anza-xyz#384) Remove Deprecated Cli Args (anza-xyz#387) Master Rebase improve simulate_bundle errors and response (anza-xyz#404) derive Clone on accountoverrides (anza-xyz#416) Add upsert to AccountOverrides (anza-xyz#419) update jito-programs (anza-xyz#430) [JIT-1661] Faster Autosnapshot (anza-xyz#436) Reverts simulate_transaction result calls to upstream (anza-xyz#446) Don't unlock accounts in TransactionBatches used during simulation (anza-xyz#449) first pass at wiring up jito-plugin (anza-xyz#428) [JIT-1713] Fix bundle's blockspace preallocation (anza-xyz#489) [JIT-1708] Fix TOC TOU condition for relayer and block engine config (anza-xyz#491) [JIT-1710] - Optimize Bundle Consumer Checks (anza-xyz#490) Add Blockhash Metrics to Bundle Committer (anza-xyz#500) add priority fee ix to mev-claim (anza-xyz#520) Update Autosnapshot (anza-xyz#548) Run MEV claims + reclaiming rent-exempt amounts in parallel. (anza-xyz#582) Update CI (anza-xyz#584) - Add recursive submodule checkouts. - Re-add solana-secondary step Add more release fixes (anza-xyz#585) Fix more release urls (anza-xyz#588) [JIT-1812] Fix blocking mutexs (anza-xyz#495) [JIT-1711] Compare the unprocessed transaction storage BundleStorage against a constant instead of VecDeque::capacity() (anza-xyz#587) Automatically rebase Jito-Solana on a periodic basis. Send message on slack during any failures or success. Fix periodic rebase anza-xyz#594 Fixes the following bugs in the periodic rebase: Sends multiple messages on failure instead of one Cancels entire job if one branch fails Ignore buildkite curl errors for rebasing and try to keep curling until job times out (anza-xyz#597) Sleep longer waiting for buildkite to start (anza-xyz#598) correctly initialize account overrides (anza-xyz#595) Fix: Ensure set contact info to UDP port instead of QUIC (anza-xyz#603) Add fast replay branch to daily rebase (anza-xyz#607) take a snapshot of all bundle accounts before sim (anza-xyz#13) (anza-xyz#615) update jito-programs submodule Add 2.0 to daily rebase (anza-xyz#626) Export agave binaries during docker build (anza-xyz#627) Buffer bundles that exceed processing time and make the allowed processing time longer (anza-xyz#611) Publish releases to S3 and GCS (anza-xyz#633) Rebase from different repos (anza-xyz#637) Point SECURITY.md to immunefi (anza-xyz#671) Loosen requirements on tip accounts touchable in BankingStage (anza-xyz#683) Separate out broadcast + retransmit shredstream (anza-xyz#703) Add packet flag for staked node (anza-xyz#705) Add auto-rebase to v2.1 (anza-xyz#739) Fix release github (anza-xyz#745) Move block_cost_limit tracking to BankingStage in preparation for SIMD-0207 (anza-xyz#753) Add precompile checks in BundleStage (anza-xyz#787) Add auto-rebase to v2.2 (anza-xyz#818) Add better error handling around missing transaction signatures for bundle id generation (anza-xyz#860) Remove unwrap from authentication (anza-xyz#861) Revert Jito-Solana WorkingBankEntry changes (anza-xyz#873) Add libclang to Dockerfile (anza-xyz#885) Remove the tip distributor code (anza-xyz#888) Rebase: Update anchor to not use deprecated crates Add TLS webpki roots back in (anza-xyz#933) Remove trusted relayer packets (anza-xyz#952) Fix shred retransmit (anza-xyz#954) Add daily v3.0 rebase (anza-xyz#972) [Master] Automatically use optimal Block Engine region (anza-xyz#974) Disable autoconfig (anza-xyz#995) Make shredstream optional (anza-xyz#997)

Lichtso added the v1.18 label Mar 25, 2024

Lichtso force-pushed the fix/failed_verification_and_closed branch 3 times, most recently from f2990ee to 84b1a11 Compare March 27, 2024 15:34

Lichtso requested review from alessandrod and pgarg66 March 27, 2024 17:42

Lichtso force-pushed the fix/failed_verification_and_closed branch 4 times, most recently from 71a38b3 to ded3c25 Compare March 29, 2024 12:43

alessandrod reviewed Apr 4, 2024

View reviewed changes

Lichtso force-pushed the fix/failed_verification_and_closed branch from 633963e to 189b852 Compare April 4, 2024 13:46

alessandrod approved these changes Apr 4, 2024

View reviewed changes

Lichtso added 8 commits April 5, 2024 11:04

Only the verifier can cause FailedVerification, everything else is Cl…

e3a2609

…osed

Removes the environments parameter from load_program_accounts().

cde499b

cargo fmt

d6612ae

Simplify invocation of deployed program

edf358a

Attempt to invoke a program before it is deployed

e6be8a8

Attempt to invoke a buffer before it is used in a deployment

4ccd1dc

Escalates Option return value of load_program_accounts() to load_prog…

c1af5b5

…ram_with_pubkey().

Review feedback

fd971ad

Lichtso force-pushed the fix/failed_verification_and_closed branch from 189b852 to fd971ad Compare April 5, 2024 09:08

alessandrod approved these changes Apr 5, 2024

View reviewed changes

Lichtso merged commit 55c05c5 into master Apr 5, 2024

Lichtso deleted the fix/failed_verification_and_closed branch April 5, 2024 11:03

mergify Bot mentioned this pull request Apr 5, 2024

v1.18: Fix - FailedVerification and Closed tombstones (backport of #419) #605

Merged

OliverNChalk pushed a commit to OliverNChalk/agave that referenced this pull request Nov 11, 2025

constrain max accounts, shrink batch size (anza-xyz#419)

5b181ab

		@@ -386,82 +385,77 @@ impl<FG: ForkGraph> TransactionBatchProcessor<FG> {
		pubkey: &Pubkey,

Conversation

Lichtso commented Mar 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problems

Summary of Changes

Uh oh!

mergify Bot commented Mar 25, 2024

Uh oh!

t-nelson commented Apr 2, 2024

Uh oh!

alessandrod commented Apr 3, 2024

Uh oh!

alessandrod left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lichtso Apr 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alessandrod left a comment

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Apr 5, 2024

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Lichtso commented Mar 25, 2024 •

edited

Loading

Lichtso Apr 4, 2024 •

edited

Loading