svm: test account loader edge cases

[2501babe]

Problem

the existing account loader has a few unique edge cases:

• if a program can be found in the cache, the executable flag on the program account retrevied from accounts-db is never checked, unless it is writable or an instruction account
• if native loader is an instruction account, it is counted against the transaction data size limit, otherwise it is not
• if any other required loader is also an instruction account, it is counted against the transaction data size limit twice
• an upgradeable program retrieved from cache has a loaded size that includes its programdata, so size totals differ if the cache is used or skipped

all of these affect consensus because they may determine whether the transaction is executed. as we are implementing a new loader without a feature gate, we must test that the new loader preserves these behaviors

Summary of Changes

add tests for them. we do this as a separate pr from the new loader so we are sure they pass for the old loader

[2501babe]

(need to wait for dependabot to upstream a dependency version update for this to pass again)

[apfitzge]

Probably good to get a second opionion on these tests, but overall they look good to me (in that they are testing the current behavior)

[jstarry]

Looks good, this is really crazy behavior. Do you mind adding some comments explaining why this happens?

From my current understanding:

1. All builtins and any account owned by a loader are added to the program cache before tx account loading
2. If a tx account loads an account which is not writable and not used as an instruction account input, we check if it's in the program cache
3. If the tx account is in the program cache, then load it as if it's a valid program for execution
4. Later if that account is invoked as a program, the bpf loaders will check the actual program cache entry to see if it's a valid program. If not, the tx is "executed" but will return an error.

[2501babe]

thats correct. the issue is the current code just checks if something exists in the cache, and then assumes its executable, so it attempts to execute and it fails. it does load the data anyway (so using the cache isnt really an optimization) but just to see if it exists, which i assume was added to fix a bug

    } else if let Some(program) = (!is_instruction_account && !is_writable)
        .then_some(())
        .and_then(|_| loaded_programs.find(account_key))
    {
        callbacks
            .get_account_shared_data(account_key)
            .ok_or(TransactionError::AccountNotFound)?;
        // Optimization to skip loading of accounts which are only used as
        // programs in top-level instructions and not passed as instruction accounts.
        LoadedTransactionAccount {
            loaded_size: program.account_size,
            account: account_shared_data_from_program(&program),
            rent_collected: 0,
        }
    } else {

checking executable on the account here requires a feature gate because (until fee-only txn are enabled) it would change whether the transaction pays fees and thus affects consensus. but i talked with pankaj the other week and he said that the status on the cache item is sufficient to determine if it is in fact executable or not (this is what i meant in the tombstone comment, ill expand it in next commit). so in the future when we fix this i believe we can just do this

        } else if let Some(ref program) = (!is_instruction_account && !is_writable)
            .then_some(())
            .and_then(|_| program_cache.find(account_key))
            .filter(|program| !program.is_tombstone())
        {   
            loaded_accounts_map.insert_cached_program(*account_key, program);
        } else if ... {

[jstarry]

Can you add separate variants of this test where the program is 1) loaded as writeable and 2) passed as an instruction account to show the difference in behavior?

[jstarry]

Can you also add a test case for loading a tx which invokes native loader directly?

[jstarry]

For posterity, native loader is an edge case because account loading for some reason doesn't reject transactions that invoke the native loader as a program despite the native loader not being a normal "builtin" (therefore doesn't get added to the program cache) as well as not actually existing onchain as an executable account.

[jstarry]

We should also have a way to set invoked programs as writable when the upgradeable loader is present because that will also bypass the cache.

[2501babe]

i dont think i understand the point about the upgradeable loader being present but ive added test cases for both data size and executable flag check which use a message with a writable, non-instruction, invoked program

[jstarry]

i dont think i understand the point about the upgradeable loader being present

Invoked programs that are loaded as writable by a transaction will be demoted to read-only unless the upgradeable loader is loaded by the transaction. Since you directly set the is_writable_account_cache yourself in the test, this isn't really relevant, never mind!

[jstarry]

Looks like process_instruction_inner is called for both the old and the upgradeable loader and will check that program cache entries are valid. Can you explain more about what you mean that the old loader doesn't check tombstones?

[2501babe]

i deleted this part of the comment since it is no longer relevant (i previously wrote a batched account loader that checked tombstones with an escape hatch to emulate old behavior, i now have a jit account loader which retains the old behavior unmodified), but sorry this was unclear, i meant the old and new account loaders, not the bpf loaders

[2501babe]

spent the rest of the week writing a jit account loader (good news: its diff against master is about 1300 fewer lines than the batch account loader), will get to your suggestions on these tests on monday!