test-validator: Cloned upgradeable programs don't work

[acheroncrypto]

Problem

Interacting with cloned upgradeable programs results in This program may not be used for executing instructions error.

This is likely related to the recent removal (solana-labs#34194) and readding (#309) of the executable flag because this used to work in 1.17.0.

Reproduction

Version: v1.18.8

1. Clone the spl-token-2022 program:

   solana-test-validator --clone-upgradeable-program TokenzQdBNbLqP5VEhdkAS6EPFLC1PHnBqCXEpPxuEb --url https://api.mainnet-beta.solana.com
   

2. Interact with the program:

   spl-token create-token --program-id TokenzQdBNbLqP5VEhdkAS6EPFLC1PHnBqCXEpPxuEb
   

   Results in error:

   Creating token BUsVqFX4dLFZiYKggKvF9XcmYgaFsBMNkTJfwiyUwvxL under program TokenzQdBNbLqP5VEhdkAS6EPFLC1PHnBqCXEpPxuEb
   Error: Client(Error { request: Some(SendTransaction), kind: RpcError(RpcResponseError { code: -32002, message: "Transaction simulation failed: This program may not be used for executing instructions", data: SendTransactionPreflightFailure(RpcSimulateTransactionResult { err: Some(InvalidProgramForExecution), logs: Some([]), accounts: None, units_consumed: Some(0), return_data: None }) }) })
   

Solution

Not sure if it's the best way but setting the program data account of cloned upgradeable programs to true fixes this issue.

This is also what --upgradeable-program option (solana-labs#30412) does:

agave/test-validator/src/lib.rs

Line 750 in b2b159a

executable: true,

[acheroncrypto]

After debugging this issue further, I found out that this broke after solana-labs#34407 (not the executable flag PRs).

solana-labs@22bfcd9 is a rather complex commit, so I don't know which part of the commit caused this issue. Fixing this behavior from the test-validator side seems to be much simpler if there is no other problems with the mentioned change.

[buffalojoec]

@acheroncrypto thanks for reporting and also digging into this!

I've been checking this out as well, and I found so far that this seems to be specific to the "cloning down" action:

• The existing --upgradeable-program argument sets the program data account as executable: true (as depicted in the issue above). I can invoke this program without issue.
• When modifying the --upgradeable-program argument to set the program data account as executable: false - as is done by the loader - I can invoke this program without issue.
• When deploying my program to the running local validator, the program data account is set to executable: false- as is done by the loader - I can invoke this program without issue.
• When deploying my program to devnet, then cloning it down using --clone-upgradeable-program, I can see the program data account is executable: false, and I get the reported issue.

As you've mentioned above, the issue also seems unrelated to the executable flag.

I'll continue checking this out.

[buffalojoec]

I'm quite surprised we haven't seen this reported yet, but I think I've narrowed it down.

The issue appears to be that the cloned program data account's deployment slot is preserved from the cluster it clones the account from, which is problematic when the programs cache goes to attempt an extraction.

agave/program-runtime/src/loaded_programs.rs

Line 908 in b01b0d0

if entry.deployment_slot <= self.latest_root_slot

At a high-level, it appears that this causes a continuous loop on Bank's replenish_program_cache, which is terminated (quickly) by the hit_max_limit check.

agave/runtime/src/bank.rs

Lines 4989 to 5002 in b01b0d0

	if let Some((key, program)) = program_to_store.take() {
	if loaded_programs_cache
	.finish_cooperative_loading_task(self.slot, key, program)
	&& limit_to_load_programs
	{
	let mut ret = LoadedProgramsForTxBatch::new_from_cache(
	self.slot,
	self.epoch,
	&loaded_programs_cache,
	);
	ret.hit_max_limit = true;
	return ret;
	}
	}

I intend to get a more descriptive and formal report, and ensure there isn't a deeper runtime issue. However, in the meantime I can assert that simply overwriting a cloned program data account's slot to 0 fixes the issue.