Upgrade Solana to 1.17.0

[acheroncrypto]

Upgrade to Solana 1.17.0

Problem

Upgrading Solana version in Anchor repo to 1.17.0(coral-xyz/anchor#2645) but auction-house program doesn't work after 1.16. We were using 1.14 to run the auction-house tests but now that we're dropping the 1.14 support, that's not an option anymore.

Problem 1

solana-labs/solana#31960 affected both the auction-house program and the metaplex-token-metadata program.

'Program hausS13jsjafwWwGqZTUQRmWyvyxn9EQpqMwV1PBBmk invoke [1]',
'Program log: Instruction: CreateAuctionHouse',
'Program hausS13jsjafwWwGqZTUQRmWyvyxn9EQpqMwV1PBBmk consumed 11789 of 200000 compute units',
'Program hausS13jsjafwWwGqZTUQRmWyvyxn9EQpqMwV1PBBmk failed: Invoked an instruction with data that is too large (12884933012 > 10240)'

The solution to the metaplex-token-metadata program's error was to compile the program with 1.14 and use the pre-built binary for tests, and the solution for the auction-house program's error was to upgrade the Solana dependencies(solana-labs/solana#31960 (comment)).

Problem 2

Upgrading Solana dependencies solved the previous error but it also resulted with a compile error on metaplex-token-vault crate.

error[E0277]: the trait bound `T: borsh::de::BorshDeserialize` is not satisfied
   --> src/utils.rs:261:21
    |
261 |     let result: T = try_from_slice_unchecked(data)?;
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^ the trait `borsh::de::BorshDeserialize` is not implemented for `T`
    |
note: required by a bound in `solana_program::borsh::try_from_slice_unchecked`
   --> src/borsh.rs:38:1
help: consider further restricting this bound
    |
250 | pub fn try_from_slice_checked<T: BorshDeserialize + borsh::de::BorshDeserialize>(
    |                                                   +++++++++++++++++++++++++++++

For more information about this error, try `rustc --explain E0277`.
error: could not compile `metaplex-token-vault` due to previous error

The solution was to remove the metaplex-token-metadata dependency and instead use the metadata feature of anchor-spl to get access to the necessary functionality(mpl-token-metadata is now exported coral-xyz/anchor#2583).

Problem 3

After solving some more compile errors, the last error was in the execute_sale instruction which was out of memory. Apparently platform-tools v1.37(solana ^1.16.0) uses more memory for the same instruction.

'Program hausS13jsjafwWwGqZTUQRmWyvyxn9EQpqMwV1PBBmk invoke [1]',
    'Program log: Instruction: ExecuteSale',
    'Program hausS13jsjafwWwGqZTUQRmWyvyxn9EQpqMwV1PBBmk consumed 3383 of 200000 compute units',
    'Program hausS13jsjafwWwGqZTUQRmWyvyxn9EQpqMwV1PBBmk failed: Access violation in stack frame 5 at address 0x200005fe8 of size 8 at BPF instruction #40071'

The solution was to use references instead of cloning and remove the rent account from accounts.