TestValidator checks usability of programs added to genesis during startup

[tao-stones]

Problem

• programs added to TestValidator's genesis isn't immediately usable. Runtime needs time to mark the program as "deployed";
• TestValidator has a hack during startup to wait for fee stabilizes.
• During this wait period, programs in TestValidator genesis usually will become ready for use; But this isn't always reliable, sometime cause test to fail due to "Program is not deployed":

Waiting for fees to stabilize 1...

thread 'main' panicked at clients/cli/tests/command.rs:278:14:
called `Result::unwrap()` on an `Err` value: Error { request: Some(SendTransaction), kind: RpcError(RpcResponseError { code: -32002, message: "Transaction simulation failed: Error processing Instruction 0: Unsupported program id", data: SendTransactionPreflightFailure(RpcSimulateTransactionResult { err: Some(InstructionError(0, UnsupportedProgramId)), logs: Some(["Program TokenzQdBNbLqP5VEhdkAS6EPFLC1PHnBqCXEpPxuEb invoke [1]", "Program is not deployed", "Program TokenzQdBNbLqP5VEhdkAS6EPFLC1PHnBqCXEpPxuEb failed: Unsupported program id"]), accounts: None, units_consumed: Some(0), return_data: None, inner_instructions: None, replacement_blockhash: None }) }) }

Summary of Changes

Actively check for "Program is not deployed", wait till program becomes usable.

Fixes #

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 90.43478% with 11 lines in your changes missing coverage. Please review.

Project coverage is 82.8%. Comparing base (ccdcdbe) to head (9ea07b5).
Report is 14 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff            @@
##           master    #6587    +/-   ##
========================================
  Coverage    82.8%    82.8%            
========================================
  Files         849      849            
  Lines      379032   379147   +115     
========================================
+ Hits       313911   314116   +205     
+ Misses      65121    65031    -90     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tao-stones]

Add tests to assert program added to genesis is available for use after TestValidator started.

To illustrate issue, if short circuit wait_for_nonzero_fee() by returning immediately (to simulate removing the hack) and disable wait_for_upgradeable_programs_deployed() (to simulate no change), these two tests will fail.

[lijunwangs]

better to use the normal log functions.

[lijunwangs]

If the program_id has been successful in previous iterations, we don't have to resend

[tao-stones]

9ea07b5 has it

[lijunwangs]

LGTM