Remove ThinClient from LocalCluster

[gregcusack]

A Followup PR to: #258.
This is the 10th PR on the way to remove ThinClient completely.

Problem

ThinClient is deprecated. Replacing with TpuClient

Summary of Changes

Replace ThinClient with TpuClient in LocalCluster
Specifically, we use a Quic TpuClient, not a UDP TpuClient

Notes

This works but I can't say for sure why it works.
I had initially replaced all of ThinClient within LocalCluster with the nonblocking version of TpuClient. I had made the switch such that transfer_with_client() calls TpuClient.try_send_transaction() which calls send_wire_transaction_to_addr(). send_wire_transaction_to_addr() uses a non-blocking connection from the connection_cache to send the transaction.

However, using the changes above, I consistently ran into an issue with any test that called add_validator() more than once. Under the hood, add_validator() calls transfer_with_client(). transfer_with_client() transfers some amount of lamports to a destination account (aka the account associated with the validator we are trying to add). However in the LocalCluster tests, transfer_with_client() would consistently fail when adding the second validator with add_validator(). It would always fail with: transport custom error: "ConnectError(EndpointStopping).

I noticed that ThinClient used a blocking connection when sending transactions with retry_transfer() (which was used to send a transaction within transfer_with_client(). So, I switched over to using the blocking TpuClient and that seems to work well. I had to make a few changes to expose the leader schedule, connection cache, and fanout slots from the nonblocking/tpu_client up to the blocking tpu_client.

I am still working on investigating. But thinks its probably a good idea to get some eyes on this since it does work as expected now.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.1%. Comparing base (ad12ea3) to head (b86668f).
⚠️ Report is 6518 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #977   +/-   ##
=======================================
  Coverage    82.1%    82.1%           
=======================================
  Files         893      893           
  Lines      236936   236931    -5     
=======================================
+ Hits       194705   194720   +15     
+ Misses      42231    42211   -20     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[gregcusack]

closed in favor of: #1300