Fix race in ci/run-sanity.sh

[ryoqun]

Problem

CI is a bit unstable on master:

https://buildkite.com/solana-labs/solana/builds/26540#3a9158a5-2de3-4b61-b25a-25e63f4400ac/2724-2898
Even if this

 while [[ $($solana_cli --url http://localhost:8899 slot --commitment recent) -eq 0 ]]; do

evaluates to:

+ [[ 1 -eq 0 ]]

, it seems that we can't create snapshot for the slot 1:

     Running `target/debug/solana-ledger-tool create-snapshot --ledger config/ledger 1 config/snapshot-ledger`
...
Error: Slot 1 is not available

Summary of Changes

Wait more.

context

frustrated while developing at #10718

[t-nelson]

Is the problem not that we're querying the slot with recent commitment and ledger-tool wants a root?

I ran into similar when plumbing --warp-slot through for rolling upgrade tests and we run wallet sanity before the deposit transaction's slot gets rooted. Can-o-worms these scripts 🙂

[ryoqun]

@t-nelson Well, ledger-tool create-snapshot should work even with non-root slots... run-sanity.sh doesn't definitely wait for the first root (slot = 32) here. My guess is that it just don't fsync for every slot. We rather forcibly shutdown a validator with the validatorexit rpc endpoint.

Let's see how this pr stabilizes. :)

[ryoqun]

@t-nelson I was lying...:

solana/runtime/src/accounts_db.rs

Line 1837 in 0550b89

**slot <= snapshot_slot && accounts_index.is_root(**slot)

non root doesn't work. And it's quite dangerous..... I'll add rooted check in ledger-tool create-snapshot...

[ryoqun]

Is the problem not that we're querying the slot with recent commitment and ledger-tool wants a root?

I ran into similar when plumbing --warp-slot through for rolling upgrade tests and we run wallet sanity before the deposit transaction's slot gets rooted. Can-o-worms these scripts slightly_smiling_face

I'll add rooted check in ledger-tool create-snapshot...

@t-nelson I was about to create a new PR for this, but it turns out ledger-tool create-snapshot does it correct:

solana/ledger-tool/src/main.rs

Line 1255 in 4b93a7c

bank.squash();

So, my quoted code does work correctly:

solana/runtime/src/accounts_db.rs

Line 1837 in 0550b89

**slot <= snapshot_slot && accounts_index.is_root(**slot)

So, I think there is no need to improve this further.