Moar rolling upgrade

[ryoqun]

Problem

testing upgrade version test is hard and required local patches to be merged into stable and beta release channels
and supporting the combination of newer net/*.sh and older validator seems tedious and basically unmaintained.

Summary of Changes

basic idea is like this

1. git reset --hard $HEAD_OF_STABLE_OR_WHATEVER_PR_ON_STABLE_BRANCH
2. build locally
3. scp binaries
4. ./net/start.sh ...
5. git reset --hard $HEAD_OF_BETA_OR_WHATEVER_PR_ON_STABLE_BRANCH
6. build locally
7. scp binaries
8. ./net/start.sh ...

This is reboot of #9212 op top of #12337 . #9212 originally closed without merging because building local version is supported without that pr. But now we want to support to building multiple versions locally. So I think this justifies the need of this pr.

Fixes #

[ryoqun]

@t-nelson Yay, this worked nicely: https://buildkite.com/solana-labs/rolling-upgrade-tests/builds/113

with

CHANNEL=local
LOCAL_BUILD_BRANCH=pull/12387/head
UPGRADE_CHANNEL=local
UPGRADE_LOCAL_BUILD_BRANCH=v1.3

So, the rolling upgrade test passed only when #12387 applied.

[ryoqun]

supporting the combination of newer net/*.sh and older validator seems tedious and basically unmaintained.

Well, this pr currently doesn't solve this part

[t-nelson]

Hey thanks for taking a stab at this. I have a somewhat embarrassing hack to get similar. We can actually avoid the local builds entirely, passing them off as optional buildkite jobs that store the release as artifacts. Then testnet-automation can resolve and download the right builds at runtime

Gross script: t-nelson@b17c75d

Buildkite:

[t-nelson]

supporting the combination of newer net/*.sh and older validator seems tedious and basically unmaintained.

Well, this pr currently doesn't solve this part

Hehe, true. I believe this is one of our, perhaps unstated, devops goals, though. We no intention of actually supporting features back to stable, just that the master/HEAD scripts executing on the stable/HEAD bins should work. This mostly just means backporting hidden stubs so clap doesn't blow up when parsing the command line.

[ryoqun]

We can actually avoid the local builds entirely, passing them off as optional buildkite jobs that store the release as artifacts. ...

@t-nelson I see. Glad to see that you had similar one. Yeah, these're very similar. So, it seems that we can polish your separate buildkite job approach. Are you going to polish it at #12337? Alternatively, I can work on polishing it if you're busy with other things. :)

[t-nelson]

@t-nelson I see. Glad to see that you had similar one. Yeah, these're very similar. So, it seems that we can polish your separate buildkite job approach. Are you going to polish it at #12337? Alternatively, I can work on polishing it if you're busy with other things. :)

I was hoping to get back to it after cleaning up the SPL Token CLI (probably EOW), but if you've got room on your plate, fee free to take over! I'll consolidate my changes into the branch pointed at #12337 (and rip out the junk debugging commits).

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[stale]

This stale pull request has been automatically closed. Thank you for your contributions.