Ci/crate publish test

[mircea-c]

Problem

Publishing crates to crates.io sometimes fails during releases leaving versions in an inconsistent state.

Summary of Changes

Add an extra task to the crate-check workflow which publishes the crates to a test repo.

Closes https://github.com/anza-xyz/devops/issues/299

[yihau]

why do we need to wait for the trigger-buildkite-pipeline step 🤔 they won't need each other's data

[mircea-c]

This is meant to stop the draft release and version bump from running in case the crate and bin publishing fails.

[yihau]

when the buildkite pipeline is triggered, the trigger-buildkite-pipeline step will show green immediately. it doesn't reflect the pipeline's result. we will need to update the flow to achieve your goal

[mircea-c]

Oh, damn, that makes sense 😞

[mircea-c]

Updated the action code. The new version of the buildkite action adds a wait functionality, so this will wait until the pipeline is finished.

[yihau]

I have, but github runners are under powered compared to our CI machines and this takes too long as it is. I'm also trying to keep the changes to the release pipeline in this PR to a minimum.

we can use more powerful GitHub-hosted machines to run it. do you have any link you can share? I’d like to know what time you got.

[yihau]

It is a lot hacky, but changing the buildkite pipeline is only done manually through the UI right now, and I don't want to touch it. Not sure I understand your comment about 2 builds for the same commit though.

oh, the 2 builds means that in this PR, the building process looks like

1. triggering the secondary pipeline with a special message
2. triggering the secondary pipeline again without the special message

so we will get 2 different build numbers in the buildkite page, but they are the same commits.

[mircea-c]

we can use more powerful GitHub-hosted machines to run it. do you have any link you can share? I’d like to know what time you got.

The last successful run from buildkite took 1h:

https://buildkite.com/anza/agave/builds/24657/steps/canvas?sid=019746d4-191b-45f9-ba67-f8af0a9415e3

We can use self hosted github runners I guess, but that's a big change.

[yihau]

we can just use github larger runner atm I guess

[yihau]

if we’re only concerned about build time, maybe we can run it only on push and place it in the same block as the coverage test 🤔

[yihau]

I think this one should happen in the main pipeline or a different pipeline instead of here. some reasons:

1. I don't think we want the crates publishing issue will block our bin shipping
2. PRs won't trigger this pipeline, which means we won't be able to catch this error immediately

[mircea-c]

We want the release to stop if there's an issue with the crate publishing. That's why I moved this back here.

This test takes over an hour to run, so it's not a good choice to have it run on every PR.

I'll do some more work on making a more targeted check for each crate that we can add to regular CI later.

[yihau]

okay. if we want to to do so, I will suggest that we can use queue: "solana" instead of queue: "release-build".

[mircea-c]

What is the benefit of moving it to the other queue?

I see there's only one agent in the release-build queue. Are you concerned about the check taking up too much time in that queue?

[yihau]

the only difference atm is that release-build has some secrets. if you don't need them, you can use other machines for reducing wait time.

[mircea-c]

Superseded by #9334