hotfix: remove gh artifact retrieval limit

[scrasmussen]

SOURCE: Soren Rasmussen, NCAR

DESCRIPTION OF CHANGES:

• CI was failing since the gh command only finds the past 30 artifact ids by default. Adding the --paginate flag lists all artifacts that exist. Note, this doesn't list old ones, just ones that can be downloaded.

ISSUE: CI was failing since the gh command only finds the past 30 artifact ids by default. After enough PR CIs were run and created artifacts, the main branch artifacts were pushed off the returned list.

ASSOCIATED PRs:

• This issue showed up in UFS-dev PR#305 #630

TESTS CONDUCTED: List tests done as appropriate. Delete if not used.

[grantfirl]

Thanks @scrasmussen. I didn't understand exactly why it was failing, although I had an inkling that it had something to do with wiping artifacts after some threshold. To get around this, I was thinking that we can just have a workflow-dispatchable CI script to generate a fresh artifact from main. The current script could work in theory, but it errors out due to not being able to download existing artifacts before uploading newly-created ones.

I'm also wondering if we should change the triggers on the RT CIs to either only be workflow dispatchable or dispatchable via label. Having them automatically run every time there is a push to a PR is probably overkill.

[scrasmussen]

@grantfirl

to get around this, I was thinking that we can just have a workflow-dispatchable CI script to generate a fresh artifact from main.

These are good points, sorry this new testing feature didn't initially quite work as expected! I had considered adding a workflow-dispatchable CI script, what if the artifacts had expired from being too old, there would be no way to create new ones... I'll add a new script to this PR to do that.

but it errors out due to not being able to download existing artifacts before uploading newly-created ones.

I'm also wondering if we should change the triggers on the RT CIs to either only be workflow dispatchable or dispatchable via label. Having them automatically run every time there is a push to a PR is probably overkill.

I think there are pros and cons to doing it each way. Should new main artifacts only be pushed after they've been compared with the previous ones? Is it safer or should we consider any commit hitting the main branch to be truth by default, it would have gone through a vetting process already.

I prefer the overkill method, but am open to this discussion! I've added these points to our Tuesday SCM meeting notes so we can talk about them

[scrasmussen]

This PR is ready for review, I can split off the artifact-creation dispatch file into a new PR if needed (it isn't really a hotfix so I'm overloading this PR, apologies)

[grantfirl]

This PR is ready for review, I can split off the artifact-creation dispatch file into a new PR if needed (it isn't really a hotfix so I'm overloading this PR, apologies)

No need, this is great. We should merge once CI finishes.