BOM release impossible due to missing Jenkins check

[alecharp]

Service(s)

GitHub

Summary

Trying to run the BOM release, the CD workflow is not executed but the Jenkins validation completed correctly.

Reproduction steps

Looking at https://github.com/jenkins-infra/verify-ci-status-action/blob/7d194d0c5785a12623f350581db5243063542f90/run.sh#L37, I cannot get the any 'Jenkins' check on the commit jenkinsci/bom@237e4af. To do that, I ran locally the command gh api -X GET -F status=completed /repos/jenkinsci/bom/commits/237e4af2d8da8da4f0a6969a6a4c007e8d662c60/check-runs | jq -e '.check_runs | .[] | .name'.
However, we can see that the check 'Jenkins' is present on the GitHub UI.

Because of this, the workflow is not performing the release. See https://github.com/jenkinsci/bom/actions/runs/11741103051/job/32709181339

[dduportal]

Looks like we run in the "more than 1000 items":

If there are more than 1000 check suites on a single git reference, this endpoint will limit check runs to the 1000 most recent check suites. To iterate over all possible check runs, use the List check suites for a Git reference endpoint and provide the check_suite_id parameter to the List check runs in a check suite endpoint.

(see https://docs.github.com/en/rest/checks/runs?apiVersion=2022-11-28#list-check-runs-for-a-git-reference)

gh api -X GET -F status=completed /repos/jenkinsci/bom/commits/237e4af2d8da8da4f0a6969a6a4c007e8d662c60/check-runs | jq -e
 '.total_count'
2004

but

gh api -X GET -F status=completed /repos/jenkinsci/bom/commits/237e4af2d8da8da4f0a6969a6a4c007e8d662c60/check-runs | jq -e '.check_runs | length'            
30

[alecharp]

yes exactly, but I guess that's partially why the -F check_name=$NAME is used, so the list is limited. But there is no Jenkins check..

[alecharp]

Because we now have a script to trigger the CD workflow, we should probably get rid of https://github.com/jenkinsci/bom/blob/237e4af2d8da8da4f0a6969a6a4c007e8d662c60/.github/workflows/cd.yaml#L6-L8

[MarkEWaite]

I think that the most recent CD workflow failure is because it detected no "important" changes. We had not labeled the 4 additions as "enhancements" so they were not considered "important" and dependency updates are not considered important by that workflow.

I've updated the labels on those additions so that they are listed as enhancements. That might fix it for this week, but it won't fix it long term, since there are many weeks when we release only with dependency updates.

[alecharp]

I tried by using the button GH but still failing when checking the CI status (https://github.com/jenkinsci/bom/actions/runs/11743295637/job/32715806970).
It's not when considering commits that it's failing but when looking that the Jenkins check on the latest commit on the master branch.

[timja]

I checked all the pages and I didn't find a Jenkins check:

❯ for i in $(seq 1 22); do echo $i && gh api -X GET -F page=$i -F status=completed '/repos/jenkinsci/bom/commits/237e4af2d8da8da4f0a6969a6a4c007e8d662c60/check-runs?per_page=100' | jq '.check_runs[] | select(.conclusion == "success") | .app.slug' | uniq; done
1
"github-actions"
2
"github-actions"
3
"github-actions"
4
"github-actions"
5
"github-actions"
6
"github-actions"
7
"github-actions"
8
"github-actions"
9
"github-actions"
10
"github-actions"
11
"github-actions"
12
"github-actions"
13
"github-actions"
14
"github-actions"
15
"github-actions"
16
"github-actions"
17
"github-actions"
18
"github-actions"
19
"github-actions"
20
"github-actions"
21
"github-actions"
22

[timja]

Possibly this:

Once these check runs exceed 1000, GitHub will start to automatically delete older check runs.

[timja]

We could skip publishing GitHub checks results on the master branch for the JUnit plugin?

[alecharp]

Possibly, but when I trigger the CD workflow, the Jenkins build (and so the check normally) finished 5min before.

[alecharp]

to move forward, either we re-run the check in Jenkins or we find a way to bypass the CI check validation

[timja]

I've started https://ci.jenkins.io/blue/organizations/jenkins/Tools%2Fbom/detail/master/3406/pipeline

[timja]

See if it works otherwise we may want to push an empty commit

[alecharp]

Thanks @timja. Let's see in 2hr if that is better. I won't have access to a computer at that time so if anyone has access to one, please run the release-manager-scripts/bom-run-cd-workflow.sh script when the build has finished..

[timja]

There's 'interesting changes' @MarkEWaite added the label, so it should auto-release without that step needed:
jenkinsci/bom#3886

[MarkEWaite]

If an "interesting change" is enough, then we could decide that the upgrade of the weekly version should always be labeled as an enhancement. That would give us at least one "interesting change" every week.

[alecharp]

it certainly is an interesting change that we have every week.

[timja]

500 error from artifact caching proxy 😭

[2024-11-08T16:41:21.492Z] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project gitlab-plugin: Could not resolve dependencies for project org.jenkins-ci.plugins:gitlab-plugin:hpi:1.8.2

[2024-11-08T16:41:21.492Z] dependency: org.jenkins-ci:trilead-ssh2:jar:build-217-jenkins-274.276.v58da_75159cb_7 (compile)

[2024-11-08T16:41:21.492Z] 	Could not transfer artifact org.jenkins-ci:trilead-ssh2:jar:build-217-jenkins-274.276.v58da_75159cb_7 from/to azure-aks-internal (http://artifact-caching-proxy.artifact-caching-proxy.svc.cluster.local:8080/): status code: 500, reason phrase: Internal Server Error (500)

https://ci.jenkins.io/blue/organizations/jenkins/Tools%2Fbom/detail/master/3406/pipeline

triggered again

[dduportal]

Looks like Artifactory did fail (and Artifact caching proxy transitively) :'( (at least as per the internal logs in datadog).

The build 3408 was successful (at least) and I see the following release published for the BOM: https://github.com/jenkinsci/bom/releases/tag/3654.v237e4a_f2d8da_

[alecharp]

Thanks everyone who help with this release.