BUG: checks Signed-Releases and Packaging returning ? when the repo actually has releases on GitHub

[diogoteles08]

Describe the bug
I'm running Scorecard v4.10.2 over projects that have releases on GitHub, but the checks Signed-Releases and Packaging return ? as Score results. As you can see on the image below, on the Packaging result I get "no published package detected" as reason, and for Signed-Releases I get "no releases found".

Reproduction steps
Steps to reproduce the behavior:

1. Run Scorecard over one of the following projects:
   • https://github.com/tensorflow/tensorflow
   • https://github.com/angular/angular
   • https://github.com/lit/lit
2. Check the Packaging and Signed-Releases result

You can evaluate directly those checks with scorecard --repo=tensorflow/tensorflow --checks=Packaging,Signed-Releases

Expected behavior
The evaluation should consider the releases on GitHub and give a punctuation accordingly.

[evverx]

systemd releases unsigned tarballs at https://github.com/systemd/systemd/releases so it should get its well-deserved 0 instead of -1. Tags are signed there though.

[evverx]

I've been thinking about this. Could it be that tarballs generated by GitHub are somehow trusted and ignored because of that? https://wiki.debian.org/Creating%20signed%20GitHub%20releases (which scorecard refers to) expects people to verify them but at least systemd tried that and decided not to go forward with that systemd/systemd#2926. At this point distros caring about those things use signed git tags but realistically most distributions take systemd from its stable repository so it doesn't matter much what this check says because it isn't applicable to how systemd is normally consumed.

[evverx]

More generally since they are generated by GitHub maybe GitHub should sign them itself (ideally once it verifies that tags those tarballs are based on are properly signed). It could also generate provenance :-)

[diogoteles08]

Related to #688

[pnacht]

Glancing at the Scorecard code, it doesn't seem to disconsider the automatically-generated assets created for a release. It may be that the API they use doesn't list such assets.

That's not unreasonable (though understandably unexpected), since those assets don't really exist. They're actually generated on-the-fly via Git every time they're requested. This actually lead to a huge breakage in early 2023 when GitHub upgraded their Git version, which had itself upgraded the library it uses to generate the tarballs.

[evverx]

those assets don't really exist

I think it's a GitHub bug. It should make sure that stuff it automatically generates and distributes via its own platform can't be tampered with and also let consumers verify that. That's the essence of the supply chain security :-)

[raghavkaul]

Related, see discussion here: https://github.com/ossf/scorecard/pull/2186/files#r951620855

[evverx]

see discussion here: https://github.com/ossf/scorecard/pull/2186/files#r951620855

I agree that scorecard should say "inconclusive" in this particular case but since those tarballs are automatically injected into releases they still should be dealt with by GitHub. Or it shouldn't generate them at all and let consumers find things with signatures instead of blindly trusting GitHub with its tarballs with unknown provenance.