GitHub Actions Checks: Submit the checks in batches

[vfonic]

This is needed because GitHub Actions Checks API limits the number of annotations to a maximum of 50 per API request. To create more than 50 annotations, you have to make multiple requests to the Update a check run endpoint.
https://docs.github.com/en/developers/apps/guides/creating-ci-tests-with-the-checks-api#step-24-collecting-rubocop-errors

Closes #6
Action fails with: Error: Invalid request. No more than 50 items are allowed; 404 were supplied.

More people having this issue (just 2 random repos, but there are more):

• Github action fails with "No more than 50 items are allowed" kamilkisiela/graphql-inspector#1840
• Failure and Error thrown when > 50 issues found krizzu/eslint-check-action#1

Their fixes are similar to mine:

• https://github.com/kamilkisiela/graphql-inspector/pull/1854/files#diff-0a970ccd424238bd8c3d97fc7bb2ec8bc75217755a2b49226505d992a576f272
• https://github.com/krizzu/eslint-check-action/pull/3/files#diff-6792c56d44770cf1d5b0125f86f9653f158c86d23ca1227509c47fd75e6a3ff6

Test steps:

1. In any repo where theme-check-action is used (preferably repo with more than 50 theme-check issues), switch to vfonic fork:

name: Theme Check
on: [push]
jobs:
  theme-check:
    name: Theme Check
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Theme Check
-        uses: shopify/theme-check-action@v1
+        uses: vfonic/theme-check-action@batch-octokit-annotations
        with:
          theme_root: '.'
          token: ${{ github.token }}

2. Push the change to some branch and check how the Action runs

I believe, but I didn't properly test to confirm, that I'm making these Checks update requests in parallel. This means that, if there are 2000 errors, this will make 2000 / 50 = 40 requests at the same time.

I tested this if it will timeout or get rate limited, and it doesn't. It does slow down as it needs to make all those requests. It slows down by approx 1s per 100 annotations (it seems to be able to make 2 requests per second(?), but I'm not sure as I'm supposedly making all of these requests at once?).
I ran this action on a repo with 2207 theme-check issues.

Here are my findings:
There's an undocumented limit of 1000 annotations. I searched GH Actions documentation and couldn't find anything about the upper limit, only 50 annotations limit per request.

I tested with updating Checks in parallel and consecutively.

Running in parallel:

await Promise.all(
    annotationsChunks.map(async (annotations) =>
      octokit.checks.update({
        // ...

Running consecutively:

annotationsChunks.forEach(
    async (annotations) =>
      await octokit.checks.update({
        // ...

Results

Parallel	Consecutively
Errors reported	Errors reported
Action run time and number of annotations	Action run time and number of annotations

There seems to be a race condition when doing Checks update. It always reports 1000 annotations. However, it always reports 1000 different annotations.
I tested this for both "parallel job" and "consecutive job" (code posted above).

It only seems that "parallel job" runs some ~10 seconds faster so I kept that code here in the PR.

In my defense for having a theme with 2000+ errors...YOU'RE WELCOME! (for the great testing material! 😅 )

[vfonic]

This used to be await octokit.checks.update({, but now I'm immediately returning the promise and awaiting for all promises to finish with Promise.all.

The above async is needed for the below await exec('theme-check --print....

[charlespwd]

In my defense for having a theme with 2000+ errors...YOU'RE WELCOME! (for the great testing material! 😅)

😂😂😂 THANK YOU! I'll push that and try to look into the limits.

I tested this if it will timeout or get rate limited, and it doesn't. It does slow down as it needs to make all those requests. It slows down by approx 1s per 100 annotations (it seems to be able to make 2 requests per second(?), but I'm not sure as I'm supposedly making all of these requests at once?).

I would hope their SDK handles this internally. Looks like they have a plugin for that, but it doesn't show up in yarn.lock or node_modules.

[charlespwd]

Looks like you are absolutely right about everything.

Added the plugin, didn't hit the limits with 2000 errors. Looks like they have a pretty large window for the limit. Something like X calls / hour.

I'll push it anyway, just in case someone has a lower limit and ends up hitting them. Thanks again for the contribution :D