coveralls.io HTTP 422 "Unprocessable Entity for url: https://coveralls.io/api/v1/jobs"

[andy-maier]

The HTTP 422 issue has already pretty wide coverage on stackoverflow, this issue tracker, and in the issue trackers of some of the clients, such as https://github.com/TheKevJames/coveralls-python.

Since a few days, it reappeared out of nowhere, on all of our repos on Github. They all use Github Actions. They all were able to deliver coverage results successfully to coveralls.io before.

After trying with a few of our repos, I went to a rather small Python project to reproduce it.

On Python 3.11 with coveralls 3.3.1, the Github Actions log shows this (From https://github.com/pywbem/nocasedict/actions/runs/4843078401/jobs/8630438981):

  coveralls
  shell: /usr/bin/bash -l {0}
  env:
    PIP_DISABLE_PIP_VERSION_CHECK: 1
    PIP_NO_PYTHON_VERSION_WARNING: 1
    pythonLocation: /opt/hostedtoolcache/Python/3.11.3/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.11.3/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.3/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.3/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.3/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.11.3/x64/lib
    GITHUB_TOKEN: ***
    COVERALLS_PARALLEL: true
    COVERALLS_FLAG_NAME: ubuntu-latest,3.11,latest
Submitting coverage to coveralls.io...
Could not submit coverage: 422 Client Error: Unprocessable Entity for url: https://coveralls.io/api/v1/jobs
resubmitting with id None
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/coveralls/api.py", line 290, in submit_report
    response.raise_for_status()
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://coveralls.io/api/v1/jobs

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/coveralls/cli.py", line 95, in main
    result = coverallz.wear()
             ^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/coveralls/api.py", line 257, in wear
    return self.submit_report(json_string)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.3/x64/lib/python3.11/site-packages/coveralls/api.py", line 293, in submit_report
    raise CoverallsException(
coveralls.exception.CoverallsException: Could not submit coverage: 422 Client Error: Unprocessable Entity for url: https://coveralls.io/api/v1/jobs

Note that this is a new message for "Unprocessable Entity", not the one described in https://docs.coveralls.io/common-issues-and-troubleshooting#issue-422-github-actions

When running coveralls in debug mode, it shows JSON output. Not sure whether this is directly what is sent to coveralls.io, but here it is:

{
    "source_files":[ ... 6 files ... ],
    "git": {
        "branch": "andy/fix-coveralls",
        "head": {
            "id": "a72f84b06050986a19a29e8e8c2224e3e39cece4",
            "author_name": "Andreas Maier",
            "author_email": "[email protected]",
            "committer_name": "GitHub",
            "committer_email": "[email protected]",
            "message": "Merge d00f3e9ba645c3e9279cb2577a1f462306aa5547 into 0d22532d98db3e48b072b6f82505ddc46e28000f"
        },
        "remotes": [
            {
                "name": "origin",
                "url": "https: //github.com/pywbem/nocasedict"
            }
        ]
    },
    "service_job_id": null,
    "repo_token": "[secure]",
    "service_name": "github-actions",
    "service_number": "4843120923",
    "service_pull_request": "150",
    "parallel": true,
    "flag_name": "ubuntu-latest,3.11,latest",
    "config_file": ".coveragerc",
    "base_dir": "",
    "src_dir": ""
}

The run for that is https://github.com/pywbem/nocasedict/actions/runs/4843120923/jobs/8630508203

Note that service_name is "github-actions", which is what coveralls-python uses by default when it detects to run on Github Actions.

I retried with the --service=github option (which is a workaround recommended in some places) but this did not change the result - the 422 result happened as before (see run https://github.com/pywbem/nocasedict/actions/runs/4843145025/jobs/8630548586). On another project where I tried that, I could verify that this causes "service_name": "github" to be set in debug mode.

Do you think it is possible to improve the error message that is returned with the 422 HTTP response so that the client programs have a chance to understand what the issue was?

Can you figure out from the coveralls debug output shown above what is possibly wrong with the input data sent to coveralls.io?

Any other ideas on why this issue suddenly reappeared?

BTW, we are not the only ones where this reappeared. There is at least one other report: TheKevJames/coveralls-python#252 (comment)

On another project (https://github.com/zhmcclient/python-zhmcclient), I tried another recommendation I found: To remove the repo from coveralls.io and to add it again. That did not fix it, either.

[andy-maier]

I found the following in https://docs.coveralls.io/api-jobs-endpoint :

Name	Description	Type
service_name (required)	The CI service or other environment in which the test suite was run. Can be anything, but certain values may trigger automatic support by certain Coveralls Integrations.	String
service_number	The build number. Will default to incrementing integers, applied chronologically to the builds on a repo.	String
service_job_id (required)	A unique identifier for the job, assigned by the service specified in "service_name".	String
service_job_number	The job number. Will default to incrementing integers based on the jobs in a build.	String

The descriptions of service_job_id and service_job_number suggest that service_job_id should be the unique job identifier created by Github Actions in the GITHUB_RUN_ID env var.

However, when looking at the source code of coveralls-python here, it sets service_number to the GITHUB_RUN_ID, and sets service_job_id to None/null. Yet, this has worked up until recently. Also, apparently the change of service_name between "github-actions" and "github" has helped people to resolve the problem in the past (but some needed to use "github-actions" and some others "github", with the reasons for that apparently not understood so far).

So my questions are:

• Is the API documentation (and my interpretation of it) correct, i.e. that service_job_id should be the unique run ID?
• If so, why did the approach coveralls-python took work at all in the past (and coveralls.io still shows the results under the GITHUB_RUN_ID values)?
• Is there any special processing happening in coveralls.io for the service names "github-actions" or "github"?
• What is the difference between the parameters service_number and service_job_number?

[andy-maier]

I experimented with the service_job_id, service_name, etc. parameters of the coveralls-io API using the https://github.com/pywbem/nocasedict Python project, by using coverall-python's ability to override these parameters from the environment, and here are the results:

Run	service_name	service_number	service_job_id	result
test 389	github	1001-num	1001-id	success
test 390	github	1002-num	null	HTTP 422
test 392	github-actions	1004-num	1004-id	HTTP 422
test 393	github-actions	1005-num	null	HTTP 422

In all cases with HTTP status 422, the HTTP message was "Unprocessable Entity for url: https://coveralls.io/api/v1/jobs"

Due to the env var based approach for overriding the default parameter values coveralls-python generates , I could not test the case where service_number is passed as null, or any case where a parameter is omitted from the request instead of being present with a value of null. But even with this limited set of tests, it seems that it is important to provide the service_job_id parameter and to use service_name=github.

[afinetooth]

@andy-maier thank you for the thorough rundown of your issue and your tests.

tl;dr

We think a recent change on our side may have caused this issue for users of coveralls-python and are looking for a fix, even though, at this point, the fix is not obvious to us.

Really glad you found a workaround, and I'm sure that will help other users in the same situation until we can resolve this, so 🙏 .

I will update this card as soon as we've made some meaningful progress and will also attempt to answer as many of your questions, above, as I can.

Details

Perhaps to clear some fog on the major disconnects here, I'll share this for now:

We're aware that a quirk in our implementation of our Github Action probably forced the workaround in coveralls-python that resets (or unsets) service_job_id. It's complicated, but comes from two details of our implementation:

1. Our Github Action claims github for service_name - and, where that value is present, it informs the Coveralls API to treat the incoming job a little differently than other jobs, due to the fact that, for our action, we use the Github Action's token (secrets.GITHUB_TOKEN) to identify the repo on our end. With other integrations, those are identified by the Coveralls Repo Token, typically passed as the value for the repo_token parameter in the incoming JSON. (In jobs processed by our Github Action, we are instead grabbing the github-token input param for this purpose.)
2. The way we use service_job_id for jobs coming through our Github Action - As you surmised from our docs here, as it pertains to parallel jobs, we generally use service_number as an identifier for the build (the parent of several jobs), and service_job_id as the identifier for each job (as one of several children of a build). But we encountered a problem implementing this model in our Github Action when we couldn't find two obvious Github Actions env vars to use to map that parent-child relationship. As a result, we settled on using env.GITHUB_RUN_ID as the closest thing to a unique identifier that we could use for a build. But with the lack of a similar identifier for a job, we chose to leverage service_job_id for both purposes. For each incoming job, we let it identify itself with service_job_id, but then, upon creating the build related to those jobs, we borrow that value and use it for the build's identifier (service_number), and then swap the incoming job identifier for an incrementing integer, such that jobs in parallel builds get named: {env.GITHUB_RUN_ID}.1, {env.GITHUB_RUN_ID.2}, {env.GITHUB_RUN_ID}.3, etc. Again, we only operate this way on jobs with the incoming parameter, service_name: github.

[afinetooth]

@andy-maier we have applied a fix for your issue and would like to know if it has resolved it.

I understand you have applied a workaround on your end, so barring the status of the original issue, please let us know if your workaround has been affected at all.

[dgw]

I just reran a previously erroring PR workflow and it passed. Hopefully @andy-maier and everyone else affected by this will find the same positive result upon retrying affected workflows!

[afinetooth]

@dgw great! Thanks for the update. Hope so too.

[anamileva]

Same here, previously failing workflow is now passing again. Thank you for the fix!

[afinetooth]

Excellent. Thanks @anamileva.

[Glusk]

I'm getting a very similar error when trying to push coverage report to Coveralls from my Travis build:

[ERROR] Failed to execute goal org.eluder.coveralls:coveralls-maven-plugin:4.3.0:report (default-cli) on project srp6-variables: Processing of input or output data failed: Report submission to Coveralls API failed with HTTP status 422: Unprocessable Entity (The commit  wasn't found on the repo anycable/anycableThe commit  wasn't found on the repo anycable/anycable) -> [Help 1]

I haven't changed any CI/CD configuration settings. The exact same configuration has worked for me as early as April 25 2023.

[woodsp-ibm]

CI in repos mentioned here #1708 (comment) are now operational again - many thx!

[afinetooth]

Thanks, @woodsp-ibm!

Hi, @Glusk. I think this is a different issue. I have started a new issue for it here:
#1711

Please come join me there.

[andy-maier]

@afinetooth Hello James,
first of all, thanks much for the detailed response and overall help!

I did not try without the circumvention (i.e. setting the COVERALLS_SERVICE_JOB_ID and COVERALLS_SERVICE_NUMBER env vars, and using Python coveralls 3.3.1) yet.

In the past few days since I opened this issue, that circumvention worked fine. However, today the coveralls.io submissions started failing again with HTTP 422 "Unprocessable Entity" without any change on our end. Actions run: https://github.com/zhmcclient/python-zhmcclient/actions/runs/4890476122/jobs/8730913695?pr=1146
I have not yet investigated it further.

Update: The new 422 must have been a temporary condition. It runs fine again, with the circumvention: https://github.com/zhmcclient/python-zhmcclient/actions/runs/4892178353/jobs/8733627978 and also when reducing the circumvention to not set COVERALLS_SERVICE_JOB_ID: https://github.com/zhmcclient/python-zhmcclient/actions/runs/4892585029/jobs/8734515698

[andy-maier]

I guess when ignoring the temporary new 422 glitch mentioned above, the bottom line is that right now it works fine both with and without specifying COVERALLS_SERVICE_JOB_ID. (We still specify COVERALLS_SERVICE_NUMBER to have a more user friendly naming for the builds/jobs on coveralls.io, and that works, too).

Should we pursue PR TheKevJames/coveralls-python#385 which provides the "service_job_id" parameter when submitting a parallel job to coveralls.io?

[afinetooth]

Hi @andy-maier thanks for the update. I'm afraid I can't account for the temporary 422, but I'm glad to hear its working now, and in particular, I'm happy you found a good approach to control the naming of your builds and jobs that suits you; it's one of the advantages of learning about the params accepted by the /jobs endpoint that you can get some more control over that aspect of your integration.

Re: TheKevJames/coveralls-python#385, while I don't feel comfortable weighing in on whether the project should accept the PR, since it's not a project we maintain, I do want to clarify some comments made by my colleague, @mrexox, there on the PR, so you know where we're coming from on those and can make further decisions about your ultimate changes:

1. Re: github vs github-actions for service_name - Long story short(ish), we "claimed" the value github, for service_name, for our (Coveralls) Github Action, to let the Coveralls API know when a job is coming through that integration (one of our official integrations), and we did that because it's the only integration where we don't identify an incoming job's repo by its Coveralls Repo Token, and instead use the Github Action token (${{ secrets.GITHUB_TOKEN }}) to identify the repo. What it means is that it matters whether you choose github or github-actions for your service_name parameter. When you want to identify the repo at Coveralls by its GITHUB_TOKEN, you should choose github, and when you want to identify it by its Coveralls Repo Token, you should choose github-action.
2. Re: using GITHUB_JOB vs GITHUB_RUN_ID for service_job_id - Previous to now we have always used ${{ env.GITHUB_RUN_ID }} for service_job_id because it's a guaranteed unique value we can leverage for both service_job_id (Job identifier), and service_number (our Build identifier)—both strings, BTW; but, of late, we are realizing that we can probably use GITHUB_JOB for service_job_id along with GITHUB_RUN_ID for service_number, and be more prescriptive, and sensible, about how to set those two identifiers and get predictable outcomes at Coveralls. On this point, I would probably hold off until we've completed our work and have ensured it's backwards compatible; or else stay in touch with @mrexox if you want to bring that to coveralls-python as soon as its very safe to do so. (I suspect we'll even rename those—while continuing to support the old param names—so it's even more obvious and easy to reason about how to use them.)

@mrexox anything to add?

[mrexox]

I can confirm everything above! The only thing to add - when using github as a service_name value you can provide both ${{ secrets.GITHUB_TOKEN }} or Coveralls Repo Token. When using github-actions you can only use Coveralls Repo Token. That's why github is a better option for service_name.

[sahiljhawar]

I am still facing this issue. My GitHub CI looks like this. This works fine for Python 3.10 but fails in Python 3.9 and 3.11:

- name: Run Coveralls
      if: ${{ success() }}
      run: |
        coveralls --service=github
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

[afinetooth]

@sahiljhawar Please see this successful configuration shared above by user @andy-maier. It may provide suggestions for additional tweaks of your current config.