Inconsistent cypress runs between github action and cypress dashboard

[MaximeHeckel]

Hi 👋

I've recently updated my cypress github workflow to run only on pull_requests against the master branch instead of push events and I'm seeing a weird side effect when opening a new Pull Requests and re-running the cypress job:

• Cypress will run the test suite of the master branch (not the head branch that is being submitted as a PR to merge on top of master). On the github status check for the cypress dashboard, I can see it's called default-group (merge) and the title of the run contains the commit hash of my head branch and the HEAD commit hash of the master branch we're merging into.

• If I rerun the same job, Cypress will run the test suite of the head branch (as expected). On the github status check for the cypress dashboard, I can see it's simply called default-group and the title of the run

It looks like Cypress Dashboard is somehow capturing 2 different events on the same github ci run (from the same workflow file). I'm pretty convinced that my update to run cypress only on pull_request events triggered.

Am I doing something wrong?

Here's a screenshot of the pull requests checks:

Here's my Github Workflow file:

name: Run End-to-End Test Suite

on:
  pull_request:
    branches:
      - master

jobs:
  build-and-e2e-test:
    runs-on: ubuntu-16.04
    strategy:
      matrix:
        node-version: [11.x]
    steps:
      - name: Checkout Commit
        uses: actions/checkout@v1
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v1
        with:
          node-version: ${{ matrix.node-version }}
      - name: Cache NPM Dependencies
        id: cache-dependencies
        uses: actions/cache@v1
        with:
          path: node_modules
          key: node-modules-${{ runner.OS }}-npm-cache-${{ hashFiles('**/yarn.lock') }}
          restore-keys: |
            node-modules-${{ runner.OS }}-npm-cache
      - name: Cache Cypress Binaries
        id: cache-cypress-binaries
        uses: actions/cache@v1
        with:
          path: ~/.cache/Cypress
          key: cypress-binaries-${{ runner.OS }}-npm-cache-${{ hashFiles('**/yarn.lock') }}
          restore-keys: |
            cypress-binaries-${{ runner.OS }}-npm-cache
      - name: Install Dependencies
        if: steps.cache-dependencies.outputs.cache-hit != 'true' || steps.cache-cypress-binaries.outputs.cache-hit != 'true'
        run: |
          yarn install --force --non-interactive
      - name: Build Application
        run: yarn build
        env:
          CI: true
      - name: Run E2E Tests without recording (if not on master branch)
        uses: cypress-io/github-action@v1
        with:
          browser: chrome
          record: true
          headless: true
          start: yarn serve-built-files
          wait-on: "http://localhost:8081"
          config: defaultCommandTimeout=100000,pageLoadTimeout=100000,watchForFileChanges=false,video=false
        env:
          NODE_ENV: "test"
          CI: true
          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}

[jennifer-shehane]

GitHub status checks will trigger differently for CI providers that utilize "Pull Requests builds". Pull request builds are generated by the CI provider (if configured) by merging the head branch with the base branch specified in the pull request (in your case - master). These are helpful in answering the question “What if this PR merged into target (base) branch”. You'll often see the commit message for the associated build look something like "Merge 2f5a7afe97740f99f5c4f5e40a3d7347bc058b30 into 1f40d78961e5d268ac072dca86176f9fb2aa7b13".

These types of builds trigger Cypress's GitHub status checks notated with the note (merge) on the status check.

If your build also runs the action against its current branch (without the generated merge 'pull request build') you would see 2 events triggered from Cypress since there are 2 actions being run.

I'm a bit unsure about whether you are saying this happens on every single PR or whether this only triggers 2 events when you 'rerun' your action. Could you clarify?

[MaximeHeckel]

Thank you for your answer!

I'm a bit unsure about whether you are saying this happens on every single PR or whether this only triggers 2 events when you 'rerun' your action. Could you clarify?

I get the 1st merge event when I open the pull request and the 2nd event only when I rerun the PR. I'd have expected it to rerun as the 1st event.

[gotbahn]

@jennifer-shehane is it possible to specify custom message, or point Cypress to proper commit sha? As I understand by default it takes github.sha, however github.event.after would serve better

Now our dashboard doesn't look good

[jennifer-shehane]

@godban We use these environment variables to display the information in the Dashboard https://on.cypress.io/continuous-integration#Git-information So, if you have some way to overwrite this when GitHub auto creates a merge commit - then you could pass that (I'm not sure if this is possible).

[gotbahn]

@jennifer-shehane no, this will lead to unexpected behaviour even if possible.

[bahmutov]

I have looked at what is going on by opening bahmutov/cypress-gh-action-example#14

The workflow is running both on push (commit) and on pull_request events

name: merge
# run this action on Push and on Pull Request
# to demonstrate the difference in commit SHA
on: [push, pull_request]

You can see the Dashboard at https://dashboard.cypress.io/projects/yz8qku and filter by tags since I pass the event name as a tag

- name: Cypress run on ${{ github.event_name }}
  uses: cypress-io/github-action@v2
  with:
    record: true
    group: merge
    tag: ${{ github.event_name }}
  env:
    CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}

Every commit with open pull request thus triggers 2 events = 2 action runs

For example here in GH Actions we see merge #3 which is a push event with the current commit title "print", and merge #4 which is the pull request that uses the original PR title "add pr workflow".

Here is what our Dashboard shows in this case

Build 164 is the recording of the "push" event, has the expected title from the commit "print" and is ok.

Build 162 is the recording of the "pull_request" event, and it uses the default commit title created by GitHub (merge commit A into B), and this is what we would like to change.

When we look closely at the build 162, its commit also is not as nice as it could be

which leads to the merge commit and not to the pull request

Solution

To me the largest problem is the commit message for pull request. Thus a good solution is to set it for pull requests as an environment variable to overwrite the Git information. You can find the file at https://github.com/bahmutov/cypress-gh-action-example/blob/master/.github/workflows/merge.yml and here is the relevant portion

- name: Cypress run on ${{ github.event_name }}
  uses: cypress-io/github-action@v2
  with:
    record: true
    group: merge
    # tag will be either "push" or "pull_request"
    tag: ${{ github.event_name }}
  env:
    CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
    # - if the event is push, the title will be undefined
    # and Cypress will get the commit message from Git information
    # - if the event is pull_request, then we set the commit
    # message to the pull request title
    COMMIT_INFO_MESSAGE: ${{ github.event.pull_request.title }}

The Dashboard shows then:

"push" event shows the commit's subject line.
"pull_request" event shows the PR title.

[gotbahn]

@bahmutov alright, it works. Thank you!

[eplatek-safyr]

@bahmutov Thanks for this solution that fixes the dashboard labels, but it breaks the PR comments on github. After applying this solution to my project I don't have cypress bot comments on my PR, after removing the COMMIT_INFO_MESSAGE variable the comments are back.

Any idea on how to keep both of those features working ?

[HartiganHM]

Checking to see if there have been any updates on this issue as I experienced the same thing as @eliasplatekdkt. The solution @bahmutov provided worked for the title but broke all Cypress status updates to the PR in question.

It would be great to configure this option with more nuance, somehow. The current title displayed from the screenshots already posted is not only overly verbose but somewhat unhelpful as to the context of the run.

[conversayShawn]

@eplatek-safyr Try adding COMMIT_INFO_SHA: ${{ github.event.pull_request.head.sha }} as an env var.

Depending on your setup, it could look something like this:

env:
 CYPRESS_PROJECT_ID: ${{ secrets.CYPRESS_PROJECT_ID }}
 CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
 GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 COMMIT_INFO_MESSAGE: ${{ github.event.pull_request.title }}
 COMMIT_INFO_SHA: ${{ github.event.pull_request.head.sha }}

[mellis481]

@bahmutov's solution above (which does successfully update the commit message) not only breaks Cypress bot comments as @eplatek-safyr points out, but it also prevented status checks that I had configured in my parallelized workflows from ever running.

The solution above of adding COMMIT_INFO_SHA: ${{ github.event.pull_request.head.sha }} appears to correct these issues for PR workflows. @conversaShawn: Is there only a need to set COMMIT_INFO_SHA for PR workflows? I have a workflow that uses the workflow_dispatch trigger to do manual deployments which I don't think I would need to add it to after setting a custom COMMIT_INFO_MESSAGE, but want to confirm.

[conversayShawn]

@mellis481 I am currently investigating

[conversayShawn]

@mellis481 After some testing, COMMIT_INFO_MESSAGE is only necessary to obtain the message in the Dashboard when using pull_request as part of the workflow. The COMMIT_INFO_SHA seems to be necessary when using pull_request or workflow_dispatch workflows to definitively determine the run has ended.

[mellis481]

@conversaShawn I'm confused why you say COMMIT_INFO_SHA is needed for workflows that use the workflow_dispatch trigger since my manually-initiated workflows are currently working without it. Plus what value would you use for it?