Go: Basic Go 1.21 support

[mbg]

Go 1.21 was released recently. This PR updates our CI to use Go 1.21 and makes the Go extractor aware of Go 1.21. It also adds support for the new built-in functions: min, max, and clear.

[marcusthelin]

Status on this? :)

[mbg]

@marcusthelin our existing CodeQL release should already be compatible with Go 1.21 and you should be able to build Go 1.21 projects with it. This PR is mostly about modelling standard library change, updating our CI, etc.

Is there anything in particular that doesn't work for you with Go 1.21?

[marcusthelin]

@marcusthelin our existing CodeQL release should already be compatible with Go 1.21 and you should be able to build Go 1.21 projects with it. This PR is mostly about modelling standard library change, updating our CI, etc.

Is there anything in particular that doesn't work for you with Go 1.21?

@mbg we keep getting the below error. Our go.mod file has version 1.21.

Our workflow:

name: CodeQL

on:
  workflow_call:
    inputs:
      working-directory:
        type: string
        required: true
      languages:
        type: string
        default: go
    secrets:
      githubToken:
        required: true

jobs:
  analyze:
    name: Analyze
    runs-on: ubuntu-latest
    timeout-minutes: 360
    permissions:
      actions: read
      contents: read
      security-events: write

    steps:
    - name: Checkout repository
      uses: actions/checkout@v3

    # Initializes the CodeQL tools for scanning.
    - name: Initialize CodeQL
      uses: github/codeql-action/init@v2
      with:
        languages: ${{ inputs.languages }}
        # If you wish to specify custom queries, you can do so here or in a config file.
        # By default, queries listed here will override any specified in a config file.
        # Prefix the list here with "+" to use these queries and those in the config file.

        # For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
        # queries: security-extended,security-and-quality

    - name: Autobuild
      uses: github/codeql-action/autobuild@v2
      with:
        working-directory: ${{ inputs.working-directory }}

    - name: Perform CodeQL Analysis
      uses: github/codeql-action/analyze@v2
      with:
        category: "/${{ inputs.working-directory }}"

[mbg]

Ah thanks for sharing that extra information @marcusthelin. The problem you run into is that the GitHub Actions runner image has Go 1.20 installed by default and so CodeQL gives you an error to tell you that there is a mismatch between the Go version in the environment and the one you expect for your project.

To remedy this, add a step to your workflow that uses the setup-go action like so:

Updated workflow

name: CodeQL

on:
  workflow_call:
    inputs:
      working-directory:
        type: string
        required: true
      languages:
        type: string
        default: go
    secrets:
      githubToken:
        required: true

jobs:
  analyze:
    name: Analyze
    runs-on: ubuntu-latest
    timeout-minutes: 360
    permissions:
      actions: read
      contents: read
      security-events: write

    steps:
    - name: Checkout repository
      uses: actions/checkout@v3

    - name: Install Go
      uses: actions/setup-go@v4
      with:
        go-version-file: go.mod # adjust this path if needed

    # Initializes the CodeQL tools for scanning.
    - name: Initialize CodeQL
      uses: github/codeql-action/init@v2
      with:
        languages: ${{ inputs.languages }}
        # If you wish to specify custom queries, you can do so here or in a config file.
        # By default, queries listed here will override any specified in a config file.
        # Prefix the list here with "+" to use these queries and those in the config file.

        # For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
        # queries: security-extended,security-and-quality

    - name: Autobuild
      uses: github/codeql-action/autobuild@v2
      with:
        working-directory: ${{ inputs.working-directory }}

    - name: Perform CodeQL Analysis
      uses: github/codeql-action/analyze@v2
      with:
        category: "/${{ inputs.working-directory }}"

[marcusthelin]

Ah thanks for sharing that extra information @marcusthelin. The problem you run into is that the GitHub Actions runner image has Go 1.20 installed by default and so CodeQL gives you an error to tell you that there is a mismatch between the Go version in the environment and the one you expect for your project.

To remedy this, add a step to your workflow that uses the setup-go action like so:

Updated workflow

name: CodeQL

on:
  workflow_call:
    inputs:
      working-directory:
        type: string
        required: true
      languages:
        type: string
        default: go
    secrets:
      githubToken:
        required: true

jobs:
  analyze:
    name: Analyze
    runs-on: ubuntu-latest
    timeout-minutes: 360
    permissions:
      actions: read
      contents: read
      security-events: write

    steps:
    - name: Checkout repository
      uses: actions/checkout@v3

    - name: Install Go
      uses: actions/setup-go@v4
      with:
        go-version-file: go.mod # adjust this path if needed

    # Initializes the CodeQL tools for scanning.
    - name: Initialize CodeQL
      uses: github/codeql-action/init@v2
      with:
        languages: ${{ inputs.languages }}
        # If you wish to specify custom queries, you can do so here or in a config file.
        # By default, queries listed here will override any specified in a config file.
        # Prefix the list here with "+" to use these queries and those in the config file.

        # For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
        # queries: security-extended,security-and-quality

    - name: Autobuild
      uses: github/codeql-action/autobuild@v2
      with:
        working-directory: ${{ inputs.working-directory }}

    - name: Perform CodeQL Analysis
      uses: github/codeql-action/analyze@v2
      with:
        category: "/${{ inputs.working-directory }}"

@mbg I've tried this now and I still see the same error. Is the CodeQL action overriding the Go version?

[mbg]

@marcusthelin I have just put together a small test repo to validate this for myself with a slightly simplified version of your workflow. It seems to work fine for me: https://github.com/mbg/go-test/actions/runs/5867141929/job/15907302196

Could you verify that the output of the "Install Go" step in your workflow is able to find the go.mod file and that it installs Go 1.21 as expected / seen in my workflow run: https://github.com/mbg/go-test/actions/runs/5867141929/job/15907302196#step:3:8

If not, you may need to adjust the argument for go-version-file. Perhaps something like this would work better in your workflow since you seem to have a workflow input for the working directory which is used to set the working directory for other steps:

      with:
        go-version-file: ${{ inputs.working-directory }}/go.mod

[marcusthelin]

@mbg I see the log in autobuilder:

Autobuilder was built with go1.20.5, environment has go1.21.0

I also see errors:

  Error: 2023/08/15 13:49:18   /opt/hostedtoolcache/go/1.21.0/x64/src/slices/sort.go:67:7: undefined: min
  Error: 2023/08/15 13:49:18   /opt/hostedtoolcache/go/1.21.0/x64/src/slices/sort.go:97:7: undefined: max

which indicates that it does not in fact run 1.21, since those are new functions in 1.21. What could be the cause?

[mbg]

I also see errors:

  Error: 2023/08/15 13:49:18   /opt/hostedtoolcache/go/1.21.0/x64/src/slices/sort.go:67:7: undefined: min
  Error: 2023/08/15 13:49:18   /opt/hostedtoolcache/go/1.21.0/x64/src/slices/sort.go:97:7: undefined: max

which indicates that it does not in fact run 1.21, since those are new functions in 1.21. What could be the cause?

Thank you for posting these error messages. In short, there are multiple versions of Go at play: the version of the Go toolchain and standard library that is installed on the runner (which is indeed 1.21 now as reported), and the version of the Go language libraries that the CodeQL Go extractor is built against / the version of the Go compiler that the extractor is built with (which is still for 1.20 in the latest CodeQL release).

This is largely expected, however, until we support the new built-ins. Once both #13923 and this PR are merged and a new CodeQL CLI release is out with those changes, those errors should go away.

In the meantime, I would expect your CodeQL analysis to still be working successfully as before (the errors in the log shouldn't stop it from proceeding). The only limitation here is that data flow / taint flow through the new built-ins isn't modelled yet until the updated models are released.

[marcusthelin]

@mbg Ah, okay! So just because I see an error under the "Security" tab (according to the screenshot I've posted previously), the scans should still work.

Thank you for helping me out on this 🙏🏼

[mbg]

Yes, sorry about that error. I can see that we are reporting an error-level diagnostic in this particular scenario that's probably not helpful. I will raise an internal issue about this to see if we can improve the reporting here. Go 1.21 is the first new Go release since we introduced the tool status page, so this is the first time we have run into the particular scenario where the system version of Go is newer than the one our tooling is built with! Thank you for your feedback here!

[smowton]

Looks good to me!

Approved by other member of the team