—env VALIDATE_ALL_CODEBASE=false does not work in docker

[RolfMoleman]

My team and I have identified that —env VALIDATE_ALL_CODEBASE=false does not work in docker since at least version v7.8.0

To Reproduce
Steps to reproduce the behavior:

Using the following docker run command (and other minor variations)

docker run \
          --volume $(pwd):/tmp/lint \
          --env GIT_AUTHORIZATION_BEARER=$(System.AccessToken) \
          --env CI=true \
          --env DISABLE_ERRORS=true \
          --env TF_BUILD=true \
          --env SYSTEM_ACCESSTOKEN=$(System.AccessToken) \
          --env SYSTEM_COLLECTIONURI=$(System.CollectionUri) \
          --env SYSTEM_TEAMPROJECT=$(System.TeamProject) \
          --env BUILD_BUILD_ID=$(Build.BuildId) \
          --env BUILD_REPOSITORY_ID=$(Build.Repository.ID) \
          --env MEGALINTER_CONFIG='/repo_template/.config/.mega-linter.yml' \
          --env VALIDATE_ALL_CODEBASE=false \
          oxsecurity/megalinter:latest

within an azure DevOps pipeline provides what appears to be a python error yet setting VALIDATE_ALL_CODEBASE: false in our github workflows when using the mwhalinter actions it works fine.

Error message:

  File "/megalinter/run.py", line 14, in <module>
    linter.run()
  File "/megalinter/MegaLinter.py", line 203, in run
    self.collect_files()
  File "/megalinter/MegaLinter.py", line 645, in collect_files
    all_files = self.list_files_git_diff()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/megalinter/MegaLinter.py", line 755, in list_files_git_diff
    repo.git.fetch("origin", f"{remote_ref}:{local_ref}")
  File "/usr/local/lib/python3.12/site-packages/GitPython-3.1.42-py3.12.egg/git/cmd.py", line 800, in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/GitPython-3.1.42-py3.12.egg/git/cmd.py", line 1386, in _call_process
    return self.execute(call, **exec_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/GitPython-3.1.42-py3.12.egg/git/cmd.py", line 1183, in execute
    raise GitCommandError(redacted_command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git fetch origin HEAD:refs/remotes/origin/HEAD
  stderr: 'fatal: unable to access 'https://dev.azure.com/<redacted>/': The requested URL returned error: 400'

Expected behavior

Megalinter to perform incremental scans

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[RolfMoleman]

@white-james @DanBerns

For visibility

[echoix]

I see that the formed URL to fetch might not be well written for azure DevOps. But can you make sure that if Azure DevOps has some permission controls to prevent fetching the repo, that the action (inside docker), is allowed to pull/fetch)? Not validating all code base means that we use git to know what files have changed.

[echoix]

@nvuillam do you have experience with Azure DevOps ? I don't really

[RolfMoleman]

@echoix @nvuillam This used to work in v7.6 and I think v7.7 (I can test and confirm) but has been seemingly broken since v7.8.

The permissions in Azure DevOps remain unchanged so it definitely seems to be a sicker image issue. I'll see what else I can find and update here.

[RolfMoleman]

@cmdmescall you might be interested in this

[nvuillam]

I'm currently investigating the issue in an azure repo

[nvuillam]

I succeed to reproduce the problem, that's a start ^^

[echoix]

Did they (azure) change their permissions or some sort?

Is the same bug now reproduced with an older image? (To see if it is the image version or the environment that changes the result)

[nvuillam]

It also crashes with 7.7.0

I wonder if it ever worked with VALIDATE_ALL_CODEBASE=false on Azure

MegaLinter now collects the files to analyse (expand for details)
Listing updated files in [/tmp/lint] using git diff.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/megalinter/run.py", line 14, in <module>
    linter.run()
  File "/megalinter/MegaLinter.py", line 203, in run
    self.collect_files()
  File "/megalinter/MegaLinter.py", line 645, in collect_files
    all_files = self.list_files_git_diff()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/megalinter/MegaLinter.py", line 755, in list_files_git_diff
    repo.git.fetch("origin", f"{remote_ref}:{local_ref}")
  File "/usr/local/lib/python3.11/site-packages/GitPython-3.1.40-py3.11.egg/git/cmd.py", line 736, in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/GitPython-3.1.40-py3.11.egg/git/cmd.py", line 1316, in _call_process
    return self.execute(call, **exec_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/GitPython-3.1.40-py3.11.egg/git/cmd.py", line 1111, in execute
    raise GitCommandError(redacted_command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git fetch origin HEAD:refs/remotes/origin/HEAD
  stderr: 'fatal: unable to access 'https://dev.azure.com/nicolasvuillamy/test-megalinter/_git/test-megalinter/': The requested URL returned error: 400'

for the moment I tried to add an extra step to persist git credentials, but it still triggers a git access error :/

      # Checkout repo
      - checkout: self
        fetchDepth: 0
        persistCredentials: true
        displayName: Git Checkout

[nvuillam]

Found it :)
And it had already been found in #1515 (comment) , we really need to update the documentation :)

Here is a working job, with 2 updates:

• checkout step
• remove --env GIT_AUTHORIZATION_BEARER=$(System.AccessToken) \: token remains set by checkout step

jobs:
  # Run MegaLinter to detect linting and security issues
  - job: MegaLinter
    pool:
      vmImage: ubuntu-latest
    steps:

      # Checkout repo
      - checkout: self
        fetchDepth: 0
        persistCredentials: true
        displayName: Git Checkout

      # Pull MegaLinter docker image
      - script: docker pull oxsecurity/megalinter:v7
        displayName: Pull MegaLinter

      # Run MegaLinter
      - script: |
          docker run -v $(System.DefaultWorkingDirectory):/tmp/lint \
            --env-file <(env | grep -e SYSTEM_ -e BUILD_ -e TF_ -e AGENT_) \
            -e SYSTEM_ACCESSTOKEN=$(System.AccessToken) \
            -e VALIDATE_ALL_CODEBASE=false \
            oxsecurity/megalinter:v7
        displayName: Run MegaLinter

      # Upload MegaLinter reports
      - task: PublishPipelineArtifact@1
        condition: succeededOrFailed()
        displayName: Upload MegaLinter reports
        inputs:
          targetPath: "$(System.DefaultWorkingDirectory)/megalinter-reports/"
          artifactName: MegaLinterReport

Result:

@RolfMoleman please can you confirm it works on your side with such updates ?

[RolfMoleman]

Hi @nvuillam i can confirm removal of

remove --env GIT_AUTHORIZATION_BEARER=$(System.AccessToken) \

Does indeed work. We've got some final tweaks to perform at our end but am happy to supply our job template for use in docs as a more complex Azure devops example demonstrating different run types based on pipeline trigger types and publishing of junit test results and sarif reports to code analysislogs if it would be beneficial?

[nvuillam]

@RolfMoleman I'm glad it now works for you :)

And that would indeed be nice to enhance the default azure template if it handles more business cases ;-) (i love the sarif one ^^)

[white-james]

@nvuillam I seem to be getting a similar error today after the latest update. It was working fine this morning

MegaLinter now collects the files to analyse Listing updated files in [/tmp/lint] using git diff. Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/megalinter/run.py", line 14, in <module> linter.run() File "/megalinter/MegaLinter.py", line 203, in run self.collect_files() File "/megalinter/MegaLinter.py", line 667, in collect_files all_files = self.list_files_git_diff() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/megalinter/MegaLinter.py", line 779, in list_files_git_diff diff = repo.git.diff(f"{default_branch_remote}...", name_only=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/GitPython-3.1.43-py3.12.egg/git/cmd.py", line 986, in <lambda> return lambda *args, **kwargs: self._call_process(name, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/GitPython-3.1.43-py3.12.egg/git/cmd.py", line 1598, in _call_process return self.execute(call, **exec_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/GitPython-3.1.43-py3.12.egg/git/cmd.py", line 1388, in execute raise GitCommandError(redacted_command, status, stderr_value, stdout_value) git.exc.GitCommandError: Cmd('git') failed due to: exit code(128) cmdline: git diff --name-only origin/HEAD... stderr: 'fatal: origin/HEAD...HEAD: no merge base'
rollback to v7.10.0 confirms be working again @RolfMoleman