Can't download external (private) repositories

[nass-altice]

Hi,

We've got an issue with downloading external terraform modules on private github namespace.

checkov -d .    --quiet   --output github_failed_only   --download-external-modules true    --framework terraform  
Error: -30 14:11:36,9[26](https://github.com/***namespace***/module-tf-cleo-admin/actions/runs/3159383417/jobs/5142549378#step:4:27) [MainThread  ] [ERROR]  failed to get git::https://github.com/***namespace***/module-tf-acm?ref=v4.0.0 because of Cmd('git') failed due to: exit code(128)
  cmdline: git clone -v --depth=1 -b v4.0.0 https://github.com/***namespace***/module-tf-acm /github/workspace/.external_modules/github.com/***namespace***/module-tf-acm/v4.0.0
  stderr: 'Cloning into '/github/workspace/.external_modules/github.com/***namespace***/module-tf-acm/v4.0.0'...
fatal: could not read Username for 'https://github.com': No such device or address
'
Error: -30 14:11:38,546 [MainThread  ] [ERROR]  failed to get git::https://github.com/***namespace***/module-tf-wafv2?ref=v3.0.0 because of Cmd('git') failed due to: exit code(1[28](https://github.com/***namespace***/module-tf-cleo-admin/actions/runs/3159383417/jobs/5142549378#step:4:29))
  cmdline: git clone -v --depth=1 -b v3.0.0 https://github.com/***namespace***/module-tf-wafv2 /github/workspace/.external_modules/github.com/***namespace***/module-tf-wafv2/v3.0.0
  stderr: 'Cloning into '/github/workspace/.external_modules/github.com/***namespace***/module-tf-wafv2/v3.0.0'...
fatal: could not read Username for 'https://github.com/': No such device or address
'

We tried with github_pat variable without success and also tried "git config" hack to replace all https://github.com occurences:

      # Workaround to allow `terraform init` cloning private repositories
      - run: git config --global url."https://oauth2:${PRIVATE_GITHUB_TOKEN}@github.com".insteadOf "https://github.com"
        env:
          PRIVATE_GITHUB_TOKEN: ${{ secrets.PRIVATE_GITHUB_TOKEN }}

Here is our github action configuration for checkov:

  static-analysis:
    runs-on: ubuntu-latest
    name: static-analysis
    steps:
      - name: Checkout repo
        uses: actions/checkout@master

      - name: Run Checkov action
        id: checkov
        uses: bridgecrewio/checkov-action@master
        with:
          quiet: true
          soft_fail: false
          framework: terraform
          output_format: github_failed_only
          download_external_modules: true # download external terraform modules from public git repositories and terraform registry
          github_pat: ${{ secrets.PRIVATE_GITHUB_TOKEN }}
          container_user: 1000

Can you help ?
If you need some details, feel free to ask.

Thanks !

[tronxd]

Hey @nass-altice, can you please try setting the GITHUB_PAT environment variable with your personal access token and try again? It seems that you set a lower case name of the variable

[nass-altice]

hey @tronxd,
Thanks for replying.
I tried with another personal access token without success and, about your second suggestion, i'm just respecting action inputs.
So, you suggest to add same variable as environment vars like this ?

      - name: Run Checkov action
        id: checkov
        uses: bridgecrewio/checkov-action@master
        env:
          GITHUB_PAT: ${{ secrets.PRIVATE_GITHUB_TOKEN }}
        with:
          quiet: true
          soft_fail: false
          framework: terraform
          output_format: github_failed_only
          download_external_modules: true
          github_pat: ${{ secrets.PRIVATE_GITHUB_TOKEN }}
          container_user: 1000

If right, i already tried :/

[tronxd]

Hey @nass-altice , thank you for the clarification.
Can you please share your terraform's module source attribute value so we can try replicate the issue on our end? You can of course omit any sensitive information first from it

[nass-altice]

Yes of course:

module "certificate_eu_west" {
  source = "github.com/***/module-tf-acm?ref=v3.0.0"

  application               = local.application
  env                       = local.env
  domain_name               = local.admin_url
  subject_alternative_names = local.subject_alternative_names
 
  providers = {
    aws      = aws
    aws.root = aws.root
  }
}

[tronxd]

Hi @nass-altice, I have inspected the configuration and we have support in checkov for unprefixed github.com URLs, so assuming your token is valid, this should work.
Can you please try scanning your configuration directly with checkov? (e.g. checkov -d <path_to_conf_dir> --framework terraform) and see if the issue reproduces? I want to understand if the issue is in checkov module loading logic or in the checkov GH action itself.

[nass-altice]

Sorry for the late ...
Yes of course ! When i run that command, it works like it does.
It's only through GH action

[tronxd]

@nass-altice Can you please try adding a run step to your GH action that is similar to your workaround?
- run: git config --global url.https://${PRIVATE_GITHUB_TOKEN}@github.com/.insteadOf https://github.com/

[nass-altice]

Yes of course ! It's what i've done.
It works for terraform in github actions but not for checkov.

[walkerk1980]

Same issue here. We use a GitHub App to get tokens for the private GH repos for all of our external module dependencies and then we use the .gitconfig to insert the tokens which are specific to each module/repo.

This works perfectly for Terraform, but not for Checkov which for some reason doesn't respect the .gitconfig settings.

      - name: Read Required Repos File
        id: get_required_repos
        env:
          REQUIRED_REPOS_FILE_PATH: required_repos.txt
        run: |
          echo "Reading ${REQUIRED_REPOS_FILE_PATH}"
          if [ ! -e "${REQUIRED_REPOS_FILE_PATH}" ]; then echo "${REQUIRED_REPOS_FILE_PATH} not found" && exit 1; fi
          export REQUIRED_REPOSITORIES=$(cat "${REQUIRED_REPOS_FILE_PATH}"| sed 's/\s\+/\n/g' | tr '\n' ',')
          echo "required_repos=$(echo $REQUIRED_REPOSITORIES)" >> $GITHUB_OUTPUT || exit 1
      - name: Generate App tokens
        id: generate_tokens
        uses: OrgNameHere/it_infra_github_app_token@v2
        with:
          app_id: ${{ secrets.IT_INFRA_READ_APP_APPID }}
          private_key: ${{ secrets.IT_INFRA_READ_APP_PRIVATEKEY }}
          # Optional (defaults to ID of the repository's installation).
          # installation_id: 1337
          # Optional (defaults to the current repository).
          repository_names_list:  ${{ steps.get_required_repos.outputs.required_repos }}
      - name: Git Config for Pulling private cross-repo TF modules
        env:
          TF_MODULES_TOKEN_DICT: ${{ steps.generate_tokens.outputs.token_dict }}
        run: |
          for module_name in $(echo "${TF_MODULES_TOKEN_DICT}"|jq -r 'keys[]')
          do
            export THIS_MODULE_TOKEN=$(echo "${TF_MODULES_TOKEN_DICT}"|jq -r .${module_name})
            echo "THIS_MODULE_TOKEN=${THIS_MODULE_TOKEN}"
            git config --global url."https://x-access-token:${THIS_MODULE_TOKEN}@github.com/OrgNameHere/${module_name}.git".insteadOf "https://github.com/OrgNameHere/${module_name}.git"
            # git config --global credential.helper store
          done

Checkov Action always fails like so:

BC_REPOSITORY_URL=https://github.com/OrgNameHere/it_infra_terraform_oii_application_sso
running checkov on directory: examples
checkov -d examples       --output sarif  --soft-fail  --download-external-modules true    --framework terraform   
Error: -[27](https://github.com/OrgNameHere/it_infra_terraform_oii_application_sso/actions/runs/4025510047/jobs/6918819710#step:7:28) 14:45:41,478 [MainThread  ] [ERROR]  failed to get git::https://github.com/OrgNameHere/it_infra_terraform_environment_settings?ref=main because of Cmd('git') failed due to: exit code(1[28](https://github.com/OrgNameHere/it_infra_terraform_oii_application_sso/actions/runs/4025510047/jobs/6918819710#step:7:29))
  cmdline: git clone -v --depth=1 -b main -- https://github.com/OrgNameHere/it_infra_terraform_environment_settings /github/workspace/examples/.external_modules/github.com/OrgNameHere/it_infra_terraform_environment_settings/main
  stderr: 'Cloning into '/github/workspace/examples/.external_modules/github.com/OrgNameHere/it_infra_terraform_environment_settings/main'...
fatal: could not read Username for 'https://github.com/': No such device or address
'

Terraform picks up the insteadOf from the .gitconfig and pulls the modules properly:

Initializing modules...
Downloading git::https://github.com/OrgNameHere/it_infra_terraform_environment_settings.git?ref=main for oii...
- oii in .terraform/modules/oii
- sso_application in ..

Here's what the module source looks like:

module "oii" {
  source      = "github.com/OrgNameHere/it_infra_terraform_environment_settings?ref=main"
  module_name = "oii"
  application = {
    name                = var.application.name
    business_unit       = var.application.business_unit
    project_code        = var.application.project_code
    technical_contact   = var.application.technical_contact
    finance_contact     = var.application.financial_contact
    ad_application_name = var.application.ad_application_name
  }
  location               = "eastus"
  compute_route_table_id = false
}

[bmorrissirromb]

I would really like it if the checkov action could use an installation access token like the output of machine-learning-apps/actions-app-token (https://github.com/machine-learning-apps/actions-app-token). Not sure if that's possible, but would resolve my issues.

[walkerk1980]

So I did some testing locally with checkov and checkov obeys .gitconfig when run locally, just not when run via checkov-action. I then noticed that the action runs in a container and so that's why it doesn't know about .gitconfig and does not respect it.

[gruebel]

Hey everyone, feel free to check my PR which just adds the docs, but it is already fixed with the latest version.

[bmorrissirromb]

Very cool! Any idea when it will be incorporated into the checkov-action for GitHub Actions?