Post Run: Error: Value cannot be null. (Parameter 'target')

[solarmosaic-kflorence]

Describe the bug
Some of our "Post job cleanup" steps are failing when using actions/cache@v2 (https://github.com/actions/cache). Unclear if the issue is related to that action or with the runner.

The error manifests as the following:

Post job cleanup.
Error: Value cannot be null. (Parameter 'target')
Post job cleanup.
Cache hit occurred on the primary key Linux-terraform-, not saving cache.
Post job cleanup.

To Reproduce
Define a composite action like the following:

name: "terraform-apply"
description: "Execute terraform apply against a working directory."
inputs:
  backend-bucket:
    description: "The name of the terraform backend state bucket."
    required: true
  backend-dynamodb-table:
    description: "The name of the DynamoDB table which stores the terraform backend state locks."
    required: true
  backend-kms-key-id:
    description: "The ID of the KMS key which is used to encrypt/decrypt the terraform backend state files."
    required: true
  input-variables-file:
    description: "The filename of a .tfvars file, relative to the working directory."
    required: false
  working-directory:
    description: "The working directory to use when running terraform apply."
    required: true
  workspace:
    default: "${{ github.event.number }}"
    description: "The terraform workspace to use."
    required: true
outputs:
  exitcode:
    description: "The exit code of the call to the terraform binary."
    value: ${{ steps.apply.outputs.exitcode }}
  stderr:
    description: "The STDERR stream of the call to the terraform binary."
    value: ${{ steps.apply.outputs.stderr }}
  stdout:
    description: "The STDOUT stream of the call to the terraform binary."
    value: ${{ steps.apply.outputs.stdout }}
runs:
  using: "composite"
  steps:
    - name: "Create Terraform cache directory"
      env:
        TF_PLUGIN_CACHE_DIR: "${{ github.workspace }}/.terraform.d/plugin-cache"
      run: mkdir --parents "$TF_PLUGIN_CACHE_DIR"
      shell: bash
    - name: "Setup Terraform cache"
      env:
        TF_PLUGIN_CACHE_DIR: "${{ github.workspace }}/.terraform.d/plugin-cache"
      uses: actions/cache@v2
      with:
        path: ${{ env.TF_PLUGIN_CACHE_DIR }}
        key: ${{ runner.os }}-terraform-${{ hashFiles('${{ inputs.working-directory }}/.terraform.lock.hcl') }}
    - uses: hashicorp/setup-terraform@v1
    - run: terraform version
      shell: bash
    - name: "Lint"
      run: terraform fmt -check -diff -recursive
      shell: bash
      working-directory: "${{ inputs.working-directory }}"
    - name: "Apply"
      id: apply
      env:
        TF_IN_AUTOMATION: "true"
        TF_PLUGIN_CACHE_DIR: "${{ github.workspace }}/.terraform.d/plugin-cache"
      working-directory: "${{ inputs.working-directory }}"
      run: |
        terraform init -input=false -no-color \
          -backend-config="bucket=${{ inputs.backend-bucket }}" \
          -backend-config="dynamodb_table=${{ inputs.backend-dynamodb-table }}" \
          -backend-config="key=${{ inputs.working-directory }}.tfstate" \
          -backend-config="kms_key_id=${{ inputs.backend-kms-key-id }}"
        terraform workspace select "${{ inputs.workspace }}" || terraform workspace new "${{ inputs.workspace }}"
        if [[ -n "${{ inputs.input-variables-file }}" ]]; then
          terraform apply -auto-approve -no-color -var-file="${{ inputs.input-variables-file }}"
        else
          terraform apply -auto-approve -no-color
        fi
        terraform output -json
      shell: bash

And then invoke that action in a workflow:

    - name: "Apply management resources"
      id: management
      uses: ./.github/actions/terraform-apply
      with:
        backend-bucket: "${{ env.TF_BACKEND_STATE_BUCKET }}"
        backend-dynamodb-table: "${{ env.TF_DYNAMO_DB_LOCK_TABLE }}"
        backend-kms-key-id: "${{ env.TF_KMS_KEY_ID }}"
        working-directory: "${{ inputs.project-path }}/infra/terraform/management"

This results in the above error during the "Post Run" phase of the job. An example run with this error: https://github.com/solarmosaic-internal/mosaic/runs/4098670986 (note: this is a private organization).

If the following is commented out from the composite action:

#    - name: "Create Terraform cache directory"
#      env:
#        TF_PLUGIN_CACHE_DIR: "${{ github.workspace }}/.terraform.d/plugin-cache"
#      run: mkdir --parents "$TF_PLUGIN_CACHE_DIR"
#      shell: bash
#    - name: "Setup Terraform cache"
#      env:
#        TF_PLUGIN_CACHE_DIR: "${{ github.workspace }}/.terraform.d/plugin-cache"
#      uses: actions/cache@v2
#      with:
#        path: ${{ env.TF_PLUGIN_CACHE_DIR }}
#        key: ${{ runner.os }}-terraform-${{ hashFiles('${{ inputs.working-directory }}/.terraform.lock.hcl') }}

The workflow succeeds without error.

Expected behavior
I expect the "Post run" steps to succeed. Additionally, if they do fail, I would expect a better indication of where the failure is happening and why. It's not clear to me if this is an issue with actions/cache, or the runner, or both.

Runner Version and Platform

Current runner version: '2.284.0'
Operating System
  Ubuntu
  20.04.3
  LTS
Virtual Environment
  Environment: ubuntu-20.04
  Version: 20211101.1
  Included Software: https://github.com/actions/virtual-environments/blob/ubuntu20/20211101.1/images/linux/Ubuntu2004-README.md
  Image Release: https://github.com/actions/virtual-environments/releases/tag/ubuntu20%2F20211101.1
Virtual Environment Provisioner
  1.0.0.0-master-20211022-1
GITHUB_TOKEN Permissions
  Actions: write
  Checks: write
  Contents: write
  Deployments: write
  Discussions: write
  Issues: write
  Metadata: read
  Packages: write
  Pages: write
  PullRequests: write
  RepositoryProjects: write
  SecurityEvents: write
  Statuses: write

What's not working?

Potentially related to #1438

Job Log Output

2021-11-03T22:03:50.0333306Z Post job cleanup.
2021-11-03T22:03:50.0386558Z Post job cleanup.
2021-11-03T22:03:50.0517077Z ##[error]Value cannot be null. (Parameter 'target')
2021-11-03T22:03:50.8698990Z Post job cleanup.
2021-11-03T22:03:51.0061804Z Cache hit occurred on the primary key Linux-terraform-, not saving cache.
2021-11-03T22:03:51.0208763Z Post job cleanup.

Runner and Worker's Diagnostic Logs

Not sure how to access diagnostic logs.

[solarmosaic-kflorence]

Okay, this definitely seems related to actions/cache. If I comment out the following our "Post Run" passes:

- name: "Create Terraform cache directory"
  env:
    TF_PLUGIN_CACHE_DIR: "${{ github.workspace }}/.terraform.d/plugin-cache"
  run: mkdir --parents "$TF_PLUGIN_CACHE_DIR"
  shell: bash
- name: "Setup Terraform cache"
  env:
    TF_PLUGIN_CACHE_DIR: "${{ github.workspace }}/.terraform.d/plugin-cache"
  uses: actions/cache@v2
  with:
    path: ${{ env.TF_PLUGIN_CACHE_DIR }}
    key: ${{ runner.os }}-terraform-${{ hashFiles('${{ inputs.working-directory }}/.terraform.lock.hcl') }}

Strangely, there don't seem to be any changes to this action for awhile and we only started seeing these failures recently. I'm not sure if the problem is with that action specifically or something related to the runner.

[TingluoHuang]

@fhammerl related to any of the recent composite action changes we did?

[solarmosaic-kflorence]

Perhaps it is from

runner/src/Runner.Worker/Handlers/NodeScriptActionHandler.cs

Line 91 in 05b8429

ArgUtil.NotNullOrEmpty(target, nameof(target));

[fhammerl]

@TingluoHuang it could be related to either #1433 or #1438.

@solarmosaic-kflorence did this workflow ever fail on the previous runner version (2.283.3) or did it ever run successfully on the current one?

[solarmosaic-kflorence]

@fhammerl I first noticed this failure yesterday in our monorepo. Previously without any changes the same workflow configuration worked as of 5 days ago (that is the last run I saw, anyways) on runner version 2.283.3.

It has never completed successfully on 2.284.0 without the workaround above.

[fhammerl]

@solarmosaic-kflorence Thanks for confirming it, we'll be tracking this issue as a regression.

As an alternative workaround, perhaps you could try invoking is as a global action?

    - name: "Apply management resources"
      id: management
      uses: owner/repo/.github/actions/terraform-apply@main # <--
      with:
...