diff --git a/.github/instructions/gh-aw-workflows.instructions.md b/.github/instructions/gh-aw-workflows.instructions.md deleted file mode 100644 index 2230e8c86492..000000000000 --- a/.github/instructions/gh-aw-workflows.instructions.md +++ /dev/null @@ -1,290 +0,0 @@ ---- -applyTo: - - ".github/workflows/*.md" - - ".github/workflows/*.lock.yml" ---- - -# gh-aw (GitHub Agentic Workflows) Guidelines - -## ๐Ÿšจ Before You Build: Prefer Built-in gh-aw Features - -**CRITICAL RULE:** Before implementing any trigger, output, scheduling, or interaction mechanism in a gh-aw workflow, check whether gh-aw has a built-in feature that does it. gh-aw extends GitHub Actions with many convenience features โ€” manually reimplementing them is always worse (more code, more bugs, missing platform integration like emoji reactions, sanitized inputs, and noise reduction). - -### Step 1: Check the anti-patterns table below -### Step 2: If not listed, check the [triggers reference](https://github.github.com/gh-aw/reference/triggers/), [frontmatter reference](https://github.github.com/gh-aw/reference/frontmatter/), and [safe-outputs reference](https://github.github.com/gh-aw/reference/safe-outputs/) -### Step 3: If a built-in exists, use it. If not, proceed with manual implementation. - -### Anti-Patterns: Manual Reimplementations to Avoid - -| If you're about to implement... | Use this built-in instead | Docs | -|---------------------------------|--------------------------|------| -| `issue_comment` + `startsWith(comment.body, '/cmd')` | `slash_command:` trigger | [Command Triggers](https://github.github.com/gh-aw/reference/command-triggers/) | -| Manual emoji reaction on triggering comment | `reaction:` field under `on:` | [Frontmatter](https://github.github.com/gh-aw/reference/frontmatter/) | -| Posting "workflow started/completed" status comments | `status-comment: true` under `on:` | [Frontmatter](https://github.github.com/gh-aw/reference/frontmatter/) | -| Fixed cron schedule (`0 9 * * 1`) for non-critical timing | `schedule: weekly on monday around 9:00` (fuzzy) | [Triggers](https://github.github.com/gh-aw/reference/triggers/) | -| Manual `if:` to skip bot-authored PRs | `skip-bots:` under `on:` | [Triggers](https://github.github.com/gh-aw/reference/triggers/) | -| Manual `if:` to skip by author role | `skip-roles:` under `on:` | [Triggers](https://github.github.com/gh-aw/reference/triggers/) | -| Manual label check + removal for one-shot commands | `label_command:` trigger | [Triggers](https://github.github.com/gh-aw/reference/triggers/) | -| Editing old comments to collapse them | `hide-older-comments: true` on `add-comment:` | [Safe Outputs](https://github.github.com/gh-aw/reference/safe-outputs/) | -| Creating no-op report issues | `noop: report-as-issue: false` | [Safe Outputs / Monitoring](https://github.github.com/gh-aw/patterns/monitoring/) | -| Auto-closing older issues from same workflow | `close-older-issues: true` on `create-issue:` | [Safe Outputs](https://github.github.com/gh-aw/reference/safe-outputs/) | -| Disabling workflow after a date | `stop-after:` under `on:` | [Triggers](https://github.github.com/gh-aw/reference/triggers/) | -| Manual approval gating | `manual-approval:` under `on:` | [Triggers](https://github.github.com/gh-aw/reference/triggers/) | -| Search-based skip logic in `steps:` | `skip-if-match:` / `skip-if-no-match:` under `on:` | [Triggers](https://github.github.com/gh-aw/reference/triggers/) | - -**Note:** gh-aw is actively developed. If a capability feels like something a framework would provide natively, check the reference docs โ€” it probably exists even if it's not in this table yet. - -## Architecture - -gh-aw workflows are authored as `.md` files with YAML frontmatter, compiled to `.lock.yml` via `gh aw compile`. The lock file is auto-generated โ€” **never edit it manually**. - -### Execution Model - -``` -activation job (renders prompt from base branch .md via runtime-import) - โ†“ -agent job: - user steps: (pre-agent, OUTSIDE firewall, has GITHUB_TOKEN) - โ†“ - platform steps: (configure git โ†’ checkout_pr_branch.cjs โ†’ install CLI) - โ†“ - agent: (INSIDE sandboxed container, NO credentials) -``` - -| Context | Has GITHUB_TOKEN | Has gh CLI | Has git creds | Can execute scripts | -|---------|-----------------|-----------|---------------|-------------------| -| `steps:` (user) | โœ… Yes | โœ… Yes | โœ… Yes | โœ… Yes โ€” **be careful** | -| Platform steps | โœ… Yes | โœ… Yes | โœ… Yes | Platform-controlled | -| Agent container | โŒ Scrubbed | โŒ Scrubbed | โŒ Scrubbed | โœ… But sandboxed | - -**โš ๏ธ Agent container credential nuance:** `GITHUB_TOKEN` and `gh` CLI credentials are scrubbed inside the agent container. However, `COPILOT_TOKEN` (used for LLM inference) is present in the environment via `--env-all`. Any subprocess (e.g., `dotnet build`, `npm install`) inherits this variable. The AWF network firewall, `redact_secrets.cjs` (post-agent log scrubbing), and the threat detection agent limit the blast radius. See [Security Boundaries](#security-boundaries) below. - -### Step Ordering (Critical) - -User `steps:` **always run before** platform-generated steps. You cannot insert user steps after platform steps. - -The platform's `checkout_pr_branch.cjs` runs with `if: (github.event.pull_request) || (github.event.issue.pull_request)` โ€” it is **skipped** for `workflow_dispatch` triggers. - -### Prompt Rendering - -The prompt is built in the **activation job** via `{{#runtime-import .github/workflows/.md}}`. This reads the `.md` file from the **base branch** workspace (before any PR checkout). The rendered prompt is uploaded as an artifact and downloaded by the agent job. - -- The agent prompt is always the base branch version โ€” fork PRs cannot alter it -- The prompt references files on disk (e.g., `SKILL.md`) โ€” those files must exist in the agent's workspace - -### Fork PR Activation Gate - -By default, `gh aw compile` automatically injects a fork guard into the activation job's `if:` condition: `head.repo.id == repository_id`. This blocks fork PRs on `pull_request` events. - -To **allow fork PRs**, add `forks: ["*"]` to the `pull_request` trigger in the `.md` frontmatter. The compiler removes the auto-injected guard from the compiled `if:` conditions. This is safe when the workflow uses the `Checkout-GhAwPr.ps1` pattern (checkout + trusted-infra restore) and the agent is sandboxed. - -## Security Boundaries - -### Key Principles (from [GitHub Security Lab](https://securitylab.github.com/resources/github-actions-preventing-pwn-requests/)) - -1. **Never execute untrusted PR code with elevated credentials.** The classic "pwn-request" attack is `pull_request_target` + checkout PR + run build scripts with `GITHUB_TOKEN`. The attack surface includes build scripts (`make`, `build.ps1`), package manager hooks (`npm postinstall`, MSBuild targets), and test runners. - -2. **Treating PR contents as passive data is safe.** Reading, analyzing, or diffing PR code is fine โ€” the danger is *executing* it. Our gh-aw workflows read code for evaluation; they never build or run it. - -3. **`pull_request_target` grants write permissions and secrets access.** This is by design โ€” the workflow YAML comes from the base branch (trusted). But any step that checks out and runs fork code in this context creates a vulnerability. - -4. **`pull_request` from forks has no secrets access.** GitHub withholds secrets because the workflow YAML comes from the fork (untrusted). This is the safe default for CI builds on fork PRs. - -5. **The `workflow_run` pattern separates privilege from code execution.** Build in an unprivileged `pull_request` job โ†’ pass artifacts โ†’ process in a privileged `workflow_run` job. This is architecturally what gh-aw does: agent runs read-only, `safe_outputs` job has write permissions. - -### gh-aw Defense Layers - -| Layer | What it does | What it doesn't do | -|-------|-------------|-------------------| -| **AWF network firewall** | Restricts outbound to allowlisted domains | Doesn't prevent reading env vars inside the container | -| **`redact_secrets.cjs`** | Scrubs known secret values from logs/artifacts post-agent | Doesn't catch encoded/obfuscated values | -| **Threat detection agent** | Reviews agent outputs before safe-outputs publishes them | Can miss novel exfiltration techniques | -| **Safe-outputs permission separation** | Write operations happen in separate job, not the agent | Agent can still request writes via safe-output tools | -| **`max: 1` on `add-comment`** | Limits agent to one comment | That one comment could contain sensitive data (mitigated by redaction) | -| **XPIA prompt** | Instructs LLM to resist prompt injection from untrusted content | LLM compliance is probabilistic, not guaranteed | -| **`pre_activation` role check** | Gates on write-access collaborators | Does not apply if `roles: all` is set | - -### Rules for gh-aw Workflow Authors - -- โœ… **DO** treat PR contents as passive data (read, analyze, diff) -- โœ… **DO** run data-gathering scripts in `steps:` (pre-agent, trusted context) not inside the agent -- โœ… **DO** use `Checkout-GhAwPr.ps1` for `workflow_dispatch` to restore trusted `.github/` from base -- โŒ **DO NOT** run `dotnet build`, `npm install`, or any build command on untrusted PR code inside the agent โ€” build tool hooks (MSBuild targets, postinstall scripts) can read `COPILOT_TOKEN` from the environment -- โŒ **DO NOT** execute workspace scripts (`.ps1`, `.sh`, `.py`) after checking out a fork PR in `steps:` โ€” those run with `GITHUB_TOKEN` -- โŒ **DO NOT** set `roles: all` on workflows that process PR content โ€” this allows any user to trigger the workflow - -## Fork PR Handling - -### The "pwn-request" Threat Model - -The classic attack requires **checkout + execution** of fork code with elevated credentials. Checkout alone is not dangerous โ€” the vulnerability is executing workspace scripts with `GITHUB_TOKEN`. - -Reference: https://securitylab.github.com/resources/github-actions-preventing-pwn-requests/ - -### Fork PR Behavior by Trigger - -| Trigger | `checkout_pr_branch.cjs` runs? | Fork handling | -|---------|-------------------------------|---------------| -| `pull_request` (default) | โœ… Yes | Blocked by auto-generated activation gate unless `forks: ["*"]` is set | -| `pull_request` + `forks: ["*"]` | โœ… Yes | โœ… Works โ€” user steps restore trusted infra before agent runs | -| `workflow_dispatch` | โŒ Skipped | โœ… Works โ€” user steps handle checkout and restore is final | -| `issue_comment` (same-repo) | โœ… Yes | โœ… Works โ€” files already on PR branch | -| `issue_comment` (fork) | โœ… Yes | โš ๏ธ Works โ€” `checkout_pr_branch.cjs` re-checks out fork branch after user steps, potentially overwriting restored infra. Acceptable because agent is sandboxed (no credentials, max 1 comment via safe-outputs). Pre-flight check catches missing `SKILL.md` if fork isn't rebased. | -| `slash_command` | โœ… Yes (compiles to `issue_comment` internally) | Same behavior as `issue_comment` above, but with platform-managed command matching, emoji reactions, and sanitized input. Prefer `slash_command:` over manual `issue_comment` + `startsWith()`. | - -### The `issue_comment` + Fork Problem - -For `/slash-command` triggers on fork PRs, `checkout_pr_branch.cjs` runs AFTER all user steps and re-checks out the fork branch. This overwrites any files restored by user steps (e.g., `.github/skills/`). A fork could include a crafted `SKILL.md` that alters the agent's evaluation behavior. - -**Accepted residual risk:** The agent runs in a sandboxed container with `GITHUB_TOKEN` and `gh` CLI credentials scrubbed. `COPILOT_TOKEN` (for LLM inference) remains in the environment but the AWF network firewall restricts outbound connections to an allowlist of domains, `redact_secrets.cjs` scrubs known secret values from logs/outputs post-agent, and the threat detection agent reviews outputs before they are published. The worst practical outcome is a manipulated evaluation comment (`safe-outputs: add-comment: max: 1`). The pre-flight check in the agent prompt catches the case where `SKILL.md` is missing entirely (fork not rebased on `main`). - -**Upstream issue:** [github/gh-aw#18481](https://github.com/github/gh-aw/issues/18481) โ€” "Using gh-aw in forks of repositories" - -### Safe Pattern: Checkout + Restore - -Use the shared `.github/scripts/Checkout-GhAwPr.ps1` script, which implements checkout + restore in a single reusable step: - -```yaml -steps: - - name: Checkout PR and restore agent infrastructure - env: - GH_TOKEN: ${{ github.token }} - PR_NUMBER: ${{ github.event.pull_request.number || inputs.pr_number }} - run: pwsh .github/scripts/Checkout-GhAwPr.ps1 -``` - -The script: -1. Verifies the PR author has write access and rejects fork PRs -2. Captures the base branch SHA before checkout -3. Checks out the PR branch via `gh pr checkout` -4. Restores `.github/skills/`, `.github/instructions/`, and `.github/copilot-instructions.md` from the base branch SHA (fatal on failure) - -**Behavior by trigger:** -- **`workflow_dispatch`**: Platform checkout is skipped, so the restore IS the final workspace state (trusted files from base branch) -- **`slash_command`** (same-repo): Platform's `checkout_pr_branch.cjs` handles checkout. Skill files typically match main unless the PR modified them. -- **`slash_command`** (fork): Platform re-checks out fork branch after user steps, overwriting restored files. Agent is sandboxed; pre-flight in the prompt catches missing `SKILL.md` - -### Anti-Patterns - -**Do NOT skip checkout for fork PRs:** - -```bash -# โŒ ANTI-PATTERN: Makes fork PRs unevaluable -if [ "$HEAD_OWNER" != "$BASE_OWNER" ]; then - echo "Skipping checkout for fork PR" - exit 0 # Agent evaluates workflow branch instead of PR -fi -``` - -Skipping checkout means the agent evaluates the wrong files. The correct approach is: always check out the PR, then restore agent infrastructure from the base branch. - -**Do NOT execute workspace code after fork checkout:** - -```yaml -# โŒ DANGEROUS: runs fork code with GITHUB_TOKEN -- name: Checkout PR - run: gh pr checkout "$PR_NUMBER" ... -- name: Run analysis - run: pwsh .github/skills/some-script.ps1 -``` - -If you need to run scripts, either: -1. Run them **before** the checkout (from the base branch) -2. Run them **inside the agent container** (sandboxed, no tokens) - -## Compilation - -```bash -# Compile after every change to the .md source -gh aw compile .github/workflows/.md - -# This updates: -# - .github/workflows/.lock.yml (auto-generated) -# - .github/aw/actions-lock.json -``` - -**Always commit the compiled lock file alongside the source `.md`.** - -## Common Patterns - -### Pre-Agent Data Prep (the `steps:` pattern) - -Use `steps:` for any operation requiring GitHub API access that the agent needs: - -```yaml -steps: - - name: Fetch PR data - env: - GH_TOKEN: ${{ github.token }} - run: | - gh pr view "$PR_NUMBER" --json title,body > pr-metadata.json - gh pr diff "$PR_NUMBER" --name-only > changed-files.txt -``` - -### Safe Outputs (Posting Comments) - -```yaml -safe-outputs: - add-comment: - max: 1 - target: "*" # Required for workflow_dispatch (no triggering PR context) -``` - -### Concurrency - -Include all trigger-specific PR number sources: - -```yaml -concurrency: - group: "my-workflow-${{ github.event.issue.number || github.event.pull_request.number || inputs.pr_number || github.run_id }}" - cancel-in-progress: true -``` - -### Noise Reduction - -Filter `pull_request` triggers to relevant paths and add a gate step: - -```yaml -on: - pull_request: - paths: - - 'src/**/tests/**' - -steps: - - name: Gate โ€” skip if no relevant files - if: github.event_name == 'pull_request' - run: | - FILES=$(gh pr diff "$PR_NUMBER" --name-only | grep -E '\.cs$' || true) - if [ -z "$FILES" ]; then exit 1; fi -``` - -Manual triggers (`workflow_dispatch`, `issue_comment`) should bypass the gate. Note: `exit 1` causes a red โŒ on non-matching PRs โ€” this is intentional (no built-in "skip" mechanism in gh-aw steps). - -## Limitations - -| What | Behavior | Workaround | -|------|----------|------------| -| User steps always before platform steps | Cannot run user code after `checkout_pr_branch.cjs` | For `issue_comment` fork PRs, accept sandboxed residual risk; see [gh-aw#18481](https://github.com/github/gh-aw/issues/18481) | -| `--allow-all-tools` in lock.yml | Emitted by `gh aw compile` | Cannot override from `.md` source | -| MCP integrity filtering | Fork PRs blocked as "unapproved" | Use `steps:` checkout instead of MCP | -| `gh` CLI inside agent | Credentials scrubbed | Use `steps:` for API calls, or MCP tools | -| `issue_comment` trigger | Requires workflow on default branch | Must merge to `main` before `/slash-commands` work | -| Duplicate runs | gh-aw sometimes creates 2 runs per dispatch | Harmless, use concurrency groups | - -### Upstream References - -- [github/gh-aw#18481](https://github.com/github/gh-aw/issues/18481) โ€” Fork support tracking issue -- [github/gh-aw#18518](https://github.com/github/gh-aw/issues/18518) โ€” Fork detection in `gh aw init` -- [github/gh-aw#18521](https://github.com/github/gh-aw/issues/18521) โ€” Fork support documentation - -## Troubleshooting - -| Symptom | Cause | Fix | -|---------|-------|-----| -| Agent evaluates wrong PR | `workflow_dispatch` checks out workflow branch | Add `gh pr checkout` in `steps:` | -| Agent can't find SKILL.md | Fork PR branch doesn't include `.github/skills/` | Rebase fork on `main`, or use `workflow_dispatch` with `pr_number` input | -| Fork PR skipped on `pull_request` | `forks: ["*"]` not in workflow frontmatter | Add `forks: ["*"]` under `pull_request:` in the `.md` source and recompile | -| `gh` commands fail in agent | Credentials scrubbed inside container | Move to `steps:` section | -| Lock file out of date | Forgot to recompile | Run `gh aw compile` | -| Integrity filtering warning | MCP reading fork PR data | Expected, non-blocking | -| `/slash-command` doesn't trigger | Workflow not on default branch | Merge to `main` first | diff --git a/.github/scripts/Checkout-GhAwPr.ps1 b/.github/scripts/Checkout-GhAwPr.ps1 deleted file mode 100644 index 1231e451be7d..000000000000 --- a/.github/scripts/Checkout-GhAwPr.ps1 +++ /dev/null @@ -1,116 +0,0 @@ -<# -.SYNOPSIS - Shared PR checkout and trusted-infra restore for gh-aw workflows. - -.DESCRIPTION - Checks out a PR branch and restores trusted agent infrastructure (skills, - instructions) from the base branch. This gives the agent the PR's code - changes with the latest skills and instructions from main. - - Currently used for workflow_dispatch triggers. For slash_command and - issue_comment triggers, the gh-aw platform's checkout_pr_branch.cjs - handles PR checkout automatically โ€” but may overwrite trusted infra - with fork-supplied files. Call this script after platform checkout to - restore trusted .github/ from the base branch. - - SECURITY: Before checkout, the script verifies the PR author has - write access (write, maintain, or admin) and rejects fork PRs. - This prevents checkout of untrusted code in privileged contexts. - - DO NOT add steps after this that run scripts from the workspace - (e.g., ./build.sh, pwsh ./script.ps1). That would create a code - execution vulnerability. See: - https://securitylab.github.com/resources/github-actions-preventing-pwn-requests/ - -.NOTES - Required environment variables (set by the calling workflow step): - GH_TOKEN - GitHub token for API access - PR_NUMBER - PR number to check out - GITHUB_REPOSITORY - owner/repo (set by GitHub Actions) - GITHUB_ENV - path to env file (set by GitHub Actions) -#> - -$ErrorActionPreference = 'Stop' - -# โ”€โ”€ Validate inputs โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - -if (-not $env:PR_NUMBER -or $env:PR_NUMBER -eq '0') { - Write-Host "No PR number available, using default checkout" - exit 0 -} - -$PrNumber = $env:PR_NUMBER - -# โ”€โ”€ Verify PR is same-repo and author has write access โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - -$RawJson = gh pr view $PrNumber --repo $env:GITHUB_REPOSITORY --json author,isCrossRepository --jq '{author: .author.login, isFork: .isCrossRepository}' -if ($LASTEXITCODE -ne 0) { - Write-Host "โŒ Failed to fetch PR #$PrNumber metadata" - exit 1 -} - -try { - $PrInfo = $RawJson | ConvertFrom-Json -} catch { - Write-Host "โŒ PR #$PrNumber returned malformed JSON: $RawJson" - exit 1 -} - -if (-not $PrInfo -or -not $PrInfo.author) { - Write-Host "โŒ PR #$PrNumber returned empty or malformed metadata" - exit 1 -} - -if ($PrInfo.isFork) { - Write-Host "โญ๏ธ PR #$PrNumber is from a fork โ€” skipping. Fork PRs are evaluated in the sandboxed agent container via the platform's checkout_pr_branch.cjs." - exit 1 -} - -$Permission = gh api "repos/$($env:GITHUB_REPOSITORY)/collaborators/$($PrInfo.author)/permission" --jq '.permission' -if ($LASTEXITCODE -ne 0) { - Write-Host "โŒ Failed to check permissions for '$($PrInfo.author)'" - exit 1 -} - -$AllowedRoles = @('admin', 'write', 'maintain') -if ($Permission -notin $AllowedRoles) { - Write-Host "โญ๏ธ PR author '$($PrInfo.author)' has '$Permission' access. workflow_dispatch only processes PRs from authors with write access." - exit 1 -} - -Write-Host "โœ… PR #$PrNumber by '$($PrInfo.author)' ($Permission access, same-repo)" - -# โ”€โ”€ Save base branch SHA โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - -$BaseSha = git rev-parse HEAD -if ($LASTEXITCODE -ne 0) { - Write-Host "โŒ Failed to get current HEAD SHA" - exit 1 -} -Add-Content -Path $env:GITHUB_ENV -Value "BASE_SHA=$BaseSha" -Write-Host "Base branch SHA: $BaseSha" - -# โ”€โ”€ Checkout PR branch โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ - -Write-Host "Checking out PR #$PrNumber..." -gh pr checkout $PrNumber --repo $env:GITHUB_REPOSITORY -if ($LASTEXITCODE -ne 0) { - Write-Host "โŒ Failed to checkout PR #$PrNumber" - exit 1 -} -Write-Host "โœ… Checked out PR #$PrNumber" -git log --oneline -1 - -# โ”€โ”€ Restore agent infrastructure from base branch โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ -# Replace skills and instructions with base branch versions to ensure the agent -# always uses trusted infrastructure from main. Uses git checkout to read files -# directly from the commit tree โ€” works in shallow clones (no history traversal). -# Restore BEFORE deleting so a failure doesn't leave the workspace without infra. - -git checkout $BaseSha -- .github/skills/ .github/instructions/ .github/copilot-instructions.md 2>&1 -if ($LASTEXITCODE -eq 0) { - Write-Host "โœ… Restored agent infrastructure from base branch ($BaseSha)" -} else { - Write-Host "โŒ Failed to restore agent infrastructure from base branch โ€” aborting to prevent running with untrusted infra" - exit 1 -} diff --git a/.github/workflows/copilot-evaluate-tests.lock.yml b/.github/workflows/copilot-evaluate-tests.lock.yml index 8f07ab6405a9..46dffc668d56 100644 --- a/.github/workflows/copilot-evaluate-tests.lock.yml +++ b/.github/workflows/copilot-evaluate-tests.lock.yml @@ -1,3 +1,5 @@ +# gh-aw-metadata: {"schema_version":"v3","frontmatter_hash":"3987c8ff8c6fc12c964100324d3531fc77e954c8823607239515783232e430c7","compiler_version":"v0.68.3","strict":true,"agent_id":"copilot","agent_model":"claude-sonnet-4.6"} +# gh-aw-manifest: {"version":1,"secrets":["COPILOT_GITHUB_TOKEN","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/checkout","sha":"de0fac2e4500dabe0009e67214ff5f5447ce83dd","version":"v6.0.2"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"373c709c69115d41ff229c7e5df9f8788daa9553","version":"v9"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"ba90f2186d7ad780ec640f364005fa24e797b360","version":"v0.68.3"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.25.20"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.25.20"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.25.20"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.2.19"},{"image":"ghcr.io/github/github-mcp-server:v0.32.0"},{"image":"node:lts-alpine"}]} # ___ _ _ # / _ \ | | (_) # | |_| | __ _ ___ _ __ | |_ _ ___ @@ -12,7 +14,7 @@ # \ /\ / (_) | | | | ( | | | | (_) \ V V /\__ \ # \/ \/ \___/|_| |_|\_\|_| |_|\___/ \_/\_/ |___/ # -# This file was automatically generated by gh-aw (v0.62.2). DO NOT EDIT. +# This file was automatically generated by gh-aw (v0.68.3). DO NOT EDIT. # # To update this file, edit the corresponding .md file and run: # gh aw compile @@ -22,7 +24,27 @@ # # Evaluates test quality, coverage, and appropriateness on PRs that add or modify tests # -# gh-aw-metadata: {"schema_version":"v2","frontmatter_hash":"cbfd75d7a699f76155135d83866eda6476cef647c384243c8ee991065a2b44d7","compiler_version":"v0.62.2","strict":true} +# Secrets used: +# - COPILOT_GITHUB_TOKEN +# - GH_AW_GITHUB_MCP_SERVER_TOKEN +# - GH_AW_GITHUB_TOKEN +# - GITHUB_TOKEN +# +# Custom actions used: +# - actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 +# - actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 +# - actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 +# - actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9 +# - actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 +# - github/gh-aw-actions/setup@ba90f2186d7ad780ec640f364005fa24e797b360 # v0.68.3 +# +# Container images used: +# - ghcr.io/github/gh-aw-firewall/agent:0.25.20 +# - ghcr.io/github/gh-aw-firewall/api-proxy:0.25.20 +# - ghcr.io/github/gh-aw-firewall/squid:0.25.20 +# - ghcr.io/github/gh-aw-mcpg:v0.2.19 +# - ghcr.io/github/github-mcp-server:v0.32.0 +# - node:lts-alpine name: "Evaluate PR Tests" "on": @@ -32,8 +54,17 @@ name: "Evaluate PR Tests" types: - created - edited + # roles: # Roles processed as role check in pre-activation job + # - admin # Roles processed as role check in pre-activation job + # - maintain # Roles processed as role check in pre-activation job + # - write # Roles processed as role check in pre-activation job workflow_dispatch: inputs: + aw_context: + default: "" + description: Agent caller context (used internally by Agentic Workflows). + required: false + type: string pr_number: description: PR number to evaluate required: true @@ -47,7 +78,7 @@ name: "Evaluate PR Tests" permissions: {} concurrency: - cancel-in-progress: true + cancel-in-progress: false group: evaluate-pr-tests-${{ github.event.issue.number || inputs.pr_number || github.run_id }} run-name: "Evaluate PR Tests" @@ -56,9 +87,10 @@ jobs: activation: needs: pre_activation if: > - (needs.pre_activation.outputs.activated == 'true') && (github.event_name == 'issue_comment' || github.event_name == 'workflow_dispatch') + needs.pre_activation.outputs.activated == 'true' && (github.event_name == 'issue_comment' || github.event_name == 'workflow_dispatch') runs-on: ubuntu-slim permissions: + actions: read contents: read discussions: write issues: write @@ -71,56 +103,61 @@ jobs: lockdown_check_failed: ${{ steps.generate_aw_info.outputs.lockdown_check_failed == 'true' }} model: ${{ steps.generate_aw_info.outputs.model }} secret_verification_result: ${{ steps.validate-secret.outputs.verification_result }} + setup-trace-id: ${{ steps.setup.outputs.trace-id }} slash_command: ${{ needs.pre_activation.outputs.matched_command }} + stale_lock_file_failed: ${{ steps.check-lock-file.outputs.stale_lock_file_failed == 'true' }} text: ${{ steps.sanitized.outputs.text }} title: ${{ steps.sanitized.outputs.title }} steps: - name: Setup Scripts - uses: github/gh-aw-actions/setup@20045bbd5ad2632b9809856c389708eab1bd16ef # v0.62.2 + id: setup + uses: github/gh-aw-actions/setup@ba90f2186d7ad780ec640f364005fa24e797b360 # v0.68.3 with: destination: ${{ runner.temp }}/gh-aw/actions + job-name: ${{ github.job }} + trace-id: ${{ needs.pre_activation.outputs.setup-trace-id }} - name: Generate agentic run info id: generate_aw_info env: GH_AW_INFO_ENGINE_ID: "copilot" GH_AW_INFO_ENGINE_NAME: "GitHub Copilot CLI" GH_AW_INFO_MODEL: "claude-sonnet-4.6" - GH_AW_INFO_VERSION: "" - GH_AW_INFO_AGENT_VERSION: "latest" - GH_AW_INFO_CLI_VERSION: "v0.62.2" + GH_AW_INFO_VERSION: "1.0.21" + GH_AW_INFO_AGENT_VERSION: "1.0.21" + GH_AW_INFO_CLI_VERSION: "v0.68.3" GH_AW_INFO_WORKFLOW_NAME: "Evaluate PR Tests" GH_AW_INFO_EXPERIMENTAL: "false" GH_AW_INFO_SUPPORTS_TOOLS_ALLOWLIST: "true" GH_AW_INFO_STAGED: "false" GH_AW_INFO_ALLOWED_DOMAINS: '["defaults"]' GH_AW_INFO_FIREWALL_ENABLED: "true" - GH_AW_INFO_AWF_VERSION: "v0.24.3" + GH_AW_INFO_AWF_VERSION: "v0.25.20" GH_AW_INFO_AWMG_VERSION: "" GH_AW_INFO_FIREWALL_TYPE: "squid" GH_AW_COMPILED_STRICT: "true" - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 with: script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); + setupGlobals(core, github, context, exec, io, getOctokit); const { main } = require('${{ runner.temp }}/gh-aw/actions/generate_aw_info.cjs'); await main(core, context); - name: Add eyes reaction for immediate feedback id: react - if: github.event_name == 'issues' || github.event_name == 'issue_comment' || github.event_name == 'pull_request_review_comment' || github.event_name == 'discussion' || github.event_name == 'discussion_comment' || (github.event_name == 'pull_request') && (github.event.pull_request.head.repo.id == github.repository_id) - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + if: github.event_name == 'issues' || github.event_name == 'issue_comment' || github.event_name == 'pull_request_review_comment' || github.event_name == 'discussion' || github.event_name == 'discussion_comment' || github.event_name == 'pull_request' && github.event.pull_request.head.repo.id == github.repository_id + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 env: GH_AW_REACTION: "eyes" with: github-token: ${{ secrets.GITHUB_TOKEN }} script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); + setupGlobals(core, github, context, exec, io, getOctokit); const { main } = require('${{ runner.temp }}/gh-aw/actions/add_reaction.cjs'); await main(); - name: Validate COPILOT_GITHUB_TOKEN secret id: validate-secret - run: ${RUNNER_TEMP}/gh-aw/actions/validate_multi_secret.sh COPILOT_GITHUB_TOKEN 'GitHub Copilot CLI' https://github.github.com/gh-aw/reference/engines/#github-copilot-default + run: bash "${RUNNER_TEMP}/gh-aw/actions/validate_multi_secret.sh" COPILOT_GITHUB_TOKEN 'GitHub Copilot CLI' https://github.github.com/gh-aw/reference/engines/#github-copilot-default env: COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_GITHUB_TOKEN }} - name: Checkout .github and .agents folders @@ -132,44 +169,56 @@ jobs: .agents sparse-checkout-cone-mode: true fetch-depth: 1 - - name: Check workflow file timestamps - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + - name: Check workflow lock file + id: check-lock-file + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 env: GH_AW_WORKFLOW_FILE: "copilot-evaluate-tests.lock.yml" + GH_AW_CONTEXT_WORKFLOW_REF: "${{ github.workflow_ref }}" with: script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); + setupGlobals(core, github, context, exec, io, getOctokit); const { main } = require('${{ runner.temp }}/gh-aw/actions/check_workflow_timestamp_api.cjs'); await main(); + - name: Check compile-agentic version + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 + env: + GH_AW_COMPILED_VERSION: "v0.68.3" + with: + script: | + const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); + setupGlobals(core, github, context, exec, io, getOctokit); + const { main } = require('${{ runner.temp }}/gh-aw/actions/check_version_updates.cjs'); + await main(); - name: Compute current body text id: sanitized - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 env: - GH_AW_ALLOWED_BOTS: copilot-swe-agent[bot] + GH_AW_ALLOWED_BOTS: "copilot-swe-agent[bot]" with: script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); + setupGlobals(core, github, context, exec, io, getOctokit); const { main } = require('${{ runner.temp }}/gh-aw/actions/compute_text.cjs'); await main(); - name: Add comment with workflow run link id: add-comment - if: github.event_name == 'issues' || github.event_name == 'issue_comment' || github.event_name == 'pull_request_review_comment' || github.event_name == 'discussion' || github.event_name == 'discussion_comment' || (github.event_name == 'pull_request') && (github.event.pull_request.head.repo.id == github.repository_id) - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + if: github.event_name == 'issues' || github.event_name == 'issue_comment' || github.event_name == 'pull_request_review_comment' || github.event_name == 'discussion' || github.event_name == 'discussion_comment' || github.event_name == 'pull_request' && github.event.pull_request.head.repo.id == github.repository_id + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 env: GH_AW_WORKFLOW_NAME: "Evaluate PR Tests" GH_AW_SAFE_OUTPUT_MESSAGES: "{\"footer\":\"\\u003e ๐Ÿงช *Test evaluation by [{workflow_name}]({run_url})*\",\"runStarted\":\"๐Ÿ”ฌ Evaluating tests on this PRโ€ฆ [{workflow_name}]({run_url})\",\"runSuccess\":\"โœ… Test evaluation complete! [{workflow_name}]({run_url})\",\"runFailure\":\"โŒ Test evaluation failed. [{workflow_name}]({run_url}) {status}\"}" with: script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); + setupGlobals(core, github, context, exec, io, getOctokit); const { main } = require('${{ runner.temp }}/gh-aw/actions/add_workflow_run_comment.cjs'); await main(); - name: Create prompt with built-in context env: GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt - GH_AW_SAFE_OUTPUTS: ${{ env.GH_AW_SAFE_OUTPUTS }} + GH_AW_SAFE_OUTPUTS: ${{ runner.temp }}/gh-aw/safeoutputs/outputs.jsonl GH_AW_EXPR_A77326CF: ${{ github.event.issue.number || inputs.pr_number }} GH_AW_GITHUB_ACTOR: ${{ github.actor }} GH_AW_GITHUB_EVENT_COMMENT_ID: ${{ github.event.comment.id }} @@ -181,17 +230,18 @@ jobs: GH_AW_GITHUB_WORKSPACE: ${{ github.workspace }} GH_AW_INPUTS_SUPPRESS_OUTPUT: ${{ inputs.suppress_output }} GH_AW_IS_PR_COMMENT: ${{ github.event.issue.pull_request && 'true' || '' }} + # poutine:ignore untrusted_checkout_exec run: | - bash ${RUNNER_TEMP}/gh-aw/actions/create_prompt_first.sh + bash "${RUNNER_TEMP}/gh-aw/actions/create_prompt_first.sh" { - cat << 'GH_AW_PROMPT_EOF' + cat << 'GH_AW_PROMPT_76b04e0a6e258a6a_EOF' - GH_AW_PROMPT_EOF + GH_AW_PROMPT_76b04e0a6e258a6a_EOF cat "${RUNNER_TEMP}/gh-aw/prompts/xpia.md" cat "${RUNNER_TEMP}/gh-aw/prompts/temp_folder_prompt.md" cat "${RUNNER_TEMP}/gh-aw/prompts/markdown.md" cat "${RUNNER_TEMP}/gh-aw/prompts/safe_outputs_prompt.md" - cat << 'GH_AW_PROMPT_EOF' + cat << 'GH_AW_PROMPT_76b04e0a6e258a6a_EOF' Tools: add_comment, missing_tool, missing_data, noop @@ -223,20 +273,18 @@ jobs: {{/if}} - GH_AW_PROMPT_EOF + GH_AW_PROMPT_76b04e0a6e258a6a_EOF cat "${RUNNER_TEMP}/gh-aw/prompts/github_mcp_tools_with_safeoutputs_prompt.md" if [ "$GITHUB_EVENT_NAME" = "issue_comment" ] && [ -n "$GH_AW_IS_PR_COMMENT" ] || [ "$GITHUB_EVENT_NAME" = "pull_request_review_comment" ] || [ "$GITHUB_EVENT_NAME" = "pull_request_review" ]; then cat "${RUNNER_TEMP}/gh-aw/prompts/pr_context_prompt.md" fi - cat << 'GH_AW_PROMPT_EOF' + cat << 'GH_AW_PROMPT_76b04e0a6e258a6a_EOF' - GH_AW_PROMPT_EOF - cat << 'GH_AW_PROMPT_EOF' {{#runtime-import .github/workflows/copilot-evaluate-tests.md}} - GH_AW_PROMPT_EOF + GH_AW_PROMPT_76b04e0a6e258a6a_EOF } > "$GH_AW_PROMPT" - name: Interpolate variables and render templates - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 env: GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt GH_AW_EXPR_A77326CF: ${{ github.event.issue.number || inputs.pr_number }} @@ -245,11 +293,11 @@ jobs: with: script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); + setupGlobals(core, github, context, exec, io, getOctokit); const { main } = require('${{ runner.temp }}/gh-aw/actions/interpolate_prompt.cjs'); await main(); - name: Substitute placeholders - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 env: GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt GH_AW_EXPR_A77326CF: ${{ github.event.issue.number || inputs.pr_number }} @@ -268,7 +316,7 @@ jobs: with: script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); + setupGlobals(core, github, context, exec, io, getOctokit); const substitutePlaceholders = require('${{ runner.temp }}/gh-aw/actions/substitute_placeholders.cjs'); @@ -294,19 +342,23 @@ jobs: - name: Validate prompt placeholders env: GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt - run: bash ${RUNNER_TEMP}/gh-aw/actions/validate_prompt_placeholders.sh + # poutine:ignore untrusted_checkout_exec + run: bash "${RUNNER_TEMP}/gh-aw/actions/validate_prompt_placeholders.sh" - name: Print prompt env: GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt - run: bash ${RUNNER_TEMP}/gh-aw/actions/print_prompt_summary.sh + # poutine:ignore untrusted_checkout_exec + run: bash "${RUNNER_TEMP}/gh-aw/actions/print_prompt_summary.sh" - name: Upload activation artifact if: success() - uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: activation path: | /tmp/gh-aw/aw_info.json /tmp/gh-aw/aw-prompts/prompt.txt + /tmp/gh-aw/github_rate_limits.jsonl + if-no-files-found: ignore retention-days: 1 agent: @@ -324,81 +376,85 @@ jobs: GH_AW_MCP_LOG_DIR: /tmp/gh-aw/mcp-logs/safeoutputs GH_AW_WORKFLOW_ID_SANITIZED: copilotevaluatetests outputs: + agentic_engine_timeout: ${{ steps.detect-copilot-errors.outputs.agentic_engine_timeout || 'false' }} checkout_pr_success: ${{ steps.checkout-pr.outputs.checkout_pr_success || 'true' }} - detection_conclusion: ${{ steps.detection_conclusion.outputs.conclusion }} - detection_success: ${{ steps.detection_conclusion.outputs.success }} + effective_tokens: ${{ steps.parse-mcp-gateway.outputs.effective_tokens }} has_patch: ${{ steps.collect_output.outputs.has_patch }} - inference_access_error: ${{ steps.detect-inference-error.outputs.inference_access_error || 'false' }} + inference_access_error: ${{ steps.detect-copilot-errors.outputs.inference_access_error || 'false' }} + mcp_policy_error: ${{ steps.detect-copilot-errors.outputs.mcp_policy_error || 'false' }} model: ${{ needs.activation.outputs.model }} + model_not_supported_error: ${{ steps.detect-copilot-errors.outputs.model_not_supported_error || 'false' }} output: ${{ steps.collect_output.outputs.output }} output_types: ${{ steps.collect_output.outputs.output_types }} + setup-trace-id: ${{ steps.setup.outputs.trace-id }} steps: - name: Setup Scripts - uses: github/gh-aw-actions/setup@20045bbd5ad2632b9809856c389708eab1bd16ef # v0.62.2 + id: setup + uses: github/gh-aw-actions/setup@ba90f2186d7ad780ec640f364005fa24e797b360 # v0.68.3 with: destination: ${{ runner.temp }}/gh-aw/actions + job-name: ${{ github.job }} + trace-id: ${{ needs.activation.outputs.setup-trace-id }} - name: Set runtime paths + id: set-runtime-paths run: | - echo "GH_AW_SAFE_OUTPUTS=${RUNNER_TEMP}/gh-aw/safeoutputs/outputs.jsonl" >> "$GITHUB_ENV" - echo "GH_AW_SAFE_OUTPUTS_CONFIG_PATH=${RUNNER_TEMP}/gh-aw/safeoutputs/config.json" >> "$GITHUB_ENV" - echo "GH_AW_SAFE_OUTPUTS_TOOLS_PATH=${RUNNER_TEMP}/gh-aw/safeoutputs/tools.json" >> "$GITHUB_ENV" + { + echo "GH_AW_SAFE_OUTPUTS=${RUNNER_TEMP}/gh-aw/safeoutputs/outputs.jsonl" + echo "GH_AW_SAFE_OUTPUTS_CONFIG_PATH=${RUNNER_TEMP}/gh-aw/safeoutputs/config.json" + echo "GH_AW_SAFE_OUTPUTS_TOOLS_PATH=${RUNNER_TEMP}/gh-aw/safeoutputs/tools.json" + } >> "$GITHUB_OUTPUT" - name: Checkout repository uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: persist-credentials: false - name: Create gh-aw temp directory - run: bash ${RUNNER_TEMP}/gh-aw/actions/create_gh_aw_tmp_dir.sh + run: bash "${RUNNER_TEMP}/gh-aw/actions/create_gh_aw_tmp_dir.sh" - name: Configure gh CLI for GitHub Enterprise - run: bash ${RUNNER_TEMP}/gh-aw/actions/configure_gh_for_ghe.sh + run: bash "${RUNNER_TEMP}/gh-aw/actions/configure_gh_for_ghe.sh" env: GH_TOKEN: ${{ github.token }} - env: GH_TOKEN: ${{ github.token }} PR_NUMBER: ${{ github.event.issue.number || inputs.pr_number }} name: Gate โ€” skip if no test source files in diff - run: "# Verify this is an open PR\nif ! STATE=$(gh pr view \"$PR_NUMBER\" --repo \"$GITHUB_REPOSITORY\" --json state --jq .state 2>&1); then\n echo \"โŒ Failed to fetch PR #$PR_NUMBER state: $STATE\"\n exit 1\nfi\nif [ \"$STATE\" != \"OPEN\" ]; then\n echo \"โญ๏ธ PR #$PR_NUMBER is $STATE โ€” skipping evaluation.\"\n exit 1\nfi\n# Try gh pr diff first; fall back to REST API only on command failure\nif DIFF_OUTPUT=$(gh pr diff \"$PR_NUMBER\" --repo \"$GITHUB_REPOSITORY\" --name-only 2>/dev/null); then\n TEST_FILES=$(echo \"$DIFF_OUTPUT\" \\\n | grep -E '\\.(cs|xaml)$' \\\n | grep -iE '(tests?/|TestCases|UnitTests|DeviceTests)' \\\n || true)\nelse\n # gh pr diff fails with HTTP 406 for PRs with 300+ files; use paginated files API\n if ! API_FILES=$(gh api \"repos/$GITHUB_REPOSITORY/pulls/$PR_NUMBER/files\" --paginate --jq '.[].filename' 2>&1); then\n echo \"โŒ gh pr diff failed and REST API fallback also failed: $API_FILES\"\n exit 1\n fi\n TEST_FILES=$(echo \"$API_FILES\" \\\n | grep -E '\\.(cs|xaml)$' \\\n | grep -iE '(tests?/|TestCases|UnitTests|DeviceTests)' \\\n || true)\nfi\nif [ -z \"$TEST_FILES\" ]; then\n echo \"โญ๏ธ No test source files (.cs/.xaml) found in PR diff. Nothing to evaluate.\"\n exit 1\nfi\necho \"โœ… Found test files to evaluate:\"\necho \"$TEST_FILES\" | head -20\n" - - env: - GH_TOKEN: ${{ github.token }} - PR_NUMBER: ${{ inputs.pr_number }} - if: github.event_name == 'workflow_dispatch' - name: Checkout PR and restore agent infrastructure - run: pwsh .github/scripts/Checkout-GhAwPr.ps1 + run: "# Verify this is an open PR\nif ! STATE=$(gh pr view \"$PR_NUMBER\" --repo \"$GITHUB_REPOSITORY\" --json state --jq .state 2>&1); then\n echo \"โŒ Failed to fetch PR #$PR_NUMBER state: $STATE\"\n exit 1\nfi\nif [ \"$STATE\" != \"OPEN\" ]; then\n echo \"โญ๏ธ PR #$PR_NUMBER is $STATE โ€” skipping evaluation.\"\n exit 1\nfi\n# Try gh pr diff first; fall back to REST API only on command failure\nif DIFF_OUTPUT=$(gh pr diff \"$PR_NUMBER\" --repo \"$GITHUB_REPOSITORY\" --name-only 2>/dev/null); then\n TEST_FILES=$(echo \"$DIFF_OUTPUT\" \\\n | grep -E '\\.(cs|xaml)$' \\\n | grep -iE '(tests?/|TestCases|UnitTests|DeviceTests)' \\\n || true)\nelse\n # gh pr diff fails with HTTP 406 for PRs with 300+ files; use paginated files API\n if ! API_FILES=$(gh api \"repos/$GITHUB_REPOSITORY/pulls/$PR_NUMBER/files\" --paginate --jq '.[].filename' 2>&1); then\n echo \"โŒ gh pr diff failed and REST API fallback also failed: $API_FILES\"\n exit 1\n fi\n TEST_FILES=$(echo \"$API_FILES\" \\\n | grep -E '\\.(cs|xaml)$' \\\n | grep -iE '(tests?/|TestCases|UnitTests|DeviceTests)' \\\n || true)\nfi\nif [ -z \"$TEST_FILES\" ]; then\n echo \"โญ๏ธ No test source files (.cs/.xaml) found in PR diff. Nothing to evaluate.\"\n exit 1\nfi\necho \"โœ… Found test files to evaluate:\"\necho \"$TEST_FILES\" | head -20" - name: Configure Git credentials env: REPO_NAME: ${{ github.repository }} SERVER_URL: ${{ github.server_url }} + GITHUB_TOKEN: ${{ github.token }} run: | git config --global user.email "github-actions[bot]@users.noreply.github.com" git config --global user.name "github-actions[bot]" git config --global am.keepcr true # Re-authenticate git with GitHub token SERVER_URL_STRIPPED="${SERVER_URL#https://}" - git remote set-url origin "https://x-access-token:${{ github.token }}@${SERVER_URL_STRIPPED}/${REPO_NAME}.git" + git remote set-url origin "https://x-access-token:${GITHUB_TOKEN}@${SERVER_URL_STRIPPED}/${REPO_NAME}.git" echo "Git configured with standard GitHub Actions identity" - name: Checkout PR branch id: checkout-pr if: | - (github.event.pull_request) || (github.event.issue.pull_request) - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + github.event.pull_request || github.event.issue.pull_request + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 env: GH_TOKEN: ${{ secrets.GH_AW_GITHUB_MCP_SERVER_TOKEN || secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} with: github-token: ${{ secrets.GH_AW_GITHUB_MCP_SERVER_TOKEN || secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); + setupGlobals(core, github, context, exec, io, getOctokit); const { main } = require('${{ runner.temp }}/gh-aw/actions/checkout_pr_branch.cjs'); await main(); - name: Install GitHub Copilot CLI - run: ${RUNNER_TEMP}/gh-aw/actions/install_copilot_cli.sh latest + run: bash "${RUNNER_TEMP}/gh-aw/actions/install_copilot_cli.sh" 1.0.21 env: GH_HOST: github.com - name: Install AWF binary - run: bash ${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh v0.24.3 + run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.25.20 - name: Determine automatic lockdown mode for GitHub MCP Server id: determine-automatic-lockdown - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9 env: GH_AW_GITHUB_TOKEN: ${{ secrets.GH_AW_GITHUB_TOKEN }} GH_AW_GITHUB_MCP_SERVER_TOKEN: ${{ secrets.GH_AW_GITHUB_MCP_SERVER_TOKEN }} @@ -407,106 +463,130 @@ jobs: const determineAutomaticLockdown = require('${{ runner.temp }}/gh-aw/actions/determine_automatic_lockdown.cjs'); await determineAutomaticLockdown(github, context, core); - name: Download container images - run: bash ${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh ghcr.io/github/gh-aw-firewall/agent:0.24.3 ghcr.io/github/gh-aw-firewall/api-proxy:0.24.3 ghcr.io/github/gh-aw-firewall/squid:0.24.3 ghcr.io/github/gh-aw-mcpg:v0.1.19 ghcr.io/github/github-mcp-server:v0.32.0 node:lts-alpine + run: bash "${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh" ghcr.io/github/gh-aw-firewall/agent:0.25.20 ghcr.io/github/gh-aw-firewall/api-proxy:0.25.20 ghcr.io/github/gh-aw-firewall/squid:0.25.20 ghcr.io/github/gh-aw-mcpg:v0.2.19 ghcr.io/github/github-mcp-server:v0.32.0 node:lts-alpine - name: Write Safe Outputs Config run: | - mkdir -p ${RUNNER_TEMP}/gh-aw/safeoutputs + mkdir -p "${RUNNER_TEMP}/gh-aw/safeoutputs" mkdir -p /tmp/gh-aw/safeoutputs mkdir -p /tmp/gh-aw/mcp-logs/safeoutputs - cat > ${RUNNER_TEMP}/gh-aw/safeoutputs/config.json << 'GH_AW_SAFE_OUTPUTS_CONFIG_EOF' - {"add_comment":{"max":1,"target":"*"},"missing_data":{},"missing_tool":{},"noop":{"max":1}} - GH_AW_SAFE_OUTPUTS_CONFIG_EOF + cat > "${RUNNER_TEMP}/gh-aw/safeoutputs/config.json" << 'GH_AW_SAFE_OUTPUTS_CONFIG_f147e6f2a8dceac4_EOF' + {"add_comment":{"hide_older_comments":true,"max":1,"target":"*"},"create_report_incomplete_issue":{},"missing_data":{},"missing_tool":{},"noop":{"max":1,"report-as-issue":"false"},"report_incomplete":{}} + GH_AW_SAFE_OUTPUTS_CONFIG_f147e6f2a8dceac4_EOF - name: Write Safe Outputs Tools - run: | - cat > ${RUNNER_TEMP}/gh-aw/safeoutputs/tools_meta.json << 'GH_AW_SAFE_OUTPUTS_TOOLS_META_EOF' - { - "description_suffixes": { - "add_comment": " CONSTRAINTS: Maximum 1 comment(s) can be added. Target: *." - }, - "repo_params": {}, - "dynamic_tools": [] - } - GH_AW_SAFE_OUTPUTS_TOOLS_META_EOF - cat > ${RUNNER_TEMP}/gh-aw/safeoutputs/validation.json << 'GH_AW_SAFE_OUTPUTS_VALIDATION_EOF' - { - "add_comment": { - "defaultMax": 1, - "fields": { - "body": { - "required": true, - "type": "string", - "sanitize": true, - "maxLength": 65000 - }, - "item_number": { - "issueOrPRNumber": true - }, - "repo": { - "type": "string", - "maxLength": 256 + env: + GH_AW_TOOLS_META_JSON: | + { + "description_suffixes": { + "add_comment": " CONSTRAINTS: Maximum 1 comment(s) can be added. Target: *. Supports reply_to_id for discussion threading." + }, + "repo_params": {}, + "dynamic_tools": [] + } + GH_AW_VALIDATION_JSON: | + { + "add_comment": { + "defaultMax": 1, + "fields": { + "body": { + "required": true, + "type": "string", + "sanitize": true, + "maxLength": 65000 + }, + "item_number": { + "issueOrPRNumber": true + }, + "reply_to_id": { + "type": "string", + "maxLength": 256 + }, + "repo": { + "type": "string", + "maxLength": 256 + } } - } - }, - "missing_data": { - "defaultMax": 20, - "fields": { - "alternatives": { - "type": "string", - "sanitize": true, - "maxLength": 256 - }, - "context": { - "type": "string", - "sanitize": true, - "maxLength": 256 - }, - "data_type": { - "type": "string", - "sanitize": true, - "maxLength": 128 - }, - "reason": { - "type": "string", - "sanitize": true, - "maxLength": 256 + }, + "missing_data": { + "defaultMax": 20, + "fields": { + "alternatives": { + "type": "string", + "sanitize": true, + "maxLength": 256 + }, + "context": { + "type": "string", + "sanitize": true, + "maxLength": 256 + }, + "data_type": { + "type": "string", + "sanitize": true, + "maxLength": 128 + }, + "reason": { + "type": "string", + "sanitize": true, + "maxLength": 256 + } } - } - }, - "missing_tool": { - "defaultMax": 20, - "fields": { - "alternatives": { - "type": "string", - "sanitize": true, - "maxLength": 512 - }, - "reason": { - "required": true, - "type": "string", - "sanitize": true, - "maxLength": 256 - }, - "tool": { - "type": "string", - "sanitize": true, - "maxLength": 128 + }, + "missing_tool": { + "defaultMax": 20, + "fields": { + "alternatives": { + "type": "string", + "sanitize": true, + "maxLength": 512 + }, + "reason": { + "required": true, + "type": "string", + "sanitize": true, + "maxLength": 256 + }, + "tool": { + "type": "string", + "sanitize": true, + "maxLength": 128 + } } - } - }, - "noop": { - "defaultMax": 1, - "fields": { - "message": { - "required": true, - "type": "string", - "sanitize": true, - "maxLength": 65000 + }, + "noop": { + "defaultMax": 1, + "fields": { + "message": { + "required": true, + "type": "string", + "sanitize": true, + "maxLength": 65000 + } + } + }, + "report_incomplete": { + "defaultMax": 5, + "fields": { + "details": { + "type": "string", + "sanitize": true, + "maxLength": 65000 + }, + "reason": { + "required": true, + "type": "string", + "sanitize": true, + "maxLength": 1024 + } } } } - } - GH_AW_SAFE_OUTPUTS_VALIDATION_EOF - node ${RUNNER_TEMP}/gh-aw/actions/generate_safe_outputs_tools.cjs + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 + with: + script: | + const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); + setupGlobals(core, github, context, exec, io, getOctokit); + const { main } = require('${{ runner.temp }}/gh-aw/actions/generate_safe_outputs_tools.cjs'); + await main(); - name: Generate Safe Outputs MCP Server Config id: safe-outputs-config run: | @@ -529,6 +609,7 @@ jobs: id: safe-outputs-start env: DEBUG: '*' + GH_AW_SAFE_OUTPUTS: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS }} GH_AW_SAFE_OUTPUTS_PORT: ${{ steps.safe-outputs-config.outputs.safe_outputs_port }} GH_AW_SAFE_OUTPUTS_API_KEY: ${{ steps.safe-outputs-config.outputs.safe_outputs_api_key }} GH_AW_SAFE_OUTPUTS_TOOLS_PATH: ${{ runner.temp }}/gh-aw/safeoutputs/tools.json @@ -537,18 +618,19 @@ jobs: run: | # Environment variables are set above to prevent template injection export DEBUG + export GH_AW_SAFE_OUTPUTS export GH_AW_SAFE_OUTPUTS_PORT export GH_AW_SAFE_OUTPUTS_API_KEY export GH_AW_SAFE_OUTPUTS_TOOLS_PATH export GH_AW_SAFE_OUTPUTS_CONFIG_PATH export GH_AW_MCP_LOG_DIR - bash ${RUNNER_TEMP}/gh-aw/actions/start_safe_outputs_server.sh + bash "${RUNNER_TEMP}/gh-aw/actions/start_safe_outputs_server.sh" - name: Start MCP Gateway id: start-mcp-gateway env: - GH_AW_SAFE_OUTPUTS: ${{ env.GH_AW_SAFE_OUTPUTS }} + GH_AW_SAFE_OUTPUTS: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS }} GH_AW_SAFE_OUTPUTS_API_KEY: ${{ steps.safe-outputs-start.outputs.api_key }} GH_AW_SAFE_OUTPUTS_PORT: ${{ steps.safe-outputs-start.outputs.port }} GITHUB_MCP_GUARD_MIN_INTEGRITY: ${{ steps.determine-automatic-lockdown.outputs.min_integrity }} @@ -570,10 +652,10 @@ jobs: export DEBUG="*" export GH_AW_ENGINE="copilot" - export MCP_GATEWAY_DOCKER_COMMAND='docker run -i --rm --network host -v /var/run/docker.sock:/var/run/docker.sock -e MCP_GATEWAY_PORT -e MCP_GATEWAY_DOMAIN -e MCP_GATEWAY_API_KEY -e MCP_GATEWAY_PAYLOAD_DIR -e MCP_GATEWAY_PAYLOAD_SIZE_THRESHOLD -e DEBUG -e MCP_GATEWAY_LOG_DIR -e GH_AW_MCP_LOG_DIR -e GH_AW_SAFE_OUTPUTS -e GH_AW_SAFE_OUTPUTS_CONFIG_PATH -e GH_AW_SAFE_OUTPUTS_TOOLS_PATH -e GH_AW_ASSETS_BRANCH -e GH_AW_ASSETS_MAX_SIZE_KB -e GH_AW_ASSETS_ALLOWED_EXTS -e DEFAULT_BRANCH -e GITHUB_MCP_SERVER_TOKEN -e GITHUB_MCP_GUARD_MIN_INTEGRITY -e GITHUB_MCP_GUARD_REPOS -e GITHUB_REPOSITORY -e GITHUB_SERVER_URL -e GITHUB_SHA -e GITHUB_WORKSPACE -e GITHUB_TOKEN -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e GITHUB_JOB -e GITHUB_ACTION -e GITHUB_EVENT_NAME -e GITHUB_EVENT_PATH -e GITHUB_ACTOR -e GITHUB_ACTOR_ID -e GITHUB_TRIGGERING_ACTOR -e GITHUB_WORKFLOW -e GITHUB_WORKFLOW_REF -e GITHUB_WORKFLOW_SHA -e GITHUB_REF -e GITHUB_REF_NAME -e GITHUB_REF_TYPE -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GH_AW_SAFE_OUTPUTS_PORT -e GH_AW_SAFE_OUTPUTS_API_KEY -v /tmp/gh-aw/mcp-payloads:/tmp/gh-aw/mcp-payloads:rw -v /opt:/opt:ro -v /tmp:/tmp:rw -v '"${GITHUB_WORKSPACE}"':'"${GITHUB_WORKSPACE}"':rw ghcr.io/github/gh-aw-mcpg:v0.1.19' + export MCP_GATEWAY_DOCKER_COMMAND='docker run -i --rm --network host -v /var/run/docker.sock:/var/run/docker.sock -e MCP_GATEWAY_PORT -e MCP_GATEWAY_DOMAIN -e MCP_GATEWAY_API_KEY -e MCP_GATEWAY_PAYLOAD_DIR -e MCP_GATEWAY_PAYLOAD_SIZE_THRESHOLD -e DEBUG -e MCP_GATEWAY_LOG_DIR -e GH_AW_MCP_LOG_DIR -e GH_AW_SAFE_OUTPUTS -e GH_AW_SAFE_OUTPUTS_CONFIG_PATH -e GH_AW_SAFE_OUTPUTS_TOOLS_PATH -e GH_AW_ASSETS_BRANCH -e GH_AW_ASSETS_MAX_SIZE_KB -e GH_AW_ASSETS_ALLOWED_EXTS -e DEFAULT_BRANCH -e GITHUB_MCP_SERVER_TOKEN -e GITHUB_MCP_GUARD_MIN_INTEGRITY -e GITHUB_MCP_GUARD_REPOS -e GITHUB_REPOSITORY -e GITHUB_SERVER_URL -e GITHUB_SHA -e GITHUB_WORKSPACE -e GITHUB_TOKEN -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e GITHUB_JOB -e GITHUB_ACTION -e GITHUB_EVENT_NAME -e GITHUB_EVENT_PATH -e GITHUB_ACTOR -e GITHUB_ACTOR_ID -e GITHUB_TRIGGERING_ACTOR -e GITHUB_WORKFLOW -e GITHUB_WORKFLOW_REF -e GITHUB_WORKFLOW_SHA -e GITHUB_REF -e GITHUB_REF_NAME -e GITHUB_REF_TYPE -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GH_AW_SAFE_OUTPUTS_PORT -e GH_AW_SAFE_OUTPUTS_API_KEY -v /tmp/gh-aw/mcp-payloads:/tmp/gh-aw/mcp-payloads:rw -v /opt:/opt:ro -v /tmp:/tmp:rw -v '"${GITHUB_WORKSPACE}"':'"${GITHUB_WORKSPACE}"':rw ghcr.io/github/gh-aw-mcpg:v0.2.19' mkdir -p /home/runner/.copilot - cat << GH_AW_MCP_CONFIG_EOF | bash ${RUNNER_TEMP}/gh-aw/actions/start_mcp_gateway.sh + cat << GH_AW_MCP_CONFIG_a5bd4f57a227c878_EOF | bash "${RUNNER_TEMP}/gh-aw/actions/start_mcp_gateway.sh" { "mcpServers": { "github": { @@ -614,7 +696,7 @@ jobs: "payloadDir": "${MCP_GATEWAY_PAYLOAD_DIR}" } } - GH_AW_MCP_CONFIG_EOF + GH_AW_MCP_CONFIG_a5bd4f57a227c878_EOF - name: Download activation artifact uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: @@ -622,7 +704,7 @@ jobs: path: /tmp/gh-aw - name: Clean git credentials continue-on-error: true - run: bash ${RUNNER_TEMP}/gh-aw/actions/clean_git_credentials.sh + run: bash "${RUNNER_TEMP}/gh-aw/actions/clean_git_credentials.sh" - name: Execute GitHub Copilot CLI id: agentic_execution # Copilot CLI tool arguments (sorted): @@ -630,9 +712,10 @@ jobs: run: | set -o pipefail touch /tmp/gh-aw/agent-step-summary.md + (umask 177 && touch /tmp/gh-aw/agent-stdio.log) # shellcheck disable=SC1003 - sudo -E awf --env-all --container-workdir "${GITHUB_WORKSPACE}" --mount "${RUNNER_TEMP}/gh-aw:${RUNNER_TEMP}/gh-aw:ro" --mount "${RUNNER_TEMP}/gh-aw:/host${RUNNER_TEMP}/gh-aw:ro" --allow-domains "api.business.githubcopilot.com,api.enterprise.githubcopilot.com,api.github.com,api.githubcopilot.com,api.individual.githubcopilot.com,api.snapcraft.io,archive.ubuntu.com,azure.archive.ubuntu.com,crl.geotrust.com,crl.globalsign.com,crl.identrust.com,crl.sectigo.com,crl.thawte.com,crl.usertrust.com,crl.verisign.com,crl3.digicert.com,crl4.digicert.com,crls.ssl.com,github.com,host.docker.internal,json-schema.org,json.schemastore.org,keyserver.ubuntu.com,ocsp.digicert.com,ocsp.geotrust.com,ocsp.globalsign.com,ocsp.identrust.com,ocsp.sectigo.com,ocsp.ssl.com,ocsp.thawte.com,ocsp.usertrust.com,ocsp.verisign.com,packagecloud.io,packages.cloud.google.com,packages.microsoft.com,ppa.launchpad.net,raw.githubusercontent.com,registry.npmjs.org,s.symcb.com,s.symcd.com,security.ubuntu.com,telemetry.enterprise.githubcopilot.com,ts-crl.ws.symantec.com,ts-ocsp.ws.symantec.com,www.googleapis.com" --log-level info --proxy-logs-dir /tmp/gh-aw/sandbox/firewall/logs --enable-host-access --image-tag 0.24.3 --skip-pull --enable-api-proxy \ - -- /bin/bash -c '/usr/local/bin/copilot --add-dir /tmp/gh-aw/ --log-level all --log-dir /tmp/gh-aw/sandbox/agent/logs/ --add-dir "${GITHUB_WORKSPACE}" --disable-builtin-mcps --allow-all-tools --allow-all-paths --prompt "$(cat /tmp/gh-aw/aw-prompts/prompt.txt)"' 2>&1 | tee -a /tmp/gh-aw/agent-stdio.log + sudo -E awf --container-workdir "${GITHUB_WORKSPACE}" --mount "${RUNNER_TEMP}/gh-aw:${RUNNER_TEMP}/gh-aw:ro" --mount "${RUNNER_TEMP}/gh-aw:/host${RUNNER_TEMP}/gh-aw:ro" --env-all --exclude-env COPILOT_GITHUB_TOKEN --exclude-env GITHUB_MCP_SERVER_TOKEN --exclude-env MCP_GATEWAY_API_KEY --allow-domains api.business.githubcopilot.com,api.enterprise.githubcopilot.com,api.github.com,api.githubcopilot.com,api.individual.githubcopilot.com,api.snapcraft.io,archive.ubuntu.com,azure.archive.ubuntu.com,crl.geotrust.com,crl.globalsign.com,crl.identrust.com,crl.sectigo.com,crl.thawte.com,crl.usertrust.com,crl.verisign.com,crl3.digicert.com,crl4.digicert.com,crls.ssl.com,github.com,host.docker.internal,json-schema.org,json.schemastore.org,keyserver.ubuntu.com,ocsp.digicert.com,ocsp.geotrust.com,ocsp.globalsign.com,ocsp.identrust.com,ocsp.sectigo.com,ocsp.ssl.com,ocsp.thawte.com,ocsp.usertrust.com,ocsp.verisign.com,packagecloud.io,packages.cloud.google.com,packages.microsoft.com,ppa.launchpad.net,raw.githubusercontent.com,registry.npmjs.org,s.symcb.com,s.symcd.com,security.ubuntu.com,telemetry.enterprise.githubcopilot.com,ts-crl.ws.symantec.com,ts-ocsp.ws.symantec.com,www.googleapis.com --log-level info --proxy-logs-dir /tmp/gh-aw/sandbox/firewall/logs --audit-dir /tmp/gh-aw/sandbox/firewall/audit --enable-host-access --image-tag 0.25.20 --skip-pull --enable-api-proxy \ + -- /bin/bash -c 'node ${RUNNER_TEMP}/gh-aw/actions/copilot_driver.cjs /usr/local/bin/copilot --add-dir /tmp/gh-aw/ --log-level all --log-dir /tmp/gh-aw/sandbox/agent/logs/ --disable-builtin-mcps --no-ask-user --allow-all-tools --allow-all-paths --add-dir "${GITHUB_WORKSPACE}" --prompt "$(cat /tmp/gh-aw/aw-prompts/prompt.txt)"' 2>&1 | tee -a /tmp/gh-aw/agent-stdio.log env: COPILOT_AGENT_RUNNER_TYPE: STANDALONE COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_GITHUB_TOKEN }} @@ -640,8 +723,8 @@ jobs: GH_AW_MCP_CONFIG: /home/runner/.copilot/mcp-config.json GH_AW_PHASE: agent GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt - GH_AW_SAFE_OUTPUTS: ${{ env.GH_AW_SAFE_OUTPUTS }} - GH_AW_VERSION: v0.62.2 + GH_AW_SAFE_OUTPUTS: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS }} + GH_AW_VERSION: v0.68.3 GITHUB_API_URL: ${{ github.api_url }} GITHUB_AW: true GITHUB_HEAD_REF: ${{ github.head_ref }} @@ -655,40 +738,28 @@ jobs: GIT_COMMITTER_EMAIL: github-actions[bot]@users.noreply.github.com GIT_COMMITTER_NAME: github-actions[bot] XDG_CONFIG_HOME: /home/runner - - name: Detect inference access error - id: detect-inference-error + - name: Detect Copilot errors + id: detect-copilot-errors if: always() continue-on-error: true - run: bash ${RUNNER_TEMP}/gh-aw/actions/detect_inference_access_error.sh + run: node "${RUNNER_TEMP}/gh-aw/actions/detect_copilot_errors.cjs" - name: Configure Git credentials env: REPO_NAME: ${{ github.repository }} SERVER_URL: ${{ github.server_url }} + GITHUB_TOKEN: ${{ github.token }} run: | git config --global user.email "github-actions[bot]@users.noreply.github.com" git config --global user.name "github-actions[bot]" git config --global am.keepcr true # Re-authenticate git with GitHub token SERVER_URL_STRIPPED="${SERVER_URL#https://}" - git remote set-url origin "https://x-access-token:${{ github.token }}@${SERVER_URL_STRIPPED}/${REPO_NAME}.git" + git remote set-url origin "https://x-access-token:${GITHUB_TOKEN}@${SERVER_URL_STRIPPED}/${REPO_NAME}.git" echo "Git configured with standard GitHub Actions identity" - name: Copy Copilot session state files to logs if: always() continue-on-error: true - run: | - # Copy Copilot session state files to logs folder for artifact collection - # This ensures they are in /tmp/gh-aw/ where secret redaction can scan them - SESSION_STATE_DIR="$HOME/.copilot/session-state" - LOGS_DIR="/tmp/gh-aw/sandbox/agent/logs" - - if [ -d "$SESSION_STATE_DIR" ]; then - echo "Copying Copilot session state files from $SESSION_STATE_DIR to $LOGS_DIR" - mkdir -p "$LOGS_DIR" - cp -v "$SESSION_STATE_DIR"/*.jsonl "$LOGS_DIR/" 2>/dev/null || true - echo "Session state files copied successfully" - else - echo "No session-state directory found at $SESSION_STATE_DIR" - fi + run: bash "${RUNNER_TEMP}/gh-aw/actions/copy_copilot_session_state.sh" - name: Stop MCP Gateway if: always() continue-on-error: true @@ -697,14 +768,14 @@ jobs: MCP_GATEWAY_API_KEY: ${{ steps.start-mcp-gateway.outputs.gateway-api-key }} GATEWAY_PID: ${{ steps.start-mcp-gateway.outputs.gateway-pid }} run: | - bash ${RUNNER_TEMP}/gh-aw/actions/stop_mcp_gateway.sh "$GATEWAY_PID" + bash "${RUNNER_TEMP}/gh-aw/actions/stop_mcp_gateway.sh" "$GATEWAY_PID" - name: Redact secrets in logs if: always() - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 with: script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); + setupGlobals(core, github, context, exec, io, getOctokit); const { main } = require('${{ runner.temp }}/gh-aw/actions/redact_secrets.cjs'); await main(); env: @@ -715,18 +786,20 @@ jobs: SECRET_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - name: Append agent step summary if: always() - run: bash ${RUNNER_TEMP}/gh-aw/actions/append_agent_step_summary.sh + run: bash "${RUNNER_TEMP}/gh-aw/actions/append_agent_step_summary.sh" - name: Copy Safe Outputs if: always() + env: + GH_AW_SAFE_OUTPUTS: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS }} run: | mkdir -p /tmp/gh-aw cp "$GH_AW_SAFE_OUTPUTS" /tmp/gh-aw/safeoutputs.jsonl 2>/dev/null || true - name: Ingest agent output id: collect_output if: always() - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 env: - GH_AW_SAFE_OUTPUTS: ${{ env.GH_AW_SAFE_OUTPUTS }} + GH_AW_SAFE_OUTPUTS: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS }} GH_AW_ALLOWED_DOMAINS: "api.business.githubcopilot.com,api.enterprise.githubcopilot.com,api.github.com,api.githubcopilot.com,api.individual.githubcopilot.com,api.snapcraft.io,archive.ubuntu.com,azure.archive.ubuntu.com,crl.geotrust.com,crl.globalsign.com,crl.identrust.com,crl.sectigo.com,crl.thawte.com,crl.usertrust.com,crl.verisign.com,crl3.digicert.com,crl4.digicert.com,crls.ssl.com,github.com,host.docker.internal,json-schema.org,json.schemastore.org,keyserver.ubuntu.com,ocsp.digicert.com,ocsp.geotrust.com,ocsp.globalsign.com,ocsp.identrust.com,ocsp.sectigo.com,ocsp.ssl.com,ocsp.thawte.com,ocsp.usertrust.com,ocsp.verisign.com,packagecloud.io,packages.cloud.google.com,packages.microsoft.com,ppa.launchpad.net,raw.githubusercontent.com,registry.npmjs.org,s.symcb.com,s.symcd.com,security.ubuntu.com,telemetry.enterprise.githubcopilot.com,ts-crl.ws.symantec.com,ts-ocsp.ws.symantec.com,www.googleapis.com" GITHUB_SERVER_URL: ${{ github.server_url }} GITHUB_API_URL: ${{ github.api_url }} @@ -734,27 +807,28 @@ jobs: with: script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); + setupGlobals(core, github, context, exec, io, getOctokit); const { main } = require('${{ runner.temp }}/gh-aw/actions/collect_ndjson_output.cjs'); await main(); - name: Parse agent logs for step summary if: always() - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 env: GH_AW_AGENT_OUTPUT: /tmp/gh-aw/sandbox/agent/logs/ with: script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); + setupGlobals(core, github, context, exec, io, getOctokit); const { main } = require('${{ runner.temp }}/gh-aw/actions/parse_copilot_log.cjs'); await main(); - name: Parse MCP Gateway logs for step summary if: always() - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + id: parse-mcp-gateway + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 with: script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); + setupGlobals(core, github, context, exec, io, getOctokit); const { main } = require('${{ runner.temp }}/gh-aw/actions/parse_mcp_gateway_log.cjs'); await main(); - name: Print firewall logs @@ -772,10 +846,26 @@ jobs: else echo 'AWF binary not installed, skipping firewall log summary' fi + - name: Parse token usage for step summary + if: always() + continue-on-error: true + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 + with: + script: | + const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); + setupGlobals(core, github, context, exec, io, getOctokit); + const { main } = require('${{ runner.temp }}/gh-aw/actions/parse_token_usage.cjs'); + await main(); + - name: Write agent output placeholder if missing + if: always() + run: | + if [ ! -f /tmp/gh-aw/agent_output.json ]; then + echo '{"items":[]}' > /tmp/gh-aw/agent_output.json + fi - name: Upload agent artifacts if: always() continue-on-error: true - uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: agent path: | @@ -783,143 +873,27 @@ jobs: /tmp/gh-aw/sandbox/agent/logs/ /tmp/gh-aw/redacted-urls.log /tmp/gh-aw/mcp-logs/ - /tmp/gh-aw/sandbox/firewall/logs/ + /tmp/gh-aw/agent_usage.json /tmp/gh-aw/agent-stdio.log /tmp/gh-aw/agent/ + /tmp/gh-aw/github_rate_limits.jsonl /tmp/gh-aw/safeoutputs.jsonl /tmp/gh-aw/agent_output.json + /tmp/gh-aw/aw-*.patch + /tmp/gh-aw/aw-*.bundle + /tmp/gh-aw/sandbox/firewall/logs/ + /tmp/gh-aw/sandbox/firewall/audit/ if-no-files-found: ignore - # --- Threat Detection (inline) --- - - name: Check if detection needed - id: detection_guard - if: always() - env: - OUTPUT_TYPES: ${{ steps.collect_output.outputs.output_types }} - HAS_PATCH: ${{ steps.collect_output.outputs.has_patch }} - run: | - if [[ -n "$OUTPUT_TYPES" || "$HAS_PATCH" == "true" ]]; then - echo "run_detection=true" >> "$GITHUB_OUTPUT" - echo "Detection will run: output_types=$OUTPUT_TYPES, has_patch=$HAS_PATCH" - else - echo "run_detection=false" >> "$GITHUB_OUTPUT" - echo "Detection skipped: no agent outputs or patches to analyze" - fi - - name: Clear MCP configuration for detection - if: always() && steps.detection_guard.outputs.run_detection == 'true' - run: | - rm -f /tmp/gh-aw/mcp-config/mcp-servers.json - rm -f /home/runner/.copilot/mcp-config.json - rm -f "$GITHUB_WORKSPACE/.gemini/settings.json" - - name: Prepare threat detection files - if: always() && steps.detection_guard.outputs.run_detection == 'true' - run: | - mkdir -p /tmp/gh-aw/threat-detection/aw-prompts - cp /tmp/gh-aw/aw-prompts/prompt.txt /tmp/gh-aw/threat-detection/aw-prompts/prompt.txt 2>/dev/null || true - cp /tmp/gh-aw/agent_output.json /tmp/gh-aw/threat-detection/agent_output.json 2>/dev/null || true - for f in /tmp/gh-aw/aw-*.patch; do - [ -f "$f" ] && cp "$f" /tmp/gh-aw/threat-detection/ 2>/dev/null || true - done - echo "Prepared threat detection files:" - ls -la /tmp/gh-aw/threat-detection/ 2>/dev/null || true - - name: Setup threat detection - if: always() && steps.detection_guard.outputs.run_detection == 'true' - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 - env: - WORKFLOW_NAME: "Evaluate PR Tests" - WORKFLOW_DESCRIPTION: "Evaluates test quality, coverage, and appropriateness on PRs that add or modify tests" - HAS_PATCH: ${{ steps.collect_output.outputs.has_patch }} - with: - script: | - const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); - const { main } = require('${{ runner.temp }}/gh-aw/actions/setup_threat_detection.cjs'); - await main(); - - name: Ensure threat-detection directory and log - if: always() && steps.detection_guard.outputs.run_detection == 'true' - run: | - mkdir -p /tmp/gh-aw/threat-detection - touch /tmp/gh-aw/threat-detection/detection.log - - name: Execute GitHub Copilot CLI - if: always() && steps.detection_guard.outputs.run_detection == 'true' - id: detection_agentic_execution - # Copilot CLI tool arguments (sorted): - # --allow-tool shell(cat) - # --allow-tool shell(grep) - # --allow-tool shell(head) - # --allow-tool shell(jq) - # --allow-tool shell(ls) - # --allow-tool shell(tail) - # --allow-tool shell(wc) - timeout-minutes: 20 - run: | - set -o pipefail - touch /tmp/gh-aw/agent-step-summary.md - # shellcheck disable=SC1003 - sudo -E awf --env-all --container-workdir "${GITHUB_WORKSPACE}" --mount "${RUNNER_TEMP}/gh-aw:${RUNNER_TEMP}/gh-aw:ro" --mount "${RUNNER_TEMP}/gh-aw:/host${RUNNER_TEMP}/gh-aw:ro" --allow-domains "api.business.githubcopilot.com,api.enterprise.githubcopilot.com,api.github.com,api.githubcopilot.com,api.individual.githubcopilot.com,github.com,host.docker.internal,raw.githubusercontent.com,registry.npmjs.org,telemetry.enterprise.githubcopilot.com" --log-level info --proxy-logs-dir /tmp/gh-aw/sandbox/firewall/logs --enable-host-access --image-tag 0.24.3 --skip-pull --enable-api-proxy \ - -- /bin/bash -c '/usr/local/bin/copilot --add-dir /tmp/gh-aw/ --log-level all --log-dir /tmp/gh-aw/sandbox/agent/logs/ --add-dir "${GITHUB_WORKSPACE}" --disable-builtin-mcps --allow-tool '\''shell(cat)'\'' --allow-tool '\''shell(grep)'\'' --allow-tool '\''shell(head)'\'' --allow-tool '\''shell(jq)'\'' --allow-tool '\''shell(ls)'\'' --allow-tool '\''shell(tail)'\'' --allow-tool '\''shell(wc)'\'' --prompt "$(cat /tmp/gh-aw/aw-prompts/prompt.txt)"' 2>&1 | tee -a /tmp/gh-aw/threat-detection/detection.log - env: - COPILOT_AGENT_RUNNER_TYPE: STANDALONE - COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_GITHUB_TOKEN }} - COPILOT_MODEL: claude-sonnet-4.6 - GH_AW_PHASE: detection - GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt - GH_AW_VERSION: v0.62.2 - GITHUB_API_URL: ${{ github.api_url }} - GITHUB_AW: true - GITHUB_HEAD_REF: ${{ github.head_ref }} - GITHUB_REF_NAME: ${{ github.ref_name }} - GITHUB_SERVER_URL: ${{ github.server_url }} - GITHUB_STEP_SUMMARY: /tmp/gh-aw/agent-step-summary.md - GITHUB_WORKSPACE: ${{ github.workspace }} - GIT_AUTHOR_EMAIL: github-actions[bot]@users.noreply.github.com - GIT_AUTHOR_NAME: github-actions[bot] - GIT_COMMITTER_EMAIL: github-actions[bot]@users.noreply.github.com - GIT_COMMITTER_NAME: github-actions[bot] - XDG_CONFIG_HOME: /home/runner - - name: Parse threat detection results - id: parse_detection_results - if: always() && steps.detection_guard.outputs.run_detection == 'true' - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 - with: - script: | - const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); - const { main } = require('${{ runner.temp }}/gh-aw/actions/parse_threat_detection_results.cjs'); - await main(); - - name: Upload threat detection log - if: always() && steps.detection_guard.outputs.run_detection == 'true' - uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0 - with: - name: detection - path: /tmp/gh-aw/threat-detection/detection.log - if-no-files-found: ignore - - name: Set detection conclusion - id: detection_conclusion - if: always() - env: - RUN_DETECTION: ${{ steps.detection_guard.outputs.run_detection }} - DETECTION_SUCCESS: ${{ steps.parse_detection_results.outputs.success }} - run: | - if [[ "$RUN_DETECTION" != "true" ]]; then - echo "conclusion=skipped" >> "$GITHUB_OUTPUT" - echo "success=true" >> "$GITHUB_OUTPUT" - echo "Detection was not needed, marking as skipped" - elif [[ "$DETECTION_SUCCESS" == "true" ]]; then - echo "conclusion=success" >> "$GITHUB_OUTPUT" - echo "success=true" >> "$GITHUB_OUTPUT" - echo "Detection passed successfully" - else - echo "conclusion=failure" >> "$GITHUB_OUTPUT" - echo "success=false" >> "$GITHUB_OUTPUT" - echo "Detection found issues" - fi conclusion: needs: - activation - agent + - detection - safe_outputs - if: (always()) && ((needs.agent.result != 'skipped') || (needs.activation.outputs.lockdown_check_failed == 'true')) + if: > + always() && (needs.agent.result != 'skipped' || needs.activation.outputs.lockdown_check_failed == 'true' || + needs.activation.outputs.stale_lock_file_failed == 'true') runs-on: ubuntu-slim permissions: contents: read @@ -930,14 +904,18 @@ jobs: group: "gh-aw-conclusion-copilot-evaluate-tests" cancel-in-progress: false outputs: + incomplete_count: ${{ steps.report_incomplete.outputs.incomplete_count }} noop_message: ${{ steps.noop.outputs.noop_message }} tools_reported: ${{ steps.missing_tool.outputs.tools_reported }} total_count: ${{ steps.missing_tool.outputs.total_count }} steps: - name: Setup Scripts - uses: github/gh-aw-actions/setup@20045bbd5ad2632b9809856c389708eab1bd16ef # v0.62.2 + id: setup + uses: github/gh-aw-actions/setup@ba90f2186d7ad780ec640f364005fa24e797b360 # v0.68.3 with: destination: ${{ runner.temp }}/gh-aw/actions + job-name: ${{ github.job }} + trace-id: ${{ needs.activation.outputs.setup-trace-id }} - name: Download agent output artifact id: download-agent-output continue-on-error: true @@ -946,52 +924,92 @@ jobs: name: agent path: /tmp/gh-aw/ - name: Setup agent output environment variable + id: setup-agent-output-env if: steps.download-agent-output.outcome == 'success' run: | mkdir -p /tmp/gh-aw/ find "/tmp/gh-aw/" -type f -print - echo "GH_AW_AGENT_OUTPUT=/tmp/gh-aw/agent_output.json" >> "$GITHUB_ENV" - - name: Process No-Op Messages + echo "GH_AW_AGENT_OUTPUT=/tmp/gh-aw/agent_output.json" >> "$GITHUB_OUTPUT" + - name: Process no-op messages id: noop - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 env: - GH_AW_AGENT_OUTPUT: ${{ env.GH_AW_AGENT_OUTPUT }} + GH_AW_AGENT_OUTPUT: ${{ steps.setup-agent-output-env.outputs.GH_AW_AGENT_OUTPUT }} GH_AW_NOOP_MAX: "1" GH_AW_WORKFLOW_NAME: "Evaluate PR Tests" + GH_AW_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + GH_AW_AGENT_CONCLUSION: ${{ needs.agent.result }} + GH_AW_NOOP_REPORT_AS_ISSUE: "false" + with: + github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} + script: | + const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); + setupGlobals(core, github, context, exec, io, getOctokit); + const { main } = require('${{ runner.temp }}/gh-aw/actions/handle_noop_message.cjs'); + await main(); + - name: Log detection run + id: detection_runs + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 + env: + GH_AW_AGENT_OUTPUT: ${{ steps.setup-agent-output-env.outputs.GH_AW_AGENT_OUTPUT }} + GH_AW_WORKFLOW_NAME: "Evaluate PR Tests" + GH_AW_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + GH_AW_DETECTION_CONCLUSION: ${{ needs.detection.outputs.detection_conclusion }} + GH_AW_DETECTION_REASON: ${{ needs.detection.outputs.detection_reason }} with: github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); - const { main } = require('${{ runner.temp }}/gh-aw/actions/noop.cjs'); + setupGlobals(core, github, context, exec, io, getOctokit); + const { main } = require('${{ runner.temp }}/gh-aw/actions/handle_detection_runs.cjs'); await main(); - - name: Record Missing Tool + - name: Record missing tool id: missing_tool - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 env: - GH_AW_AGENT_OUTPUT: ${{ env.GH_AW_AGENT_OUTPUT }} + GH_AW_AGENT_OUTPUT: ${{ steps.setup-agent-output-env.outputs.GH_AW_AGENT_OUTPUT }} + GH_AW_MISSING_TOOL_CREATE_ISSUE: "true" GH_AW_WORKFLOW_NAME: "Evaluate PR Tests" with: github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); + setupGlobals(core, github, context, exec, io, getOctokit); const { main } = require('${{ runner.temp }}/gh-aw/actions/missing_tool.cjs'); await main(); - - name: Handle Agent Failure + - name: Record incomplete + id: report_incomplete + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 + env: + GH_AW_AGENT_OUTPUT: ${{ steps.setup-agent-output-env.outputs.GH_AW_AGENT_OUTPUT }} + GH_AW_REPORT_INCOMPLETE_CREATE_ISSUE: "true" + GH_AW_WORKFLOW_NAME: "Evaluate PR Tests" + with: + github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} + script: | + const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); + setupGlobals(core, github, context, exec, io, getOctokit); + const { main } = require('${{ runner.temp }}/gh-aw/actions/report_incomplete_handler.cjs'); + await main(); + - name: Handle agent failure id: handle_agent_failure if: always() - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 env: - GH_AW_AGENT_OUTPUT: ${{ env.GH_AW_AGENT_OUTPUT }} + GH_AW_AGENT_OUTPUT: ${{ steps.setup-agent-output-env.outputs.GH_AW_AGENT_OUTPUT }} GH_AW_WORKFLOW_NAME: "Evaluate PR Tests" GH_AW_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} GH_AW_AGENT_CONCLUSION: ${{ needs.agent.result }} GH_AW_WORKFLOW_ID: "copilot-evaluate-tests" + GH_AW_ENGINE_ID: "copilot" GH_AW_SECRET_VERIFICATION_RESULT: ${{ needs.activation.outputs.secret_verification_result }} GH_AW_CHECKOUT_PR_SUCCESS: ${{ needs.agent.outputs.checkout_pr_success }} GH_AW_INFERENCE_ACCESS_ERROR: ${{ needs.agent.outputs.inference_access_error }} + GH_AW_MCP_POLICY_ERROR: ${{ needs.agent.outputs.mcp_policy_error }} + GH_AW_AGENTIC_ENGINE_TIMEOUT: ${{ needs.agent.outputs.agentic_engine_timeout }} + GH_AW_MODEL_NOT_SUPPORTED_ERROR: ${{ needs.agent.outputs.model_not_supported_error }} GH_AW_LOCKDOWN_CHECK_FAILED: ${{ needs.activation.outputs.lockdown_check_failed }} + GH_AW_STALE_LOCK_FILE_FAILED: ${{ needs.activation.outputs.stale_lock_file_failed }} GH_AW_SAFE_OUTPUT_MESSAGES: "{\"footer\":\"\\u003e ๐Ÿงช *Test evaluation by [{workflow_name}]({run_url})*\",\"runStarted\":\"๐Ÿ”ฌ Evaluating tests on this PRโ€ฆ [{workflow_name}]({run_url})\",\"runSuccess\":\"โœ… Test evaluation complete! [{workflow_name}]({run_url})\",\"runFailure\":\"โŒ Test evaluation failed. [{workflow_name}]({run_url}) {status}\"}" GH_AW_GROUP_REPORTS: "false" GH_AW_FAILURE_REPORT_AS_ISSUE: "true" @@ -1000,85 +1018,232 @@ jobs: github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); + setupGlobals(core, github, context, exec, io, getOctokit); const { main } = require('${{ runner.temp }}/gh-aw/actions/handle_agent_failure.cjs'); await main(); - - name: Handle No-Op Message - id: handle_noop_message - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 - env: - GH_AW_AGENT_OUTPUT: ${{ env.GH_AW_AGENT_OUTPUT }} - GH_AW_WORKFLOW_NAME: "Evaluate PR Tests" - GH_AW_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} - GH_AW_AGENT_CONCLUSION: ${{ needs.agent.result }} - GH_AW_NOOP_MESSAGE: ${{ steps.noop.outputs.noop_message }} - GH_AW_NOOP_REPORT_AS_ISSUE: "false" - with: - github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} - script: | - const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); - const { main } = require('${{ runner.temp }}/gh-aw/actions/handle_noop_message.cjs'); - await main(); - name: Update reaction comment with completion status id: conclusion - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 env: - GH_AW_AGENT_OUTPUT: ${{ env.GH_AW_AGENT_OUTPUT }} + GH_AW_AGENT_OUTPUT: ${{ steps.setup-agent-output-env.outputs.GH_AW_AGENT_OUTPUT }} GH_AW_COMMENT_ID: ${{ needs.activation.outputs.comment_id }} GH_AW_COMMENT_REPO: ${{ needs.activation.outputs.comment_repo }} GH_AW_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} GH_AW_WORKFLOW_NAME: "Evaluate PR Tests" GH_AW_AGENT_CONCLUSION: ${{ needs.agent.result }} - GH_AW_DETECTION_CONCLUSION: ${{ needs.agent.outputs.detection_conclusion }} + GH_AW_DETECTION_CONCLUSION: ${{ needs.detection.outputs.detection_conclusion }} + GH_AW_DETECTION_REASON: ${{ needs.detection.outputs.detection_reason }} GH_AW_SAFE_OUTPUT_MESSAGES: "{\"footer\":\"\\u003e ๐Ÿงช *Test evaluation by [{workflow_name}]({run_url})*\",\"runStarted\":\"๐Ÿ”ฌ Evaluating tests on this PRโ€ฆ [{workflow_name}]({run_url})\",\"runSuccess\":\"โœ… Test evaluation complete! [{workflow_name}]({run_url})\",\"runFailure\":\"โŒ Test evaluation failed. [{workflow_name}]({run_url}) {status}\"}" with: github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); + setupGlobals(core, github, context, exec, io, getOctokit); const { main } = require('${{ runner.temp }}/gh-aw/actions/notify_comment_error.cjs'); await main(); + detection: + needs: + - activation + - agent + if: > + always() && needs.agent.result != 'skipped' && (needs.agent.outputs.output_types != '' || needs.agent.outputs.has_patch == 'true') + runs-on: ubuntu-latest + permissions: + contents: read + outputs: + detection_conclusion: ${{ steps.detection_conclusion.outputs.conclusion }} + detection_reason: ${{ steps.detection_conclusion.outputs.reason }} + detection_success: ${{ steps.detection_conclusion.outputs.success }} + steps: + - name: Setup Scripts + id: setup + uses: github/gh-aw-actions/setup@ba90f2186d7ad780ec640f364005fa24e797b360 # v0.68.3 + with: + destination: ${{ runner.temp }}/gh-aw/actions + job-name: ${{ github.job }} + trace-id: ${{ needs.activation.outputs.setup-trace-id }} + - name: Download agent output artifact + id: download-agent-output + continue-on-error: true + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: agent + path: /tmp/gh-aw/ + - name: Setup agent output environment variable + id: setup-agent-output-env + if: steps.download-agent-output.outcome == 'success' + run: | + mkdir -p /tmp/gh-aw/ + find "/tmp/gh-aw/" -type f -print + echo "GH_AW_AGENT_OUTPUT=/tmp/gh-aw/agent_output.json" >> "$GITHUB_OUTPUT" + - name: Checkout repository for patch context + if: needs.agent.outputs.has_patch == 'true' + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + persist-credentials: false + # --- Threat Detection --- + - name: Clean stale firewall files from agent artifact + run: | + rm -rf /tmp/gh-aw/sandbox/firewall/logs + rm -rf /tmp/gh-aw/sandbox/firewall/audit + - name: Download container images + run: bash "${RUNNER_TEMP}/gh-aw/actions/download_docker_images.sh" ghcr.io/github/gh-aw-firewall/agent:0.25.20 ghcr.io/github/gh-aw-firewall/api-proxy:0.25.20 ghcr.io/github/gh-aw-firewall/squid:0.25.20 + - name: Check if detection needed + id: detection_guard + if: always() + env: + OUTPUT_TYPES: ${{ needs.agent.outputs.output_types }} + HAS_PATCH: ${{ needs.agent.outputs.has_patch }} + run: | + if [[ -n "$OUTPUT_TYPES" || "$HAS_PATCH" == "true" ]]; then + echo "run_detection=true" >> "$GITHUB_OUTPUT" + echo "Detection will run: output_types=$OUTPUT_TYPES, has_patch=$HAS_PATCH" + else + echo "run_detection=false" >> "$GITHUB_OUTPUT" + echo "Detection skipped: no agent outputs or patches to analyze" + fi + - name: Clear MCP configuration for detection + if: always() && steps.detection_guard.outputs.run_detection == 'true' + run: | + rm -f /tmp/gh-aw/mcp-config/mcp-servers.json + rm -f /home/runner/.copilot/mcp-config.json + rm -f "$GITHUB_WORKSPACE/.gemini/settings.json" + - name: Prepare threat detection files + if: always() && steps.detection_guard.outputs.run_detection == 'true' + run: | + mkdir -p /tmp/gh-aw/threat-detection/aw-prompts + cp /tmp/gh-aw/aw-prompts/prompt.txt /tmp/gh-aw/threat-detection/aw-prompts/prompt.txt 2>/dev/null || true + cp /tmp/gh-aw/agent_output.json /tmp/gh-aw/threat-detection/agent_output.json 2>/dev/null || true + for f in /tmp/gh-aw/aw-*.patch; do + [ -f "$f" ] && cp "$f" /tmp/gh-aw/threat-detection/ 2>/dev/null || true + done + for f in /tmp/gh-aw/aw-*.bundle; do + [ -f "$f" ] && cp "$f" /tmp/gh-aw/threat-detection/ 2>/dev/null || true + done + echo "Prepared threat detection files:" + ls -la /tmp/gh-aw/threat-detection/ 2>/dev/null || true + - name: Setup threat detection + if: always() && steps.detection_guard.outputs.run_detection == 'true' + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 + env: + WORKFLOW_NAME: "Evaluate PR Tests" + WORKFLOW_DESCRIPTION: "Evaluates test quality, coverage, and appropriateness on PRs that add or modify tests" + HAS_PATCH: ${{ needs.agent.outputs.has_patch }} + with: + script: | + const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); + setupGlobals(core, github, context, exec, io, getOctokit); + const { main } = require('${{ runner.temp }}/gh-aw/actions/setup_threat_detection.cjs'); + await main(); + - name: Ensure threat-detection directory and log + if: always() && steps.detection_guard.outputs.run_detection == 'true' + run: | + mkdir -p /tmp/gh-aw/threat-detection + touch /tmp/gh-aw/threat-detection/detection.log + - name: Install GitHub Copilot CLI + run: bash "${RUNNER_TEMP}/gh-aw/actions/install_copilot_cli.sh" 1.0.21 + env: + GH_HOST: github.com + - name: Install AWF binary + run: bash "${RUNNER_TEMP}/gh-aw/actions/install_awf_binary.sh" v0.25.20 + - name: Execute GitHub Copilot CLI + if: always() && steps.detection_guard.outputs.run_detection == 'true' + id: detection_agentic_execution + # Copilot CLI tool arguments (sorted): + timeout-minutes: 20 + run: | + set -o pipefail + touch /tmp/gh-aw/agent-step-summary.md + (umask 177 && touch /tmp/gh-aw/threat-detection/detection.log) + # shellcheck disable=SC1003 + sudo -E awf --container-workdir "${GITHUB_WORKSPACE}" --mount "${RUNNER_TEMP}/gh-aw:${RUNNER_TEMP}/gh-aw:ro" --mount "${RUNNER_TEMP}/gh-aw:/host${RUNNER_TEMP}/gh-aw:ro" --env-all --exclude-env COPILOT_GITHUB_TOKEN --allow-domains api.business.githubcopilot.com,api.enterprise.githubcopilot.com,api.github.com,api.githubcopilot.com,api.individual.githubcopilot.com,github.com,host.docker.internal,telemetry.enterprise.githubcopilot.com --log-level info --proxy-logs-dir /tmp/gh-aw/sandbox/firewall/logs --audit-dir /tmp/gh-aw/sandbox/firewall/audit --enable-host-access --image-tag 0.25.20 --skip-pull --enable-api-proxy \ + -- /bin/bash -c 'node ${RUNNER_TEMP}/gh-aw/actions/copilot_driver.cjs /usr/local/bin/copilot --add-dir /tmp/gh-aw/ --log-level all --log-dir /tmp/gh-aw/sandbox/agent/logs/ --disable-builtin-mcps --no-ask-user --allow-all-tools --add-dir "${GITHUB_WORKSPACE}" --prompt "$(cat /tmp/gh-aw/aw-prompts/prompt.txt)"' 2>&1 | tee -a /tmp/gh-aw/threat-detection/detection.log + env: + COPILOT_AGENT_RUNNER_TYPE: STANDALONE + COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_GITHUB_TOKEN }} + COPILOT_MODEL: claude-sonnet-4.6 + GH_AW_PHASE: detection + GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt + GH_AW_VERSION: v0.68.3 + GITHUB_API_URL: ${{ github.api_url }} + GITHUB_AW: true + GITHUB_HEAD_REF: ${{ github.head_ref }} + GITHUB_REF_NAME: ${{ github.ref_name }} + GITHUB_SERVER_URL: ${{ github.server_url }} + GITHUB_STEP_SUMMARY: /tmp/gh-aw/agent-step-summary.md + GITHUB_WORKSPACE: ${{ github.workspace }} + GIT_AUTHOR_EMAIL: github-actions[bot]@users.noreply.github.com + GIT_AUTHOR_NAME: github-actions[bot] + GIT_COMMITTER_EMAIL: github-actions[bot]@users.noreply.github.com + GIT_COMMITTER_NAME: github-actions[bot] + XDG_CONFIG_HOME: /home/runner + - name: Upload threat detection log + if: always() && steps.detection_guard.outputs.run_detection == 'true' + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: detection + path: /tmp/gh-aw/threat-detection/detection.log + if-no-files-found: ignore + - name: Parse and conclude threat detection + id: detection_conclusion + if: always() + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 + env: + RUN_DETECTION: ${{ steps.detection_guard.outputs.run_detection }} + GH_AW_DETECTION_CONTINUE_ON_ERROR: "true" + with: + script: | + const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); + setupGlobals(core, github, context, exec, io, getOctokit); + const { main } = require('${{ runner.temp }}/gh-aw/actions/parse_threat_detection_results.cjs'); + await main(); + pre_activation: if: github.event_name == 'issue_comment' || github.event_name == 'workflow_dispatch' runs-on: ubuntu-slim outputs: - activated: ${{ (steps.check_membership.outputs.is_team_member == 'true') && (steps.check_command_position.outputs.command_position_ok == 'true') }} + activated: ${{ steps.check_membership.outputs.is_team_member == 'true' && steps.check_command_position.outputs.command_position_ok == 'true' }} matched_command: ${{ steps.check_command_position.outputs.matched_command }} + setup-trace-id: ${{ steps.setup.outputs.trace-id }} steps: - name: Setup Scripts - uses: github/gh-aw-actions/setup@20045bbd5ad2632b9809856c389708eab1bd16ef # v0.62.2 + id: setup + uses: github/gh-aw-actions/setup@ba90f2186d7ad780ec640f364005fa24e797b360 # v0.68.3 with: destination: ${{ runner.temp }}/gh-aw/actions + job-name: ${{ github.job }} - name: Check team membership for command workflow id: check_membership - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 env: - GH_AW_REQUIRED_ROLES: admin,maintainer,write - GH_AW_ALLOWED_BOTS: copilot-swe-agent[bot] + GH_AW_REQUIRED_ROLES: "admin,maintain,write" + GH_AW_ALLOWED_BOTS: "copilot-swe-agent[bot]" with: github-token: ${{ secrets.GITHUB_TOKEN }} script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); + setupGlobals(core, github, context, exec, io, getOctokit); const { main } = require('${{ runner.temp }}/gh-aw/actions/check_membership.cjs'); await main(); - name: Check command position id: check_command_position - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 env: GH_AW_COMMANDS: "[\"evaluate-tests\"]" with: script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); + setupGlobals(core, github, context, exec, io, getOctokit); const { main } = require('${{ runner.temp }}/gh-aw/actions/check_command_position.cjs'); await main(); safe_outputs: - needs: agent - if: ((!cancelled()) && (needs.agent.result != 'skipped')) && (needs.agent.outputs.detection_success == 'true') + needs: + - activation + - agent + - detection + if: (!cancelled()) && needs.agent.result != 'skipped' && needs.detection.result == 'success' runs-on: ubuntu-slim permissions: contents: read @@ -1088,6 +1253,9 @@ jobs: timeout-minutes: 15 env: GH_AW_CALLER_WORKFLOW_ID: "${{ github.repository }}/copilot-evaluate-tests" + GH_AW_DETECTION_CONCLUSION: ${{ needs.detection.outputs.detection_conclusion }} + GH_AW_DETECTION_REASON: ${{ needs.detection.outputs.detection_reason }} + GH_AW_EFFECTIVE_TOKENS: ${{ needs.agent.outputs.effective_tokens }} GH_AW_ENGINE_ID: "copilot" GH_AW_ENGINE_MODEL: "claude-sonnet-4.6" GH_AW_SAFE_OUTPUT_MESSAGES: "{\"footer\":\"\\u003e ๐Ÿงช *Test evaluation by [{workflow_name}]({run_url})*\",\"runStarted\":\"๐Ÿ”ฌ Evaluating tests on this PRโ€ฆ [{workflow_name}]({run_url})\",\"runSuccess\":\"โœ… Test evaluation complete! [{workflow_name}]({run_url})\",\"runFailure\":\"โŒ Test evaluation failed. [{workflow_name}]({run_url}) {status}\"}" @@ -1104,9 +1272,12 @@ jobs: process_safe_outputs_temporary_id_map: ${{ steps.process_safe_outputs.outputs.temporary_id_map }} steps: - name: Setup Scripts - uses: github/gh-aw-actions/setup@20045bbd5ad2632b9809856c389708eab1bd16ef # v0.62.2 + id: setup + uses: github/gh-aw-actions/setup@ba90f2186d7ad780ec640f364005fa24e797b360 # v0.68.3 with: destination: ${{ runner.temp }}/gh-aw/actions + job-name: ${{ github.job }} + trace-id: ${{ needs.activation.outputs.setup-trace-id }} - name: Download agent output artifact id: download-agent-output continue-on-error: true @@ -1115,12 +1286,14 @@ jobs: name: agent path: /tmp/gh-aw/ - name: Setup agent output environment variable + id: setup-agent-output-env if: steps.download-agent-output.outcome == 'success' run: | mkdir -p /tmp/gh-aw/ find "/tmp/gh-aw/" -type f -print - echo "GH_AW_AGENT_OUTPUT=/tmp/gh-aw/agent_output.json" >> "$GITHUB_ENV" + echo "GH_AW_AGENT_OUTPUT=/tmp/gh-aw/agent_output.json" >> "$GITHUB_OUTPUT" - name: Configure GH_HOST for enterprise compatibility + id: ghes-host-config shell: bash run: | # Derive GH_HOST from GITHUB_SERVER_URL so the gh CLI targets the correct @@ -1130,25 +1303,27 @@ jobs: echo "GH_HOST=${GH_HOST}" >> "$GITHUB_ENV" - name: Process Safe Outputs id: process_safe_outputs - uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8 + uses: actions/github-script@373c709c69115d41ff229c7e5df9f8788daa9553 # v9 env: - GH_AW_AGENT_OUTPUT: ${{ env.GH_AW_AGENT_OUTPUT }} + GH_AW_AGENT_OUTPUT: ${{ steps.setup-agent-output-env.outputs.GH_AW_AGENT_OUTPUT }} GH_AW_ALLOWED_DOMAINS: "api.business.githubcopilot.com,api.enterprise.githubcopilot.com,api.github.com,api.githubcopilot.com,api.individual.githubcopilot.com,api.snapcraft.io,archive.ubuntu.com,azure.archive.ubuntu.com,crl.geotrust.com,crl.globalsign.com,crl.identrust.com,crl.sectigo.com,crl.thawte.com,crl.usertrust.com,crl.verisign.com,crl3.digicert.com,crl4.digicert.com,crls.ssl.com,github.com,host.docker.internal,json-schema.org,json.schemastore.org,keyserver.ubuntu.com,ocsp.digicert.com,ocsp.geotrust.com,ocsp.globalsign.com,ocsp.identrust.com,ocsp.sectigo.com,ocsp.ssl.com,ocsp.thawte.com,ocsp.usertrust.com,ocsp.verisign.com,packagecloud.io,packages.cloud.google.com,packages.microsoft.com,ppa.launchpad.net,raw.githubusercontent.com,registry.npmjs.org,s.symcb.com,s.symcd.com,security.ubuntu.com,telemetry.enterprise.githubcopilot.com,ts-crl.ws.symantec.com,ts-ocsp.ws.symantec.com,www.googleapis.com" GITHUB_SERVER_URL: ${{ github.server_url }} GITHUB_API_URL: ${{ github.api_url }} - GH_AW_SAFE_OUTPUTS_HANDLER_CONFIG: "{\"add_comment\":{\"hide_older_comments\":true,\"max\":1,\"target\":\"*\"},\"missing_data\":{},\"missing_tool\":{},\"noop\":{\"max\":1,\"report-as-issue\":\"false\"}}" + GH_AW_SAFE_OUTPUTS_HANDLER_CONFIG: "{\"add_comment\":{\"hide_older_comments\":true,\"max\":1,\"target\":\"*\"},\"create_report_incomplete_issue\":{},\"missing_data\":{},\"missing_tool\":{},\"noop\":{\"max\":1,\"report-as-issue\":\"false\"},\"report_incomplete\":{}}" with: github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} script: | const { setupGlobals } = require('${{ runner.temp }}/gh-aw/actions/setup_globals.cjs'); - setupGlobals(core, github, context, exec, io); + setupGlobals(core, github, context, exec, io, getOctokit); const { main } = require('${{ runner.temp }}/gh-aw/actions/safe_output_handler_manager.cjs'); await main(); - - name: Upload safe output items + - name: Upload Safe Outputs Items if: always() - uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: - name: safe-output-items - path: /tmp/gh-aw/safe-output-items.jsonl + name: safe-outputs-items + path: | + /tmp/gh-aw/safe-output-items.jsonl + /tmp/gh-aw/temporary-id-map.json if-no-files-found: ignore diff --git a/.github/workflows/copilot-evaluate-tests.md b/.github/workflows/copilot-evaluate-tests.md index 7752f29a6fea..93235d7f9fc9 100644 --- a/.github/workflows/copilot-evaluate-tests.md +++ b/.github/workflows/copilot-evaluate-tests.md @@ -21,6 +21,7 @@ on: required: false type: boolean default: false + roles: [admin, maintain, write] bots: - "copilot-swe-agent[bot]" @@ -62,7 +63,7 @@ network: defaults concurrency: group: "evaluate-pr-tests-${{ github.event.issue.number || inputs.pr_number || github.run_id }}" - cancel-in-progress: true + cancel-in-progress: false timeout-minutes: 20 @@ -104,23 +105,6 @@ steps: fi echo "โœ… Found test files to evaluate:" echo "$TEST_FILES" | head -20 - - # For slash_command triggers, the gh-aw platform's checkout_pr_branch.cjs runs - # AFTER all user steps and overlays the PR branch onto the workspace. This means - # fork PRs can supply their own .github/skills/ and .github/instructions/. - # We cannot restore trusted infra here because the platform checkout runs later. - # Mitigation: agent is sandboxed (no credentials), max 1 comment via safe-outputs, - # and the agent prompt includes a pre-flight check that catches missing SKILL.md. - # See: .github/instructions/gh-aw-workflows.instructions.md "The issue_comment + Fork Problem" - - # For workflow_dispatch, the platform skips checkout entirely โ€” this step is the - # only thing that gets the PR code onto disk and restores trusted infra from main. - - name: Checkout PR and restore agent infrastructure - if: github.event_name == 'workflow_dispatch' - env: - GH_TOKEN: ${{ github.token }} - PR_NUMBER: ${{ inputs.pr_number }} - run: pwsh .github/scripts/Checkout-GhAwPr.ps1 --- # Evaluate PR Tests @@ -132,7 +116,7 @@ Invoke the **evaluate-pr-tests** skill: read and follow `.github/skills/evaluate - **Repository**: ${{ github.repository }} - **PR Number**: ${{ github.event.issue.number || inputs.pr_number }} -The PR branch has been checked out for you. All files from the PR are available locally. +The PR is available via MCP tools. Use `gh pr view` and `gh pr diff` to access PR data. ## Pre-flight check