Fix PR-triage eval trigger: dispatch evaluation.yml instead of bot-applied label - #746
Merged
Merged
Conversation
The triage worker added the 'evaluate-now' label via GITHUB_TOKEN, but label events emitted by GITHUB_TOKEN do not start workflows (GitHub's recursion guard), so evaluation.yml's pull_request_target:[labeled] entry point never fired for the bot (repro: PR dotnet#745). workflow_dispatch and repository_dispatch are the only token-initiated events exempt from that guard. The worker now dispatches evaluation.yml directly via 'gh workflow run' with a pr_number input, routed through the existing gate job so the path is identical to /evaluate. A dispatched run's head_sha is the default branch (not the PR head), so idempotency now matches the deterministic run name 'Evaluate PR #<n> @ <sha7>'. The 'evaluate-now' label remains a valid human entry point. Worker granted actions:write for 'gh workflow run'.
Contributor
|
Note This PR is from a fork and modifies infrastructure files ( Changes to infrastructure typically need to be submitted from a branch in Please consider recreating this PR from an upstream branch. If you don't have push access to |
JanKrivanek
marked this pull request as ready for review
June 10, 2026 16:30
JanKrivanek
enabled auto-merge (squash)
June 10, 2026 16:30
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the PR-triage automation so that when a PR is deemed ready-for-eval, the worker triggers evaluation.yml via workflow_dispatch (instead of relying on a bot-applied label that can’t emit workflow-triggering events under GitHub’s recursion guard).
Changes:
- Add
workflow_dispatchinputs toevaluation.ymland routepr_numberdispatches through the existinggatePR pipeline/concurrency group. - Update the triage worker script to dispatch
evaluation.ymlviagh workflow runand enhance idempotency detection for dispatch-triggered runs. - Grant the triage worker
actions: writeso it can dispatchevaluation.yml, and update design docs to reflect the new trigger path.
Show a summary per file
| File | Description |
|---|---|
docs/design/pr-triage-workflows.md |
Updates architecture + evaluation entry-point documentation to reflect dispatch-based triggering. |
.github/workflows/pr-triage.yml |
Expands permissions to allow dispatching evaluation.yml from the worker. |
.github/workflows/evaluation.yml |
Adds dispatch inputs and updates gate/concurrency logic to support triage dispatch entry point. |
.github/scripts/pr-triage-act.sh |
Switches eval triggering from label application to workflow dispatch; adds dispatch-run idempotency lookup. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 4/4 changed files
- Comments generated: 1
Contributor
|
👋 @JanKrivanek — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the |
Member
Author
|
/evaluate |
github-actions
Bot
added
pr-state/ready-for-eval
Evangelink
approved these changes
Jun 12, 2026
Contributor
Skill Validation Results
[1] (Isolated) Quality unchanged but weighted score is -0.8% due to: efficiency metrics Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
This was referenced Jun 12, 2026
JanKrivanek
pushed a commit
that referenced
this pull request
Jun 12, 2026
…ment (#759) The run-name added in #746 is an unquoted plain scalar containing ` #{0}`. In YAML, a space followed by '#' starts a comment, so everything from '#{0} ...' onward was stripped, leaving an unterminated ${{ }} expression. The file still parses as YAML (yaml.safe_load succeeds) but GitHub Actions rejects it ("This run likely failed because of a workflow file issue"), which broke every evaluation run on main after #746 merged. Wrapping the value in double quotes keeps the full expression intact; verified with actionlint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
YuliiaKovalova
added a commit
that referenced
this pull request
Jun 16, 2026
…rkflow-YAML breakage) (#760) * Fix evaluation.yml: quote run-name so '#' is not parsed as a YAML comment The run-name added in #746 is an unquoted plain scalar containing ` #{0}`. In YAML, a space followed by '#' starts a comment, so everything from '#{0} ...' onward was stripped, leaving an unterminated ${{ }} expression. The file still parses as YAML (yaml.safe_load succeeds) but GitHub Actions rejects it ("This run likely failed because of a workflow file issue"), which broke every evaluation run on main after #746 merged. Wrapping the value in double quotes keeps the full expression intact; verified with actionlint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add authoring-github-workflows skill + actionlint CI gate Prevents the class of bug that broke evaluation on main (#746): an unquoted `${{ }}` workflow expression containing `#`, which YAML treats as a comment and silently truncates, producing a file that parses as YAML but that GitHub Actions refuses to run ("This run likely failed because of a workflow file issue", no jobs started). Two layers: - Knowledge: .agents/skills/authoring-github-workflows/SKILL.md teaches the `#`-as-comment trap and other quoting rules, and to validate with actionlint (plain YAML linters accept the truncated form). - Enforcement: .github/workflows/actionlint.yml runs a pinned actionlint on hand-authored workflows for any PR/push touching .github/workflows. Generated gh-aw files (DO NOT EDIT headers) are skipped; shellcheck/pyflakes are off so the gate focuses on workflow/expression errors. .github/actionlint.yaml declares the repo's custom runner labels (ubuntu-slim, windows-11-arm). Verified locally: actionlint exits 0 on all hand-authored workflows, and exits 1 on the original unquoted run-name, confirming the gate catches it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix markdownlint MD038: remove spaces inside code spans in SKILL.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments: actionlint config trigger, curl -f, syntactic-vs-semantic scope Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Harden actionlint workflow: pin checkout to SHA, disable persisted creds, verify actionlint checksum Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When the PR-triage worker decides a PR is
ready-for-eval, it added theevaluate-nowlabel and relied onevaluation.yml'spull_request_target: [labeled]trigger to start evaluation. That trigger neverfires for the bot.
GitHub's recursion guard: events emitted by the default
GITHUB_TOKENdo notstart new workflow runs — and
labeledis one of them. The worker applies thelabel as
github-actions[bot](itsGITHUB_TOKEN), so thelabeledwebhook issuppressed and no evaluation runs.
Live repro — PR #745
github-actions[bot]evaluate-now/evaluateThe same worker run (
workflow_dispatch, dispatched by the batch viaGITHUB_TOKEN) is itself an A/B proof of the guard:workflow_dispatch→ fired (the worker executed)labeled(the label it added) → did not fireworkflow_dispatchandrepository_dispatchare the only token-initiated eventsexempt from the recursion guard.
Fix
The worker now dispatches
evaluation.ymldirectly viagh workflow run(workflow_dispatch) with a
pr_numberinput, instead of adding a label thatcan't fire. The dispatch is routed through the existing
gatejob, so the pathis identical to
/evaluate(same permission checks, PR fetch, fork handling,concurrency group, commit status, and result comment).
evaluation.yml: newworkflow_dispatchinputspr_number+head_sha;gate.ifand the concurrency group gain a third entry point;gatereads andnumerically validates
pr_number(via env, no interpolation);discoveronlyruns a full/plugin eval for a no-pr_number dispatch.
pr-triage-act.sh:do_eval_triggerdispatchesevaluation.ymlratherthan labeling. Idempotency (
eval_run_exists_for_head) gains a second path:a dispatched run's
head_shais the default branch (not the PR head), so it ismatched by the deterministic run name
Evaluate PR #<n> @ <sha7>.pr-triage.yml: worker grantedactions: write(required forgh workflow run).evaluate-nowlabel remains a valid human entry point (a humanadding the label is not subject to the recursion guard).
Verification
(see the A/B table above) —
workflow_dispatchviaGITHUB_TOKENalreadydrives the batch → worker chain.
evaluation.yml+pr-triage.ymlparse cleanly;pr-triage-act.shpassesbash -n.Docs
Updated pr-triage-workflows.md (architecture
diagram + the three
evaluation.ymlentry points).