Skip to content

fix(workflows): complete #3016 interpolation triage follow-up - #3023

Merged
stranske merged 17 commits into
mainfrom
cursor/issue-3016-interpolation-triage-followup
Aug 10, 2026
Merged

fix(workflows): complete #3016 interpolation triage follow-up#3023
stranske merged 17 commits into
mainfrom
cursor/issue-3016-interpolation-triage-followup

Conversation

@stranske

@stranske stranske commented Aug 10, 2026

Copy link
Copy Markdown
Owner

Source: Issue #3016

Closes #3016

Automated Status Summary

Scope

Scope section missing from source issue.

Context for Agent

Related Issues/PRs

Tasks

  • Fix the free-text exemplars first, since these break on ordinary input: maint-70-fix-integration-formatting.yml:124 (inputs.commit_message) and reusable-codex-run.yml (inputs.codex_args) — move each to a step-level env: entry and reference "$VAR" in the script.
  • Fix maint-69-sync-labels.yml's 3× ${{ inputs.repos }} in the "Determine target repos" step the same way.
  • Triage remaining .github/workflows/*.yml inventory: convert run:/script: interpolations to env: indirection where free-form; document deliberate exceptions inline where provably constrained (e.g. boolean or number).
  • Add a repository guard so the pattern cannot silently return — a test that parses every file in .github/workflows/ and fails on ${{ inputs.* }} / ${{ github.event.* }} inside a run:/script: block scalar, with an explicit allowlist for reviewed exceptions.

Acceptance criteria

  • Named test: tests/workflows/test_no_untrusted_interpolation.py::test_no_untrusted_expressions_in_script_bodies — parses each workflow, walks run:/script: block scalars, and asserts no inputs./github.event. expression appears except those in an explicit, commented allowlist.
  • Deliberate-break → revert: re-introduce git commit -m "${{ inputs.commit_message }}" in maint-70-fix-integration-formatting.yml → confirm the named test FAILS → revert.
  • Behavioural check for the exemplar: dispatch maint-70 with commit_message set to a"b and confirm the step succeeds (today it produces a broken git commit command). Note this dispatch is currently blocked by [P1][ops] All agent issue-automation workflows halt at action_required with zero jobs — agents:auto-pilot is a no-op #3009.

Summary by CodeRabbit

  • Bug Fixes

    • Improved workflow handling of user-provided and event-based values without changing existing behavior.
    • Preserved repository selection, dry-run handling, limits, synchronization, autofix, and coverage validation functionality.
  • Documentation

    • Added guidance for identifying and reviewing workflow script interpolations.
  • Tests

    • Expanded regression checks to detect unsafe or unreviewed workflow expressions, including indexed and default-value forms.
  • Chores

    • Removed obsolete execution metadata.

Document 103 reviewed constrained script interpolations, convert remaining
free-text inputs to env indirection, and extend the regression guard so new
inputs.* / github.event.* hits cannot land without an explicit allowlist entry.

Co-authored-by: Cursor <cursoragent@cursor.com>
Copilot AI lite review requested due to automatic review settings August 10, 2026 04:41
@cursor

cursor Bot commented Aug 10, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@stranske stranske added agent:codex Agent-created issues from Codex agents:keepalive Use to initiate keepalive functionality with agents autofix Opt-in automated formatting & lint remediation labels Aug 10, 2026
@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 40 seconds

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: cc5e2150-7888-44ca-9e4f-2744b6fa14bd

📥 Commits

Reviewing files that changed from the base of the PR and between 4f2da0b and a725b5d.

📒 Files selected for processing (1)
  • docs/workflows/script-interpolation-triage.md
📝 Walkthrough

Walkthrough

Selected workflow scripts now receive inputs through environment variables. The interpolation regression test scans workflow and event expressions, while documentation defines review rules. The campaign contract test verifies environment-based sync-hash handling.

Changes

Workflow interpolation hardening

Layer / File(s) Summary
Workflow environment indirection
.github/workflows/agents-keepalive-branch-sync.yml, .github/workflows/maint-72-fix-pr-body-conflicts.yml, .github/workflows/maint-82-sync-dependency-campaign.yml, .github/workflows/maint-45-cosmetic-repair.yml, .github/workflows/reusable-10-ci-python.yml, .github/workflows/reusable-18-autofix.yml
Selected workflow inputs now pass through environment variables. The autofix workflow validates single-line commit prefixes before persisting them.
Interpolation regression guard
tests/workflows/test_no_untrusted_interpolation.py, docs/workflows/script-interpolation-triage.md
The test normalizes workflow expressions, detects banned free-text and unreviewed constrained interpolations, and covers indexed and wrapped forms. The documentation records triage and allowlist rules.
Campaign contract validation
tests/workflows/test_maint82_sync_campaign_contract.py
The contract test verifies that the campaign maps the sync hash into the step environment and reads it through process.env.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

Suggested labels: verify:compare, follow-up, agent:retry

🚥 Pre-merge checks | ✅ 2 | ❌ 3

❌ Failed checks (3 warnings)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning The PR adds the guard and fixes several workflows, but the provided changes omit required exemplars such as maint-70, reusable-codex-run, and maint-69. Convert the remaining required free-text interpolations to env: indirection and verify the named regression test fails on a deliberate unsafe reintroduction.
Out of Scope Changes check ⚠️ Warning The deletion of langsmith-fleet-worker-attempt.json is unrelated to workflow interpolation handling and is outside issue #3016 scope. Restore the artifact, or link its removal to a separate issue and submit it separately.
Docstring Coverage ⚠️ Warning Docstring coverage is 30.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the workflow interpolation-triage follow-up and references issue #3016.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch cursor/issue-3016-interpolation-triage-followup

Comment @coderabbitai help to get the list of available commands.

@stranske-keepalive

stranske-keepalive Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

🤖 Keepalive Loop Status

PR #3023 | Agent: Codex | Iteration 0/12

Current State

Metric Value
Iteration progress [----------] 0/12
Action run (agent-run-failed)
Agent status ❌ AGENT FAILED
Gate success
Tasks 0/7 complete
Timeout 45 min (default)
Timeout usage 11m elapsed (26%, 34m remaining)
Keepalive ✅ enabled
Autofix ❌ disabled

Last Codex Run

Result Value
Status ❌ AGENT FAILED
Reason agent-run-failed
Exit code unknown
Failures 1/3 before pause

To retry immediately:

  • Add the agent:retry label to this PR

Or wait for the next successful Gate run to automatically retry.

🔍 Failure Classification

| Error type | infrastructure |
| Error category | transient |
| Suggested recovery | Capture logs and context; retry once and escalate if the issue persists. |

⚠️ Failure Tracking

| Consecutive failures | 1/3 |
| Reason | agent-run-failed |

@stranske-keepalive

stranske-keepalive Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor
Keepalive Work Log (click to expand)
# Time (UTC) Agent Action Result Files Tasks Progress Commit Gate
0 2026-08-10 04:41:45 Codex wait (gate-pending-transient) skipped 0 0/7
0 2026-08-10 04:42:29 Codex wait (gate-cancelled-transient-transient) skipped 0 0/7 cancelled
0 2026-08-10 04:43:31 Codex run (agent-run-skipped) skipped 0 0/7 cancelled
0 2026-08-10 04:47:07 Codex run (agent-run-failed) failure 2 file(s) 0 0/7 0284784 cancelled
0 2026-08-10 04:50:24 Codex run (agent-run-failed) failure 2 file(s) 0 0/7 56e1435 cancelled
0 2026-08-10 04:54:26 Codex run (agent-run-failed-repeat) failure 4 file(s) 0 0/7 7e6599b cancelled
0 2026-08-10 04:55:02 Codex skip (needs-human) skipped 0 0/7
0 2026-08-10 04:55:45 Codex skip (needs-human) skipped 0 0/7 cancelled
0 2026-08-10 04:56:32 Codex skip (needs-human) skipped 0 0/7 cancelled
0 2026-08-10 04:57:14 Codex skip (needs-human) skipped 0 0/7 cancelled
0 2026-08-10 04:58:00 Codex skip (needs-human) retry skipped 0 0/7
0 2026-08-10 05:04:07 Codex skip (needs-human) skipped 0 0/7 failure
0 2026-08-10 05:08:10 Codex skip (needs-human) skipped 0 0/7 cancelled
0 2026-08-10 05:12:24 Codex skip (needs-human) skipped 0 0/7 failure
0 2026-08-10 05:18:59 Codex skip (needs-human) skipped 0 0/7 failure
0 2026-08-10 05:26:20 Codex skip (needs-human) skipped 0 0/7 failure
0 2026-08-10 05:30:21 Codex run (agent-run-failed-repeat) retry failure 4 file(s) 0 0/7 754d8d7
0 2026-08-10 05:31:08 Codex skip (needs-human) skipped 0 0/7
0 2026-08-10 05:31:51 Codex skip (needs-human) skipped 0 0/7 cancelled
0 2026-08-10 05:32:32 Codex skip (needs-human) skipped 0 0/7 cancelled
0 2026-08-10 05:36:08 Codex skip (needs-human) skipped 0 0/7 failure
0 2026-08-10 05:41:40 Codex run (agent-run-failed-repeat) retry failure 3 file(s) 0 0/7
0 2026-08-10 05:42:32 Codex skip (needs-human) skipped 0 0/7
0 2026-08-10 05:43:15 Codex skip (needs-human) skipped 0 0/7 cancelled
0 2026-08-10 05:44:03 Codex skip (needs-human) skipped 0 0/7 cancelled
0 2026-08-10 05:48:20 Codex skip (needs-human) skipped 0 0/7 failure
0 2026-08-10 05:53:56 Codex skip (needs-human) skipped 0 0/7 success
0 2026-08-10 06:30:44 Codex run (agent-run-failed-repeat) retry failure 3 file(s) 0 0/7
0 2026-08-10 06:31:36 Codex skip (needs-human) skipped 0 0/7
0 2026-08-10 06:32:08 Codex skip (needs-human) skipped 0 0/7 cancelled
0 2026-08-10 06:32:58 Codex skip (needs-human) skipped 0 0/7 cancelled
0 2026-08-10 06:36:37 Codex skip (needs-human) skipped 0 0/7 success
0 2026-08-10 06:38:57 Codex wait (gate-pending-transient) skipped 0 0/7
0 2026-08-10 06:49:50 Codex run (agent-run-failed) failure 1 file(s) 0 0/7 b35cf56 success

@github-actions

github-actions Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Runner dispatch state for codex on PR #3023. Do not edit.

@stranske
stranske temporarily deployed to agent-high-privilege August 10, 2026 04:42 — with GitHub Actions Inactive

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e526b6f6f7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

return [
_normalize_expression(body)
for body in _actions_expression_bodies(script)
if re.search(r"\b(inputs\.|github\.event\.)", body)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Detect bracket-form workflow inputs in the guard

The new scanner only recognizes dot notation, so an interpolation such as ${{ inputs['repos'] }} or ${{ github['event']['issue']['body'] }} produces no hit and bypasses both the banned-expression check and the reviewed allowlist. This weakens the previous repository scan, whose _references_untrusted_input path explicitly recognized bracket-form inputs; use that parser or extend this matcher so changing a banned interpolation to equivalent bracket syntax still fails the regression test.

Useful? React with 👍 / 👎.

@agents-workflows-bot
agents-workflows-bot Bot temporarily deployed to agent-high-privilege August 10, 2026 04:44 Inactive

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR completes the #3016 “untrusted interpolation in run:/script: bodies” triage follow-up by documenting the reviewed inventory, expanding the regression test to enforce an explicit allowlist, and converting remaining free-text exemplars to step-level env: indirection in affected workflows.

Changes:

  • Expanded the regression test to (a) ban specific free-text inputs.* fields in script bodies and (b) require an explicit allowlist entry for every remaining inputs.* / github.event.* expression found in run: / with.script: scalars.
  • Added repository documentation for the triage and the allowlist/test gate.
  • Updated several workflows to pass previously-interpolated values via env: and read them from shell/process.env.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tests/workflows/test_no_untrusted_interpolation.py Adds banned free-text list + reviewed allowlist gate for expressions inside script bodies.
docs/workflows/script-interpolation-triage.md Documents the repo-wide triage results and the intended test gate behavior.
.github/workflows/reusable-18-autofix.yml Moves inputs.commit_prefix consumption out of script interpolation into step env: indirection.
.github/workflows/maint-82-sync-dependency-campaign.yml Moves github-script interpolations to env: + process.env reads.
.github/workflows/maint-72-fix-pr-body-conflicts.yml Moves inputs.target_repo consumption to step env: indirection.
.github/workflows/agents-keepalive-branch-sync.yml Moves inputs.head_is_fork / inputs.head_repository consumption to step env: indirection.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +213 to +218
def _script_interpolation_hits(script: str) -> list[str]:
return [
_normalize_expression(body)
for body in _actions_expression_bodies(script)
if re.search(r"\b(inputs\.|github\.event\.)", body)
]
Comment on lines +203 to +210
def _is_banned_free_text(body: str) -> bool:
normalized = _normalize_expression(body)
return any(
normalized == banned
or normalized.startswith(f"{banned} ")
or normalized.startswith(f"{banned}||")
for banned in BANNED_FREE_TEXT_EXPRESSIONS
)
Comment on lines +153 to +157
- name: Set commit prefix env
id: prefix
run: echo "AUTOFIX_COMMIT_PREFIX=${{ inputs.commit_prefix }}" >> "$GITHUB_ENV"
env:
COMMIT_PREFIX: ${{ inputs.commit_prefix }}
run: echo "AUTOFIX_COMMIT_PREFIX=$COMMIT_PREFIX" >> "$GITHUB_ENV"
@agents-workflows-bot
agents-workflows-bot Bot temporarily deployed to agent-high-privilege August 10, 2026 04:48 Inactive

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/reusable-18-autofix.yml:
- Around line 155-157: Validate COMMIT_PREFIX for carriage-return or line-feed
characters before writing it to GITHUB_ENV, rejecting multiline input. In the
step that exports AUTOFIX_COMMIT_PREFIX, use printf for the environment
assignment rather than echo, while preserving the existing input value for valid
single-line prefixes.

In `@tests/workflows/test_no_untrusted_interpolation.py`:
- Around line 203-218: Update `_script_interpolation_hits` to detect indexed
context references as well as dotted `inputs` and `github.event` references,
including `inputs['commit_message']` and `github['event']['issue']['title']`.
Preserve existing matches and add regression cases covering both indexed forms.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 5835a4d7-5335-4e8e-ba39-9a2859010d74

📥 Commits

Reviewing files that changed from the base of the PR and between 676798f and f6e4c31.

📒 Files selected for processing (7)
  • .github/workflows/agents-keepalive-branch-sync.yml
  • .github/workflows/maint-72-fix-pr-body-conflicts.yml
  • .github/workflows/maint-82-sync-dependency-campaign.yml
  • .github/workflows/reusable-18-autofix.yml
  • docs/workflows/script-interpolation-triage.md
  • langsmith-fleet-worker-attempt.json
  • tests/workflows/test_no_untrusted_interpolation.py

Comment thread .github/workflows/reusable-18-autofix.yml Outdated
Comment thread tests/workflows/test_no_untrusted_interpolation.py
@agents-workflows-bot
agents-workflows-bot Bot temporarily deployed to agent-high-privilege August 10, 2026 04:51 Inactive
@stranske-keepalive stranske-keepalive Bot added agent:needs-attention Agent needs human review or intervention needs-human Requires human intervention or review labels Aug 10, 2026
@agents-workflows-bot
agents-workflows-bot Bot temporarily deployed to agent-high-privilege August 10, 2026 04:56 Inactive
@stranske
stranske temporarily deployed to agent-high-privilege August 10, 2026 05:39 — with GitHub Actions Inactive
@stranske-keepalive stranske-keepalive Bot added agent:needs-attention Agent needs human review or intervention needs-human Requires human intervention or review labels Aug 10, 2026
@stranske-keepalive
stranske-keepalive Bot temporarily deployed to agent-high-privilege August 10, 2026 05:43 Inactive
@agents-workflows-bot
agents-workflows-bot Bot temporarily deployed to agent-high-privilege August 10, 2026 05:49 Inactive

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/reusable-10-ci-python.yml:
- Around line 2001-2010: Update the Python coverage-threshold validation in
validate_number to reject non-finite parsed values such as NaN and infinities
before range checks or comparisons. Add a regression case covering coverage-min:
NaN, ensuring it fails validation and is not passed to coverage_trend.py as a
valid minimum.

In `@docs/workflows/script-interpolation-triage.md`:
- Line 13: Update the reviewed-interpolation count in the documentation table
and the remaining constrained-tuples statement to use the same value derived
from REVIEWED_SCRIPT_INTERPOLATIONS, replacing the inconsistent hardcoded counts
while preserving the surrounding wording.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 2e727061-aa3c-4d06-9692-71cbab6d71c0

📥 Commits

Reviewing files that changed from the base of the PR and between f6e4c31 and 4f2da0b.

📒 Files selected for processing (7)
  • .github/workflows/maint-45-cosmetic-repair.yml
  • .github/workflows/reusable-10-ci-python.yml
  • .github/workflows/reusable-18-autofix.yml
  • docs/workflows/script-interpolation-triage.md
  • langsmith-fleet-worker-attempt.json
  • tests/workflows/test_maint82_sync_campaign_contract.py
  • tests/workflows/test_no_untrusted_interpolation.py
💤 Files with no reviewable changes (1)
  • langsmith-fleet-worker-attempt.json

Comment thread .github/workflows/reusable-10-ci-python.yml
| --- | ---: | --- |
| Free-text inputs moved to `env:` in #3020 | 3 fields | `commit_message`, `codex_args`, `repos` |
| Additional free-text fixes in this PR | 4 fields | `target_repo`, `commit_prefix`, `head_repository`, campaign script outputs |
| Reviewed constrained interpolations | 97 | Explicit allowlist in `test_no_untrusted_interpolation.py` |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Reconcile the reviewed-interpolation count.

Line 13 states 97 reviewed constrained interpolations. Line 32 states 103 remaining constrained tuples. Use the count from REVIEWED_SCRIPT_INTERPOLATIONS consistently in both locations.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/workflows/script-interpolation-triage.md` at line 13, Update the
reviewed-interpolation count in the documentation table and the remaining
constrained-tuples statement to use the same value derived from
REVIEWED_SCRIPT_INTERPOLATIONS, replacing the inconsistent hardcoded counts
while preserving the surrounding wording.

@agents-workflows-bot

agents-workflows-bot Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

🤖 Bot Comment Handler

  • Agent: codex
  • Bot comments to address: 6

The agent has been assigned to this PR to address the bot review comments.

Instructions for agent

  1. Implement suggested fixes that improve the code
  2. Skip suggestions that don't apply (note why in your response)

The bot comment handler workflow has prepared context in the artifacts.

@stranske

Copy link
Copy Markdown
Owner Author

Closer update: reconciled the reviewed-interpolation count to the current allowlist (103) in a725b5d; python3.12 -m pytest -q tests/workflows/test_no_untrusted_interpolation.py passed (17). The non-finite coverage-threshold report was audited as a valid pre-existing hardening concern but out of scope for #3016; it has a thread reply and no PR behavior was widened.

@stranske
stranske temporarily deployed to agent-high-privilege August 10, 2026 06:26 — with GitHub Actions Inactive
@stranske stranske added agent:retry Add to trigger agent retry after rate limit or pause and removed needs-human Requires human intervention or review agent:needs-attention Agent needs human review or intervention labels Aug 10, 2026
@stranske-keepalive stranske-keepalive Bot removed the agent:retry Add to trigger agent retry after rate limit or pause label Aug 10, 2026
@stranske
stranske temporarily deployed to agent-high-privilege August 10, 2026 06:29 — with GitHub Actions Inactive
@agents-workflows-bot agents-workflows-bot Bot added agent:needs-attention Agent needs human review or intervention needs-human Requires human intervention or review labels Aug 10, 2026
@agents-workflows-bot
agents-workflows-bot Bot temporarily deployed to agent-high-privilege August 10, 2026 06:32 Inactive
@github-actions

Copy link
Copy Markdown
Contributor

Provider Comparison Report

Provider Summary

Provider Model Verdict Confidence Summary
openai gpt-5.6-terra PASS 88% The changes move workflow values out of shell-script interpolation and into step-level environment variables, with shell references quoted appropriately. The added repository-wide interpolation gua...
anthropic claude-sonnet-5 PASS 55% The PR converts several remaining workflow files (agents-keepalive-branch-sync.yml, maint-45-cosmetic-repair.yml, maint-72, maint-82, reusable-10-ci-python.yml, reusable-18-autofix.yml) from direct...
📋 Full Provider Details (click to expand)

openai

  • Model: gpt-5.6-terra
  • Verdict: PASS
  • Confidence: 88%
  • Scores:
    • Correctness: 9.0/10
    • Completeness: 9.0/10
    • Quality: 9.0/10
    • Testing: 9.0/10
    • Risks: 9.0/10
  • Summary: The changes move workflow values out of shell-script interpolation and into step-level environment variables, with shell references quoted appropriately. The added repository-wide interpolation guard is substantial and directly targets the required failure mode across workflow run/script bodies, while the triage documentation and explicit reviewed-exception mechanism provide maintainable rationale for constrained cases. The guard would detect reintroduction of the cited direct commit-message interpolation pattern. The unavailable dispatch behavioral check is explicitly blocked by the referenced upstream issue and does not undermine the code-level remediation.

anthropic

  • Model: claude-sonnet-5
  • Verdict: PASS
  • Confidence: 55%
  • Scores:
    • Correctness: 7.0/10
    • Completeness: 6.0/10
    • Quality: 6.0/10
    • Testing: 7.0/10
    • Risks: 6.0/10
  • Summary: The PR converts several remaining workflow files (agents-keepalive-branch-sync.yml, maint-45-cosmetic-repair.yml, maint-72, maint-82, reusable-10-ci-python.yml, reusable-18-autofix.yml) from direct ${{ inputs.* }}/${{ github.event.* }} interpolation in run: blocks to safer step-level env: var indirection, matching the intended pattern from the linked issue [P2] 83 untrusted ${{ }} expressions interpolated into run:/script: bodies across 34 workflows — rework to env: indirection #3016. It adds a substantial guard test (tests/workflows/test_no_untrusted_interpolation.py, +583 lines) intended to satisfy the named acceptance-criteria test, plus a new triage documentation file (docs/workflows/script-interpolation-triage.md) documenting exceptions, consistent with the 'triage remaining workflows' task. The core exemplar fixes (maint-70, reusable-codex-run.yml, maint-69) are absent from this diff, which is expected given the PR is framed as a 'follow-up' completing prior work, but this could not be independently verified. Some included files (config/model_eval_corpus_staging.json, deleted langsmith-fleet-worker-attempt.json) appear unrelated to the stated scope. Given visible changes correctly implement the intended security pattern and CI on the merge commit is green/in-progress with no failures, this is assessed as a reasonable, largely complete follow-up, with moderate residual uncertainty about full test coverage due to diff truncation.
  • Concerns:
    • The PR diff does not touch maint-70-fix-integration-formatting.yml, reusable-codex-run.yml, or maint-69-sync-labels.yml, which the acceptance criteria call out as the primary 'fix first' exemplars — this strongly implies those were addressed in a prior merge (the original [P2] 83 untrusted ${{ }} expressions interpolated into run:/script: bodies across 34 workflows — rework to env: indirection #3016 work) and this PR is purely the completion/follow-up pass, but this cannot be fully confirmed from the diff alone.
    • Several files in the diff (config/model_eval_corpus_staging.json added, langsmith-fleet-worker-attempt.json deleted) appear unrelated to the stated interpolation-fix scope, raising questions about scope creep or accidental inclusion of unrelated artifacts.
    • The full content of the heavily-expanded tests/workflows/test_no_untrusted_interpolation.py (+583/-11) was truncated in the provided diff, so the named test function, its allowlist logic, and the deliberate-break/revert verification described in the acceptance criteria could not be independently confirmed line-by-line.
    • The behavioral dispatch check for maint-70 with commit_message='a"b' remains blocked by [P1][ops] All agent issue-automation workflows halt at action_required with zero jobs — agents:auto-pilot is a no-op #3009 as explicitly acknowledged — this is an accepted, documented limitation rather than a defect in this PR.
    • Shown env-var conversions (agents-keepalive-branch-sync.yml, maint-45-cosmetic-repair.yml) are correctly implemented, replacing risky ${{ inputs.* }}/${{ github.event.* }} interpolation in run: blocks with step-level env: vars and shell $VAR references, which is the correct security pattern.

Agreement

  • Verdict: PASS (all providers)

Disagreement

Dimension openai anthropic
Correctness 9.0/10 7.0/10
Completeness 9.0/10 6.0/10
Quality 9.0/10 6.0/10
Testing 9.0/10 7.0/10
Risks 9.0/10 6.0/10

Unique Insights

  • openai: The changes move workflow values out of shell-script interpolation and into step-level environment variables, with shell references quoted appropriately. The added repository-wide interpolation guard is substantial and directly targets the required failure mode across workflow run/script bodies,...
  • anthropic: The PR diff does not touch maint-70-fix-integration-formatting.yml, reusable-codex-run.yml, or maint-69-sync-labels.yml, which the acceptance criteria call out as the primary 'fix first' exemplars — this strongly implies those were addressed in a prior merge (the original [P2] 83 untrusted ${{ }} expressions interpolated into run:/script: bodies across 34 workflows — rework to env: indirection #3016 work) and this PR is purely the completion/follow-up pass, but this cannot be fully confirmed from the diff alone.; Several files in the diff (config/model_eval_corpus_staging.json added, langsmith-fleet-worker-attempt.json deleted) appear unrelated to the stated interpolation-fix scope, raising questions about scope creep or accidental inclusion of unrelated artifacts.; The full content of the heavily-expanded tests/workflows/test_no_untrusted_interpolation.py (+583/-11) was truncated in the provided diff, so the named test function, its allowlist logic, and the deliberate-break/revert verification described in the acceptance criteria could not be independently confirmed line-by-line.; The behavioral dispatch check for maint-70 with commit_message='a"b' remains blocked by [P1][ops] All agent issue-automation workflows halt at action_required with zero jobs — agents:auto-pilot is a no-op #3009 as explicitly acknowledged — this is an accepted, documented limitation rather than a defect in this PR.; Shown env-var conversions (agents-keepalive-branch-sync.yml, maint-45-cosmetic-repair.yml) are correctly implemented, replacing risky ${{ inputs.* }}/${{ github.event.* }} interpolation in run: blocks with step-level env: vars and shell $VAR references, which is the correct security pattern.

🔍 LangSmith Traces

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:codex Agent-created issues from Codex agents:keepalive Use to initiate keepalive functionality with agents autofix:patch autofix Opt-in automated formatting & lint remediation verify:compare Compare multiple LLM evaluations

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[P2] 83 untrusted ${{ }} expressions interpolated into run:/script: bodies across 34 workflows — rework to env: indirection

3 participants