Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .github/workflows/agents-issue-format-guard.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,14 @@ jobs:
jq -r '.body // ""' issue.json > body.md
exempt=false
held=false
# A closed issue has nothing left to format. Routing one to the optimizer

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Update the documented exemption contract

This changes the consumer-facing guard contract, but docs/ci/WORKFLOWS.md:157 still says the guard validates manual dispatch and lists only durable/wontfix and bot issues as exempt, while .github/sync-manifest.yml:95 retains the same incomplete description. Document the closed-issue exemption alongside the synchronized workflow change so operators and consumers see the behavior the template now implements.

AGENTS.md reference: AGENTS.md:L62-L65

Useful? React with 👍 / 👎.

# produces a body edit, that edit re-fires this guard, and the pair loops
# against work that is already delivered — see Fine-Art-Archive#464, which
# kept formatting for 17.5 hours after it was closed.
if [[ "$(jq -r '.state // "" | ascii_downcase' issue.json)" == "closed" ]]; then

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Recheck state before mutating or dispatching

When an issue closes after the Resolve issue snapshot but before later steps execute, this exemption remains false for the run. The stale-format step at lines 120-123 can then remove agents:formatted, and although the routing step deliberately refreshes live.json at lines 138-140, it checks only hold labels before dispatching the optimizer. Recheck the live state at both side-effect boundaries so a run already in progress cannot continue modifying or routing an issue after closure.

Useful? React with 👍 / 👎.

echo "Issue is closed — nothing to format."
exempt=true
fi
Comment on lines +64 to +71

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Check the re-fetched issue state before dispatch.

Both workflows only check the state in issue.json. If the issue closes before the route step fetches live.json, the workflow can still label, dispatch, and comment on the closed issue.

  • .github/workflows/agents-issue-format-guard.yml#L64-L71: stop the route step when live.json.state is closed.
  • templates/consumer-repo/.github/workflows/agents-issue-format-guard.yml#L64-L71: apply the same route-step guard through the workflow sync source.
Proposed guard after the live.json fetch
           gh issue view "$NUMBER" --repo "$GITHUB_REPOSITORY" \
             --json number,body,labels,state,author > live.json
+          if [[ "$(jq -r '.state // "" | ascii_downcase' live.json)" == "closed" ]]; then
+            echo "Issue is now closed — skipping optimizer dispatch."
+            exit 0
+          fi
           jq -r '.body // ""' live.json > body.md
📍 Affects 2 files
  • .github/workflows/agents-issue-format-guard.yml#L64-L71 (this comment)
  • templates/consumer-repo/.github/workflows/agents-issue-format-guard.yml#L64-L71
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/agents-issue-format-guard.yml around lines 64 - 71, Update
the route step in both .github/workflows/agents-issue-format-guard.yml lines
64-71 and
templates/consumer-repo/.github/workflows/agents-issue-format-guard.yml lines
64-71 to check the re-fetched live.json state after the live issue fetch; when
live.json.state is closed, set the existing exemption and stop routing before
labeling, dispatching, or commenting. Keep both workflow copies synchronized.

if jq -e '[.labels[].name] | any(. == "tracker:durable" or . == "wontfix")' issue.json >/dev/null; then exempt=true; fi
if jq -e '[.labels[].name] | any(. == "agents:auto-pilot-pause" or . == "needs-human")' issue.json >/dev/null; then held=true; fi
if jq -er '.author.login // ""' issue.json | grep -qiE '(\[bot\]|^renovate$|^dependabot$)'; then exempt=true; fi
Expand Down
4 changes: 2 additions & 2 deletions langsmith-fleet-worker-attempt.json
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
{
"agent": "codex",
"cli_version": "0.144.1",
"emitted_at": "2026-08-08T15:51:23.873676Z",
"emitted_at": "2026-08-08T23:54:44.735714Z",
"execution_profile": "codex-default",
"fallback_models": [
"gpt-5.5"
],
"operation_role": "worker",
"pr_number": "2985",
"pr_number": "2997",
"requested_model": "gpt-5.6-terra",
"runner": "reusable-codex-run",
"schema": "langsmith-fleet/v1",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,14 @@ jobs:
jq -r '.body // ""' issue.json > body.md
exempt=false
held=false
# A closed issue has nothing left to format. Routing one to the optimizer
# produces a body edit, that edit re-fires this guard, and the pair loops
# against work that is already delivered — see Fine-Art-Archive#464, which
# kept formatting for 17.5 hours after it was closed.
if [[ "$(jq -r '.state // "" | ascii_downcase' issue.json)" == "closed" ]]; then
echo "Issue is closed — nothing to format."
exempt=true
fi
Comment on lines +64 to +71

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Re-check the live issue state before optimizer dispatch.

The initial closed-state check does not protect a run when the issue closes before the route step re-fetches live.json. That route can still dispatch the optimizer for a closed issue.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@templates/consumer-repo/.github/workflows/agents-issue-format-guard.yml`
around lines 64 - 71, Update the optimizer dispatch route to re-check the
freshly fetched live issue state from live.json immediately before dispatch, not
only the initial issue.json state. When the live state is closed, skip optimizer
routing and mark the issue exempt using the existing control flow.

if jq -e '[.labels[].name] | any(. == "tracker:durable" or . == "wontfix")' issue.json >/dev/null; then exempt=true; fi
if jq -e '[.labels[].name] | any(. == "agents:auto-pilot-pause" or . == "needs-human")' issue.json >/dev/null; then held=true; fi
if jq -er '.author.login // ""' issue.json | grep -qiE '(\[bot\]|^renovate$|^dependabot$)'; then exempt=true; fi
Expand Down
123 changes: 123 additions & 0 deletions tests/workflows/test_issue_format_guard_closed_issues.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
"""The format guard must not process closed issues.

These tests execute the guard's real "Resolve issue" step, lifted straight out of
the workflow YAML, against a fixture issue payload. A closed issue routed to the
optimizer produces a body edit, that edit re-fires the guard, and the pair loops
against already-delivered work: Fine-Art-Archive#464 kept formatting for 17.5
hours after it was closed.
"""

from __future__ import annotations

import shutil
import subprocess
from pathlib import Path
Comment on lines +12 to +14

import pytest
import yaml

REPO_ROOT = Path(__file__).resolve().parents[2]
GUARD_PATH = REPO_ROOT / ".github/workflows/agents-issue-format-guard.yml"
CONSUMER_GUARD_PATH = (
REPO_ROOT / "templates/consumer-repo/.github/workflows/agents-issue-format-guard.yml"
)
GUARD_PATHS = (GUARD_PATH, CONSUMER_GUARD_PATH)

JQ_AVAILABLE = shutil.which("jq") is not None
skip_if_no_jq = pytest.mark.skipif(not JQ_AVAILABLE, reason="jq required by the guard step")


def _resolve_step_script(path: Path) -> str:
workflow = yaml.safe_load(path.read_text(encoding="utf-8"))
for step in workflow["jobs"]["check"]["steps"]:
if step.get("name") == "Resolve issue":
return step["run"]
raise AssertionError(f"'Resolve issue' step not found in {path}")


def _run_resolve_step(tmp_path: Path, script: str, issue_json: str) -> dict[str, str]:
"""Run the real step with `gh` stubbed to return the fixture payload."""
bin_dir = tmp_path / "bin"
bin_dir.mkdir()
fixture = tmp_path / "fixture.json"
fixture.write_text(issue_json, encoding="utf-8")

# The step's only external call is `gh issue view … > issue.json`.
gh_stub = bin_dir / "gh"
gh_stub.write_text(f'#!/usr/bin/env bash\ncat "{fixture}"\n', encoding="utf-8")
gh_stub.chmod(0o755)

github_output = tmp_path / "github_output"
github_output.touch()

completed = subprocess.run(
["bash", "-c", script],
cwd=tmp_path,
text=True,
capture_output=True,
env={
"PATH": f"{bin_dir}:{shutil.os.environ['PATH']}",
"GITHUB_OUTPUT": str(github_output),
"GITHUB_REPOSITORY": "stranske/Fine-Art-Archive",
"NUMBER": "464",
"GH_TOKEN": "stub",
},
)
assert completed.returncode == 0, f"step failed: {completed.stderr}"

outputs: dict[str, str] = {}
for line in github_output.read_text(encoding="utf-8").splitlines():
if "=" in line:
key, _, value = line.partition("=")
outputs[key] = value
return outputs


# The #464 shape: closed, and carrying none of the pre-existing exemption signals.
CLOSED_ISSUE = """
{
"number": 464,
"state": "CLOSED",
"body": "## Tasks\\n- [ ] do a thing\\n\\n## Acceptance Criteria\\n- [ ] it is done\\n",
"labels": [{"name": "agents:format"}, {"name": "agents:formatted"}],
"author": {"login": "stranske"}
}
"""

OPEN_ISSUE = CLOSED_ISSUE.replace('"state": "CLOSED"', '"state": "OPEN"')


@skip_if_no_jq
@pytest.mark.parametrize("guard_path", GUARD_PATHS, ids=lambda p: str(p.relative_to(REPO_ROOT)))
def test_closed_issue_is_exempt(tmp_path: Path, guard_path: Path) -> None:
outputs = _run_resolve_step(tmp_path, _resolve_step_script(guard_path), CLOSED_ISSUE)
assert outputs["exempt"] == "true", (
"a closed issue must be exempt; otherwise the guard keeps routing it to the "
"optimizer and the pair loops against delivered work"
)


@skip_if_no_jq
@pytest.mark.parametrize("guard_path", GUARD_PATHS, ids=lambda p: str(p.relative_to(REPO_ROOT)))
def test_open_issue_is_still_processed(tmp_path: Path, guard_path: Path) -> None:
"""The exemption must be narrow: an equivalent open issue still gets validated."""
outputs = _run_resolve_step(tmp_path, _resolve_step_script(guard_path), OPEN_ISSUE)
assert outputs["exempt"] == "false"
assert outputs["held"] == "false"


@skip_if_no_jq
@pytest.mark.parametrize("guard_path", GUARD_PATHS, ids=lambda p: str(p.relative_to(REPO_ROOT)))
def test_closed_state_is_matched_case_insensitively(tmp_path: Path, guard_path: Path) -> None:
"""`gh` returns CLOSED; the REST API returns closed. Both must exempt."""
lowercase = CLOSED_ISSUE.replace('"state": "CLOSED"', '"state": "closed"')
outputs = _run_resolve_step(tmp_path, _resolve_step_script(guard_path), lowercase)
assert outputs["exempt"] == "true"


def test_both_guard_copies_carry_the_closed_check() -> None:
"""Root and consumer template must not drift on this check."""
for path in GUARD_PATHS:
assert "ascii_downcase" in _resolve_step_script(
path
), f"{path} lacks the closed-state check"
Loading