Skip to content

test(actions): make setup-python contract upgrade-safe #2854

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
stranske merged 4 commits into from
Jul 31, 2026
Merged

test(actions): make setup-python contract upgrade-safe #2854

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
bf699b0
test(actions): validate setup-python contract semantically
stranske Jul 31, 2026
d6521fe
test(actions): require valid setup-python major refs
stranske Jul 31, 2026
99a7b1d
test(actions): execute shadow workflow on pin changes
stranske Jul 31, 2026
3246f8d
style(tests): format shadow workflow contract
stranske Jul 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .github/workflows/health-69-consumer-sync-shadow-evidence.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,19 @@ name: Health 69 Consumer Sync Shadow Evidence

on:
workflow_dispatch:
pull_request:
paths:
- '.github/workflows/health-69-consumer-sync-shadow-evidence.yml'
- '.github/sync-manifest.yml'
- 'scripts/sync_manifest_compiler.py'
- 'scripts/build_consumer_sync_shadow_handoff.py'
- 'templates/consumer-repo/**'
schedule:
- cron: '17 8 * * 1'
push:
branches: [main]
paths:
- '.github/workflows/health-69-consumer-sync-shadow-evidence.yml'
- '.github/sync-manifest.yml'
- 'scripts/sync_manifest_compiler.py'
- 'scripts/build_consumer_sync_shadow_handoff.py'
Expand Down
14 changes: 10 additions & 4 deletions tests/scripts/test_consumer_sync_shadow_handoff.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
from __future__ import annotations

import json
import re
import subprocess
import sys
from pathlib import Path
Expand Down Expand Up @@ -123,9 +124,8 @@ def test_cli_reports_invalid_plan_without_writing_handoff(tmp_path: Path) -> Non

def test_workflow_has_no_write_or_apply_surface() -> None:
root = Path(__file__).parents[2]
workflow = (
root / ".github" / "workflows" / "health-69-consumer-sync-shadow-evidence.yml"
).read_text(encoding="utf-8")
workflow_path = root / ".github" / "workflows" / "health-69-consumer-sync-shadow-evidence.yml"
workflow = workflow_path.read_text(encoding="utf-8")

assert "contents: read" in workflow
assert "contents: write" not in workflow
Expand All @@ -135,7 +135,13 @@ def test_workflow_has_no_write_or_apply_surface() -> None:
assert "write_authority" not in workflow.lower() or "Write authority: false" in workflow
assert "persist-credentials: false" in workflow
assert "actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1" in workflow
assert "actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1" in workflow
setup_python_refs = re.findall(
r"uses:\s+actions/setup-python@[0-9a-f]{40}\s+# v\d+\b",
Comment thread
stranske marked this conversation as resolved.
workflow,
)
assert len(setup_python_refs) == 1
assert "pull_request:" in workflow
assert f"- '{workflow_path.relative_to(root).as_posix()}'" in workflow
Comment on lines +138 to +144

@coderabbitai coderabbitai Bot Jul 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Validate actual workflow steps, not matching text.

The regex counts only references that already match the pinned format. A second unpinned actions/setup-python@v6 reference can remain undetected. A matching string inside a comment or run: | block can also satisfy the assertion.

Parse the workflow and inspect each step's uses value. Assert that exactly one actions/setup-python step exists and that its reference has the required 40-character SHA and version comment. Validate the trigger and path registration from parsed YAML as well.

As per path instructions, synced workflow checks must reject unpinned third-party actions.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/scripts/test_consumer_sync_shadow_handoff.py` around lines 138 - 144,
Replace the regex-based checks in the workflow validation test with YAML parsing
and inspect actual workflow steps, ensuring exactly one step uses
actions/setup-python and its uses value contains a 40-character SHA plus version
comment. Validate pull_request and the registered workflow path from parsed
YAML, and reject any unpinned third-party action references in the synced
workflow.

Source: Path instructions

assert "actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a" in workflow
assert "pyyaml==6.0.2" in workflow
assert (
Expand Down
13 changes: 12 additions & 1 deletion tests/workflows/test_template_drift_workflow.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
from __future__ import annotations

import re
from pathlib import Path

import yaml
Expand All @@ -19,5 +20,15 @@ def test_template_drift_workflow_installs_pyyaml_before_checker() -> None:
)
prior_steps = steps[:checker_index]

assert any(step.get("uses") == "actions/setup-python@v6" for step in prior_steps)
setup_python_steps = [
step
for step in prior_steps
if re.fullmatch(r"actions/setup-python@v\d+", str(step.get("uses", "")))
]

assert len(setup_python_steps) == 1
assert any(
re.fullmatch(r"actions/setup-python@v\d+", str(step.get("uses", "")))
for step in prior_steps
)
Comment on lines +23 to +33

@coderabbitai coderabbitai Bot Jul 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Assert interpreter setup before the PyYAML installation.

The test verifies that both setup-python and pip install pyyaml occur before the checker. It does not verify their relative order.

If the installation runs first, PyYAML can be installed for a different interpreter. The checker can then fail after setup-python selects the configured interpreter. Record both step indexes and assert:

setup_python_index < pyyaml_install_index < checker_index

As per path instructions, Python changes must prioritize correctness and test coverage.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/workflows/test_template_drift_workflow.py` around lines 23 - 33, Update
the workflow test around the existing setup_python_steps validation to locate
the step indexes for setup-python, the PyYAML installation, and the checker,
then assert setup_python_index < pyyaml_install_index < checker_index. Preserve
the existing checks that exactly one setup-python step exists and that the
required steps are present.

Source: Path instructions

assert any("pip install pyyaml" in step.get("run", "").lower() for step in prior_steps)
Loading