Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions .github/workflows/health-69-consumer-sync-shadow-evidence.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
name: Health 69 Consumer Sync Shadow Evidence
Comment thread
stranske marked this conversation as resolved.
Comment thread
stranske marked this conversation as resolved.

on:
workflow_dispatch:
schedule:
- cron: '17 8 * * 1'
push:
branches: [main]
paths:
- '.github/sync-manifest.yml'
- 'scripts/sync_manifest_compiler.py'
- 'scripts/build_consumer_sync_shadow_handoff.py'
- 'templates/consumer-repo/**'

permissions:
contents: read

concurrency:
group: consumer-sync-shadow-evidence
cancel-in-progress: false

jobs:
produce:
name: Produce read-only shadow evidence
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false

- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: '3.14'

- name: Install compiler dependency
run: python -m pip install "pyyaml==6.0.2"

- name: Compile typed consumer sync plan
run: |
mkdir -p consumer-sync-shadow-evidence
python scripts/sync_manifest_compiler.py \
--manifest .github/sync-manifest.yml \
--output-json consumer-sync-shadow-evidence/consumer-sync-plan.json

- name: Build non-authorizing Orchestrator handoff
env:
SHADOW_RUN_REF: >-
github-actions:${{ github.repository }}:${{ github.run_id }}:${{ github.run_attempt }}
run: |
python scripts/build_consumer_sync_shadow_handoff.py \
--plan consumer-sync-shadow-evidence/consumer-sync-plan.json \
--run-ref "$SHADOW_RUN_REF" \
--output consumer-sync-shadow-evidence/handoff.json

- name: Upload shadow evidence
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: consumer-sync-shadow-evidence-${{ github.run_id }}-${{ github.run_attempt }}
path: consumer-sync-shadow-evidence/
retention-days: 30

- name: Summarize shadow boundary
run: |
python - <<'PY' >> "$GITHUB_STEP_SUMMARY"
import json
from pathlib import Path

handoff = json.loads(
Path('consumer-sync-shadow-evidence/handoff.json').read_text()
)
print('## Consumer sync shadow evidence')
print(f"- Plan: {handoff['plan_id']}")
print(f"- Copy entries: {handoff['entry_count']}")
print(f"- Removals: {handoff['removal_count']}")
print('- Supervision: shadow')
print('- Write authority: false')
print('- Promotion allowed: false')
PY
1 change: 1 addition & 0 deletions docs/WORKFLOW_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ _Inline Gate helper_
- **`health-50-security-scan.yml`** — Security scanning workflow triggered on push, PR, and schedule. Runs CodeQL vulnerability scans using the configured PAT priority list without minting additional GitHub App tokens.
- **`health-67-integration-sync-check.yml`** — Daily + event-driven comparison between `templates/integration-repo/` and the Workflows-Integration-Tests repo. Manual dispatchers can now toggle the CI/versions/input checks individually to avoid cloning/running sections they don't need while still filing drift issues when enabled checks detect problems.
- **`health-68-consumer-sync-drift.yml`** — Daily + change-triggered drift detector for consumer repos. Uses the shared `scripts/list_registered_consumer_repos.py` helper to build the repo list (also accepts a manual override) before running `scripts/check_consumer_sync_drift.py`, and files/updates the `consumer-sync` issue when inconsistencies surface.
- **`health-69-consumer-sync-shadow-evidence.yml`** — Weekly, change-triggered, and manual read-only producer for the typed consumer-sync plan plus a non-authorizing Orchestrator handoff artifact. It has `contents: read` only and cannot mutate consumers or promote the shadow capability.
- **`health-71-sync-health-check.yml`** — Daily monitor that inspects the recent `maint-68-sync-consumer-repos` runs. Manual dispatches can tweak `lookback_runs`/`max_age_hours`, and the workflow now reuses `scripts/list_registered_consumer_repos.py` (instead of ad-hoc parsing) without minting an extra GitHub App token.
- **`health-72-template-sync.yml`** — Keeps manifest-declared exact template-sync files in sync between Workflows and the consumer template. For PRs from this repo it auto-runs `scripts/sync_templates.sh` + pushes deltas, then validates via `scripts/validate_template_sync.py`, relying only on the installation token.
- **`health-75-api-rate-diagnostic.yml`** — Hourly rate-limit snapshotter that polls every configured PAT/App pool, posts a tabular summary, and (on-demand) can fan out to load-balancer simulations and consumer-repo churn reports. Manual dispatch inputs gate the expensive legs (`include_consumer_repos`, `run_load_sharing_checks`, `verify_actions_access`) so that day-to-day runs stay light on API calls while still allowing deep dives when quota pressure crops up.
Expand Down
3 changes: 2 additions & 1 deletion docs/ci/WORKFLOWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Diagram labels that start with `.` are shorthand for files under `.github/workfl
- **PR checks:** [Gate](../../.github/workflows/pr-00-gate.yml) fans out to the reusable Python CI matrix and Docker smoke tests before its inline `summary` job publishes the commit status and PR comment. The **Gate summary job** keeps that follow-up comment updated with the latest artifacts.
- **Autofix path:** When Gate reports a failure, it dispatches `autofix_gate_failure`; [agents-autofix-dispatcher.yml](../../.github/workflows/agents-autofix-dispatcher.yml) receives that event and routes eligible PRs into [agents-autofix-loop.yml](../../.github/workflows/agents-autofix-loop.yml), while [Reusable 18 Autofix](../../.github/workflows/reusable-18-autofix.yml) remains a direct-call helper for hygiene-fix or patch-artifact callers. The diagram edge from Gate to Autofix Dispatch represents this repository dispatch hop, not a direct call into reusable autofix.
- **Agents control plane:** Successful Gate runs dispatch the [Agents 70 Orchestrator](../../.github/workflows/agents-70-orchestrator.yml), which coordinates the [Codex belt](../../.github/workflows/agents-71-codex-belt-dispatcher.yml) hand-off (dispatcher → worker → conveyor) and runs the built-in keepalive sweep unless the repository-level `keepalive:paused` label or `keepalive_enabled` flag disables it. The orchestrator summary exposes whether the pause label was detected and records the exact label name through the `keepalive_pause_label` output so downstream jobs can echo the control state.
- **Health checks:** The [Health 4x suite](../../.github/workflows/health-40-repo-selfcheck.yml), [Health 40 Sweep](../../.github/workflows/health-40-sweep.yml), [Health 41](../../.github/workflows/health-41-repo-health.yml), [Health 42](../../.github/workflows/health-42-actionlint.yml), [Health 43](../../.github/workflows/health-43-ci-signature-guard.yml), [Health 44](../../.github/workflows/health-44-gate-branch-protection.yml), [Health 46 Codex Auth Check](../../.github/workflows/health-codex-auth-check.yml), [Health 50 Security Scan](../../.github/workflows/health-50-security-scan.yml), [Health 51 Actions SAST](../../.github/workflows/health-51-zizmor.yml), [Health 52 Semgrep Scan](../../.github/workflows/health-52-semgrep.yml), [Health 67 Integration Sync Check](../../.github/workflows/health-67-integration-sync-check.yml), [Health 68 Consumer Sync Drift Check](../../.github/workflows/health-68-consumer-sync-drift.yml), and [Health 70 Validate Sync Manifest](../../.github/workflows/health-70-validate-sync-manifest.yml) workflows provide scheduled drift detection, enforcement snapshots, auth token monitoring, security scanning, and sync drift detection.
- **Health checks:** The [Health 4x suite](../../.github/workflows/health-40-repo-selfcheck.yml), [Health 40 Sweep](../../.github/workflows/health-40-sweep.yml), [Health 41](../../.github/workflows/health-41-repo-health.yml), [Health 42](../../.github/workflows/health-42-actionlint.yml), [Health 43](../../.github/workflows/health-43-ci-signature-guard.yml), [Health 44](../../.github/workflows/health-44-gate-branch-protection.yml), [Health 46 Codex Auth Check](../../.github/workflows/health-codex-auth-check.yml), [Health 50 Security Scan](../../.github/workflows/health-50-security-scan.yml), [Health 51 Actions SAST](../../.github/workflows/health-51-zizmor.yml), [Health 52 Semgrep Scan](../../.github/workflows/health-52-semgrep.yml), [Health 67 Integration Sync Check](../../.github/workflows/health-67-integration-sync-check.yml), [Health 68 Consumer Sync Drift Check](../../.github/workflows/health-68-consumer-sync-drift.yml), [Health 69 Consumer Sync Shadow Evidence](../../.github/workflows/health-69-consumer-sync-shadow-evidence.yml), and [Health 70 Validate Sync Manifest](../../.github/workflows/health-70-validate-sync-manifest.yml) workflows provide scheduled drift detection, read-only shadow evidence, enforcement snapshots, auth token monitoring, security scanning, and sync drift detection.
Comment thread
stranske marked this conversation as resolved.

Start with the [Workflow System Overview](WORKFLOW_SYSTEM.md) for the
bucket-level summary, the [keep vs retire roster](WORKFLOW_SYSTEM.md#final-topology-keep-vs-retire), and policy checklist. Return
Expand Down Expand Up @@ -188,6 +188,7 @@ Scheduled health jobs keep the automation ecosystem aligned:
* [`health-53-scorecard.yml`](../../.github/workflows/health-53-scorecard.yml) runs the OpenSSF Scorecard supply-chain posture scan (report-only/advisory) and uploads findings to code scanning (push to main, `branch_protection_rule`, weekly schedule, manual dispatch). The weekly repo-review coordinator also consumes the published Scorecard API results as a human-gated supplemental candidate source (`scorecard-scan.json`).
* [`health-67-integration-sync-check.yml`](../../.github/workflows/health-67-integration-sync-check.yml) validates that Workflows-Integration-Tests repo stays in sync with templates (push, `repository_dispatch`, daily schedule).
* [`health-68-consumer-sync-drift.yml`](../../.github/workflows/health-68-consumer-sync-drift.yml) detects drift in registered consumer repos (template/manifest changes, daily schedule, manual dispatch).
* [`health-69-consumer-sync-shadow-evidence.yml`](../../.github/workflows/health-69-consumer-sync-shadow-evidence.yml) publishes the typed plan and a read-only, non-promoting handoff artifact for the existing local Orchestrator shadow capability (weekly schedule, main changes, manual dispatch).
* [`health-70-validate-sync-manifest.yml`](../../.github/workflows/health-70-validate-sync-manifest.yml) validates that sync-manifest.yml is complete - ensures all sync-able files are declared (PR, push).
* [`health-71-sync-health-check.yml`](../../.github/workflows/health-71-sync-health-check.yml) monitors sync workflow health daily - creates issues if all recent runs failed or sync is stale (daily schedule, manual dispatch).
* [`health-72-template-sync.yml`](../../.github/workflows/health-72-template-sync.yml) validates that manifest-declared exact template-sync files are in sync with their consumer template copies (PR, push on exact-sync source/template changes).
Expand Down
1 change: 1 addition & 0 deletions docs/ci/WORKFLOW_SYSTEM.md
Original file line number Diff line number Diff line change
Expand Up @@ -726,6 +726,7 @@ Keep this table handy when you are triaging automation: it confirms which workfl
| **Health 53 Scorecard** (`health-53-scorecard.yml`, maintenance bucket) | `push` (main), `branch_protection_rule`, `schedule` (weekly), `workflow_dispatch` | Run the OpenSSF Scorecard supply-chain posture scan (report-only/advisory) and upload findings to code scanning. Published API results are consumed by the weekly repo-review Scorecard scan (`scorecard-scan.json`) as a human-gated candidate source. | ⚪ Automatic/scheduled | [Scorecard runs](https://github.com/stranske/Workflows/actions/workflows/health-53-scorecard.yml) |
| **Health 67 Integration Sync Check** (`health-67-integration-sync-check.yml`, maintenance bucket) | `push` (templates), `repository_dispatch`, `schedule` (daily) | Validate that Workflows-Integration-Tests repo stays in sync with templates. Creates issues when drift detected. | ⚪ Automatic/scheduled | [Integration sync runs](https://github.com/stranske/Workflows/actions/workflows/health-67-integration-sync-check.yml) |
| **Health 68 Consumer Sync Drift Check** (`health-68-consumer-sync-drift.yml`, maintenance bucket) | `push` (templates + manifest + scripts), `schedule` (daily), `workflow_dispatch` | Detect template/manifest drift across registered consumer repos. Creates or updates issues when drift is detected. | ⚪ Automatic/scheduled | [Consumer drift runs](https://github.com/stranske/Workflows/actions/workflows/health-68-consumer-sync-drift.yml) |
| **Health 69 Consumer Sync Shadow Evidence** (`health-69-consumer-sync-shadow-evidence.yml`, maintenance bucket) | `push` (manifest, compiler, handoff producer, or consumer templates), `schedule` (weekly), `workflow_dispatch` | Compile the typed sync plan and publish a bounded read-only Orchestrator handoff artifact. The workflow has no consumer write authority and cannot promote the capability. | ⚪ Automatic/scheduled/manual | [Consumer sync shadow evidence runs](https://github.com/stranske/Workflows/actions/workflows/health-69-consumer-sync-shadow-evidence.yml) |
| **Health 70 Validate Sync Manifest** (`health-70-validate-sync-manifest.yml`, maintenance bucket) | `pull_request`, `push` | Validate that sync-manifest.yml includes all sync-able files. Fails PRs that add workflows/prompts/scripts without updating manifest. | ⚪ Required on PRs | [Manifest validation runs](https://github.com/stranske/Workflows/actions/workflows/health-70-validate-sync-manifest.yml) |
| **Health 71 Sync Health Check** (`health-71-sync-health-check.yml`, maintenance bucket) | `schedule` (daily), `workflow_dispatch` | Monitor sync workflow health and create issues when all recent runs failed or sync is stale. | ⚪ Scheduled/manual | [Sync health check runs](https://github.com/stranske/Workflows/actions/workflows/health-71-sync-health-check.yml) |
| **Health 72 Template Sync** (`health-72-template-sync.yml`, maintenance bucket) | `pull_request`, `push` (exact-sync sources/templates) | Validate that manifest-declared exact template-sync files match their `templates/consumer-repo/` copies. | ⚪ Required on PRs | [Template sync validation runs](https://github.com/stranske/Workflows/actions/workflows/health-72-template-sync.yml) |
Expand Down
8 changes: 8 additions & 0 deletions docs/ops/CONSUMER_REPO_MAINTENANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,14 @@ stable `effect_fingerprint`; the plan adds `manifest_sha256` and `plan_id`.
Directory content hashes are computed from a sorted relative-path/content
inventory, so identical inputs produce byte-identical JSON.

`health-69-consumer-sync-shadow-evidence.yml` publishes that plan with a
`workflows.consumer-sync-shadow-handoff/v1` envelope for the existing local
Orchestrator capability `capability:reference-sync-hygiene-test-gate`. The
handoff is explicitly `shadow`, `write_authority=false`, and
`promotion_allowed=false`; classification, counterexamples, expiry, rollback,
and promotion blockers remain owned by Orchestrator's
`consumer_sync_shadow.py` dashboard.

To validate the manifest locally:

```bash
Expand Down
4 changes: 2 additions & 2 deletions langsmith-fleet-worker-attempt.json
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
{
"agent": "codex",
"cli_version": "0.125.0",
"emitted_at": "2026-07-11T03:38:47.089880Z",
"emitted_at": "2026-07-11T05:26:17.399554Z",
"execution_profile": "codex-default",
"fallback_models": [
"gpt-5.4"
],
"operation_role": "worker",
"pr_number": "2756",
"pr_number": "2757",
"requested_model": "gpt-5.5",
"runner": "reusable-codex-run",
"schema": "langsmith-fleet/v1",
Expand Down
101 changes: 101 additions & 0 deletions scripts/build_consumer_sync_shadow_handoff.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
#!/usr/bin/env python3
"""Build a bounded handoff for Orchestrator's consumer-sync shadow rail."""

from __future__ import annotations

import argparse
import hashlib
import json
import re
from pathlib import Path
from typing import Any

try:
from scripts.sync_manifest_compiler import PLAN_SCHEMA
except ImportError:
from sync_manifest_compiler import PLAN_SCHEMA # type: ignore[no-redef]


HANDOFF_SCHEMA = "workflows.consumer-sync-shadow-handoff/v1"
CAPABILITY_ID = "capability:reference-sync-hygiene-test-gate"
SHA256_RE = re.compile(r"^sha256:[0-9a-f]{64}$")
RUN_REF_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:/#@-]{0,255}$")
EFFECT_ALLOWLIST = ("create", "update", "remove", "skip", "no_change")


class ShadowHandoffError(ValueError):
pass


def _stable_hash(namespace: str, value: Any) -> str:
encoded = json.dumps(value, sort_keys=True, separators=(",", ":")).encode()
return "sha256:" + hashlib.sha256(namespace.encode() + b"\0" + encoded).hexdigest()


def build_handoff(plan: Any, *, run_ref: str) -> dict[str, Any]:
"""Validate the compiler identity and emit a deterministic, non-authorizing handoff."""
if not isinstance(plan, dict) or set(plan) != {
"schema",
"version",
"plan_id",
"manifest_sha256",
"entries",
"removals",
}:
raise ShadowHandoffError("invalid_consumer_sync_plan_fields")
if plan.get("schema") != PLAN_SCHEMA or plan.get("version") != 1:
raise ShadowHandoffError("unsupported_consumer_sync_plan_schema")
if not SHA256_RE.fullmatch(str(plan.get("plan_id") or "")):
raise ShadowHandoffError("invalid_consumer_sync_plan_id")
if not SHA256_RE.fullmatch(str(plan.get("manifest_sha256") or "")):
raise ShadowHandoffError("invalid_consumer_sync_manifest_hash")
if not isinstance(plan.get("entries"), list) or not isinstance(plan.get("removals"), list):
raise ShadowHandoffError("consumer_sync_plan_collections_not_arrays")
if not RUN_REF_RE.fullmatch(run_ref):
raise ShadowHandoffError("invalid_shadow_run_ref")
if any(
marker in run_ref.lower() for marker in ("token", "secret", "password", "api-key", "apikey")
):
raise ShadowHandoffError("secret_like_shadow_run_ref")
core = {
"schema": HANDOFF_SCHEMA,
"version": 1,
"capability_id": CAPABILITY_ID,
"plan_schema": PLAN_SCHEMA,
"plan_id": plan["plan_id"],
"manifest_sha256": plan["manifest_sha256"],
"entry_count": len(plan["entries"]),
"removal_count": len(plan["removals"]),
"plan_filename": "consumer-sync-plan.json",
"run_ref": run_ref,
"supervision_mode": "shadow",
"write_authority": False,
"promotion_allowed": False,
"effect_allowlist": list(EFFECT_ALLOWLIST),
"kill_switch": "ORCH_REFERENCE_WORKFLOW_DISABLED=1",
"consumer": "Orchestrator/consumer_sync_shadow.py",
}
return {**core, "handoff_id": _stable_hash("consumer-sync-shadow-handoff", core)}


def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--plan", type=Path, required=True)
parser.add_argument("--run-ref", required=True)
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args(argv)
try:
plan = json.loads(args.plan.read_text(encoding="utf-8"))
handoff = build_handoff(plan, run_ref=args.run_ref)
except (OSError, json.JSONDecodeError, ShadowHandoffError) as exc:
parser.error(str(exc))
args.output.write_text(
json.dumps(handoff, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
print(json.dumps(handoff, sort_keys=True))
return 0
Comment thread
stranske marked this conversation as resolved.


if __name__ == "__main__":
raise SystemExit(main())
Loading
Loading