Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 17 additions & 12 deletions .github/workflows/compliance-audit-and-improvement.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# Job 3: Claude analyzes both datasets in six phases:
# Phase 1-3: Load data, categorize findings, research root causes.
# Phase 4: Evaluate against industry best practices & emerging capabilities.
# Phase 5: Create actionable issues per repo (claude label for agent pickup).
# Phase 5: Create actionable issues per repo (dev-lead label for agent pickup).
# Phase 6: Summary report.
# Standard: https://github.com/${{ github.repository_owner }}/.github/tree/main/standards
name: Org Standards Compliance Audit
Expand Down Expand Up @@ -52,6 +52,7 @@ jobs:
repos_with_findings: ${{ steps.audit.outputs.repos_with_findings }}
issues_added: ${{ steps.audit.outputs.issues_added }}
issues_existing: ${{ steps.audit.outputs.issues_existing }}
issues_retriggered: ${{ steps.audit.outputs.issues_retriggered }}
issues_removed: ${{ steps.audit.outputs.issues_removed }}
steps:
- name: Checkout .github repo
Expand Down Expand Up @@ -83,12 +84,14 @@ jobs:
if [ -f "$COUNTS_FILE" ]; then
ISSUES_ADDED=$(jq '.added' "$COUNTS_FILE")
ISSUES_EXISTING=$(jq '.existing' "$COUNTS_FILE")
ISSUES_RETRIGGERED=$(jq '.retriggered // 0' "$COUNTS_FILE")
ISSUES_REMOVED=$(jq '.removed' "$COUNTS_FILE")
else
ISSUES_ADDED=0; ISSUES_EXISTING=0; ISSUES_REMOVED=0
ISSUES_ADDED=0; ISSUES_EXISTING=0; ISSUES_RETRIGGERED=0; ISSUES_REMOVED=0
fi
echo "issues_added=$ISSUES_ADDED" >> "$GITHUB_OUTPUT"
echo "issues_existing=$ISSUES_EXISTING" >> "$GITHUB_OUTPUT"
echo "issues_retriggered=$ISSUES_RETRIGGERED" >> "$GITHUB_OUTPUT"
echo "issues_removed=$ISSUES_REMOVED" >> "$GITHUB_OUTPUT"

- name: Write step summary
Expand Down Expand Up @@ -273,7 +276,7 @@ jobs:

# -----------------------------------------------------------------------
# Job 3: Combined analysis — Claude reviews both datasets
# Creates actionable issues in the appropriate repo with the claude label.
# Creates actionable issues in the appropriate repo with the dev-lead label.
# -----------------------------------------------------------------------
analyze:
name: Analyze & Create Issues (Claude)
Expand Down Expand Up @@ -332,6 +335,7 @@ jobs:
- Repos with findings: ${{ needs.audit.outputs.repos_with_findings }}
- Issues added (new): ${{ needs.audit.outputs.issues_added }}
- Issues existing (updated): ${{ needs.audit.outputs.issues_existing }}
- Issues re-triggered (dev-lead re-engaged on persistent findings): ${{ needs.audit.outputs.issues_retriggered }}
- Issues removed (resolved): ${{ needs.audit.outputs.issues_removed }}

### Health Survey (runtime telemetry)
Expand Down Expand Up @@ -445,7 +449,7 @@ jobs:
exclusively **proposed new or improved standards** for the org. For each top
opportunity (max 2-3), create a standards proposal issue in `${{ github.repository_owner }}/.github`:
- Title: `Standards: <concise description of the proposed standard>`
- Labels: `claude,enhancement`
- Labels: `dev-lead,enhancement`
- Body must include:
- **Proposed Standard** — the specific policy, workflow, or configuration to adopt
- **Rationale** — why this matters, linked to the feasibility/impact/urgency assessment
Expand All @@ -461,10 +465,10 @@ jobs:
- Issues affecting multiple repos or org-wide concerns go in `${{ github.repository_owner }}/.github`
- Standards improvement proposals go in `${{ github.repository_owner }}/.github`

**IMPORTANT: Every issue MUST have the `claude` label** so it gets picked up for implementation.
Ensure the `claude` label exists in the target repo before creating the issue:
**IMPORTANT: Every issue MUST have the `dev-lead` label** so it gets picked up for implementation.
Ensure the `dev-lead` label exists in the target repo before creating the issue:
```bash
gh label create claude --repo ${{ github.repository_owner }}/<repo> --color "8B5CF6" --description "For Claude agent pickup" 2>/dev/null || true
gh label create dev-lead --repo ${{ github.repository_owner }}/<repo> --color "8B5CF6" --description "For dev-lead agent pickup" 2>/dev/null || true
```

Additional labels by type: `bug`, `security`, `ci`, `automation`, `enhancement`, `documentation`
Expand All @@ -477,7 +481,7 @@ jobs:
```bash
gh issue create --repo ${{ github.repository_owner }}/<target-repo> \
--title "<severity prefix>: <concise title>" \
--label "claude,<other-labels>" \
--label "dev-lead,<other-labels>" \
--body "<structured body>"
```

Expand Down Expand Up @@ -513,9 +517,9 @@ jobs:
per-repo issues; your job is to identify systemic patterns and create higher-level
issues for them.
- If a similar issue exists, add a comment with latest findings instead
- When commenting on existing issues, also ensure the `claude` label is present:
- When commenting on existing issues, also ensure the `dev-lead` label is present:
```bash
gh issue edit <number> --repo ${{ github.repository_owner }}/<repo> --add-label claude
gh issue edit <number> --repo ${{ github.repository_owner }}/<repo> --add-label dev-lead
```

**Before writing the Phase 6 summary**, gather linked PR data for all issues you
Expand Down Expand Up @@ -556,6 +560,7 @@ jobs:
|--------|-------|
| Added (new) | ${{ needs.audit.outputs.issues_added }} |
| Existing (updated) | ${{ needs.audit.outputs.issues_existing }} |
| Re-triggered (dev-lead re-engaged) | ${{ needs.audit.outputs.issues_retriggered }} |
| Removed (resolved) | ${{ needs.audit.outputs.issues_removed }} |

Group by compliance issue type — one subsection per distinct check/finding type,
Expand Down Expand Up @@ -597,9 +602,9 @@ jobs:
## Rules

- **Do not fix code or push changes.** Analysis and issue creation only.
- **Do not close or modify existing issues** beyond adding the `claude` label.
- **Do not close or modify existing issues** beyond adding the `dev-lead` label.
- **Do not create PRs.** Only create issues with actionable recommendations.
- **Every issue gets the `claude` label.** No exceptions.
- **Every issue gets the `dev-lead` label.** No exceptions.
- **Repo-specific issues go in that repo.** Org-wide issues go in `.github`.
- **Be specific.** Include run IDs, URLs, and exact error messages.
- **Deduplicate aggressively.** One well-written issue beats five vague ones.
Expand Down
48 changes: 37 additions & 11 deletions scripts/compliance-audit.sh
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ ISSUE_COUNTS_FILE="$REPORT_DIR/issue-counts.json"
ISSUES_ADDED=0
ISSUES_EXISTING=0
ISSUES_REMOVED=0
ISSUES_RETRIGGERED=0

REQUIRED_WORKFLOWS=(ci.yml sonarcloud.yml dev-lead.yml dependabot-automerge.yml dependency-audit.yml agent-shield.yml pr-review-mention.yml)
# Note: codeql.yml is intentionally NOT in REQUIRED_WORKFLOWS. CodeQL is now
Expand Down Expand Up @@ -130,6 +131,11 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=lib/push-protection.sh
. "$SCRIPT_DIR/lib/push-protection.sh"

# Shared dev-lead retrigger helpers — dl_dev_lead_active() and
# dl_cycle_trigger_label(). Used to re-engage dev-lead on persistent findings.
# shellcheck source=lib/dev-lead-retrigger.sh
. "$SCRIPT_DIR/lib/dev-lead-retrigger.sh"

# ---------------------------------------------------------------------------
# Ecosystem detection
# ---------------------------------------------------------------------------
Expand Down Expand Up @@ -1155,9 +1161,9 @@ ensure_audit_label() {
--description "$AUDIT_LABEL_DESC" \
--color "$AUDIT_LABEL_COLOR" \
--force 2>/dev/null || true
gh label create "claude" \
gh label create "dev-lead" \
--repo "$ORG/$repo" \
--description "For Claude agent pickup" \
--description "For dev-lead agent pickup" \
--color "8B5CF6" \
--force 2>/dev/null || true
}
Expand Down Expand Up @@ -1214,14 +1220,33 @@ This finding is still open.
**Detail:** $detail

**Standard:** [$standard_ref](https://github.com/$ORG/.github/blob/main/$standard_ref)" 2>/dev/null || update_ok=false
# Ensure claude label is present on pre-existing issues regardless
gh issue edit "$existing" --repo "$ORG/$repo" --add-label "claude" 2>/dev/null || true
if [ "$update_ok" = "true" ]; then
info "Updated existing issue #$existing in $repo for: $check"
ISSUES_EXISTING=$((ISSUES_EXISTING + 1))
else
warn "Failed to update existing issue #$existing in $repo for: $check"
fi

# Re-engage dev-lead on findings that PERSIST across audits.
#
# dev-lead listens on issues:labeled and fires only once per label
# application. The `dev-lead` label is already present on a pre-existing
# issue, so a plain --add-label is a no-op that emits no event and dev-lead
# is never re-triggered. To give it a fresh chance to produce a fix PR we
# cycle the label (remove + re-add) — UNLESS dev-lead is already working
# this issue (open dev-lead PR or `in-progress` label), in which case we
# leave it alone (the label is already present, so no action is needed).
if dl_dev_lead_active "$ORG" "$repo" "$existing"; then
info "Existing issue #$existing in $repo — dev-lead already active, not re-triggering"
Comment thread
don-petry marked this conversation as resolved.
elif dl_cycle_trigger_label "$ORG" "$repo" "$existing" "dev-lead" "$DRY_RUN"; then
info "Re-triggered dev-lead on persistent issue #$existing in $repo for: $check"
ISSUES_RETRIGGERED=$((ISSUES_RETRIGGERED + 1))
else
# Cycle failed mid-way — ensure the label is at least present so the issue
# is not left without its trigger label.
gh issue edit "$existing" --repo "$ORG/$repo" --add-label "dev-lead" 2>/dev/null || true
warn "Failed to re-trigger dev-lead on issue #$existing in $repo for: $check"
fi
# Record existing issue for umbrella
jq --null-input \
--arg repo "$repo" \
Expand Down Expand Up @@ -1294,11 +1319,11 @@ ${remediation_steps}
*This issue was automatically created by the [weekly compliance audit](https://github.com/${ORG}/.github/blob/main/.github/workflows/compliance-audit.yml).*"

local issue_url
# Individual finding issues get both compliance-audit and claude labels so agents can pick them up.
# Individual finding issues get both compliance-audit and dev-lead labels so agents can pick them up.
issue_url=$(gh issue create --repo "$ORG/$repo" \
--title "$search_title" \
--label "$AUDIT_LABEL" \
--label "claude" \
--label "dev-lead" \
--body "$body" 2>/dev/null || echo "")

if [ -n "$issue_url" ]; then
Expand Down Expand Up @@ -1438,7 +1463,7 @@ Findings are grouped by remediation category. Address each category together to
umbrella_url=$(gh issue create --repo "$ORG/.github" \
--title "$title" \
--label "$AUDIT_LABEL" \
--label "claude" \
--label "dev-lead" \
--body "$body" 2>/dev/null || echo "")

if [ -n "$umbrella_url" ]; then
Expand Down Expand Up @@ -1796,7 +1821,7 @@ main() {
done

# Create one umbrella issue per audit run grouping all findings by remediation category.
# Both individual issues and the umbrella get the `claude` label for agent pickup.
# Both individual issues and the umbrella get the `dev-lead` label for agent pickup.
create_umbrella_issue

# Append per-check issue links and related open PRs to the step summary
Expand All @@ -1808,8 +1833,8 @@ main() {

# Write issue-management counts and append to summary (conditional on issue management running)
if [ "$CREATE_ISSUES" = "true" ] && [ "$DRY_RUN" != "true" ]; then
printf '{"added":%d,"existing":%d,"removed":%d}\n' \
"$ISSUES_ADDED" "$ISSUES_EXISTING" "$ISSUES_REMOVED" > "$ISSUE_COUNTS_FILE"
printf '{"added":%d,"existing":%d,"removed":%d,"retriggered":%d}\n' \
"$ISSUES_ADDED" "$ISSUES_EXISTING" "$ISSUES_REMOVED" "$ISSUES_RETRIGGERED" > "$ISSUE_COUNTS_FILE"
cat >> "$SUMMARY_FILE" <<HEREDOC

## Issue Management
Expand All @@ -1818,10 +1843,11 @@ main() {
|--------|-------|
| Added (new) | $ISSUES_ADDED |
| Existing (updated) | $ISSUES_EXISTING |
| Re-triggered (dev-lead re-engaged) | $ISSUES_RETRIGGERED |
| Removed (resolved) | $ISSUES_REMOVED |
HEREDOC
else
printf '{"added":0,"existing":0,"removed":0}\n' > "$ISSUE_COUNTS_FILE"
printf '{"added":0,"existing":0,"removed":0,"retriggered":0}\n' > "$ISSUE_COUNTS_FILE"
cat >> "$SUMMARY_FILE" <<HEREDOC

## Issue Management
Expand Down
Loading
Loading