Add automated quarterly metrics collection and reporting

.claude/skills/analyze-quarterly-metrics/SKILL.md

@@ -0,0 +1,200 @@
+---
+name: analyze-quarterly-metrics
+description: This skill should be used when the user asks to "analyze quarterly metrics", "analyze the quarter", "generate quarterly report", "quarterly analysis", mentions a specific quarter like "2026-Q2", or discusses trends, risks, and growth opportunities for Linux System Roles metrics.
+---
+
+# Quarterly Metrics Analysis Skill
+
+This skill analyzes quarterly metrics data and generates a comprehensive report with insights, trends, risks, and recommendations.
+
+## What to do
+
+1. **Determine if the quarter is complete or in-progress:**
+   - Get today's date and compare to the quarter being analyzed
+   - If analyzing current or future quarter: Add a disclaimer that data is PARTIAL/INCOMPLETE
+   - Adjust your interpretation: low numbers may just mean "not much time has passed yet", not a crisis
+   - For partial quarters: focus on trends and rates rather than absolute numbers
+
+2. **Load the raw data** for the specified quarter:
+   - Read the summary CSV files: `data/github_prs_summary.csv`, `data/github_issues_summary.csv`, `data/galaxy_legacy_summary.csv`, `data/galaxy_collections_summary.csv`
+   - Extract data for the specified quarter and the previous 3-4 quarters for comparison
+   - If `data/{{quarter}}/galaxy_legacy.csv` exists, read it for per-role download analysis
+
+3. **Calculate key metrics** from the raw data:
+   - **PR Merge Rate**: PRs Merged / (PRs Created - PRs Open) × 100 (excludes PRs still under review)
+   - **External Acceptance Rate**: External PRs Merged / (External PRs Created - External PRs Open) × 100
+   - **External Contribution %**: External PRs Created / PRs Created × 100
+   - **Issue Resolution Rate**: Issues Closed / Issues Created × 100
+   - **QoQ Growth rates**: Compare current quarter to previous quarter
+   - **Fastest growing roles**: If per-role data exists, identify top gainers by comparing to previous quarter
+
+4. **Analyze the data** and generate a detailed report with these sections:
+
+   ### Executive Summary (2-3 sentences)
+   - High-level overview of the quarter's performance
+   - Most significant achievement or concern
+
+   ### Key Findings (Quick-Scan Section)
+   **Improvements:**
+   - Notable positive changes and improvements from previous quarter
+   - Metrics that show upward trends
+
+   **Critical Concerns:**
+   1. Most urgent issues requiring immediate attention (ranked by severity)
+   2. Trending problems that could impact project health
+   3. Metrics showing significant decline
+
+   **Top Recommendations:**
+   1. **Immediate:** Critical actions needed within 2 weeks
+   2. **Short-term:** Actions needed through end of quarter
+   3. **Ongoing:** Continuous improvement areas
+
+   ### Key Metrics Overview
+   - Present the main numbers (PRs, Issues, Downloads)
+   - Compare to previous quarter (QoQ change)
+   - Compare to same quarter last year (YoY if available)
+
+   ### Trend Analysis
+   - **PR Activity**: Are PRs increasing/decreasing? Merge rate trends?
+   - **External Contributions**: Growing or declining? Acceptance rate healthy?
+   - **Issue Management**: Resolution rate? Backlog growing?
+   - **Galaxy Downloads**: Which collections/roles are trending? Growth rate sustainable?
+
+   ### Highlights & Achievements
+   - What went well this quarter?
+   - Notable improvements in metrics
+   - Fastest growing roles or areas
+
+   ### Risks & Concerns
+   - Declining trends that need attention
+   - Bottlenecks or capacity issues
+   - Quality concerns (low acceptance rates, etc.)
+   - Areas falling behind
+
+   ### Growth Opportunities
+   - Underutilized roles with potential
+   - Areas showing momentum
+   - External contributor engagement opportunities
+
+   ### Recommendations
+   - 2-4 specific, actionable recommendations based on the data
+   - Focus on addressing risks and capturing opportunities
+
+5. **Be specific with numbers**: Always cite actual metrics, percentages, and comparisons. Don't use vague language like "significant" without quantifying it.
+
+6. **Be smart about partial quarters**:
+   - If the quarter is incomplete, DO NOT flag low absolute numbers as risks
+   - Focus on rates (merge rate, acceptance rate) rather than volumes for partial data
+   - Only flag things as concerns if they represent actual problems, not just "we're only 2 weeks into the quarter"
+   - Make it clear in the Executive Summary if data is partial
+
+7. **Validate the report for accuracy**:
+   - Re-read all numeric claims in your draft
+   - For each number: verify it matches the correct CSV column/row
+   - For each comparison: verify the direction (higher/lower) matches the math
+   - Check for contradictions: "up from X, but below X" is impossible
+   - If you find errors, fix them before saving
+
+8. **Automatically save the report**:
+   - Save to `reports/{{quarter}}-analysis.md`
+   - Create the reports directory if it doesn't exist
+   - **Safe to overwrite**: If the report file already exists, overwrite it (reports are tracked in git, so previous versions are preserved)
+   - Notify the user where the file was saved
+
+9. **Output the analysis** directly to the user in Markdown format.
+
+## Guidelines
+
+- **Detect partial quarters**: Compare today's date to the quarter end date. If analyzing an incomplete quarter, prominently note this and adjust your analysis
+- **Compare to historical data**: Always reference previous quarters for context
+- **Identify patterns**: Look for multi-quarter trends, not just single-quarter changes
+- **Be balanced**: Include both positive and negative findings
+- **Be actionable**: Recommendations should be specific and implementable
+- **Consider seasonality**: Note if quarterly patterns are typical or anomalous
+- **Highlight outliers**: Call out unusual spikes or drops in any metric
+- **Don't cry wolf on partial data**: For incomplete quarters, only flag true risks (bad rates, declining trends), not low volumes that are expected mid-quarter
+- **Key Findings should be scannable**: The Key Findings section should provide a quick executive summary that busy stakeholders can read in 30 seconds. Keep it concise with 2-3 improvements, 3-4 critical concerns, and 3 top recommendations
+
+### Data Accuracy and Validation
+
+**CRITICAL: Verify all numbers before making claims**
+
+1. **Use correct data sources**:
+   - When discussing a specific collection (e.g., fedora.linux_system_roles), use that collection's column, NOT the Total Downloads column
+   - When discussing totals, clearly state "total across all collections"
+   - Double-check: Does the number in your sentence match the CSV cell you're referencing?
+
+2. **Verify comparison direction**:
+   - If A > B: use "higher than", "above", "exceeds", "increased from"
+   - If A < B: use "lower than", "below", "decreased from", "down from"
+   - If A ≈ B: use "similar to", "comparable to", "roughly equal to"
+   - **NEVER** say "A is well below B" when A > B or vice versa
+
+3. **Validate calculations**:
+   - QoQ growth = ((Current - Previous) / Previous) × 100
+   - If growth is positive, use "increased" or "up"; if negative, use "decreased" or "down"
+   - Check sign: positive growth means increase, negative growth means decrease
+
+4. **Before finalizing the report**:
+   - Re-read each numeric claim
+   - Verify the number matches the data source (correct CSV column/row)
+   - Verify the comparison direction matches the math (higher/lower/equal)
+   - Check for internal contradictions (e.g., "up from X, but well below X")
+
+## Example invocations
+
+**Via skill invocation:**
+User types: `/analyze-quarterly-metrics 2026-Q2`
+
+**Via natural language:**
+- "Analyze the quarterly metrics for 2026-Q2"
+- "Generate a quarterly report for Q2 2026"
+- "What do the metrics show for this quarter?"
+
+You should:
+1. Extract the quarter from the user's request or args (format: YYYY-QN, e.g., 2026-Q2)
+2. Determine if the quarter is complete or in-progress based on today's date
+3. Read all the raw metrics data from CSV files for that quarter
+4. Calculate derived metrics from the raw data (merge rates, growth rates, etc.)
+5. Read historical data for comparison (previous 3-4 quarters)
+6. Generate the comprehensive analysis (being smart about partial data)
+7. Validate the report: verify all numbers match data sources and comparisons are correct
+8. Automatically save to `reports/{{quarter}}-analysis.md`
+9. Present it to the user in Markdown format and notify where it was saved
+
+## Important
+
+- **Always check if the quarter is complete**: Compare today's date to quarter end
+  - Q1: January 1 - March 31
+  - Q2: April 1 - June 30
+  - Q3: July 1 - September 30
+  - Q4: October 1 - December 31
+- For partial quarters: add clear disclaimer in Executive Summary and adjust risk assessment
+- **Always calculate metrics from CSV data** - don't rely on pre-computed derived_metrics.json
+- If the quarter directory doesn't exist (like 2026-Q1), work with summary CSV data only
+- For per-role analysis, check if `data/{{quarter}}/galaxy_legacy.csv` exists
+- Always show your reasoning and cite specific data points
+- **Automatically save** the report to `reports/{{quarter}}-analysis.md` (create directory if needed)
+- Safe to overwrite existing report files - they're tracked in git
+- Notify the user where the file was saved
+
+## Metric Calculation Formulas
+
+Use these formulas when calculating metrics from the raw CSV data:
+
+**PR Metrics:**
+- Merge Rate = (PRs Merged) / (PRs Created - PRs Open) × 100
+  - Excludes PRs still under review from the calculation
+- External Acceptance = (External PRs Merged) / (External PRs Created - External PRs Open) × 100  
+  - Excludes external PRs still under review
+- External % = (External PRs Created) / (PRs Created) × 100
+- QoQ Growth = ((Current - Previous) / Previous) × 100
+
+**Issue Metrics:**
+- Resolution Rate = (Issues Closed) / (Issues Created) × 100
+- External % = (External Issues Created) / (Issues Created) × 100
+- QoQ Growth = ((Current - Previous) / Previous) × 100
+
+**Galaxy Metrics:**
+- Legacy QoQ Growth = ((Current Total - Previous Total) / Previous Total) × 100
+- Collections QoQ Growth = ((Current Total - Previous Total) / Previous Total) × 100

.github/workflows/quarterly-metrics.yml

@@ -0,0 +1,171 @@
+name: Quarterly Metrics Report
+
+on:
+  # Scheduled: Run on the last day of each quarter at 9am UTC
+  schedule:
+    - cron: '0 9 31 3,12 *'  # March 31, December 31 at 9am UTC
+    - cron: '0 9 30 6,9 *'   # June 30, September 30 at 9am UTC
+
+  # Manual trigger with optional parameters
+  workflow_dispatch:
+    inputs:
+      quarter:
+        description: 'Quarter (e.g., 2025-Q1)'
+        required: false
+        type: string
+      date_range:
+        description: 'Date range (e.g., 2025-01-01..2025-03-31)'
+        required: false
+        type: string
+
+jobs:
+  generate-report:
+    runs-on: ubuntu-latest
+
+    permissions:
+      contents: write        # Required to commit and push changes
+      pull-requests: write   # Required to create pull requests
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: '3.11'
+          cache: 'pip'
+
+      - name: Install Python dependencies
+        run: |
+          pip install -r requirements.txt
+
+      - name: Install GitHub CLI
+        run: |
+          # gh CLI is pre-installed on ubuntu-latest runners
+          gh --version
+
+      - name: Determine quarter and date range
+        id: quarter
+        env:
+          INPUT_QUARTER: ${{ github.event.inputs.quarter }}
+          INPUT_DATE_RANGE: ${{ github.event.inputs.date_range }}
+        run: |
+          # Use input if provided, otherwise calculate current quarter
+          if [ -n "$INPUT_QUARTER" ]; then
+            QUARTER="$INPUT_QUARTER"
+          else
+            # Calculate quarter from current date
+            YEAR=$(date +%Y)
+            MONTH=$(date +%m)
+            Q=$(( (MONTH - 1) / 3 + 1 ))
+            QUARTER="${YEAR}-Q${Q}"
+          fi
+          echo "quarter=${QUARTER}" >> $GITHUB_OUTPUT
+
+          # Determine date range
+          if [ -n "$INPUT_DATE_RANGE" ]; then
+            DATE_RANGE="$INPUT_DATE_RANGE"
+          else
+            # Auto-calculate date range based on quarter
+            YEAR=$(echo ${QUARTER} | cut -d'-' -f1)
+            Q_NUM=$(echo ${QUARTER} | cut -d'Q' -f2)
+
+            case ${Q_NUM} in
+              1)
+                DATE_RANGE="${YEAR}-01-01..${YEAR}-03-31"
+                ;;
+              2)
+                DATE_RANGE="${YEAR}-04-01..${YEAR}-06-30"
+                ;;
+              3)
+                DATE_RANGE="${YEAR}-07-01..${YEAR}-09-30"
+                ;;
+              4)
+                DATE_RANGE="${YEAR}-10-01..${YEAR}-12-31"
+                ;;
+            esac
+          fi
+          echo "date_range=${DATE_RANGE}" >> $GITHUB_OUTPUT
+
+      - name: Collect GitHub statistics
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          QUARTER: ${{ steps.quarter.outputs.quarter }}
+          DATE_RANGE: ${{ steps.quarter.outputs.date_range }}
+        run: bash scripts/collect_all_github_stats.sh
+
+      - name: Collect Galaxy statistics
+        env:
+          GALAXY_API_KEY: ${{ secrets.GALAXY_API_KEY }}
+          QUARTER: ${{ steps.quarter.outputs.quarter }}
+        run: python3 scripts/collect_galaxy_stats.py
+
+      - name: Update quarterly summary files
+        env:
+          QUARTER: ${{ steps.quarter.outputs.quarter }}
+        run: python3 scripts/update_quarterly_summary.py
+
+      - name: Generate graphs
+        env:
+          QUARTER: ${{ steps.quarter.outputs.quarter }}
+        run: python3 scripts/generate_graphs.py
+
+      - name: Create Pull Request with results
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          QUARTER: ${{ steps.quarter.outputs.quarter }}
+        run: |
+          git config user.name "github-actions[bot]"
+          git config user.email "github-actions[bot]@users.noreply.github.com"
+
+          git add data/ reports/
+
+          # Check if there are changes to commit
+          if git diff --staged --quiet; then
+            echo "No changes to commit"
+            exit 0
+          fi
+
+          # Create or switch to branch for this quarter
+          BRANCH_NAME="metrics/${QUARTER}"
+          if git ls-remote --exit-code --heads origin "$BRANCH_NAME" >/dev/null 2>&1; then
+            echo "Branch $BRANCH_NAME exists, updating it"
+            git fetch origin "$BRANCH_NAME"
+            git checkout -B "$BRANCH_NAME" "origin/$BRANCH_NAME"
+          else
+            echo "Creating new branch $BRANCH_NAME"
+            git checkout -b "$BRANCH_NAME"
+          fi
+
+          # Commit changes
+          git commit -m "Add metrics for ${QUARTER}" -m "Generated by GitHub Actions workflow" -m "- Data: data/${QUARTER}/" -m "- Graphs: reports/images/"
+
+          # Push the branch
+          git push --set-upstream origin "$BRANCH_NAME"
+
+          # Create Pull Request if it doesn't exist
+          PR_URL=$(gh pr list --head "$BRANCH_NAME" --base main --json url -q '.[0].url')
+          if [ -n "$PR_URL" ]; then
+            echo "PR already exists: $PR_URL"
+          else
+            gh pr create \
+              --title "Quarterly Metrics - ${QUARTER}" \
+              --body "Automated metrics collection for ${QUARTER}. **Data:** \`data/${QUARTER}/\` **Graphs:** \`reports/images/\` After merging, generate analysis: \`/analyze-quarterly-metrics ${QUARTER}\`" \
+              --base main \
+              --head "$BRANCH_NAME"
+          fi
+
+      - name: Create workflow summary
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          QUARTER: ${{ steps.quarter.outputs.quarter }}
+        run: |
+          echo "## Quarterly Metrics - ${QUARTER}" >> $GITHUB_STEP_SUMMARY
+          echo "" >> $GITHUB_STEP_SUMMARY
+          PR_URL=$(gh pr view metrics/${QUARTER} --json url -q .url 2>/dev/null || echo "")
+          if [ -n "$PR_URL" ]; then
+            echo "**[Review Pull Request](${PR_URL})**" >> $GITHUB_STEP_SUMMARY
+          else
+            echo "Data: \`data/${QUARTER}/\`" >> $GITHUB_STEP_SUMMARY
+          fi

.gitignore

@@ -0,0 +1,2 @@
+__pycache__/
+venv/