chore: add Renovate config with git-submodules tracking

[ralphbean]

Summary

• Add renovate.json to enable automated dependency updates
• Enable git-submodules so Renovate tracks agent-eval-harness upstream changes
• Mirror fullsend repo's base config (recommended preset, automerge patches, ignore self-references)

Test plan

• Renovate onboarding succeeds after merge
• Renovate opens a PR when opendatahub-io/agent-eval-harness pushes a new commit to main

🤖 Generated with Claude Code

[qodo-code-review]

PR Summary by Qodo

Add Renovate config to track git submodule updates

⚙️ Configuration changes 🕐 Less than 10 minutes

AI Description

• Add a Renovate configuration to enable automated dependency maintenance PRs.
• Enable git-submodules scanning so the agent-eval-harness submodule stays current.
• Autocomplete low-risk patch/pin updates via automerge with conservative PR rate limiting.

Diagram

graph TD
  R{{"Renovate Bot"}} --> C["renovate.json"] --> Repo["Repository"] --> PR["Renovate PRs"]
  Repo --> Sub["git submodule: agent-eval-harness"]

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Scheduled GitHub Action to bump submodule pointer

• ➕ Full control over update cadence and branching policy
• ➕ No external Renovate onboarding/setup required
• ➖ Custom maintenance burden (scripts, auth, error handling)
• ➖ Harder to extend consistently to other dependency ecosystems later

2. Dependabot-only approach

• ➕ Native GitHub integration and familiar workflow
• ➖ Limited/unsupported git-submodule update capabilities compared to Renovate
• ➖ Less flexible rule system for cross-ecosystem automation

Recommendation: The PR’s approach (Renovate with git-submodules enabled) is the most scalable option: it directly supports submodule tracking and can be extended to other dependency managers with consistent policy controls (rate limiting, automerge rules, ignore rules). The alternatives either add bespoke maintenance (scheduled workflow) or lack strong submodule support (Dependabot-only).

Files changed (1) +23 / -0

Other (1) +23 / -0

renovate.jsonAdd Renovate baseline config and enable git-submodules +23/-0

Add Renovate baseline config and enable git-submodules

• Introduces a Renovate configuration extending the recommended preset, enabling platform automerge for patch/pin updates, and limiting PR creation rate. Enables git-submodules scanning so Renovate can propose updates when the agent-eval-harness submodule’s upstream advances.

renovate.json

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 11:03 AM UTC · Completed 11:15 AM UTC
Commit: 2596da2 · View workflow run →

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 55 rules

✅ Skills: 4 invoked

  code-review

  code-implementation

  pr-review

  docs-review

1. No linked issue for Renovate ✗ Dismissed 📜 Skill insight § Compliance

Description

This PR introduces a new Renovate configuration (including automerge behavior and submodule
tracking) with 20+ added lines but the PR description does not link an authorizing issue/ADR.
Non-trivial infrastructure/config changes require explicit authorization via a linked issue.

Code

renovate.json[R1-23]

+{
+  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
+  "extends": ["config:recommended"],
+  "automergeType": "pr",
+  "platformAutomerge": true,
+  "prHourlyLimit": 1,
+  "packageRules": [
+    {
+      "description": "Automerge low-risk updates (patch bumps and pin updates)",
+      "matchUpdateTypes": ["patch", "pin"],
+      "automerge": true
+    },
+    {
+      "description": "Ignore fullsend self-references (own reusable workflows and actions)",
+      "matchManagers": ["github-actions"],
+      "matchPackageNames": ["/^fullsend-ai\\//"],
+      "enabled": false
+    }
+  ],
+  "git-submodules": {
+    "enabled": true
+  }
+}

Relevance

⭐⭐ Medium

Some compliance reviews ask for linked issue/ADR (PR#25 partially accepted), but similar “link
authorization” note in PR#29 stayed undetermined.

PR-#25
PR-#29

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The checklist requires a linked issue/ADR for non-trivial changes. The diff adds an entire new
renovate.json (23 lines) enabling Renovate behaviors like platformAutomerge and
git-submodules, which constitutes a non-trivial change without explicit authorization linkage.

renovate.json[1-23]
Skill: pr-review

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The PR introduces a non-trivial change (new Renovate automation configuration) but lacks an explicit authorization signal via a linked issue/ADR in the PR description.

## Issue Context
Per compliance, non-trivial changes (20+ changed lines or structural changes) must include a linked issue authorizing the work.

## Fix Focus Areas
- renovate.json[1-23]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[fullsend-ai-review]

Review

Findings

High

• [API contract violation / intent-description-mismatch] renovate.json:21 — The git-submodules manager tracks the URL in .gitmodules, which points to https://github.com/ralphbean/agent-eval-harness.git (a personal fork). The PR description and test plan claim Renovate will track opendatahub-io/agent-eval-harness, but Renovate will actually track the ralphbean fork. If the fork diverges from or falls behind the upstream repo, Renovate will not track the intended upstream.
  Remediation: Either update .gitmodules to point to https://github.com/opendatahub-io/agent-eval-harness.git, or correct the PR description and test plan to reflect that Renovate tracks the ralphbean fork. If the fork carries intentional patches, document this.

Low

• [missing-authorization] renovate.json — No linked issue is visible in the PR metadata. While AGENTS.md §3 states every changed line must trace to issue scope, this is agent-facing guidance and does not strictly apply to human-authored PRs. Consider linking an issue for traceability.

• [scope-label-mismatch] — PR is labeled fullsend-fix but the PR title uses chore: type, which is the correct conventional commit type for tooling/dependency maintenance. There is a semantic mismatch between the label and the commit type.

• [architectural-coordination-risk] renovate.json — AGENTS.md §6 documents lockstep versioning with fullsend and centrally managed workflows. Automated dependency updates could potentially conflict with centralized release orchestration. The config already mitigates this by excluding fullsend-ai/* GitHub Actions, and submodule updates are not automerged (only patch and pin types are).

[fullsend-ai-review]

See the review comment for full details.

[fullsend-ai-coder]

🤖 Fix · ⚠️ Cancelled · Started 11:16 AM UTC · Ended 11:21 AM UTC
Commit: bd28f41 · View workflow run →

[ralphbean]

/fs-fix the .gitmodules fix is in #146

[fullsend-ai-coder]

🤖 Finished Fix · ✅ Success · Started 11:22 AM UTC · Completed 11:28 AM UTC
Commit: bd28f41 · View workflow run →

[fullsend-ai-coder]

🔧 Fix agent — iteration 1 (human-triggered)

No code changes needed. The single High finding (git-submodules tracking the ralphbean fork instead of opendatahub-io) is already addressed by companion PR #146, which updates .gitmodules. The three Low findings are informational — the review itself notes they are either non-applicable to human-authored PRs or already mitigated by the existing config. Human instruction confirmed the .gitmodules fix is in PR #146.

Disagreed (4):

1. API contract violation / intent-description-mismatch: git-submodules manager tracks ralphbean fork URL in .gitmodules instead of opendatahub-io/agent-eval-harness: The reviewer's observation is factually correct — .gitmodules currently points to the ralphbean fork. However, the human instruction confirms this is already being fixed in companion PR chore: point agent-eval-harness submodule to opendatahub-io #146 (chore/fix-submodule-url), which updates .gitmodules to point to opendatahub-io/agent-eval-harness and advances the submodule pointer. Modifying .gitmodules in this PR would create a merge conflict with PR chore: point agent-eval-harness submodule to opendatahub-io #146. The correct resolution is to merge PR chore: point agent-eval-harness submodule to opendatahub-io #146 first (or concurrently), which resolves the intent-description mismatch.
2. missing-authorization: no linked issue in PR metadata: The review itself notes this is agent-facing guidance (AGENTS.md §3) that does not strictly apply to human-authored PRs. This is a human-authored PR for routine tooling configuration. No code change is applicable.
3. scope-label-mismatch: PR labeled fullsend-fix but title uses chore: type: The fullsend-fix label is a workflow trigger label, not a semantic scope label. The chore: commit type correctly describes tooling/dependency maintenance. Label management is outside the fix agent's scope — labels are managed by the post-script and workflow automation.
4. architectural-coordination-risk: automated dependency updates could conflict with centralized release orchestration: The review itself acknowledges that the config already mitigates this risk by excluding fullsend-ai/* GitHub Actions and not automerging submodule updates (only patch and pin types are automerged). This is an informational note, not an actionable code change.

Tests: passed

_{Updated by fullsend fix agent}

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 11:54 AM UTC · Completed 12:01 PM UTC
Commit: 2596da2 · View workflow run →

[fullsend-ai-retro]

PR #147 added a Renovate configuration to the fullsend-ai/agents repo — a 23-line, single-file config change authored by a human (ralphbean). The review agent raised 1 High and 3 Low findings. The High finding (git-submodules tracking a personal fork URL in .gitmodules) was factually correct but context-blind: the author had already created companion PR #146 to fix that issue. The CHANGES_REQUESTED verdict auto-triggered a fix agent run that was cancelled when the author manually invoked /fs-fix with context about PR #146. A second fix agent run correctly determined no code changes were needed. The 3 Low findings were all dismissed as noise. A human reviewer approved, and the PR merged ~50 minutes after opening. Two proposals target cross-PR awareness and low-signal finding suppression.

Proposals filed

• Review agent should query for related open PRs before finalizing findings that reference files outside the PR diff (in fullsend-ai/agents)
• Review agent should suppress low-value compliance findings on trivial config-only PRs from repo members (in fullsend-ai/agents)