ci(functional-tests): add config.yaml for local agent resolution

[ralphbean]

Summary

• Adds config.yaml with allowed_remote_resources to fix script URL resolution in fullsend v0.29.0
• Registers all agents with local harness paths so functional tests validate PR branch content instead of the @v0 release tag

Context

Without config.yaml, fullsend v0.29.0 has an empty allowed_remote_resources allowlist (a bug fixed in the next release). This caused all functional test cases to fail immediately with:

URL "https://raw.githubusercontent.com/fullsend-ai/agents/.../scripts/pre-triage.sh" is not in allowed_remote_resources

Registering agents as local paths also fixes a semantic problem: without explicit registration, fullsend run resolves agents from fullsend-ai/agents@v0 (the latest release tag), meaning functional tests were never actually testing the PR's changes.

Test plan

• functional-tests CI check passes with actual agent execution (not short-circuited)

🤖 Generated with Claude Code

[qodo-code-review]

PR Summary by Qodo

Add fullsend config.yaml to allow remote scripts and use local harness agents

⚙️ Configuration changes 🐞 Bug fix 🕐 10-20 Minutes

AI Description

• Add fullsend config.yaml to populate allowed_remote_resources for v0.29.0 functional tests
• Allow raw GitHub URLs used by agent scripts to avoid immediate test failures
• Register agents via local harness/*.yaml so CI tests PR branch content, not agents@v0

Diagram

graph TD
  CI["CI: functional-tests"] --> FS["fullsend run"] --> CFG["config.yaml"] --> ARR["allowed_remote_resources"] --> RAW["raw.githubusercontent.com"]
  CFG --> AG["Local harness agents"]
  AG --> H1["harness/*.yaml"]
  subgraph Legend
    direction LR
    _ci[CI job] ~~~ _cfg[Config file] ~~~ _ext{{External host}}
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Pin fullsend to a version with the allowlist bug fixed

• ➕ Avoids needing repository-specific allowlist configuration for this regression
• ➕ Keeps security policy driven by upstream defaults
• ➖ May not be feasible if CI is intentionally testing/standardizing on v0.29.0
• ➖ Still doesn’t address the separate issue of agents resolving from agents@v0 unless config/flags are added anyway

2. Pass allowlist and local agent sources via CLI/CI flags instead of a repo file

• ➕ No persistent config file in the repository root
• ➕ Can scope the behavior strictly to the functional-tests job
• ➖ More brittle/verbose CI wiring; easier to drift across jobs
• ➖ Harder to reproduce locally without duplicating CI flag logic

Recommendation: Keeping an explicit config.yaml in-repo is a good approach here: it fixes the v0.29.0 allowlist regression and, more importantly, makes agent resolution deterministic (local harness sources) so functional tests actually validate the PR branch. Consider the version-bump alternative if the project prefers minimizing allowlisted remote access, but local agent registration still needs an explicit config somewhere.

Files changed (1) +11 / -0

Other (1) +11 / -0

config.yamlAdd fullsend config with remote allowlist and local agent sources +11/-0

Add fullsend config with remote allowlist and local agent sources

• Introduces a root 'config.yaml' declaring 'allowed_remote_resources' for raw GitHub content used by agent scripts. Registers each agent using local 'harness/*.yaml' sources so functional tests run against the PR’s agent definitions rather than resolving 'fullsend-ai/agents@v0'.

config.yaml

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 8:05 PM UTC · Ended 8:08 PM UTC
Commit: e8381e3 · View workflow run →

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 8:09 PM UTC · Ended 8:13 PM UTC
Commit: e8381e3 · View workflow run →

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📜 Skill insights (1)

Context used

✅ Compliance rules (platform): 55 rules

✅ Skills: 4 invoked

  code-review

  code-implementation

  pr-review

  docs-review

1. Tests skip config changes ✓ Resolved 🐞 Bug ≡ Correctness

Description

Adding root-level config.yaml changes functional-test behavior, but functional-tests.yml does not
include config.yaml in either push.paths or the PR file-change relevance regex, so PRs that only
touch this file will skip functional tests. This can let a broken allowlist/agent registration merge
without any functional execution validating it.

Code

config.yaml[R1-11]

+version: "1"
+allowed_remote_resources:
+  - "https://raw.githubusercontent.com/fullsend-ai/fullsend/"
+  - "https://raw.githubusercontent.com/fullsend-ai/agents/"
+agents:
+  - source: harness/triage.yaml
+  - source: harness/code.yaml
+  - source: harness/fix.yaml
+  - source: harness/review.yaml
+  - source: harness/retro.yaml
+  - source: harness/prioritize.yaml

Relevance

⭐⭐⭐ High

Functional-tests workflow has explicit path/regex gating “SYNC-WITH” (PR #31); missing config.yaml
likely addressed.

PR-#31
PR-#76

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The workflow only runs functional tests when certain paths are modified; neither the push path
filter nor the PR file-change regex includes config.yaml, so this PR’s change can be treated as
“not relevant” and skip tests.

.github/workflows/functional-tests.yml[10-21]
.github/workflows/functional-tests.yml[81-116]
config.yaml[1-11]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Functional tests are skipped when only `config.yaml` changes, because the workflow’s `push.paths` and PR change-detection regex don’t include it.

## Issue Context
This PR introduces a new top-level `config.yaml` intended to affect `fullsend run` behavior during functional tests.

## Fix Focus Areas
- .github/workflows/functional-tests.yml[10-21]
- .github/workflows/functional-tests.yml[81-116]

## Proposed fix
1. Add `config.yaml` (and optionally `.fullsend/config.yaml` if that’s also supported) to the `on.push.paths` list.
2. Add `^config\.yaml$` (and optionally `^\.fullsend/config\.yaml$`) to the `grep -qE` relevance regex so PRs that change config trigger functional tests.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Overbroad remote allowlist 🐞 Bug ⛨ Security

Description

config.yaml allowlists entire raw.githubusercontent.com prefixes for fullsend-ai/fullsend and
fullsend-ai/agents, allowing fullsend to fetch any file at any ref under those repos. Because
pre/post scripts run on the GitHub Actions runner (not just in the sandbox) and functional tests run
in pull_request_target with secrets, this unnecessarily expands runner-side remote code-fetch
surface.

Code

config.yaml[R2-4]

+allowed_remote_resources:
+  - "https://raw.githubusercontent.com/fullsend-ai/fullsend/"
+  - "https://raw.githubusercontent.com/fullsend-ai/agents/"

Relevance

⭐⭐ Medium

Least-privilege CI security feedback sometimes accepted/partial (PR #25), but no
allowlist-tightening precedent found.

PR-#25

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The new config explicitly allows broad remote URL prefixes. Functional tests run on
pull_request_target with secrets and invoke fullsend run with --fullsend-dir $FULLSEND_DIR,
meaning the checked-out PR’s config governs what remote resources fullsend may fetch; additionally,
the repo documents that pre/post scripts execute on the runner (where secrets exist).

config.yaml[2-4]
.github/workflows/functional-tests.yml[3-6]
.github/workflows/functional-tests.yml[245-255]
eval/scripts/run-fullsend.sh[69-76]
README.md[33-41]
harness/triage.yaml[24-29]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`allowed_remote_resources` is currently a broad prefix allowlist that can permit fetching arbitrary content from entire repos/refs.

## Issue Context
Functional tests run in `pull_request_target` and pass secrets into the run; the agent pipeline executes pre/post scripts on the runner.

## Fix Focus Areas
- config.yaml[1-4]
- .github/workflows/functional-tests.yml[3-6]
- .github/workflows/functional-tests.yml[245-255]

## Proposed fix
1. Narrow the allowlist to the minimal required subpaths (e.g., only the specific directories/files that fullsend actually needs to fetch).
2. Prefer pinning to immutable refs (commit SHA or a specific release tag path) instead of allowing any ref.
3. If feasible, generate/overwrite the allowlist in the workflow from a trusted base-branch source (or validate the PR’s config) so PRs can’t broaden it in secret-bearing `pull_request_target` runs.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. config.yaml quoting style inconsistent 📜 Skill insight ⚙ Maintainability

Description

The new config.yaml uses quoted scalars (e.g., version: "1" and quoted URL strings), while
existing YAML in the same directory (repo root) primarily uses unquoted scalars. This violates the
requirement that new config files match existing style conventions, increasing maintenance friction
and formatting drift.

Code

config.yaml[R1-4]

+version: "1"
+allowed_remote_resources:
+  - "https://raw.githubusercontent.com/fullsend-ai/fullsend/"
+  - "https://raw.githubusercontent.com/fullsend-ai/agents/"

Relevance

⭐ Low

Style-only consistency changes often rejected (workflow .yml→.yaml naming/style rejected in PR #25).

PR-#25

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance ID 1538351 requires new YAML config files to match the style (including quoting
conventions) of existing config files in the same directory. In this PR, config.yaml introduces
quoted scalars, while the existing root YAML file .pre-commit-config.yaml uses unquoted scalars
for similar URL values.

config.yaml[1-4]
.pre-commit-config.yaml[3-6]
Skill: code-implementation

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`config.yaml` uses a different quoting convention than existing YAML files in the repo root (notably `.pre-commit-config.yaml`). The compliance rule requires new config files to match the style of existing files in the same directory.

## Issue Context
In `.pre-commit-config.yaml`, URL values and most scalars are unquoted unless quoting is required. In `config.yaml`, several scalars are quoted without an obvious need.

## Fix Focus Areas
- config.yaml[1-4]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[qodo-code-review]

3. Overbroad remote allowlist 🐞 Bug ⛨ Security

config.yaml allowlists entire raw.githubusercontent.com prefixes for fullsend-ai/fullsend and
fullsend-ai/agents, allowing fullsend to fetch any file at any ref under those repos. Because
pre/post scripts run on the GitHub Actions runner (not just in the sandbox) and functional tests run
in pull_request_target with secrets, this unnecessarily expands runner-side remote code-fetch
surface.

Agent Prompt

## Issue description
`allowed_remote_resources` is currently a broad prefix allowlist that can permit fetching arbitrary content from entire repos/refs.

## Issue Context
Functional tests run in `pull_request_target` and pass secrets into the run; the agent pipeline executes pre/post scripts on the runner.

## Fix Focus Areas
- config.yaml[1-4]
- .github/workflows/functional-tests.yml[3-6]
- .github/workflows/functional-tests.yml[245-255]

## Proposed fix
1. Narrow the allowlist to the minimal required subpaths (e.g., only the specific directories/files that fullsend actually needs to fetch).
2. Prefer pinning to immutable refs (commit SHA or a specific release tag path) instead of allowing any ref.
3. If feasible, generate/overwrite the allowlist in the workflow from a trusted base-branch source (or validate the PR’s config) so PRs can’t broaden it in secret-bearing `pull_request_target` runs.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 8:13 PM UTC · Ended 8:20 PM UTC
Commit: e8381e3 · View workflow run →

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 8:20 PM UTC · Ended 8:26 PM UTC
Commit: e8381e3 · View workflow run →

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 8:27 PM UTC · Ended 8:36 PM UTC
Commit: e8381e3 · View workflow run →

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 8:37 PM UTC · Ended 8:41 PM UTC
Commit: e8381e3 · View workflow run →

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 8:42 PM UTC · Ended 8:52 PM UTC
Commit: e8381e3 · View workflow run →

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 8:53 PM UTC · Completed 9:07 PM UTC
Commit: 4467568 · View workflow run →

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 8:56 PM UTC · Completed 9:02 PM UTC
Commit: 4467568 · View workflow run →

[fullsend-ai-retro]

PR #78 was a human-authored CI configuration fix (7 commits in 48 minutes) that added config.yaml for local agent resolution and fixed the functional-tests workflow. The fullsend review agent was dispatched 8 times but completed 0 reviews: 7 runs were cancelled by rapid pushes, and the 8th started 1 minute before the PR was self-merged. Qodo's review (which completed) identified a valid path-filter gap that the author fixed. This is a well-documented pattern — existing issue fullsend-ai/fullsend#1014 tracks debouncing review dispatch. Two proposals filed as evidence for existing issues.

Proposals filed

• Evidence for #1014: 8 review dispatches with 0 completions on 48-minute rapid-iteration PR #78 (in fullsend-ai/fullsend)
• Evidence for #74: review run 8 on PR #78 started 45 seconds before merge, findings discarded (in fullsend-ai/agents)

[fullsend-ai-review]

Review skipped — this PR is already merged.

The /fs-review command only reviews open pull requests.

_{Posted by fullsend post-review check}