feat: add experiment numbering and metadata conventions

[waynesun09]

Summary

• Rename all 22 experiments to sequential NNNN-short-description format (0001–0022)
• Add YAML frontmatter (title, status, topics) to every experiment
• Update README with experiment index table
• Create 0000-experiment-template/ with standard sections (Hypothesis, Approach, Results, Conclusion)
• Add hack/lint-experiment-numbers and hack/lint-experiment-frontmatter lint scripts
• Wire experiment lint hooks into .pre-commit-config.yaml
• Add AGENTS.md and CLAUDE.md with repo-wide conventions

Modeled after the ADR system in fullsend-ai/fullsend — same lint script patterns, frontmatter validation, and pre-commit enforcement.

Migration table

Current	New
003-agent-outage-fire-drill.md	0001-agent-outage-fire-drill.md
adr46-claude-scanner/	0002-claude-scanner/
adr46-scanner/	0003-scanner/
006-meta-loop-self-improving-engine/	0004-meta-loop-self-improving-engine/
agent-scoped-tools-triage/	0005-agent-scoped-tools-triage/
code-agent-evaluation/	0006-code-agent-evaluation/
github-actions-agent-runtime-mvp/	0007-github-actions-agent-runtime-mvp/
guardrails-eval/	0008-guardrails-eval/
hermes-security-patterns/	0009-hermes-security-patterns/
host-side-api-server/	0010-host-side-api-server/
integration-service-design-drift/	0011-integration-service-design-drift/
model-armor-vs-agent-triage/	0012-model-armor-vs-agent-triage/
openshell-policy-bypass/	0013-openshell-policy-bypass/
openshell-sandbox-evaluation.md	0014-openshell-sandbox-evaluation.md
prompt-injection-defense/	0015-prompt-injection-defense/
promptfoo-eval/	0016-promptfoo-eval/
reasoning-monitor/	0017-reasoning-monitor/
runner-hello-world/	0018-runner-hello-world/
skills/	0019-skills/
target-repo-skills/	0020-target-repo-skills/
tool-scoping/	0021-tool-scoping/
67-claude-github-app-auth/	0022-claude-github-app-auth/

Statuses: 18 Concluded, 4 Active

Valid status	Meaning
Active	In progress or being iterated on
Concluded	Finished, results documented
Abandoned	Stopped before completion
Merged	Results adopted into the main project

Test plan

• ./hack/lint-experiment-numbers — 22 entries, no duplicates, sequential 0001–0022
• python3 ./hack/lint-experiment-frontmatter — 22 files, all valid
• pre-commit run --all-files — experiment hooks pass

Closes #34

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 7:37 PM UTC · Ended 7:38 PM UTC
Commit: 0123b0b · View workflow run →

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 7:41 PM UTC · Ended 7:43 PM UTC
Commit: 0123b0b · View workflow run →

[qodo-code-review]

PR Summary by Qodo

Add numbered experiment conventions with frontmatter + pre-commit linting

✨ Enhancement ⚙️ Configuration changes 📝 Documentation 🕐 20-40 Minutes

AI Description

• Standardize experiment names to four-digit numbering across directories and standalone markdown.
• Add YAML frontmatter conventions (title/status/topics) and a reusable experiment template.
• Enforce conventions via new lint scripts wired into pre-commit hooks.

Diagram

graph TD
  Dev(["Contributor"]) --> PC["pre-commit"] --> LNum[["lint numbers"]] --> Exp[("Experiment entries")]
  PC --> LFM[["lint frontmatter"]] --> Exp
  Docs["AGENTS.md" ] --> PC
  subgraph Legend
    direction LR
    _p(["Person"]) ~~~ _h["Hook"] ~~~ _s[["Script"]] ~~~ _e[("Entries")]
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Single unified Python linter

• ➕ One implementation/runtime to maintain (shared file discovery, reporting, exit codes).
• ➕ Easier to add future checks (e.g., required sections, topic taxonomy).
• ➖ Less consistent with the ADR pattern being mirrored (bash+python split).
• ➖ Requires reworking existing pre-commit wiring and developer expectations.

2. Make linters filename-driven (pass_filenames: true)

• ➕ Faster on large repos; checks only staged/changed experiment entries.
• ➕ Avoids scanning the entire repo root on every commit.
• ➖ Harder to reliably detect global invariants like duplicate experiment numbers without a full scan.
• ➖ May miss issues introduced by renames not included in the current staged set.

3. Relax Python version pinning in the script header

• ➕ Avoids friction on machines without Python 3.12 if the code remains 3.10+ compatible.
• ➕ Better alignment with pre-commit’s managed venv behavior.
• ➖ If future checks rely on 3.12-only features, you lose the explicit guardrail.

Recommendation: Current approach (repo-scanning linters + pre-commit enforcement) is the right tradeoff for ensuring global invariants like unique numbering and consistent metadata. Consider either relaxing the embedded requires-python >=3.12 marker (if not truly required) or documenting the minimum supported Python for contributors to avoid environment surprises.

Files changed (38) +668 / -60

Refactor (1) +91 / -40

server.pyPure formatting cleanup in repo provisioner server +91/-40

Pure formatting cleanup in repo provisioner server

• Applies formatting-only changes: import ordering, multi-line literals/collections, and line wrapping. No behavior changes intended; improves readability and consistency.

0012-host-side-api-server/servers/repo-provisioner/server.py

Tests (1) +0 / -1

test_hooks.pyRemove stray whitespace line in unit test +0/-1

Remove stray whitespace line in unit test

• Removes an empty line between imports in a test to keep formatting tidy after the experiment renumbering sweep.

0019-reasoning-monitor/tests/test_hooks.py

Documentation (33) +310 / -19

README.mdIntroduce a standard experiment template with required frontmatter and sections +26/-0

Introduce a standard experiment template with required frontmatter and sections

• Adds a reusable template experiment README with YAML frontmatter placeholders and standardized sections (Hypothesis, Approach, Results, Conclusion). Intended as the starting point for new experiments and excluded from lint checks.

0000-experiment-template/README.md

AGENTS.mdDocument repository-wide experiment conventions and contributor rules +82/-0

Document repository-wide experiment conventions and contributor rules

• Adds a single source of truth for experiment numbering, required frontmatter, valid statuses, and lint hook behavior. Also documents commit message conventions, DCO expectations, and general contribution guardrails.

AGENTS.md

CLAUDE.mdPoint agent instructions to AGENTS.md as the canonical ruleset +3/-0

Point agent instructions to AGENTS.md as the canonical ruleset

• Adds a short file directing agent workflows to follow AGENTS.md as the authoritative guidance.

CLAUDE.md

0003-agent-outage-fire-drill.mdAdd YAML frontmatter to experiment entry +8/-0

Add YAML frontmatter to experiment entry

• Adds standardized YAML frontmatter (title/status/topics) to the standalone experiment markdown file to comply with the new conventions.

0003-agent-outage-fire-drill.md

README.mdAdd YAML frontmatter to experiment README +8/-0

Add YAML frontmatter to experiment README

• Adds standardized YAML frontmatter (title/status/topics) to the experiment README to comply with the new conventions.

0004-claude-scanner/README.md

README.mdAdd YAML frontmatter to experiment README +8/-0

Add YAML frontmatter to experiment README

• Adds standardized YAML frontmatter (title/status/topics) to the experiment README to comply with the new conventions.

0005-scanner/README.md

README.mdAdd YAML frontmatter to experiment README +9/-0

Add YAML frontmatter to experiment README

• Adds standardized YAML frontmatter (title/status/topics) to the experiment README to comply with the new conventions.

0006-meta-loop-self-improving-engine/README.md

README.mdAdd YAML frontmatter to experiment README +9/-0

Add YAML frontmatter to experiment README

• Adds standardized YAML frontmatter (title/status/topics) to the experiment README to comply with the new conventions.

0007-agent-scoped-tools-triage/README.md

EXPERIMENT.mdAdd YAML frontmatter to experiment entry file +8/-0

Add YAML frontmatter to experiment entry file

• Adds standardized YAML frontmatter (title/status/topics) to the experiment entry markdown (EXPERIMENT.md) to comply with the new conventions.

0008-code-agent-evaluation/EXPERIMENT.md

README.mdAdd YAML frontmatter to experiment README +8/-0

Add YAML frontmatter to experiment README

• Adds standardized YAML frontmatter (title/status/topics) to the experiment README to comply with the new conventions.

0009-github-actions-agent-runtime-mvp/README.md

README.mdAdd YAML frontmatter to experiment README +8/-0

Add YAML frontmatter to experiment README

• Adds standardized YAML frontmatter (title/status/topics) to the experiment README to comply with the new conventions.

0010-guardrails-eval/README.md

README.mdAdd YAML frontmatter to experiment README +8/-0

Add YAML frontmatter to experiment README

• Adds standardized YAML frontmatter (title/status/topics) to the experiment README to comply with the new conventions.

0011-hermes-security-patterns/README.md

README.mdAdd YAML frontmatter to experiment README +8/-0

Add YAML frontmatter to experiment README

• Adds standardized YAML frontmatter (title/status/topics) to the experiment README to comply with the new conventions.

0012-host-side-api-server/README.md

results.mdNormalize markdown formatting in results output +2/-2

Normalize markdown formatting in results output

• Removes trailing double-space line breaks in endpoint lines to standardize markdown formatting under the renamed experiment structure.

0012-host-side-api-server/results/agent-baked-instructions-restricted-3395961-1780273527/iteration-1/output/results.md

results.mdNormalize markdown formatting in results output +6/-6

Normalize markdown formatting in results output

• Removes trailing double-space line breaks in multiple fields to standardize markdown formatting under the renamed experiment structure.

0012-host-side-api-server/results/agent-baked-instructions-restricted-4080040-1779243702/iteration-1/output/results.md

results.mdNormalize markdown formatting in results output +1/-1

Normalize markdown formatting in results output

• Removes trailing double-space line breaks on the Date line to standardize markdown formatting.

0012-host-side-api-server/results/agent-openapi-discovery-full-4038775-1779243123/iteration-1/output/results.md

results.mdNormalize markdown formatting in results output +1/-1

Normalize markdown formatting in results output

• Removes trailing double-space line breaks on the Date line to standardize markdown formatting.

0012-host-side-api-server/results/agent-openapi-discovery-restricted-3445554-1780274427/iteration-1/output/results.md

results.mdNormalize markdown formatting in results output +1/-1

Normalize markdown formatting in results output

• Removes trailing double-space line breaks on the Date line to standardize markdown formatting.

0012-host-side-api-server/results/agent-openapi-discovery-restricted-4097617-1779243949/iteration-1/output/results.md

results.mdNormalize markdown formatting in results output +2/-2

Normalize markdown formatting in results output

• Removes trailing double-space line breaks in endpoint lines to standardize markdown formatting.

0012-host-side-api-server/results/agent-tooluse-discovery-full-4061051-1779243439/iteration-1/output/results.md

results.mdNormalize markdown formatting in results output +3/-3

Normalize markdown formatting in results output

• Removes trailing double-space line breaks on Date/Base URL lines to standardize markdown formatting.

0012-host-side-api-server/results/agent-tooluse-discovery-restricted-3481065-1780274959/iteration-1/output/results.md

results.mdNormalize markdown formatting in results output +3/-3

Normalize markdown formatting in results output

• Removes trailing double-space line breaks on Target lines to standardize markdown formatting.

0012-host-side-api-server/results/agent-tooluse-discovery-restricted-4121470-1779244303/iteration-1/output/results.md

README.mdAdd YAML frontmatter to experiment README +8/-0

Add YAML frontmatter to experiment README

• Adds standardized YAML frontmatter (title/status/topics) to the experiment README to comply with the new conventions.

0013-integration-service-design-drift/README.md

README.mdAdd YAML frontmatter to experiment README +8/-0

Add YAML frontmatter to experiment README

• Adds standardized YAML frontmatter (title/status/topics) to the experiment README to comply with the new conventions.

0014-model-armor-vs-agent-triage/README.md

README.mdAdd YAML frontmatter to experiment README +8/-0

Add YAML frontmatter to experiment README

• Adds standardized YAML frontmatter (title/status/topics) to the experiment README to comply with the new conventions.

0015-openshell-policy-bypass/README.md

0016-openshell-sandbox-evaluation.mdAdd YAML frontmatter to experiment entry +8/-0

Add YAML frontmatter to experiment entry

• Adds standardized YAML frontmatter (title/status/topics) to the standalone experiment markdown file to comply with the new conventions.

0016-openshell-sandbox-evaluation.md

README.mdAdd YAML frontmatter to experiment README +8/-0

Add YAML frontmatter to experiment README

• Adds standardized YAML frontmatter (title/status/topics) to the experiment README to comply with the new conventions.

0017-prompt-injection-defense/README.md

README.mdAdd YAML frontmatter to experiment README +8/-0

Add YAML frontmatter to experiment README

• Adds standardized YAML frontmatter (title/status/topics) to the experiment README to comply with the new conventions.

0018-promptfoo-eval/README.md

README.mdAdd YAML frontmatter to experiment README +8/-0

Add YAML frontmatter to experiment README

• Adds standardized YAML frontmatter (title/status/topics) to the experiment README to comply with the new conventions.

0019-reasoning-monitor/README.md

README.mdAdd YAML frontmatter to experiment README +8/-0

Add YAML frontmatter to experiment README

• Adds standardized YAML frontmatter (title/status/topics) to the experiment README to comply with the new conventions.

0020-runner-hello-world/README.md

README.mdAdd new Skills experiment entry with required frontmatter +10/-0

Add new Skills experiment entry with required frontmatter

• Introduces an experiment entry README for Skills with required YAML frontmatter and a brief description, aligning it with the numbering and metadata conventions.

0021-skills/README.md

README.mdAdd YAML frontmatter to experiment README +8/-0

Add YAML frontmatter to experiment README

• Adds standardized YAML frontmatter (title/status/topics) to the experiment README to comply with the new conventions.

0022-target-repo-skills/README.md

README.mdAdd YAML frontmatter to experiment README +8/-0

Add YAML frontmatter to experiment README

• Adds standardized YAML frontmatter (title/status/topics) to the experiment README to comply with the new conventions.

0023-tool-scoping/README.md

README.mdAdd YAML frontmatter to experiment README +8/-0

Add YAML frontmatter to experiment README

• Adds standardized YAML frontmatter (title/status/topics) to the experiment README to comply with the new conventions.

0067-claude-github-app-auth/README.md

Other (3) +267 / -0

.pre-commit-config.yamlAdd pre-commit hooks for experiment naming and frontmatter validation +15/-0

Add pre-commit hooks for experiment naming and frontmatter validation

• Registers two new hooks: one script hook for numbering/naming rules and one Python hook for YAML frontmatter validation. Both are scoped to root entries matching the '^[0-9]{4}-' convention and run repo-wide (no filenames passed).

.pre-commit-config.yaml

lint-experiment-numbersAdd numbering/naming linter for experiment entries +114/-0

Add numbering/naming linter for experiment entries

• Introduces a bash linter that enforces 'NNNN-...' naming, detects duplicate numbers, and rejects leading zeros in titles/H1 headings. Scans root-level experiment directories and standalone '.md' entries, excluding the '0000-' template.

hack/lint-experiment-numbers

lint-experiment-frontmatterAdd YAML frontmatter validator for experiment entry markdown +138/-0

Add YAML frontmatter validator for experiment entry markdown

• Introduces a Python linter that locates each experiment’s entry markdown (README.md/EXPERIMENT.md/standalone .md) and validates required fields ('title', 'status'), allowed status values, and 'topics' list typing. Produces per-file OK/FAIL output with aggregated errors.

hack/lint-experiment-frontmatter

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📎 Requirement gaps (0) 📜 Skill insights (0)

Context used

✅ Tickets: 🎫 Add experiment numbering and metadata conventions

✅ Compliance rules (platform): 9 rules

✅ Skills: writing-how-to

1. lint-experiment-numbers wrong shebang ✓ Resolved 📜 Skill insight ☼ Reliability

Description

The new hack/lint-experiment-numbers script starts with #!/bin/bash instead of the required
#!/usr/bin/env bash. This can break portability and violates the mandated script header
convention.

Code

hack/lint-experiment-numbers[1]

+#!/bin/bash

Evidence

PR Compliance ID 1062126 requires shell scripts to start with #!/usr/bin/env bash. The added
script begins with #!/bin/bash, violating the rule.

hack/lint-experiment-numbers[1-1]
Skill: writing-how-to

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`hack/lint-experiment-numbers` uses `#!/bin/bash` but the compliance requirement mandates `#!/usr/bin/env bash` as the first line of any shell script.

## Issue Context
This script was added in this PR and is used as a repo lint hook.

## Fix Focus Areas
- hack/lint-experiment-numbers[1-1]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. lint-experiment-numbers missing strict mode line ✓ Resolved 📜 Skill insight ☼ Reliability

Description

set -euo pipefail is not placed immediately after the shebang; it appears later after
blank/comment lines. This violates the requirement that strict mode must be the second line of the
script.

Code

hack/lint-experiment-numbers[R1-13]

+#!/bin/bash
+
+# lint-experiment-numbers - Check experiment naming and numbering conventions
+#
+# Validates:
+# - Experiment directories/files match ^[0-9]{4}- pattern
+# - No duplicate experiment numbers
+# - No leading zeros in titles or H1 headings
+#
+# Adapted from fullsend-ai/fullsend hack/lint-adr-numbers
+
+set -euo pipefail
+

Evidence

PR Compliance ID 1062125 requires set -euo pipefail on the second line after the shebang. In
hack/lint-experiment-numbers, line 2 is blank and strict mode is only enabled at line 12.

hack/lint-experiment-numbers[1-13]
Skill: writing-how-to

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The script must have `set -euo pipefail` as the second line (immediately after the shebang), but it currently appears after a blank line and comments.

## Issue Context
Compliance requires strict mode to be enabled at the top of the script for consistent failure behavior.

## Fix Focus Areas
- hack/lint-experiment-numbers[1-13]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. GNU sort flag used ✓ Resolved 🐞 Bug ☼ Reliability

Description

hack/lint-experiment-numbers pipes NUL-delimited paths into sort -z, which is not supported by
all sort implementations; on those systems the hook will error and block commits. This is
avoidable because experiment entry names don’t require NUL-safe sorting here.

Code

hack/lint-experiment-numbers[R22-28]

+while IFS= read -r -d '' entry; do
+    experiments+=("$entry")
+done < <(find . -maxdepth 1 \( -type d -o -type f -name "*.md" \) \
+    -regex '.*/[0-9].*' \
+    ! -name "0000-experiment-template" \
+    ! -path "./0000-experiment-template" \
+    -print0 | sort -z)

Evidence

The discovery pipeline explicitly uses -print0 | sort -z, which depends on GNU sort supporting
-z.

hack/lint-experiment-numbers[22-28]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`hack/lint-experiment-numbers` currently uses `sort -z` in the discovery pipeline. `sort -z` is GNU-specific and can break the pre-commit hook on platforms with BSD userland tools.

### Issue Context
The hook is wired into `.pre-commit-config.yaml` and will run for contributors; avoiding GNU-only flags improves portability.

### Fix Focus Areas
- hack/lint-experiment-numbers[22-28]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. Nonportable grep regex ✓ Resolved 🐞 Bug ≡ Correctness

Description

hack/lint-experiment-numbers uses grep with a \+ quantifier but without -E, which is not
portable and may not match as intended on non-GNU grep. This can cause the hook to miss leading-zero
headings and reduce enforcement consistency across environments.

Code

hack/lint-experiment-numbers[R100-103]

+    heading_line=$(grep '^# 0[0-9]\+\. ' "$md_file" 2>/dev/null || true)
+    if [[ -n "$heading_line" ]]; then
+        error "$name: H1 heading has leading zeros (found: $heading_line). Use \"# $expected_number. ...\" instead."
+        has_errors=1

Evidence

The script uses a grep pattern containing \+ while calling plain grep (BRE mode), which is not
guaranteed to treat \+ as a quantifier.

hack/lint-experiment-numbers[100-103]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The heading validation uses a `grep` pattern containing `\+` but does not enable ERE mode (`-E`). This is not portable and can fail to match properly depending on the platform `grep` implementation.

### Issue Context
This hook exists to enforce experiment title/heading formatting conventions; inconsistent regex behavior undermines that purpose.

### Fix Focus Areas
- hack/lint-experiment-numbers[100-103]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

5. Broken experiment links 🐞 Bug ⚙ Maintainability

Description

Some experiment docs still link to the old pre-rename experiment directory names (e.g.
../model-armor-vs-agent-triage/, ../prompt-injection-defense/), which will be broken after this
PR’s renames. This degrades navigation and makes cross-referencing experiments unreliable.

Code

0010-guardrails-eval/README.md[11]

Evaluates prompt injection scanning tools as pre-LLM guardrails for fullsend's autonomous SDLC pipeline. Extends the [Model Armor experiment](../model-armor-vs-agent-triage/) (PR #117) with additional attack payloads and alternative scanning backends.

Evidence

The referenced files contain relative links to unprefixed paths (e.g.
../model-armor-vs-agent-triage/), while the actual experiment directories in this branch are
prefixed (e.g. 0014-model-armor-vs-agent-triage, 0017-prompt-injection-defense).

0010-guardrails-eval/README.md[11-31]
0011-hermes-security-patterns/README.md[239-242]
0019-reasoning-monitor/README.md[9-14]
0014-model-armor-vs-agent-triage/README.md[17-20]
0014-model-armor-vs-agent-triage/README.md[1-10]
0017-prompt-injection-defense/README.md[1-10]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
After experiments were renamed to `NNNN-...` directories, several markdown links still reference the old unprefixed directory names.

### Issue Context
These are relative links between experiment directories; renaming the directories requires updating the link targets.

### Fix Focus Areas
- 0010-guardrails-eval/README.md[11-31]
- 0011-hermes-security-patterns/README.md[239-242]
- 0019-reasoning-monitor/README.md[9-14]
- 0014-model-armor-vs-agent-triage/README.md[17-20]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

6. Title regex too narrow ✓ Resolved 🐞 Bug ≡ Correctness

Description

hack/lint-experiment-numbers only flags leading-zero title: values when the YAML scalar is
double-quoted, so valid YAML variants like unquoted or single-quoted titles can bypass the check.
This creates a gap in the “no leading zeros in titles” rule enforcement.

Code

hack/lint-experiment-numbers[R94-98]

+    title_line=$(grep '^title:' "$md_file" 2>/dev/null || true)
+    if [[ $title_line =~ ^title:\ *\"0[0-9]+\. ]]; then
+        error "$name: title has leading zeros (found: $title_line). Use \"$expected_number. ...\" instead."
+        has_errors=1
+    fi

Evidence

The regex explicitly requires a double quote after title:, so it won’t catch equivalent YAML
scalars written without double quotes.

hack/lint-experiment-numbers[94-98]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The leading-zero title check is implemented as a raw text regex that only matches `title: "0...` (double quotes). YAML allows unquoted and single-quoted scalars, which would not be detected.

### Issue Context
Because this is a linter, missing a syntax variant means conventions can drift without being caught.

### Fix Focus Areas
- hack/lint-experiment-numbers[94-98]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 7:46 PM UTC · Ended 7:47 PM UTC
Commit: 0123b0b · View workflow run →

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 7:51 PM UTC · Ended 7:59 PM UTC
Commit: 0123b0b · View workflow run →

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 8:02 PM UTC · Ended 8:13 PM UTC
Commit: 0123b0b · View workflow run →

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 8:17 PM UTC · Completed 8:24 PM UTC
Commit: ba62b5c · View workflow run →

[fullsend-ai-review]

Review — approve

Scope: Renames 22 experiments to sequential NNNN-short-description format, adds YAML frontmatter (title, status, topics) to every experiment, creates 0000-experiment-template/, adds three lint scripts with pre-commit integration, updates README with an index table, adds AGENTS.md/CLAUDE.md, and adds a doc site build workflow. Closes #34.

Analysis

Change composition: 410 files changed — ~400 are pure renames (directory restructuring), 8 new files, 2 modified. The substantive new code is in the lint scripts, pre-commit configuration, CI workflow, and documentation.

Lint scripts (hack/lint-experiment-numbers, hack/lint-experiment-frontmatter, hack/lint-experiment-index):

• Correctly validate naming conventions (^[0-9]{4}-), duplicate detection, frontmatter fields (title, status), valid status values, and README index synchronization.
• The frontmatter linter properly uses yaml.safe_load (not yaml.load), avoiding arbitrary code execution.
• The index linter cross-checks disk entries against README table rows bidirectionally.

Pre-commit integration (.pre-commit-config.yaml):

• Hooks correctly use pass_filenames: false since the scripts scan the repo root.
• File filters (files: ^[0-9]{4}-) scope hook execution to experiment entries only.
• The frontmatter hook uses language: python with additional_dependencies: ["pyyaml>=6.0"], which is valid for local hooks.

CI workflow (.github/workflows/site-build.yml):

• Actions pinned to full SHAs with version comments — good supply-chain practice.
• Permissions scoped to contents: read — minimal privilege.
• Uses npm ci --ignore-scripts to mitigate install-script attacks.
• Concurrency group prevents redundant builds.

Documentation (AGENTS.md, CLAUDE.md, README.md):

• AGENTS.md provides clear, actionable conventions for experiment creation, naming, frontmatter, statuses, commit format, and DCO requirements.
• README index table is complete (22 experiments) and consistent with the migration table in the PR body.

No findings at or above the configured severity threshold (medium).

---

Protected paths detected — this PR modifies files under one or more
protected paths. The review agent cannot approve PRs that touch these paths.
A human reviewer must approve this PR.

Protected files in this PR:

• .github/workflows/site-build.yml
• .pre-commit-config.yaml

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 12:05 PM UTC · Completed 12:13 PM UTC
Commit: ba62b5c · View workflow run →

[fullsend-ai-retro]

Retro: PR #35 — Add experiment numbering and metadata conventions

Workflow went well. This was a human-authored PR by waynesun09 that renamed ~300 files to establish numbered experiment conventions, added YAML frontmatter, lint scripts, and pre-commit hooks. The PR was opened Jul 1 19:34 UTC, reviewed by both Qodo and fullsend review agents, approved by human reviewer rh-hemartin Jul 2 06:07 UTC, and merged Jul 2 12:03 UTC.

Timeline

1. 19:34 — PR opened with ~300+ changed files (mostly renames)
2. 19:34–19:51 — 5 review runs cancelled due to cancel-in-progress concurrency as new commits were pushed
3. 19:45 — Qodo bot posted 6 findings (wrong shebang, missing strict mode, GNU-only sort -z, nonportable grep, narrow title regex, broken cross-links)
4. 19:58 — Human pushed fix commit addressing Qodo findings
5. 20:13 — Human pushed second fix commit (index lint hook, more broken links)
6. 20:17–20:24 — Review agent run fix: remove pull_request_review from dispatch-review condition #6 completed successfully, APPROVED with protected-path flag (requires-manual-review)
7. 06:07 Jul 2 — Human approved
8. 12:03 Jul 2 — Merged

Assessment

• Review quality: Good. The review agent correctly recognized ~400 renames, validated lint scripts, checked CI workflow for SHA-pinned actions and minimal permissions, and flagged protected paths (.github/workflows/, .pre-commit-config.yaml) for human review. No false positives.
• Rework rate: N/A. Human-authored PR; no agent code generation involved.
• Token cost: Moderate waste from 5 cancelled runs. Each cancelled run bootstrapped an agent before being terminated. This is a known issue with extensive existing coverage.
• Time to resolution: Reasonable. ~16.5 hours from open to merge, with ~10 hours waiting for human review overnight.

Existing issue coverage

All improvement opportunities identified are already tracked in fullsend-ai/fullsend:

• Review debouncing / cancellation noise: #1014, #1452, #1422, #1331, #2399
• Stale bot comment cleanup: #1209
• File move detection: #2019
• Protected-path severity on human PRs: #1551
• Pre-review skip to save bootstrap tokens: #2809

No novel proposals warranted — existing issues comprehensively cover the identified improvement areas.