Experiment: target repo skills in triage

target-repo-skills/.gitignore

@@ -0,0 +1 @@
+.experiment-state

target-repo-skills/HOW_TO.md

@@ -0,0 +1,81 @@
+## Purpose
+
+Run the target-repo-skills experiment to determine whether a target
+repository's `.claude/skills/` are discovered by the fullsend triage agent
+inside an OpenShell sandbox.
+
+## Requirements
+
+| Requirement | Link |
+|-------------|------|
+| fullsend CLI | Built from fullsend repo: `go build ./cmd/fullsend` |
+| gh CLI | https://cli.github.com/ |
+| git | https://git-scm.com/ |
+| GCP credentials | `gcloud auth application-default login` |
+
+### Environment variables
+
+| Variable | Description |
+|----------|-------------|
+| `FULLSEND_DIR` | Path to the fullsend scaffold directory (e.g., `/path/to/fullsend/internal/scaffold/fullsend-repo`) |
+| `ANTHROPIC_VERTEX_PROJECT_ID` | GCP project ID with Vertex AI access |
+| `CLOUD_ML_REGION` | GCP region for Vertex AI (e.g., `us-east5`) |
+| `GOOGLE_APPLICATION_CREDENTIALS` | Path to GCP service account key or ADC file |
+| `TARGET_ORG` | (Optional) GitHub org/user for the synthetic repo. Defaults to `maruiz93` |
+| `TARGET_REPO` | (Optional) Repo name. Defaults to `experiment-target-repo-skills` |
+
+## Steps
+
+1. Navigate to the experiment directory:
+   ```bash
+   cd target-repo-skills
+   ```
+
+2. Set required environment variables:
+   ```bash
+   export FULLSEND_DIR=/path/to/fullsend/internal/scaffold/fullsend-repo
+   export ANTHROPIC_VERTEX_PROJECT_ID=your-project-id
+   export CLOUD_ML_REGION=us-east5
+   export GOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/application_default_credentials.json
+   ```
+
+3. Create the synthetic target repo and file the test issue:
+   ```bash
+   ./setup-target-repo.sh
+   ```
+
+4. Run the experiment:
+   ```bash
+   ./run.sh
+   ```
+
+## Expected Output
+
+- `results/control/agent-result.json` — triage output without the skill
+- `results/control/transcript.jsonl` — Claude session transcript without the skill
+- `results/treatment/agent-result.json` — triage output with the skill
+- `results/treatment/transcript.jsonl` — Claude session transcript with the skill
+- The treatment transcript should contain `triage-guidance` in the system prompt
+  if skill discovery works
+- The treatment `agent-result.json` should use labels from the skill's taxonomy
+  (`area:api`, `priority:critical`, `type:bug`)
+
+## Viewing Transcripts
+
+Use [claude-replay](https://www.npmjs.com/package/claude-replay) to view the
+session transcripts as interactive HTML replays:
+
+```bash
+# Control run (no skill)
+claude-replay results/control/transcript.jsonl --title "Control Run" --serve
+
+# Treatment run (with triage-guidance skill)
+claude-replay results/treatment/transcript.jsonl --title "Treatment Run" --serve
+```
+
+Or generate static HTML files:
+
+```bash
+claude-replay results/control/transcript.jsonl --title "Control Run" -o control-replay.html
+claude-replay results/treatment/transcript.jsonl --title "Treatment Run" -o treatment-replay.html
+```

target-repo-skills/README.md

@@ -0,0 +1,115 @@
+# Experiment: Target Repository Skills in Triage
+
+## Hypothesis
+
+When `fullsend run triage` executes Claude inside the target repository
+directory, Claude discovers and uses `.claude/skills/` from that repository —
+even though `CLAUDE_CONFIG_DIR` is set to `/tmp/claude-config` (the sandbox
+config directory for fullsend's own agent and skill definitions).
+
+## Background
+
+Fullsend provisions agents inside OpenShell sandboxes. The bootstrap sequence
+sets `CLAUDE_CONFIG_DIR=/tmp/claude-config` where it copies the agent definition
+and harness-declared skills. Claude is then invoked with `cd <target-repo-dir>`.
+
+Claude Code discovers project-level skills from `.claude/skills/` in the current
+working directory's git root. This experiment tests whether that CWD-based
+discovery still works when `CLAUDE_CONFIG_DIR` is overridden.
+
+## Method
+
+- **Independent variable:** Presence of `.claude/skills/triage-guidance/SKILL.md`
+  in the target repository.
+- **Control:** Triage run against a synthetic Go REST API repo without
+  `.claude/skills/`.
+- **Treatment:** Same run, same issue, but with a triage guidance skill committed
+  to the repo.
+
+All other inputs are held constant: same codebase, same GitHub issue, same
+fullsend scaffold (agent, harness, policy, scripts), same model.
+
+### Versions
+
+| Component | Version |
+|-----------|---------|
+| fullsend | `de54df2` (built from source) |
+| OpenShell | 0.0.36 |
+| Sandbox image | `ghcr.io/fullsend-ai/fullsend-sandbox:latest` |
+| Harness | `internal/scaffold/fullsend-repo/harness/triage.yaml` |
+
+### Evidence
+
+1. **JSONL transcript** (primary) — shows whether the skill appears in Claude's
+   system prompt.
+2. **Triage JSON output** (secondary) — shows whether the agent's labeling
+   behavior changed.
+
+## Results
+
+### Control
+
+- **Severity:** high
+- **Labels:** none (no skill-specific labels referenced)
+- **Overall clarity score:** 0.89
+- Full output: [`results/control/agent-result.json`](results/control/agent-result.json)
+
+### Treatment
+
+- **Severity:** critical
+- **Labels:** `area:api`, `area:data`, `priority:critical`, `type:bug`
+- **Overall clarity score:** 0.90
+- Reasoning explicitly cites: _"Per triage guidance"_
+- Full output: [`results/treatment/agent-result.json`](results/treatment/agent-result.json)
+
+### Skill Discovery
+
+**Yes.** The treatment transcript shows 3 mentions of `triage-guidance`:
+
+1. The agent called `Skill("triage-guidance")` during triage.
+2. The skill content was loaded from `/tmp/workspace/repo/.claude/skills/triage-guidance`.
+3. The skill instructions appeared in the agent's context.
+
+The control transcript contains 0 mentions of `triage-guidance`.
+
+```bash
+$ grep -c 'triage-guidance' results/treatment/transcript.jsonl
+3
+$ grep -c 'triage-guidance' results/control/transcript.jsonl
+0
+```
+
+## Conclusion
+
+**Hypothesis confirmed.** Target repository `.claude/skills/` are discovered and
+used by the fullsend triage agent inside OpenShell sandboxes, even though
+`CLAUDE_CONFIG_DIR` is overridden to `/tmp/claude-config`.
+
+Claude Code's CWD-based project skill discovery (`cd <target-repo> && claude`)
+operates independently of `CLAUDE_CONFIG_DIR`. The agent discovers skills from
+the git root of its working directory regardless of where user-level config
+points.
+
+**Behavioral impact:** The triage guidance skill changed the agent's output in
+two measurable ways:
+
+1. **Severity escalation:** "high" → "critical" (matching the skill's rule that
+   panic/500 errors are `priority:critical`)
+2. **Structured labeling:** The treatment output includes explicit label
+   references (`area:api`, `area:data`, `priority:critical`, `type:bug`) that
+   match the skill's taxonomy exactly
+
+The triage agent comments (control and treatment) can be seen on the target
+repo issue: https://github.com/maruiz93/experiment-target-repo-skills/issues/1
+
+**Note:** The treatment agent referenced the skill's taxonomy labels (`area:api`,
+`priority:critical`, etc.) in its comment text, but these were not assigned to
+the GitHub issue. The `post-triage.sh` script only applies workflow labels
+(`ready-to-code`, `needs-info`, `duplicate`) — it does not parse or apply
+taxonomy labels from the agent's output. Extracting and assigning skill-defined
+labels is a potential improvement to the post-triage pipeline.
+
+## See Also
+
+- [Design spec](docs/2026-04-28-target-repo-skills-design.md)
+- [HOW_TO](HOW_TO.md)