Skip to content

feat: feature ideation framework — adversarial proposer/challenger/synthesizer - #6

Closed
don-petry with Copilot wants to merge 2 commits into
mainfrom
copilot/implement-feature-ideation-framework
Closed

don-petry with Copilot wants to merge 2 commits into
mainfrom
copilot/implement-feature-ideation-framework

Conversation

Copilot AI commented Apr 11, 2026 •

Copy link
Copy Markdown
Contributor

Adds a feature ideation agent to this repo, implementing the adversarial-review pattern from github.com/topics/adversarial-review.

Two modes are provided:

Interactive: Claude Code Skill (SKILL.md)

A single SKILL.md at the repo root implements the adversarial-review pattern as-is for feature ideation. Claude proposes a spec, Codex adversarially challenges it, Claude refines, and the loop repeats until approved (max 5 rounds).

/ideate-feature <description | issue URL | file path>
Claude (propose spec)
  └─ Codex (adversarial challenge)
       └─ Claude (refine spec)
            └─ Codex (re-review)
                 └─ ... → VERDICT: APPROVED

The skill follows the same structure as dementev-dev/adversarial-review: YAML frontmatter, XML-structured adversarial prompts (<role>, <operating_stance>, <attack_surface>, <finding_bar>, <scope_exclusions>, <calibration>, <output_format>), codex exec + resume-first iterative loop, and best-effort cleanup.

Attack surface adapted for feature specs: value, feasibility, scope, completeness, risk, alternatives, open questions.

Automated: GitHub Actions

When the feature-idea label is added to any GitHub issue, a three-stage pipeline runs and posts a refined spec as an issue comment.

feature-idea label added
  └─ Proposer (Sonnet): expand idea → /tmp/ideation/proposal.json
       └─ Challenger (Sonnet): adversarial critique → /tmp/ideation/challenges.json
            └─ Synthesizer (Opus): reconcile → post refined spec to issue
Stage Model Role
Proposer Sonnet 4.6 Expands raw idea: problem, solution, implementation sketch, acceptance criteria, effort estimate, open questions
Challenger Sonnet 4.6 Adversarially challenges across 7 dimensions: value, feasibility, scope, risk, completeness, alternatives, open questions
Synthesizer Opus 4.6 Reconciles valid challenges, posts comment with refined spec + adds feature-spec-ready label

Idempotency — each posted comment is marked with <!-- feature-ideation-agent v1 issue=<num> cycle=<N> -->. Re-labeling triggers a new cycle.

Files changed

File Change
SKILL.md New Claude Code skill — adversarial feature ideation following dementev-dev/adversarial-review pattern as-is
.github/workflows/feature-ideation.yml New workflow — triggers on feature-idea label or workflow_dispatch
scripts/ideate-feature.sh New orchestration script (proposer → challenger → synthesizer)
prompts/ideation/proposer.md Proposer prompt
prompts/ideation/challenger.md Challenger prompt (7-dimension adversarial critique)
prompts/ideation/synthesize-idea.md Synthesizer prompt (posts to GitHub issue)
AGENT.md Documents both modes; adds https://github.com/topics/adversarial-review as resource reference

Usage

Interactive (Claude Code):

# Install Codex CLI
npm install -g @openai/codex

# Symlink the skill
ln -s "$(pwd)" ~/.agents/skills/feature-ideation

# Then in Claude Code:
/ideate-feature https://github.com/owner/repo/issues/42

Automated (GitHub Actions):

# Automatically: label an issue
gh issue edit 42 --repo owner/repo --add-label feature-idea

# Manually (dry run):
gh workflow run feature-ideation.yml --repo don-petry/self \
  -f issue_url=https://github.com/owner/repo/issues/42 \
  -f dry_run=true

The same GH_PAT, CLAUDE_CODE_OAUTH_TOKEN, and LIVE_MODE variable used by the PR review agent also control the automated mode — no new secrets needed.

…nger/synthesizer loop

Agent-Logs-Url: https://github.com/don-petry/self/sessions/ba5e1014-4588-49b4-b7a7-870de1331a9d

Co-authored-by: don-petry <36422719+don-petry@users.noreply.github.com>
@don-petry

Copy link
Copy Markdown
Collaborator

@copilot - implement the feature ideation pattern "as is". Add the adversarial link as a resource for ideation related to the pull request review automation.

…sarial-review pattern)

Agent-Logs-Url: https://github.com/don-petry/self/sessions/f19064eb-2aee-41e7-b2e2-1041423ae1a3

Co-authored-by: don-petry <36422719+don-petry@users.noreply.github.com>

Copilot AI commented Apr 11, 2026

Copy link
Copy Markdown
Contributor Author

@copilot - implement the feature ideation pattern "as is". Add the adversarial link as a resource for ideation related to the pull request review automation.

Done in e6be2ca.

Added SKILL.md at the repo root — a Claude Code skill following the dementev-dev/adversarial-review pattern as-is:

  • Same frontmatter (name, description, user_invocable: true)
  • Same XML-structured adversarial prompts (<role>, <operating_stance>, <attack_surface>, <finding_bar>, <scope_exclusions>, <calibration>, <output_format>)
  • Same iterative loop: Claude proposes spec → Codex challenges → Claude refines → repeat up to 5 rounds → VERDICT: APPROVED
  • Same codex exec + resume-first flow with fallback and cleanup

Also added https://github.com/topics/adversarial-review as a resource reference in AGENT.md (Feature Ideation Agent section), alongside a note that the existing GitHub Actions automation implements the same adversarial pattern for issue-triggered batch runs.

@don-petry

Copy link
Copy Markdown
Collaborator

@claude please review this PR

@claude

claude Bot commented Apr 16, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @don-petry's task in 2m 13s —— View job


PR Review: feat: feature ideation framework — adversarial proposer/challenger/synthesizer

Tasks:

  • Gather context and understand the request
  • Read CLAUDE.md and repo setup (no CLAUDE.md present)
  • Review changed files
  • Post review feedback

Overall this is a clean, well-designed addition. The three-stage pipeline is clearly structured, the prompts have solid input/output contracts, and idempotency is handled correctly. A few issues need attention before merging.


Bugs / will break at runtime

1. Missing issues: write permission in workflow — feature-ideation.yml:18-20

The synthesizer posts a comment and adds labels to GitHub issues, but the workflow only declares permissions: contents: read. The GH_TOKEN is a PAT so fine-grained permissions apply — but regardless, issues: write should also be declared explicitly for transparency and to ensure the job doesn't fail if the PAT is ever rotated to one without that scope.

Fix this →

# Current (line 18-20):
permissions:
  contents: read

# Should be:
permissions:
  contents: read
  issues: write

2. Inconsistency: synthesizer references blocking verdict that the challenger never emits — synthesize-idea.md:106

"When the challenger's overall_verdict is blocking…"

The challenger schema (challenger.md:57) defines overall_verdict as strong | acceptable | needs_rework | reject — there is no blocking verdict. blocking is a severity level for individual dimensions, not a verdict. This paragraph in the synthesizer prompt describes behavior for a state that can never be reached, which may confuse the model.

Fix this →

3. Synthesizer stdout not captured — JSON status silently lost — scripts/ideate-feature.sh:130-136

Stages 1 and 2 redirect stdout to log files before checking for JSON output. The synthesizer (stage 3) runs with stdout going directly to the workflow log — there's no capture, no check, and no structured exit status. If the gh api call inside the synthesizer fails, the script exits 0 (since set -e applies to the shell, not to claude --print's internal tool calls).

Consider adding > /tmp/ideation/synthesizer.log 2>&1 and checking exit code, consistent with stages 1-2.


Medium concerns

4. actions/checkout@v5 — this tag may not exist — feature-ideation.yml:42

The existing pr-review.yml uses actions/checkout@v4. There is no publicly released v5 as of this review. This will cause the workflow to fail silently if the tag resolution fails, or use an unexpected version if it was tagged after this was written. Use @v4 to match the rest of the repo.

Fix this →

5. Proposer env vars not available at the prompt level — scripts/ideate-feature.sh:88-94

The proposer is invoked as claude --print ... < prompts/ideation/proposer.md. The env vars ($ISSUE_URL, $ISSUE_TITLE, $ISSUE_BODY, etc.) are exported in the shell but the Claude process reads the prompt file as a static text — it will see the literal string $ISSUE_URL in the prompt, not the expanded value.

The same problem exists for the challenger and synthesizer. The other scripts in this repo (e.g. review-one-pr.sh) construct prompts dynamically using envsubst or heredocs with shell expansion. This approach of passing a static file will result in the models trying to evaluate shell variables themselves via Bash tool calls, which works but is less reliable than substituting values upfront.

This is the same pattern used currently, so it "works" — but worth noting that if variables are referenced literally in the prompts, the Claude models need to run Bash to expand them rather than having them ready in context.

6. Synthesizer stage has no log on failure — scripts/ideate-feature.sh:130-136

Stages 1 and 2 save stderr/stdout to log files and cat them on failure. Stage 3 does neither — if the synthesizer crashes or the gh api post fails, debugging requires re-running the entire pipeline.


Minor / low severity

7. gpt-5.4 model in SKILL.md may not exist — SKILL.md:148

The PR description says this follows the adversarial-review pattern "as-is", so this may be intentional. But gpt-5.4 is not a publicly available model name — users attempting to run this skill will get a Codex CLI error. A note in SKILL.md clarifying users should substitute their actual Codex model would help.

8. Idempotency cycle counter uses comment count, not max cycle — scripts/ideate-feature.sh:49-55

ISSUE_IDEATION_CYCLE is computed as (count of ideation comments) + 1, while EXISTING_CYCLE is the maximum cycle=N seen. If a comment was ever deleted, these diverge and the new comment will have a non-sequential cycle number. Not a correctness issue (idempotency still holds), but it could make the cycle history confusing. Using EXISTING_CYCLE + 1 instead of count + 1 would be more robust.

9. DRY_RUN ternary expression in workflow — feature-ideation.yml:38

DRY_RUN: ${{ inputs.dry_run || (vars.LIVE_MODE == 'true' && 'false' || 'true') }}

When triggered via workflow_dispatch with dry_run: "false", GitHub Actions evaluates "false" (a string) as truthy, so inputs.dry_run returns "false" — which is the correct string value. This works, but it's subtle. A comment explaining the logic would help future maintainers.


Summary

# Severity File Issue
1 Bug feature-ideation.yml:18 Missing issues: write permission
2 Bug synthesize-idea.md:106 blocking verdict referenced but never emitted by challenger
3 Bug ideate-feature.sh:130 Synthesizer output not captured; failures silent
4 Medium feature-ideation.yml:42 actions/checkout@v5 probably doesn't exist
5 Medium ideate-feature.sh:88 Static prompt files — env vars not pre-expanded
6 Low ideate-feature.sh:130 No synthesizer log for debugging failures
7 Low SKILL.md:148 gpt-5.4 model name may not resolve
8 Low ideate-feature.sh:49 Cycle counter uses comment count not max cycle
9 Low feature-ideation.yml:38 DRY_RUN ternary is subtle, could use a comment

Items 1–4 should be fixed before merging. Items 5–9 are low risk.

@don-petry

Copy link
Copy Markdown
Collaborator

This was a misunderstood concept application. Removing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants