Experiment: triage skill comparison with oh-my-claudecode and oh-my-openagent adapters - #170
Closed
ralphbean wants to merge 11 commits into
Closed
Experiment: triage skill comparison with oh-my-claudecode and oh-my-openagent adapters#170ralphbean wants to merge 11 commits into
ralphbean wants to merge 11 commits into
Conversation
…d oh-my-openagent adapters Design and implement an experiment comparing 5 triage questioning strategies adapted from third-party coding agent skills for async GitHub issue triage. Strategies under evaluation: - superpowers-brainstorming (obra/superpowers): judgment-based, one question at a time - omc-deep-interview (oh-my-claudecode): mathematical ambiguity gating with challenge modes - omo-prometheus (oh-my-openagent): phased engineer-style interview - structured-triage: checklist baseline - socratic-refinement: open-ended Socratic probing Includes 3 fictional bug scenarios at different quality levels, file-based simulation scripts, a GitHub-native adapter, and an independent judge agent with a weighted scoring rubric. Builds on PR #169 by incorporating oh-my-claudecode deep-interview and oh-my-openagent Prometheus strategies that the prior attempt missed. Related: #126 Assisted-by: OpenCode claude-opus-4-6@default
ralphbean
force-pushed
the
agent-triage-skill-comparison-v2
branch
from
April 5, 2026 23:39
960cba7 to
5aee0de
Compare
ralphbean
force-pushed
the
agent-triage-skill-comparison-v2
branch
from
April 6, 2026 00:55
5aee0de to
1707151
Compare
Adds analyze-scenario.sh to compare judge assessments across strategies for each scenario, with a system prompt for structured analysis. The single-trial script now also emits a human-readable conversation.md alongside the JSON, and the summarizer renders cross-strategy analysis when available. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Each scenario x strategy cell now runs N independent trials, stored under trial-1/ through trial-N/. The summarizer aggregates scores as mean +/- stddev, making it possible to observe strategy consistency across repeated interactions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Raw results from the first run of all 3 scenarios x 5 strategies. These predate the multi-trial refactoring so each cell has n=1. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a Results section to the README with strategy rankings, key findings from 10 trials per cell, and six concrete recommendations for the triage agent design in issue #126. Updates the output structure docs and file index for the multi-trial layout. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Successor to triage-skill-comparison, addressing methodological limitations identified in the first study: - 10 scenarios (up from 3) spanning 7 bug archetypes - Reporter realism profiles (cooperative/typical/difficult) - Separate judge model via --judge-model flag (default: sonnet) - Resolve/ask prompt reframed to reduce premature closure bias - Adapters normalized to ~40-50 lines each - Efficiency split into thoroughness + economy - Reliability tracked separately from quality scores - Acceptable diagnostic paths for fairer accuracy scoring Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Accepts an existing results directory and skips completed work in each phase: seed app, trials (by trial-metadata.json), judging (by judge-assessment.json), and cross-strategy analysis. Summary is always regenerated to reflect the full dataset. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Summarizes the 10×5×10 experiment run: strategy rankings, what separates top from bottom performers (hypothesis-driven questioning, causal dating, dual-structure questions), scenario-specific patterns, and six concrete implications for the Story 3 triage agent design. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
ralphbean
marked this pull request as ready for review
April 16, 2026 02:39
Site previewPreview: https://ac426f3f-site.fullsend-ai.workers.dev Commit: |
Member
Author
|
So, this is really hard to review with all of the Are those helpful to understand what's going on? or, are they noise that should be git rm'd and squashed from the history? |
Member
|
Hello! We are moving experiments to another repo because we are bloating this one, please reopen this PR at https://github.com/fullsend-ai/experiments. Thanks |
This was referenced Jul 7, 2026
Closed
This was referenced Jul 12, 2026
This was referenced Jul 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Designs and implements an experiment comparing 5 triage questioning strategies adapted from third-party coding agent skills for asynchronous GitHub issue triage via comment dialogue. Related to #126 (Story 3: Triage Agent).
Builds on #169 by incorporating the oh-my-claudecode deep-interview and oh-my-openagent Prometheus strategies that the prior attempt missed.
What this adds
5 triage strategy adapters
superpowers-brainstormingomc-deep-interviewomo-prometheusstructured-triagesocratic-refinement3 fictional bug scenarios, orchestration scripts, GitHub adapter, judge agent
See
experiments/triage-skill-comparison/README.mdfor full details.Key insight
Non-interactive mode (
-p) naturally forces the behavior needed for issue comments. No hooks or skill modifications required.