Skip to content

feat: success pattern tracking with model routing integration (t102.3) #779

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
marcusquinn merged 2 commits into from
Feb 9, 2026
+658 −123
Merged

feat: success pattern tracking with model routing integration (t102.3) #779

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b93b8de
feat: success pattern tracking with model routing integration (t102.3)
marcusquinn Feb 9, 2026
40f932d
Merge branch 'main' into feature/t102.3
marcusquinn Feb 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 27 additions & 8 deletions .agents/memory/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -261,24 +261,43 @@ FTS5 keyword search remains the default and works without any setup.

## Pattern Tracking

Track what works and what fails across task types and models:
Track what works and what fails across task types, models, and approaches.
Patterns are captured automatically by the supervisor after task completion,
and can also be recorded manually.

```bash
# Record a pattern
# Record a pattern with full metadata
pattern-tracker-helper.sh record --outcome success --task-type bugfix \
--model sonnet --description "Structured debugging found root cause"
--model sonnet --task-id t102.3 --duration 120 \
--description "Structured debugging found root cause"

# Get suggestions for a task
# Get suggestions for a task (includes model routing hints)
pattern-tracker-helper.sh suggest "refactor the auth middleware"

# View pattern statistics
# Get model recommendation based on historical success rates
pattern-tracker-helper.sh recommend --task-type bugfix

# View pattern statistics (includes supervisor-generated patterns)
pattern-tracker-helper.sh stats

# Or use the /patterns command
/patterns refactor
# Generate a comprehensive report
pattern-tracker-helper.sh report

# Export patterns for analysis
pattern-tracker-helper.sh export --format json > patterns.json

# Or use slash commands
/patterns refactor # Suggest patterns for a task
/patterns report # Full report
/patterns recommend bugfix # Model recommendation
/route "fix auth bug" # Model routing (now includes pattern data)
```

See `scripts/pattern-tracker-helper.sh` for full documentation.
**Automatic capture**: The supervisor stores `SUCCESS_PATTERN` and `FAILURE_PATTERN`
entries after each task evaluation, tagged with model tier, duration, and retry count.
This data feeds into the `recommend` command for data-driven model routing.

See `scripts/pattern-tracker-helper.sh help` for full documentation.

## Memory Graduation (Sharing Learnings)

Expand Down
28 changes: 21 additions & 7 deletions .agents/scripts/commands/patterns.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Show success/failure patterns from memory to guide task approach
description: Show success/failure patterns from memory to guide task approach and model routing
agent: Build+
mode: subagent
model: haiku
Expand All @@ -11,31 +11,45 @@ Arguments: $ARGUMENTS

## Instructions

1. If arguments are provided, use them as a task description to find relevant patterns:
1. If arguments contain "report", show the comprehensive report:

```bash
~/.aidevops/agents/scripts/pattern-tracker-helper.sh report
```

2. If arguments contain "recommend", show model recommendation:

```bash
~/.aidevops/agents/scripts/pattern-tracker-helper.sh recommend "$ARGUMENTS"
```

3. If other arguments are provided, use them as a task description to find relevant patterns:

```bash
~/.aidevops/agents/scripts/pattern-tracker-helper.sh suggest "$ARGUMENTS"
```

2. If no arguments, show overall pattern statistics and recent patterns:
4. If no arguments, show overall pattern statistics and recent patterns:

```bash
~/.aidevops/agents/scripts/pattern-tracker-helper.sh stats
~/.aidevops/agents/scripts/pattern-tracker-helper.sh analyze --limit 5
```

3. Present the results with actionable guidance:
5. Present the results with actionable guidance:
- Highlight what approaches have worked for similar tasks
- Warn about approaches that have failed
- Suggest the optimal model tier based on pattern data

4. If no patterns exist yet, explain how to start recording:
6. If no patterns exist yet, explain how to start recording:

```text
No patterns recorded yet. Patterns are recorded automatically during
development loops, or manually with:
No patterns recorded yet. Patterns are recorded automatically by the
supervisor after task completion, or manually with:

pattern-tracker-helper.sh record --outcome success \
--task-type bugfix --model sonnet \
--description "Structured debugging approach found root cause quickly"

Available commands: suggest, recommend, analyze, stats, report, export
Copy link
Contributor

@coderabbitai coderabbitai bot Feb 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Command list may be incomplete.

The available commands list mentions suggest, recommend, analyze, stats, report, export, but the example above (lines 50-52) demonstrates the record command. Should record be included in the available commands list for completeness?

📝 Suggested fix to include record command 
-Available commands: suggest, recommend, analyze, stats, report, export
+Available commands: record, suggest, recommend, analyze, stats, report, export
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
Available commands: suggest, recommend, analyze, stats, report, export
Available commands: record, suggest, recommend, analyze, stats, report, export
🤖 Prompt for AI Agents 
In @.agents/scripts/commands/patterns.md at line 54, The "Available commands:
suggest, recommend, analyze, stats, report, export" list omits the `record`
command shown in the example; update that command list to include `record` so it
matches the example usage (i.e., add `record` to the string "Available commands:
...") and verify any surrounding documentation references to the same command
list (the example block demonstrating `record`) are consistent.

```
23 changes: 18 additions & 5 deletions .agents/scripts/commands/route.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Suggest optimal model tier for a task description
description: Suggest optimal model tier for a task description using rules + pattern history
agent: Build+
mode: subagent
model: haiku
Expand All @@ -11,27 +11,40 @@ Task: $ARGUMENTS

## Instructions

1. Read `tools/context/model-routing.md` for the routing rules and tier definitions.
1. First, check pattern history for data-driven insights:

2. Analyze the task description against the routing rules:
```bash
~/.aidevops/agents/scripts/pattern-tracker-helper.sh recommend "$ARGUMENTS"
```
Comment on lines +14 to +18
Copy link
Contributor

@coderabbitai coderabbitai bot Feb 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

recommend is invoked with the raw task description, but the command only filters by --task-type, not free text.

cmd_recommend in pattern-tracker-helper.sh uses --task-type for SQL filtering. Positional arguments are stored in task_desc but are only displayed — they don't influence the query. So invoking recommend "$ARGUMENTS" (where $ARGUMENTS is a free-form task description like "refactor the auth middleware") will show unfiltered global model stats, not task-specific recommendations.

If the intent is task-type-specific routing, the agent would need to extract a task type from $ARGUMENTS first. If global stats are acceptable here, consider adding a comment clarifying that.

🤖 Prompt for AI Agents 
In @.agents/scripts/commands/route.md around lines 14 - 18, The documentation
invokes recommend with free-form "$ARGUMENTS" but cmd_recommend in
pattern-tracker-helper.sh only filters by --task-type (positional task_desc is
only displayed), so update the call or docs: either modify the script to extract
a task type from ARGUMENTS before calling recommend (e.g., derive and pass
--task-type from ARGUMENTS) or change the example invocation to pass an explicit
task type variable instead of raw free text; reference cmd_recommend,
pattern-tracker-helper.sh, recommend, ARGUMENTS, and task_desc when making the
change and add a short clarifying comment explaining whether global stats are
acceptable or a task-type must be provided.


2. Read `tools/context/model-routing.md` for the routing rules and tier definitions.

3. Analyze the task description against the routing rules:
- **Complexity**: Simple transform vs reasoning vs novel design
- **Context size**: Small focused task vs large codebase sweep
- **Output type**: Classification vs code vs architecture

3. Output a recommendation in this format:
4. Combine pattern history with routing rules:
- If pattern data exists and shows a clear winner (>75% success rate with 3+ samples), weight it heavily
- If pattern data is sparse or inconclusive, rely on routing rules
- If pattern data contradicts routing rules, note the conflict and explain

5. Output a recommendation in this format:

```text
Recommended: {tier} ({model_name})
Reason: {one-line justification}
Cost: ~{relative}x vs sonnet baseline
Pattern data: {success_rate}% success rate from {N} samples (or "no data")
```

4. If the task is ambiguous, suggest the tier and note what would push it up or down:
6. If the task is ambiguous, suggest the tier and note what would push it up or down:

```text
Recommended: sonnet (claude-sonnet-4)
Reason: Code modification with moderate reasoning
Cost: ~1x baseline
Pattern data: 85% success rate from 12 samples

Could be haiku if: the change is a simple rename/reformat
Could be opus if: the change requires architectural decisions
Expand Down
Loading