diff --git a/.agents/memory/README.md b/.agents/memory/README.md index b6d27eb4..8de492b4 100644 --- a/.agents/memory/README.md +++ b/.agents/memory/README.md @@ -261,24 +261,43 @@ FTS5 keyword search remains the default and works without any setup. ## Pattern Tracking -Track what works and what fails across task types and models: +Track what works and what fails across task types, models, and approaches. +Patterns are captured automatically by the supervisor after task completion, +and can also be recorded manually. ```bash -# Record a pattern +# Record a pattern with full metadata pattern-tracker-helper.sh record --outcome success --task-type bugfix \ - --model sonnet --description "Structured debugging found root cause" + --model sonnet --task-id t102.3 --duration 120 \ + --description "Structured debugging found root cause" -# Get suggestions for a task +# Get suggestions for a task (includes model routing hints) pattern-tracker-helper.sh suggest "refactor the auth middleware" -# View pattern statistics +# Get model recommendation based on historical success rates +pattern-tracker-helper.sh recommend --task-type bugfix + +# View pattern statistics (includes supervisor-generated patterns) pattern-tracker-helper.sh stats -# Or use the /patterns command -/patterns refactor +# Generate a comprehensive report +pattern-tracker-helper.sh report + +# Export patterns for analysis +pattern-tracker-helper.sh export --format json > patterns.json + +# Or use slash commands +/patterns refactor # Suggest patterns for a task +/patterns report # Full report +/patterns recommend bugfix # Model recommendation +/route "fix auth bug" # Model routing (now includes pattern data) ``` -See `scripts/pattern-tracker-helper.sh` for full documentation. +**Automatic capture**: The supervisor stores `SUCCESS_PATTERN` and `FAILURE_PATTERN` +entries after each task evaluation, tagged with model tier, duration, and retry count. +This data feeds into the `recommend` command for data-driven model routing. + +See `scripts/pattern-tracker-helper.sh help` for full documentation. ## Memory Graduation (Sharing Learnings) diff --git a/.agents/scripts/commands/patterns.md b/.agents/scripts/commands/patterns.md index e4967291..d31e290c 100644 --- a/.agents/scripts/commands/patterns.md +++ b/.agents/scripts/commands/patterns.md @@ -1,5 +1,5 @@ --- -description: Show success/failure patterns from memory to guide task approach +description: Show success/failure patterns from memory to guide task approach and model routing agent: Build+ mode: subagent model: haiku @@ -11,31 +11,45 @@ Arguments: $ARGUMENTS ## Instructions -1. If arguments are provided, use them as a task description to find relevant patterns: +1. If arguments contain "report", show the comprehensive report: + +```bash +~/.aidevops/agents/scripts/pattern-tracker-helper.sh report +``` + +2. If arguments contain "recommend", show model recommendation: + +```bash +~/.aidevops/agents/scripts/pattern-tracker-helper.sh recommend "$ARGUMENTS" +``` + +3. If other arguments are provided, use them as a task description to find relevant patterns: ```bash ~/.aidevops/agents/scripts/pattern-tracker-helper.sh suggest "$ARGUMENTS" ``` -2. If no arguments, show overall pattern statistics and recent patterns: +4. If no arguments, show overall pattern statistics and recent patterns: ```bash ~/.aidevops/agents/scripts/pattern-tracker-helper.sh stats ~/.aidevops/agents/scripts/pattern-tracker-helper.sh analyze --limit 5 ``` -3. Present the results with actionable guidance: +5. Present the results with actionable guidance: - Highlight what approaches have worked for similar tasks - Warn about approaches that have failed - Suggest the optimal model tier based on pattern data -4. If no patterns exist yet, explain how to start recording: +6. If no patterns exist yet, explain how to start recording: ```text -No patterns recorded yet. Patterns are recorded automatically during -development loops, or manually with: +No patterns recorded yet. Patterns are recorded automatically by the +supervisor after task completion, or manually with: pattern-tracker-helper.sh record --outcome success \ --task-type bugfix --model sonnet \ --description "Structured debugging approach found root cause quickly" + +Available commands: suggest, recommend, analyze, stats, report, export ``` diff --git a/.agents/scripts/commands/route.md b/.agents/scripts/commands/route.md index f203356f..ee33d89c 100644 --- a/.agents/scripts/commands/route.md +++ b/.agents/scripts/commands/route.md @@ -1,5 +1,5 @@ --- -description: Suggest optimal model tier for a task description +description: Suggest optimal model tier for a task description using rules + pattern history agent: Build+ mode: subagent model: haiku @@ -11,27 +11,40 @@ Task: $ARGUMENTS ## Instructions -1. Read `tools/context/model-routing.md` for the routing rules and tier definitions. +1. First, check pattern history for data-driven insights: -2. Analyze the task description against the routing rules: +```bash +~/.aidevops/agents/scripts/pattern-tracker-helper.sh recommend "$ARGUMENTS" +``` + +2. Read `tools/context/model-routing.md` for the routing rules and tier definitions. + +3. Analyze the task description against the routing rules: - **Complexity**: Simple transform vs reasoning vs novel design - **Context size**: Small focused task vs large codebase sweep - **Output type**: Classification vs code vs architecture -3. Output a recommendation in this format: +4. Combine pattern history with routing rules: + - If pattern data exists and shows a clear winner (>75% success rate with 3+ samples), weight it heavily + - If pattern data is sparse or inconclusive, rely on routing rules + - If pattern data contradicts routing rules, note the conflict and explain + +5. Output a recommendation in this format: ```text Recommended: {tier} ({model_name}) Reason: {one-line justification} Cost: ~{relative}x vs sonnet baseline +Pattern data: {success_rate}% success rate from {N} samples (or "no data") ``` -4. If the task is ambiguous, suggest the tier and note what would push it up or down: +6. If the task is ambiguous, suggest the tier and note what would push it up or down: ```text Recommended: sonnet (claude-sonnet-4) Reason: Code modification with moderate reasoning Cost: ~1x baseline +Pattern data: 85% success rate from 12 samples Could be haiku if: the change is a simple rename/reformat Could be opus if: the change requires architectural decisions diff --git a/.agents/scripts/pattern-tracker-helper.sh b/.agents/scripts/pattern-tracker-helper.sh index d5c44105..abf20441 100755 --- a/.agents/scripts/pattern-tracker-helper.sh +++ b/.agents/scripts/pattern-tracker-helper.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # pattern-tracker-helper.sh - Track and analyze success/failure patterns -# Extends memory-helper.sh with pattern-specific analysis +# Extends memory-helper.sh with pattern-specific analysis and routing decisions # # Usage: # pattern-tracker-helper.sh record --outcome success --task-type "code-review" \ @@ -9,7 +9,10 @@ # --model haiku --description "Haiku missed edge cases in complex refactor" # pattern-tracker-helper.sh analyze [--task-type TYPE] [--model MODEL] # pattern-tracker-helper.sh suggest "task description" +# pattern-tracker-helper.sh recommend --task-type "bugfix" # pattern-tracker-helper.sh stats +# pattern-tracker-helper.sh export [--format json|csv] +# pattern-tracker-helper.sh report # pattern-tracker-helper.sh help set -euo pipefail @@ -17,14 +20,23 @@ set -euo pipefail # Configuration SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" || exit source "${SCRIPT_DIR}/shared-constants.sh" +init_log_file readonly SCRIPT_DIR readonly MEMORY_HELPER="$SCRIPT_DIR/memory-helper.sh" +readonly MEMORY_DB="${AIDEVOPS_MEMORY_DIR:-$HOME/.aidevops/.agent-workspace/memory}/memory.db" -log_info() { echo -e "${BLUE}[INFO]${NC} $*"; } -log_success() { echo -e "${GREEN}[OK]${NC} $*"; } -log_warn() { echo -e "${YELLOW}[WARN]${NC} $*"; } -log_error() { echo -e "${RED}[ERROR]${NC} $*" >&2; } +# All pattern-related memory types (dedicated + supervisor-generated) +# All pattern-related memory types (dedicated + supervisor-generated) +# Use via: local types_sql="$PATTERN_TYPES_SQL" then sqlite3 ... "$types_sql" ... +# Or inline in single-line sqlite3 calls where variable expansion works correctly +PATTERN_TYPES="'SUCCESS_PATTERN','FAILURE_PATTERN','WORKING_SOLUTION','FAILED_APPROACH','ERROR_FIX'" +readonly PATTERN_TYPES + +log_info() { echo -e "${BLUE}[INFO]${NC} $*"; return 0; } +log_success() { echo -e "${GREEN}[OK]${NC} $*"; return 0; } +log_warn() { echo -e "${YELLOW}[WARN]${NC} $*"; return 0; } +log_error() { echo -e "${RED}[ERROR]${NC} $*" >&2; return 0; } # Valid task types for pattern tracking readonly VALID_TASK_TYPES="code-review refactor bugfix feature docs testing deployment security architecture planning research content seo" @@ -32,6 +44,27 @@ readonly VALID_TASK_TYPES="code-review refactor bugfix feature docs testing depl # Valid model tiers readonly VALID_MODELS="haiku flash sonnet pro opus" +####################################### +# Ensure memory database exists +# Returns 0 if DB exists, 1 if not +####################################### +ensure_db() { + if [[ ! -f "$MEMORY_DB" ]]; then + log_warn "No memory database found at: $MEMORY_DB" + log_info "Run 'memory-helper.sh store' to initialize the database." + return 1 + fi + return 0 +} + +####################################### +# SQL-escape a value (double single quotes) +####################################### +sql_escape() { + local val="$1" + echo "${val//\'/\'\'}" +} + ####################################### # Record a success or failure pattern ####################################### @@ -41,6 +74,9 @@ cmd_record() { local model="" local description="" local tags="" + local task_id="" + local duration="" + local retries="" while [[ $# -gt 0 ]]; do case "$1" in @@ -49,6 +85,9 @@ cmd_record() { --model) model="$2"; shift 2 ;; --description|--desc) description="$2"; shift 2 ;; --tags) tags="$2"; shift 2 ;; + --task-id) task_id="$2"; shift 2 ;; + --duration) duration="$2"; shift 2 ;; + --retries) retries="$2"; shift 2 ;; *) if [[ -z "$description" ]]; then description="$1" @@ -76,16 +115,16 @@ cmd_record() { # Validate task type if provided if [[ -n "$task_type" ]]; then - local type_pattern=" $task_type " - if [[ ! " $VALID_TASK_TYPES " =~ $type_pattern ]]; then + local type_check=" $task_type " + if [[ ! " $VALID_TASK_TYPES " =~ $type_check ]]; then log_warn "Non-standard task type: $task_type (standard: $VALID_TASK_TYPES)" fi fi # Validate model if provided if [[ -n "$model" ]]; then - local model_pattern=" $model " - if [[ ! " $VALID_MODELS " =~ $model_pattern ]]; then + local model_check=" $model " + if [[ ! " $VALID_MODELS " =~ $model_check ]]; then log_warn "Non-standard model: $model (standard: $VALID_MODELS)" fi fi @@ -102,12 +141,18 @@ cmd_record() { local all_tags="pattern" [[ -n "$task_type" ]] && all_tags="$all_tags,$task_type" [[ -n "$model" ]] && all_tags="$all_tags,model:$model" + [[ -n "$task_id" ]] && all_tags="$all_tags,$task_id" + [[ -n "$duration" ]] && all_tags="$all_tags,duration:$duration" + [[ -n "$retries" ]] && all_tags="$all_tags,retries:$retries" [[ -n "$tags" ]] && all_tags="$all_tags,$tags" # Build content with structured metadata local content="$description" [[ -n "$task_type" ]] && content="[task:$task_type] $content" [[ -n "$model" ]] && content="$content [model:$model]" + [[ -n "$task_id" ]] && content="$content [id:$task_id]" + [[ -n "$duration" ]] && content="$content [duration:${duration}s]" + [[ -n "$retries" && "$retries" != "0" ]] && content="$content [retries:$retries]" # Store via memory-helper.sh "$MEMORY_HELPER" store \ @@ -122,6 +167,7 @@ cmd_record() { ####################################### # Analyze patterns from memory +# Uses direct SQLite for reliability ####################################### cmd_analyze() { local task_type="" @@ -137,76 +183,70 @@ cmd_analyze() { esac done + ensure_db || return 0 + echo "" echo -e "${CYAN}=== Pattern Analysis ===${NC}" echo "" - # Fetch success patterns - local success_query="SUCCESS_PATTERN" - [[ -n "$task_type" ]] && success_query="$success_query task:$task_type" - [[ -n "$model" ]] && success_query="$success_query model:$model" + # Build WHERE clause for filtering + local where_success="type IN ('SUCCESS_PATTERN', 'WORKING_SOLUTION')" + local where_failure="type IN ('FAILURE_PATTERN', 'FAILED_APPROACH', 'ERROR_FIX')" + + if [[ -n "$task_type" ]]; then + local escaped_type + escaped_type=$(sql_escape "$task_type") + where_success="$where_success AND (tags LIKE '%${escaped_type}%' OR content LIKE '%task:${escaped_type}%')" + where_failure="$where_failure AND (tags LIKE '%${escaped_type}%' OR content LIKE '%task:${escaped_type}%')" + fi + if [[ -n "$model" ]]; then + local escaped_model + escaped_model=$(sql_escape "$model") + where_success="$where_success AND (tags LIKE '%model:${escaped_model}%' OR content LIKE '%model:${escaped_model}%')" + where_failure="$where_failure AND (tags LIKE '%model:${escaped_model}%' OR content LIKE '%model:${escaped_model}%')" + fi + + # Success patterns echo -e "${GREEN}Success Patterns:${NC}" local success_results - success_results=$("$MEMORY_HELPER" recall --query "$success_query" --type SUCCESS_PATTERN --limit "$limit" --json 2>/dev/null || echo "[]") + success_results=$(sqlite3 "$MEMORY_DB" "SELECT content FROM learnings WHERE $where_success ORDER BY created_at DESC LIMIT $limit;" 2>/dev/null || echo "") - local success_count - if command -v jq &>/dev/null; then - success_count=$(echo "$success_results" | jq 'length' 2>/dev/null || echo "0") - if [[ "$success_count" -gt 0 ]]; then - echo "$success_results" | jq -r '.[] | " + \(.content)"' 2>/dev/null - else - echo " (none recorded)" - fi + if [[ -n "$success_results" ]]; then + while IFS= read -r line; do + echo " + $line" + done <<< "$success_results" else - if [[ "$success_results" != "[]" && -n "$success_results" ]]; then - echo "$success_results" - else - echo " (none recorded)" - success_count=0 - fi + echo " (none recorded)" fi echo "" - # Fetch failure patterns - local failure_query="FAILURE_PATTERN" - [[ -n "$task_type" ]] && failure_query="$failure_query task:$task_type" - [[ -n "$model" ]] && failure_query="$failure_query model:$model" - + # Failure patterns echo -e "${RED}Failure Patterns:${NC}" local failure_results - failure_results=$("$MEMORY_HELPER" recall --query "$failure_query" --type FAILURE_PATTERN --limit "$limit" --json 2>/dev/null || echo "[]") + failure_results=$(sqlite3 "$MEMORY_DB" "SELECT content FROM learnings WHERE $where_failure ORDER BY created_at DESC LIMIT $limit;" 2>/dev/null || echo "") - local failure_count - if command -v jq &>/dev/null; then - failure_count=$(echo "$failure_results" | jq 'length' 2>/dev/null || echo "0") - if [[ "$failure_count" -gt 0 ]]; then - echo "$failure_results" | jq -r '.[] | " - \(.content)"' 2>/dev/null - else - echo " (none recorded)" - fi + if [[ -n "$failure_results" ]]; then + while IFS= read -r line; do + echo " - $line" + done <<< "$failure_results" else - if [[ "$failure_results" != "[]" && -n "$failure_results" ]]; then - echo "$failure_results" - else - echo " (none recorded)" - failure_count=0 - fi + echo " (none recorded)" fi echo "" - # Summary + # Summary counts + local success_count failure_count + success_count=$(sqlite3 "$MEMORY_DB" "SELECT COUNT(*) FROM learnings WHERE $where_success;" 2>/dev/null || echo "0") + failure_count=$(sqlite3 "$MEMORY_DB" "SELECT COUNT(*) FROM learnings WHERE $where_failure;" 2>/dev/null || echo "0") + echo -e "${CYAN}Summary:${NC}" - echo " Successes: ${success_count:-0}" - echo " Failures: ${failure_count:-0}" - if [[ -n "$task_type" ]]; then - echo " Task type: $task_type" - fi - if [[ -n "$model" ]]; then - echo " Model: $model" - fi + echo " Successes: $success_count" + echo " Failures: $failure_count" + [[ -n "$task_type" ]] && echo " Task type: $task_type" + [[ -n "$model" ]] && echo " Model: $model" echo "" return 0 } @@ -226,21 +266,31 @@ cmd_suggest() { echo -e "${CYAN}=== Pattern Suggestions for: \"$task_desc\" ===${NC}" echo "" - # Search for relevant success patterns + # Search for relevant success patterns via FTS5 echo -e "${GREEN}What has worked before:${NC}" local success_results success_results=$("$MEMORY_HELPER" recall --query "$task_desc" --type SUCCESS_PATTERN --limit 5 --json 2>/dev/null || echo "[]") + # Also search WORKING_SOLUTION (supervisor-generated) + local working_results + working_results=$("$MEMORY_HELPER" recall --query "$task_desc" --type WORKING_SOLUTION --limit 3 --json 2>/dev/null || echo "[]") + + local found_success=false if command -v jq &>/dev/null; then - local success_count - success_count=$(echo "$success_results" | jq 'length' 2>/dev/null || echo "0") - if [[ "$success_count" -gt 0 ]]; then - echo "$success_results" | jq -r '.[] | " + \(.content) (score: \(.score // "N/A"))"' 2>/dev/null - else - echo " (no matching success patterns)" + local count + count=$(echo "$success_results" | jq 'length' 2>/dev/null || echo "0") + if [[ "$count" -gt 0 ]]; then + echo "$success_results" | jq -r '.[] | " + \(.content)"' 2>/dev/null + found_success=true + fi + count=$(echo "$working_results" | jq 'length' 2>/dev/null || echo "0") + if [[ "$count" -gt 0 ]]; then + echo "$working_results" | jq -r '.[] | " + \(.content)"' 2>/dev/null + found_success=true fi - else - echo " (install jq for formatted output)" + fi + if [[ "$found_success" == false ]]; then + echo " (no matching success patterns)" fi echo "" @@ -250,16 +300,131 @@ cmd_suggest() { local failure_results failure_results=$("$MEMORY_HELPER" recall --query "$task_desc" --type FAILURE_PATTERN --limit 5 --json 2>/dev/null || echo "[]") + local failed_results + failed_results=$("$MEMORY_HELPER" recall --query "$task_desc" --type FAILED_APPROACH --limit 3 --json 2>/dev/null || echo "[]") + + local found_failure=false if command -v jq &>/dev/null; then - local failure_count - failure_count=$(echo "$failure_results" | jq 'length' 2>/dev/null || echo "0") - if [[ "$failure_count" -gt 0 ]]; then - echo "$failure_results" | jq -r '.[] | " - \(.content) (score: \(.score // "N/A"))"' 2>/dev/null + local count + count=$(echo "$failure_results" | jq 'length' 2>/dev/null || echo "0") + if [[ "$count" -gt 0 ]]; then + echo "$failure_results" | jq -r '.[] | " - \(.content)"' 2>/dev/null + found_failure=true + fi + count=$(echo "$failed_results" | jq 'length' 2>/dev/null || echo "0") + if [[ "$count" -gt 0 ]]; then + echo "$failed_results" | jq -r '.[] | " - \(.content)"' 2>/dev/null + found_failure=true + fi + fi + if [[ "$found_failure" == false ]]; then + echo " (no matching failure patterns)" + fi + + echo "" + + # Model recommendation based on patterns + _show_model_hint "$task_desc" + + return 0 +} + +####################################### +# Recommend model tier based on pattern history +# Queries patterns tagged with model info and calculates success rates +####################################### +cmd_recommend() { + local task_type="" + local task_desc="" + + while [[ $# -gt 0 ]]; do + case "$1" in + --task-type) task_type="$2"; shift 2 ;; + *) + if [[ -z "$task_desc" ]]; then + task_desc="$1" + else + task_desc="$task_desc $1" + fi + shift + ;; + esac + done + + ensure_db || { + echo "" + echo -e "${CYAN}=== Model Recommendation ===${NC}" + echo "" + echo " No pattern data available. Default recommendation: sonnet" + echo " Record patterns to enable data-driven routing." + echo "" + return 0 + } + + echo "" + echo -e "${CYAN}=== Model Recommendation ===${NC}" + echo "" + + # Build filter clause + local filter="" + if [[ -n "$task_type" ]]; then + local escaped_type + escaped_type=$(sql_escape "$task_type") + filter="AND (tags LIKE '%${escaped_type}%' OR content LIKE '%task:${escaped_type}%')" + echo -e " Task type: ${WHITE}$task_type${NC}" + fi + if [[ -n "$task_desc" ]]; then + echo -e " Description: ${WHITE}$task_desc${NC}" + fi + echo "" + + # Query success/failure counts per model tier + echo -e "${CYAN}Model Performance (from pattern history):${NC}" + echo "" + printf " %-10s %8s %8s %10s\n" "Model" "Success" "Failure" "Rate" + printf " %-10s %8s %8s %10s\n" "-----" "-------" "-------" "----" + + local best_model="" best_rate=0 has_data=false + + for model_tier in $VALID_MODELS; do + local model_filter="AND (tags LIKE '%model:${model_tier}%' OR content LIKE '%model:${model_tier}%')" + + local successes failures + successes=$(sqlite3 "$MEMORY_DB" "SELECT COUNT(*) FROM learnings WHERE type IN ('SUCCESS_PATTERN', 'WORKING_SOLUTION') $model_filter $filter;" 2>/dev/null || echo "0") + failures=$(sqlite3 "$MEMORY_DB" "SELECT COUNT(*) FROM learnings WHERE type IN ('FAILURE_PATTERN', 'FAILED_APPROACH', 'ERROR_FIX') $model_filter $filter;" 2>/dev/null || echo "0") + + local total=$((successes + failures)) + if [[ "$total" -gt 0 ]]; then + has_data=true + local rate + rate=$(( (successes * 100) / total )) + printf " %-10s %8d %8d %9d%%\n" "$model_tier" "$successes" "$failures" "$rate" + + # Track best model (prefer higher success rate, break ties with more data) + if [[ "$rate" -gt "$best_rate" ]] || { [[ "$rate" -eq "$best_rate" ]] && [[ "$total" -gt 0 ]]; }; then + best_rate=$rate + best_model=$model_tier + fi else - echo " (no matching failure patterns)" + printf " %-10s %8s %8s %10s\n" "$model_tier" "-" "-" "no data" + fi + done + + echo "" + + # Recommendation + if [[ "$has_data" == true && -n "$best_model" ]]; then + echo -e " ${GREEN}Recommended: ${WHITE}$best_model${GREEN} (${best_rate}% success rate)${NC}" + + # Add context about the recommendation + if [[ "$best_rate" -lt 50 ]]; then + echo -e " ${YELLOW}Warning: Low success rate across all models. Consider reviewing task approach.${NC}" + elif [[ "$best_rate" -lt 75 ]]; then + echo -e " ${YELLOW}Note: Moderate success rate. Consider using a higher-tier model for complex tasks.${NC}" fi else - echo " (install jq for formatted output)" + echo -e " ${YELLOW}No pattern data for model comparison. Default: sonnet${NC}" + echo " Record patterns with --model flag to enable data-driven routing." fi echo "" @@ -274,29 +439,39 @@ cmd_stats() { echo -e "${CYAN}=== Pattern Statistics ===${NC}" echo "" - # Count by type using direct SQLite queries (FTS5 search unreliable for type filtering) - local memory_db="${AIDEVOPS_MEMORY_DIR:-$HOME/.aidevops/.agent-workspace/memory}/memory.db" - - if [[ ! -f "$memory_db" ]]; then - echo " No memory database found." - return 0 - fi + ensure_db || return 0 + # Count by dedicated pattern types local success_count failure_count - success_count=$(sqlite3 "$memory_db" "SELECT COUNT(*) FROM learnings WHERE type = 'SUCCESS_PATTERN';" 2>/dev/null || echo "0") - failure_count=$(sqlite3 "$memory_db" "SELECT COUNT(*) FROM learnings WHERE type = 'FAILURE_PATTERN';" 2>/dev/null || echo "0") + success_count=$(sqlite3 "$MEMORY_DB" "SELECT COUNT(*) FROM learnings WHERE type = 'SUCCESS_PATTERN';" 2>/dev/null || echo "0") + failure_count=$(sqlite3 "$MEMORY_DB" "SELECT COUNT(*) FROM learnings WHERE type = 'FAILURE_PATTERN';" 2>/dev/null || echo "0") + + # Count supervisor-generated patterns + local working_count failed_count error_count + working_count=$(sqlite3 "$MEMORY_DB" "SELECT COUNT(*) FROM learnings WHERE type = 'WORKING_SOLUTION' AND tags LIKE '%supervisor%';" 2>/dev/null || echo "0") + failed_count=$(sqlite3 "$MEMORY_DB" "SELECT COUNT(*) FROM learnings WHERE type = 'FAILED_APPROACH' AND tags LIKE '%supervisor%';" 2>/dev/null || echo "0") + error_count=$(sqlite3 "$MEMORY_DB" "SELECT COUNT(*) FROM learnings WHERE type = 'ERROR_FIX' AND tags LIKE '%supervisor%';" 2>/dev/null || echo "0") + + echo " Dedicated patterns:" + echo " SUCCESS_PATTERN: $success_count" + echo " FAILURE_PATTERN: $failure_count" + echo "" + echo " Supervisor-generated:" + echo " WORKING_SOLUTION: $working_count" + echo " FAILED_APPROACH: $failed_count" + echo " ERROR_FIX: $error_count" + echo "" - echo " Success patterns: $success_count" - echo " Failure patterns: $failure_count" - echo " Total patterns: $(( success_count + failure_count ))" + local total=$(( success_count + failure_count + working_count + failed_count + error_count )) + echo " Total trackable patterns: $total" echo "" - # Show task type breakdown by querying tags directly + # Show task type breakdown echo " Task types with patterns:" local found_any=false for task_type in $VALID_TASK_TYPES; do local type_count - type_count=$(sqlite3 "$memory_db" "SELECT COUNT(*) FROM learnings WHERE type IN ('SUCCESS_PATTERN', 'FAILURE_PATTERN') AND tags LIKE '%${task_type}%';" 2>/dev/null || echo "0") + type_count=$(sqlite3 "$MEMORY_DB" "SELECT COUNT(*) FROM learnings WHERE type IN ($PATTERN_TYPES) AND (tags LIKE '%${task_type}%' OR content LIKE '%task:${task_type}%');" 2>/dev/null || echo "0") if [[ "$type_count" -gt 0 ]]; then echo " $task_type: $type_count" found_any=true @@ -306,6 +481,237 @@ cmd_stats() { echo " (none recorded with task types)" fi echo "" + + # Show model tier breakdown + echo " Model tiers with patterns:" + local found_model=false + for model_tier in $VALID_MODELS; do + local model_count + model_count=$(sqlite3 "$MEMORY_DB" "SELECT COUNT(*) FROM learnings WHERE type IN ($PATTERN_TYPES) AND (tags LIKE '%model:${model_tier}%' OR content LIKE '%model:${model_tier}%');" 2>/dev/null || echo "0") + if [[ "$model_count" -gt 0 ]]; then + echo " $model_tier: $model_count" + found_model=true + fi + done + if [[ "$found_model" == false ]]; then + echo " (none recorded with model tiers)" + fi + echo "" + + # Success rate + local total_success=$((success_count + working_count)) + local total_failure=$((failure_count + failed_count + error_count)) + local total_all=$((total_success + total_failure)) + if [[ "$total_all" -gt 0 ]]; then + local overall_rate=$(( (total_success * 100) / total_all )) + echo " Overall success rate: ${overall_rate}% ($total_success/$total_all)" + fi + echo "" + return 0 +} + +####################################### +# Export patterns as JSON or CSV +####################################### +cmd_export() { + local format="json" + + while [[ $# -gt 0 ]]; do + case "$1" in + --format|-f) format="$2"; shift 2 ;; + *) shift ;; + esac + done + + ensure_db || return 1 + + # SQL for pattern types (used in queries below) + local types_sql="'SUCCESS_PATTERN','FAILURE_PATTERN','WORKING_SOLUTION','FAILED_APPROACH','ERROR_FIX'" + + case "$format" in + json) + # Use sqlite3 -json for proper JSON output + local query="SELECT l.id, l.type, l.content, l.tags, l.confidence, l.created_at, COALESCE(a.access_count, 0) as access_count FROM learnings l LEFT JOIN learning_access a ON l.id = a.id WHERE l.type IN ($types_sql) ORDER BY l.created_at DESC;" + sqlite3 -json "$MEMORY_DB" "$query" 2>/dev/null || echo "[]" + ;; + csv) + echo "id,type,content,tags,confidence,created_at,access_count" + local csv_query="SELECT l.id, l.type, l.content, l.tags, l.confidence, l.created_at, COALESCE(a.access_count, 0) FROM learnings l LEFT JOIN learning_access a ON l.id = a.id WHERE l.type IN ($types_sql) ORDER BY l.created_at DESC;" + sqlite3 -csv "$MEMORY_DB" "$csv_query" 2>/dev/null + ;; + *) + log_error "Unknown format: $format (use json or csv)" + return 1 + ;; + esac + return 0 +} + +####################################### +# Generate a summary report of patterns +####################################### +cmd_report() { + ensure_db || return 0 + + echo "" + echo -e "${CYAN}=== Pattern Tracking Report ===${NC}" + echo -e " Generated: $(date -u '+%Y-%m-%d %H:%M:%S UTC')" + echo "" + + # Overall counts + local total_patterns + total_patterns=$(sqlite3 "$MEMORY_DB" "SELECT COUNT(*) FROM learnings WHERE type IN ($PATTERN_TYPES);" 2>/dev/null || echo "0") + echo " Total patterns tracked: $total_patterns" + + if [[ "$total_patterns" -eq 0 ]]; then + echo "" + echo " No patterns recorded yet. Patterns are captured:" + echo " - Automatically by the supervisor after task completion" + echo " - Manually via: pattern-tracker-helper.sh record ..." + echo "" + return 0 + fi + + # Date range + local oldest newest + oldest=$(sqlite3 "$MEMORY_DB" "SELECT MIN(created_at) FROM learnings WHERE type IN ($PATTERN_TYPES);" 2>/dev/null || echo "unknown") + newest=$(sqlite3 "$MEMORY_DB" "SELECT MAX(created_at) FROM learnings WHERE type IN ($PATTERN_TYPES);" 2>/dev/null || echo "unknown") + echo " Date range: $oldest to $newest" + echo "" + + # Success vs failure breakdown + local success_total failure_total + success_total=$(sqlite3 "$MEMORY_DB" "SELECT COUNT(*) FROM learnings WHERE type IN ('SUCCESS_PATTERN', 'WORKING_SOLUTION');" 2>/dev/null || echo "0") + failure_total=$(sqlite3 "$MEMORY_DB" "SELECT COUNT(*) FROM learnings WHERE type IN ('FAILURE_PATTERN', 'FAILED_APPROACH', 'ERROR_FIX');" 2>/dev/null || echo "0") + + echo -e " ${GREEN}Successes: $success_total${NC}" + echo -e " ${RED}Failures: $failure_total${NC}" + + local total_sf=$((success_total + failure_total)) + if [[ "$total_sf" -gt 0 ]]; then + local rate=$(( (success_total * 100) / total_sf )) + echo " Success rate: ${rate}%" + fi + echo "" + + # Top failure reasons (most common failure content patterns) + echo -e "${RED}Most Common Failure Patterns:${NC}" + local top_failures + top_failures=$(sqlite3 "$MEMORY_DB" " + SELECT content, COUNT(*) as cnt + FROM learnings + WHERE type IN ('FAILURE_PATTERN', 'FAILED_APPROACH', 'ERROR_FIX') + GROUP BY content + ORDER BY cnt DESC + LIMIT 5; + " 2>/dev/null || echo "") + + if [[ -n "$top_failures" ]]; then + while IFS='|' read -r content cnt; do + # Truncate long content + if [[ ${#content} -gt 80 ]]; then + content="${content:0:77}..." + fi + echo " ($cnt) $content" + done <<< "$top_failures" + else + echo " (none)" + fi + echo "" + + # Model tier performance + echo -e "${CYAN}Model Tier Performance:${NC}" + printf " %-10s %8s %8s %10s\n" "Model" "Success" "Failure" "Rate" + printf " %-10s %8s %8s %10s\n" "-----" "-------" "-------" "----" + + for model_tier in $VALID_MODELS; do + local model_filter="AND (tags LIKE '%model:${model_tier}%' OR content LIKE '%model:${model_tier}%')" + local m_success m_failure + m_success=$(sqlite3 "$MEMORY_DB" "SELECT COUNT(*) FROM learnings WHERE type IN ('SUCCESS_PATTERN', 'WORKING_SOLUTION') $model_filter;" 2>/dev/null || echo "0") + m_failure=$(sqlite3 "$MEMORY_DB" "SELECT COUNT(*) FROM learnings WHERE type IN ('FAILURE_PATTERN', 'FAILED_APPROACH', 'ERROR_FIX') $model_filter;" 2>/dev/null || echo "0") + + local m_total=$((m_success + m_failure)) + if [[ "$m_total" -gt 0 ]]; then + local m_rate=$(( (m_success * 100) / m_total )) + printf " %-10s %8d %8d %9d%%\n" "$model_tier" "$m_success" "$m_failure" "$m_rate" + fi + done + echo "" + + # Recent patterns (last 5) + echo -e "${CYAN}Recent Patterns (last 5):${NC}" + local recent + recent=$(sqlite3 -separator '|' "$MEMORY_DB" " + SELECT type, content, created_at + FROM learnings + WHERE type IN ($PATTERN_TYPES) + ORDER BY created_at DESC + LIMIT 5; + " 2>/dev/null || echo "") + + if [[ -n "$recent" ]]; then + while IFS='|' read -r type content created_at; do + local icon="?" + case "$type" in + SUCCESS_PATTERN|WORKING_SOLUTION) icon="${GREEN}+${NC}" ;; + FAILURE_PATTERN|FAILED_APPROACH|ERROR_FIX) icon="${RED}-${NC}" ;; + esac + # Truncate long content + if [[ ${#content} -gt 70 ]]; then + content="${content:0:67}..." + fi + echo -e " $icon $content" + echo " ($created_at)" + done <<< "$recent" + else + echo " (none)" + fi + echo "" + return 0 +} + +####################################### +# Internal: Show model hint based on pattern data +# Used by suggest command to add routing context +####################################### +_show_model_hint() { + local task_desc="$1" + + if ! ensure_db 2>/dev/null; then + return 0 + fi + + # Check if any patterns have model data + local model_patterns + model_patterns=$(sqlite3 "$MEMORY_DB" "SELECT COUNT(*) FROM learnings WHERE type IN ($PATTERN_TYPES) AND (tags LIKE '%model:%' OR content LIKE '%model:%');" 2>/dev/null || echo "0") + + if [[ "$model_patterns" -gt 0 ]]; then + echo -e "${CYAN}Model Routing Hint:${NC}" + + local best_model="" best_rate=0 + for model_tier in $VALID_MODELS; do + local model_filter="AND (tags LIKE '%model:${model_tier}%' OR content LIKE '%model:${model_tier}%')" + local m_success m_failure + m_success=$(sqlite3 "$MEMORY_DB" "SELECT COUNT(*) FROM learnings WHERE type IN ('SUCCESS_PATTERN', 'WORKING_SOLUTION') $model_filter;" 2>/dev/null || echo "0") + m_failure=$(sqlite3 "$MEMORY_DB" "SELECT COUNT(*) FROM learnings WHERE type IN ('FAILURE_PATTERN', 'FAILED_APPROACH', 'ERROR_FIX') $model_filter;" 2>/dev/null || echo "0") + + local m_total=$((m_success + m_failure)) + if [[ "$m_total" -gt 2 ]]; then + local m_rate=$(( (m_success * 100) / m_total )) + if [[ "$m_rate" -gt "$best_rate" ]]; then + best_rate=$m_rate + best_model=$model_tier + fi + fi + done + + if [[ -n "$best_model" ]]; then + echo " Based on pattern history, $best_model has the highest success rate (${best_rate}%)" + else + echo " Not enough model-tagged patterns for a recommendation yet." + fi + echo "" + fi return 0 } @@ -323,7 +729,10 @@ COMMANDS: record Record a success or failure pattern analyze Analyze patterns by task type or model suggest Get suggestions based on past patterns for a task - stats Show pattern statistics + recommend Recommend model tier based on historical success rates + stats Show pattern statistics (includes supervisor patterns) + export Export patterns as JSON or CSV + report Generate a comprehensive pattern report help Show this help RECORD OPTIONS: @@ -331,6 +740,9 @@ RECORD OPTIONS: --task-type Task category (code-review, refactor, bugfix, etc.) --model Model used (haiku, flash, sonnet, pro, opus) --description What happened (required) + --task-id Task identifier (e.g., t102.3) + --duration How long the task took + --retries Number of retries before completion --tags Additional comma-separated tags ANALYZE OPTIONS: @@ -338,27 +750,42 @@ ANALYZE OPTIONS: --model Filter by model tier --limit Max results per category (default: 20) +RECOMMEND OPTIONS: + --task-type Filter recommendation by task type + +EXPORT OPTIONS: + --format Output format (default: json) + VALID TASK TYPES: code-review, refactor, bugfix, feature, docs, testing, deployment, security, architecture, planning, research, content, seo EXAMPLES: - # Record a success + # Record a success with full metadata pattern-tracker-helper.sh record --outcome success \ - --task-type code-review --model sonnet \ - --description "Structured checklist caught 3 bugs" + --task-type code-review --model sonnet --task-id t102.3 \ + --duration 120 --description "Structured checklist caught 3 bugs" # Record a failure pattern-tracker-helper.sh record --outcome failure \ --task-type refactor --model haiku \ --description "Haiku missed edge cases in complex refactor" + # Get model recommendation for a task type + pattern-tracker-helper.sh recommend --task-type bugfix + # Analyze patterns for a task type pattern-tracker-helper.sh analyze --task-type bugfix # Get suggestions for a new task pattern-tracker-helper.sh suggest "refactor the auth middleware" + # Export patterns as JSON + pattern-tracker-helper.sh export --format json > patterns.json + + # Generate a report + pattern-tracker-helper.sh report + # View statistics pattern-tracker-helper.sh stats EOF @@ -376,7 +803,10 @@ main() { record) cmd_record "$@" ;; analyze) cmd_analyze "$@" ;; suggest) cmd_suggest "$@" ;; + recommend) cmd_recommend "$@" ;; stats) cmd_stats ;; + export) cmd_export "$@" ;; + report) cmd_report ;; help|--help|-h) cmd_help ;; *) log_error "Unknown command: $command" diff --git a/.agents/scripts/supervisor-helper.sh b/.agents/scripts/supervisor-helper.sh index 423cfe24..c4ccc163 100755 --- a/.agents/scripts/supervisor-helper.sh +++ b/.agents/scripts/supervisor-helper.sh @@ -7665,6 +7665,7 @@ $task_memories" # Store a failure pattern in memory after evaluation # Called when a task fails, is blocked, or retries # Tags with supervisor context for future recall +# Uses FAILURE_PATTERN type for pattern-tracker integration (t102.3) ####################################### store_failure_pattern() { local task_id="$1" @@ -7679,7 +7680,7 @@ store_failure_pattern() { # Only store meaningful failure patterns (not transient retries) case "$outcome_type" in blocked|failed) - local memory_type="FAILED_APPROACH" + true # Always store these ;; retry) # Only store retry patterns if they indicate a recurring issue @@ -7689,23 +7690,43 @@ store_failure_pattern() { return 0 ;; esac - local memory_type="ERROR_FIX" ;; *) return 0 ;; esac + # Look up model tier from task record for pattern routing (t102.3) + local model_tier="" + local task_model + task_model=$(db "$SUPERVISOR_DB" "SELECT model FROM tasks WHERE id = '$(sql_escape "$task_id")';" 2>/dev/null || echo "") + if [[ -n "$task_model" ]]; then + # Extract tier name from model string (e.g., "anthropic/claude-opus-4-6" -> "opus") + case "$task_model" in + *haiku*) model_tier="haiku" ;; + *flash*) model_tier="flash" ;; + *sonnet*) model_tier="sonnet" ;; + *opus*) model_tier="opus" ;; + *pro*) model_tier="pro" ;; + esac + fi + + # Build structured content for pattern-tracker compatibility local content="Supervisor task $task_id ($outcome_type): $outcome_detail" if [[ -n "$description" ]]; then - content="$content | Task: $description" + content="[task:feature] $content | Task: $description" fi + [[ -n "$model_tier" ]] && content="$content [model:$model_tier]" + + # Build tags with model info for pattern-tracker queries + local tags="supervisor,pattern,$task_id,$outcome_type,$outcome_detail" + [[ -n "$model_tier" ]] && tags="$tags,model:$model_tier" "$MEMORY_HELPER" store \ --auto \ - --type "$memory_type" \ + --type "FAILURE_PATTERN" \ --content "$content" \ - --tags "supervisor,$task_id,$outcome_type,$outcome_detail" \ + --tags "$tags" \ 2>/dev/null || true log_info "Stored failure pattern in memory: $task_id ($outcome_type: $outcome_detail)" @@ -7715,6 +7736,7 @@ store_failure_pattern() { ####################################### # Store a success pattern in memory after task completion # Records what worked for future reference +# Uses SUCCESS_PATTERN type for pattern-tracker integration (t102.3) ####################################### store_success_pattern() { local task_id="$1" @@ -7725,26 +7747,63 @@ store_success_pattern() { return 0 fi + # Look up model tier and timing from task record (t102.3) + local escaped_id + escaped_id=$(sql_escape "$task_id") + local model_tier="" + local task_model duration_info retries + task_model=$(db "$SUPERVISOR_DB" "SELECT model FROM tasks WHERE id = '$escaped_id';" 2>/dev/null || echo "") + retries=$(db "$SUPERVISOR_DB" "SELECT retries FROM tasks WHERE id = '$escaped_id';" 2>/dev/null || echo "0") + + # Calculate duration if timestamps available + local started completed duration_secs="" + started=$(db "$SUPERVISOR_DB" "SELECT started_at FROM tasks WHERE id = '$escaped_id';" 2>/dev/null || echo "") + completed=$(db "$SUPERVISOR_DB" "SELECT completed_at FROM tasks WHERE id = '$escaped_id';" 2>/dev/null || echo "") + if [[ -n "$started" && -n "$completed" ]]; then + local start_epoch end_epoch + start_epoch=$(date -j -f "%Y-%m-%dT%H:%M:%SZ" "$started" "+%s" 2>/dev/null || date -d "$started" "+%s" 2>/dev/null || echo "") + end_epoch=$(date -j -f "%Y-%m-%dT%H:%M:%SZ" "$completed" "+%s" 2>/dev/null || date -d "$completed" "+%s" 2>/dev/null || echo "") + if [[ -n "$start_epoch" && -n "$end_epoch" ]]; then + duration_secs=$((end_epoch - start_epoch)) + fi + fi + + # Extract tier name from model string + if [[ -n "$task_model" ]]; then + case "$task_model" in + *haiku*) model_tier="haiku" ;; + *flash*) model_tier="flash" ;; + *sonnet*) model_tier="sonnet" ;; + *opus*) model_tier="opus" ;; + *pro*) model_tier="pro" ;; + esac + fi + + # Build structured content for pattern-tracker compatibility local content="Supervisor task $task_id completed successfully" if [[ -n "$detail" && "$detail" != "no_pr" ]]; then content="$content | PR: $detail" fi if [[ -n "$description" ]]; then - content="$content | Task: $description" + content="[task:feature] $content | Task: $description" fi - - # Get retry count for context - local retries=0 - retries=$(db "$SUPERVISOR_DB" "SELECT retries FROM tasks WHERE id = '$(sql_escape "$task_id")';" 2>/dev/null || echo 0) + [[ -n "$model_tier" ]] && content="$content [model:$model_tier]" + [[ -n "$duration_secs" ]] && content="$content [duration:${duration_secs}s]" if [[ "$retries" -gt 0 ]]; then - content="$content | Succeeded after $retries retries" + content="$content [retries:$retries]" fi + # Build tags with model and duration info for pattern-tracker queries + local tags="supervisor,pattern,$task_id,complete" + [[ -n "$model_tier" ]] && tags="$tags,model:$model_tier" + [[ -n "$duration_secs" ]] && tags="$tags,duration:$duration_secs" + [[ "$retries" -gt 0 ]] && tags="$tags,retries:$retries" + "$MEMORY_HELPER" store \ --auto \ - --type "WORKING_SOLUTION" \ + --type "SUCCESS_PATTERN" \ --content "$content" \ - --tags "supervisor,$task_id,complete" \ + --tags "$tags" \ 2>/dev/null || true log_info "Stored success pattern in memory: $task_id"