t1098: Wire compare-models to read live pattern data#1637
Conversation
Adds live success rate data from pattern-tracker-helper.sh memory DB to compare-models-helper.sh. Pattern data appears in: - cmd_list: tier success rates appended after pricing table - cmd_compare: per-model success rate badges in capabilities section - cmd_recommend: success rates alongside pricing for recommended models - cmd_capabilities: success rates in tier mapping display - cmd_patterns: new dedicated command for focused pattern data view Integration reads directly from the shared memory.db (same schema as pattern-tracker-helper.sh) — no data duplication. Gracefully degrades when no pattern data exists.
|
Warning Rate limit exceeded
⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary of ChangesHello @marcusquinn, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the Highlights
Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
🔍 Code Quality Report�[0;35m[MONITOR]�[0m Code Review Monitoring Report �[0;34m[INFO]�[0m Latest Quality Status: �[0;34m[INFO]�[0m Recent monitoring activity: 📈 Current Quality Metrics
Generated on: Wed Feb 18 02:56:15 UTC 2026 Generated by AI DevOps Framework Code Review Monitoring |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request effectively integrates live pattern data from the pattern-tracker-helper.sh into the compare-models-helper.sh script, adding valuable real-world performance metrics to the model comparison tool. The new patterns command is a great addition for a focused view on model success rates. The code is well-structured and handles the absence of data gracefully.
My review focuses on a couple of areas for improvement:
- Code Duplication: There's an opportunity to refactor duplicated database query logic into a shared helper function.
- Shell Scripting Robustness: A suggestion is made to use a bash array for iterating over model tiers to make the script more robust and compliant with ShellCheck best practices, as required by the repository's style guide.
Overall, these are solid enhancements that make the tool more data-driven.
| # Same DB as pattern-tracker-helper.sh — no duplication of storage. | ||
|
|
||
| readonly PATTERN_DB="${AIDEVOPS_MEMORY_DIR:-$HOME/.aidevops/.agent-workspace/memory}/memory.db" | ||
| readonly PATTERN_VALID_MODELS="haiku flash sonnet pro opus" |
There was a problem hiding this comment.
Using a space-separated string for PATTERN_VALID_MODELS and relying on word-splitting in for loops (in get_all_tier_patterns on line 113 and cmd_list on line 286) can be fragile and is flagged by ShellCheck (SC2043). The style guide (rule 14) requires zero ShellCheck violations. It's more robust to define this as a bash array. The loops should then be updated to iterate over the array elements safely (e.g., for tier in "${PATTERN_VALID_MODELS[@]}"; do).
| readonly PATTERN_VALID_MODELS="haiku flash sonnet pro opus" | |
| readonly -a PATTERN_VALID_MODELS=(haiku flash sonnet pro opus) |
References
- ShellCheck zero violations required -- targeted inline disables with reason comments only (link)
There was a problem hiding this comment.
Fixed: converted PATTERN_VALID_MODELS to a bash array (readonly -a) and updated all 3 loop sites to use "\${PATTERN_VALID_MODELS[@]}". ShellCheck passes clean.
| # Get pattern data | ||
| local filter="" | ||
| if [[ -n "$task_type" ]]; then | ||
| filter="AND (tags LIKE '%${task_type}%' OR content LIKE '%task:${task_type}%')" | ||
| fi | ||
| local model_filter="AND (tags LIKE '%model:${tier}%' OR content LIKE '%model:${tier}%')" | ||
|
|
||
| local successes failures | ||
| successes=$(sqlite3 "$PATTERN_DB" "SELECT COUNT(*) FROM learnings WHERE type IN ('SUCCESS_PATTERN', 'WORKING_SOLUTION') $model_filter $filter;" 2>/dev/null || echo "0") | ||
| failures=$(sqlite3 "$PATTERN_DB" "SELECT COUNT(*) FROM learnings WHERE type IN ('FAILURE_PATTERN', 'FAILED_APPROACH', 'ERROR_FIX') $model_filter $filter;" 2>/dev/null || echo "0") |
There was a problem hiding this comment.
This block of code to get success and failure counts from the database is duplicated from the get_tier_success_rate function. To improve maintainability and follow the DRY (Don't Repeat Yourself) principle, this logic should be extracted into a new internal helper function. Both get_tier_success_rate and cmd_patterns can then call this new function.
For example, you could create a function like this:
_get_tier_pattern_counts() {
local tier="$1"
local task_type="${2:-}"
[[ -f "$PATTERN_DB" ]] || { echo "0|0"; return 0; }
local filter=""
if [[ -n "$task_type" ]]; then
filter="AND (tags LIKE '%${task_type}%' OR content LIKE '%task:${task_type}%')"
fi
local model_filter="AND (tags LIKE '%model:${tier}%' OR content LIKE '%model:${tier}%')"
local successes failures
successes=$(sqlite3 "$PATTERN_DB" "SELECT COUNT(*) FROM learnings WHERE type IN ('SUCCESS_PATTERN', 'WORKING_SOLUTION') $model_filter $filter;" 2>/dev/null || echo "0")
failures=$(sqlite3 "$PATTERN_DB" "SELECT COUNT(*) FROM learnings WHERE type IN ('FAILURE_PATTERN', 'FAILED_APPROACH', 'ERROR_FIX') $model_filter $filter;" 2>/dev/null || echo "0")
echo "${successes}|${failures}"
return 0
}Then get_tier_success_rate and cmd_patterns can be simplified by calling _get_tier_pattern_counts.
References
- In shell scripts, extract repeated logic into an internal helper function to improve maintainability. This applies even for standalone scripts where external
sourcedependencies are avoided.
There was a problem hiding this comment.
Fixed: extracted _get_tier_pattern_counts() internal helper that returns successes|failures. Both get_tier_success_rate and cmd_patterns now call it instead of duplicating the query logic.
🔍 Code Quality Report�[0;35m[MONITOR]�[0m Code Review Monitoring Report �[0;34m[INFO]�[0m Latest Quality Status: �[0;34m[INFO]�[0m Recent monitoring activity: 📈 Current Quality Metrics
Generated on: Wed Feb 18 03:07:46 UTC 2026 Generated by AI DevOps Framework Code Review Monitoring |
|
…d t1094.1 (t1107) All t1094 deliverables confirmed implemented: - Prompt strategy tracking → t1095 (pr:#1629) - Output quality gradient + failure categorization → t1096 (pr:#1632) - Token usage → t1095 (pr:#1629) - A/B comparison → t1098+t1099 (pr:#1637, pr:#1634) - Prompt-repeat strategy → t1097 (pr:#1631) - Build-agent reference → t1094.1 (pr:#1633) t1094 parent ready to be marked complete: verified:2026-02-18
1 participant
Summary
pattern-tracker-helper.sh(memory.db) intocompare-models-helper.shoutputlist,compare,recommend, andcapabilitiescommands alongside static pricing/capability specspatternscommand for a focused view of model performance with optional--task-typefilteringChanges
compare-models-helper.sh:has_pattern_data(),get_tier_success_rate(),model_id_to_tier(),format_pattern_badge(),get_all_tier_patterns()cmd_list: Appends tier success rates after pricing tablecmd_compare: Shows per-model success rate badges in capabilities sectioncmd_recommend: Includes success rates alongside pricing for recommended models + pattern insights sectioncmd_capabilities: Shows success rates in tier mapping displaycmd_patterns: New dedicated command showing tier performance table with pricingcompare-models.md: Updated agent doc with pattern data workflow step and examplesExample Output
Verification
--task-typefilteringRef #1623