t1220: AI-powered semantic dedup for task creation#1863
Conversation
Replace keyword-only duplicate detection with two-layer approach: 1. Fast keyword pre-filter scans open tasks for word overlap (free, instant) 2. Sonnet AI call verifies semantic similarity (~$0.002 per check) This prevents the supervisor from creating near-duplicate tasks when the AI generates slightly different titles for the same underlying issue. The keyword pre-filter catches candidates, then sonnet makes the final call — understanding that 'Investigate X failures' and 'Add diagnostics for X' are the same work. Falls back to keyword-only (3+ match threshold) if AI is unavailable. New config: AI_SEMANTIC_DEDUP_USE_AI, AI_SEMANTIC_DEDUP_TIMEOUT
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
WalkthroughThis change implements a two-layer semantic deduplication system for task creation in the AI-actions automation script. It introduces AI-powered duplicate detection with configurable thresholds, timeout settings, and fallback behavior, replacing single-function logic with a keyword pre-filter and optional AI verification approach. Changes
Sequence DiagramsequenceDiagram
participant TC as Task Creation Flow
participant KPF as Keyword Pre-Filter
participant AI as AI Service<br/>(Claude/Sonnet)
participant FB as Fallback Logic
participant DB as Task Database
TC->>KPF: proposed_title
KPF->>DB: extract keyword matches
DB-->>KPF: candidates (top 5)
KPF-->>TC: candidate list
alt AI_SEMANTIC_DEDUP_USE_AI enabled
TC->>AI: proposed_title + candidates
alt AI responds within timeout
AI-->>TC: JSON {duplicate, task_id, confidence}
TC->>TC: evaluate AI result
alt Duplicate confirmed
TC-->>TC: skip creation, return existing
else No duplicate
TC-->>TC: proceed with creation
end
else AI timeout/unavailable
TC->>FB: fallback to keyword logic
FB->>FB: apply strict threshold (3+)
FB-->>TC: keyword-only result
end
else AI disabled
TC->>FB: use keyword-only logic
FB-->>TC: fallback result
end
Estimated Code Review Effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🔍 Code Quality Report�[0;35m[MONITOR]�[0m Code Review Monitoring Report �[0;34m[INFO]�[0m Latest Quality Status: �[0;34m[INFO]�[0m Recent monitoring activity: 📈 Current Quality Metrics
Generated on: Thu Feb 19 01:40:20 UTC 2026 Generated by AI DevOps Framework Code Review Monitoring |
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
.agents/scripts/supervisor/ai-actions.sh (1)
375-379: Docstring misstates the output format — minor documentation inconsistency.The header says
"task_id|title_excerpt"(2 fields) but the actual output (Line 437) istask_id|match_count|excerpt(3 fields). Callers already usecut -d'|' -f3-correctly, but the doc will mislead future readers.📝 Proposed fix
-# Newline-separated list of "task_id|title_excerpt" for candidates +# Newline-separated list of "task_id|match_count|excerpt" for candidates (sorted by match_count desc, top 5)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.agents/scripts/supervisor/ai-actions.sh around lines 375 - 379, The docstring/header currently says the output is "task_id|title_excerpt" but the actual echo prints three pipe-separated fields (task_id, match_count, excerpt) using variables like match_count and excerpt; update the header to document the correct format (e.g., "task_id|match_count|title_excerpt" or similar) so it matches the echo that emits task_id|${match_count}|${excerpt} and callers that use cut -d'|' -f3-.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.agents/scripts/supervisor/ai-actions.sh:
- Around line 493-497: resolve_model always returns 0 and ignores the second
parameter, so the current "|| { log_warn ...; return 1; }" branch is dead and
the extra "$ai_cli" arg is unused; change the call to ai_model=$(resolve_model
"sonnet") (drop "$ai_cli"), then explicitly check the captured ai_model (e.g. if
[ -z "$ai_model" ]; then log_warn "AI Actions: semantic dedup AI check skipped —
sonnet model unavailable"; return 1; fi) so the script correctly handles a
missing model; reference resolve_model, ai_model, ai_cli and log_warn when
making the change.
- Around line 592-614: The current branch returns immediately when AI is enabled
but _ai_semantic_dedup_check fails, which skips the intended keyword-only
fallback; change the logic in the AI block so that if ai_result is set (success)
you print and return 0, but if _ai_semantic_dedup_check exits non‑zero (AI
unavailable/unreachable) you do NOT return 1 — instead log that AI was
unavailable (mention AI_SEMANTIC_DEDUP_USE_AI and ai_result) and let execution
fall through to the existing keyword-only fallback that uses candidates /
best_id / best_count; this ensures the 3+ keyword guard runs when the AI CLI is
down.
- Around line 519-537: Replace the fragile default-format + ANSI-strip + grep
JSON extraction with a robust JSON path: when calling opencode (check ai_cli and
the opencode invocation that sets ai_result using portable_timeout, ai_timeout
and ai_model), use --format json and pipe its output through jq to extract the
object (instead of the sed+grep sequence), and similarly for the claude branch
(claude_model/ai_result). Then remove the brittle json_block grep logic and
instead set json_block by parsing ai_result with jq (or by collapsing multiline
JSON and piping to jq) so pretty-printed responses are handled reliably.
---
Nitpick comments:
In @.agents/scripts/supervisor/ai-actions.sh:
- Around line 375-379: The docstring/header currently says the output is
"task_id|title_excerpt" but the actual echo prints three pipe-separated fields
(task_id, match_count, excerpt) using variables like match_count and excerpt;
update the header to document the correct format (e.g.,
"task_id|match_count|title_excerpt" or similar) so it matches the echo that
emits task_id|${match_count}|${excerpt} and callers that use cut -d'|' -f3-.
| local ai_model | ||
| ai_model=$(resolve_model "sonnet" "$ai_cli" 2>/dev/null) || { | ||
| log_warn "AI Actions: semantic dedup AI check skipped — sonnet model unavailable" | ||
| return 1 | ||
| } |
There was a problem hiding this comment.
Dead error handler — resolve_model always returns 0.
Per ai-reason.sh, resolve_model unconditionally returns 0 for any input (all case branches end with return 0). The || { log_warn "sonnet model unavailable"; return 1; } branch is unreachable dead code. The warning will never be emitted; if the model string is ever wrong, the failure surfaces only at the AI CLI invocation level. The extra $ai_cli argument passed to resolve_model is also silently ignored since the function only accepts $1.
🔧 Proposed fix
- local ai_model
- ai_model=$(resolve_model "sonnet" "$ai_cli" 2>/dev/null) || {
- log_warn "AI Actions: semantic dedup AI check skipped — sonnet model unavailable"
- return 1
- }
+ local ai_model
+ ai_model=$(resolve_model "sonnet")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| local ai_model | |
| ai_model=$(resolve_model "sonnet" "$ai_cli" 2>/dev/null) || { | |
| log_warn "AI Actions: semantic dedup AI check skipped — sonnet model unavailable" | |
| return 1 | |
| } | |
| local ai_model | |
| ai_model=$(resolve_model "sonnet") |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.agents/scripts/supervisor/ai-actions.sh around lines 493 - 497,
resolve_model always returns 0 and ignores the second parameter, so the current
"|| { log_warn ...; return 1; }" branch is dead and the extra "$ai_cli" arg is
unused; change the call to ai_model=$(resolve_model "sonnet") (drop "$ai_cli"),
then explicitly check the captured ai_model (e.g. if [ -z "$ai_model" ]; then
log_warn "AI Actions: semantic dedup AI check skipped — sonnet model
unavailable"; return 1; fi) so the script correctly handles a missing model;
reference resolve_model, ai_model, ai_cli and log_warn when making the change.
| if [[ "$ai_cli" == "opencode" ]]; then | ||
| ai_result=$(portable_timeout "$ai_timeout" opencode run \ | ||
| -m "$ai_model" \ | ||
| --format default \ | ||
| --title "dedup-check-$$" \ | ||
| "$prompt" 2>/dev/null || echo "") | ||
| # Strip ANSI escape codes | ||
| ai_result=$(printf '%s' "$ai_result" | sed 's/\x1b\[[0-9;]*[mGKHF]//g; s/\x1b\[[0-9;]*[A-Za-z]//g; s/\x1b\]//g; s/\x07//g') | ||
| else | ||
| local claude_model="${ai_model#*/}" | ||
| ai_result=$(portable_timeout "$ai_timeout" claude \ | ||
| -p "$prompt" \ | ||
| --model "$claude_model" \ | ||
| --output-format text 2>/dev/null || echo "") | ||
| fi | ||
|
|
||
| # Parse the response — extract JSON from potentially noisy output | ||
| local json_block="" | ||
| json_block=$(printf '%s' "$ai_result" | grep -oE '\{[^}]+\}' | head -1) |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Find all opencode invocations in shell scripts
echo "=== All opencode invocations ==="
rg -n 'opencode' -A 4 -B 1 --type shRepository: marcusquinn/aidevops
Length of output: 50379
🏁 Script executed:
#!/bin/bash
# Also check for any opencode documentation or config files
echo "=== Files mentioning opencode ==="
rg -l 'opencode' --type sh --type txt --type md
echo ""
echo "=== Any opencode help or docs ==="
find . -type f \( -name "*opencode*" -o -name "*openai*" \) 2>/dev/null | head -20Repository: marcusquinn/aidevops
Length of output: 6684
🏁 Script executed:
#!/bin/bash
# Check for JSON extraction patterns elsewhere in the codebase
echo "=== JSON extraction patterns in shell scripts ==="
rg -n 'grep.*\{' --type sh -A 2 -B 2Repository: marcusquinn/aidevops
Length of output: 50377
🏁 Script executed:
#!/bin/bash
# Check the actual code at the specified lines
echo "=== ai-actions.sh lines 519-537 ==="
sed -n '515,540p' .agents/scripts/supervisor/ai-actions.shRepository: marcusquinn/aidevops
Length of output: 1036
🏁 Script executed:
#!/bin/bash
# Search for --format default usage
echo "=== Uses of --format default ==="
rg -n '\-\-format\s+default' --type sh
echo ""
echo "=== Uses of --format json ==="
rg -n '\-\-format\s+json' --type sh | head -20Repository: marcusquinn/aidevops
Length of output: 2716
🏁 Script executed:
#!/bin/bash
# Check for opencode documentation files
echo "=== Check opencode.md documentation ==="
if [[ -f ".agents/tools/opencode/opencode.md" ]]; then
head -100 ".agents/tools/opencode/opencode.md"
fiRepository: marcusquinn/aidevops
Length of output: 2924
🏁 Script executed:
#!/bin/bash
# Look for any test/example patterns of opencode run with format flag
echo "=== All opencode run patterns ==="
rg 'opencode\s+run' --type sh -A 3 | head -60Repository: marcusquinn/aidevops
Length of output: 3702
Use --format json with jq for robust JSON extraction, not --format default with grep.
Two issues with the current approach:
1. Suboptimal flag choice (Lines 519–524). The code uses --format default (standard OpenCode flag, not undocumented) with ANSI stripping and then grep -oE '\{[^}]+\}' to extract JSON. This works for single-line JSON but fails on pretty-printed responses. Elsewhere in the codebase (batch-cleanup-helper.sh, ai-reason.sh, agent-test-helper.sh), the pattern is to use --format json with jq, which is more robust.
2. grep pattern fragility (Line 537). The regex \{[^}]+\} requires the entire JSON object to sit on a single line. If Claude returns pretty-printed JSON (with newlines inside the braces), the extraction silently fails, returning empty, and the function falls back to keyword-only dedup.
🔧 Proposed fix
- ai_result=$(portable_timeout "$ai_timeout" opencode run \
- -m "$ai_model" \
- --format default \
- --title "dedup-check-$$" \
- "$prompt" 2>/dev/null || echo "")
- # Strip ANSI escape codes
- ai_result=$(printf '%s' "$ai_result" | sed 's/\x1b\[[0-9;]*[mGKHF]//g; s/\x1b\[[0-9;]*[A-Za-z]//g; s/\x1b\]//g; s/\x07//g')
+ ai_result=$(portable_timeout "$ai_timeout" opencode run \
+ -m "$ai_model" \
+ --format json \
+ --title "dedup-check-$$" \
+ "$prompt" 2>/dev/null || echo "")
# Parse the response — extract JSON from potentially noisy output
local json_block=""
- json_block=$(printf '%s' "$ai_result" | grep -oE '\{[^}]+\}' | head -1)
+ json_block=$(printf '%s' "$ai_result" | jq -r '.content[] | select(.type=="text") | .text' 2>/dev/null | grep -oE '\{[^}]+\}' | head -1)Or, matching the pattern from ai-reason.sh, collapse multi-line JSON before extraction if sticking with --format default:
- json_block=$(printf '%s' "$ai_result" | grep -oE '\{[^}]+\}' | head -1)
+ json_block=$(printf '%s' "$ai_result" | tr '\n' ' ' | grep -oE '\{[^}]+\}' | head -1)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.agents/scripts/supervisor/ai-actions.sh around lines 519 - 537, Replace the
fragile default-format + ANSI-strip + grep JSON extraction with a robust JSON
path: when calling opencode (check ai_cli and the opencode invocation that sets
ai_result using portable_timeout, ai_timeout and ai_model), use --format json
and pipe its output through jq to extract the object (instead of the sed+grep
sequence), and similarly for the claude branch (claude_model/ai_result). Then
remove the brittle json_block grep logic and instead set json_block by parsing
ai_result with jq (or by collapsing multiline JSON and piping to jq) so
pretty-printed responses are handled reliably.
| # Step 2: AI semantic check (if enabled and CLI available) | ||
| if [[ "${AI_SEMANTIC_DEDUP_USE_AI:-true}" == "true" ]]; then | ||
| local ai_result | ||
| if ai_result=$(_ai_semantic_dedup_check "$title" "$candidates"); then | ||
| printf '%s' "$ai_result" | ||
| return 0 | ||
| fi | ||
| # AI said not a duplicate or was unavailable — trust the AI over keywords | ||
| log_info "AI Actions: semantic dedup: AI did not confirm duplicate, allowing task creation" | ||
| return 1 | ||
| fi | ||
|
|
||
| # Fallback: keyword-only mode (AI disabled) — require higher threshold | ||
| local best_id best_count | ||
| best_id=$(printf '%s' "$candidates" | head -1 | cut -d'|' -f1) | ||
| best_count=$(printf '%s' "$candidates" | head -1 | cut -d'|' -f2) | ||
|
|
||
| # Without AI confirmation, require 3+ keyword matches (stricter) | ||
| if [[ -n "$best_id" && "$best_count" -ge 3 ]]; then | ||
| log_info "AI Actions: semantic dedup (keyword-only fallback): $best_id matches with $best_count keywords" | ||
| printf '%s' "$best_id" | ||
| return 0 | ||
| fi |
There was a problem hiding this comment.
AI-unavailable path skips keyword fallback — contradicts stated behavior.
When AI_SEMANTIC_DEDUP_USE_AI=true and the AI CLI is unreachable, _ai_semantic_dedup_check returns 1, the outer if on line 595 fails, and the function immediately returns 1 at line 601 — allowing task creation with zero dedup protection, even though keyword candidates were found.
The keyword-only fallback block at lines 604–614 is only reachable when AI_SEMANTIC_DEDUP_USE_AI=false. The PR description explicitly states "Falls back to keyword-only mode (3+ match threshold) if AI is unavailable or disabled", but the current code only falls back if disabled. An AI CLI outage effectively disables dedup entirely.
If the intended behavior is to always use the keyword guard as a last line of defence when AI is unavailable, consider:
🔧 Proposed fix to honour the stated fallback contract
# Step 2: AI semantic check (if enabled and CLI available)
if [[ "${AI_SEMANTIC_DEDUP_USE_AI:-true}" == "true" ]]; then
local ai_result
if ai_result=$(_ai_semantic_dedup_check "$title" "$candidates"); then
printf '%s' "$ai_result"
return 0
fi
- # AI said not a duplicate or was unavailable — trust the AI over keywords
- log_info "AI Actions: semantic dedup: AI did not confirm duplicate, allowing task creation"
- return 1
+ # AI said not a duplicate — trust the AI verdict
+ # (If AI was unavailable, _ai_semantic_dedup_check already logged a warning;
+ # fall through to keyword-only as a safety net)
+ log_info "AI Actions: semantic dedup: AI did not confirm duplicate, checking keyword threshold"
fi
# Fallback: keyword-only mode (AI disabled or unavailable) — require higher threshold📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| # Step 2: AI semantic check (if enabled and CLI available) | |
| if [[ "${AI_SEMANTIC_DEDUP_USE_AI:-true}" == "true" ]]; then | |
| local ai_result | |
| if ai_result=$(_ai_semantic_dedup_check "$title" "$candidates"); then | |
| printf '%s' "$ai_result" | |
| return 0 | |
| fi | |
| # AI said not a duplicate or was unavailable — trust the AI over keywords | |
| log_info "AI Actions: semantic dedup: AI did not confirm duplicate, allowing task creation" | |
| return 1 | |
| fi | |
| # Fallback: keyword-only mode (AI disabled) — require higher threshold | |
| local best_id best_count | |
| best_id=$(printf '%s' "$candidates" | head -1 | cut -d'|' -f1) | |
| best_count=$(printf '%s' "$candidates" | head -1 | cut -d'|' -f2) | |
| # Without AI confirmation, require 3+ keyword matches (stricter) | |
| if [[ -n "$best_id" && "$best_count" -ge 3 ]]; then | |
| log_info "AI Actions: semantic dedup (keyword-only fallback): $best_id matches with $best_count keywords" | |
| printf '%s' "$best_id" | |
| return 0 | |
| fi | |
| # Step 2: AI semantic check (if enabled and CLI available) | |
| if [[ "${AI_SEMANTIC_DEDUP_USE_AI:-true}" == "true" ]]; then | |
| local ai_result | |
| if ai_result=$(_ai_semantic_dedup_check "$title" "$candidates"); then | |
| printf '%s' "$ai_result" | |
| return 0 | |
| fi | |
| # AI said not a duplicate — trust the AI verdict | |
| # (If AI was unavailable, _ai_semantic_dedup_check already logged a warning; | |
| # fall through to keyword-only as a safety net) | |
| log_info "AI Actions: semantic dedup: AI did not confirm duplicate, checking keyword threshold" | |
| fi | |
| # Fallback: keyword-only mode (AI disabled or unavailable) — require higher threshold | |
| local best_id best_count | |
| best_id=$(printf '%s' "$candidates" | head -1 | cut -d'|' -f1) | |
| best_count=$(printf '%s' "$candidates" | head -1 | cut -d'|' -f2) | |
| # Without AI confirmation, require 3+ keyword matches (stricter) | |
| if [[ -n "$best_id" && "$best_count" -ge 3 ]]; then | |
| log_info "AI Actions: semantic dedup (keyword-only fallback): $best_id matches with $best_count keywords" | |
| printf '%s' "$best_id" | |
| return 0 | |
| fi |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.agents/scripts/supervisor/ai-actions.sh around lines 592 - 614, The current
branch returns immediately when AI is enabled but _ai_semantic_dedup_check
fails, which skips the intended keyword-only fallback; change the logic in the
AI block so that if ai_result is set (success) you print and return 0, but if
_ai_semantic_dedup_check exits non‑zero (AI unavailable/unreachable) you do NOT
return 1 — instead log that AI was unavailable (mention AI_SEMANTIC_DEDUP_USE_AI
and ai_result) and let execution fall through to the existing keyword-only
fallback that uses candidates / best_id / best_count; this ensures the 3+
keyword guard runs when the AI CLI is down.
1 participant
Summary
Architecture
_keyword_prefilter_open_tasks()— Fast, free keyword scan. Extracts distinctive words from proposed title, counts overlap with open tasks. Returns top 5 candidates with 2+ keyword matches._ai_semantic_dedup_check()— Calls sonnet viaresolve_ai_cli()+resolve_model("sonnet"). Sends focused prompt with proposed title + candidate list. AI returns{"duplicate": true, "existing_task": "tXXXX", "confidence": "high|medium"}or{"duplicate": false}. 30s timeout._check_similar_open_task()— Orchestrator: runs keyword pre-filter first (instant, free), then if candidates found andAI_SEMANTIC_DEDUP_USE_AI=true(default), calls sonnet for final verdict.Test Results
t1113duplicate, high confidenceConfig
Cost
~$0.002 per dedup check (only fires when AI proposes task creation, ~5-10 times per pulse cycle max).
Closes #1862
Summary by CodeRabbit