M1/decision relevance ruler

[jinhongkuan]

Summary by CodeRabbit

• New Features

  • Ingest now reports grounding metrics (grounded count and grounded percentage) and supports per-source grounding breakdowns
  • End-to-end decision-relevance evaluation and a headless extraction workflow for measuring extraction + grounding

• Documentation

  • Added README and schema guidance for ground-truth extraction fixtures

• Tests

  • Large suite of tests and fixtures for extraction, matching, metrics, and grounding

• Chores

  • Fixture regeneration utility for extraction fixtures

[coderabbitai]

Warning

Rate limit exceeded

@jinhongkuan has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 49 minutes and 21 seconds before requesting another review.

Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 49 minutes and 21 seconds.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 036641bd-e291-42db-81d9-fdeafc104ec4

📥 Commits

Reviewing files that changed from the base of the PR and between 0c1dac9 and cfbf490.

📒 Files selected for processing (1)

• tests/fixtures/expected/decisions.py

📝 Walkthrough

Walkthrough

ValidateSymbolsTool now retains the DB instance; ingest handler computes and logs grounded counts and percentage and populates new IngestStats fields; ledger gains grounding breakdown query; a headless extraction driver, LLM/rapidfuzz matchers, metrics, fixtures, regeneration tooling, and evaluation runner with tests were added.

Changes

Cohort / File(s)	Summary
Validate & Ingest code_locator/tools/validate_symbols.py, contracts.py, handlers/ingest.py	ValidateSymbolsTool now stores the provided db on self._db. IngestStats gains grounded and grounded_pct. handle_ingest computes grounded/grounded_pct, clamps values, logs [ingest] complete, and populates the new stats fields.
Ledger Query ledger/queries.py	Added get_grounding_breakdown(...) async helper that aggregates intents by source_ref (grounded/ungrounded/total) and computes grounded_pct; supports optional source_type/source_scope filters.
Headless Extraction Driver tests/_extract_headless.py, tests/test_extract_headless.py	New headless extractor that extracts Step 1 from SKILL.md, forces Anthropic tool-based extraction, caches results, and includes parsing, cache, and error-handling tests.
Extraction Matching & Metrics tests/_extraction_matcher.py, tests/_extraction_metrics.py, tests/test_extraction_metrics.py	Added LLM-backed 1:1 matcher with cache, rapidfuzz fallback, deterministic parsing, compute_extraction_metrics and aggregate_extraction_metrics, plus unit tests covering matchers and metrics.
Evaluation Runner & Regression Tests tests/eval_decision_relevance.py, tests/test_phase2_ledger.py	New CLI evaluation runner orchestrating extract→ingest→metrics across repos/transcripts, aggregates per-repo/overall grounding and extraction metrics, enforces regression gates; tests assert grounding stats and ledger query behavior.
Fixture Registry & Fixtures tests/fixtures/expected/decisions.py, tests/fixtures/extraction/README.md, tests/fixtures/extraction/*.json	Added TRANSCRIPT_SOURCES mapping, documentation, and numerous M1 extraction JSON fixtures with metadata and decisions/action_items.
Fixture Regeneration Tooling tests/regen_extraction_fixtures.py	Added script to regenerate extraction fixtures via the headless extractor with CLI options (--all, --source-ref, --model, --force, --dry-run).
Test Harness Additions tests/test_extract_headless.py, tests/test_extraction_metrics.py, tests/test_phase2_ledger.py	Extensive tests for extractor parsing, cache behavior, matcher selection, metric aggregation, and grounding stats assertions.

Sequence Diagram(s)

sequenceDiagram
    participant Handler as Ingest Handler
    participant Tool as ValidateSymbolsTool
    participant DB as Database
    participant Ledger as Ledger Queries

    Handler->>+Handler: handle_ingest(payload)
    Handler->>+Tool: validate_symbols(...)
    Tool->>+DB: lookup_by_file() via self._db
    DB-->>-Tool: symbol results
    Tool-->>-Handler: validation results

    Handler->>Handler: compute intents_created, ungrounded
    Handler->>Handler: grounded = max(0, intents_created - ungrounded)
    Handler->>Handler: grounded_pct = grounded/intents_created or 0.0
    Handler->>+Ledger: persist/return IngestStats (including grounded, grounded_pct)
    Ledger-->>-Handler: stats returned
    Handler-->>-Handler: return result

Loading

sequenceDiagram
    participant Eval as Eval Runner
    participant Extract as Headless Extractor
    participant API as Anthropic API
    participant Ingest as Ingest Handler
    participant Metrics as Metrics Computer

    Eval->>+Eval: load transcripts & plan
    loop per transcript
        Eval->>+Extract: extract_from_current_skill(transcript)
        Extract->>Extract: check cache
        alt cache hit
            Extract-->>Eval: return cached decisions
        else cache miss
            Extract->>+API: send tool-based extraction request
            API-->>-Extract: tool_use response
            Extract->>Extract: parse & cache
            Extract-->>Eval: return extracted decisions
        end

        Eval->>+Ingest: handle_ingest(payload)
        Ingest-->>-Eval: return grounding stats

        Eval->>+Metrics: compute_extraction_metrics(extracted, fixture)
        Metrics-->>-Eval: return precision/recall/F1
    end
    Eval->>Eval: aggregate per-repo & overall metrics
    Eval->>Eval: enforce regression gates -> exit code

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~65 minutes

Possibly related PRs

• refactor: BicameralContext — request-scoped snapshot isolation #4 — Related refactor touching grounding and code-locator adapter flows; aligns with retaining ValidateSymbolsTool._db to enable DB lookups during auto-grounding.

Poem

🐰 I nibbled the fixtures, found carrots of truth,

Hopped through MD steps and cached each sweet tooth,
Grounded the intents, watched percentages bloom,
Metrics and tests guard the warren and room,
A rabbit applauds: tidy, robust, and sleuth.

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 51.02% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check	❓ Inconclusive	The title 'M1/decision relevance ruler' is vague and does not clearly summarize the main changes in the pull request.	Consider a more descriptive title that captures the primary objective, such as 'Add M1 decision-relevance evaluation framework with grounding metrics and extraction validation' or 'Implement extraction-based decision grounding evaluation system'.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch m1/decision-relevance-ruler

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 5

🧹 Nitpick comments (1)

tests/_extract_headless.py (1)

166-169: Chain the exception for better traceability.

When re-raising as a new exception type, use from exc to preserve the original traceback chain. This helps with debugging when JSON parsing fails.

Proposed fix

     try:
         parsed = json.loads(text)
     except json.JSONDecodeError as exc:
-        raise ValueError(f"extractor did not return valid JSON: {exc}\nbody={body[:500]}")
+        raise ValueError(f"extractor did not return valid JSON: {exc}\nbody={body[:500]}") from exc

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/_extract_headless.py` around lines 166 - 169, The except block around
json.loads(text) currently raises a new ValueError without chaining the original
json.JSONDecodeError; update the except handler for json.JSONDecodeError (the
block catching exc) to re-raise the ValueError using "from exc" so the original
traceback is preserved (i.e., change the raise in the try/except that follows
json.loads(text) to use exception chaining).

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@ledger/queries.py`:
- Around line 571-580: The code currently treats any non-"ungrounded" status
(including empty or missing status) as grounded; update the logic in the block
that obtains status and updates the bucket (the variables status,
buckets.setdefault, and keys "grounded"/"ungrounded"/"total") so that only an
explicit "grounded" value increments b["grounded"], and all other cases
(including empty string or missing status) increment b["ungrounded"] (or are
handled as ungrounded) before updating totals and grounded_pct.

In `@tests/_extraction_metrics.py`:
- Around line 88-103: The current greedy loop matching items from actual to
unmatched_expected (variables actual, unmatched_expected, matched) using
fuzz.token_set_ratio and popping the best per iteration causes order-dependent
results; replace this with an order-independent max-cardinality / max-weight
bipartite matching: build a weight matrix of scores between each actual and each
expected (using fuzz.token_set_ratio and MATCH_THRESHOLD), run a max-weight
matching (e.g., Hungarian algorithm or
networkx.algorithms.matching.max_weight_matching) to select globally optimal
pairs, then populate matched, unmatched_actual, and unmatched_expected from that
matching instead of popping inside the greedy loop. Ensure ties below
MATCH_THRESHOLD are treated as non-matches.

In `@tests/eval_decision_relevance.py`:
- Around line 244-248: Wrap the json.loads(args.multi_repo) call in a try/except
to catch json.JSONDecodeError so a malformed --multi-repo value produces a clean
CLI error instead of a traceback; specifically, when parsing into repo_map (the
repo_map: dict[str, str] = json.loads(args.multi_repo) line), catch
json.JSONDecodeError, print a descriptive message to stderr (e.g. "error:
invalid --multi-repo JSON") and return the same error tuple ({}, 2) as the empty
case.

In `@tests/fixtures/extraction/saleor-checkout-extensibility.json`:
- Around line 50-51: Replace the ambiguous empty string owner value with a
proper null to represent an unassigned owner: find the JSON object containing
the "text" value "Deliver base implementation for review by end of sprint;
check-in Thursday on the plugin changes (critical path)" and set its "owner"
property from "" to null so consumers don't need extra normalization logic.

In `@tests/fixtures/extraction/vendure-custom-fields.json`:
- Around line 49-50: Change the JSON fixture entry where the object has "text":
"All PRs reviewed by Wednesday" and "owner": "" so that the owner field is
normalized to null; specifically replace the empty-string value for the "owner"
property with a JSON null to ensure schema consistency across fixtures.

---

Nitpick comments:
In `@tests/_extract_headless.py`:
- Around line 166-169: The except block around json.loads(text) currently raises
a new ValueError without chaining the original json.JSONDecodeError; update the
except handler for json.JSONDecodeError (the block catching exc) to re-raise the
ValueError using "from exc" so the original traceback is preserved (i.e., change
the raise in the try/except that follows json.loads(text) to use exception
chaining).

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 443dbd1e-4916-4f3c-be13-116bede51987

📥 Commits

Reviewing files that changed from the base of the PR and between 4d46607 and a23dba3.

📒 Files selected for processing (22)

• code_locator/tools/validate_symbols.py
• contracts.py
• handlers/ingest.py
• ledger/queries.py
• tests/_extract_headless.py
• tests/_extraction_metrics.py
• tests/eval_decision_relevance.py
• tests/fixtures/expected/decisions.py
• tests/fixtures/extraction/README.md
• tests/fixtures/extraction/medusa-payment-timeout.json
• tests/fixtures/extraction/medusa-plugin-migration.json
• tests/fixtures/extraction/medusa-webhook-notifications.json
• tests/fixtures/extraction/saleor-checkout-extensibility.json
• tests/fixtures/extraction/saleor-graphql-permissions.json
• tests/fixtures/extraction/saleor-order-workflows.json
• tests/fixtures/extraction/vendure-channel-pricing.json
• tests/fixtures/extraction/vendure-custom-fields.json
• tests/fixtures/extraction/vendure-search-reindexing.json
• tests/regen_extraction_fixtures.py
• tests/test_extract_headless.py
• tests/test_extraction_metrics.py
• tests/test_phase2_ledger.py

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Do not treat empty/missing status as grounded.

At Line 571-580, status == "" (or missing) currently increments grounded, which can overstate grounded_pct and skew eval outcomes.

Suggested fix

-        status = str(row.get("status") or "")
+        status = str(row.get("status") or "")
         b = buckets.setdefault(
             ref,
             {"source_ref": ref, "grounded": 0, "ungrounded": 0, "total": 0, "grounded_pct": 0.0},
         )
         b["total"] += 1
-        if status == "ungrounded":
+        if not status or status == "ungrounded":
             b["ungrounded"] += 1
         else:
             b["grounded"] += 1

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@ledger/queries.py` around lines 571 - 580, The code currently treats any
non-"ungrounded" status (including empty or missing status) as grounded; update
the logic in the block that obtains status and updates the bucket (the variables
status, buckets.setdefault, and keys "grounded"/"ungrounded"/"total") so that
only an explicit "grounded" value increments b["grounded"], and all other cases
(including empty string or missing status) increment b["ungrounded"] (or are
handled as ungrounded) before updating totals and grounded_pct.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Order-dependent greedy matching can mis-score TP/FP/FN.

At Line 88, matching extracted decisions sequentially and popping matches at Line 102 makes outcomes dependent on input order. For a regression gate, this can undercount true positives on equivalent outputs.

Proposed fix (order-independent max-cardinality matching)

 def compute_extraction_metrics(
@@
-    unmatched_expected = list(expected)
-    matched: list[tuple[str, str, float]] = []
-    unmatched_actual: list[str] = []
-
-    for a in actual:
-        if not unmatched_expected:
-            unmatched_actual.append(a)
-            continue
-        # Find best match in remaining expected pool
-        best_i = -1
-        best_score = -1.0
-        for i, e in enumerate(unmatched_expected):
-            score = fuzz.token_set_ratio(a, e)
-            if score > best_score:
-                best_score = score
-                best_i = i
-        if best_score >= MATCH_THRESHOLD:
-            matched.append((a, unmatched_expected[best_i], best_score))
-            unmatched_expected.pop(best_i)
-        else:
-            unmatched_actual.append(a)
+    # Build thresholded bipartite graph (actual -> expected), neighbors sorted by score desc.
+    adj: list[list[int]] = []
+    score_map: dict[tuple[int, int], float] = {}
+    for ai, a in enumerate(actual):
+        candidates: list[tuple[int, float]] = []
+        for ei, e in enumerate(expected):
+            s = fuzz.token_set_ratio(a, e)
+            if s >= MATCH_THRESHOLD:
+                candidates.append((ei, s))
+                score_map[(ai, ei)] = s
+        candidates.sort(key=lambda t: t[1], reverse=True)
+        adj.append([ei for ei, _ in candidates])
+
+    # Maximum-cardinality bipartite matching (Kuhn DFS).
+    match_e_to_a = [-1] * len(expected)
+
+    def _dfs(ai: int, seen: set[int]) -> bool:
+        for ei in adj[ai]:
+            if ei in seen:
+                continue
+            seen.add(ei)
+            prev_ai = match_e_to_a[ei]
+            if prev_ai == -1 or _dfs(prev_ai, seen):
+                match_e_to_a[ei] = ai
+                return True
+        return False
+
+    for ai in range(len(actual)):
+        _dfs(ai, set())
+
+    matched_actual = {ai for ai in match_e_to_a if ai != -1}
+    matched_expected_idx = {ei for ei, ai in enumerate(match_e_to_a) if ai != -1}
+
+    matched: list[tuple[str, str, float]] = []
+    for ei, ai in enumerate(match_e_to_a):
+        if ai != -1:
+            matched.append((actual[ai], expected[ei], score_map[(ai, ei)]))
+
+    unmatched_actual = [a for i, a in enumerate(actual) if i not in matched_actual]
+    unmatched_expected = [e for i, e in enumerate(expected) if i not in matched_expected_idx]

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/_extraction_metrics.py` around lines 88 - 103, The current greedy loop
matching items from actual to unmatched_expected (variables actual,
unmatched_expected, matched) using fuzz.token_set_ratio and popping the best per
iteration causes order-dependent results; replace this with an order-independent
max-cardinality / max-weight bipartite matching: build a weight matrix of scores
between each actual and each expected (using fuzz.token_set_ratio and
MATCH_THRESHOLD), run a max-weight matching (e.g., Hungarian algorithm or
networkx.algorithms.matching.max_weight_matching) to select globally optimal
pairs, then populate matched, unmatched_actual, and unmatched_expected from that
matching instead of popping inside the greedy loop. Ensure ties below
MATCH_THRESHOLD are treated as non-matches.

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Handle malformed --multi-repo JSON with a clean CLI error.

Line 244 can throw json.JSONDecodeError and produce a traceback instead of a controlled exit path.

Proposed fix

-    repo_map: dict[str, str] = json.loads(args.multi_repo) if args.multi_repo else {}
+    try:
+        repo_map: dict[str, str] = json.loads(args.multi_repo) if args.multi_repo else {}
+    except json.JSONDecodeError as exc:
+        print(f"error: --multi-repo must be valid JSON: {exc}", file=sys.stderr)
+        return {}, 2

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/eval_decision_relevance.py` around lines 244 - 248, Wrap the
json.loads(args.multi_repo) call in a try/except to catch json.JSONDecodeError
so a malformed --multi-repo value produces a clean CLI error instead of a
traceback; specifically, when parsing into repo_map (the repo_map: dict[str,
str] = json.loads(args.multi_repo) line), catch json.JSONDecodeError, print a
descriptive message to stderr (e.g. "error: invalid --multi-repo JSON") and
return the same error tuple ({}, 2) as the empty case.

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Use null instead of empty string for unassigned owner.

At Line 50-51, owner: "" is ambiguous and forces extra normalization logic. Use null for missing owner values.

Suggested fix

-      "owner": ""
+      "owner": null

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	"text": "Deliver base implementation for review by end of sprint; check-in Thursday on the plugin changes (critical path)",
	"owner": ""
	"text": "Deliver base implementation for review by end of sprint; check-in Thursday on the plugin changes (critical path)",
	"owner": null

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/fixtures/extraction/saleor-checkout-extensibility.json` around lines 50
- 51, Replace the ambiguous empty string owner value with a proper null to
represent an unassigned owner: find the JSON object containing the "text" value
"Deliver base implementation for review by end of sprint; check-in Thursday on
the plugin changes (critical path)" and set its "owner" property from "" to null
so consumers don't need extra normalization logic.

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Normalize missing owner as null.

At Line 49-50, prefer owner: null over owner: "" for schema consistency across fixtures.

Suggested fix

-      "owner": ""
+      "owner": null

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	"text": "All PRs reviewed by Wednesday",
	"owner": ""
	"text": "All PRs reviewed by Wednesday",
	"owner": null

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/fixtures/extraction/vendure-custom-fields.json` around lines 49 - 50,
Change the JSON fixture entry where the object has "text": "All PRs reviewed by
Wednesday" and "owner": "" so that the owner field is normalized to null;
specifically replace the empty-string value for the "owner" property with a JSON
null to ensure schema consistency across fixtures.

[coderabbitai]

Actionable comments posted: 3

♻️ Duplicate comments (1)

tests/_extraction_metrics.py (1)

70-98: ⚠️ Potential issue | 🟠 Major

Rapidfuzz fallback is still order-dependent.

This loop still greedily pops the best remaining expected for each actual, so runs that fall back to rapidfuzz can undercount TP / overcount FP based only on list order. That keeps the offline metric unstable; this path still needs a global 1:1 bipartite matching instead of greedy removal.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/_extraction_metrics.py` around lines 70 - 98, The _rapidfuzz_match
function currently does greedy per-actual selection causing order-dependent
matches; replace the greedy removal with a global 1:1 bipartite matching: build
a score matrix using fuzz.token_set_ratio between every actual (rows) and
expected (columns), run a maximum-weight bipartite matching (e.g., Hungarian
algorithm / scipy.optimize.linear_sum_assignment or a max-weight matching
routine) to get globally optimal actual->expected pairs, then convert matched
pairs to the (actual_idx, expected_idx) list but only accept matches whose score
>= MATCH_THRESHOLD (unmatched or below-threshold become (actual_idx, None));
keep the function signature and return format unchanged.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tests/_extraction_matcher.py`:
- Around line 217-223: The RuntimeError raised after the HTTP POST is echoing
API key-derived data (api_key, key_prefix, key_len); remove any key-derived
fields from the exception text. Update the error construction that references
resp.status_code, resp.text, model, key_prefix, and key_len so it only includes
non-secret diagnostics (e.g., resp.status_code, resp.text[:500], and model) and
does not reference api_key, key_prefix, key_len, or ANTROP... constants.
- Around line 127-136: The cache key generation is ambiguous and missing
prompt/tool versioning: change _list_sha to produce an unambiguous hash of the
list (e.g., include item lengths or a separator encoding so ["a","b\nc"] !=
["a\nb","c"]) and update _cache_path to incorporate SYSTEM_PROMPT,
USER_PROMPT_TEMPLATE and MATCHER_TOOL (or their current version/sha) into the
key string used before calling _sha so llm_match() won't return stale hits when
prompts or tools change; update references to _list_sha and _cache_path
accordingly.

In `@tests/_extraction_metrics.py`:
- Line 51: The comment on the MATCH_THRESHOLD constant contains a Unicode
en-dash; replace the en-dash (–) with a plain ASCII hyphen (-) in the inline
comment after MATCH_THRESHOLD so the comment reads "rapidfuzz token_set_ratio,
0-100 scale" and resolves the RUF003 lint warning.

---

Duplicate comments:
In `@tests/_extraction_metrics.py`:
- Around line 70-98: The _rapidfuzz_match function currently does greedy
per-actual selection causing order-dependent matches; replace the greedy removal
with a global 1:1 bipartite matching: build a score matrix using
fuzz.token_set_ratio between every actual (rows) and expected (columns), run a
maximum-weight bipartite matching (e.g., Hungarian algorithm /
scipy.optimize.linear_sum_assignment or a max-weight matching routine) to get
globally optimal actual->expected pairs, then convert matched pairs to the
(actual_idx, expected_idx) list but only accept matches whose score >=
MATCH_THRESHOLD (unmatched or below-threshold become (actual_idx, None)); keep
the function signature and return format unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 5c255b50-0c6e-45de-b5a9-b1e762779ec6

📥 Commits

Reviewing files that changed from the base of the PR and between a23dba3 and 42c3a07.

📒 Files selected for processing (4)

• tests/_extraction_matcher.py
• tests/_extraction_metrics.py
• tests/eval_decision_relevance.py
• tests/test_extraction_metrics.py

🚧 Files skipped from review as they are similar to previous changes (2)

• tests/test_extraction_metrics.py
• tests/eval_decision_relevance.py

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Make the cache key unambiguous and prompt-versioned.

Line 131 hashes "\n".join(items), so inputs like ["a", "b\nc"] and ["a\nb", "c"] collide. Lines 135-136 also ignore SYSTEM_PROMPT, USER_PROMPT_TEMPLATE, and MATCHER_TOOL, so prompt/tool edits will silently reuse stale judge outputs. Because llm_match() returns cache hits before calling the API, this can produce wrong metrics.

🔧 Proposed fix

 def _sha(text: str) -> str:
     return hashlib.sha256(text.encode("utf-8")).hexdigest()
 
 
+MATCHER_CACHE_VERSION = _sha(
+    json.dumps(
+        {
+            "system": SYSTEM_PROMPT,
+            "user_template": USER_PROMPT_TEMPLATE,
+            "tool": MATCHER_TOOL,
+        },
+        sort_keys=True,
+        ensure_ascii=False,
+        separators=(",", ":"),
+    )
+)
+
+
 def _list_sha(items: list[str]) -> str:
     # Order matters for cache key — same-content-different-order lists
     # should map to different cache entries so we don't silently reuse a
     # stale matching when the runner reshuffles.
-    return _sha("\n".join(items))
+    return _sha(json.dumps(items, ensure_ascii=False, separators=(",", ":")))
 
 
 def _cache_path(model: str, actual_sha: str, expected_sha: str) -> Path:
-    key = f"{model}|{actual_sha}|{expected_sha}"
+    key = f"{MATCHER_CACHE_VERSION}|{model}|{actual_sha}|{expected_sha}"
     return MATCH_CACHE_DIR / f"{_sha(key)}.json"

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/_extraction_matcher.py` around lines 127 - 136, The cache key
generation is ambiguous and missing prompt/tool versioning: change _list_sha to
produce an unambiguous hash of the list (e.g., include item lengths or a
separator encoding so ["a","b\nc"] != ["a\nb","c"]) and update _cache_path to
incorporate SYSTEM_PROMPT, USER_PROMPT_TEMPLATE and MATCHER_TOOL (or their
current version/sha) into the key string used before calling _sha so llm_match()
won't return stale hits when prompts or tools change; update references to
_list_sha and _cache_path accordingly.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Do not echo API key material into the exception text.

Line 223 includes part of ANTHROPIC_API_KEY in every 4xx failure. Even partial credentials should not land in CI logs or artifacts; keep the model/status diagnostics and drop the key-derived fields.

🔒 Proposed fix

-    key_prefix = api_key[:12] if api_key else "(empty)"
     with httpx.Client(timeout=REQUEST_TIMEOUT_S) as client:
         resp = client.post(ANTHROPIC_API_URL, headers=headers, json=payload)
         if resp.status_code >= 400:
             raise RuntimeError(
                 f"Matcher API error {resp.status_code}: {resp.text[:500]} "
-                f"(model={model}, key_prefix={key_prefix}..., key_len={len(api_key)})"
+                f"(model={model})"
             )

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	key_prefix = api_key[:12] if api_key else "(empty)"
	with httpx.Client(timeout=REQUEST_TIMEOUT_S) as client:
	resp = client.post(ANTHROPIC_API_URL, headers=headers, json=payload)
	if resp.status_code >= 400:
	raise RuntimeError(
	f"Matcher API error {resp.status_code}: {resp.text[:500]} "
	f"(model={model}, key_prefix={key_prefix}..., key_len={len(api_key)})"
	with httpx.Client(timeout=REQUEST_TIMEOUT_S) as client:
	resp = client.post(ANTHROPIC_API_URL, headers=headers, json=payload)
	if resp.status_code >= 400:
	raise RuntimeError(
	f"Matcher API error {resp.status_code}: {resp.text[:500]} "
	f"(model={model})"
	)

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/_extraction_matcher.py` around lines 217 - 223, The RuntimeError raised
after the HTTP POST is echoing API key-derived data (api_key, key_prefix,
key_len); remove any key-derived fields from the exception text. Update the
error construction that references resp.status_code, resp.text, model,
key_prefix, and key_len so it only includes non-secret diagnostics (e.g.,
resp.status_code, resp.text[:500], and model) and does not reference api_key,
key_prefix, key_len, or ANTROP... constants.

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Replace the Unicode dash in this comment.

Line 51 uses –, which Ruff flags as RUF003. Swap it for a plain - so the new file stays lint-clean.

🧹 Proposed fix

-MATCH_THRESHOLD = 70  # rapidfuzz token_set_ratio, 0–100 scale
+MATCH_THRESHOLD = 70  # rapidfuzz token_set_ratio, 0-100 scale

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	MATCH_THRESHOLD = 70 # rapidfuzz token_set_ratio, 0–100 scale
	MATCH_THRESHOLD = 70 # rapidfuzz token_set_ratio, 0-100 scale

🧰 Tools

🪛 Ruff (0.15.9)

[warning] 51-51: Comment contains ambiguous – (EN DASH). Did you mean - (HYPHEN-MINUS)?

(RUF003)

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/_extraction_metrics.py` at line 51, The comment on the MATCH_THRESHOLD
constant contains a Unicode en-dash; replace the en-dash (–) with a plain ASCII
hyphen (-) in the inline comment after MATCH_THRESHOLD so the comment reads
"rapidfuzz token_set_ratio, 0-100 scale" and resolves the RUF003 lint warning.