test(e2e): Claude Enterprise cc_tool_accept + cc_tool_acceptance (#1440) - #1509
Conversation
|
Warning Review limit reached
Next review available in: 28 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📥 CommitsReviewing files that changed from the base of the PR and between 3c4ca32d18ac79cec60a7f4bb06b95cabc02a8e1 and 5434af9. 📒 Files selected for processing (7)
📝 WalkthroughWalkthroughAdds the ChangesClaude Enterprise bronze layer and metrics e2e
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related issues
Suggested reviewers
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
src/ingestion/tests/e2e/metrics/ai_cc_tool_acceptance.test.yaml (1)
24-28: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueThe re-sync duplicate does not actually guard the ratio against a dedup regression.
cc_tool_acceptanceis100 × Σaccepted ÷ Σoffered, which is invariant under row duplication (a doubled row scales numerator and denominator equally → 100/160 still 62.5). So ifLIMIT 1 BYregressed, this case would still pass. Dedup is genuinely exercised byai_cc_tool_accept(a sum), so coverage isn't lost — but consider asserting an additional non-ratio field here (e.g., the underlying accepted/offered sum or count) if you want this spec to independently catch a dedup break.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/ingestion/tests/e2e/metrics/ai_cc_tool_acceptance.test.yaml` around lines 24 - 28, The duplicate re-sync row in ai_cc_tool_acceptance does not validate dedup because cc_tool_acceptance is a ratio metric and remains unchanged when rows are duplicated. Update the acceptance test data in ai_cc_tool_acceptance.test.yaml so this case also asserts a non-ratio signal—such as the underlying accepted/offered sum or row count—using the existing templates/claude_enterprise_users.yaml reference and the unique_key/date fields to ensure a LIMIT 1 BY regression would be caught.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/ingestion/tests/e2e/metrics/ai_cc_tool_acceptance.test.yaml`:
- Around line 24-28: The duplicate re-sync row in ai_cc_tool_acceptance does not
validate dedup because cc_tool_acceptance is a ratio metric and remains
unchanged when rows are duplicated. Update the acceptance test data in
ai_cc_tool_acceptance.test.yaml so this case also asserts a non-ratio
signal—such as the underlying accepted/offered sum or row count—using the
existing templates/claude_enterprise_users.yaml reference and the
unique_key/date fields to ensure a LIMIT 1 BY regression would be caught.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 97b83e7f-7768-4d0e-84e0-30508147f522
📥 Commits
Reviewing files that changed from the base of the PR and between 5d8a577 and 3c4ca32d18ac79cec60a7f4bb06b95cabc02a8e1.
📒 Files selected for processing (7)
src/ingestion/connectors/ai/claude-enterprise/dbt/claude_enterprise__ai_dev_usage.sqlsrc/ingestion/scripts/create-bronze-placeholders.shsrc/ingestion/tests/e2e/conftest.pysrc/ingestion/tests/e2e/metrics/ai_cc_tool_accept.test.yamlsrc/ingestion/tests/e2e/metrics/ai_cc_tool_acceptance.test.yamlsrc/ingestion/tests/e2e/metrics/schemas/bronze_claude_enterprise.claude_enterprise_users.yamlsrc/ingestion/tests/e2e/metrics/templates/claude_enterprise_users.yaml
The last two AI-bullet metrics that claude_team can't supply (it sets
tool_use_accepted/offered = NULL): both come from the Claude Enterprise admin
Analytics API, where tool_use_accepted ← code_tool_accepted_count and
tool_use_offered ← code_tool_accepted_count + code_tool_rejected_count.
- cc_tool_accept — sum metric; alice 50 · bob 20 · carol 10 → median 20,
range [10, 50] (dedup holds).
- cc_tool_acceptance — reconstructed ratio 100 × Σ accepted ÷ Σ offered;
alice 50 of 80 → 62.5 (value-only).
Pipeline: bronze_claude_enterprise.claude_enterprise_users →
claude_enterprise__ai_dev_usage (tool='claude_code') → silver class_ai_dev_usage
→ gold insight.ai_bullet_rows → IC Bullet AI (…0013).
Supporting:
- bronze placeholder for claude_enterprise_users (+ schema + template). Note:
user_id MUST be distinct per seat — the staging model dedups with
`LIMIT 1 BY tenant, source, user_id, date`, so a shared/NULL user_id collapses
the cohort to one row and the team median is wrong.
- claude_enterprise__ai_dev_usage: same empty-table watermark guard as the other
AI staging models (INTERVAL 3 DAY underflows over an empty `this`).
- conftest: reset staging.claude_enterprise__ai_dev_usage at session start.
Verified locally: 2 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Sergei Mozhaev <mozhaev.dev@gmail.com>
3c4ca32 to
e2d5e36
Compare
2 participants
What
The final two AI-Adoption bullet metrics for #1440 —
cc_tool_acceptandcc_tool_acceptance. These are the Claude Code tool-suggestion accept count and acceptance rate, which claude_team cannot supply (it setstool_use_accepted/tool_use_offeredto NULL). They come from the Claude Enterprise admin Analytics API instead.ai_cc_tool_acceptcc_tool_accepttool_use_accepted ← code_tool_accepted_countai_cc_tool_acceptancecc_tool_acceptance100 × Σ accepted ÷ Σ offered,offered ← accepted + rejectedPipeline:
bronze_claude_enterprise.claude_enterprise_users→claude_enterprise__ai_dev_usage(tool='claude_code') → silverclass_ai_dev_usage→ goldinsight.ai_bullet_rows→ IC Bullet AI (…0013).Values:
cc_tool_acceptalice 50 · bob 20 · carol 10 → value 50, median 20, range [10, 50] (dedup holds).cc_tool_acceptancealice accepted 50 of 80 offered → 62.5 (value-only).Supporting changes
claude_enterprise_users(+ schema + template, mirroring the connector InlineSchemaLoader).user_idis set distinct per seat in the template: the staging model dedups withLIMIT 1 BY tenant, source, user_id, date, so a shared/NULLuser_idcollapses the whole cohort to one row and the team median is wrong (caught in testing).claude_enterprise__ai_dev_usage: the same empty-table watermark guard as the other AI staging models —… - INTERVAL 3 DAYunderflows the Date range over an emptythis(the rig resets staging between tests), filtering out every row. Short-circuited with(SELECT count() FROM this) = 0 OR ….staging.claude_enterprise__ai_dev_usageat session start (warm-rerun determinism).Coverage
This completes the AI-Adoption bullet (#1440) — all 23 served metric_keys now have an e2e spec across the cursor (#1468), claude_team (#1491), chatgpt_team (#1501), and this Claude Enterprise PR.
Verification
./e2e.sh test -k ai_cc_tool→ 2 passed.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes