fix(stats): unify success rate formula across admin and public metrics - #2988
Conversation
Public /data used app_log device-day dedup (~70%) while admin used version_usage install/(install+fail). global_stats cron also used the wrong get/(install+get) ratio. Align all paths on install/(install+fail).
📝 WalkthroughWalkthroughThe update metrics now calculate success rates from install/set outcomes and failures. Public live metrics use ChangesUpdate success-rate metrics
Estimated code review effort: 3 (Moderate) | ~25 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
Comment |
Merging this PR will not alter performance
Comparing 
Footnotes
|
|
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Risk: medium. Cursor Bugbot passed with no findings, but this is above the low-risk approval threshold (public/admin success-rate formula change plus a stats SQL migration). Leaving a non-blocking comment and assigning human reviewers.
Sent by Cursor Approval Agent: Pull Request Approver External
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
supabase/functions/_backend/plugin_runtime/utils/cloudflare.ts (1)
2849-2897: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy liftPreserve the public
success_ratecontract.These changes replace device-day outcomes with raw event outcomes while retaining the same public field. Existing clients cannot select the previous semantics.
Keep the legacy field behavior for existing clients. Add a versioned endpoint or a new install-based field for the new calculation.
supabase/functions/_backend/plugin_runtime/utils/cloudflare.ts#L2849-L2897: preserve the legacy public response or gate the install-based calculation by API or plugin version.supabase/functions/_backend/utils/cloudflare.ts#L3098-L3146: apply the same compatibility behavior in the matching deployment target.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@supabase/functions/_backend/plugin_runtime/utils/cloudflare.ts` around lines 2849 - 2897, The public success_rate calculation in the analytics response must retain its legacy device-day outcome semantics for existing clients. In supabase/functions/_backend/plugin_runtime/utils/cloudflare.ts lines 2849-2897, gate the raw install-based calculation by API or plugin version, or expose it under a new install-based field/versioned endpoint while preserving the existing success_rate field. Apply the same compatibility change in supabase/functions/_backend/utils/cloudflare.ts lines 3098-3146.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@supabase/functions/_backend/plugin_runtime/utils/cloudflare.ts`:
- Around line 2849-2897: The public success_rate calculation in the analytics
response must retain its legacy device-day outcome semantics for existing
clients. In supabase/functions/_backend/plugin_runtime/utils/cloudflare.ts lines
2849-2897, gate the raw install-based calculation by API or plugin version, or
expose it under a new install-based field/versioned endpoint while preserving
the existing success_rate field. Apply the same compatibility change in
supabase/functions/_backend/utils/cloudflare.ts lines 3098-3146.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 40cb8e01-ec31-4145-bb72-6a2713076b2c
📒 Files selected for processing (5)
supabase/functions/_backend/plugin_runtime/utils/cloudflare.tssupabase/functions/_backend/utils/cloudflare.tssupabase/functions/_backend/utils/supabase.tssupabase/migrations/20260811083707_fix_update_stats_success_rate_formula.sqltests/public-live-update-metrics.unit.test.ts
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
Cap-go/capacitor-updater(manual)
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
2 issues found across 5 files
Confidence score: 3/5
- In
supabase/functions/_backend/utils/cloudflare.ts, the globalsuccess_ratedenominator is still filtered byapp.get > 0, so install/fail-only slices are dropped and the top-line rate can be artificially high; this can misstate reliability trends and regressions — compute totals without that filter so all outcome types are counted. - In
supabase/functions/_backend/utils/cloudflare.ts, headline success rate and platform/country/version rates are sourced from different Analytics Engine datasets with different failure taxonomies, which can produce conflicting numbers for the same period and erode trust in dashboards — align the datasets/taxonomy (or add explicit normalization) before relying on these metrics.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="supabase/functions/_backend/utils/cloudflare.ts">
<violation number="1" location="supabase/functions/_backend/utils/cloudflare.ts:2127">
P1: Global success_rate still excludes install/fail outcomes whenever their app has no `get` event in the same 60-second slice, because the total is reduced after the existing `app.get > 0` filter. Removing that filter (or filtering only on the outcome fields) would make the new install/(install+fail) calculation cover all version_usage outcomes and match the admin/global_stats view.</violation>
<violation number="2" location="supabase/functions/_backend/utils/cloudflare.ts:3099">
P2: The headline success rate and the per-dimension (platform/country/version) success rates are now computed from two different Analytics Engine datasets with different failure taxonomies. The daily/total headline reads VERSION_USAGE (install/fail — a single generic 'fail' action), while the breakdown queries read APP_LOG (set vs. the granular PUBLIC_FAILURE_ACTIONS list). Unless those two datasets are guaranteed to record byte-identical event streams, the published headline and the dimensional success rates will disagree, which is exactly the cross-source discrepancy this PR is meant to remove. Consider driving the headline and the breakdowns from the same source (e.g. compute the daily/total aggregates from the same app_log/set + failure-actions query as the breakdowns, or derive the breakdowns from version_usage), so a single consistent success rate is published.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| .map((app) => { | ||
| const totalEvents = app.set + app.get | ||
| const successRate = Number(Number(totalEvents > 0 ? (app.get / totalEvents) * 100 : 100).toFixed(2)) | ||
| const totalOutcomes = app.set + app.failed |
There was a problem hiding this comment.
P1: Global success_rate still excludes install/fail outcomes whenever their app has no get event in the same 60-second slice, because the total is reduced after the existing app.get > 0 filter. Removing that filter (or filtering only on the outcome fields) would make the new install/(install+fail) calculation cover all version_usage outcomes and match the admin/global_stats view.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At supabase/functions/_backend/utils/cloudflare.ts, line 2127:
<comment>Global success_rate still excludes install/fail outcomes whenever their app has no `get` event in the same 60-second slice, because the total is reduced after the existing `app.get > 0` filter. Removing that filter (or filtering only on the outcome fields) would make the new install/(install+fail) calculation cover all version_usage outcomes and match the admin/global_stats view.</comment>
<file context>
@@ -2124,8 +2124,8 @@ export async function getUpdateStatsCF(c: Context): Promise<UpdateStats> {
.map((app) => {
- const totalEvents = app.set + app.get
- const successRate = Number(Number(totalEvents > 0 ? (app.get / totalEvents) * 100 : 100).toFixed(2))
+ const totalOutcomes = app.set + app.failed
+ const successRate = Number(Number(totalOutcomes > 0 ? (app.set / totalOutcomes) * 100 : 100).toFixed(2))
return {
</file context>
| const outcomeBase = `SELECT ${day} AS date, index1 AS app_id, blob1 AS device_id, max(if(blob2 = 'set', 1, 0)) AS succeeded, max(if(blob2 IN (${failureActions}), 1, 0)) AS failed, argMax(blob5, timestamp) AS platform, argMax(blob6, timestamp) AS country, argMax(blob7, timestamp) AS plugin_version FROM app_log WHERE ${window} AND (blob2 = 'set' OR blob2 IN (${failureActions})) GROUP BY date, app_id, device_id` | ||
| const outcomesQuery = `SELECT date, sum(succeeded) AS successes, sum(if(succeeded = 0, failed, 0)) AS failures FROM (${outcomeBase}) GROUP BY date` | ||
| const appLogOutcomeFilter = `(blob2 = 'set' OR blob2 IN (${failureActions}))` | ||
| const dailySuccessQuery = `SELECT ${day} AS date, sum(if(blob3 = 'install', 1, 0)) AS installs, sum(if(blob3 = 'fail', 1, 0)) AS fails FROM version_usage WHERE ${window} GROUP BY date ORDER BY date ASC` |
There was a problem hiding this comment.
P2: The headline success rate and the per-dimension (platform/country/version) success rates are now computed from two different Analytics Engine datasets with different failure taxonomies. The daily/total headline reads VERSION_USAGE (install/fail — a single generic 'fail' action), while the breakdown queries read APP_LOG (set vs. the granular PUBLIC_FAILURE_ACTIONS list). Unless those two datasets are guaranteed to record byte-identical event streams, the published headline and the dimensional success rates will disagree, which is exactly the cross-source discrepancy this PR is meant to remove. Consider driving the headline and the breakdowns from the same source (e.g. compute the daily/total aggregates from the same app_log/set + failure-actions query as the breakdowns, or derive the breakdowns from version_usage), so a single consistent success rate is published.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At supabase/functions/_backend/utils/cloudflare.ts, line 3099:
<comment>The headline success rate and the per-dimension (platform/country/version) success rates are now computed from two different Analytics Engine datasets with different failure taxonomies. The daily/total headline reads VERSION_USAGE (install/fail — a single generic 'fail' action), while the breakdown queries read APP_LOG (set vs. the granular PUBLIC_FAILURE_ACTIONS list). Unless those two datasets are guaranteed to record byte-identical event streams, the published headline and the dimensional success rates will disagree, which is exactly the cross-source discrepancy this PR is meant to remove. Consider driving the headline and the breakdowns from the same source (e.g. compute the daily/total aggregates from the same app_log/set + failure-actions query as the breakdowns, or derive the breakdowns from version_usage), so a single consistent success rate is published.</comment>
<file context>
@@ -3095,17 +3095,17 @@ export async function getPublicLiveUpdateMetricsCF(c: Context, referenceDate = n
- const outcomeBase = `SELECT ${day} AS date, index1 AS app_id, blob1 AS device_id, max(if(blob2 = 'set', 1, 0)) AS succeeded, max(if(blob2 IN (${failureActions}), 1, 0)) AS failed, argMax(blob5, timestamp) AS platform, argMax(blob6, timestamp) AS country, argMax(blob7, timestamp) AS plugin_version FROM app_log WHERE ${window} AND (blob2 = 'set' OR blob2 IN (${failureActions})) GROUP BY date, app_id, device_id`
- const outcomesQuery = `SELECT date, sum(succeeded) AS successes, sum(if(succeeded = 0, failed, 0)) AS failures FROM (${outcomeBase}) GROUP BY date`
+ const appLogOutcomeFilter = `(blob2 = 'set' OR blob2 IN (${failureActions}))`
+ const dailySuccessQuery = `SELECT ${day} AS date, sum(if(blob3 = 'install', 1, 0)) AS installs, sum(if(blob3 = 'fail', 1, 0)) AS fails FROM version_usage WHERE ${window} GROUP BY date ORDER BY date ASC`
const failuresQuery = `SELECT action, count() AS devices FROM (SELECT ${day} AS date, blob2 AS action, index1 AS app_id, blob1 AS device_id FROM app_log WHERE ${window} AND blob2 IN (${failureActions}) GROUP BY date, action, app_id, device_id) GROUP BY action`
const platformsShareQuery = `SELECT platform, count() AS devices FROM (SELECT double1 AS platform, index1 AS app_id, blob1 AS device_id FROM device_usage WHERE ${window} AND double1 IN (0.0, 1.0, 2.0) GROUP BY platform, app_id, device_id) GROUP BY platform`
</file context>
* fix(stats): restore device-day success rate for admin and public Revert #2988 raw install/(install+fail) event counting that collapsed both KPIs to ~24%. Public /data and admin global_stats now share the app_log device-day formula again. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(stats): drop unused plugin_runtime device-day helper logsnag_insights calls utils/cloudflare getDeviceDaySuccessRateCF only. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
2 participants
Summary (AI generated)
/datalive update metrics now compute headline success rate fromversion_usageusinginstall / (install + fail), matching the admin dashboard formula.getUpdateStatsCF,getUpdate_statsSQL, andgetUpdateStatsSB, which previously usedget / (install + get)— not a real install success rate.app_loginstead of device-day dedup that produced a separate ~70% number.Motivation (AI generated)
The marketing
/datapage and admin dashboard showed different success rates because they used different datasets and formulas. A prior fix aligned public stats withglobal_stats, but a follow-up PR reintroduced a separate Analytics Engine device-day calculation fromapp_log. Meanwhile, the cron snapshot stored inglobal_statsusedget/(install+get), which is also semantically wrong for "successful update installations".Business Impact (AI generated)
Capgo can now publish one consistent, correct install success rate on the marketing page and admin dashboard. This removes misleading lower public numbers and prevents overstating success via the old get-ratio formula.
Test Plan (AI generated)
npx vitest run tests/public-live-update-metrics.unit.test.ts tests/public-stats.unit.test.ts tests/cron-email-stats-backtest.unit.test.ts/private/website_stats/live_updatessuccess_rate with adminsuccess_ratemetric for the same 30-day windowglobal_statscron run stores updated success_rate values after migrationGenerated with AI
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Summary by CodeRabbit
Bug Fixes
Tests