Skip to content

fix(admin): sum global stats costs in numeric - #3547

Merged
steebchen merged 1 commit into
mainfrom
fix/global-stats-float4-sums
Aug 12, 2026
Merged

steebchen merged 1 commit into
mainfrom
fix/global-stats-float4-sums

Conversation

@steebchen

@steebchen steebchen commented Aug 11, 2026 •

Copy link
Copy Markdown
Member

Problem

On the admin Global Stats page the cost card does not add up: the headline total and the per-mode / per-kind slices printed underneath it are the same rows grouped two ways, yet they disagree by double-digit dollars at all-time scale. Narrow the view to a small enough slice and it reconciles.

Request counts reconcile everywhere; only money drifts, and only at scale. That is the tell.

Every cost column on the stats tables is real (float4), and Postgres accumulates SUM(real) in float4 too — sum(real) → real, about 7 significant digits. Once a running sum is large enough its ulp exceeds a cent, so the result depends on the order and grouping of the rows. The headline (one SUM over everything) and the composition slices (SUM per mode / per kind) therefore diverge. Small slices reconcile because they stay well inside float4's range.

Reproduced standalone:

with r as (select (i % 4) as bucket, 1234.5678::real as c from generate_series(1,800) i)
select (select sum(c)::text from r),                                       -- 987650.7
       (select sum(s)::text from (select sum(c) s from r group by bucket) g), -- 987652.7
       (select sum(c::float8::numeric)::text from r);                      -- 987654.19921875

Three different answers for the same 800 rows.

Fix

Read path (GET /admin/global-stats) — accumulate cost sums in NUMERIC, which is exact, so any grouping of a row set yields a bit-identical total.

The cast chain matters and is not decorative: real::numeric routes through float4's 6-digit display form (1234.5678::real::numeric → 1234.57), losing more than the float4 sum did. real::double precision is the exact stored value, so the cast is real → float8 → numeric, then back to float8 for JSON.

Also stopped narrowing the all-time request/error/cache counts back to int4 — this endpoint is cross-tenant and all-time, so the count will outgrow 2^31 eventually and fail hard.

Write path (apps/worker hourly aggregation) — same bug a layer down: an hour is tens of thousands of log rows, summed in float4 before being stored, so the buckets were already wrong before anything read them. Now cast to double precision first; the target column is still float4, so only the final value is rounded rather than every partial sum along the way. This covers global_*_stats, project_hourly_*, api_key_hourly_* and provider_key_hourly_stats.

While there: data_storage_cost was being summed as cast(... as real) even though log.data_storage_cost is numeric — it was downcast for no reason.

Not in this PR

  • No column-type migration. The stats tables keep real columns. Per-row rounding is ~5e-8 of the row's own value; the error that was visible came overwhelmingly from the accumulation, which is what this fixes.
  • The other SUM(<real column>) sites across analytics.ts, the model/mapping history rollups and stats-calculator.ts have the same latent bug, but they are per-project or per-model and orders of magnitude smaller, so the drift is currently invisible. Left for a follow-up rather than bundled into a bug fix.
  • The "Unattributed" bucket is a separate defect — org_kind was never backfilled when the column was added, and used_mode only partially. That is addressed in feat(scripts): backfill global stats attribution #3549.

Verification

New spec large all-time sums stay exact and grouping-independent builds a synthetic production-magnitude fixture (700 days × 4 mode/kind buckets) and asserts the headline matches the mode composition, the kind composition, and the timeseries.

Against the unfixed code it reproduces the reported symptom at scale:

AssertionError: expected 12098983.5 to be close to 12099147.748046875,
received difference is 164.248046875, but expected 0.005
  • apps/api — 930 tests pass (70 files)
  • apps/worker — 114 tests pass (9 files)
  • pnpm format, pnpm build clean

No screenshots: this is a numeric correctness fix behind an existing card, nothing about the layout changes.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Improved accuracy of cost and usage totals in global statistics.
    • Prevented rounding drift when aggregating large volumes of decimal-based costs.
    • Preserved fractional request, error, and cache counts instead of narrowing them to whole numbers.
    • Updated cost breakdowns across modes, providers, discounts, and storage metrics for more reliable totals.

Copilot AI lite review requested due to automatic review settings August 11, 2026 19:59
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@coderabbitai

coderabbitai Bot commented Aug 11, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 11a8da6f-e1dc-4a9b-ab11-09640e761958

📥 Commits

Reviewing files that changed from the base of the PR and between 0f2c22c and 29a6343.

📒 Files selected for processing (1)
  • apps/api/src/routes/admin.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • apps/api/src/routes/admin.ts

Walkthrough

The change updates admin and worker cost aggregation to use double-precision intermediates. Count aggregates now use float8. New tests validate 700-day global statistics totals against float4-rounded stored costs.

Changes

Cost aggregation precision

Layer / File(s) Summary
Admin statistics aggregation
apps/api/src/routes/admin.ts
Adds sumMoney for precise cost totals and changes request, error, and cache counts to float8.
Worker statistics aggregation
apps/worker/src/services/project-stats-aggregator.ts
Uses double-precision accumulation for project, modality, discount, storage, credit, API-key, and provider-key costs.
Large-scale aggregation validation
apps/api/src/routes/admin-global-stats.spec.ts
Adds bulk fixtures, cleanup, baseline accounting, and a 700-day aggregation test.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

  • theopenco/llmgateway#3247: Modifies shared cost aggregation in apps/api/src/routes/admin.ts for storage-cost metrics and billing.
  • theopenco/llmgateway#3453: Modifies global and project statistics aggregation areas that this change updates for greater numeric precision.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the admin Global Stats cost-summation fix, which matches the primary change.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/global-stats-float4-sums

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Cost columns on the stats tables are float4, and Postgres accumulates
SUM(real) in float4 as well — roughly 7 significant digits. At the
all-time scale the global stats page shows, the running sum's ulp is
larger than a cent, so the result depends on how the rows happened to
be grouped: the headline total and the per-mode / per-kind slices of
the same rows drifted apart by double-digit dollars.

Sum through numeric (via double precision, because real::numeric goes
through float4's 6-digit display form and loses even more) so every
grouping of a row set yields the identical total. Also stop narrowing
the all-time request/error/cache counts back to int4.

The worker had the same problem on the write side: an hour's worth of
logs was summed in float4 before being stored, so the buckets were
already wrong. Cast to double precision there too — this also fixes
data storage cost, which was being rounded from numeric to float4.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@steebchen
steebchen force-pushed the fix/global-stats-float4-sums branch from 0f2c22c to 29a6343 Compare August 11, 2026 22:02
@steebchen
steebchen merged commit 5714ebb into main Aug 12, 2026
12 checks passed
@steebchen
steebchen deleted the fix/global-stats-float4-sums branch August 12, 2026 10:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants