fix(submit): reject implausible submitted totals - #557
Merged
junhoyeo merged 1 commit intoMay 24, 2026
Conversation
Validation * Validation tier: Tier 3 — submit validation changes persisted leaderboard ingestion semantics. * git diff --check: PASS * git diff --cached --check: PASS * npx vitest run __tests__/api/submit.test.ts __tests__/api/submitAuth.test.ts: PASS, 37 tests * npx vitest run: PASS, 186 tests across 21 files * npx eslint src/lib/validation/submission.ts __tests__/api/submit.test.ts __tests__/api/submitAuth.test.ts: PASS * Cyrillic scan on changed files: PASS, no output * Ledger: not applicable — not required for selected validation tier/change family. * Version: not applicable — not required for selected validation tier/change family. * Not run: Rust test suite — frontend-only ingestion validation change. * Observed: npx tsc --noEmit --pretty false fails on pre-existing unrelated embed test fixture and asset import type errors outside this diff. Rollback * git revert HEAD
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
junhoyeo
approved these changes
May 24, 2026
junhoyeo
left a comment
junhoyeo
left a comment
Owner
There was a problem hiding this comment.
LGTM — SHA-256 idempotency and canonicalization are solid. Follow-up: make thresholds env-overridable.
This was referenced May 25, 2026
junhoyeo
added a commit
that referenced
this pull request
May 27, 2026
…ium-tool-call` (#612) * fix(submit): bypass cost-without-tokens check for Cursor legacy premium-tool-call Cursor's pre-2025-05 usage exports include `premium-tool-call` rows that are billed per tool invocation and carry no token attribution at all (input/output/cache columns are empty). PR #557 introduced a sanity check that rejects any submission where cost > 0 and tokens === 0, which permanently locks out any user with historical Cursor data — even a few cents of legacy tool-call charges block the entire upload (we observed a real submission where ~$45 of premium-tool-call rows blocked $45,618 of otherwise valid data, ~0.1% of the payload). Allow `cursor/premium-tool-call` specifically by adding a single legacy carve-out. The check still fires for every other `(client, model)` combination, including other cursor models, so genuine parser regressions remain visible. Also enrich the cost-without-tokens error messages so operators can read the offending row directly out of the CLI output: - Client-level errors now include the client, modelId, providerId, full cost (`cost=$X.XXXX`), and the full token breakdown (`input/output/cacheRead/cacheWrite/reasoning`). - Day-level errors list which clients on that day were responsible (`offending clients: cursor/premium-tool-call (provider=cursor) cost=$2.0500; ...`). - Submission-summary errors include the same per-client offender list. Constraint: cannot drop Cursor `premium-tool-call` rows in the CLI parser because their cost is a legitimate spend the user wants reflected in totals. Rejected: Drop tokenless rows in the CLI scanner | loses real cost attribution that the leaderboard depends on. Rejected: Loosen the check globally (cost > 0 && tokens === 0 → warning) | re-opens the original PR #557 vulnerability (implausible cost-only submissions). Confidence: high Scope-risk: narrow Directive: Only add new entries to CURSOR_LEGACY_TOKENLESS_MODELS for Cursor billing events that legitimately lack token attribution — never for parser bugs (fix the parser instead). Not-tested: A Cursor user whose entire history is exclusively `premium-tool-call` (summary-level legacy subtraction path is exercised only by unit tests, not by an end-to-end submit). * fix(submit): exclude Cursor legacy cost from cost-per-million sanity cap Address Codex review (P1) on PR #612: when a legacy `cursor/premium-tool-call` row shares a day with a small amount of token-bearing usage, the day-level branch falls through to the cost-per-million check because `day.totals.tokens > 0`. Using the full day cost (legacy + real) as the numerator meant tiny token counts tripped the $10k/M ceiling even though the legacy row is supposed to be skipped — e.g. $2.05 in legacy tool calls plus 100 normal tokens computed $20,600/M and was rejected. Subtract the legacy tokenless Cursor cost before applying the cost-per-million cap at both day and summary levels (was previously only subtracted for the cost-without-tokens branch). The cost-without-tokens branch behavior is unchanged because both branches now read from the same `checkableCost` value. Add a regression test (`excludes cursor legacy cost from the cost-per-million sanity cap`) that pins the exact mixed-day shape from the review. Confidence: high Scope-risk: narrow Directive: Any future cost-related sanity check on day/summary aggregates must also subtract `legacyCost` first — wire it off the same `checkableCost` local instead of recomputing. * fix(submit): use float-epsilon when subtracting Cursor legacy cost Address cubic review (P2) on PR #612: the new legacy-cost subtraction used strict `> 0` float comparison at both the day and summary cost-without-tokens checks. Floating-point summation residue could trip the check on a valid all-legacy submission — e.g. `0.1 + 0.2 === 0.30000000000000004` while IEEE `0.3 ≈ 0.299999999999999988`, so `totalCost - legacyClientCost ≈ 5.5e-17 > 0` even though the user truly has $0.30 of legacy cost and nothing else. Introduce `LEGACY_COST_FLOAT_EPSILON = 1e-6` and use it as the threshold on both check sites. 1e-6 is well below any realistic LLM charge, so any legitimate non-legacy cost still trips the check; only FP rounding noise is absorbed. Add a regression test (`tolerates floating-point residue when subtracting cursor legacy cost`) that constructs the exact `0.1 + 0.2` vs `0.3` scenario and asserts the residue is positive but the submission still validates. Confidence: high Scope-risk: narrow Directive: Any cost-vs-cost float comparison on aggregated sums should use LEGACY_COST_FLOAT_EPSILON (or a comparable epsilon) — strict `> 0` is a footgun on summed IEEE 754 values.
Owner
|
@IvGolovach this has been merged to v3.0.0: https://github.com/junhoyeo/tokscale/releases/tag/v3.0.0 thanks for the contribution! |
leecoder
pushed a commit
to leecoder/tokscale
that referenced
this pull request
May 29, 2026
…ium-tool-call` (junhoyeo#612) * fix(submit): bypass cost-without-tokens check for Cursor legacy premium-tool-call Cursor's pre-2025-05 usage exports include `premium-tool-call` rows that are billed per tool invocation and carry no token attribution at all (input/output/cache columns are empty). PR junhoyeo#557 introduced a sanity check that rejects any submission where cost > 0 and tokens === 0, which permanently locks out any user with historical Cursor data — even a few cents of legacy tool-call charges block the entire upload (we observed a real submission where ~$45 of premium-tool-call rows blocked $45,618 of otherwise valid data, ~0.1% of the payload). Allow `cursor/premium-tool-call` specifically by adding a single legacy carve-out. The check still fires for every other `(client, model)` combination, including other cursor models, so genuine parser regressions remain visible. Also enrich the cost-without-tokens error messages so operators can read the offending row directly out of the CLI output: - Client-level errors now include the client, modelId, providerId, full cost (`cost=$X.XXXX`), and the full token breakdown (`input/output/cacheRead/cacheWrite/reasoning`). - Day-level errors list which clients on that day were responsible (`offending clients: cursor/premium-tool-call (provider=cursor) cost=$2.0500; ...`). - Submission-summary errors include the same per-client offender list. Constraint: cannot drop Cursor `premium-tool-call` rows in the CLI parser because their cost is a legitimate spend the user wants reflected in totals. Rejected: Drop tokenless rows in the CLI scanner | loses real cost attribution that the leaderboard depends on. Rejected: Loosen the check globally (cost > 0 && tokens === 0 → warning) | re-opens the original PR junhoyeo#557 vulnerability (implausible cost-only submissions). Confidence: high Scope-risk: narrow Directive: Only add new entries to CURSOR_LEGACY_TOKENLESS_MODELS for Cursor billing events that legitimately lack token attribution — never for parser bugs (fix the parser instead). Not-tested: A Cursor user whose entire history is exclusively `premium-tool-call` (summary-level legacy subtraction path is exercised only by unit tests, not by an end-to-end submit). * fix(submit): exclude Cursor legacy cost from cost-per-million sanity cap Address Codex review (P1) on PR junhoyeo#612: when a legacy `cursor/premium-tool-call` row shares a day with a small amount of token-bearing usage, the day-level branch falls through to the cost-per-million check because `day.totals.tokens > 0`. Using the full day cost (legacy + real) as the numerator meant tiny token counts tripped the $10k/M ceiling even though the legacy row is supposed to be skipped — e.g. $2.05 in legacy tool calls plus 100 normal tokens computed $20,600/M and was rejected. Subtract the legacy tokenless Cursor cost before applying the cost-per-million cap at both day and summary levels (was previously only subtracted for the cost-without-tokens branch). The cost-without-tokens branch behavior is unchanged because both branches now read from the same `checkableCost` value. Add a regression test (`excludes cursor legacy cost from the cost-per-million sanity cap`) that pins the exact mixed-day shape from the review. Confidence: high Scope-risk: narrow Directive: Any future cost-related sanity check on day/summary aggregates must also subtract `legacyCost` first — wire it off the same `checkableCost` local instead of recomputing. * fix(submit): use float-epsilon when subtracting Cursor legacy cost Address cubic review (P2) on PR junhoyeo#612: the new legacy-cost subtraction used strict `> 0` float comparison at both the day and summary cost-without-tokens checks. Floating-point summation residue could trip the check on a valid all-legacy submission — e.g. `0.1 + 0.2 === 0.30000000000000004` while IEEE `0.3 ≈ 0.299999999999999988`, so `totalCost - legacyClientCost ≈ 5.5e-17 > 0` even though the user truly has $0.30 of legacy cost and nothing else. Introduce `LEGACY_COST_FLOAT_EPSILON = 1e-6` and use it as the threshold on both check sites. 1e-6 is well below any realistic LLM charge, so any legitimate non-legacy cost still trips the check; only FP rounding noise is absorbed. Add a regression test (`tolerates floating-point residue when subtracting cursor legacy cost`) that constructs the exact `0.1 + 0.2` vs `0.3` scenario and asserts the residue is positive but the submission still validates. Confidence: high Scope-risk: narrow Directive: Any cost-vs-cost float comparison on aggregated sums should use LEGACY_COST_FLOAT_EPSILON (or a comparable epsilon) — strict `> 0` is a footgun on summed IEEE 754 values.
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #554.
This PR adds server-side ingestion guardrails for self-reported submit payloads so internally consistent but fabricated totals cannot go straight into persisted leaderboard data.
Why
POST /api/submitcurrently accepts payloads as long as they are structurally valid and internally consistent. That leaves a concrete gap where a payload can report extreme daily token totals, extreme cost, or cost without meaningful token usage, then get persisted intodaily_breakdownand recomputed into public leaderboard totals.Diff scope
The validator now rejects non-finite numeric values, unsafe integer counters, implausible daily/submission token totals, implausible daily/submission costs, cost submitted with zero tokens, extreme cost-per-million-token values, and client/day token or cost mismatches that would otherwise be recalculated into persisted totals.
generateSubmissionHash()now uses a SHA-256 hash over canonicalized usage content instead of a shape-only djb2 hash. It intentionally excludesgeneratedAtso an idempotent resubmit of identical usage data keeps the same hash while changed totals, costs, models, dates, or client breakdowns produce a different hash.Branch integrity
Base branch:
main.Validated base SHA:
ca7e4784752046630f22d28b9e39ddfe386370a5.Ahead/behind:
0 behind / 1 ahead.Merge base:
ca7e4784752046630f22d28b9e39ddfe386370a5.origin/mainis an ancestor of this branch, so the diff was computed against the fetched base.Commit integrity
Introduced commit:
The final PR diff contains only submit validation and directly related frontend tests.
Diff hygiene
Changed files:
git diff --check origin/main...HEAD: PASS, no output.No migrations, generated artifacts, local environment files, secrets, credentials, build output, or unrelated files are included.
Validation mode and proof
Validation mode: Mode 3 — shared frontend ingestion validation affects persisted leaderboard data.
Local validation:
Not run locally:
Observed unrelated local typecheck issue:
Ledger and version proof
Ledger: not applicable — not required for this change family.
Version: not applicable — not required for this change family.
Migration notes
Not applicable — no database migration changed.
CI context confirmation
Not applicable — no CI workflow or required context names changed.
Remote checks observed after PR open:
Runtime safety
No new background jobs, locks, queues, external calls, or database writes were added.
This change only tightens validation before the existing transaction path can persist submitted usage.
No invariant regression introduced.
Documentation integrity
Not applicable — no user-facing commands, runbooks, release procedure, or deployment procedure changed.
Rollback plan
Rollback: revert this PR.
DB downgrade: not applicable.
Data repair: not applicable.
Operational caveats: none known.
Known residual risks
This PR intentionally does not add exact model-price-table validation because Tokscale pricing currently lives in Rust code and is not exposed to the Next.js frontend as a shared pricing artifact. Exact cost-vs-pricing verification should be a follow-up built around a deliberate shared pricing boundary rather than duplicating Rust pricing logic in the frontend.