Fix analytics history and log token accounting

[htkarabulut]

Summary

Two narrow fixes on top of release/v3.8.6:

1. Analytics history (src/app/api/usage/analytics/route.ts, src/lib/usage/aggregateHistory.ts, src/lib/db/cleanup.ts, src/shared/validation/settingsSchemas.ts)
   • The usage analytics endpoint read only usage_history, so once the configured raw-retention window (default 30 days) elapses and cleanup deletes those rows, the dashboard showed empty/short history even though daily_usage_summary already held the aggregated data created by aggregateHistory.
   • Endpoint now UNIONs usage_history (recent, raw) with daily_usage_summary (older, aggregated) in a single source query, then groups by day / provider / model / serviceTier.
   • The v3.8.6 service-tier columns are preserved. Older daily_usage_summary rows that pre-date service-tier tracking are reported as 'standard'.
   • Cleanup and aggregation now share the same retention setting so the union boundary matches the deletion boundary.
2. Logs token accounting (src/lib/usage/tokenAccounting.ts)
   • getLoggedInputTokens previously hit the input_tokens branch first and added cache_read_input_tokens + cache_creation_input_tokens on top. For Claude streaming responses translated to OpenAI shape, the translator emits both prompt_tokens (already containing the full input + cache totals) and keeps input_tokens as a compatibility alias. The old order double-counted the cache portion — e.g. a 600k-token request was logged as ~1.2M TI on the Logs page.
   • New order: tokens.input → prompt_tokens → (input_tokens + cache fields). The third branch is unchanged and still handles raw Anthropic / anthropic-compatible-cc streaming where input_tokens is only the non-cached portion.
   • Regression test added: prompt_tokens=600000 + input_tokens=600000 + cache_read_input_tokens=600000 → 600000 (not 1_200_000).

Related Issues

• Related to [RFC] Time-Based Aggregation Engine for Cost/Analytics Pages #1937 (time-based aggregation engine — analytics union completes the read path)
• Related to [RFC] Database Performance Optimization - User-Configurable Aggregation & Cleanup #1935 (DB performance / retention umbrella)
• Related to feat: detailed token tracking + fix Anthropic input undercount #1017 (Anthropic input undercount — fix refines its streaming branch)
• Related to fix(usage): include cache tokens in input token counts #822 (getLoggedInputTokens trusting prompt_tokens as the full input total)
• Related to fix(cost): prevent double-billing of cache_creation_input_tokens #2522 (cache-token double-count family — same convention)

Validation

• npm run lint — 0 errors, 2985 pre-existing warnings repo-wide; zero findings on files touched by this PR
• npm run test:unit — targeted: node --import tsx --import ./open-sse/utils/setupPolyfill.ts --test tests/unit/token-accounting-input-fix.test.ts → 9 pass / 0 fail
• npm run test:coverage
• Coverage is still >= 60% for statements, lines, functions, and branches
• SonarQube PR analysis is green or any remaining issues are explicitly documented below

Tests Added Or Updated

• tests/unit/token-accounting-input-fix.test.ts — rewritten to import the real getLoggedInputTokens from src/lib/usage/tokenAccounting.ts (previously asserted against a local re-implementation that masked the bug) and extended with a Claude-streaming regression: mixed prompt_tokens + input_tokens + cache_read_input_tokens must not double-count.
  No other test files changed. The analytics history fix is exercised through the existing aggregateHistory / cleanup paths; a dedicated test for the union behavior is a sensible follow-up if reviewers prefer.

Coverage Notes

• src/lib/usage/tokenAccounting.ts: the new branch ordering is covered by the regression test plus the existing cases for the tokens.input / input_tokens paths. No coverage regression expected in this file.
• src/app/api/usage/analytics/route.ts, src/lib/usage/aggregateHistory.ts, src/lib/db/cleanup.ts, src/shared/validation/settingsSchemas.ts: changes are additive (new union arm, new retention wiring, schema fields). Existing analytics integration tests still apply; a follow-up unit covering the boundary day where raw rows have just been cleaned but daily_usage_summary still has them would close the remaining gap.

Reviewer Notes

• Service-tier preservation: the union query keeps the v3.8.6 service_tier columns. Rows coming from daily_usage_summary that pre-date service-tier tracking are surfaced as 'standard'. If you’d rather expose them as NULL / 'unknown', that’s a one-line change in the union projection.
• Retention boundary: the analytics union assumes aggregateHistory has already populated daily_usage_summary for any day older than rawDataRetentionDays. If aggregation is disabled or lagging, those days will still appear empty — same behavior as before for that subset; this PR doesn’t change the aggregation cadence.
• Token accounting scope: the precedence change is intentionally only in the shared logging helper. Translators that emit both prompt_tokens and input_tokens as compatibility aliases are left alone, since downstream OpenAI-format consumers rely on that dual shape.
• Out of scope, flagged for follow-up: the CC-compatible executor (open-sse/executors/base.ts) appears to forward context-1m-2025-08-07 beta headers without the model gating that claudeIdentity.ts performs for native Claude OAuth. We observed a [429] Usage credits are required for long context requests. from a Sonnet request via the CC bridge that is consistent with that behavior. Not touched here.

[gemini-code-assist]

Code Review

This pull request introduces a mechanism to roll up historical usage data from usage_history into daily_usage_summary before cleanup, and updates the analytics route to query a unified source merging raw and aggregated data. It also refines token accounting to prefer prompt_tokens over input_tokens to prevent double-counting. Feedback on these changes highlights critical SQL injection vulnerabilities and double-counting risks in the analytics route due to direct string interpolation and lack of boundary checks. Additionally, the reviewer noted that deleting sidebar settings from the schema will cause validation failures, proceeding with cleanup after a failed rollup risks data loss, and using DATE(timestamp) in the rollup query bypasses database indexes.

[gemini-code-assist]

SQL Injection & Data Leakage / Double Counting Vulnerability

1. SQL Injection: Direct string interpolation of sinceIso and untilIso splits into the SQL query (date >= '${sinceDate}') creates a SQL injection vulnerability because these values originate from user-controlled query parameters (startDate and endDate).
2. Double Counting & Data Leakage: Without a boundary check, the query will select overlapping rows from both usage_history and daily_usage_summary for the recent period. Furthermore, since daily_usage_summary does not store API key or connection details, any query filtered by API key will leak other keys' aggregated usage for the overlapping period.

Fix: Use parameter binding for the date parameters and restrict the queries using the rawCutoff boundary to ensure non-overlapping datasets.

    const rawConditions: string[] = [];
    if (needsAggregated) {
      rawConditions.push("timestamp >= @rawCutoff");
      params.rawCutoff = rawCutoffIso;
    } else if (sinceIso) {
      rawConditions.push("timestamp >= @since");
    }
    if (untilIso) rawConditions.push("timestamp <= @until");
    if (apiKeyWhere) rawConditions.push(apiKeyWhere);
    const rawWhere = rawConditions.length > 0 ? `WHERE ${rawConditions.join(" AND ")}` : "";

    // Aggregated rows only span dates within the requested window and strictly before the raw cutoff
    // to prevent double-counting and data leakage (since api_key/connection filters don't apply to daily_usage_summary).
    const aggConditions: string[] = [];
    if (sinceIso) {
      aggConditions.push("date >= @sinceDate");
      params.sinceDate = sinceIso.split("T")[0];
    }
    if (untilIso) {
      aggConditions.push("date <= @untilDate");
      params.untilDate = untilIso.split("T")[0];
    }
    aggConditions.push("date < @rawCutoffDate");
    params.rawCutoffDate = rawCutoffIso.split("T")[0];
    const aggWhere = aggConditions.length > 0 ? `WHERE ${aggConditions.join(" AND ")}` : "";

References

1. Never write raw SQL in routes — always go through src/lib/db/ domain modules. ^(link)

[gemini-code-assist]

SQL Injection & Double Counting in Preset Query

This block suffers from the same SQL injection and double-counting vulnerabilities as the main query. Direct string interpolation is used for presetSinceIso splits, and there is no boundary check to prevent overlapping data between usage_history and daily_usage_summary.

Fix: Apply parameter binding and restrict the datasets using the rawCutoff boundary.

        const presetRawConds: string[] = [];
        if (presetNeedsAggregated) {
          presetRawConds.push("timestamp >= @presetRawCutoff");
          presetParams.presetRawCutoff = rawCutoffIso;
        } else if (presetSinceIso) {
          presetRawConds.push("timestamp >= @presetSince");
          presetParams.presetSince = presetSinceIso;
        }
        if (apiKeyWhere) {
          presetRawConds.push(apiKeyWhere);
          Object.assign(presetParams, params);
        }
        const presetRawWhere =
          presetRawConds.length > 0 ? `WHERE ${presetRawConds.join(" AND ")}` : "";

        const presetAggConds: string[] = [];
        if (presetSinceIso) {
          presetAggConds.push("date >= @presetSinceDate");
          presetParams.presetSinceDate = presetSinceIso.split("T")[0];
        }
        presetAggConds.push("date < @presetRawCutoffDate");
        presetParams.presetRawCutoffDate = rawCutoffIso.split("T")[0];
        const presetAggWhere =
          presetAggConds.length > 0 ? `WHERE ${presetAggConds.join(" AND ")}` : "";

References

1. Never write raw SQL in routes — always go through src/lib/db/ domain modules. ^(link)

[gemini-code-assist]

Accidental Deletion of Sidebar Settings Schema Import

The import of SIDEBAR_SECTIONS was deleted from settingsSchemas.ts. This will cause a compilation/runtime error once the deleted sidebar fields are restored to the schema.

Fix: Restore the import of SIDEBAR_SECTIONS.

Suggested change

	import { HIDEABLE_SIDEBAR_ITEM_IDS } from "@/shared/constants/sidebarVisibility";
	import { HIDEABLE_SIDEBAR_ITEM_IDS, SIDEBAR_SECTIONS } from "@/shared/constants/sidebarVisibility";

[gemini-code-assist]

Restore Sidebar Settings Schema Fields

The sidebar settings fields (sidebarSectionOrder, sidebarItemOrder, and sidebarActivePreset) were deleted from updateSettingsSchema. Since no corresponding UI changes are made in this PR, any attempts by the UI to save or update these settings will now fail Zod validation with a 400 Bad Request error.

Fix: Restore the deleted fields to the schema.

  hiddenSidebarItems: z.array(z.enum(HIDEABLE_SIDEBAR_ITEM_IDS)).optional(),
  sidebarSectionOrder: z
    .array(z.enum(SIDEBAR_SECTIONS.map((s) => s.id) as [string, ...string[]]))
    .optional(),
  sidebarItemOrder: z.record(z.string(), z.array(z.string().max(100))).optional(),
  sidebarActivePreset: z.enum(["all", "minimal", "developer", "admin"]).nullable().optional(),
  comboConfigMode: z.enum(COMBO_CONFIG_MODES).optional(),

[gemini-code-assist]

Data Loss Risk on Rollup Failure

If rollupUsageHistoryBeforeDate fails (e.g., due to database locks or disk issues), the cleanup process currently logs the error but still proceeds to execute DELETE FROM usage_history. This will result in permanent loss of raw usage data without any aggregated backup.

Furthermore, the catch block here is dead code because rollupUsageHistoryBeforeDate internally catches its own errors and returns an AggregationResult object with errors > 0 instead of throwing.

Fix: Check the returned rollupResult.errors and abort the deletion if any errors occurred.

  const rollupResult = await rollupUsageHistoryBeforeDate(cutoffDateStr);
  if (rollupResult.errors > 0) {
    console.error("[Cleanup] Aborting usage_history deletion because rollup failed.");
    result.errors += rollupResult.errors;
    return result;
  }

[gemini-code-assist]

Performance Bottleneck: Index Bypassed in WHERE Clause

Using DATE(timestamp) < ? in the WHERE clause prevents SQLite from using any index on the timestamp column, forcing a full table scan of usage_history. Since usage_history can grow extremely large, this will cause severe performance degradation during cleanup.

Since beforeDate is a standard YYYY-MM-DD string, a direct string comparison timestamp < ? is perfectly correct and allows SQLite to perform an efficient index range scan.

Fix: Compare timestamp directly.

Suggested change

	WHERE DATE(timestamp) < ?
	WHERE timestamp < ?

[diegosouzapw]

Thank you @unitythemaker for your contribution! This has been successfully merged into the release branch (with date parameters in the analytics route query parameterized to secure against SQL injection) and will be included in the next release.