feat(dashboard): continuous call-log export to pluggable destinations (BigQuery first)

[dpozimski]

Summary

The Logs tab keeps request history in SQLite, bounded by rotation and retention, so anything older than the window is gone. This adds a scheduled export that ships those same records to an external store before that happens.

A destination is a plugin. It declares a Zod config schema, a field descriptor the dashboard renders its form from, which config keys are secret, and a client with test() / prepare() / send(). Adding Datadog or Grafana Loki later means one file under src/lib/logExport/destinations/ and one line in the registry. The runner, the REST layer and the UI never learn what a destination is.

Google BigQuery is the first one. It talks the REST API directly with a self-signed RS256 assertion rather than pulling @google-cloud/bigquery and google-auth-library into the tree, which is about 100 lines of node:crypto and fetch and keeps the dependency count where it was.

Some details worth knowing:

• The cursor is call_logs.rowid, not timestamp. saveCallLog accepts a caller-supplied timestamp, so a slow request can be written after a faster one that started later, and a timestamp cursor would step over it. A cursor above MAX(rowid) means the table was purged and rowids restarted, so the runner rewinds instead of going permanently blind.
• The cursor moves only after send() resolves. A failed batch is retried on the next run. The guarantee is at-least-once plus destination-side dedup, not exactly-once, and every row carries the call-log id as its insertId so a resent batch collapses on BigQuery's side.
• A partial failure comes back as HTTP 200 with a non-empty insertErrors[]. That throws. Treating it as success is how you advance the cursor past rows BigQuery never took.
• The batch size is a cursor unit, not an HTTP one. send() chunks on row count and serialised bytes, closing at 500 rows or 9 MB, whichever comes first. Row count alone is not enough once payloads ship: 500 rows carrying prompts can be tens of megabytes against insertAll's 10 MB cap.
• Service-account keys are encrypted with the existing field encryption and never returned by any endpoint. Because encrypt() is a silent passthrough without STORAGE_ENCRYPTION_KEY, storing a destination that carries a credential is refused with a 400 when the key is absent, following the guard the Telegram webhook already uses.

Payloads are a separate, opt-in step. By default the export carries the summary fields the Logs list shows. A destination can set includeBodies to additionally ship what the Logs detail pane shows: the request and response bodies plus the pipeline's route decision, client raw request, normalised OpenAI request, provider-dialect request, provider response and client response. This is prompt content, so it is off by default and chosen per destination. Both the export and the dashboard read through getCallLogById, so the two cannot drift, and payloads arrive already PII-sanitised and secret-redacted with noLog calls carrying none.

The auto-created BigQuery table is day-partitioned on timestamp and clustered on api_key_name, provider, model, status, ordered by how call logs are actually filtered. Partition retention is configurable. Both apply at creation, so an existing table keeps its layout.

New env var: OMNIROUTE_LOG_EXPORT_CRON, default 0 * * * *. Docs in docs/frameworks/LOG-EXPORT.md.

A build failure this PR introduced, and the panic that hid it

Every page under docs/ is compiled by fumadocs-mdx, whose schema requires a title. The new
docs/frameworks/LOG-EXPORT.md shipped without frontmatter, so npm run build failed:

Error: [MDX] invalid frontmatter in /app/docs/frameworks/LOG-EXPORT.md:
- title: Invalid input: expected string, received undefined

Turbopack never prints that message. It panics inside ImportTracer::get_traces while building
the import trace for the issue it is about to report, so the build dies with
internal error: entered unreachable code: there must be a path to a root
(turbopack-core/src/module_graph/mod.rs:746) and the cause is swallowed. Building with the
documented escape hatch --build-arg OMNIROUTE_USE_TURBOPACK=0 surfaces it immediately.

Frontmatter added; the production Docker image now builds and runs. The Turbopack panic itself is
independent of this change and still reproduces on the base tip with the frontmatter fixed in
place, so it is an upstream bug worth its own issue, not something this PR can address.

Related Issues

Validation

• Change type: DB + UI + other (background job, REST)
• Focused tests and category gates from the golden path
• npm run lint
• Reconciled with the current active release base (release/v3.8.51)
• Production-code changes include new automated tests in this PR

node --import tsx/esm --test tests/unit/log-export-*.test.ts      62 pass, 0 fail
npx eslint <changed files>                                        clean
npx tsc --noEmit -p tsconfig.json                                 clean for changed files
npm run check:route-validation:t06                                PASS
npm run check:openapi-routes                                      OK (276 paths)
npm run check:openapi-coverage                                    PASS 39.7% (up from 38.8%)
npm run check:openapi-security-tiers                              PASS
npm run check:env-doc-sync                                        PASS
npm run check:docs-sync / docs-counts / doc-links                 PASS
npm run i18n:check-ui-coverage                                    PASS (42 locales)
npm run check:file-size / cycles / error-helper                   OK

Full CI on the current head is green: all four unit shards, Fast Quality Gates, Docs Gates, ESLint, Vitest, Merge integrity and both semgrep jobs.

Two gate failures from the first push are fixed here. check:db-rules requires every src/lib/db/ module to be re-exported from localDb.ts, which logExportDestinations was not; and the locale parity suites require every en.json key to exist in all 42 locales, so sidebar.logExport and sidebar.logExportSubtitle are now translated rather than English-only.

Live validation: the production image, a real provider, a real BigQuery table

Built the production image from the repo Dockerfile (--target runner-web), ran it, pointed it at
a real Cursor API connection and a real BigQuery dataset, and drove the whole path through the
shipped HTTP surface. 21 checks, all green, repeatable on a fresh table name each run:

1.  dashboard login
2.  POST /api/providers  cursor-api connection  -> 201, upstream test valid in 425ms
3.  POST /api/keys                              -> 201
4.  POST /api/log-export/destinations           -> 201
    response body contains no private key; the stored config column is enc:v1:
5.  POST /destinations/{id}/test                -> ok, authenticated as the service account
6.  GET  /api/log-export/status                 -> cron "0 * * * *", UTC
7.  3 live chat completions through /v1/chat/completions on cua/auto
    answers came back "alpha" / "bravo" / "charlie"
8.  GET /api/usage/call-logs                    -> 16 rows
9.  POST /destinations/{id}/run                 -> exported=16 batches=1
10. BigQuery holds 16 rows, 0 duplicate ids, every dashboard id present
11. rerun with no new calls                     -> exported=0, BigQuery still 16
12. one more call, rerun                        -> exported=1, BigQuery 17, still 0 duplicates
13. auto-created table has all 47 columns, day-partitioned and clustered
14. field-for-field match against the Logs tab: 17 rows compared, 0 mismatches

A sample exported row, showing the null and boolean handling that matters:

{"id":"1787997867123-e1ba1e","model":"auto","requested_model":"cursor-api/auto",
 "provider":"cursor-api","status":200,"tokens_in":56,"tokens_out":25,
 "tokens_cache_read":null,"model_pinned":false,"api_key_name":"log-export-e2e",
 "error_summary":null,"duration_ms":2743}

Separately, to prove the schedule fires rather than merely being registered, a second container ran
with OMNIROUTE_LOG_EXPORT_CRON="* * * * *". One live call, then nothing was triggered by hand:
the job ran on its own, job_runs recorded status=success, and the rows landed in BigQuery.

Two defects from earlier runs against the real API are fixed here: a table created moments before is
not yet visible to the streaming endpoint and answers 404 for a few seconds, so that 404 is retried
but only when the run itself created the table; and the retry's sleep used an unref'd timer, which
let the process exit out from under an in-flight retry.

Tests Added Or Updated

• tests/unit/log-export-runner.test.ts (new) cursor advance, batching, retry after failure, max_rows_per_run, purge rewind, concurrent-run guard, migration 170 upgrade path, and for payloads: nothing ships with includeBodies off (asserted against the wire, not the record), both client and provider views ship when on, oversized payloads truncate and flag, a missing artifact exports its summary instead of failing the batch
• tests/unit/log-export-bigquery.test.ts (new) field mapping vs table schema, insertId dedup, insertErrors on HTTP 200, chunking at 500, transient retry, terminal 403, fresh-table 404, token cache, clustering and partition retention on the created table, byte-aware chunking under multi-megabyte payloads, and a single oversized row still shipping rather than wedging the cursor
• tests/unit/log-export-routes.test.ts (new) CRUD, validation, 404s, secret redaction, placeholder round trip, encryption-key guard
• tests/unit/log-export-secrets.test.ts (new) encrypt, decrypt, redact, merge
• tests/unit/log-export-extensibility.test.ts (new) a fake destination drives the whole runner with no BigQuery code loaded, plus a grep assertion that no destination-specific vocabulary leaked into the 17 generic files
• tests/unit/sidebar-visibility.test.ts updated for the new sidebar entry

Coverage Notes

Everything under src/lib/logExport/, src/lib/db/logExportDestinations.ts, src/lib/usage/callLogExportSource.ts, src/lib/jobs/logExportJob.ts and src/app/api/log-export/ is covered by the five new suites, including the failure paths. The dashboard components under src/app/(dashboard)/dashboard/log-export/ have no unit tests and follow the existing dashboard pages in that respect.

Reviewer Notes

• Migration 170_log_export_destinations.sql adds one table and one index. Nothing else is touched, and the upgrade path is covered by a test that drops the table and its ledger row and reopens the database. This takes the repo to 167 migrations, so the three docs that hardcode the count and the 42 llm.txt mirrors are updated.
• The job is registered in initCloudSync, so OMNIROUTE_DISABLE_BACKGROUND_SERVICES and the automated-test guard already switch it off. A fresh install has no destinations and exports nothing.
• Nothing runs until an operator configures a destination and enables it.
• Payloads are opt-in per destination (includeBodies, off by default). When on, the export additionally carries the request and response bodies plus the pipeline's client and provider views, read through getCallLogById so it cannot drift from the Logs detail pane. They arrive already PII-sanitised and secret-redacted, and a call made with a noLog API key stores none, so there is nothing to export. maxBodyBytes truncates rather than drops and flags the row. Streamed chunk deltas are not exported.
• The registry has two test-only exports (__registerLogExportDestinationTypeForTest / __reset...), matching the pattern already used by __resetJobRegistry. They exist so the extensibility test can prove the pipeline works with no BigQuery code loaded. Happy to drop them if you would rather keep the registry closed.
• POST /destinations/{id}/run returns skipped: true rather than a 409 when a run for that destination is already in flight. Easy to change if you prefer the status code.

[dpozimski]

All gates are green except Build (advisory), which fails exit 143 (SIGTERM at 13 minutes) with no compile output.

That job is fork-PR-only and continue-on-error: true, and quality.yml already documents the condition in its own comments: build.yml went workflow_dispatch-only on 2026-08-29 because "the hosted runner cannot build this tree in any profile, 8/8 recent fork PRs included", and 6/6 sampled failures were shutdown signals with "zero OOM, zero build errors". So it is the runner, not this branch.

I did not want to leave the production build unverified on that basis, so I built and ran it locally instead:

docker build --target runner-web --build-arg OMNIROUTE_USE_TURBOPACK=0 -t omni-local .
docker run -d --env-file env -p 20128:20128 omni-local

The image builds, boots, and serves. Against it: a real cursor-api connection, three live completions on cua/auto, 16 call-log rows exported to a real BigQuery table with no duplicates, a rerun exporting nothing, an incremental run picking up exactly one, and a 37-column day-partitioned table matching the Logs tab field for field. A second container with OMNIROUTE_LOG_EXPORT_CRON="* * * * *" confirmed the schedule fires on its own.

Building with OMNIROUTE_USE_TURBOPACK=0 is also what exposed the frontmatter bug fixed in this branch. Turbopack panics inside ImportTracer::get_traces while formatting the import trace for an issue, so the underlying message never reaches the log. Worth knowing when a build here dies with internal error: entered unreachable code and nothing else.

[dpozimski]

Live run: dashboard and BigQuery evidence

Everything below came from the production image built from this branch's Dockerfile, running
against a real Cursor connection and a real BigQuery dataset. The config path was driven through
the browser, not the API.

The destinations page

Hourly schedule, the call-log high-water mark, and per-destination state with Test / Run now /
Disable / Edit / Delete.

ui-configured-prompts in that list was created by filling this form in the browser, then tested
and run from the same page: Test returned Authenticated as …iam.gserviceaccount.com. Table call_logs_from_ui will be created on the first export., and Run now reported
exported 9 rows · 0 pending · cursor 9.

The config form

Scrolled to the payload controls. The toggle is unchecked by default; ticking it reveals the
guidance and the per-field byte cap.

Reading the destination back through the API after creating it in the UI:

{"includeBodies": true, "maxBodyBytes": 262144, "enabled": true,
 "table": "…omniroute_live_e2e.call_logs_from_ui", "retentionDays": 30,
 "credential": "__stored__"}

The exported data

omniroute-call-logs-sample.csv
is 9 real rows pulled back out of BigQuery with SELECT *, all 47 columns. One row, abridged:

requested_model : cursor-api/auto
status          : 200      tokens_in 62   tokens_out 20    duration_ms 2750
request_body    : {"model":"cursor-api/auto","messages":[{"role":"user",
                   "content":"Reply with exactly this word: capybara-1788010347897"}],"max_tokens":30}
response_body   : {"choices":[{"message":{"content":"capybara-1788010347897"}}], …}
pipeline_client_request   : 1153 bytes
pipeline_provider_request :  796 bytes  {"url":"https://agentn.global.api5.cursor.sh/agent.…
pipeline_provider_response:  562 bytes
pipeline_client_response  :  439 bytes
bodies_truncated: false

Opening that same call id in the Logs detail pane returns the identical prompt and the same four
pipeline stages, which is the parity this feature is meant to hold.

Payload presence by API key in that table:

(none)          rows= 4   without payload=4     internal calls, no payload captured
bodies-nolog    rows= 2   without payload=2     noLog key: summary exports, payloads do not
bodies-normal   rows= 2   without payload=0
debug-key       rows= 1   without payload=0

The noLog row is the one worth checking: it still exports its summary, and carries no prompt.

Table layout, read back from BigQuery

timePartitioning : {"type":"DAY","field":"timestamp","expirationMs":"7776000000"}
clustering       : ["api_key_name","provider","model","status"]
schema           : 47 columns

One dependency to be aware of

The four pipeline_* columns only populate when pipeline capture is on
(call_log_pipeline_enabled, off by default). My first live run failed exactly those four
assertions and passed everything else: the export was correct, the data simply had not been
captured. Both states are visible in the CSV, since the two oldest rows predate enabling it.
request_body and response_body do not depend on that setting.

Safety

The CSV was scanned for service-account keys, bearer tokens, JWTs, provider keys and unredacted
auth headers before publishing. authorization reads [REDACTED] in the captured headers, which
is the existing payload protection applied at write time rather than anything the exporter does.

Assets live on a separate evidence/log-export-11945 branch of the fork and are not part of this
diff.

[dpozimski]

@diegosouzapw hi, pls code review. in summary i propose to have exporters to some external datasets (1st implementation - bigquery)