fix(sqlite): enable WAL + working busy timeout + _txlock=immediate to stop concurrent write lockouts - #7030
Merged
Conversation
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
Included review availability: Your plan provides up to 10 included reviews per hour; 7 remain after this review. WalkthroughSQLite now uses pragma-based busy waiting, WAL journaling, and immediate transaction locking. A regression test runs concurrent subscription read-then-write transactions and asserts that no transaction errors occur. ChangesSQLite concurrency handling
Estimated code review effort: 3 (Moderate) | ~20 minutes Merge Risk: ⚪ Minimal · up to This PR updates SQLite concurrency settings and adds a focused regression test; no actionable merge-blocking risk remains beyond normal checks and review. Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Linked Issues checkExplanation The changes directly address issue
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
… stop concurrent write lockouts SQLite under concurrent load (e.g. several /v1/responses requests settling quota and writing logs while background tasks update subscriptions) surfaces as `database is locked (SQLITE_BUSY)` and can cascade into `cannot start a transaction within a transaction` (QuantumNous#6805). Two root causes, both in the default DSN: 1. The busy timeout never actually applied. The DSN used the plain `_busy_timeout=30000` form, which modernc.org/sqlite (the driver behind github.com/glebarez/sqlite, used in production) silently ignores — only `_pragma=busy_timeout(...)` is honored. The effective timeout stayed at SQLite's 5s default, exactly the ~5s wait reported in QuantumNous#6805 before the lock error. 2. Even with a working busy timeout, a transaction that first SELECTs (establishing a read snapshot) and then writes can hit SQLITE_BUSY_SNAPSHOT when another connection commits in between; the busy handler does not cover that case, so the write fails instantly. The new DSN fixes both and adds WAL: - _pragma=busy_timeout(30000): makes the 30s busy timeout effective - _pragma=journal_mode(WAL): readers never block the single writer - _txlock=immediate: BEGIN IMMEDIATE takes the write lock up front, so writers serialize through the busy timeout instead of dying on a stale snapshot Regression test: model/sqlite_concurrency_test.go drives 8 connections x 50 read-then-write transactions. With the old DSN it fails with SQLITE_BUSY, with the new DSN it passes with zero errors. SQLite only; MySQL/PostgreSQL are unaffected.
LinineTy
force-pushed
the
pr/6805-sqlite-txlock
branch
from
August 26, 2026 20:51
70b44ba to
9ca2c5d
Compare
Calcium-Ion
approved these changes
Aug 30, 2026
5 tasks
chunfeng789
added a commit
to chunfeng789/new-api
that referenced
this pull request
Aug 30, 2026
* fix(web): restore admin unbinding for built-in providers (QuantumNous#6987) * fix(web): align admin binding types Refs QuantumNous#6985 * test(web): restore animation mock * fix(billing): 修复时间规则恒真表达式导致倍率全天生效 (QuantumNous#6934) Co-authored-by: seefs001 <i@seefs.me> * fix(docker): add relaykit go.mod to dev build context (QuantumNous#7072) * feat(task): replace built-in task adaptors with a sandboxed JS plugin system (QuantumNous#7076) * fix(relay): 请求参数校验错误返回 HTTP 400 (QuantumNous#6774) * fix(relay): return 400 for invalid request parameters * fix(web): recheck setup status after page reload (QuantumNous#6968) * feat(auth): encrypt password login transport Closes QuantumNous#6743 * feat(chat): add AQBot preset (QuantumNous#7079) * feat(auth): make password encryption opt-in QuantumNous#6743 * feat(task): resolve channel-mapped aliases and case variants for plugin models Channel model_mapping keys exposed in a channel's model list now act as first-class aliases for task-plugin models across the whole line: - Derived alias view (model/task_model_alias.go): built from enabled channels' model_mapping, chain-following with cycle detection, declared names always win, cross-plugin conflicts dropped. Rebuilt on channel cache refresh, registry generation change, and a 60s TTL. - Request path: PinTaskPluginEndpoint resolves declared-name case folds and mapping aliases before endpoint lookup (never rewriting the body until the endpoint is claimed), pins with MappedModel, and the decode contract accepts alias echoes without loosening model ownership for normal pins. Legacy /v1/tasks submit folds case variants the same way. Fixes aliases on POST /v1/responses silently falling through to the main relay against task channels. - Mapping order: ModelMappedHelper now runs before the plugin submit hook builds and caches the upstream body, so channel model_mapping actually reaches the upstream request. Plugins receive the mapped name as ctx.upstreamModel in both decode and submit contexts. - Billing: identity stays the origin name; when the alias has no tiered expression, the selected channel's mapping tail expression applies. Pricing page and billing-expr smoke tests resolve aliases to the owning plugin's usage schema. - Case folding: ASCII-only fold with exact-match priority; same-plugin and cross-plugin fold collisions rejected at registration. - Plugins: model-keyed rate tables, req_key derivation, and combo validation in doubao/kling/jimeng/hailuo/vidu/sunoapi now key on ctx.upstreamModel || ctx.model; render/echo paths keep ctx.model. * fix(model): disable PostgreSQL prepared statements for pooler compatibility GORM v1.25.2 closes cached prepared statements asynchronously on any SQL error and immediately re-Parses the same deterministic name (pgx's stmt_<sha256>) on the same client connection. Transaction-pooling proxies (PgBouncer >=1.21 with max_prepared_statements, Neon, Supabase) respond with FATAL "prepared statement name is already in use" (SQLSTATE 08P01) and drop the connection. PreferSimpleProtocol only disables pgx's implicit prepare and never covered GORM's explicit PrepareStmt cache. - PostgreSQL now runs with PrepareStmt disabled entirely; named prepared statements are fundamentally session state and cannot be made safe under transaction pooling. Parse/plan cost is noise for this workload. - Upgrade gorm to v1.25.12 so MySQL/SQLite statement caches (still enabled) no longer churn close/re-prepare on ordinary SQL errors; v1.25.9+ restricts eviction to driver.ErrBadConn. Deliberately not v1.26+, whose LRU eviction has an open use-after-close race (#7831). - sanitizeDBError now attaches a remediation hint on 08P01/42P05 so affected deployments can self-diagnose from the log line. * fix(ali): honor image response format (QuantumNous#5513) (QuantumNous#7048) * feat(web): factory task plugins update only with the system Marketplace install/upgrade on a factory-served plugin actually created a permanent override shadowing every future built-in release. The card now shows an informational "Updates with the system" badge instead of the action, while keeping the built-in vs marketplace version line and the upgradable state badge visible. Deliberate overrides are untouched: upload and marketplace actions on overridden or third-party plugins behave as before, and the plugins table now hints when an override lags behind the shipped built-in version so operators know deleting it restores the newer factory plugin. * fix(subscription): 无有效订阅时前端如实显示「仅用订阅」偏好 (QuantumNous#6222) (QuantumNous#7086) Co-authored-by: Claude <noreply@anthropic.com> * fix(sqlite): enable WAL + working busy timeout + _txlock=immediate to stop concurrent write lockouts (QuantumNous#7030) * fix(sqlite): enable WAL + working busy timeout + _txlock=immediate to stop concurrent write lockouts * fix(model): return string from JSON column Valuers for pg simple protocol With PrepareStmt disabled, PostgreSQL queries run over pgx's simple protocol, which encodes every []byte parameter as a bytea hex literal ('\x...'). driver.Valuer implementations returning []byte from json.Marshal therefore fail json-column writes with SQLSTATE 22P02 (reported on the channels UPDATE path via ChannelInfo). Reproduced against a live PostgreSQL 16: []byte Valuer into a json column fails under simple protocol, string succeeds; []byte into a text column silently stores the hex literal (no such path exists in the repo today — audited all Valuers, json.RawMessage fields, and raw SQL call sites). - ChannelInfo, Properties, TaskPrivateData, JSONValue Value() now return string; zero-value nil semantics unchanged. Task.Data (bare json.RawMessage) is unaffected — database/sql's default converter already passes it as expected. - Their Scan() counterparts now accept both []byte and string via a shared jsonScanBytes helper: SQLite returns string for these columns once Value() emits string, and the old []byte-only assertions silently zeroed the field (caught by the model test suite). - Add regression tests locking both contracts: json-column Valuers must return string (or nil for zero values), Scanners must accept []byte and string. Verified end-to-end against PostgreSQL 16 with the real model types: Channel create/update/read-back, Task json fields, PrefillGroup items. * fix(relay): bound the wait for upstream response headers (fixes unbounded heap growth → OOM) (QuantumNous#6949) * fix(relay): bound the wait for upstream response headers (fixes unbounded heap growth) The relay transport sets a dial timeout, a TLS handshake timeout and an expect-continue timeout, but nothing bounds how long it waits for the upstream *response headers* after the request has been written. An upstream that accepts the connection and then never answers -- without sending FIN/RST, which is what happens when a NAT/firewall silently drops the flow or the provider hangs -- parks the goroutine in net/http.(*persistConn).roundTrip forever. That goroutine keeps the whole request alive, which in practice means three copies of the request body stay reachable for the lifetime of the process: the raw bytes from io.ReadAll in CreateBodyStorageFromReader, the decoded messages held as json.RawMessage, and the re-marshalled upstream body from common.Marshal. BodyStorageCleanup cannot help here: it runs after c.Next() returns, and for these requests c.Next() never returns. Measured on v1.0.0-rc.23 in production (see QuantumNous#6947 for the full evidence): - 23 goroutines stuck in persistConn.roundTrip on a single 40h-old instance, blocked between 353 and 1894 minutes (5.9h to 31.5h) - 96.9% of the live heap, sampled after a forced GC, attributable to those three body copies (HeapAlloc 892 MiB surviving three GC cycles; HeapObjects dropping 30x while bytes dropped only 25%) - the live floor grows with uptime: 33.7 MiB at 0.1h, 89.2 at 13.8h, 510.0 at 40.1h, 955.2 at 146.8h, OOMKilled at 172.9h -- same image, same config, same load Doubling the memory limit and adding GOMEMLIMIT only moved the OOM from 132h to 172.9h. RELAY_TIMEOUT (http.Client.Timeout) cannot be used for this: it covers the whole response read and would cut legitimate long streaming calls, which is why it defaults to 0. ResponseHeaderTimeout only bounds the wait for the headers; streaming after they arrive is unaffected. The default is deliberately generous. Non-streaming upstreams usually send the response headers only once generation has finished, so the value has to leave room for a long completion. 1800s is 12x shorter than the shortest hang observed here while leaving several times the headroom a normal non-streaming request needs; 0 restores the previous unbounded behaviour. The assignment goes next to the other transport.* lines rather than inside the else branch: newRelayHTTPTransport() normally takes the http.DefaultTransport.Clone() path, and DefaultTransport does not set ResponseHeaderTimeout either. This repo already sets ResponseHeaderTimeout on its other outbound transports (controller/model_sync.go, controller/ratio_sync.go); the relay path appears to have been missed. Refs QuantumNous#6947. Likely also the root cause of QuantumNous#6731, which reported the same symptom (production OOM on /v1/responses after ~64h) but was closed for template reasons. * review: clamp overflowing timeout values and switch the test to testify Addresses the two CodeRabbit findings on this PR. Overflow (common/init.go:113): a RELAY_RESPONSE_HEADER_TIMEOUT beyond ~9.2e9 seconds overflows time.Duration and can wrap into a *tiny positive* timeout, which would cut every relay request instead of only the stuck ones. The value is now clamped before the conversion, with regression tests for both the negative and the overflowing input. I did not add fail-on-startup validation for negative values, for two reasons: the existing `if seconds > 0` guard already treats them as "disabled", and the neighbouring env-driven timeouts in this file are less strict still -- RelayIdleConnTimeout is converted with no guard at all. Failing startup on a bad value would be a behaviour change out of step with the rest of the file; happy to add it if you'd prefer that direction repo-wide. Test style: switched to testify (require.Equal / require.Zero / require.Positive), which is what every other test under service/ uses. go build, go vet and go test ./common/... ./service/... pass. (`go build ./...` fails on the `web/dist` embed both with and without this change -- the frontend bundle is not checked in.) * fix initialize database * fix(model): drop leftover prefill_groups unique constraints before AutoMigrate (QuantumNous#7100) * Revert "fix(model): drop leftover prefill_groups unique constraints before Au…" (QuantumNous#7101) This reverts commit 69a41ee. * fix(model): drop leftover prefill_groups unique constraints before AutoMigrate --------- Co-authored-by: zcxads666 <128150298+zcxads666@users.noreply.github.com> Co-authored-by: seefs001 <i@seefs.me> Co-authored-by: Uladzislau <53997152+VladKabiak@users.noreply.github.com> Co-authored-by: Calcium-Ion <i@caion.me> Co-authored-by: Alex Xiang <ax2@zicode.com> Co-authored-by: Seefs <40468931+seefs001@users.noreply.github.com> Co-authored-by: 憧憬Licoy <licoycn@gmail.com> Co-authored-by: PuppetKL <154485567+PuppetKL@users.noreply.github.com> Co-authored-by: ruiyunzhao <91191418+CR-Yun@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Xayinn <129403670+LinineTy@users.noreply.github.com> Co-authored-by: txgo <tianxi.liu@gmail.com>
drwoodck
pushed a commit
to drwoodck/new-api
that referenced
this pull request
Sep 2, 2026
… stop concurrent write lockouts (QuantumNous#7030) * fix(sqlite): enable WAL + working busy timeout + _txlock=immediate to stop concurrent write lockouts
yiranxiaohui
pushed a commit
to yiranxiaohui/new-api
that referenced
this pull request
Sep 2, 2026
… stop concurrent write lockouts (QuantumNous#7030) * fix(sqlite): enable WAL + working busy timeout + _txlock=immediate to stop concurrent write lockouts (cherry picked from commit 1751f43)
kiminbaek
pushed a commit
to kiminbaek/new-api
that referenced
this pull request
Sep 5, 2026
… stop concurrent write lockouts (QuantumNous#7030) * fix(sqlite): enable WAL + working busy timeout + _txlock=immediate to stop concurrent write lockouts
12 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📝 变更描述 / Description
SQLite 在并发负载下（多个 /v1/responses 同时结算额度、更新 Token、写请求日志，后台任务同时更新订阅/系统实例/任务锁）会报
database is locked (SQLITE_BUSY)，并可级联成cannot start a transaction within a transaction（#6805）。根因有两个，都在默认 DSN 里：
1. busy timeout 从来没真正生效。 原 DSN 是
one-api.db?_busy_timeout=30000，但生产用的纯 Go 驱动 modernc.org/sqlite（即 github.com/glebarez/sqlite）静默忽略_busy_timeout=这种裸参数，只认_pragma=busy_timeout(...)。所以实际超时一直是 SQLite 默认的 5 秒——正好对应 #6805 里描述的「等待约 5 秒后触发 SQLITE_BUSY」。2. 即使 busy timeout 生效，先读后写的事务仍会瞬间失败。 事务内先 SELECT（建立读快照）再写时，若另一连接在此期间提交写入，会命中
SQLITE_BUSY_SNAPSHOT；busy handler 不覆盖该场景，所以不管超时设多长都会立刻报错。新 DSN 一次解决两件事并补上 WAL：
_pragma=busy_timeout(30000)— 让 30s busy 超时真正生效_pragma=journal_mode(WAL)— 读不阻塞单写者，autocommit SELECT 保持并发_txlock=immediate— 事务开头即拿写锁（BEGIN IMMEDIATE），写者靠 busy timeout 排队，不再死于陈旧快照回归测试
model/sqlite_concurrency_test.go：8 连接 × 50 次「事务内先读后写」。旧 DSN 下必现 SQLITE_BUSY 失败，新 DSN 下 0 错误通过。仅影响 SQLite；MySQL/PostgreSQL 不受影响。
🚀 变更类型 / Type of change
🔗 关联任务 / Related Issue
✅ 提交前检查项 / Checklist
go build ./...通过；go test ./model/ -run TestSQLiteConcurrentReadThenWriteNoBusy通过；已用旧 DSN 验证测试能复现 SQLITE_BUSY。📸 运行证明 / Proof of Work
go1.26.1，本地 SQLite：
go build ./...✅go test ./model/ -run TestSQLiteConcurrentReadThenWriteNoBusy✅ 0 错误database is locked (5) (SQLITE_BUSY)，复现 database is locked (SQLITE_BUSY) cannot start a transaction within a transaction #6805Summary by CodeRabbit
Bug Fixes
Tests