Index messages by (session_id, created_timestamp, id) to stop on-disk sort storms

[vincenzopalazzo]

Issue: #11055 — filed for this PR per the contribution workflow in AGENTS.md; not yet triaged to Ready, so this PR is a proposal attached to it rather than agreed work.

Rebased onto main.

Verification after rebase

• Conflict resolved by hand: main added messages_read_back_in_the_order_they_arrived in the same block; both tests are kept.
• Re-confirmed the premise still holds on current main: the read path is still ORDER BY created_timestamp, id and the index is still absent. main's new per-insert SELECT MAX(created_timestamp) FROM messages WHERE session_id = ? in add_message is additionally covered by this index.
• cargo test -p goose --lib session::session_manager — 44 passed.
• cargo clippy -p goose --lib --tests -- -D warnings — clean.

---

Summary

• Loading a session's messages runs WHERE session_id = ? ORDER BY created_timestamp, id, but no index covered created_timestamp — the only timestamp index (idx_messages_timestamp) is on a different column (timestamp).
• SQLite fell back to an on-disk TEMP B-TREE sort dragging each row's content_json through temp files. With large tool responses stored inline (observed: a single ~479 MB toolResponse), this produced multi-GB write storms that exhausted macOS disk-write limits and stalled goose — confirmed by a disk writes resource event (34.36 GB in ~55 min, 100% of samples in sqlite3VdbeSorterWrite → vdbeSorterFlushPMA → pwrite).
• Adds a composite index (session_id, created_timestamp, id) so the planner walks the index instead of sorting. Verified the plan goes from USE TEMP B-TREE FOR ORDER BY → SEARCH messages USING INDEX idx_messages_session_created. Added in both the fresh-schema path and a new migration (v16).

Review Notes

• Production-safety review passed (1 round). Additive index only — no data transform.
• CREATE INDEX IF NOT EXISTS is idempotent; fresh DBs (via create_schema) and existing v15 DBs (via migration 16) both covered, mirroring idx_sessions_parent.

Decision Log

Hardest decision: Keeping idx_messages_session rather than dropping it. The new composite index (session_id, …) covers any pure WHERE session_id = ? lookup, so idx_messages_session is now largely redundant and costs one write per INSERT. I kept it to keep this PR a pure, zero-risk addition — the session-listing LEFT JOIN relies on it, and removing it would change that query plan. Dropping it is a follow-up optimization.

Alternatives rejected:

• Bare (created_timestamp) index: would help a global ORDER BY created_timestamp but not the dominant per-session WHERE session_id = ? ORDER BY created_timestamp, id (still needs a sort after filtering). The composite index covers both filter and ordering.
• Dropping idx_messages_session: less write amplification, but risks the session-listing JOIN planner and is out of scope for a crash fix.

Least confident about: One-time migration cost on pathologically large DBs. CREATE INDEX scans all rows once (it only reads the indexed columns, so the resulting index is tiny — ~3 MB for ~100k rows), but on a ~1GB DB it is a brief lock at next launch. Acceptable and one-time.

Test plan

• New regression test test_messages_session_created_index_avoids_disk_sort asserts the index exists and EXPLAIN QUERY PLAN uses it with no TEMP B-TREE (fails without the index).
• Full session::session_manager suite (40 tests) passes.
• cargo fmt, cargo clippy -p goose --all-targets -- -D warnings clean.

Note

Fixes the crash mechanism. The oversized toolResponse that bloated the DB to 985 MB is a separate data-hygiene issue worth a follow-up (cap/externalize oversized tool results at write time). Existing bloated DBs can be trimmed with VACUUM after truncating the offending row.

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 58e6f704c3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Defer implementation until the issue is Ready

For this upstream change, the reviewed commit message explicitly says issue #11055 is “not yet triaged to Ready,” yet this line begins implementing it. Pause the implementation until the issue reaches Ready and its design, constraints, and verification plan are agreed; proceeding now violates the repository’s contribution workflow.

AGENTS.md reference: AGENTS.md:L9-L12

Useful? React with 👍 / 👎.

[lifeizhou-ap]

it is in "Ready" status now