fix: reduce FTS index left() cap from 800k to 250k chars to stay within tsvector limit - #4057
Merged
Merged
Conversation
Contributor
Author
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughAdds an unexported ftsInputCharLimit constant and applies it to three PostgreSQL GIN index definitions and the matching to_tsvector calls in query predicates so indexed expressions and search predicates use the same left(..., ftsInputCharLimit) truncation. ChangesFull-text search truncation alignment
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 golangci-lint (2.12.2)level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain main module or its selected dependencies" Comment |
impoiler
changed the title
left() cap from 800k to 250k chars to stay within tsvector limit
impoiler
marked this pull request as ready for review
June 4, 2026 14:24
Contributor
Confidence Score: 4/5Safe to merge for deployments where the old indexes failed to build; deployments with previously successful The constant introduction and query-predicate alignment are correct. The gap is that framework/logstore/migrations.go — the Important Files Changed
Reviews (9): Last reviewed commit: "fix/logs-content-fts-tsvector-overflow" | Re-trigger Greptile |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@framework/logstore/migrations.go`:
- Around line 2407-2417: The FTS index expressions added in migrations.go use
to_tsvector('simple', left(..., 250000)) but the ContentSearch predicates in
framework/logstore/rdb.go still use uncapped to_tsvector('simple',
content_summary|arguments|result), so Postgres may not use the new indexes;
update the query predicates in the ContentSearch implementation to use the exact
same expressions (e.g. to_tsvector('simple', left(content_summary, 250000)),
to_tsvector('simple', left(arguments, 250000)), to_tsvector('simple',
left(result, 250000))) or factor a shared SQL fragment/constant used by both the
creator (migrations) and the search queries so the expressions match exactly and
the GIN expression indexes are used.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: f7aca1d3-5f92-46fd-9b94-208ad3b982a5
📒 Files selected for processing (1)
framework/logstore/migrations.go
impoiler
marked this pull request as draft
June 4, 2026 14:41
impoiler
force-pushed
the
06-04-fix_logs-content-fts-tsvector-overflow
branch
2 times, most recently
from
June 5, 2026 06:13
b5d941f to
eada2eb
Compare
impoiler
marked this pull request as ready for review
June 5, 2026 06:20
18 tasks
impoiler
force-pushed
the
06-04-fix_logs-content-fts-tsvector-overflow
branch
from
June 5, 2026 12:03
eada2eb to
257b83c
Compare
Merged
18 tasks
impoiler
force-pushed
the
06-04-fix_logs-content-fts-tsvector-overflow
branch
from
June 5, 2026 12:21
257b83c to
104ba11
Compare
impoiler
force-pushed
the
06-04-fix_logs-content-fts-tsvector-overflow
branch
from
June 5, 2026 13:12
104ba11 to
d649120
Compare
impoiler
force-pushed
the
graphite-base/4057
branch
from
June 5, 2026 13:12
1afd0ad to
c829950
Compare
impoiler
changed the base branch from
graphite-base/4057
to
06-05-fix_bug_add_alias_filter_support_for_matview_queries
June 5, 2026 13:12
This was referenced Jun 5, 2026
impoiler
force-pushed
the
06-04-fix_logs-content-fts-tsvector-overflow
branch
from
June 5, 2026 13:51
b954d74 to
6195969
Compare
impoiler
force-pushed
the
06-05-fix_bug_add_alias_filter_support_for_matview_queries
branch
from
June 5, 2026 13:51
9be784d to
513f10e
Compare
impoiler
force-pushed
the
06-04-fix_logs-content-fts-tsvector-overflow
branch
from
June 5, 2026 13:53
6195969 to
a2935d2
Compare
Contributor
Merge activity
|
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@framework/logstore/migrations.go`:
- Around line 345-355: Don't drop live filter matviews in
migrationRecreateFilterTeamBUMatViews and
migrationRecreateFilterCustomersMatView during triggerMigrations; instead avoid
DROP MATERIALIZED VIEW in those functions and either (a) perform an atomic
replace (create the new matview under a temporary name and then SWAP or ALTER to
replace the live name), or (b) defer wiring these migrations so ensureMatViews
(or the post-startup repair path) performs the rebuild/CREATE safely after the
new pod is ready. Update migrationRecreateFilterTeamBUMatViews and
migrationRecreateFilterCustomersMatView to remove unconditional DROP statements
and implement one of the atomic-replace or defer strategies so no cluster-wide
relation-missing window occurs while older replicas still serve requests.
- Around line 2039-2048: The Rollback function is dropping large Postgres GIN
indexes with plain DROP INDEX which can block; update the two tx.Exec calls in
the Rollback (the ones referencing idx_logs_team_ids_gin and
idx_logs_business_unit_ids_gin) to use "DROP INDEX CONCURRENTLY IF EXISTS
<index>" and ensure this operation is executed outside a transaction (confirm
the migration has transactions disabled or the migration runner will run
non-transactionally) so the CONCURRENTLY option is valid and avoids blocking.
In `@framework/logstore/rdb.go`:
- Around line 131-143: The SRF calls in teamOrBUFanoutFrom can error on
non-array text values; change the jsonb_array_elements_text inputs (the
occurrences that use arrIDs and arrNames) to be wrapped in a CASE that yields
the column cast to jsonb only when it IS NOT NULL AND IS JSON ARRAY, otherwise
return '[]'::jsonb; e.g. replace l.%[1]s::jsonb and l.%[2]s::jsonb with a CASE
expression that guards and returns '[]'::jsonb for invalid values, leaving the
rest of the query (including the UNION branch using idCol and scalarName)
unchanged so malformed legacy values no longer cause the SRF to run on bad
input.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 8b1840ac-922c-4535-9002-ea5b39b648ab
📒 Files selected for processing (2)
framework/logstore/migrations.goframework/logstore/rdb.go
Contributor
There was a problem hiding this comment.
Caution
Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@framework/logstore/migrations.go`:
- Around line 345-355: Don't drop live filter matviews in
migrationRecreateFilterTeamBUMatViews and
migrationRecreateFilterCustomersMatView during triggerMigrations; instead avoid
DROP MATERIALIZED VIEW in those functions and either (a) perform an atomic
replace (create the new matview under a temporary name and then SWAP or ALTER to
replace the live name), or (b) defer wiring these migrations so ensureMatViews
(or the post-startup repair path) performs the rebuild/CREATE safely after the
new pod is ready. Update migrationRecreateFilterTeamBUMatViews and
migrationRecreateFilterCustomersMatView to remove unconditional DROP statements
and implement one of the atomic-replace or defer strategies so no cluster-wide
relation-missing window occurs while older replicas still serve requests.
- Around line 2039-2048: The Rollback function is dropping large Postgres GIN
indexes with plain DROP INDEX which can block; update the two tx.Exec calls in
the Rollback (the ones referencing idx_logs_team_ids_gin and
idx_logs_business_unit_ids_gin) to use "DROP INDEX CONCURRENTLY IF EXISTS
<index>" and ensure this operation is executed outside a transaction (confirm
the migration has transactions disabled or the migration runner will run
non-transactionally) so the CONCURRENTLY option is valid and avoids blocking.
In `@framework/logstore/rdb.go`:
- Around line 131-143: The SRF calls in teamOrBUFanoutFrom can error on
non-array text values; change the jsonb_array_elements_text inputs (the
occurrences that use arrIDs and arrNames) to be wrapped in a CASE that yields
the column cast to jsonb only when it IS NOT NULL AND IS JSON ARRAY, otherwise
return '[]'::jsonb; e.g. replace l.%[1]s::jsonb and l.%[2]s::jsonb with a CASE
expression that guards and returns '[]'::jsonb for invalid values, leaving the
rest of the query (including the UNION branch using idCol and scalarName)
unchanged so malformed legacy values no longer cause the SRF to run on bad
input.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 8b1840ac-922c-4535-9002-ea5b39b648ab
📒 Files selected for processing (2)
framework/logstore/migrations.goframework/logstore/rdb.go
🛑 Comments failed to post (3)
framework/logstore/migrations.go (2)
345-355:
⚠️ Potential issue | 🟠 Major | 🏗️ Heavy liftDon't drop live filter matviews before the deferred rebuild runs.
migrationRecreateFilterTeamBUMatViewsandmigrationRecreateFilterCustomersMatViewonlyDROP MATERIALIZED VIEW; they rely onensureMatViewslater to recreate them. BecausetriggerMigrations()runs before the new pod starts serving while older replicas may still be answering filter-data requests, the first upgraded pod removesmv_filter_teams,mv_filter_business_units, andmv_filter_customerscluster-wide and leaves a relation-missing window until the post-startup repair finishes. That's the same rolling-deploy failure mode the nearbymigrationSplitFilterDataMatViewcomment explicitly avoids. Keep these views in place until the repair path can replace them atomically, or defer wiring these migrations until the rebuild can happen without a gap.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@framework/logstore/migrations.go` around lines 345 - 355, Don't drop live filter matviews in migrationRecreateFilterTeamBUMatViews and migrationRecreateFilterCustomersMatView during triggerMigrations; instead avoid DROP MATERIALIZED VIEW in those functions and either (a) perform an atomic replace (create the new matview under a temporary name and then SWAP or ALTER to replace the live name), or (b) defer wiring these migrations so ensureMatViews (or the post-startup repair path) performs the rebuild/CREATE safely after the new pod is ready. Update migrationRecreateFilterTeamBUMatViews and migrationRecreateFilterCustomersMatView to remove unconditional DROP statements and implement one of the atomic-replace or defer strategies so no cluster-wide relation-missing window occurs while older replicas still serve requests.
2039-2048:
⚠️ Potential issue | 🟡 Minor | ⚡ Quick winUse
DROP INDEX CONCURRENTLYin these rollback paths.These are large Postgres GIN indexes on
logs, and the migrations are already non-transactional. PlainDROP INDEXcan still wait on active sessions and create avoidable blocking during downgrade/rollback.♻️ Proposed fix
Rollback: func(tx *gorm.DB) error { tx = tx.WithContext(ctx) if tx.Dialector.Name() == "postgres" { - if err := tx.Exec("DROP INDEX IF EXISTS idx_logs_team_ids_gin").Error; err != nil { + if err := tx.Exec("DROP INDEX CONCURRENTLY IF EXISTS idx_logs_team_ids_gin").Error; err != nil { return fmt.Errorf("failed to drop team_ids GIN index: %w", err) } - if err := tx.Exec("DROP INDEX IF EXISTS idx_logs_business_unit_ids_gin").Error; err != nil { + if err := tx.Exec("DROP INDEX CONCURRENTLY IF EXISTS idx_logs_business_unit_ids_gin").Error; err != nil { return fmt.Errorf("failed to drop business_unit_ids GIN index: %w", err) } } return nil }, @@ Rollback: func(tx *gorm.DB) error { tx = tx.WithContext(ctx) if tx.Dialector.Name() == "postgres" { - if err := tx.Exec("DROP INDEX IF EXISTS idx_logs_customer_ids_gin").Error; err != nil { + if err := tx.Exec("DROP INDEX CONCURRENTLY IF EXISTS idx_logs_customer_ids_gin").Error; err != nil { return fmt.Errorf("failed to drop customer_ids GIN index: %w", err) } } return nil },Based on learnings: Applies to framework/persistence/migrations/**/*.{go,sql} : Whenever a migration is added or changed, verify it avoids deadlocks and long blocking locks on large tables. Index creation in migrations must be concurrent where the database supports it.
Also applies to: 3587-3592
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@framework/logstore/migrations.go` around lines 2039 - 2048, The Rollback function is dropping large Postgres GIN indexes with plain DROP INDEX which can block; update the two tx.Exec calls in the Rollback (the ones referencing idx_logs_team_ids_gin and idx_logs_business_unit_ids_gin) to use "DROP INDEX CONCURRENTLY IF EXISTS <index>" and ensure this operation is executed outside a transaction (confirm the migration has transactions disabled or the migration runner will run non-transactionally) so the CONCURRENTLY option is valid and avoids blocking.framework/logstore/rdb.go (1)
131-143:
⚠️ Potential issue | 🟠 Major | ⚡ Quick win🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail # 1) Verify whether array-shape constraints exist for fan-out columns. rg -n "team_ids|team_names|customer_ids|customer_names|business_unit_ids|business_unit_names|CHECK|IS JSON ARRAY" framework/logstore/migrations.go framework/logstore/tables.go # 2) Confirm all jsonb_array_elements_text call sites in this store path. rg -n "jsonb_array_elements_text\\(" framework/logstore/rdb.goRepository: maximhq/bifrost
Length of output: 3506
🏁 Script executed:
#!/bin/bash set -euo pipefail # Look for any CHECK constraints / DDL enforcement for JSON-array shape on the fan-out columns. rg -n "CHECK \\(|jsonb_array_elements_text|IS JSON ARRAY" framework/logstore/migrations.go # Specifically search for constraints mentioning the columns. rg -n "team_ids|customer_ids|business_unit_ids" framework/logstore/migrations.go | head -n 200 # Inspect nearby definitions in tables.go for DB column types/tags. sed -n '1,120p' framework/logstore/tables.go sed -n '220,320p' framework/logstore/tables.go # Print the exact snippet in rdb.go around the reported lines for context. sed -n '110,170p' framework/logstore/rdb.goRepository: maximhq/bifrost
Length of output: 16530
🏁 Script executed:
#!/bin/bash set -euo pipefail # Inspect Log struct fields for storage types/tags around the team/customer/bu columns. sed -n '1,120p' framework/logstore/tables.go | cat sed -n '120,220p' framework/logstore/tables.go | cat rg -n "TeamIDs\\b|CustomerIDs\\b|BusinessUnitIDs\\b|team_ids\\b|customer_ids\\b|business_unit_ids\\b" framework/logstore/tables.go # Search for DDL/constraints enforcement in migrations. rg -n "CREATE TABLE logs|ALTER TABLE logs|CHECK \\(|team_ids|customer_ids|business_unit_ids" framework/logstore/migrations.go # Specifically look for CHECK constraints on those columns. rg -n "CHECK \\(|team_ids|customer_ids|business_unit_ids" framework/logstore/migrations.go | head -n 200Repository: maximhq/bifrost
Length of output: 20377
🏁 Script executed:
#!/bin/bash set -euo pipefail # 1) Find migrations that add/alter team_ids/customer_ids/business_unit_ids columns (DDL). rg -n "ALTER TABLE logs .*team_ids|ADD COLUMN team_ids|team_ids.*type|customer_ids.*type|business_unit_ids.*type" framework/logstore/migrations.go | head -n 200 # 2) Find where those columns are backfilled/updated (may show expected JSON array shape enforcement). rg -n "team_ids|customer_ids|business_unit_ids" framework/logstore/migrations.go | head -n 200 # 3) Inspect serialization/deserialization for these JSON text fields in tables.go. sed -n '520,700p' framework/logstore/tables.go sed -n '820,940p' framework/logstore/tables.goRepository: maximhq/bifrost
Length of output: 41
🏁 Script executed:
#!/bin/bash set -u # 1) Show all occurrences for these column names in migrations (trimmed). rg -n "team_ids|customer_ids|business_unit_ids" framework/logstore/migrations.go | head -n 200 # 2) Look for any ALTER/ADD/CREATE statements involving logs table + those columns. rg -n "(ALTER TABLE logs|ADD COLUMN|ALTER COLUMN|DROP COLUMN|CREATE INDEX|CREATE TABLE logs).*?(team_ids|customer_ids|business_unit_ids)" framework/logstore/migrations.go | head -n 200 || true # 3) Check if any migrations add CHECK constraints involving JSON. rg -n "CHECK \\(|IS JSON ARRAY|jsonb_array_elements_text|::jsonb" framework/logstore/migrations.go | head -n 200Repository: maximhq/bifrost
Length of output: 2134
🏁 Script executed:
#!/bin/bash set -euo pipefail # Locate logs table DDL / column creation in migrations.go (if any). rg -n "CREATE TABLE (IF NOT EXISTS )?logs" framework/logstore/migrations.go || true rg -n "logs.*team_ids|team_ids.*logs|logs.*customer_ids|customer_ids.*logs|logs.*business_unit_ids|business_unit_ids.*logs" framework/logstore/migrations.go | head -n 200 || true # Inspect how Log.SerializeFields/DeserializeFields handle TeamIDs/CustomerIDs/BusinessUnitIDs. sed -n '560,690p' framework/logstore/tables.go sed -n '840,930p' framework/logstore/tables.goRepository: maximhq/bifrost
Length of output: 7960
🏁 Script executed:
#!/bin/bash set -euo pipefail # 1) Find where these columns are set/updated in the Go code (besides Log.SerializeFields). rg -n "team_ids\\b|customer_ids\\b|business_unit_ids\\b" framework/logstore --glob='*.go' | head -n 200 # 2) Look for migrations/backfills that populate these columns from scalar/legacy columns. rg -n "team_ids\\b|customer_ids\\b|business_unit_ids\\b" framework/logstore/migrations.go | head -n 200 # 3) Specifically look for SQL that sets them (UPDATE/SET/to_json/jsonb/json_build_array patterns). rg -n "(UPDATE|SET|jsonb_build_array|json_build_array|to_jsonb|\\[\\]|\\barray\\b|::jsonb)" framework/logstore/migrations.go | rg -n "(team_ids|customer_ids|business_unit_ids)" || trueRepository: maximhq/bifrost
Length of output: 6300
🏁 Script executed:
#!/bin/bash set -euo pipefail # Find where "IS JSON ARRAY" is used at runtime in matviews or SQL builders. rg -n "team_ids IS JSON ARRAY|team_ids\\s+IS\\s+JSON\\s+ARRAY|customer_ids IS JSON ARRAY|business_unit_ids IS JSON ARRAY" framework/logstore | head -n 200 # Inspect the fan-out aggregation SQL builder and any matview SQL selection that likely handles scalar-vs-array. sed -n '1,120p' framework/logstore/multi_team_matview_test.go sed -n '1,120p' framework/logstore/multi_team_filter_test.go # Inspect rdb.go for multiValueDimensionFilterSQL and for any other similar SRF usage with JSON guards. sed -n '1,260p' framework/logstore/rdb.goRepository: maximhq/bifrost
Length of output: 20030
Guard fan-out SRF JSON casts in teamOrBUFanoutFrom
In
framework/logstore/rdb.go(teamOrBUFanoutFrom), thejsonb_array_elements_text(l.%[1]s::jsonb)/jsonb_array_elements_text(l.%[2]s::jsonb)SRFs run in the FROM clause before theWHERE l.%[1]s IS NOT NULL AND l.%[1]s IS JSON ARRAYguard. If any non-NULL legacy/malformed value exists inteam_ids/customer_ids/business_unit_ids(stored astype:text), the query can error and abort. Wrap the SRF inputs with aCASEso invalid/non-array values map to'[]'::jsonb.💡 Suggested fix
- SELECT t.value AS dim_id, COALESCE(n.value, '') AS dim_name - FROM jsonb_array_elements_text(l.%[1]s::jsonb) WITH ORDINALITY AS t(value, ord) - LEFT JOIN jsonb_array_elements_text(l.%[2]s::jsonb) WITH ORDINALITY AS n(value, ord) ON n.ord = t.ord + SELECT t.value AS dim_id, COALESCE(n.value, '') AS dim_name + FROM jsonb_array_elements_text( + CASE + WHEN l.%[1]s IS NOT NULL AND l.%[1]s IS JSON ARRAY THEN l.%[1]s::jsonb + ELSE '[]'::jsonb + END + ) WITH ORDINALITY AS t(value, ord) + LEFT JOIN jsonb_array_elements_text( + CASE + WHEN l.%[2]s IS NOT NULL AND l.%[2]s IS JSON ARRAY THEN l.%[2]s::jsonb + ELSE '[]'::jsonb + END + ) WITH ORDINALITY AS n(value, ord) ON n.ord = t.ord🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@framework/logstore/rdb.go` around lines 131 - 143, The SRF calls in teamOrBUFanoutFrom can error on non-array text values; change the jsonb_array_elements_text inputs (the occurrences that use arrIDs and arrNames) to be wrapped in a CASE that yields the column cast to jsonb only when it IS NOT NULL AND IS JSON ARRAY, otherwise return '[]'::jsonb; e.g. replace l.%[1]s::jsonb and l.%[2]s::jsonb with a CASE expression that guards and returns '[]'::jsonb for invalid values, leaving the rest of the query (including the UNION branch using idCol and scalarName) unchanged so malformed legacy values no longer cause the SRF to run on bad input.
akshaydeo
changed the base branch from
06-05-fix_bug_add_alias_filter_support_for_matview_queries
to
graphite-base/4057
June 5, 2026 14:19
This was referenced Jun 5, 2026
18 tasks
akshaydeo
added a commit
that referenced
this pull request
Jun 6, 2026
## Summary This PR bumps the Go toolchain version from `1.26.3` to `1.26.4` across all modules and CI workflows, and cuts a new release (`core` v1.5.17, `framework` v1.3.17, `transports` v1.5.9, `plugins/compat` v0.1.16, `plugins/governance` v1.5.17, and associated plugin versions) incorporating a large batch of features and fixes accumulated since the previous release. ## Changes - **Go 1.26.4** — Updated `go-version` in all GitHub Actions workflows (`e2e-tests`, `helm-release`, `pr-tests`, `release-cli`, `release-pipeline`, `snyk`) and all `go.mod` files (core, framework, transports, cli, all plugins, examples, and test modules). - **Core (v1.5.17)** — OpenAI compaction support, multi-customer logs and usage tracking, multiple team/business unit support, `request_headers` wildcard pattern capture for OTel and Maxim plugins, xAI `x_search` tool, fetch URL validation with SSRF hardening, `file://` pricing URL scheme, virtual key provider fan-out filtering, and a broad set of fixes including Anthropic prompt cache key, empty thinking block stripping, OpenAI stream usage event cleanup, Gemini numeric schema constraints, stale connection retries, Azure Claude diagnostic strip, and passthrough budget handling. - **Framework (v1.3.17)** — Scope-aware budgets and limits wired from model configs, provider-level governance, multiple customer budget support with `calendar_aligned` windows, paginated virtual key fetch, `config.json` source-of-truth flow, FTS index cap reduction, sync worker drift fix, cascade deletes for model configs, and high-scale virtual key flow improvements. - **Transports (v1.5.9)** — Full changelog covering all of the above plus UI improvements (log navigation, customer detail sheet, `BudgetDisplay` component, inline loading shell, materialized view alias), SCIM provisioning fields, Helm/config schema additions (`roles`, `per_user_oauth`), client IP resolution from forwarded headers, and dependency upgrades (`recharts` to 3.8.1, `golang.org/x` CVE remediation). - **Plugins** — `governance` v1.5.17 adds team budget/rate-limit exporters, ghost node reconciliation fix, and VK double usage counting fix; `logging` v1.5.17 adds wildcard header capture and file attachment rendering; `otel` v1.2.17 adds `disable_content_logging` and multiple collectors support; `maxim` v1.6.17 adds `request_headers` wildcard capture; `compat` v0.1.16 fixes `max_tokens` preservation during param filtering. ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [x] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [x] Providers/Integrations - [x] Plugins - [x] UI (React) - [ ] Docs ## How to test ```sh # Verify Go version go version # should report go1.26.4 # Run core tests cd core && go test ./... # Run framework tests cd framework && go test ./... # Run transports tests cd transports && go test ./... # Run plugin tests cd plugins/governance && go test ./... cd plugins/logging && go test ./... cd plugins/otel && go test ./... # UI cd ui pnpm i pnpm build pnpm test ``` ## Breaking changes - [ ] Yes - [x] No ## Related issues #4053, #4066, #4041, #4012, #3976, #3947, #3991, #4045, #3957, #3938, #3937, #3939, #3981, #3998, #3997, #4092, #4091, #4079, #4080, #4086, #3929, #3994, #4028, #3970, #3919, #3861, #3664, #3999, #4088, #4070, #4051, #4043, #4057, #4023, #3941, #3955, #4024, #3956, #3967, #3925, #3992, #3900 ## Security considerations - Fetch URL validation hardened against SSRF by tightening IP checks for private networks and link-local addresses (#4092, #3947, #3991). - Transitive `golang.org/x` dependencies (crypto, net, sys, text) bumped to address Docker Scout CVEs (#3900). ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [x] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [x] I verified the CI pipeline passes locally if applicable <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * OpenAI compaction, multi-customer/team logstore support, request-header wildcard capture, enhanced governance (provider-level & scope-aware limits), disable-content-logging option, support for multiple OpenTelemetry collectors, SSRF hardening and URL validation. * **Chores** * Bumped Go toolchain across modules and updated component/plugin version releases. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Merged
akshaydeo
pushed a commit
that referenced
this pull request
Jun 7, 2026
…thin tsvector limit (#4057) ## Summary Fixes a `CREATE INDEX CONCURRENTLY` failure when building the full-text search GIN indexes on the `logs` and `mcp_tool_logs` tables. Postgres aborts index creation with: ``` ERROR: string is too long for tsvector (1118246 bytes, max 1048575 bytes) ``` The `left(col, 800000)` cap counts **characters**, but `to_tsvector` enforces a **1,048,575-byte** limit on its **output**. Multi-byte UTF-8 / high-lexeme content can expand 800k chars well past 1 MB, aborting the build and leaving full-text search to fall back to slow sequential scans. ## Changes - Lowered the `left()` input cap from `800000` → `250000` chars on all three FTS GIN indexes: - `idx_logs_content_summary_fts` (logs.content_summary) - `idx_mcp_logs_arguments_fts` (mcp_tool_logs.arguments) - `idx_mcp_logs_result_fts` (mcp_tool_logs.result) - 250k chars keeps even worst-case content (short unique tokens, full multi-byte) safely under the 1 MB tsvector ceiling, while still covering ~50 pages of searchable text per log entry. ## Type of change - [x] Bug fix - [ ] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh go build ./framework/logstore/ ``` On a Postgres deployment, confirm the indexes build successfully on startup (via `ensurePerformanceIndexes`) without the tsvector size error, including on databases that previously failed: ```sql \di idx_logs_content_summary_fts \di idx_mcp_logs_arguments_fts \di idx_mcp_logs_result_fts ``` Note: `CREATE INDEX CONCURRENTLY IF NOT EXISTS` is a no-op if an index already exists, so any invalid index left behind by the prior failed build should be dropped before the corrected one is created. ## Breaking changes - [ ] Yes - [x] No ## Related issues N/A ## Security considerations None. Affects only index input truncation length. ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [ ] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved full-text search reliability by enforcing a consistent input-length cap for indexed content and matching queries, preventing mismatches or failures with very large text fields and ensuring search predicates align with index definitions. Non-PostgreSQL (LIKE-based) search behavior remains unchanged. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
akshaydeo
added a commit
that referenced
this pull request
Jun 7, 2026
## ✨ Features - **OpenAI Compaction** — Added OpenAI conversation compaction support across core, framework, logging, and the API surface (#4053) - **Multi-Customer & Org Hierarchy** — Logs and usage tracking now support multiple customers, teams, and business units, including business unit CRUD, team assignment, and governance endpoints in the OpenAPI spec (#4066, #4041, #4082) - **Provider-Level Governance** — Budgets & limits are now scope-aware and can be applied at the virtual-key top level and per provider, wired from the model configs table, with UI filters for scope and providers (#3938, #3937, #3939, #3981, #3962) - **Customer Budgets** — Customers support multiple budgets and `calendar_aligned` budget windows (#3998, #3997) - **Virtual Key Attribution & Controls** — Added a `created_by` user attribution column and a `blacklisted_models` column for virtual key provider configs (#3672, #3653) - **Request Header Capture** — OTel and Maxim observability plugins capture `request_headers` by pattern, with wildcard support (e.g. `x-custom-*`); logging gained the same wildcard header capture (#4012, #3958) - **OTel Content Controls & Collectors** — New `disable_content_logging` option drops message/tool content from exported spans, plus support for multiple OTel collectors (#4064, #3894) - **xAI x_search** — Added xAI `x_search` tool support (#3976) - **URL Validation** — Added fetch URL validation with private-network configuration and link-local blocking (#3947, #3991) - **File Scheme Pricing URLs** — Pricing source URLs now accept the `file://` scheme for air-gapped and self-hosted deployments (#4045) - **Paginated Virtual Keys** — Virtual key fetching is paginated to handle deployments with very large numbers of keys (#3957) - **Client IP Resolution** — Resolve client IP from `X-Forwarded-For`/`X-Real-IP` headers - **SCIM Provisioning** — Added `attributeType`/`attributeValue` SCIM provisioning fields - **Helm/Config Schema** — Added `roles` RBAC governance config and `per_user_oauth` MCP auth to the Helm chart and config schema (#4004, #4009) - **Log Navigation UI** — Added a "View logs" menu item to customer, team, and virtual key tables, clickable links in log detail views, a customer detail sheet, and a reusable `BudgetDisplay` component (#4073, #4054, #4026, #4055) - **Faster First Paint** — Added an inline loading shell to `#root` before React mounts (#4063) - **Materialized View Alias** — Added an `alias` column to the materialized view with filter support (#4078) ## 🐞 Fixed - **Fetch URL IP Checks** — Hardened fetch URL IP checks against SSRF (#4092) - **Mantle Model Matching** — Broadened Mantle model matching to all `gpt` variants (#4091) - **Empty Thinking Blocks** — Strip thinking blocks when the signature is empty (#4079) - **OpenAI Stream Usage** — Removed usage from the `responses.created` event in the OpenAI stream (#4080) - **Prompt Cache Key** — Set the prompt cache key from the Anthropic integration (#4086) - **Upstream Failure Status** — Map upstream connection failures to 502 instead of 400 (#3929) (thanks [@chris-colinsky](https://github.com/chris-colinsky)!) - **Gemini Schema Constraints** — Accept numeric schema integer constraints for Gemini (#3994) (thanks [@yanhao98](https://github.com/yanhao98)!) - **Files Provider Param** — Accept the `?provider=` query param on `GET /v1/files` (#3971) (thanks [@alexef](https://github.com/alexef)!) - **Optional Batch Model** — Made the `model` field optional on `POST /v1/batches` (#3973) (thanks [@alexef](https://github.com/alexef)!) - **Helm Azure Config** — Added missing `azure_key_config` fields to the Helm schema (#3996) (thanks [@axelray-dev](https://github.com/axelray-dev)!) - **Text Completion Chunk Model** — Added the missing `Model` field to `TextCompletionChunkResponse` (#3970) (thanks [@kuishou68](https://github.com/kuishou68)!) - **MCP Inline stdio Env** — MCP stdio server configs accept inline environment variable assignments (#3861) (thanks [@Shushmitaaaa](https://github.com/Shushmitaaaa)!) - **Orphaned Tool Results** — Orphaned tool results in the OpenAI to Anthropic conversion flow are no longer rejected by the Anthropic API (#3919) - **Node Usage Reconciliation** — Added a monotonic `inc_number` log cursor so node usage reconciliation does not skip late async log writes (#3664) - **Bedrock Output Assessments** — Corrected the type of `outputAssessments` in Bedrock responses (#4028) - **Model Pool Pricing Reloads** — Preserve non-pricing model pool entries across pricing reloads (#3999) - **Ghost Node Reconciliation** — Replicate the VK hierarchy flow for ghost node reconciliation (#4088) - **VK Double Usage Counting** — Fixed double usage counting when creating a virtual key (#4070) - **Model Config Lifecycle** — Cascade deletes for model configs and removal of stale in-memory model configs (#4051, #4043) - **FTS Index Cap** — Reduced the FTS index `left()` cap from 800k to 250k chars to stay within the tsvector limit (#4057) - **Sync Worker Drift** — Reduced the sync worker ticker period to 5m to prevent threshold drift (#4023) - **Passthrough** — Fixed passthrough budgets, gated passthrough models per VK, model extraction for Azure passthrough, and restricted fallbacks/provider selection to the VK boundary (#3941, #3988, #3983, #3924) - **Provider Response Headers** — Strip provider response headers and add a content-type filter (#3955, #4024) - **Stream Handling** — Drain non-SSE stream readers and retry stale connections (#3956, #3967) - **Azure Claude** — Strip Azure diagnostic property for Claude models (#3925) - **Compat max_tokens** — Preserve chat `max_tokens` during param filtering (#3992) - **Raw Request Flag** — Removed the raw request flag from providers that don't support it (#4058) - **UI Fixes** — Standardized page container layout, virtual key model configs UI, and dashboard chart tooltips (#4046, #4052, #4044) ## 🔧 Maintenance - **Dependency Upgrades** — Bumped transitive `golang.org/x` dependencies (crypto, net, sys, text) for Docker Scout CVE remediation and `recharts` to 3.8.1; cascaded version bumps across all modules (#3900, #4003)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a
CREATE INDEX CONCURRENTLYfailure when building the full-text search GIN indexes on thelogsandmcp_tool_logstables. Postgres aborts index creation with:The
left(col, 800000)cap counts characters, butto_tsvectorenforces a 1,048,575-byte limit on its output. Multi-byte UTF-8 / high-lexeme content can expand 800k chars well past 1 MB, aborting the build and leaving full-text search to fall back to slow sequential scans.Changes
left()input cap from800000→250000chars on all three FTS GIN indexes:idx_logs_content_summary_fts(logs.content_summary)idx_mcp_logs_arguments_fts(mcp_tool_logs.arguments)idx_mcp_logs_result_fts(mcp_tool_logs.result)Type of change
Affected areas
How to test
On a Postgres deployment, confirm the indexes build successfully on startup (via
ensurePerformanceIndexes) without the tsvector size error, including on databases that previously failed:Note:
CREATE INDEX CONCURRENTLY IF NOT EXISTSis a no-op if an index already exists, so any invalid index left behind by the prior failed build should be dropped before the corrected one is created.Breaking changes
Related issues
N/A
Security considerations
None. Affects only index input truncation length.
Checklist
docs/contributing/README.mdand followed the guidelinesSummary by CodeRabbit