add source of truth flow for config.json

[akshaydeo]

Summary

Introduces a source_of_truth field to config.json that allows operators to make config.json sections authoritative during startup reconciliation. When set to "config.json", any section explicitly present in the file becomes the single source of truth — database-only rows for that section are pruned. When omitted or set to "split" (the default), existing merge behavior is preserved.

Changes

• Added source_of_truth field to ConfigData with two modes: "split" (default, existing behavior) and "config.json" (file-authoritative).
• Added presentSections and presentGovernanceSections tracking maps populated during UnmarshalJSON so that explicitly-present-but-empty sections (e.g., "providers": {}) can be distinguished from absent sections.
• Added sectionPresent and governanceSectionPresent helpers on ConfigData to query section presence.
• Introduced syncAuthoritativeProvidersInStore which, under config.json mode, deletes DB-only providers and DB-only keys within kept providers inside a single transaction.
• Introduced processAuthoritativeProvider as the authoritative counterpart to processProvider, preserving runtime-only fields (status, description) from the DB while making the file's key list canonical.
• Introduced syncMCPConfigFromFile which replaces stored MCP clients with exactly those declared in config.json, deleting DB-only clients and upserting file clients.
• Introduced syncPluginsFromFile which replaces stored plugins with exactly those declared in config.json, deleting DB-only plugins and upserting file plugins within a transaction.
• Introduced pruneGovernanceConfigToFile which, after the normal governance merge, removes DB-only rows for each governance collection that was explicitly present in the file (virtual keys, routing rules, pricing overrides, model configs, teams, customers, providers, budgets, rate limits).
• Added source_of_truth to config.schema.json as an enum of ["split", "config.json"] with schema validation.
• Added a schema candidate path for tests running from transports/bifrost-http/lib/.
• Fixed MockConfigStore.DeleteMCPClientConfig and DeletePlugin to actually remove entries so sync tests can assert on store state.

Type of change

• Bug fix
• Feature
• Refactor
• Documentation
• Chore/CI

Affected areas

• Core (Go)
• Transports (HTTP)
• Providers/Integrations
• Plugins
• UI (React)
• Docs

How to test

go test ./transports/bifrost-http/lib/... -run TestConfigDataSourceOfTruth
go test ./transports/bifrost-http/lib/... -run TestSourceOfTruthConfigJSON
go test ./transports/bifrost-http/lib/... -run TestSQLite_SourceOfTruthConfigJSON
go test ./transports/bifrost-http/lib/... -run TestConfigSchemaSourceOfTruthValidation
go test ./transports/bifrost-http/lib/...

New config.json field:

Field	Type	Values	Default
source_of_truth	string	"split", "config.json"	"split"

To enable authoritative mode, add to config.json:

{
  "source_of_truth": "config.json",
  "providers": { ... },
  "governance": { "budgets": [...] }
}

Only sections explicitly present in the file will be pruned in the database. Sections omitted from the file leave database rows untouched regardless of mode.

Breaking changes

• Yes
• No

Default behavior ("split") is unchanged. Operators must explicitly opt in to "config.json" mode.

Related issues

Security considerations

Provider API keys that exist only in the database will be permanently deleted when source_of_truth: "config.json" is set and the providers section is present in the file. Operators should ensure all required keys are declared in config.json before enabling this mode.

Checklist

• I read docs/contributing/README.md and followed the guidelines
• I added/updated tests where appropriate
• I updated documentation where needed
• I verified builds succeed (Go and UI)
• I verified the CI pipeline passes locally if applicable

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: def0f759-3cf2-4ba2-9832-5ccac03939aa

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR adds a configurable source_of_truth mode (split or config.json) that toggles exact "replace-and-prune" reconciliation for providers, MCP clients, plugins, and governance when file sections are explicitly present. It also makes deletion APIs accept an optional variadic GORM transaction to allow transactional batching, and includes schema, unmarshalling, unit, and integration test updates.

Changes

Source-of-Truth Config Reconciliation

Layer / File(s)	Summary
Transaction API Enhancement framework/configstore/store.go, framework/configstore/rdb.go	ConfigStore methods DeleteTeam, DeleteCustomer, and DeleteModelConfig now accept optional variadic tx ...*gorm.DB. RDBConfigStore reuses a provided tx or begins its own transaction and performs nulling of foreign keys, primary delete, and deletion of linked governance rows.
Source-of-Truth Schema and ConfigData Contract transports/bifrost-http/lib/config.go, transports/config.schema.json	Adds SourceOfTruthSplit and SourceOfTruthConfigJSON constants, ConfigData.SourceOfTruth, presence-tracking of top-level/governance sections, unmarshalling helpers, and schema source_of_truth enum (split/config.json, default split).
Provider Reconciliation and Authoritative Helper transports/bifrost-http/lib/config.go	When authoritative and providers present, builds authoritative provider entries (assigns key IDs, validates aliases, computes hashes, preserves existing status/description) and persists via a transactional sync that prunes DB-only providers/keys before upserting.
MCP Client Reconciliation transports/bifrost-http/lib/config.go	When authoritative and mcp present, syncMCPConfigFromFile reconciles clients by name/ID, assigns IDs, computes per-client hashes, upserts matching clients, and prunes DB-only clients transactionally.
Plugin Reconciliation transports/bifrost-http/lib/config.go	When authoritative and plugins present (including explicit empty), syncPluginsFromFile transactionally prunes DB-only plugins and upserts file-declared plugins, updating in-memory state only after the durable transaction succeeds.
Governance Reconciliation and Schema Validation transports/bifrost-http/lib/config.go, transports/bifrost-http/lib/validator.go	After governance merge, when authoritative and a governance collection is explicitly present, pruneGovernanceConfigToFile transactionally deletes DB-only rows for those collections and adjusts provider mapping links; validator adds local schema discovery path.
MockConfigStore Updates and Configuration Tests transports/bifrost-http/lib/config_test.go	MockConfigStore delete method signatures now accept optional transactions; mock delete implementations prune in-memory MCP clients and plugins. Unit tests cover source_of_truth semantics, and integration tests verify DB pruning and exact-sync behavior.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Suggested reviewers

• danpiths
• roroghost17

Poem

In a burrow of code where configs grow,
I nudge the truth where the mappings flow.
Split or file, the DB trims its wine,
Pruned and tidy — oh, how fine! 🐇✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'add source of truth flow for config.json' clearly and concisely describes the main feature being introduced: a source-of-truth mechanism for config.json configuration files.
Description check	✅ Passed	The pull request description follows the template structure with comprehensive sections including Summary, Changes, Type of change, Affected areas, How to test, Breaking changes, Security considerations, and a completed Checklist.
Docstring Coverage	✅ Passed	Docstring coverage is 91.43% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 06-02-add_source_of_truth_flow_for_config.json

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[akshaydeo]

• add source of truth flow for config.json #3968 👈 (View in Graphite)
• stale connection retries #3967
• dev

This stack of pull requests is managed by Graphite. Learn more about stacking.

[greptile-apps]

Confidence Score: 4/5

Safe to merge for the default split path; the new config.json SOT mode works correctly for the tested scenarios but has a nuanced interaction in the governance pruning transaction that is worth validating under more complex DB states before broad rollout.

The governance pruning transaction in pruneGovernanceConfigToFile deletes teams and customers before it processes budgets and rate-limits. Because DeleteTeam cascades owned budgets via the governance_budgets.team_id FK, and DeleteCustomer explicitly deletes its owned budget/rate-limit rows, those rows can be removed by cascade before the budgets/rate-limits prune block attempts to delete them again — which would return not-found and cause the transaction to fail with logger.Fatal on startup. The bulk-entity integration test avoids this scenario by creating teams/customers with no owned budgets, so the test passes but the failure path remains uncovered.

transports/bifrost-http/lib/config.go — specifically pruneGovernanceConfigToFile and the ordering of the teams/customers/budgets/rate-limits prune blocks within its transaction

Important Files Changed

Filename	Overview
transports/bifrost-http/lib/config.go	Core SOT reconciliation logic: adds syncAuthoritativeProvidersInStore, syncMCPConfigFromFile, syncPluginsFromFile, pruneGovernanceConfigToFile, and processAuthoritativeProvider. Logic is largely correct; pruneGovernanceConfigToFile wraps governance pruning in a single transaction with logger.Fatal on failure. One consistency gap: syncPluginsFromFile calls GetPlugins(ctx) without the enclosing tx handle.
framework/configstore/rdb.go	Refactors DeleteTeam, DeleteCustomer, DeleteModelConfig to accept variadic tx ...*gorm.DB using the self-recursion pattern already used by DeleteBudget/DeleteRateLimit, enabling these methods to participate in the outer governance-prune transaction.
framework/configstore/store.go	Interface signatures for DeleteTeam, DeleteCustomer, DeleteModelConfig updated to add variadic tx ...*gorm.DB — minimal, focused change consistent with the variadic pattern used across the rest of the interface.
transports/bifrost-http/lib/config_test.go	Adds unit tests for section-presence tracking, schema validation, MCP/plugin pruning with mock store, and SQLite-backed integration tests for provider+key pruning and bulk governance pruning. Mock DeleteMCPClientConfig and DeletePlugin now actually mutate mock state. Mock DeleteTeam/DeleteCustomer/DeleteModelConfig return nil without mutating, which is fine for tests that use SQLite for governance assertions.
transports/bifrost-http/lib/validator.go	Adds ../../config.schema.json candidate path so schema-validation tests running from transports/bifrost-http/lib/ can locate the schema file.
transports/config.schema.json	Adds source_of_truth as an enum field ([split, config.json], default split) — consistent with normalizeSourceOfTruth in the Go code.

_{Reviews (7): Last reviewed commit: "add source of truth flow for config.json" | Re-trigger Greptile}

[akshaydeo]

@coderabbitai full review

[coderabbitai]

✅ Actions performed

Full review triggered.

[coderabbitai]

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@framework/configstore/rdb.go`:
- Around line 3471-3479: The functions (e.g., DeleteTeam, and the similar
DeleteCustomerConfig/DeleteModelConfig blocks flagged) currently dereference
tx[0] after only checking length, which will panic if someone calls the function
with a nil transaction (e.g., DeleteTeam(ctx,id,nil)); modify each function to
explicitly check that when len(tx) > 0 the provided tx[0] is not nil before
using it and return a clear error if it is nil (or alternatively treat nil as
"no transaction" and start a new transaction); ensure txDB := tx[0] is only
assigned after the nil check and update all call sites in those functions (e.g.,
the code paths using txDB.WithContext(ctx) or transaction) accordingly.

In `@transports/bifrost-http/lib/config.go`:
- Around line 1121-1129: The current authoritative sync loop using
isConfigJSONSourceOfTruth() calls processAuthoritativeProvider and, on error,
skips adding that provider into authoritativeProviders which causes the existing
DB provider to be pruned; change the error path so that when
processAuthoritativeProvider returns an error you preserve the existing provider
config by inserting existingCfg (from providersInConfigStore[provider]) into
authoritativeProviders if exists, or alternatively abort the authoritative sync
by returning the error; update the loop around processAuthoritativeProvider (and
the similar block at the later location mentioned) to ensure
authoritativeProviders always contains either the new validated config or the
existingCfg for that provider to avoid accidental deletion of DB-only keys.
- Around line 1717-1761: The delete loop is mistakenly pruning existing MCP
client rows when a corresponding file entry fails validation because keepIDs is
only set after mcpClientConfigToTable and configstore.GenerateMCPClientHash
succeed; fix by marking the matched existing client as kept up-front and/or
skipping the prune pass if any file entry fails: in the file iteration over
fileMCPConfig.ClientConfigs, when you compute existing := existingByName[...] /
existingByID[...] immediately set keepIDs[existing.ID] = true (if existing !=
nil && existing.ID != "") before calling mcpClientConfigToTable or
GenerateMCPClientHash, and additionally track a boolean like hadValidationError
which you set when you hit an error and then guard the later
config.ConfigStore.DeleteMCPClientConfig loop with if !hadValidationError { ...
} to avoid accidental deletions.
- Around line 2333-2340: The unchanged virtual keys path is passing
vk.MCPConfigs with unresolved mcp_client_name entries (MCPClientID==0) into
reconcileVirtualKeyAssociations, which can create or delete wrong associations;
before calling reconcileVirtualKeyAssociations for any virtual key (including
those in the unchanged branch that iterate configData.Governance.VirtualKeys),
ensure you run the same resolution step used for new/changed keys — call
resolveMCPConfigClientIDs (or the equivalent resolution routine) to populate
MCPClientID values on vk.MCPConfigs (using mergeGovernanceConfig’s logic) so
reconcileVirtualKeyAssociations receives fully-resolved MCPClientIDs and does
not create/delete incorrect client-id 0 associations.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: b7047f89-0de7-4432-b025-a83bbe93fae7

📥 Commits

Reviewing files that changed from the base of the PR and between 66d1cee and e80068e.

📒 Files selected for processing (6)

• framework/configstore/rdb.go
• framework/configstore/store.go
• transports/bifrost-http/lib/config.go
• transports/bifrost-http/lib/config_test.go
• transports/bifrost-http/lib/validator.go
• transports/config.schema.json

👮 Files not reviewed due to content moderation or server errors (1)

• transports/bifrost-http/lib/config_test.go

[akshaydeo]

Merge activity

• Jun 2, 6:44 AM UTC: A user started a stack merge that includes this pull request via Graphite.
• Jun 2, 6:45 AM UTC: @akshaydeo merged this pull request with Graphite.

The base branch was changed.