-
Notifications
You must be signed in to change notification settings - Fork 0
feat(data): add fail-closed schema migration ledger #500
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
seonghobae
wants to merge
61
commits into
develop
Choose a base branch
from
refactor/schema-migration-ledger-433
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
61 commits
Select commit
Hold shift + click to select a range
096dcf3
test(data): specify schema migration state boundary
seonghobae d2b5cde
test(data): run schema migration state contract in CI
seonghobae 7070a29
feat(data): add fail-closed schema migration ledger
seonghobae 88cb945
feat(data): enforce migration-state guard at startup
seonghobae cf0d0f0
docs(data): record schema migration ledger boundary
seonghobae ec8b74e
docs(changelog): record schema migration guard
seonghobae 5c50012
test(data): cover migration guard failure branches
seonghobae 3eb58ba
test(data): reject legacy recreation on canonical schema
seonghobae 6cbded1
fix(data): inspect schema before legacy bootstrap
seonghobae 2832fba
fix(data): fail before legacy schema recreation
seonghobae a8901a6
test(data): cover pre-bootstrap schema inspection
seonghobae fa2581b
docs(data): record pre-bootstrap fail-closed boundary
seonghobae 782c8ca
test(data): reject impossible migration rollback history
seonghobae 9682288
fix(data): reject backward migration ledger history
seonghobae 6386009
test(data): cover monotonic migration ledger history
seonghobae c1752f2
docs(data): document monotonic migration history
seonghobae ba5d119
docs(changelog): record migration history guard
seonghobae d36200c
fix(data): make migration state map prototype-free
seonghobae b7e4a2e
merge(data): reconcile migration ledger with protected develop
seonghobae b98c3f5
merge(data): reconcile protected XML import baseline
seonghobae 85ee768
merge(data): reconcile schema migration ledger with current develop
seonghobae 74dd422
test(data): expose swallowed legacy column migration failures
seonghobae 5624541
test(data): register legacy column migration regression
seonghobae 345202f
fix(data): fail closed on legacy column migration errors
seonghobae 26932ce
fix(data): use catalog-first compatibility migrations
seonghobae 525e995
docs(data): record fail-closed compatibility DDL
seonghobae 44bd114
docs(data): document catalog-first legacy column upgrades
seonghobae ad4166b
merge(develop): reconcile migration ledger with OpenCode config
seonghobae 2a4e118
merge(develop): reconcile schema migration ledger with adaptive attri…
seonghobae 2af6d29
fix(stack): reconcile schema migration ledger with protected develop
seonghobae 1bca59e
test(data): reject incompatible legacy columns
seonghobae 41925c0
fix(data): validate legacy column definitions
seonghobae 1902554
test(ci): require database startup coverage
seonghobae 9c1dcba
fix(ci): cover database startup control flow
seonghobae 7751daf
merge(develop): preserve Playwright 1.62.1 in schema ledger slice
seonghobae 1087eb2
test(schema): require streamed migration metadata
seonghobae 4eeb8ad
test(schema): register streaming regressions
seonghobae 1c3a76a
fix(schema): stream migration catalog evidence
seonghobae f9410c3
test(data): require atomic initial schema bootstrap
seonghobae c50e7d1
test(data): cover atomic bootstrap contract
seonghobae 8c5d023
fix(data): make initial schema bootstrap atomic
seonghobae 348b105
fix(data): wire atomic schema bootstrap
seonghobae 84d0177
docs(data): document atomic bootstrap recovery
seonghobae f5e82a4
test(data): reject malformed migration ledger schema
seonghobae 864887c
test(data): register migration ledger schema regression
seonghobae 502bc13
fix(data): validate migration ledger schema
seonghobae 0ab9a74
test(data): model validated ledger metadata in streaming fixture
seonghobae f2aeb67
test(data): align ledger fixtures with enforced schema
seonghobae 3a25bc1
test(data): reject ledger timestamp default drift
seonghobae 35a0cfe
fix(data): enforce ledger timestamp default contract
seonghobae aaa3566
test(data): align streaming ledger schema fixture
seonghobae f63b55c
test(data): reproduce rolling-start migration lock
seonghobae 033d960
test(data): register rolling-start lock regression
seonghobae c51d11d
fix(data): avoid no-op startup write locks
seonghobae db0df09
fix(data): preserve concurrent ledger idempotence
seonghobae 5777834
test(data): fix persisted-ledger streaming expectation
seonghobae b15cbbe
test(data): reproduce skipped additive bootstrap
seonghobae 6bab0a5
fix(data): preserve additive schema bootstrap
seonghobae 37e27e7
test(data): prove migrated startup skips ledger DDL
seonghobae 9f2e681
fix(data): avoid redundant migration-ledger DDL
seonghobae b7ef5c5
fix(data): reject unknown schema objects
seonghobae File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,161 @@ | ||
| # Schema migration ledger and fail-closed generation guard | ||
|
|
||
| ## Decision | ||
|
|
||
| Issue #433 requires the existing single-word SQLite object names to move to the | ||
| repository's two-or-more-word `snake_case` contract without serving a database | ||
| that is half old-schema and half canonical-schema. The first bounded | ||
| expand/verify slice introduces an append-only `schema_migrations` ledger and a | ||
| framework-independent generation classifier before any destructive rename is | ||
| attempted. | ||
|
|
||
| The guard recognizes exactly two complete generations for the ten tables in the | ||
| rename plan: | ||
|
|
||
| - legacy: `users`, `orgs`, `memberships`, `projects`, `invites`, `webhooks`, | ||
| `baselines`, `comments`, `sprints`, `attachments`; | ||
| - canonical: `user_accounts`, `organization_records`, | ||
| `organization_memberships`, `project_records`, `invitation_records`, | ||
| `webhook_endpoints`, `project_baselines`, `project_comments`, | ||
| `project_sprints`, `project_attachments`. | ||
|
|
||
| The `schema_migrations` table is migration metadata and is excluded from the | ||
| generation catalog. Any other unrecognized table, missing table, old/new | ||
| mixture, or duplicated generation is an invalid startup state and raises | ||
| `SchemaMigrationStateError` before request handling begins. This prevents an | ||
| otherwise complete generation from being trusted alongside an object that the | ||
| query layer has not verified. The existing non-renamed tables | ||
| (`webhook_deliveries`, `audit_log`, `api_tokens`, `project_revisions`, | ||
| `share_tokens`, and `project_seen`) are explicitly allowlisted as stable | ||
| application objects. | ||
|
|
||
| ## Pre-bootstrap boundary | ||
|
|
||
| The database catalog is inspected **before** the legacy `CREATE TABLE IF NOT | ||
| EXISTS` and additive `ALTER TABLE` statements run. Only a genuinely empty | ||
| database may initialize the legacy schema from scratch. An existing database | ||
| must already be one complete known generation; mixed, incomplete, ledger-only, | ||
| or otherwise ambiguous states fail closed before legacy bootstrap can mutate | ||
| them. | ||
|
|
||
| The complete legacy `CREATE TABLE`/`CREATE INDEX` bootstrap is executed inside | ||
| one explicit SQLite transaction. If a first deployment is terminated while that | ||
| transaction is open, SQLite recovery returns the catalog to its pre-transaction | ||
| state; if a statement fails synchronously, ScopeWeave explicitly rolls the | ||
| transaction back before rethrowing the causal error. This prevents a brand-new | ||
| database from being stranded as an otherwise unrecoverable partial legacy | ||
| generation. A partial generation that already exists before startup remains | ||
| fail-closed and requires verified recovery rather than heuristic completion. | ||
|
|
||
| The historical compatibility columns `users.token_version`, | ||
| `projects.archived`, and `projects.methodology` are also catalog-driven. Startup | ||
| queries `PRAGMA table_info` and runs `ALTER TABLE` only when the exact column is | ||
| absent. This replaces the former blanket `try/catch` pattern that treated every | ||
| SQLite exception as "already there". With expected idempotence removed from the | ||
| exception path, read-only, disk-I/O, locking, corruption, or other genuine DDL | ||
| failures propagate and stop startup instead of leaving a partially upgraded | ||
| schema available to request handling. | ||
|
|
||
| A complete canonical generation is deliberately identified and recorded, but | ||
| this application version still uses legacy table names in its query layer. | ||
| Therefore startup records/verifies `canonical_schema_v2` and then fails with a | ||
| stable "canonical schema generation is not yet supported by this application | ||
| version" error. It does **not** recreate legacy tables over the canonical | ||
| schema. Serving a canonical database becomes valid only in the later issue #433 | ||
| slice that migrates the application query/data-access layer as part of the same | ||
| reviewed cutover. | ||
|
|
||
| ## Ledger contract | ||
|
|
||
| `schema_migrations` contains a stable `migration_key`, a low-cardinality | ||
| `state_code`, and an application timestamp. Repeated startup is idempotent via | ||
| `INSERT OR IGNORE`. Every persisted row must be one of the known exact | ||
| migration-key/state pairs before the current generation is recorded. Unknown | ||
| keys or corrupted state codes fail closed rather than being treated as harmless | ||
| history. | ||
|
|
||
| The ledger is monotonic across the naming migration. A canonical database may | ||
| retain the earlier `legacy_schema_v1` record and append `canonical_schema_v2`. | ||
| The reverse is forbidden: if the schema catalog is legacy while the ledger says | ||
| `canonical_schema_v2` has ever been reached, startup fails because that state is | ||
| consistent with an unsupported reverse migration, stale restore, or incomplete | ||
| recovery. Backup restoration must restore schema and migration history to one | ||
| coherent point rather than silently moving the schema backward under newer | ||
| ledger evidence. | ||
|
|
||
| After insertion, the persisted state is read back and compared with the fresh | ||
| schema-catalog classification so a corrupted current-generation record cannot | ||
| silently bless the database. | ||
|
|
||
| This slice deliberately does **not** rename application tables, create legacy | ||
| compatibility views, or claim PostgreSQL adapter readiness. Those operations | ||
| remain subsequent issue #433 work and must use the ledger/guard as the | ||
| precondition for an atomic expand/verify/contract cutover. | ||
|
|
||
| ## Failure and recovery boundary | ||
|
|
||
| A partially renamed database is not automatically repaired on startup. Serving | ||
| mixed names would make query routing ambiguous and could split reads and writes | ||
| between generations. The process therefore fails closed. Operators must restore | ||
| a verified backup or complete the reviewed migration before restarting. | ||
|
|
||
| The later rename executor must run with foreign keys enabled, modern SQLite | ||
| rename propagation semantics, pre/post `PRAGMA integrity_check` and | ||
| `PRAGMA foreign_key_check`, deterministic interruption tests, restart evidence, | ||
| and a restore rehearsal. Reverse renames are not a substitute for backup | ||
| recovery, and a restore that moves from canonical history back to legacy schema | ||
| must restore the ledger from the same verified recovery point. | ||
|
|
||
| ## Verification | ||
|
|
||
| `tests/unit/schema-migration-state.test.mjs` covers: | ||
|
|
||
| - idempotent legacy ledger creation; | ||
| - a distinct canonical-generation ledger record; | ||
| - preservation of valid legacy history when the canonical record is appended; | ||
| - rejection of unknown or corrupted migration-ledger identities/states; | ||
| - rejection of an otherwise complete generation paired with an unknown | ||
| application table; | ||
| - rejection of a canonical ledger record paired with a legacy schema generation; | ||
| - fail-closed mixed-generation detection; | ||
| - incomplete schema rejection; | ||
| - the complete ten-object legacy and canonical inventories; | ||
| - pre-bootstrap classification of pristine, legacy, canonical, and invalid | ||
| databases; | ||
| - atomic legacy-bootstrap commit behavior, rollback after a real mid-script | ||
| SQLite DDL failure, input validation, and preservation of the causal error even | ||
| if best-effort rollback itself fails; and | ||
| - a real `server/db.mjs` subprocess regression proving canonical startup fails | ||
| for the truthful query-layer reason without recreating any legacy table. | ||
|
|
||
| `tests/unit/legacy-column-migration.test.mjs` additionally proves that the three | ||
| historical additive columns are created idempotently from SQLite catalog | ||
| evidence, preserve their production defaults, reject incomplete adapters, and | ||
| propagate a real database error rather than converting it into false idempotent | ||
| success. | ||
|
|
||
| The production module is registered in the Istanbul coverage command. The API | ||
| and existing server tests exercise normal startup integration through | ||
| `server/db.mjs`, while the canonical-database subprocess exercises the opposite | ||
| fail-closed boundary against a persisted SQLite file. | ||
|
|
||
| ## Rollback | ||
|
|
||
| Before any table rename ships, rollback of this slice is non-destructive: revert | ||
| the startup guard and module. The extra `schema_migrations` table is inert data | ||
| and may remain in an existing database. Once a future rename migration is | ||
| released, rollback must follow that migration's backup/restore runbook instead | ||
| of deleting ledger history. | ||
|
|
||
| ## References | ||
|
|
||
| International Organization for Standardization. (2023). *Information | ||
| technology—Database languages SQL—Part 1: Framework (SQL/Framework)* | ||
| (ISO/IEC Standard No. 9075-1:2023). | ||
| https://www.iso.org/standard/76583.html | ||
|
|
||
| SQLite Consortium. (2026). *ALTER TABLE*. | ||
| https://sqlite.org/lang_altertable.html | ||
|
|
||
| SQLite Consortium. (2026). *PRAGMA statements*. | ||
| https://sqlite.org/pragma.html |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.