Skip to content

fix(db): unblock PostgreSQL upgrade to v0.7.x (sqlalchemy<2.1 + autocommit_block migrations) - #1904

Merged
nicoloboschi merged 2 commits into
mainfrom
fix/pg-upgrade-sqlalchemy-21
Jun 1, 2026
Merged

fix(db): unblock PostgreSQL upgrade to v0.7.x (sqlalchemy<2.1 + autocommit_block migrations)#1904
nicoloboschi merged 2 commits into
mainfrom
fix/pg-upgrade-sqlalchemy-21

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Summary

Fixes the v0.6.2v0.7.x PostgreSQL upgrade path reported in #1902, which failed during startup migrations in two sequential ways.

1. Missing psycopg DBAPI

We ship only psycopg2-binary, but the dependency pin was sqlalchemy>=2.0.44 with no upper bound, so a fresh PyPI install resolves SQLAlchemy 2.1. SQLAlchemy 2.1 changed the default postgresql:// driver from psycopg2 to psycopg (v3). Since to_libpq_url() produces a driver-less postgresql:// URL, the migration engine then tried to import psycopg3 and failed with ModuleNotFoundError: No module named 'psycopg'.

Fix: cap sqlalchemy>=2.0.44,<2.1 so psycopg2 stays the default driver — the line that's locked (2.0.44) and tested in CI. (The repo's own uv.lock was already on 2.0.44, which is why CI never caught this — only fresh PyPI installs hit it.)

2. CREATE/DROP INDEX CONCURRENTLY inside a transaction block

Seven migrations escaped Alembic's migration transaction with a hand-rolled op.execute("COMMIT") trick. That happens to work on psycopg2 but breaks on psycopg/SQLAlchemy 2.1 (the env you land in after working around #1), where the next statement re-opens a transaction and PostgreSQL rejects the CONCURRENTLY DDL with ActiveSqlTransaction.

Fix: convert all seven to with op.get_context().autocommit_block():, matching the project's own newer b8c9d0e1f2a3 migration. The e9b2c7d1f3a4 entity-link cleanup's DO $$ … COMMIT … $$ batch loop is wrapped too, since procedural COMMIT also requires autocommit.

Migrations converted:

  • a2b3c4d5e6f8, b3c4d5e6f7g8, c1a2b3d4e5f6, d2e3f4a5b6c7, d4e5f6g7h8i9, e1b2c3d4f5a6, e9b2c7d1f3a4

Regression guard

Two lint-style tests in test_migration_shape.py run over every migration so this class of bug can't return:

  • test_migration_uses_autocommit_block_not_manual_commit — bans op.execute("COMMIT").
  • test_migration_concurrently_ddl_runs_in_autocommit_block — any migration running CONCURRENTLY DDL must open an autocommit_block().

Testing

  • pytest tests/test_migration_shape.py → 140 passed, 70 skipped (the new guards).
  • pytest tests/test_admin_backup_restore.py → 6 passed — this builds a full schema by running the entire migration chain (including the converted CONCURRENTLY migrations) on real pg0, exercising the autocommit_block path end-to-end.
  • ./scripts/hooks/lint.sh clean.

Notes

  • The <2.1 cap is a deliberate stopgap. Adopting SQLAlchemy 2.1 later just means adding psycopg[binary] (or pinning postgresql+psycopg2:// in to_libpq_url) and lifting the ceiling — a comment in pyproject.toml flags this.
  • The CI/test path runs on psycopg2/SQLAlchemy 2.0; the psycopg3/2.1 path can't be exercised here, but autocommit_block is the documented-correct approach for both drivers.
  • The issue's third point (batched/retryable entity-link delete) is already handled — e9b2c7d1f3a4 chunks the delete at 50k rows; no change needed.

Closes #1902

…mmit_block

Fixes the v0.6.2 -> v0.7.x PostgreSQL upgrade path reported in #1902, which
failed in two ways:

1. Missing psycopg DBAPI. We ship only psycopg2-binary, but `sqlalchemy>=2.0.44`
   allowed SQLAlchemy 2.1, which changed the default `postgresql://` driver from
   psycopg2 to psycopg (v3). A bare PyPI install then failed migrations with
   "No module named 'psycopg'". Cap to `>=2.0.44,<2.1` so psycopg2 stays the
   default driver (the tested/locked line) until psycopg3 is adopted.

2. CONCURRENTLY inside a transaction block. Seven migrations escaped Alembic's
   migration transaction with the hand-rolled `op.execute("COMMIT")` trick. That
   happens to work on psycopg2 but breaks on psycopg/SQLAlchemy 2.1, where the
   next statement re-opens a transaction and PostgreSQL rejects CREATE/DROP
   INDEX CONCURRENTLY. Convert all seven to `op.get_context().autocommit_block()`,
   matching the existing b8c9d0e1f2a3 migration. The e9b2c7d1f3a4 entity-link
   cleanup's `DO $$ ... COMMIT ... $$` batch loop is wrapped too, since
   procedural COMMIT also requires autocommit.

Add two lint-style guard tests in test_migration_shape.py so this class of bug
can't be reintroduced: one bans `op.execute("COMMIT")`, the other requires any
migration running CONCURRENTLY DDL to open an autocommit_block().
@nicoloboschi
nicoloboschi merged commit b7f267b into main Jun 1, 2026
153 of 154 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PostgreSQL upgrade to v0.7.1 fails: missing psycopg and CONCURRENTLY migrations run inside transaction blocks

1 participant