Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Add type hints to schema deltas #15497

Merged
merged 12 commits into from
Apr 27, 2023
Merged

Add type hints to schema deltas #15497

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
689db6a
Remove unneeded untyped-defs.
clokep Apr 26, 2023
8894bef
Add some type ignores so we can check main/cache.py.
clokep Apr 26, 2023
3e29d56
Add some ignores to databases/__init__.py
clokep Apr 26, 2023
33fb232
Newsfragment
clokep Apr 26, 2023
556855c
Rename a file to avoid a conflict.
clokep Apr 26, 2023
9c6e91e
Remove no-op functions.
clokep Apr 26, 2023
85e4bb3
Add missing function parameters.
clokep Apr 26, 2023
91f74c2
Fix-up type hints.
clokep Apr 26, 2023
7978dc9
Use LoggingTransaction instead of Cursor since we use execute_batch.
clokep Apr 26, 2023
e76a736
No more excluded files from mypy.
clokep Apr 26, 2023
fbacc65
Newsfragment
clokep Apr 26, 2023
e8bc91a
Merge remote-tracking branch 'origin/develop' into clokep/type-hints-2
clokep Apr 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions synapse/storage/schema/main/delta/20/pushers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,3 @@ def run_create(cur, database_engine, *args, **kwargs):
cur.execute("DROP TABLE pushers")
cur.execute("ALTER TABLE pushers2 RENAME TO pushers")
logger.info("Moved %d pushers to new table", count)


def run_upgrade(*args, **kwargs):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is safe to remove, because we do a hasattr check:

synapse/synapse/storage/prepare_database.py

Line 510 in 9bb2eac

if not is_empty and hasattr(module, "run_upgrade"):

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should have called that out, sorry. 👍

pass
4 changes: 0 additions & 4 deletions synapse/storage/schema/main/delta/25/fts.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,3 @@ def run_create(cur, database_engine, *args, **kwargs):
)

cur.execute(sql, ("event_search", progress_json))


def run_upgrade(*args, **kwargs):
pass
4 changes: 0 additions & 4 deletions synapse/storage/schema/main/delta/27/ts.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,3 @@ def run_create(cur, database_engine, *args, **kwargs):
)

cur.execute(sql, ("event_origin_server_ts", progress_json))


def run_upgrade(*args, **kwargs):
pass
4 changes: 0 additions & 4 deletions synapse/storage/schema/main/delta/31/pushers_0.py
View file
Open in desktop
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because of the way mypy assigns python files to modules (it essentially searches the directory tree until it finds a __init__.py) we need all Python schema deltas to be unique.

This was the only conflict; I played with some of the options for changing this behavior (namespace packages, etc.) but it seemed easiest to just rename the single offender.

This is from a really really old schema so I think renaming it won't break anything, but I'm not 100% confident on that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only breakage I can see: someone who is still on schema 31 might end up trying to apply the renamed migration after it was already run, if they upgraded directly to a Synapse version with this change.

This looks to be around 7 years old now. I think we should be blasé about breaking setups which are that old and suffer from known security problems.

Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,3 @@ def run_create(cur, database_engine, *args, **kwargs):
cur.execute("DROP TABLE pushers")
cur.execute("ALTER TABLE pushers2 RENAME TO pushers")
logger.info("Moved %d pushers to new table", count)


def run_upgrade(cur, database_engine, *args, **kwargs):
pass
4 changes: 0 additions & 4 deletions synapse/storage/schema/main/delta/31/search_update.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,3 @@ def run_create(cur, database_engine, *args, **kwargs):
)

cur.execute(sql, ("event_search_order", progress_json))


def run_upgrade(cur, database_engine, *args, **kwargs):
pass
4 changes: 0 additions & 4 deletions synapse/storage/schema/main/delta/33/event_fields.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,3 @@ def run_create(cur, database_engine, *args, **kwargs):
)

cur.execute(sql, ("event_fields_sender_url", progress_json))


def run_upgrade(cur, database_engine, *args, **kwargs):
pass
4 changes: 0 additions & 4 deletions synapse/storage/schema/main/delta/34/cache_stream.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,3 @@ def run_create(cur, database_engine, *args, **kwargs):

for statement in get_statements(CREATE_TABLE.splitlines()):
cur.execute(statement)


def run_upgrade(cur, database_engine, *args, **kwargs):
pass
4 changes: 0 additions & 4 deletions synapse/storage/schema/main/delta/34/received_txn_purge.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,3 @@ def run_create(cur, database_engine, *args, **kwargs):
cur.execute("DELETE FROM received_transactions")

cur.execute("CREATE INDEX received_transactions_ts ON received_transactions(ts)")


def run_upgrade(cur, database_engine, *args, **kwargs):
pass
4 changes: 0 additions & 4 deletions synapse/storage/schema/main/delta/37/remove_auth_idx.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,3 @@ def run_create(cur, database_engine, *args, **kwargs):

for statement in get_statements(drop_constraint.splitlines()):
cur.execute(statement)


def run_upgrade(cur, database_engine, *args, **kwargs):
pass
4 changes: 0 additions & 4 deletions synapse/storage/schema/main/delta/42/user_dir.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,3 @@ def run_create(cur, database_engine, *args, **kwargs):
cur.execute(statement)
else:
raise Exception("Unrecognized database engine")


def run_upgrade(*args, **kwargs):
pass
4 changes: 0 additions & 4 deletions synapse/storage/schema/main/delta/48/group_unique_indexes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,3 @@ def run_create(cur, database_engine, *args, **kwargs):

for statement in get_statements(FIX_INDEXES.splitlines()):
cur.execute(statement)


def run_upgrade(*args, **kwargs):
pass
4 changes: 0 additions & 4 deletions synapse/storage/schema/main/delta/56/unique_user_filter_index.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,6 @@
"""


def run_upgrade(cur, database_engine, *args, **kwargs):
pass


def run_create(cur, database_engine, *args, **kwargs):
if isinstance(database_engine, PostgresEngine):
select_clause = """
Expand Down
4 changes: 0 additions & 4 deletions synapse/storage/schema/main/delta/58/06dlols_unique_idx.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,6 @@
logger = logging.getLogger(__name__)


def run_upgrade(*args, **kwargs):
pass


def run_create(cur: Cursor, database_engine: BaseDatabaseEngine, *args, **kwargs):
# some instances might already have this index, in which case we can skip this
if isinstance(database_engine, PostgresEngine):
Expand Down
4 changes: 0 additions & 4 deletions synapse/storage/schema/main/delta/58/11user_id_seq.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,3 @@ def run_create(cur, database_engine, *args, **kwargs):

next_id = find_max_generated_user_id_localpart(cur) + 1
cur.execute("CREATE SEQUENCE user_id_seq START WITH %s", (next_id,))


def run_upgrade(*args, **kwargs):
pass
4 changes: 0 additions & 4 deletions synapse/storage/schema/main/delta/59/01ignored_user.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,6 @@
logger = logging.getLogger(__name__)


def run_upgrade(cur: Cursor, database_engine: BaseDatabaseEngine, *args, **kwargs):
pass


def run_create(cur: Cursor, database_engine: BaseDatabaseEngine, *args, **kwargs):
logger.info("Creating ignored_users table")
execute_statements_from_stream(cur, StringIO(_create_commands))
Expand Down
4 changes: 0 additions & 4 deletions synapse/storage/schema/main/delta/61/03recreate_min_depth.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,3 @@ def run_create(cur: Cursor, database_engine: BaseDatabaseEngine, *args, **kwargs
(6103, 'replace_room_depth_min_depth', '{}', 'populate_room_depth2')
"""
)


def run_upgrade(cur: Cursor, database_engine: BaseDatabaseEngine, *args, **kwargs):
pass
4 changes: 0 additions & 4 deletions synapse/storage/schema/state/delta/47/state_group_seq.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,3 @@ def run_create(cur, database_engine, *args, **kwargs):
start_val = row[0] + 1

cur.execute("CREATE SEQUENCE state_group_id_seq START WITH %s", (start_val,))


def run_upgrade(*args, **kwargs):
pass