Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add custom migration runner, forbid some diesel migration commands, fix old migrations #4673

Open
wants to merge 96 commits into
base: main
Choose a base branch
from

Conversation

dullbananas
Copy link
Collaborator

@dullbananas dullbananas commented Apr 28, 2024

Reverting and testing migrations will be done with lemmy_server migration commands, and diesel migration commands will do nothing and fail when trying to run or revert migrations.

Benefits:

  • Changes in replaceable_schema will no longer require a corresponding migration
  • Migrations will now be run and reverted without the "r" schema applied, instead of needing to work both with and without it
  • All migrations, not just the newest one, will be tested with schema diff (moved to Improve migration diff check and fix old migrations #5204)
  • Multiple lemmy_server processes trying to run migrations at the same time will now be prevented
  • Name and duration of each migration will be displayed when run in lemmy_server

#4333 (comment)

The new tests (even just reverting all migrations) revealed some mistakes in old migrations, so I fixed those, and I also discovered that 'now'::timestamp in old views used the timestamp of the view's creation instead of the current timestamp.

@dullbananas dullbananas changed the title Custom migration runner Custom migration runner (diesel migration no longer allowed) Apr 28, 2024
@dullbananas dullbananas changed the title Custom migration runner (diesel migration no longer allowed) Custom migration runner (some diesel migration commands no longer allowed) May 11, 2024
@dullbananas dullbananas changed the title Custom migration runner (some diesel migration commands no longer allowed) Add custom migration runner, forbid some diesel migration commands, fix old migrations May 13, 2024
@dullbananas dullbananas marked this pull request as ready for review May 28, 2024 14:24
@dessalines
Copy link
Member

Sry about being slow to review this, I'd like it to come after the 0.19.4 release.

@Nutomic Nutomic added this to the 0.20.0 milestone Sep 10, 2024
Copy link
Member

@dessalines dessalines left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The editing (even older broken migrations) seems a bit scary (but you did explain why you need to do this.). And I don't understand the reason for all of the custom code for diff checking.

But overall I think this is a good idea.

when: *slow_check_paths

check_diesel_migration:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't check / test the migrations anymore in CI?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was moved to the unit tests in schema_setup.rs


const PATTERN_LEN: usize = 19;

pub fn check_dump_diff(before: String, after: String, label: &str) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is so much custom code necessary for checking a diff?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The added complexity is now in #5204 instead of this PR, and I will try to make the comments more clear about the reason in that PR

@@ -220,14 +221,36 @@ BEGIN
post_aggregates_view
WHERE
id = NEW.post_id;
-- Force the hot rank as zero on week-older posts
-- Update the comment hot_ranks as of last week
UPDATE
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are really old migrations that are updating tables and views that aren't used anymore, whats going on?

@dullbananas
Copy link
Collaborator Author

I've wanted to split this PR so the changed diff behavior and the migration changes it requires are separate, but I've been putting it off. I will do that soon.

For the migration changes, I now think the diff checker should only check migrations after the migration that removes all views. That should allow removing most of the migration changes. I will work on this for a while after I make the PR with the simpler changes.

@dullbananas dullbananas marked this pull request as draft September 16, 2024 17:54
@dullbananas dullbananas marked this pull request as ready for review November 20, 2024 06:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants