Conversation
This comment has been minimized.
This comment has been minimized.
src/invidious/database/migrator.cr
Outdated
| ) | ||
| SQL | ||
|
|
||
| @db.exec("DROP TABLE backup.#{table_name}") |
There was a problem hiding this comment.
The problem I see with that is that it might nuke a perfectly valid backup if invidious is interrupted mid-migration and the user tries to run it again immediately after.
There was a problem hiding this comment.
so there's stage 1,2,3. the migration can get interrupted pre-1,pre-2,pre-3 & post-3.
pre-1 : The migration wouldn't be run (hopefully upon re-attempting migration the migration is question will still be queued up)
pre-2 : At this point, there exists a table backup.cool_table - it is empty. An interuption now, would leave a empty table that'd be re-populated upon re-attempt
pre-3 : at this point, there is a backup.cool_table and a lack of backup.cool_table (this latter one being cool_tables version right before migration). At this point, critical data is destroyed and we must replenish it ASAP - an interuption now would be fatal
Ok - I see what you're saying, what if I created a copy of backup.cool_table (Pre-Migration's Version) earlier on?
There was a problem hiding this comment.
Okay, I see! Though I still don't understand something:
Stage 1: You create the backup table (if it doesn't already exist)
Stage 2: You remove that backup table
Stage 3: You create a new backup table from the content of the regular table
If my understanding is correct, stage 1 is useless.
There was a problem hiding this comment.
The following is based off the assumption that former broquemonsieur would write SQL that looks redundant/hacky but actually is doing the right thing due to SQL's odd and vague limitations where you cannot do something that should be simple or the only way to achieve a simple operation (that seemingly should be a one-liner) is the combo of two ops (a.k.a. two-liner). Present day broquemonsieur has failing memory but can vouch for this ^ :)
Meet Sonic, a new Invidious host. At this point in her users have a few playlists saved and Sonic has not migrated her DB...ever. Her users playlists look like this:
| title | author |
|---|---|
| Mark Hamill Moments | kilowattwomxn |
| Awesome Trailers | movieDad |
Sonic migrates her DB:
Stage 1 - Creates a table backup.playlists. It is empty
Stage 2 - This op done nothing (it fails silently). (If Sonic were to have a backup of her users' playlists already there, it would be destroyed now to make way for a snapshot of her user's present-day playlists) tl;dr This stages deletes the prior version of backup.playlists, not the one you just read out in Stage 1 (Yes, SQL forums agree this is weird that SQL has to be written this way lol)
Stage 3 - This op populates backup.playlists with
| title | author |
|---|---|
| Mark Hamill Moments | kilowattwomxn |
| Awesome Trailers | movieDad |
Meet Amy, a jaded Invidious host who already has a backup for her users' playlists.
Amy migrates her DB:
you get the idea :)
That should clear it up (mainly I was trying to explain for my recollection to myself lol). But basically, the system is vulnerable . At time this comment is sent, I begin efforts to prevent memory loss given an asteroid strike at any point in time
There was a problem hiding this comment.
Hi! Reporting back from my coding session. Now, we only destroy the "second most recent" backup. Whenever an asteroid strikes, the "most recent" backup will always be preserved.
There was a problem hiding this comment.
Asteroid strikes are useless analogies.
Meet Dale, he is chilling in Mexican beach propping a MacBook Air on pool tube. He is SSH'd to his Invidious server and tries to do DB migration - a major part of which is DB backup.
The hotel Wifi could go out, the MacBook could get wet or the Server itself could shutdown at any point during this critical time and still Dale's users would not lose
- "most recent" backup
- regular DB tables
The "2nd most recent" backup, however, is susceptible to being lost in the event of a crash but when you hear "2nd most recent" think "safety cap for most recent" - it takes the bullet for it
src/invidious/database/migrator.cr
Outdated
| ) | ||
| SQL | ||
|
|
||
| @db.exec("DROP TABLE backup.#{table_name}") |
There was a problem hiding this comment.
Okay, I see! Though I still don't understand something:
Stage 1: You create the backup table (if it doesn't already exist)
Stage 2: You remove that backup table
Stage 3: You create a new backup table from the content of the regular table
If my understanding is correct, stage 1 is useless.
| table_names_request = <<-SQL | ||
| SELECT tablename FROM pg_catalog.pg_tables | ||
| WHERE schemaname = 'public' | ||
| SQL |
There was a problem hiding this comment.
Also, does this copy views, data types, and other database elements?
There was a problem hiding this comment.
That SQL query retrieves only table names from the pg_tables view in PostgreSQL. It does not include views, data types, or other database elements.
Views: Views are stored in the pg_views catalog.
Data Types: Data types are found in the pg_type catalog.
Other Database Elements: Other elements, such as indexes or sequences, are in different catalog tables like pg_indexes and pg_sequences.
If you need to include views or other database elements, additional queries targeting the respective catalog tables would be required.
3 participants
the first step in addressing automatic migration
#4190 (comment)