Comic Rework, New Scanner, Foundation Overahul (is this a full release?)

[therobbiedavis]

This is a MASSIVE PR containing over 170 dev hours and 30K lines of code. This brings support for flexible number support (chapter 0, chapter -1, chapter ∞), a new Parsing architecture, and a new library ComicVine that is for the hardcode comic collectors and aligns with best practices from Mylar/ComicVine/CBL.

As this PR is so large, expect all external apps to not work with the nighties until developers have time to adapt. This PR also requires up to 10 minutes to run mandatory migrations. You must wait this whole time. Stopping prematurely will lead to data loss or a corrupted DB.

This PR also includes a lot of manual migrations to keep reading progress, especially around loose leaf chapters and specials. I spent 20 hours working the migrations to try my best to retain as much progress as possible. Kavita will export a progress_export.csv in your config folder in case anything is missed during the migrations or you loose data after the force scan.

You MUST do Force Scans on all libraries for things to work correctly. This is mandatory!

Added

• Added: Chapters now sort by SortOrder. SortOrder is generated from chapter number, but can be overridden (in a future update). This allows for custom sorting for issues that aren't purely numerical. SortOrder will handle cases like 19, 19.HU, 19.BD, 20 without any user intervention.
• Added: You can now have Chapter 0, -1, etc. and Volume 0
• Added: This fully supports chapters with non-numeric issue numbers, like Alpha, 19.HU, etc.
• Added: Added a new library type: ComicVine which is aimed at users of Mylar or have well tagged libraries following ComicVine. This library works different and can support multiple folders with the same series parallel to library root.
• Added: New Series Relationship of Annual.
• Added: Imprints, Locations, and Teams are now read from ComicInfo, will display on the UI and can be filtered.
• Added: (Scanner) Added a new LowestFolderPath. This is the lowest path that contains all the series files and is used by ScanFolder/Series to avoid extra work.
• Added: There is a one-time progress export to config/progress_export.csv in case of data loss.

Changed

• Changed: Loose Leaf volumes are now encoded with the number -100000 instead of 0. This now means users can have a Volume 0 without issue.
• Changed: Specials are now stored in their own volume with number 100000.
• Changed: Streamlined a lot of the code for generation of numbers for the UI
• Changed: Specials will now order by SP marker numbers, else fallback to natural sort
• Changed: Moved comicinfo parsing into the default parser so that it can override data earlier.
• Changed: Clean up a log message when no files found, to ensure the admin checks the Library settings as well.
• Changed: Search will now search against a chapter range and will also fallback if there is no title (comicinfo title) to the range.
• Changed: Image Library is now more aligned to reported usage (Fixes Folder structure with raw images mostly doesn't work #2769, Fixes Incorrect volume / chapter numbering when using raw images #2762)
• Changed: Changed chapter title format to Chapter {number} - {title} (assuming title exists), else Chapter {number}. For Books, nothing has changed.
• Changed: Series Detail will collapse automatically on desktop if there are too many tags.
• Changed: Comic/ComicVine libraries will never show Storyline or Volumes (unless there are no issues/specials or multiple volumes exist)
• Changed: Minor changes to how PDFs parse to make them a tad more smart. Nothing major, will do the overhaul later.
• Changed: Brought back parallel processing of series in the scanner.
• Changed: Trim strings from ComicInfo more aggressively to prevent weird parsing for things like Number.
• Changed: Optimized a lot of the scanner and cleaned up the code deeply.
• Changed: Adjusted the special parsing for Annuals to account for 'Series Annual \d'
• Changed: Dependency and security updates
• Changed: Added a flag to the CBL import to use ComicVine parsing for CBL Import (and set to true as all CBLs to my knowledge are using this new library).
• Changed: Kavita now handles non-float numbers from CBL Imports (including special cases for comics)
• Changed: (Kavita+) During library setting modal, changing type will now automatically change the Allow Scrobbling control, if the underlying library type is not Kavita+ applicable.

Fixed

• Fixed: Fixed a bug with looping around to volume 1 at the end of a reading session when it shouldn't
• Fixed: Fixed a case where an exception was thrown, scanner wasn't existing and instead invoked metadata service with bad entities
• Fixed: Fixed some bad styling on Chapter metadata (drawer)
• Fixed: Fixed a missing case where parser wasn't setting Special Volume correctly (no impact)
• Fixed: Fixed a bad localization on edit reading list modal.
• Fixed: Fixed another check around rescheduling Kavita+ tasks once a valid check came through. There was a lack of a checking mechanism and Kavita was doing more than needed.
• Fixed: I believe the Foreign Key constraint issue is solved once and for all (Fixed FOREIGN KEY constraint failed #2747)
• Fixed: Fixed a rendering bug on CBL validation where the UI reports success but also a warning message appears
• Fixed: Fixed a bug where Translators couldn't be updated in Edit Series Modal.
• Fixed: Fixed an edge case where some bad data can exist in Kavita and the priming of the tag service could fail.

Removed

• Removed .kavitaignore file support as Library Exclude patterns are much easier to use and flexible enough.

API

• VolumeDto reverted the float on Number field back to int
• ChapterDto Range will be the chapter number. If a special, will be the special name. MinNumber can always be checked for the special encoding
• Encoding: Special (100000), Loose Leaf Volume (-100000).
• stats/user/reading-history has a dto change, making ChapterNumber a float instead of a string.
• Tachiyomi routes now use their own Dto to avoid having to break apis there.
• Added SortOrder on the ChapterDto. This is a float that can be used to order the chapters within the Volume. By default, all APIs will already order by this.
• SeriesMetadataDto (for updating series metadata) now has Teams, Locations, Imprints and their lock fields: teamLocked, locationLocked, imprintLocked.

External Apps

External app developers have been notified already of changes

• CDisplayEx
  • Single volumes will show Vol X #-100000
  • Special Volumes are showing Vol. 100000
  • Special titles aren't rendering
• Tachiyomi/Mihon
  • Loose leaf chapters will show Vol 100000
• Komf
  • Series metadata update may fail as there are new fields required.

Closes #2542