v0.6.2

What's Changed

• Support Torznab query by season and episode by @mgdigital in #130

Full Changelog: v0.6.1...v0.6.2

v0.6.1

What's Changed

• Torznab imdbid optimisation by @mgdigital in #129: A simple optimisation that significantly improves Torznab performance when searching by IMDB ID.

Full Changelog: v0.6.0...v0.6.1

v0.6.0

This release introduces a significant optimisation of aggregations (counts).

It takes advantage of the fact that a Postgres query plan can tell you the cost of a query up-front, along with a rough estimate of the count based on indexes. All count queries now have a "budget", defaulting to 5,000. If the budget is exceeded according to the query plan, then the estimate will be returned (and the UI will display an estimate symbol ~ next to the associated count), otherwise the query will be executed and an exact count will be returned.

The accuracy of the estimate seems to be within 10-20% of the exact count in most cases - though accuracy depends on selected filter criteria and what is being counted, I've noticed bigger discrepancies but overall it seems like an acceptable trade-off.

The background cache warmer has been removed and aggregations are now real time again (the cache warmer was at best a short term mitigation while I figured out a better solution). The cache TTL has been reduced to 10 minutes. It was previously increased to allow the cache warmer to be run less frequently.

There are also some adjustments to the indexes that improve performance and the accuracy of estimations. For large indexes the migration may take a while to run - please allow it to run uninterrupted: in my tests on 12 million torrents it took 15 minutes.

What's Changed

• Use budgeted count for aggregations by @mgdigital in #128

Full Changelog: v0.5.1...v0.6.0

v0.5.1

What's Changed

• Fix issue with original language not being attached to content by @mgdigital in #123

Full Changelog: v0.5.0...v0.5.1

v0.5.0

v0.5.0 focuses on text search performance. The way torrents are indexed and queried for text search has been completely reworked. See the updated docsite for more details on the new search syntax: https://bitmagnet.io/tutorials/text-search.html

Upgrading

The v0.5.0 upgrade process is fully automatic (unless you've used the /import endpoint, read below). All torrents must be reprocessed before they will work with text search. Just wait for the background job queue to work through them, then enjoy better and faster search!

If you have a large library the queue may take several hours, or even a day or more to work through. To check on progress, visit the /metrics endpoint and note metrics that contain process_torrent. The key ones being:

asynq_tasks_enqueued_total{queue="process_torrent",state="completed"}
asynq_tasks_enqueued_total{queue="process_torrent",state="pending"}
asynq_tasks_failed_total{queue="process_torrent"}

Until the queue has worked down you won't see any newly crawled torrents (though the crawler will be running!)

When you start the container for the first time you should see a progress bar for queueing the reprocess - please don't interrupt the process until you see the completed progress bar. If the process is interrupted you may have to requeue manually using the command bitmagnet reprocess --classifyMode=skip.

Note: I'm aware there are still some performance issues that need addressing for aggregations. These will be addressed separately - this release focuses on text search.

If you've used the /import endpoint, you will lose any classification hints that were part of the import. That is, if torrents were assigned to particular content types, video resolutions etc, this information will be lost and the classifier will attempt to classify from scratch without these hints. To recover the hints, the original import data must be reimported - it was necessary to separate these hints from the final classification result as the 2 things were previously conflated, making it impossible to separate out the data as part of this upgrade.

IMPORTANT: the format for the /import endpoint has changed slightly, please check the updated documentation at https://bitmagnet.io/tutorials/import.html (note there is a slight change to the jq query in the final code snippet).

What's Changed

• Bump golang.org/x/crypto from 0.14.0 to 0.17.0 by @dependabot in #91
• Lazy instantiation for services with side effects by @mgdigital in #98
• Search query string rework by @mgdigital in #96
• Cache tweaks by @mgdigital in #99
• Add reclassify command by @mgdigital in #100
• Disable Gorm easer plugin which sometimes causes zero results by @mgdigital in #101
• Bump follow-redirects from 1.15.2 to 1.15.4 in /webui by @dependabot in #102
• Replace problematic chars in tsvector by @mgdigital in #103
• Fix search ordering by @mgdigital in #104
• Fix index out of range by @mgdigital in #105
• Refactor torrent processing by @mgdigital in #106
• Add 0.5.0 upgrade hook to queue reprocessing of all torrents by @mgdigital in #107
• Add missing criteria for collections by @mgdigital in #110
• Video classifier fixes by @mgdigital in #111
• Upgrade dependencies by @mgdigital in #112
• Reprocess tweaks by @mgdigital in #116
• Torrent processor optimisations by @mgdigital in #119
• Torznab: disable total count; fix index out of range by @mgdigital in #120
• v0.5.0 documentation updates by @mgdigital in #108
• Reduce crawler persistence batch size by @mgdigital in #121

Full Changelog: v0.4.1...v0.5.0

v0.5.0-beta.2

Thanks for giving this beta a try and reporting any issues!

v0.5.0 focuses on text search performance. The way torrents are indexed and queried for text search has been completely reworked; the final release will include some documentation updates for the website that will go into further detail.

Upgrading

To try the beta, set your image tag to ghcr.io/bitmagnet-io/bitmagnet:0.5.0-beta.2.

What's Changed

Beta 2 builds upon Beta 1 with the following tweaks and fixes:

• Add missing criteria for collections by @mgdigital in #110 (fixes a bug with filtering by genre)
• Video classifier fixes by @mgdigital in #111 (video-related attributes were being incorrectly assigned to unknown content types)
• Upgrade dependencies by @mgdigital in #112
• Reprocess tweaks by @mgdigital in #116: This is the most important change in Beta 2, it significantly reduces the time needed for reprocessing all torrents (should be roughly 2x faster), by not making API calls that aren't strictly necessary. For users who have not configured a personal TMDB API key, this change is pretty much essential as the reprocessing time will be bottlenecked by the rate limiting on this API.

If you've already installed Beta 1

You don't need to do anything, but if your queue is still working away and you'd like to take advantage of the faster reprocessing you'll need to follow these steps.
If you've already installed Beta 1, AND you're using the default TMDB API key, then it's highly recommended to follow these steps:

1. Upgrade to Beta 2
2. Clear your queue manually by flushing Redis (e.g. docker exec -it bitmagnet-redis redis-cli flushall)
3. Re-queue your torrents using the CLI command reprocess --classifyMode=skip (e.g. docker exec -it bitmagnet bitmagnet reprocess --classifyMode=skip)

Full Changelog: v0.5.0-beta.1...v0.5.0-beta.2

v0.5.0-beta.1

Thanks for giving this beta a try and reporting any issues!

v0.5.0 focuses on text search performance. The way torrents are indexed and queried for text search has been completely reworked; the final release will include some documentation updates for the website that will go into further detail.

Upgrading

To try the beta, set your image tag to ghcr.io/bitmagnet-io/bitmagnet:0.5.0-beta.1.

The v0.5.0 upgrade process is fully automatic (unless you've used the /import endpoint, read below). All torrents must be reprocessed before they will work with text search. Just wait for the background job queue to work through them, then enjoy better and faster search!

The queue may take several hours to work down if you have a large library. To check on progress, visit the /metrics endpoint and note metrics that begin asynq_tasks_enqueued_total{queue="process_torrent". Until the queue has worked down you won't see any newly crawled torrents (though the crawler will be running!)

When you start the container for the first time you should see a progress bar for queueing the reprocess - please don't interrupt the process until you see the completed progress bar. If the process is interrupted you may have to requeue manually using the command bitmagnet reprocess.

Note: I'm aware there are still some performance issues that need addressing for aggregations. These will be addressed separately - this release focuses on text search.

If you've used the /import endpoint, you will lose any classification hints that were part of the import. That is, if torrents were assigned to particular content types, video resolutions etc, this information will be lost and the classifier will attempt to classify from scratch without these hints. To recover the hints, the original import data must be reimported - it was necessary to separate these hints from the final classification result as the 2 things were previously conflated, making it impossible to separate out the data as part of this upgrade. IMPORTANT: the format for the /import endpoint has changed slightly, and the documentation for this on a branch here during the beta phase: https://github.com/bitmagnet-io/bitmagnet/blob/0.5.0-docs/bitmagnet.io/tutorials/import.md (note there is a slight change to the jq query in the final code snippet).

What's Changed

• Bump golang.org/x/crypto from 0.14.0 to 0.17.0 by @dependabot in #91
• Lazy instantiation for services with side effects by @mgdigital in #98
• Search query string rework by @mgdigital in #96
• Cache tweaks (increase TTL and cache warmer interval) by @mgdigital in #99
• Add reclassify command by @mgdigital in #100
• Disable Gorm easer plugin which sometimes causes zero results by @mgdigital in #101
• Bump follow-redirects from 1.15.2 to 1.15.4 in /webui by @dependabot in #102
• Fix search ordering by @mgdigital in #104
• Refactor torrent processing by @mgdigital in #106
• Add 0.5.0 upgrade hook to queue reprocessing of all torrents by @mgdigital in #107

Full Changelog: v0.4.1...v0.5.0-beta.1

v0.4.1

What's Changed

• Remove asynqmon by @mgdigital in #93

Remove asynqmon altogether: The new version hasn't been published properly, and the previous version doesn't support the Redis wrapper.

Full Changelog: v0.4.0...v0.4.1

v0.4.0

What's Changed

• Add gluetun configuration example by @mgdigital in #83
• Downgrade asynqmon by @mgdigital in #92
• Reduce aggregations in web UI by @mgdigital in #87 (Fix #85):
  • Accurate counts in the web UI will only be shown for the top-level filters; when deep-filtering, the counts will show a less than or equal sign: ≤, indicating the number is a "maximum possible" rather than an accurate count; I'm still not sure about the UX aspect of showing the ≤ which could be confusing if you're not sure what it means, but I figured it's better than showing nothing at all - open to suggestions!
  • A background process keeps the in-memory query cache warm for the top-level aggregations (by default every 10 minutes), allowing them to be served instantly
  • The query cache TTL has been increased to 20 minutes; improved performance is preferable to seeing the most up-to-date information
  • Pagination has been refactored to account for not knowing the total number of pages; internally the search engine will request 1 more item than it needs to know if there's a next page to advance to
  • Some general refactoring of the web app

Full Changelog: v0.3.1...v0.4.0

v0.3.1

What's Changed

• Split webUI query by @mgdigital in #82

Split the web UI query into 2 parts; one part fetches items and the other fetches aggregations (counts). This makes the UI seem more responsive as we don't have to wait for the aggregations to render the item results.

Full Changelog: v0.3.0...v0.3.1