diff --git a/docs/Writerside/snippets/compose-file.yaml b/docs/Writerside/snippets/compose-file.yaml index 1ffacc3..b68a469 100644 --- a/docs/Writerside/snippets/compose-file.yaml +++ b/docs/Writerside/snippets/compose-file.yaml @@ -24,9 +24,10 @@ services: condition: service_healthy postgres: - image: postgres:17.1 + image: postgres:17.2-alpine container_name: postgres restart: unless-stopped + shm_size: 2G environment: PGDATA: /var/lib/postgresql/data/pgdata POSTGRES_USER: postgres diff --git a/docs/Writerside/snippets/default-settings.json b/docs/Writerside/snippets/default-settings.json index 8351f97..6bfa32c 100644 --- a/docs/Writerside/snippets/default-settings.json +++ b/docs/Writerside/snippets/default-settings.json @@ -26,7 +26,9 @@ "Imdb": { "EnableImportMatching": true, "EnableEndpoint": true, - "MinimumScoreMatch": 0.85 + "MinimumScoreMatch": 0.85, + "UseAllCores": false, + "NumberOfCores": 2 }, "Ingestion": { "ZurgInstances": [], @@ -47,4 +49,4 @@ "BatchSize": 5000 } } -} \ No newline at end of file +} diff --git a/docs/Writerside/topics/Configuration.md b/docs/Writerside/topics/Configuration.md index 324c91d..4c02c46 100644 --- a/docs/Writerside/topics/Configuration.md +++ b/docs/Writerside/topics/Configuration.md @@ -108,6 +108,14 @@ _Default: `true`_ _The minimum score match for IMDB search results. Closer to 1 means a more exact match has to occur. A value between 1 and 0._ _Default: `0.85`_ +**Imdb.UseAllCores** +_Should title matching use all available processor cores on the machine?_ +_Default: `false`_ + +**Imdb.NumberOfCores** +_The number of processor cores to use for parallel operations during matching_ +_Default: `2`_ + ### Ingestion Configuration **Ingestion.ZurgInstances** _A list of Zurg instances to scrape from._ diff --git a/docs/Writerside/topics/Scraper.md b/docs/Writerside/topics/Scraper.md index 37478be..b23223e 100644 --- a/docs/Writerside/topics/Scraper.md +++ b/docs/Writerside/topics/Scraper.md @@ -18,7 +18,7 @@ To run the scraper manually, you can use the following command: docker compose stop zilean docker run -it --rm \ - -v ./settings.json:/app/data/settings.json \ + -v ./settings.json:/app/data/settings.json \ --name zileanscraper \ ipromknight/zilean:latest \ /app/scraper @@ -28,41 +28,42 @@ This will run the scraper, and output the results to the console. The following arguments are available when running the scraper: -| Argument | Description | -|---------------------|-------------------------------------------------------------------------------------------------------| -| `dmm-sync` | Run the DMM scraper to import DMM Hash Lists. | -| `generic-sync` | Run the Ingestion syncer to sync from Zilean and Zurg instances in your config. | -| `resync-imdb -s` | Force re-ingest IMDB metadata. | -| `resync-imdb -s -t` | Force re-ingest IMDB metadata, then try to update any entries in your database that do not have an id | +| Argument | Description | +|---------------------|------------------------------------------------------------------------------------------------------------| +| `dmm-sync` | Run the DMM scraper to import DMM Hash Lists. | +| `generic-sync` | Run the Ingestion syncer to sync from Zilean and Zurg instances in your config. | +| `resync-imdb -s` | Force re-ingest IMDB metadata. | +| `resync-imdb -s -t` | Force re-ingest IMDB metadata, then try to update any entries in your database that do not have an imdb id | +| `resync-imdb -t` | Try to update any entries in your database that do not have an imdb id | +| `resync-imdb -s -a` | Force re-ingest IMDB metadata, then Rematch / update any entries in your database with a new ImdbID | +| `resync-imdb -a` | Rematch / update any entries in your database with a new ImdbID | ## Examples ### Running the IMDB Resync with Title Update ```bash -docker run -it --rm \ - -v ./settings.json:/app/data/settings.json \ - --name zileanscraper \ - ipromknight/zilean:latest \ +docker exec -it zilean \ /app/scraper resync-imdb -s -t ``` +### Running the IMDB Resync to rematch ALL Titles + +```bash +docker exec -it zilean \ + /app/scraper resync-imdb -a +``` + ### Running the DMM Sync ```bash -docker run -it --rm \ - -v ./settings.json:/app/data/settings.json \ - --name zileanscraper \ - ipromknight/zilean:latest \ +docker exec -it zilean \ /app/scraper dmm-sync ``` ### Running the Generic Sync ```bash -docker run -it --rm \ - -v ./settings.json:/app/data/settings.json \ - --name zileanscraper \ - ipromknight/zilean:latest \ +docker exec -it zilean \ /app/scraper generic-sync ```