refactor(http): remove retry logic, add Amazon provider and cache cleanup

.github/workflows/continuous-integration-test.yml

@@ -4,6 +4,9 @@ on: [push, pull_request, workflow_dispatch]
 jobs:
   test:
     runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      statuses: write
     strategy:
       matrix:
         python-version: ["3.10", "3.11", "3.12"]
@@ -38,3 +41,4 @@ jobs:
       continue-on-error: true
       with:
         format: cobertura
+        github-token: ${{ secrets.GITHUB_TOKEN }}

INSTALL.md

@@ -31,29 +31,25 @@ in your Flask/Invenio configuration:
 ```python
 RERO_INVENIO_THUMBNAILS_PROVIDERS = [
     "files",
-    "open library",
-    "bnf",
-    "dnb",
     "google books",
     "google api",
+    "amazon",
+    "bnf",
+    "internet archive",
+    "open library",
 ]
 
 # Directory used by FilesProvider
 RERO_INVENIO_THUMBNAILS_FILES_DIR = "/path/to/thumbnails"
 RERO_INVENIO_THUMBNAILS_CACHE_EXPIRE = 3600
-# Retry configuration (defaults shown)
-RERO_INVENIO_THUMBNAILS_RETRY_ENABLED = True
-RERO_INVENIO_THUMBNAILS_RETRY_ATTEMPTS = 5
-RERO_INVENIO_THUMBNAILS_RETRY_BACKOFF_MULTIPLIER = 0.5
-RERO_INVENIO_THUMBNAILS_RETRY_BACKOFF_MIN = 1
-RERO_INVENIO_THUMBNAILS_RETRY_BACKOFF_MAX = 10
 ```
 
-You can also disable retries globally (useful in tests) via environment variable:
-
-```bash
-export RERO_THUMBNAILS_DISABLE_RETRIES=true
-```
+> **DNB / MVB licence requirement**: The `dnb` provider is **not** enabled by
+> default. Its cover images are sourced from VLB (Verzeichnis Lieferbarer
+> Bücher, operated by MVB GmbH) and are subject to copyright. To enable it,
+> add `"dnb"` to your instance-specific `RERO_INVENIO_THUMBNAILS_PROVIDERS`
+> list. A valid data licence agreement with MVB is required.
+> Contact: kundenservice@mvb-online.de
 
 ## Application integration
 

README.md

@@ -29,10 +29,9 @@ RERO Invenio extension to discover book thumbnail URLs from multiple providers.
 ## Features
 
 - **Multiple Providers**: Chainable providers query external services in order and return the first available thumbnail URL
-- **Built-in Providers**: FilesProvider (local files), OpenLibraryProvider, BnfProvider, DnbProvider, GoogleBooksProvider, GoogleApiProvider
+- **Built-in Providers**: FilesProvider (local files), GoogleBooksProvider, GoogleApiProvider, AmazonProvider, DnbProvider, BnfProvider, InternetArchiveProvider, OpenLibraryProvider
 - **Plugin Architecture**: Extensible via entry points - register custom providers without modifying core code
 - **Smart Caching**: Redis-based caching with configurable TTL
-- **Robust HTTP Handling**: Configurable retry logic with exponential backoff for external providers
 - **RESTful API**: JSON endpoint for thumbnail URL retrieval
 
 ## Custom Providers
@@ -75,24 +74,30 @@ Configure the providers and files dir in your application config:
 
 ```python
 # Provider configuration (optional - if not set, all discovered providers are used)
-RERO_INVENIO_THUMBNAILS_PROVIDERS = ["files", "open library", "bnf", "dnb", "google books", "google api"]
+RERO_INVENIO_THUMBNAILS_PROVIDERS = [
+    "files",
+    "google books",
+    "google api",
+    "amazon",
+    "bnf",
+    "internet archive",
+    "open library",
+]
 
 # Files provider configuration
 RERO_INVENIO_THUMBNAILS_FILES_DIR = "/path/to/thumbnails"
 
 # Cache configuration
 RERO_INVENIO_THUMBNAILS_CACHE_EXPIRE = 3600
-
-# Retry config (defaults shown)
-RERO_INVENIO_THUMBNAILS_RETRY_ENABLED = True
-RERO_INVENIO_THUMBNAILS_RETRY_ATTEMPTS = 5
-RERO_INVENIO_THUMBNAILS_RETRY_BACKOFF_MULTIPLIER = 0.5
-RERO_INVENIO_THUMBNAILS_RETRY_BACKOFF_MIN = 1
-RERO_INVENIO_THUMBNAILS_RETRY_BACKOFF_MAX = 10
-# Disable retries globally via env (e.g., for tests)
-# export RERO_THUMBNAILS_DISABLE_RETRIES=true
 ```
 
+> **DNB / MVB license requirement**: The `dnb` provider is **not** enabled by
+> default. Its cover images are sourced from VLB (Verzeichnis Lieferbarer
+> Bücher, operated by MVB GmbH) and are subject to copyright. To enable it,
+> add `"dnb"` to your instance-specific `RERO_INVENIO_THUMBNAILS_PROVIDERS`
+> list. A valid data licence agreement with MVB is required.
+> Contact: kundenservice@mvb-online.de
+
 Initialize the extension:
 
 ```python

docs/get_thumbnail_url.mmd

@@ -11,7 +11,10 @@ flowchart TD
     C -->|no| G[read PROVIDERS list\nfrom app.config]
 
     G --> H{providers\nempty?}
-    H -->|yes| Y[return None, None]
+    H -->|yes| Y{cached=True?}
+    Y -->|yes| Y1[cache.set JSON null+null]
+    Y1 --> Y2([return None, None])
+    Y -->|no| Y2
 
     H -->|no| I[for each provider_name]
     I --> J{name in\nPROVIDERS?}
@@ -32,11 +35,12 @@ flowchart TD
     P -->|no| Z
 
     I -->|exhausted| R{cached=True?}
-    R -->|yes| S[cache.set\nJSON null+last_provider]
-    S --> T([return None, last_provider])
+    R -->|yes| S[cache.set\nJSON null+null]
+    S --> T([return None, None])
     R -->|no| T
 
     subgraph providers [registered providers – entry points resolved at registry build]
+        AMAZON[AmazonProvider\namazon]
         BNF[BnfProvider\nbnf]
         DNB[DnbProvider\ndnb]
         FILES[FilesProvider\nfiles]

pyproject.toml

@@ -3,7 +3,7 @@
 # ---------------------------------------------------------------------------
 
 [build-system]
-requires = ["uv_build"]
+requires = ["uv_build>=0.6,<0.12"]
 build-backend = "uv_build"
 
 [tool.uv.build-backend]
@@ -31,9 +31,9 @@ dependencies = [
     "invenio-app>=2.3.0",
     "invenio-base>=2.4.0",
     "invenio-cache (>=2.0.0,<3.0.0)",
+    "isbnlib>=3.10.14",
     "pillow>=11.3.0",
     "requests (>=2.23.0)",
-    "tenacity>=8.0.0",
 ]
 
 [dependency-groups]
@@ -67,6 +67,7 @@ dnb = "rero_invenio_thumbnails.contrib.dnb.api:DnbProvider"
 "google books" = "rero_invenio_thumbnails.contrib.google_books.api:GoogleBooksProvider"
 "google api" = "rero_invenio_thumbnails.contrib.google_api.api:GoogleApiProvider"
 "internet archive" = "rero_invenio_thumbnails.contrib.internet_archive.api:InternetArchiveProvider"
+amazon = "rero_invenio_thumbnails.contrib.amazon.api:AmazonProvider"
 
 # ---------------------------------------------------------------------------
 # Tool configurations

rero_invenio_thumbnails/__init__.py

@@ -23,7 +23,6 @@
     - Chainable provider pattern (first match wins)
     - Redis caching integration via invenio_cache
     - HTTP blueprint endpoint for thumbnail serving
-    - Retry/backoff support for external provider queries
     - Comprehensive error handling and logging
 
 Usage:
@@ -42,24 +41,30 @@
 from importlib.metadata import version
 
 from .api import get_thumbnail_url
+from .contrib.amazon.api import AmazonProvider
 from .contrib.bnf.api import BnfProvider
 from .contrib.dnb.api import DnbProvider
 from .contrib.files.api import FilesProvider
 from .contrib.google_api.api import GoogleApiProvider
 from .contrib.google_books.api import GoogleBooksProvider
+from .contrib.internet_archive.api import InternetArchiveProvider
 from .contrib.open_library.api import OpenLibraryProvider
+from .contrib.utils import clean_all_cache
 from .ext import REROInvenioThumbnails
 
 __version__ = version("rero-invenio-thumbnails")
 
 __all__ = (
+    "AmazonProvider",
     "BnfProvider",
     "DnbProvider",
     "FilesProvider",
     "GoogleApiProvider",
     "GoogleBooksProvider",
+    "InternetArchiveProvider",
     "OpenLibraryProvider",
     "REROInvenioThumbnails",
     "__version__",
+    "clean_all_cache",
     "get_thumbnail_url",
 )

rero_invenio_thumbnails/api.py

@@ -22,9 +22,8 @@
     - Multi-provider support with fallback chain
     - Plugin-based architecture via entry points
     - Redis caching via invenio_cache
-    - Image validation (minimum 10x10 pixels)
+    - Image validation (minimum 50x50 pixels by default)
     - Dynamic image resizing
-    - Configurable retry logic for HTTP requests
 
 Provider Discovery:
     Providers are automatically discovered via the
@@ -38,7 +37,6 @@
 """
 
 import json
-from contextlib import suppress
 from importlib.metadata import entry_points
 
 from flask import current_app
@@ -143,11 +141,13 @@ def get_thumbnail_url(isbn, cached=True):
         # Try to get from cache
         if (cached_result := cache.get(cache_key)) is not None:
             # Cached result is JSON: {"url": "...", "provider": "..."}
-            with suppress(json.JSONDecodeError, AttributeError, TypeError):
+            try:
                 data = json.loads(cached_result)
                 url = data.get("url")
                 provider = data.get("provider")
                 return url, provider
+            except (json.JSONDecodeError, AttributeError, TypeError) as e:
+                current_app.logger.debug(f"Malformed cache entry for ISBN {isbn}, re-fetching: {e}")
 
     # Get cache timeout
     timeout = current_app.config.get("RERO_INVENIO_THUMBNAILS_CACHE_EXPIRE", DEFAULT_CACHE_EXPIRE)
@@ -173,9 +173,7 @@ def get_thumbnail_url(isbn, cached=True):
             return url, returned_provider
 
     # Cache None result to avoid repeated failed lookups
-    # Use last provider name or None
-    last_provider = providers[-1] if providers else None
     if cached:
-        cache_data = json.dumps({"url": None, "provider": last_provider})
+        cache_data = json.dumps({"url": None, "provider": None})
         cache.set(cache_key, cache_data, timeout=timeout)
-    return None, last_provider
+    return None, None

rero_invenio_thumbnails/config.py

@@ -38,12 +38,16 @@
 # List of thumbnail providers to query in order (first match wins).
 # Default provider order optimized for typical usage patterns.
 # Override in instance config to customize provider precedence.
+#
+# Note: "dnb" is NOT included by default because its cover images are sourced
+# from VLB (operated by MVB GmbH) and require a paid data licence.  To enable
+# it, add "dnb" to your instance-specific RERO_INVENIO_THUMBNAILS_PROVIDERS.
 RERO_INVENIO_THUMBNAILS_PROVIDERS = [
     "files",
-    "bnf",
-    "dnb",
     "google books",
     "google api",
+    "bnf",
+    "amazon",
     "internet archive",
     "open library"  # Open Library is last because it has frequent timeouts
 ]
@@ -58,18 +62,6 @@
 # Set to 0 to disable HTTP caching
 RERO_INVENIO_THUMBNAILS_HTTP_CACHE_MAX_AGE = 86400
 
-# HTTP Retry Configuration
-# Enable/disable automatic retries for failed HTTP requests to external providers
-RERO_INVENIO_THUMBNAILS_RETRY_ENABLED = True
-
-# Maximum number of retry attempts before giving up
-RERO_INVENIO_THUMBNAILS_RETRY_ATTEMPTS = 5
-
-# Exponential backoff multiplier (seconds between retries grow exponentially)
-RERO_INVENIO_THUMBNAILS_RETRY_BACKOFF_MULTIPLIER = 0.5
-
-# Minimum wait time between retries (seconds)
-RERO_INVENIO_THUMBNAILS_RETRY_BACKOFF_MIN = 1
-
-# Maximum wait time between retries (seconds)
-RERO_INVENIO_THUMBNAILS_RETRY_BACKOFF_MAX = 10
+# Default HTTP request timeout as (connect, read) in seconds for cover image fetches.
+# Increase if providers are timing out; decrease for faster fallback to next provider.
+RERO_INVENIO_THUMBNAILS_HTTP_TIMEOUT = (2, 10)

rero_invenio_thumbnails/contrib/__init__.py

@@ -29,9 +29,12 @@
 Available Providers:
     bnf: Bibliothèque nationale de France provider
         - Uses BNF catalogue API
-        - Converts ISBN to ARK identifiers
         - Covers for French publications since 2010
 
+    dnb: Deutsche Nationalbibliothek provider
+        - Uses DNB/MVB cover service
+        - Covers for German publications
+
     files: Local file storage provider
         - Searches local file system
         - Supports JPG, PNG, JPEG formats
@@ -47,14 +50,17 @@
         - JSONP callback format
         - Public access without authentication
 
+    internet_archive: Internet Archive provider
+        - Resolves ISBN to OCAID via search API
+        - Free, no authentication required
+
     open_library: Open Library covers API provider
         - Free and open-source
         - Configurable sizes (S, M, L)
         - Creative Commons licensed covers
 
     utils: Shared utility functions
         - clean_isbn(): Remove hyphens and spaces from ISBN
-        - fetch_with_retries(): HTTP requests with retry logic
         - validate_image_content(): Image validation (format, dimensions)
         - handle_provider_errors(): Standardized error handling decorator
 
@@ -66,10 +72,10 @@ def __init__(self, **config):
                 # Initialize provider with configuration
                 pass
 
-            def get_thumbnail_url(self, isbn: str) -> str | None:
+            def get_thumbnail_url(self, isbn):
                 # Retrieve thumbnail URL for ISBN
                 # :param isbn: ISBN-10 or ISBN-13 (with or without hyphens)
-                # :returns: Thumbnail URL if found, None otherwise
+                # :returns: tuple (url_or_None, provider_name)
                 pass
 
 Usage Pattern::
@@ -82,8 +88,9 @@ def get_thumbnail_url(self, isbn: str) -> str | None:
     providers = [BnfProvider(), OpenLibraryProvider()]
 
     for provider in providers:
-        if url := provider.get_thumbnail_url(isbn):
-            print(f"Found: {url}")
+        url, provider_name = provider.get_thumbnail_url(isbn)
+        if url:
+            print(f"Found: {url} from {provider_name}")
             break
     else:
         print("No thumbnail found")
@@ -95,26 +102,15 @@ def get_thumbnail_url(self, isbn: str) -> str | None:
     - Catches and logs all other exceptions
     - Returns None on any error for graceful fallback
 
-Retry Logic:
-    HTTP requests use fetch_with_retries() with:
-    - Exponential backoff between retries
-    - Configurable retry attempts (default: 5)
-    - Disabled during testing for performance
-    - Handles transient network failures
-
 Image Validation:
     All providers validate images using validate_image_content():
     - Checks for non-empty content
     - Validates image format with PIL
-    - Verifies minimum dimensions (10x10px default)
+    - Verifies minimum dimensions (50x50px default)
     - Filters out placeholder images
 
 Configuration:
     Provider behavior can be configured via Flask configuration:
-    - RERO_INVENIO_THUMBNAILS_RETRY_ATTEMPTS
-    - RERO_INVENIO_THUMBNAILS_RETRY_BACKOFF_MULTIPLIER
-    - RERO_INVENIO_THUMBNAILS_RETRY_BACKOFF_MIN
-    - RERO_INVENIO_THUMBNAILS_RETRY_BACKOFF_MAX
     - RERO_INVENIO_THUMBNAILS_FILES_DIR (for files provider)
 
 Note: