Infinite "FixThumbnailsWithoutMetadata" scan with "Fixed 0 ThumbnailBook" #1338

noiret · 2023-12-09T14:49:08Z

Steps to reproduce

Komga directly run the task "FixThumbnailsWithoutMetadata" and the problem start.
The problem seems to occur when upgrading the docker image from 1.8.3 to 1.8.4 (latest).
Rollback 1.8.4 to 1.8.3 and 1.7.2 without improvement.

Expected behavior

Komga start and task "FixThumbnailsWithoutMetadata" completed successfully.

Actual behavior

The scan "FixThumbnailsWithoutMetadata" never end and fix no "ThumbnailBook" (=> produce high CPU usage).

Logs

Logs : komga-manual-1.log

Komga version

1.8.4, 1.8.3 and 1.7.2

Operating system

Debian 12

Installation method

Docker

Other details

Environement variables :

- name: LOGGING_LEVEL_ORG_GOTSON_KOMGA
  value: TRACE
- name: JAVA_TOOL_OPTIONS
  value: >-
    -Xmx8g -XX:+UseContainerSupport -XX:MaxRAMPercentage=70.0
    -XX:InitialRAMPercentage=40.0 -XX:MinRAMPercentage=40.0

Deployed on kubernetes (k3s).

Library "scan on boot" and "periodic scan" are disabled with no improvement.
Seem not related with #1287 issue.

Acknowledgements

I have searched the existing issues (open AND closed) and this is a new ticket, NOT a duplicate or related to another open issue.
I have written a short but informative title.
I have checked the FAQ.
I have updated the app to the latest version.
I will fill out all of the requested information in this form.

The text was updated successfully, but these errors were encountered:

gotson · 2023-12-11T02:13:56Z

The logs are from 1.7.2. Please run the latest version and provide log file.

noiret · 2023-12-11T09:01:02Z

Hello, logs with 1.8.4 (latest) komga-1-8-4.log

gotson · 2023-12-11T09:13:02Z

thanks.

Difficult to see what's going on, i would need to have more information on what's in your DB.

Can you execute the following query in the DB via sqlite3 or any DB explorer and provide the results ?

select BOOK_ID,
       length(THUMBNAIL) as thumbnail_size,
       url,
       type,
       WIDTH,
       HEIGHT,
       MEDIA_TYPE,
       FILE_SIZE
from THUMBNAIL_BOOK
where FILE_SIZE = 0
   or MEDIA_TYPE = ''
   or WIDTH = 0
   or HEIGHT = 0
;

noiret · 2023-12-11T09:29:23Z

The CSV export from database.sqlite (semicolon separated) : Query-database-komga.csv

gotson · 2023-12-11T12:16:33Z

All the thumbnails listed are generated, but they don't have any data.

Can you try to request the thumbnail for one of those books, see what's the api returns?

noiret · 2023-12-11T12:49:02Z

With the first id in the list i get a 404 (using GET api/v1/books/{bookid}/thumbnail route) :

 cache-control: max-age=0,must-revalidate,no-transform,private 
 content-length: 0 
 date: Mon,11 Dec 2023 12:36:16 GMT 
 strict-transport-security: max-age=31536000 ; includeSubDomains 
 vary: Origin,Access-Control-Request-Method,Access-Control-Request-Headers 
 x-content-type-options: nosniff 
 x-frame-options: SAMEORIGIN 
 x-xss-protection: 0

(HTTP 200 with book id not listed in the previous query)

gotson · 2023-12-11T14:28:04Z

Those books probably need to be analyzed again. Are they all part of the same library?

noiret · 2023-12-11T16:11:17Z

Yes, I only have one large library.
I have scheduled analysis on it.

noiret · 2023-12-12T09:29:48Z

The analysis is complete, but the problem seems to persist

2023-12-12T09:27:17.867Z  INFO 1 --- [taskProcessor-2] o.g.komga.application.tasks.TaskHandler  : Task FixThumbnailsWithoutMetadata(priority='0') executed in 243.737688ms
2023-12-12T09:27:17.871Z  INFO 1 --- [taskProcessor-4] o.g.komga.application.tasks.TaskHandler  : Executing task: FixThumbnailsWithoutMetadata(priority='0')
2023-12-12T09:27:18.119Z  INFO 1 --- [taskProcessor-4] o.g.k.domain.service.ThumbnailLifecycle  : Fetched 1000 ThumbnailBook to fix, total: 3405
2023-12-12T09:27:18.119Z  INFO 1 --- [taskProcessor-4] o.g.k.domain.service.ThumbnailLifecycle  : Fixed 0 ThumbnailBook in 247.927913ms
2023-12-12T09:27:18.119Z  INFO 1 --- [taskProcessor-4] o.g.komga.application.tasks.TaskEmitter  : Sending task: FixThumbnailsWithoutMetadata(priority='0')
2023-12-12T09:27:18.122Z  INFO 1 --- [taskProcessor-4] o.g.komga.application.tasks.TaskHandler  : Task FixThumbnailsWithoutMetadata(priority='0') executed in 250.661296ms
2023-12-12T09:27:18.125Z  INFO 1 --- [taskProcessor-1] o.g.komga.application.tasks.TaskHandler  : Executing task: FixThumbnailsWithoutMetadata(priority='0')
2023-12-12T09:27:18.436Z  INFO 1 --- [taskProcessor-1] o.g.k.domain.service.ThumbnailLifecycle  : Fetched 1000 ThumbnailBook to fix, total: 3405
2023-12-12T09:27:18.436Z  INFO 1 --- [taskProcessor-1] o.g.k.domain.service.ThumbnailLifecycle  : Fixed 0 ThumbnailBook in 310.778248ms
2023-12-12T09:27:18.436Z  INFO 1 --- [taskProcessor-1] o.g.komga.application.tasks.TaskEmitter  : Sending task: FixThumbnailsWithoutMetadata(priority='0')
2023-12-12T09:27:18.439Z  INFO 1 --- [taskProcessor-1] o.g.komga.application.tasks.TaskHandler  : Task FixThumbnailsWithoutMetadata(priority='0') executed in 313.307328ms
2023-12-12T09:27:18.442Z  INFO 1 --- [taskProcessor-6] o.g.komga.application.tasks.TaskHandler  : Executing task: FixThumbnailsWithoutMetadata(priority='0')
2023-12-12T09:27:18.688Z  INFO 1 --- [taskProcessor-6] o.g.k.domain.service.ThumbnailLifecycle  : Fetched 1000 ThumbnailBook to fix, total: 3405
2023-12-12T09:27:18.688Z  INFO 1 --- [taskProcessor-6] o.g.k.domain.service.ThumbnailLifecycle  : Fixed 0 ThumbnailBook in 246.405242ms
2023-12-12T09:27:18.689Z  INFO 1 --- [taskProcessor-6] o.g.komga.application.tasks.TaskEmitter  : Sending task: FixThumbnailsWithoutMetadata(priority='0')
2023-12-12T09:27:18.691Z  INFO 1 --- [taskProcessor-6] o.g.komga.application.tasks.TaskHandler  : Task FixThumbnailsWithoutMetadata(priority='0') executed in 248.853406ms
2023-12-12T09:27:18.694Z  INFO 1 --- [taskProcessor-5] o.g.komga.application.tasks.TaskHandler  : Executing task: FixThumbnailsWithoutMetadata(priority='0')
2023-12-12T09:27:18.938Z  INFO 1 --- [taskProcessor-5] o.g.k.domain.service.ThumbnailLifecycle  : Fetched 1000 ThumbnailBook to fix, total: 3405
2023-12-12T09:27:18.938Z  INFO 1 --- [taskProcessor-5] o.g.k.domain.service.ThumbnailLifecycle  : Fixed 0 ThumbnailBook in 244.136625ms
2023-12-12T09:27:18.939Z  INFO 1 --- [taskProcessor-5] o.g.komga.application.tasks.TaskEmitter  : Sending task: FixThumbnailsWithoutMetadata(priority='0')
2023-12-12T09:27:18.941Z  INFO 1 --- [taskProcessor-5] o.g.komga.application.tasks.TaskHandler  : Task FixThumbnailsWithoutMetadata(priority='0') executed in 246.762927ms
2023-12-12T09:27:18.945Z  INFO 1 --- [taskProcessor-3] o.g.komga.application.tasks.TaskHandler  : Executing task: FixThumbnailsWithoutMetadata(priority='0')
2023-12-12T09:27:19.198Z  INFO 1 --- [taskProcessor-3] o.g.k.domain.service.ThumbnailLifecycle  : Fetched 1000 ThumbnailBook to fix, total: 3405
2023-12-12T09:27:19.198Z  INFO 1 --- [taskProcessor-3] o.g.k.domain.service.ThumbnailLifecycle  : Fixed 0 ThumbnailBook in 252.760988ms
2023-12-12T09:27:19.198Z  INFO 1 --- [taskProcessor-3] o.g.komga.application.tasks.TaskEmitter  : Sending task: FixThumbnailsWithoutMetadata(priority='0')
2023-12-12T09:27:19.200Z  INFO 1 --- [taskProcessor-3] o.g.komga.application.tasks.TaskHandler  : Task FixThumbnailsWithoutMetadata(priority='0') executed in 255.517088ms
2023-12-12T09:27:19.204Z  INFO 1 --- [taskProcessor-4] o.g.komga.application.tasks.TaskHandler  : Executing task: FixThumbnailsWithoutMetadata(priority='0')
< ... >

gotson · 2023-12-12T09:49:47Z

Can you locate one of the book for which it does not work, analyze it, and see if there's anything in the logs ?

noiret · 2023-12-12T10:02:20Z

Yes, here is the log : analyse-book-komga.log

gotson · 2023-12-12T10:07:48Z

I suspect the books do not have a cover at all. Could you send me one of those books ?

noiret · 2023-12-12T10:14:29Z

You can donwload the book here : https://wormhole.app/0lpap#rQdfYhJxj1ZwOB-u3RpScw

noiret · 2023-12-12T10:19:03Z

For information, i also have ZIP archives in the exported list

gotson · 2023-12-12T10:23:33Z

For information, i also have ZIP archives in the exported list

I'm not sure to understand that statement.

noiret · 2023-12-12T10:34:49Z

For example, the book with the ID "0CDYKE6554E8P" in the list ( #1338 (comment)) is not an EPUB file but an archive : https://wormhole.app/5zR8R#ayTN1Hw3dbBRm7eOhS8TFQ

noiret · 2023-12-12T11:10:30Z

For me, the problem doesn't seem to be linked to these files (these files have been there for years without issue).
What's new is that Komga tries to generate thumbnails on these files in a loop and produce high CPU usage.

gotson · 2023-12-12T11:10:49Z

For example, the book with the ID "0CDYKE6554E8P" in the list ( #1338 (comment)) is not an EPUB file but an archive : https://wormhole.app/5zR8R#ayTN1Hw3dbBRm7eOhS8TFQ

That file has no images in it, thus should show as in error. Can you confirm?

noiret · 2023-12-12T11:17:55Z

That file has no images in it, thus should show as in error. Can you confirm?

Just to be clear, the archive does not contain an image at the root, but a PDF with images.

noiret · 2023-12-12T11:25:39Z

And no, the file is not listed in media analysis page

gotson · 2023-12-12T11:31:08Z

That file has no images in it, thus should show as in error. Can you confirm?

Just to be clear, the archive does not contain an image at the root, but a PDF with images.

Pdf inside archives are not supported.

And no, the file is not listed in media analysis page

Can you check the book media status in the API for that book?

gotson · 2023-12-12T11:52:26Z

Actually for that book if you could provide the json from /api/v1/books/{bookId} that would help

noiret · 2023-12-12T11:59:05Z

Actually for that book if you could provide the json from /api/v1/books/{bookId} that would help

{
  "id": "0CDYKE6554E8P",
  "seriesId": "0CDYKE6554E8N",
  "seriesTitle": "ARTeam",
  "libraryId": "0CDYKDSJN4BKY",
  "name": "ARTeam_eZine_Number1",
  "url": "/mnt/livres/Zines/ARTeam/ARTeam_eZine_Number1.rar",
  "number": 3,
  "created": "2023-05-21T09:47:38Z",
  "lastModified": "2023-12-11T16:18:51Z",
  "fileLastModified": "2020-12-23T16:43:16Z",
  "sizeBytes": 3129556,
  "size": "3 MiB",
  "media": {
    "status": "READY",
    "mediaType": "application/x-rar-compressed; version=4",
    "pagesCount": 1,
    "comment": "",
    "mediaProfile": "DIVINA"
  },
  "metadata": {
    "title": "ARTeam_eZine_Number1",
    "titleLock": false,
    "summary": "",
    "summaryLock": false,
    "number": "3",
    "numberLock": false,
    "numberSort": 3,
    "numberSortLock": false,
    "releaseDate": null,
    "releaseDateLock": false,
    "authors": [],
    "authorsLock": false,
    "tags": [],
    "tagsLock": false,
    "isbn": "",
    "isbnLock": false,
    "links": [],
    "linksLock": false,
    "created": "2023-05-21T09:47:38Z",
    "lastModified": "2023-12-11T16:18:51Z"
  },
  "readProgress": null,
  "deleted": false,
  "fileHash": "9c94c119e39f67bb262e97b50fc08160",
  "oneshot": false
}

gotson · 2023-12-12T12:30:22Z

How about /api/v1/books/{bookId}/pages?

noiret · 2023-12-12T12:56:25Z

How about /api/v1/books/{bookId}/pages?

[
  {
    "number": 1,
    "fileName": "ARTeam.eZine.Number1\\Supplements\\Breaking Protocol\\Src\\Bruteus\\Bruteus\\icon.ico",
    "mediaType": "image/vnd.microsoft.icon",
    "width": null,
    "height": null,
    "sizeBytes": 1150,
    "size": "1.1 KiB"
  }
]

gotson · 2023-12-12T14:29:10Z

Oh that's funny. We'll you probably shouldn't have that kind of files in your komga library.

I'll have a look at how I can fix your issue with the never ending tasks.

gotson · 2023-12-13T09:09:34Z

I've been looking at this, turns out an empty thumnail is created and saved when no poster can be found. This should not happen and i'll fix this.

I will also do a database cleanup in the next release to remove those empty thumbnails, which should prevent the fixThumbnails task to loop through them.

noiret · 2023-12-13T10:39:59Z

Thanks for your support 👍

github-actions · 2023-12-15T03:37:26Z

🎉 This issue has been resolved in 1.9.0 (Release Notes)

noiret added the triage label Dec 9, 2023

gotson added troubleshooting and removed triage labels Dec 11, 2023

gotson closed this as completed in 15920b7 Dec 13, 2023

github-actions bot added the released label Dec 15, 2023

github-actions bot locked as resolved and limited conversation to collaborators Jan 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Infinite "FixThumbnailsWithoutMetadata" scan with "Fixed 0 ThumbnailBook" #1338

Infinite "FixThumbnailsWithoutMetadata" scan with "Fixed 0 ThumbnailBook" #1338

noiret commented Dec 9, 2023 •

edited

Loading

gotson commented Dec 11, 2023

noiret commented Dec 11, 2023

gotson commented Dec 11, 2023

noiret commented Dec 11, 2023

gotson commented Dec 11, 2023

noiret commented Dec 11, 2023

gotson commented Dec 11, 2023

noiret commented Dec 11, 2023

noiret commented Dec 12, 2023

gotson commented Dec 12, 2023

noiret commented Dec 12, 2023

gotson commented Dec 12, 2023

noiret commented Dec 12, 2023

noiret commented Dec 12, 2023

gotson commented Dec 12, 2023

noiret commented Dec 12, 2023

noiret commented Dec 12, 2023 •

edited

Loading

gotson commented Dec 12, 2023

noiret commented Dec 12, 2023

noiret commented Dec 12, 2023

gotson commented Dec 12, 2023

gotson commented Dec 12, 2023

noiret commented Dec 12, 2023

gotson commented Dec 12, 2023

noiret commented Dec 12, 2023

gotson commented Dec 12, 2023

gotson commented Dec 13, 2023

noiret commented Dec 13, 2023

github-actions bot commented Dec 15, 2023

Infinite "FixThumbnailsWithoutMetadata" scan with "Fixed 0 ThumbnailBook" #1338

Infinite "FixThumbnailsWithoutMetadata" scan with "Fixed 0 ThumbnailBook" #1338

Comments

noiret commented Dec 9, 2023 • edited Loading

Steps to reproduce

Expected behavior

Actual behavior

Logs

Komga version

Operating system

Installation method

Other details

Acknowledgements

gotson commented Dec 11, 2023

noiret commented Dec 11, 2023

gotson commented Dec 11, 2023

noiret commented Dec 11, 2023

gotson commented Dec 11, 2023

noiret commented Dec 11, 2023

gotson commented Dec 11, 2023

noiret commented Dec 11, 2023

noiret commented Dec 12, 2023

gotson commented Dec 12, 2023

noiret commented Dec 12, 2023

gotson commented Dec 12, 2023

noiret commented Dec 12, 2023

noiret commented Dec 12, 2023

gotson commented Dec 12, 2023

noiret commented Dec 12, 2023

noiret commented Dec 12, 2023 • edited Loading

gotson commented Dec 12, 2023

noiret commented Dec 12, 2023

noiret commented Dec 12, 2023

gotson commented Dec 12, 2023

gotson commented Dec 12, 2023

noiret commented Dec 12, 2023

gotson commented Dec 12, 2023

noiret commented Dec 12, 2023

gotson commented Dec 12, 2023

gotson commented Dec 13, 2023

noiret commented Dec 13, 2023

github-actions bot commented Dec 15, 2023

noiret commented Dec 9, 2023 •

edited

Loading

noiret commented Dec 12, 2023 •

edited

Loading