People's thumbnails don't get regenerated from the job that is supposed to regenerate them

[NyaomiDEV]

The bug

This is a lie.

The OS that Immich Server is running on

Arch

Version of Immich Server

v1.102.3

Version of Immich Mobile App

Not applicable

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

version: "3.8"
services:
  immich_server:
    image: ghcr.io/immich-app/immich-server:release
    command:
      - start.sh
      - immich
    volumes:
      - /data/immich/photos:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    environment:
      DB_HOSTNAME: postgres
      DB_USERNAME: postgres
      DB_PASSWORD: postgres
      DB_DATABASE_NAME: immich
      REDIS_HOSTNAME: redis
    depends_on:
      - redis
      - postgres
    networks:
      - default
      - nginx_default
    restart: always
  immich_microservices:
    image: ghcr.io/immich-app/immich-server:release
    command:
      - start.sh
      - microservices
    volumes:
      - /data/immich/photos:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    environment:
      DB_HOSTNAME: postgres
      DB_USERNAME: postgres
      DB_PASSWORD: postgres
      DB_DATABASE_NAME: immich
      REDIS_HOSTNAME: redis
    depends_on:
      - redis
      - postgres
    networks:
      - default
    restart: always
  immich_machine_learning:
    image: ghcr.io/immich-app/immich-machine-learning:release
    volumes:
      - model-cache:/cache
    environment:
      DB_HOSTNAME: postgres
      DB_USERNAME: postgres
      DB_PASSWORD: postgres
      DB_DATABASE_NAME: immich
      REDIS_HOSTNAME: redis
    networks:
      - default
    restart: always
  redis:
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:51d6c56749a4243096327e3fb964a48ed92254357108449cb6e23999c37773c5
    networks:
      - default
    restart: always
  postgres:
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: postgres
      POSTGRES_USER: postgres
      POSTGRES_DB: immich
    volumes:
      - /data/immich/db:/var/lib/postgresql/data
    networks:
      - default
    restart: always
volumes:
  model-cache: null
networks:
  default: null
  nginx_default:
    external: true

Your .env content

nothing here

Reproduction steps

1. Delete some faces from the thumbnail cache
2. Try to regenerate them

Relevant log output

No response

Additional information

No response

[alextran1502]

I think we moved this to the face-detection job, correct? @mertalev

[mertalev]

The facial recognition job just queues person thumbnail generation. The thumbnail generation itself is the same as always and queued the same way here when clicking All or Missing. All person thumbnails are generated in the thumbnail generation queue.

@NyaomiDEV Can you elaborate on why you think it isn't regenerating person thumbnails? Just deleting the files won't re-generate them when running Missing, if this is what you tested. That queues assets based on data in the database, not by checking each asset. All would re-generate everything, so that should indeed re-generate a deleted thumbnail.

[NyaomiDEV]

@mertalev In my testing I pressed "all" and it didn't regenerate any of them. Since I deleted the old thumbnails manually, if there was at least one thumbnail to be generated I would've catched that, but there was nothing at all. In the night time (EU here) I scheduled a facial recognition job to redo everything from scratch, though I should still check the results.

---

Okay so I checked and re-running facial detection and recognition for all files actually mitigates this issue; point is I was lucky to never have used the people feature so I didn't assign any names or merged anyone yet. I can't imagine the stress of doing this when you've already tracked that sort of stuff.

[NyaomiDEV]

OH! Yeah... people's thumbnails are still a Jpeg.

[alextran1502]

I don't think we mention anywhere that facial thumbnails can be generated with different extensions. In the release note, it was about the preview (small) and the thumbnail (large) for an asset

[NyaomiDEV]

I don't think we mention anywhere that facial thumbnails can be generated with different extensions. In the release note, it was about the preview (small) and the thumbnail (large) for an asset

True that, however this means that I shouldn't have deleted all the Jpegs... that's my fault. However, I'd love to ask for this feature for the sake of consolidation.

This still doesn't mean that the bug isn't there, as they didn't get regenerated when I ran the thumbnails job.

[mertalev]

All not regenerating the thumbnails is weird. A few things come to mind:

1. We don't handle the preview size being different from when the faces were detected (see handleGeneratePersonThumbnail incorrectly crops if thumbnail is regenerated with a different size #8317). If you changed it then at least some of the jobs would fail because of invalid bounding boxes.

2. Some of the person thumbnails might have been based on the motion part of a live photo. Person thumbnails are cropped from preview images. Since you deleted the existing previews and thumbnail generation now skips these assets, jobs to generate person thumbnails will fail in this case since they're trying to crop images that don't exist.

Could you check the logs in the microservices container to see if there are any errors during thumbnail generation?

Also yes, the format change should ideally apply to person thumbnails as well. It can just use the thumbnail setting for this.

[NyaomiDEV]

Could you check the logs in the microservices container to see if there are any errors during thumbnail generation?

Sadly I had to bring down and up the server again, and in doing so I missed a bunch of logs, as I am not keeping them on disk.

The only thing I remember about the microservices container is that it sometimes timed out, so I brought down the number of concurrent tasks, thinking Postgres couldn't probably handle that traffic. I rebuilt the thumbnail cache since, and still, no thumbnails; also, if that was the case I would get at least some thumbnails, whereas all my other ones would've been spotty.

Of course, I could just re-delete all the Jpegs again and try it out yet again, but I'd kinda want to not reschedule another face recognition task later tonight.

[mertalev]

That's fine! I'll see if I can reproduce this. If the person thumbnails do get regenerated under normal circumstances, then it means the jobs are failing in your case.

[Chaphasilor]

Just wanted to chime in saying that I have this very same issue for a few weeks now, I just didn't get around to creating an issue.

Had to delete thumbnails when my SSD filled up and services stopped working, and ever since all person thumnails are gone. Thumbnails can be "restored" by selecting a new featured photo, but that would mean doing that for hundreds of persons.

Since I deleted all thumbnails, I also definitely ran the "All" job and not just "Missing".

Sadly I've made heavy use of the persons-feature, and a lot of work would be lost by re-running face recognition for all assets.

Let me know if I can help in any way!

[mertalev]

I tested and the person thumbnails really are queued and generated when running thumbnail generation on all assets. Setting the logging to verbose and running All shows Cropping face for person log entries after some time. I think the manual deletion of images is probably the source of the issues here: we don't expect these files to be touched by anything but the server.

[NyaomiDEV]

I think the manual deletion of images is probably the source of the issues here: we don't expect these files to be touched by anything but the server.

Yeah, but also they show up as missing, so why is the server not regenerating them? The thumbnail cache should be safe to delete, for all the reasons imaginable. It's not actually important data.

As it is, closing the issue without knowing why specifically this happens is a bit counterproductive. If at least we know exactly why that happens, someone could make a MR working around or fixing the main issue.

[Chaphasilor]

If you say thumbnail generation is working correctly and the database is simply in a bad state after the external deletion, maybe there should be a way to repair the missing assets somehow? So that afterwards the thumbnails can be re-generated normally :)

[mertalev]

why is the server not regenerating them

The jobs are most likely failing for the reasons above, which we're already aware of. Sharing the full verbose logs after running All would be helpful to see if the issue is distinct from those.

[Chaphasilor]

I'd like to avoid doing a full rescan until #8760 is fixed, otherwise I might end up with a full SSD again. But once that is the case I'd be happy to provide logs!

[mertalev]

That should be fixed via #9126, but you're right that it hasn't made it into a release yet.

[nathan-lamy]

Not fixed in v1.103.1, I regenerated ALL thumbnails and still doesn't have people faces showing up :(

[nathan-lamy]

After digging through the log, there's only one message that could correspond to this issue :

[JobService] Unable to run job handler (thumbnailGeneration/queue-generate-thumbnails): QueryFailedError: insert or update on table "person" violates foreign key constraint "FK_2bbabe31656b6778c6b87b61023"
[JobService] QueryFailedError: insert or update on table "person" violates foreign key constraint "FK_2bbabe31656b6778c6b87b61023"

Should I open a new issue?

[mertalev]

That's an interesting log. Encountering that error would prevent any later people from getting queued for thumbnail generation.

[mertalev]

Using the ghcr.io/immich-app/immich-server:pr-9195 image for immich-microservices and re-running should fix that issue. Would you be able to try it?

[nathan-lamy]

That's an interesting log. Encountering that error would prevent any later people from getting queued for thumbnail generation.

Exactly! I've seen this error only ONE time in the log but ~ hundred people don't have a thumbnail...

Using the ghcr.io/immich-app/immich-server:pr-9195 image for immich-microservices and re-running should fix that issue. Would you be able to try it?

Yes I'll try, thank you! Do I have to regenerate all thumbnails or only missing ones would do it?

[mertalev]

Yes I'll try, thank you! Do I have to regenerate all thumbnails or only missing ones would do it?

I think it'd have to be all thumbnails for the person thumbnails to get regenerated.

[Chaphasilor]

Just wanted to confirm that after finally having enough disk space to regenerate thumbnails again, once the job completed all person thumbnails were restored! That was on 1.105.1.

Thanks for the fix!