server error when deleting a user #10467

FireBird4 · 2024-06-18T18:36:04Z

The bug

I tried to delete a user, but get the errors in the server logs every day at midnight.
The datebase also throws some errors.
The files of the user are deleted in the filesystem but not in the database.
The repair section in immich also shows a lot of files.

User Management site:

Repair site:

immich_server logs (with highlighting):

The OS that Immich Server is running on

Synology DiskStation Manager (DSM)

Version of Immich Server

v1.106.4

Version of Immich Mobile App

v1.106.3 build.143

Platform with the issue

Server
Web
Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    ports:
      - 2283:3001
    networks:
      - immich
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    environment:
      - MACHINE_LEARNING_MODEL_TTL=0
    volumes:
      - model-cache:/cache
    env_file:
      - stack.env
    networks:
      - immich
    restart: always

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:d6c2911ac51b289db208767581a5d154544f2b2fe4914ea5056443f62dc6e900
    healthcheck:
      test: redis-cli ping || exit 1
    volumes:
      - redisdata:/data
    networks:
      - immich
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    env_file:
      - stack.env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    volumes:
      - ${PGDATA_LOCATION}:/var/lib/postgresql/data
    networks:
      - immich
    restart: always

volumes:
  model-cache:
    driver: local
    driver_opts:
      device: ${MODEL_CACHE_LOCATION}
      o: bind
      type: local
  redisdata:
    driver: local
    driver_opts:
      device: ${REDIS_LOCATION}
      o: bind
      type: local

networks: 
  immich:
    name: immich
    ipam:
      driver: default
      config:
        - subnet: "172.20.0.0/16"

Your .env content

UPLOAD_LOCATION=/volume1/docker/Immich/library
IMMICH_VERSION=v1.106.4
TYPESENSE_API_KEY=****
DB_PASSWORD=****
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis
MODEL_CACHE_LOCATION=/volume1/docker/Immich/model_cache
PGDATA_LOCATION=/volume1/docker/Immich/pgdata
TSDATA_LOCATION=/volume1/docker/Immich/tsdata
REDIS_LOCATION=/volume1/docker/Immich/redis

Reproduction steps

1. Delete user on the User Management site
2. Wait until user would get deleted

Relevant log output

immich_server:
[Nest] 7  - 06/16/2024, 12:00:00 AM     LOG [Microservices:UserService] Deleting user: f37dd582-87d0-4783-8dfd-b6e0c043aff5
[Nest] 7  - 06/16/2024, 12:00:00 AM    WARN [Microservices:UserService] Removing user from filesystem: upload/library/f37dd582-87d0-4783-8dfd-b6e0c043aff5
[Nest] 7  - 06/16/2024, 12:00:00 AM    WARN [Microservices:UserService] Removing user from filesystem: upload/upload/f37dd582-87d0-4783-8dfd-b6e0c043aff5
[Nest] 7  - 06/16/2024, 12:00:00 AM    WARN [Microservices:UserService] Removing user from filesystem: upload/profile/f37dd582-87d0-4783-8dfd-b6e0c043aff5
[Nest] 7  - 06/16/2024, 12:00:00 AM    WARN [Microservices:UserService] Removing user from filesystem: upload/thumbs/f37dd582-87d0-4783-8dfd-b6e0c043aff5
[Nest] 7  - 06/16/2024, 12:00:00 AM    WARN [Microservices:UserService] Removing user from filesystem: upload/encoded-video/f37dd582-87d0-4783-8dfd-b6e0c043aff5
[Nest] 7  - 06/16/2024, 12:00:00 AM    WARN [Microservices:UserService] Removing user from database: f37dd582-87d0-4783-8dfd-b6e0c043aff5
[Nest] 7  - 06/16/2024, 12:00:01 AM   ERROR [Microservices:JobService] Unable to run job handler (backgroundTask/user-deletion): QueryFailedError: update or delete on table "assets" violates foreign key constraint "FK_91704e101438fd0653f582426dc" on table "asset_stack"
[Nest] 7  - 06/16/2024, 12:00:01 AM   ERROR [Microservices:JobService] QueryFailedError: update or delete on table "assets" violates foreign key constraint "FK_91704e101438fd0653f582426dc" on table "asset_stack"
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async DeleteQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/DeleteQueryBuilder.js:52:33)
    at async SubjectExecutor.executeRemoveOperations (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:461:17)
    at async SubjectExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:104:9)
    at async EntityPersistExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/EntityPersistExecutor.js:140:21)
    at async UserService.handleUserDelete (/usr/src/app/dist/services/user.service.js:158:9)
    at async /usr/src/app/dist/services/job.service.js:148:36
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 7  - 06/16/2024, 12:00:01 AM   ERROR [Microservices:JobService] Object:
{
  "id": "f37dd582-87d0-4783-8dfd-b6e0c043aff5"
}


immich_postgres:
2024-06-18 00:00:01.071 UTC [24923] ERROR:  update or delete on table "assets" violates foreign key constraint "FK_91704e101438fd0653f582426dc" on table "asset_stack"
2024-06-18 00:00:01.071 UTC [24923] DETAIL:  Key (id)=(5610a364-8eeb-465d-a41e-245504eedb31) is still referenced from table "asset_stack".
2024-06-18 00:00:01.071 UTC [24923] STATEMENT:  DELETE FROM "users" WHERE "id" = $1
2024-06-18 00:04:45.504 UTC [24949] FATAL:  role "root" does not exist
2024-06-18 00:09:46.150 UTC [24974] FATAL:  role "root" does not exist
2024-06-18 00:14:46.861 UTC [25000] FATAL:  role "root" does not exist
2024-06-18 00:19:47.788 UTC [25025] FATAL:  role "root" does not exist
2024-06-18 00:24:48.403 UTC [25051] FATAL:  role "root" does not exist
2024-06-18 00:29:49.195 UTC [25076] FATAL:  role "root" does not exist
...

Additional information

No response

Buco7854 · 2024-06-25T22:10:33Z

Exactly the same here, here are my logs but might not be more helpful. Same thing in the repair section.
For now it happens when I import some file using https://github.com/simulot/immich-go but it might be related to a broader issue.
Might also be related to #8207

[Nest] 7  - 06/25/2024, 10:01:40 PM     LOG [Microservices:UserService] Deleting user: af47f895-69c8-4893-9f8e-943aae60299b
[Nest] 7  - 06/25/2024, 10:01:40 PM    WARN [Microservices:UserService] Removing user from filesystem: upload/library/remi
[Nest] 7  - 06/25/2024, 10:01:40 PM    WARN [Microservices:UserService] Removing user from filesystem: upload/upload/af47f895-69c8-4893-9f8e-943aae60299b
[Nest] 17  - 06/25/2024, 10:01:43 PM     LOG [Api:EventRepository] Websocket Disconnect: Wtci2iBYqJc0lg5wAAAP
[Nest] 17  - 06/25/2024, 10:01:43 PM     LOG [Api:EventRepository] Websocket Connect:    PifiYcOm1URAciyyAAAR
[Nest] 7  - 06/25/2024, 10:01:44 PM    WARN [Microservices:UserService] Removing user from filesystem: upload/profile/af47f895-69c8-4893-9f8e-943aae60299b
[Nest] 7  - 06/25/2024, 10:01:44 PM    WARN [Microservices:UserService] Removing user from filesystem: upload/thumbs/af47f895-69c8-4893-9f8e-943aae60299b
[Nest] 7  - 06/25/2024, 10:01:48 PM    WARN [Microservices:UserService] Removing user from filesystem: upload/encoded-video/af47f895-69c8-4893-9f8e-943aae60299b
[Nest] 7  - 06/25/2024, 10:01:49 PM    WARN [Microservices:UserService] Removing user from database: af47f895-69c8-4893-9f8e-943aae60299b
[Nest] 7  - 06/25/2024, 10:01:49 PM   ERROR [Microservices:JobService] Unable to run job handler (backgroundTask/user-deletion): QueryFailedError: update or delete on table "assets" violates foreign key constraint "FK_91704e101438fd0653f582426dc" on table "asset_stack"
[Nest] 7  - 06/25/2024, 10:01:49 PM   ERROR [Microservices:JobService] QueryFailedError: update or delete on table "assets" violates foreign key constraint "FK_91704e101438fd0653f582426dc" on table "asset_stack"
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async DeleteQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/DeleteQueryBuilder.js:52:33)
    at async SubjectExecutor.executeRemoveOperations (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:461:17)
    at async SubjectExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:104:9)
    at async EntityPersistExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/EntityPersistExecutor.js:140:21)
    at async UserService.handleUserDelete (/usr/src/app/dist/services/user.service.js:158:9)
    at async /usr/src/app/dist/services/job.service.js:148:36
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 7  - 06/25/2024, 10:01:49 PM   ERROR [Microservices:JobService] Object:
{
  "id": "af47f895-69c8-4893-9f8e-943aae60299b",
  "force": true
}

jrasm91 · 2024-06-26T00:19:07Z

This is a bug with stacked assets which have a circular dependency that we aren't handling. Assets belong to a stack and a stack has a primary asset.

In the short term just unstack all the stacked assets for that user and then it should get deleted.

FireBird4 · 2024-06-26T05:26:17Z

I cannot login in with the deleted account to unstack the assets.
And I think I had no stacked assets in that account.

Buco7854 · 2024-06-26T10:15:09Z

I deleted them manually in the database, the table is asset_stack.

I tried to look for any stacked asset before starting the account deletion process but i could not see one in the web ui, It can be me that just didnt see them tho.

I'm just preventing immich go to stack image for now -create-stacks=FALSE. But that's kind of a shame.

Still immich is great thanks!

jrasm91 · 2024-06-26T11:39:58Z

This should be an easy bug to fix, so I would just continue using stacks. Well fix it in the next release or so.

Buco7854 · 2024-06-26T12:00:15Z

OK, thanks 😁

alextran1502 · 2024-06-27T19:33:05Z

Fixed in #10642

jrasm91 self-assigned this Jun 26, 2024

alextran1502 closed this as completed Jun 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

server error when deleting a user #10467

server error when deleting a user #10467

FireBird4 commented Jun 18, 2024 •

edited

Loading

Buco7854 commented Jun 25, 2024 •

edited

Loading

jrasm91 commented Jun 26, 2024

FireBird4 commented Jun 26, 2024 •

edited

Loading

Buco7854 commented Jun 26, 2024 •

edited

Loading

jrasm91 commented Jun 26, 2024

Buco7854 commented Jun 26, 2024

alextran1502 commented Jun 27, 2024

server error when deleting a user #10467

server error when deleting a user #10467

Comments

FireBird4 commented Jun 18, 2024 • edited Loading

The bug

The OS that Immich Server is running on

Version of Immich Server

Version of Immich Mobile App

Platform with the issue

Your docker-compose.yml content

Your .env content

Reproduction steps

Relevant log output

Additional information

Buco7854 commented Jun 25, 2024 • edited Loading

jrasm91 commented Jun 26, 2024

FireBird4 commented Jun 26, 2024 • edited Loading

Buco7854 commented Jun 26, 2024 • edited Loading

jrasm91 commented Jun 26, 2024

Buco7854 commented Jun 26, 2024

alextran1502 commented Jun 27, 2024

FireBird4 commented Jun 18, 2024 •

edited

Loading

Buco7854 commented Jun 25, 2024 •

edited

Loading

FireBird4 commented Jun 26, 2024 •

edited

Loading

Buco7854 commented Jun 26, 2024 •

edited

Loading