Unable to start immich after upgrade (v1.102.3 -> v1.107.2) and funky/bad pgvector state due to face_search not yet being created

[lgo]

The bug

I just got around to upgrading things and immich didn't want to come up right away. I didn't keep a close track of what I did which I'm regretting now as I might have broke another thing in the process that made this a bit more complicated[0]. This was my crash log:

Detected CPU Cores: 40
Starting api worker
Starting microservices worker
[Nest] 7  - 07/05/2024, 4:02:58 AM     LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 17  - 07/05/2024, 4:02:58 AM     LOG [Api:EventRepository] Initialized websocket server
[Nest] 7  - 07/05/2024, 4:03:01 AM    WARN [Microservices:DatabaseRepository] Could not reindex index face_index. Attempting to auto-fix.
[Nest] 7  - 07/05/2024, 4:03:02 AM    WARN [Microservices:DatabaseService] Could not run vector reindexing checks. If the extension was updated, please restart the Postgres instance.
TypeError: Cannot read properties of undefined (reading 'dimsize')
    at DatabaseRepository.getDimSize (/usr/src/app/dist/repositories/database.repository.js:162:31)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async DatabaseRepository.reindex (/usr/src/app/dist/repositories/database.repository.js:96:33)
    at async /usr/src/app/dist/services/database.service.js:126:21
    at async /usr/src/app/dist/repositories/database.repository.js:177:23
microservices worker exited with code 1

Looking at the code, there's this re-indexing step (potentially due to my broken pgvector state / weirdness).

immich/server/src/repositories/database.repository.ts

Lines 87 to 88 in e32ce82

	await this.reindex(VectorIndex.CLIP);
	await this.reindex(VectorIndex.FACE);

but the REINDEX INDEX is failing, hitting the auto-fix, then failing and crashing on getDimSize.

immich/server/src/repositories/database.repository.ts

Line 102 in 6030349

const dimSize = await this.getDimSize(table);

importantly, I didn't have the face_search table which was added in v1.107.0 so this pre-migration step is always going to fail on getDimSize.

I was able to upgrade to v1.106.4, have the auto-fix complete, and then successfully upgrade past v1.107.0.

---

[0]: I thought my database was probably in an OK state for pgvector as it was already upgraded to 0.2.0 (specifically tensorchord/pgvecto-rs:pg16-v0.2.0).

I was getting some errors where I mindlessly ran some upgrade Postgres commands related to the 0.1.11 to 0.2.0 upgrade (manually set the version, attempt upgrade, etc.). I ended up running a few more things to get immich to make a bit more progress on boot:

-- in immich database
UPDATE pg_catalog.pg_extension SET extversion = '0.2.0' WHERE extname = 'vectors';
-- in `postgres` database. this was failing for a bit
GRANT SELECT ON TABLE pg_vector_index_stat to <immuch_user>

Everything else in the report is after having broken pgvector then unwedge it with that.

The OS that Immich Server is running on

Debian

Version of Immich Server

v1.107.2

Version of Immich Mobile App

n/a

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

services:
  {{ service_name }}:
    container_name: {{ service_name }}
    image: ghcr.io/immich-app/immich-server:{{ service_version }}
    volumes:
      - {{ document_root }}/immich:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    environment:
      IMMICH_VERSION: {{ service_version }}
      DB_HOSTNAME: ...
      REDIS_HOSTNAME: ...
      DB_USERNAME: ...
      DB_PASSWORD: ...
      DB_DATABASE_NAME: ...
    networks:
      - postgres_postgresnet
      - redis_redisnet
    restart: unless-stopped

networks:
  traefik_network:
    external: true
  postgres_postgresnet:
    external: true
  redis_redisnet:
    external: true

Your .env content

n/a

Reproduction steps

n/a

Relevant log output

No response

Additional information

No response

[lgo]

hmm, unfortunately after getting this unwedged with doing the upgrade in steps I realized I had no information on the REINDEX error which wasn't logged and likely can't reproduce. I still don't quite know what the first error was that caused me to go down this rabbit hole, but I suspect it was the crash as reported or the lack of permissions on pg_vector_index_stat.

[mmomjian]

Seems like it’s working now, and we already have a fix for some of this. Thanks.