feat(server): improve checkAlbumAccess query performance

[skatsubo]

Description

Fix slow SQL query in checkAlbumAccess() caused by the array overlap operator array && array:

• '&&' cannot use indexes, hence PG planner cannot use this condition effectively on asset. So it resorts to SeqScan on album_asset and/or asset (thus more records -> slower the query).
• '&&' is a little bit more CPU hungry vs simple comparison, although this difference manifests only for large number of IDs in the checked array, so it's of low importance.

Fixes #22389.
In essence, with partner sharing enabled, partner's photos take longer to load (in timeline or direct view) if I have a lot of assets in my albums. (Somewhat counterintuitive at first).

Original auto_explain from #22389 (49K rows in album_asset)

2025-09-26 06:45:49.354 UTC [57] LOG:  duration: 1740.506 ms  plan:
	Query Text: select "asset"."id", "asset"."livePhotoVideoId" from "album" inner join "album_asset" as "albumAssets" on "album"."id" = "albumAssets"."albumsId" inner join "asset" on "asset"."id" = "albumAssets"."assetsId" and "asset"."deletedAt" is null left join "album_user" as "albumUsers" on "albumUsers"."albumsId" = "album"."id" left join "user" on "user"."id" = "albumUsers"."usersId" and "user"."deletedAt" is null where array["asset"."id", "asset"."livePhotoVideoId"] && array[$1]::uuid[]  and ("album"."ownerId" = $2 or "user"."id" = $3) and "album"."deletedAt" is null
	Nested Loop  (cost=18.48..2141.31 rows=1 width=32)
	  ->  Hash Left Join  (cost=18.05..1242.69 rows=198 width=16)
	        Hash Cond: ("albumUsers"."usersId" = "user".id)
	        Filter: ((album."ownerId" = 'cb10fabd-5fce-497e-ab95-f29b6af0d0bb'::uuid) OR ("user".id = 'cb10fabd-5fce-497e-ab95-f29b6af0d0bb'::uuid))
	        ->  Nested Loop Left Join  (cost=16.01..1239.58 rows=395 width=48)
	              ->  Hash Join  (cost=13.83..1198.86 rows=259 width=48)
	                    Hash Cond: ("albumAssets"."albumsId" = album.id)
	                    ->  Seq Scan on album_asset "albumAssets"  (cost=0.00..1054.35 rows=49235 width=32)
	                    ->  Hash  (cost=13.80..13.80 rows=2 width=32)
	                          ->  Seq Scan on album  (cost=0.00..13.80 rows=2 width=32)
	                                Filter: ("deletedAt" IS NULL)
	              ->  Memoize  (cost=2.18..9.29 rows=3 width=32)
	                    Cache Key: album.id
	                    Cache Mode: logical
	                    ->  Bitmap Heap Scan on album_user "albumUsers"  (cost=2.17..9.28 rows=3 width=32)
	                          Recheck Cond: ("albumsId" = album.id)
	                          ->  Bitmap Index Scan on "album_user_albumsId_idx"  (cost=0.00..2.17 rows=3 width=0)
	                                Index Cond: ("albumsId" = album.id)
	        ->  Hash  (cost=2.02..2.02 rows=2 width=16)
	              ->  Seq Scan on "user"  (cost=0.00..2.02 rows=2 width=16)
	                    Filter: ("deletedAt" IS NULL)
	  ->  Index Scan using asset_pkey on asset  (cost=0.42..4.54 rows=1 width=32)
	        Index Cond: (id = "albumAssets"."assetsId")
	        Filter: (("deletedAt" IS NULL) AND (ARRAY[id, "livePhotoVideoId"] && '{50ffba07-2b85-449e-9a9b-3ba1c02482ee}'::uuid[]))

How Has This Been Tested?

Test on my Immich instance

Tested the fix on my local Immich instance: shared partner assets are loading faster.

EXPLAIN with the fix

explain (analyze, verbose, buffers, settings)
select "asset"."id", "asset"."livePhotoVideoId" from "album" 
inner join "album_asset" as "albumAssets" on "album"."id" = "albumAssets"."albumsId" 
inner join "asset" on "asset"."id" = "albumAssets"."assetsId" and "asset"."deletedAt" is null 
left join "album_user" as "albumUsers" on "albumUsers"."albumsId" = "album"."id" 
left join "user" on "user"."id" = "albumUsers"."usersId" and "user"."deletedAt" is null 
where 
      ("asset"."id" in ('d011dd26-f03e-4fe1-8cc4-9244d81a308d') or "asset"."livePhotoVideoId" in ('d011dd26-f03e-4fe1-8cc4-9244d81a308d')) 
  and ("album"."ownerId" = 'dc14ff4e-654b-4edb-9f6c-3a6a0f59f3f1' or "user"."id" = 'dc14ff4e-654b-4edb-9f6c-3a6a0f59f3f1') 
  and "album"."deletedAt" is null;
                                                                                QUERY PLAN                                                                                 
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Nested Loop Left Join  (cost=13.29..29.90 rows=580 width=32) (actual time=0.210..0.213 rows=0 loops=1)
   Output: asset.id, asset."livePhotoVideoId"
   Filter: ((album."ownerId" = 'dc14ff4e-654b-4edb-9f6c-3a6a0f59f3f1'::uuid) OR ("user".id = 'dc14ff4e-654b-4edb-9f6c-3a6a0f59f3f1'::uuid))
   Buffers: shared hit=15
   ->  Nested Loop  (cost=8.99..18.05 rows=1 width=64) (actual time=0.209..0.211 rows=0 loops=1)
         Output: album.id, album."ownerId", asset.id, asset."livePhotoVideoId"
         Buffers: shared hit=15
         ->  Bitmap Heap Scan on public.asset  (cost=8.71..12.73 rows=1 width=32) (actual time=0.095..0.097 rows=1 loops=1)
               Output: asset.id, asset."livePhotoVideoId"
               Recheck Cond: ((asset.id = 'd011dd26-f03e-4fe1-8cc4-9244d81a308d'::uuid) OR (asset."livePhotoVideoId" = 'd011dd26-f03e-4fe1-8cc4-9244d81a308d'::uuid))
               Filter: (asset."deletedAt" IS NULL)
               Heap Blocks: exact=1
               Buffers: shared hit=6
               ->  BitmapOr  (cost=8.71..8.71 rows=1 width=0) (actual time=0.082..0.083 rows=0 loops=1)
                     Buffers: shared hit=5
                     ->  Bitmap Index Scan on "asset_id_stackId_idx"  (cost=0.00..4.42 rows=1 width=0) (actual time=0.050..0.050 rows=1 loops=1)
                           Index Cond: (asset.id = 'd011dd26-f03e-4fe1-8cc4-9244d81a308d'::uuid)
                           Buffers: shared hit=3
                     ->  Bitmap Index Scan on "asset_livePhotoVideoId_idx"  (cost=0.00..4.29 rows=1 width=0) (actual time=0.030..0.030 rows=0 loops=1)
                           Index Cond: (asset."livePhotoVideoId" = 'd011dd26-f03e-4fe1-8cc4-9244d81a308d'::uuid)
                           Buffers: shared hit=2
         ->  Nested Loop  (cost=0.28..5.32 rows=1 width=48) (actual time=0.110..0.111 rows=0 loops=1)
               Output: album.id, album."ownerId", "albumAssets"."assetsId"
               Buffers: shared hit=9
               ->  Seq Scan on public.album  (cost=0.00..1.01 rows=1 width=32) (actual time=0.015..0.020 rows=4 loops=1)
                     Output: album.id, album."ownerId"
                     Filter: (album."deletedAt" IS NULL)
                     Buffers: shared hit=1
               ->  Index Only Scan using album_asset_pkey on public.album_asset "albumAssets"  (cost=0.28..4.30 rows=1 width=32) (actual time=0.021..0.021 rows=0 loops=4)
                     Output: "albumAssets"."albumsId", "albumAssets"."assetsId"
                     Index Cond: (("albumAssets"."albumsId" = album.id) AND ("albumAssets"."assetsId" = asset.id))
                     Heap Fetches: 0
                     Buffers: shared hit=8
   ->  Nested Loop Left Join  (cost=4.30..11.80 rows=3 width=32) (never executed)
         Output: "albumUsers"."albumsId", "user".id
         Inner Unique: true
         ->  Bitmap Heap Scan on public.album_user "albumUsers"  (cost=4.17..11.28 rows=3 width=32) (never executed)
               Output: "albumUsers"."albumsId", "albumUsers"."usersId"
               Recheck Cond: ("albumUsers"."albumsId" = album.id)
               ->  Bitmap Index Scan on "album_user_albumsId_idx"  (cost=0.00..4.17 rows=3 width=0) (never executed)
                     Index Cond: ("albumUsers"."albumsId" = album.id)
         ->  Index Scan using user_pkey on public."user"  (cost=0.13..0.17 rows=1 width=16) (never executed)
               Output: "user".id
               Index Cond: ("user".id = "albumUsers"."usersId")
               Filter: ("user"."deletedAt" IS NULL)
 Planning:
   Buffers: shared hit=346
 Planning Time: 5.202 ms
 Execution Time: 0.462 ms

Synthetic tests

I ran simplified EXPLAINs on synthetic data to compare plans and performance of &&, IN, ANY.

The script below creates a minimal table with 100K records and runs simplified SELECTs using &&, IN, ANY and various number of checked IDs.

Script to test `&&`, `IN`, `ANY` on synthetic data

#!/usr/bin/env bash

image=ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:c44be5f2871c59362966d71eab4268170eb6f5653c0e6170184e72b38ffdf107
container=testdb
db_storage_type=HDD

# explain command and queries to explain
explain="explain (analyze, verbose, buffers)"
# &&    where array[id, "livePhotoVideoId"] && array['10758605-...','d5533ca8-...']::uuid[]
query_overlap="select id, \"livePhotoVideoId\" from a where array[id, \"livePhotoVideoId\"] && array[:uuid_csv]::uuid[]"
# in    where id in ('10758605-...','d5533ca8-...') or "livePhotoVideoId" in ('10758605-...','d5533ca8-...')
query_in="select id, \"livePhotoVideoId\" from a where id in (:uuid_csv) or \"livePhotoVideoId\" in (:uuid_csv)"
# any   where id = any(array['10758605-...','d5533ca8-...']::uuid[]) or "livePhotoVideoId" = any(array['10758605-...','d5533ca8-...']::uuid[])
query_any="select id, \"livePhotoVideoId\" from a where id = any(array[:uuid_csv]::uuid[]) or \"livePhotoVideoId\" = any(array[:uuid_csv]::uuid[])"

# functions
psql_cmd() {
  docker exec -i "$container" psql -U postgres --tuples-only "$@"
}

run_testdb() {
  echo "==== create testdb ===="
  docker run -d --rm --name "$container" -e DB_STORAGE_TYPE="$db_storage_type" -e DB_PASSWORD=postgres "$image"
  sleep 2
  until docker exec "$container" pg_isready ; do
    sleep 1
  done
}

fin_testdb() {
  echo "==== remove testdb ===="
  docker stop "$container"
}

create_dataset() {
  echo "==== create dataset ===="

  known_asset_id=$(docker exec "$container" cat /proc/sys/kernel/random/uuid)
  known_live_id=$(docker exec "$container" cat /proc/sys/kernel/random/uuid)

  query_create=$(cat <<EOF
    CREATE UNLOGGED TABLE a (
      id uuid DEFAULT gen_random_uuid() PRIMARY KEY,
      "livePhotoVideoId" uuid
    );

    -- every 10th record is live photo
    INSERT INTO a ("livePhotoVideoId")
    SELECT CASE WHEN random() < 0.1 THEN gen_random_uuid() END
    FROM generate_series(1, 99998);

    -- insert known IDs to be queried later
    INSERT INTO a (id) VALUES ('$known_asset_id');
    INSERT INTO a ("livePhotoVideoId") VALUES ('$known_live_id');

    CREATE INDEX "a_livePhotoVideoId_idx" ON a USING btree ("livePhotoVideoId");
    ANALYZE a;

    -- just bunch of uuids to be queried
    CREATE UNLOGGED TABLE id (
      id uuid DEFAULT gen_random_uuid() PRIMARY KEY
    );
    INSERT INTO id 
    SELECT
    FROM generate_series(1, 5000);
EOF
)
  psql_cmd <<<"$query_create"
}

gen_uuid_csv() {
  uuid_count=$1
  if [[ $uuid_count -eq 1 ]]; then
    uuid_csv="'$known_asset_id'"
  else
    query_uuid="select :'known_asset_id' id
      union (select :'known_live_id' limit (:uuid_count - 1))
      union (select id::text from id limit (:uuid_count - 2))
      limit (:uuid_count)"
    query_uuid_csv="select string_agg('''' || id::text || '''', ', ') from ($query_uuid) t"
    uuid_csv=$(psql_cmd -Aq -v known_asset_id="$known_asset_id" -v known_live_id="$known_live_id" -v uuid_count="$uuid_count" <<<"$query_uuid_csv")
  fi
}

explain() {
  uuid_count=$1
  gen_uuid_csv "$uuid_count"

  echo "==== explain: 'in' uuid_count: $uuid_count ===="
  psql_cmd -v uuid_csv="$uuid_csv" <<<"$explain $query_in"

  echo "==== explain: 'any' uuid_count: $uuid_count ===="
  psql_cmd -v uuid_csv="$uuid_csv" <<<"$explain $query_any"

  echo "==== explain: '&&' uuid_count: $uuid_count ===="
  psql_cmd -v uuid_csv="$uuid_csv" <<<"$explain $query_overlap"
}

# main
run_testdb
create_dataset
explain 1
explain 2
explain 50
explain 500 | cut -c-200
explain 3000 | cut -c-200
fin_testdb

Output: compare && / IN / ANY for 1 checked ID

==== explain: 'in' uuid_count: 1 ====
 Bitmap Heap Scan on public.a  (cost=8.73..16.40 rows=2 width=32) (actual time=0.029..0.030 rows=1 loops=1)
   Output: id, "livePhotoVideoId"
   Recheck Cond: ((a.id = 'a9e7388c-229a-4b92-88e1-bdd993026017'::uuid) OR (a."livePhotoVideoId" = 'a9e7388c-229a-4b92-88e1-bdd993026017'::uuid))
   Heap Blocks: exact=1
   Buffers: shared hit=4 read=2
   ->  BitmapOr  (cost=8.73..8.73 rows=2 width=0) (actual time=0.027..0.027 rows=0 loops=1)
         Buffers: shared hit=3 read=2
         ->  Bitmap Index Scan on a_pkey  (cost=0.00..4.43 rows=1 width=0) (actual time=0.019..0.019 rows=1 loops=1)
               Index Cond: (a.id = 'a9e7388c-229a-4b92-88e1-bdd993026017'::uuid)
               Buffers: shared hit=3
         ->  Bitmap Index Scan on "a_livePhotoVideoId_idx"  (cost=0.00..4.30 rows=1 width=0) (actual time=0.008..0.008 rows=0 loops=1)
               Index Cond: (a."livePhotoVideoId" = 'a9e7388c-229a-4b92-88e1-bdd993026017'::uuid)
               Buffers: shared read=2
 Planning:
   Buffers: shared hit=73 read=1
 Planning Time: 0.145 ms
 Execution Time: 0.076 ms

==== explain: 'any' uuid_count: 1 ====
 Bitmap Heap Scan on public.a  (cost=8.73..16.39 rows=2 width=32) (actual time=0.025..0.025 rows=1 loops=1)
   Output: id, "livePhotoVideoId"
   Recheck Cond: ((a.id = ANY ('{a9e7388c-229a-4b92-88e1-bdd993026017}'::uuid[])) OR (a."livePhotoVideoId" = ANY ('{a9e7388c-229a-4b92-88e1-bdd993026017}'::uuid[])))
   Heap Blocks: exact=1
   Buffers: shared hit=6
   ->  BitmapOr  (cost=8.73..8.73 rows=2 width=0) (actual time=0.023..0.023 rows=0 loops=1)
         Buffers: shared hit=5
         ->  Bitmap Index Scan on a_pkey  (cost=0.00..4.43 rows=1 width=0) (actual time=0.020..0.021 rows=1 loops=1)
               Index Cond: (a.id = ANY ('{a9e7388c-229a-4b92-88e1-bdd993026017}'::uuid[]))
               Buffers: shared hit=3
         ->  Bitmap Index Scan on "a_livePhotoVideoId_idx"  (cost=0.00..4.30 rows=1 width=0) (actual time=0.002..0.002 rows=0 loops=1)
               Index Cond: (a."livePhotoVideoId" = ANY ('{a9e7388c-229a-4b92-88e1-bdd993026017}'::uuid[]))
               Buffers: shared hit=2
 Planning:
   Buffers: shared hit=133 read=1
 Planning Time: 0.216 ms
 Execution Time: 0.063 ms

==== explain: '&&' uuid_count: 1 ====
 Seq Scan on public.a  (cost=0.00..1810.00 rows=500 width=32) (actual time=10.956..10.956 rows=1 loops=1)
   Output: id, "livePhotoVideoId"
   Filter: (ARRAY[a.id, a."livePhotoVideoId"] && '{a9e7388c-229a-4b92-88e1-bdd993026017}'::uuid[])
   Rows Removed by Filter: 99999
   Buffers: shared hit=567
 Planning:
   Buffers: shared hit=120
 Planning Time: 0.176 ms
 Execution Time: 10.984 ms

Output: compare && / IN / ANY for 500 checked IDs

==== explain: 'in' uuid_count: 500 ====
 Seq Scan on public.a  (cost=2.50..2562.50 rows=998 width=32) (actual time=6.833..6.834 rows=2 loops=1)
   Output: id, "livePhotoVideoId"
   Filter: ((a.id = ANY ('{1b3e299a-15c7-4b01-b9d7-fdc396913f7d,78fe5bbb-de0a-48d2-a24d-533b3bd537b8,959276f6-3cb3-4d24-a267-f6c4767e5b03,a2707f03-4927-4380-8afb-4565826210a8,7619425c-17e2-4551-8163-3
   Rows Removed by Filter: 99998
   Buffers: shared hit=560
 Planning:
   Buffers: shared hit=137
 Planning Time: 0.429 ms
 Execution Time: 6.862 ms

==== explain: 'any' uuid_count: 500 ====
 Seq Scan on public.a  (cost=2.50..2562.50 rows=998 width=32) (actual time=6.857..6.858 rows=2 loops=1)
   Output: id, "livePhotoVideoId"
   Filter: ((a.id = ANY ('{1b3e299a-15c7-4b01-b9d7-fdc396913f7d,78fe5bbb-de0a-48d2-a24d-533b3bd537b8,959276f6-3cb3-4d24-a267-f6c4767e5b03,a2707f03-4927-4380-8afb-4565826210a8,7619425c-17e2-4551-8163-3
   Rows Removed by Filter: 99998
   Buffers: shared hit=560
 Planning:
   Buffers: shared hit=137
 Planning Time: 0.416 ms
 Execution Time: 6.886 ms

==== explain: '&&' uuid_count: 500 ====
 Seq Scan on public.a  (cost=0.00..1810.00 rows=91843 width=32) (actual time=211.565..211.569 rows=2 loops=1)
   Output: id, "livePhotoVideoId"
   Filter: (ARRAY[a.id, a."livePhotoVideoId"] && '{1b3e299a-15c7-4b01-b9d7-fdc396913f7d,78fe5bbb-de0a-48d2-a24d-533b3bd537b8,959276f6-3cb3-4d24-a267-f6c4767e5b03,a2707f03-4927-4380-8afb-4565826210a8,7
   Rows Removed by Filter: 99998
   Buffers: shared hit=567
 Planning:
   Buffers: shared hit=120
 Planning Time: 0.266 ms
 Execution Time: 211.606 ms

Related

I've noticed a recent PR with access check performance improvements - #21827. Seemingly it does not overlap with this PR because it applies the changes at higher level and do not touch checkAlbumAccess internals.

Out of scope

A similar clause in checkSharedLinkAccess()

.where(
  sql`array["asset"."id", "asset"."livePhotoVideoId", "albumAssets"."id", "albumAssets"."livePhotoVideoId"]`,
  '&&',
  sql`array[${sql.join([...assetIds])}]::uuid[] `,
)

Please describe to which degree, if any, an LLM was used in creating this pull request.

To research/compare &&, IN(), ANY() in Postgres.