Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(server): don't crash when refreshing large libraries #7934

Merged
merged 122 commits into from
Aug 28, 2024
Merged
Changes from 1 commit
Commits
Show all changes
122 commits
Select commit Hold shift + click to select a range
5e497e5
add job to check for offline files
etnoy Mar 13, 2024
8bb73d6
fix lint
etnoy Mar 13, 2024
247429c
only check for offline when using checkForOffline
etnoy Mar 13, 2024
0803458
improve tests
etnoy Mar 14, 2024
f68bcf0
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Mar 14, 2024
d09d4d3
remove old test
etnoy Mar 14, 2024
5b581ce
wip
etnoy Mar 14, 2024
3b0d993
remove trie
etnoy Mar 14, 2024
4ba95bb
refactor batches
etnoy Mar 14, 2024
95b57f0
also check offline status
etnoy Mar 14, 2024
68a4925
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Mar 14, 2024
d8dd1fb
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Mar 14, 2024
311d7d5
fix spelling
etnoy Mar 14, 2024
d7a78e5
don't do offline scan
etnoy Mar 14, 2024
fa3a70a
rename scan to check
etnoy Mar 15, 2024
8bcee7f
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Mar 15, 2024
ff47d55
fix job statuses
etnoy Mar 15, 2024
380ae35
fix lint
etnoy Mar 15, 2024
f7f30a5
cleanup
etnoy Mar 15, 2024
e26f8b4
add test
etnoy Mar 15, 2024
5ae4fb8
open-api
etnoy Mar 15, 2024
f8039a7
fix test
etnoy Mar 15, 2024
a3ce28b
fix spinner
etnoy Mar 16, 2024
5bc22d5
reset text
etnoy Mar 16, 2024
3f56cbe
don't double batch
etnoy Mar 16, 2024
d8830e2
fix comments from mert
etnoy Mar 16, 2024
0bf31bd
remove tries
etnoy Mar 16, 2024
9b5a0a9
fix tests
etnoy Mar 16, 2024
649358c
fix e2e
etnoy Mar 17, 2024
5905fce
fix test
etnoy Mar 17, 2024
65d3990
fix test
etnoy Mar 17, 2024
890d488
add tests
etnoy Mar 17, 2024
f201773
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Mar 17, 2024
a5e6e90
fix lint
etnoy Mar 17, 2024
32b8630
fix e2e
etnoy Mar 17, 2024
17c9f0b
Merge remote-tracking branch 'origin' into feat/offline-files-job
etnoy Mar 18, 2024
0070b83
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Mar 18, 2024
69ce4e8
interweave scans
etnoy Mar 19, 2024
104ffdd
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Mar 19, 2024
3c5eb92
fix errors
etnoy Mar 19, 2024
6f34013
fix messages
etnoy Mar 19, 2024
e6c7618
fix test
etnoy Mar 19, 2024
7db36ea
add mock
etnoy Mar 19, 2024
b969fc7
fix sql
etnoy Mar 19, 2024
f28f899
fix e2e
etnoy Mar 19, 2024
8941071
use library batch size
etnoy Mar 19, 2024
482645e
Merge branch 'main' into feat/offline-files-job
alextran1502 Mar 20, 2024
76241e0
save -> update
etnoy Mar 20, 2024
bedb494
add file extensions
etnoy Mar 20, 2024
b288743
update specs
etnoy Mar 20, 2024
9a70787
test for import paths
etnoy Mar 20, 2024
f5073a1
check import paths when testing offline
etnoy Mar 20, 2024
11eb7f7
fix lint
etnoy Mar 20, 2024
7858cf4
normalize import path
etnoy Mar 20, 2024
ea47cb8
remove console logs
etnoy Mar 20, 2024
5cdf31c
decrease batch size to 1000
etnoy Mar 20, 2024
3fc6e82
add test for import path
etnoy Mar 20, 2024
cba8019
add test for already-online assets
etnoy Mar 20, 2024
94b9b4d
Merge remote-tracking branch 'origin' into feat/offline-files-job
etnoy Mar 20, 2024
f48992b
Merge remote-tracking branch 'origin' into feat/offline-files-job
etnoy Mar 20, 2024
38e2cde
fix merge
etnoy Mar 20, 2024
7043f2f
Merge remote-tracking branch 'origin' into feat/offline-files-job
etnoy Mar 20, 2024
6fd511e
fix lint
etnoy Mar 20, 2024
fbee95b
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Mar 22, 2024
17216e1
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Mar 25, 2024
2398edf
add library job back
etnoy Mar 25, 2024
2c4b174
add offline job to correct queue
etnoy Mar 25, 2024
f2c7235
library spec compiles now
etnoy Mar 25, 2024
5745010
move one test to new e2e
etnoy Mar 25, 2024
52a19b6
fix comments
etnoy Mar 25, 2024
2d40c85
fix comments
etnoy Mar 25, 2024
cfbad58
fix lint
etnoy Mar 25, 2024
37ad05f
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Mar 26, 2024
586bf19
refactor path validation
etnoy Mar 26, 2024
5515f57
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Mar 30, 2024
ee0fedf
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Apr 1, 2024
17f2adb
fix loop bug
etnoy Apr 1, 2024
0348372
remove logging
etnoy Apr 1, 2024
a352f7e
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Apr 4, 2024
b218e0d
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Apr 5, 2024
2a9f615
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Apr 9, 2024
b3bb5c5
expect responses
etnoy Apr 9, 2024
d095f48
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Jul 24, 2024
75e4fc5
fix asset mock
etnoy Jul 25, 2024
be0804f
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Jul 25, 2024
2b1d348
take the straightforward approach
etnoy Jul 25, 2024
c079e38
use generator correctly
etnoy Jul 25, 2024
71fcb75
fix vitest on file edit
etnoy Jul 26, 2024
b2e75aa
bump vitest to 1.6.0
etnoy Jul 26, 2024
039c480
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Jul 26, 2024
5decc21
Merge branch 'chore/vitest-160' of https://github.com/immich-app/immi…
etnoy Jul 26, 2024
7ff68f6
test for offline check
etnoy Jul 26, 2024
7502516
add e2e tests for offlining assets depending on import path
etnoy Jul 26, 2024
a675973
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Jul 26, 2024
759d4cc
cleanup e2e test after finish
etnoy Jul 26, 2024
184cf1e
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Jul 27, 2024
9354baf
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Aug 11, 2024
7e09f95
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Aug 14, 2024
f93af3e
cleanup library service
etnoy Aug 14, 2024
407d6ac
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Aug 23, 2024
329def7
paginate the walk generator
etnoy Aug 23, 2024
166b7cb
fix tests
etnoy Aug 23, 2024
11b5b38
fix typo
etnoy Aug 23, 2024
06b98b0
refactoring handleOfflineCheck
etnoy Aug 23, 2024
169bf7e
better testing of handleOfflineCheck
etnoy Aug 23, 2024
09b1f4d
fix lint
etnoy Aug 23, 2024
b82d264
handle large library deletions
etnoy Aug 23, 2024
7f1e5d9
dont check if library is deleted
etnoy Aug 23, 2024
a30a104
fix mock
etnoy Aug 23, 2024
914667d
add a 100k page size to library
etnoy Aug 25, 2024
20bcdc7
fix loading animation
etnoy Aug 26, 2024
f7f833b
better log messages
etnoy Aug 26, 2024
dd853dd
Better logging for offline asset removal
etnoy Aug 27, 2024
88b6499
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Aug 27, 2024
779b1ee
fix sql and tests
etnoy Aug 27, 2024
b9532cb
fix number format
etnoy Aug 27, 2024
4033d15
Merge branch 'main' of https://github.com/immich-app/immich into feat…
etnoy Aug 27, 2024
eb613ff
Remove submodule
etnoy Aug 27, 2024
ccff383
fix format
etnoy Aug 27, 2024
c587740
chore: cleanup
jrasm91 Aug 28, 2024
937b41c
chore: merge main
jrasm91 Aug 28, 2024
bd96c07
chore: fix tests
jrasm91 Aug 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
don't do offline scan
etnoy committed Mar 14, 2024
commit d7a78e5f25faf8a7c96c4985cd46df0283a03468
2 changes: 0 additions & 2 deletions server/src/domain/library/library.service.ts
Original file line number Diff line number Diff line change
@@ -689,8 +689,6 @@ export class LibraryService extends EventEmitter {
.filter((validation) => validation.isValid)
.map((validation) => validation.importPath);

await this.handleQueueOfflineScan(job);

const generator = this.storageRepository.walk({
pathsToCrawl: validImportPaths,
exclusionPatterns: library.exclusionPatterns,