[BUG] Exif Data not processed for large video files #4349
Comments
|
Hello is this issue still there? |
|
Hi Alex, yes, the issue is still present. I performed a full EXIF scan today and the large video, where I noticed the issue, still does not have any metadata. |
|
Issue still exists. This is the log message Immich gives at the end of exif extraction:
|
|
I can confirm this behaviour. You have to add exiftool -api LargeFileSupport=1 ........ when exiftool reads the metadata! |
|
Old bug. Devs ignore it. |
|
Also experienced this with a simple 30 minute long 1080p 30fps video recorded on a pixel 5. It's about 6GB Not only does the video date fall back to when I uploaded it and not have the GPS location, the video length appears as 0:00. Also, might be related to the video showing as "not backed up" (cloud icon crossed out) on my mobile app, even though it successfully backed up and I checked that the file on the server has the same MD5, the app seems to think it wasn't backed up. If it's comparing date taken via metadata then this might be connected. Oddly the successfully uploaded video which shows up as May 5 doesn't show up as a duplicate in the app which has the video as May 4th. Edit: Oddly logging out and back in to rebuild the timeline fixes the sync status issue, it now shows the video under the incorrect date but shows cloud+local so it seems to have realized the backed up copy is the same as the local copy |
add it where? |
|
SOLUTION - SIMPLE WORKAROUND photostructure/exiftool-vendored.js#109 (comment) This was the solution, very easy fix was to create your ExifTool config file with the options you need.
.ExifTool_config in the /app/node_modules/exiftool-vendored.pl/bin directory
I am running Immich in a venv, on bare metal myself.
restart your containers, or in my case |
|
@TransRapid Did you just refresh the metadata of the specific asset after doing that to get it to work? I'm using docker so I mapped the .ExifTool_config file into /usr/src/app/node_modules/exiftool-vendored.pl, which exists on immich_server and immich_microservices, but not immich_machine_learning, so I skipped that one. I also mapped it into ~ (/root) and I can successfully cat the file to see the contents, it's owned by root and 660 permissions. I restarted the containers and refreshed the metadata of the large file but it still shows 0:00 for the duration unfortunately. |
|
I can't remember if I refreshed the data or had just removed it prior then tried to reupload, but I think I had re-uploaded the file. I would try downloading and removing the file from Immich entirely. Try removing it from the trash too . Also The other consideration might be some SQL filesize limit. Also, try turning on verbose logging within the Administration settings from the web interface. Then do something like How are you importing these files? Have you tried a new large file? What do your logs show when you try this? |
|
@TransRapid So I figured it out! And yes I was also only putting the config in the home directory ( At any rate, that didn't work (even when reuploading) and I think the reason why is that exiftool may not be looking in What did work though was the folder at So the workaround for docker users should just be: added to immich_microservices (I just added it to the main container too so I don't forget to keep the workaround next update when the main container starts doing the microservices stuff). And of course I haven't tested whether refreshing metadata on the bugged items fixes it since I switched to deleting, emptying trash, and reuploading like you did just to make sure I was properly testing the fix. Hope this helps anybody else with this issue until a built-in fix is implemented. |
|
Ah okay so the winning location to place the config file in any case seems to be in the same directory as your exiftool executable. I did check and that is the same area I have mine in. I don't know why I didn't catch that here when I posted. I will update mine as well for anyone else finding this. |
|
Fixed via #9894 |
|
Hi, |
|
You need to re-run metadata extraction for the affect assets. |
|
So it actually looks like this bug wasn't fixed by #9894, at least with my 6GB file. I removed the .ExifTool_config file and upgraded to 1.106.1, deleted the large video and emptied trash, then re-uploaded it and it presented the same problem as before. 0:00 length, dated as today when I uploaded it, and no GPS location. restoring the .ExifTool_config file and using the "Refresh metadata" option fixed it. (It's nice to know that just using the refresh button is enough to test this since I had been re-uploading the file every time before) Looking at the debug log when refreshing the metadata confirms this: |
|
Fixed by using workaround : Already refreshed metadata with no success. |
|
So I did a little more debugging, using And so I was able to replicate the issue by running locally: Then I can execute a sequence of commands like: What seems to be happening is that after So the original solution of passing the args when constructing exiftool doesn't seem viable, and exiftool-vendored doesn't seem to have any ability to load in an argument to be used each time we use it, only args to spawn the initial process, so the solution probably is to pass the largefilesupport flag in each place we call exiftool functions. So digging through immich and exiftool-vendored code, we use exiftool-vendored in 3 places for The Thankfully, the diff --git a/server/src/repositories/metadata.repository.ts b/server/src/repositories/metadata.repository.ts
index 5baf07829..eca7a0bef 100644
--- a/server/src/repositories/metadata.repository.ts
+++ b/server/src/repositories/metadata.repository.ts
@@ -21,26 +21,27 @@ export class MetadataRepository implements IMetadataRepository {
) {
this.logger.setContext(MetadataRepository.name);
}
async teardown() {
await exiftool.end();
}
readTags(path: string): Promise<ImmichTags | null> {
return exiftool
.read(path, undefined, {
...DefaultReadTaskOptions,
+ optionalArgs: ['-api', 'largefilesupport=1'],
defaultVideosToUTC: true,
backfillTimezones: true,
inferTimezoneFromDatestamps: true,
useMWG: true,
numericTags: [...DefaultReadTaskOptions.numericTags, 'FocalLength'],
/* eslint unicorn/no-array-callback-reference: off, unicorn/no-array-method-this-argument: off */
geoTz: (lat, lon) => geotz.find(lat, lon)[0],
})
.catch((error) => {
this.logger.warn(`Error reading exif data (${path}): ${error}`, error?.stack);
return null;
}) as Promise<ImmichTags | null>;
}
I think this means the previous change can be reverted since we don't need to maintain an instance of exiftool and can just call the methods statically. Also it looks like I created a PR with this fix here: #10167 |
|
Fix is now merged and I can confirm that it works using the latest container built from the main branch and no more workaround, just refresh metadata and then reload the page 🎉 |
|
I actually left my symbolic link that I placed there originally. It is a good option to have additional controls if need be. It references the same one I use outside of Immich. |
The bug
Dear Team,
first of all: thank you so much for your amazing work!
I fell over the issue of immich not extracting metadata for a large video file (6.4 GB) within my external library. When I manually checked the exif data with exiftool, I got the following output:
ExifTool Version Number : 12.40 File Name : 2023-09-23_13-48-19.MOV Directory : . File Size : 6.4 GiB File Modification Date/Time : 2023:09:24 20:29:33+02:00 File Access Date/Time : 2023:10:04 10:14:35+02:00 File Inode Change Date/Time : 2023:09:24 20:29:33+02:00 File Permissions : -rwxr-xr-x File Type : MOV File Type Extension : mov MIME Type : video/quicktime Major Brand : Apple QuickTime (.MOV/QT) Minor Version : 0.0.0 Compatible Brands : qt Warning : End of processing at large atom (LargeFileSupport not enabled)When adding the parameter
-api largefilesupport=1metadata was extracted as expected. Is it possible to enable large file support in immich as well?Many thanks once again!
The OS that Immich Server is running on
Ubuntu 22.04.3
Version of Immich Server
v1.81.1
Version of Immich Mobile App
v1.80
Platform with the issue
Your docker-compose.yml content
Your .env content
Reproduction steps
Additional information
No response
The text was updated successfully, but these errors were encountered: