Warning: 'End of processing at large atom (LargeFileSupport not enabled)'

[medikoo]

Describe the bug

I've tried to retrieve EXIF data from 2GB+ movie file, and got End of processing at large atom (LargeFileSupport not enabled) warning in return data.

Is there any way to enable large file support?

Environment (please complete the following information):

• macOs 12.1
• Node.JS: 14.19.1

[mceachen]

When you build your instance of ExifTool, you can give it arguments. The default is ["-stay_open", "True", "-@", "-"]: you could try ["-api", "largefilesupport=1", "-stay_open", "True", "-@", "-"]:

import { DefaultExiftoolArgs, ExifTool } from "exiftool-vendored"

const et = new ExifTool({
  exiftoolArgs: ["-api", "largefilesupport=1", ...DefaultExiftoolArgs]
})

(please tell me if this works for you!)

[medikoo]

@mceachen great thanks for the response. Just tried that, and adding exiftoolArgs as you've proposed doesn't introduce any difference, I still get same warning

[mceachen]

You can also try setting up a config file for exiftool. see this forum post for details:

https://exiftool.org/forum/index.php?topic=3916.0

(closing as this is a config issue for exiftool rather than a coding issue for this module.)

[medikoo]

(closing as this is a config issue for exiftool rather than a coding issue for this module.)

This plugin embeds exiftool (it's not that the user installs it separately and then uses this module with it). So it'll be good to provide some hint on how this embedded ExifTool can be additionally configured. Currently, it's not clear to me.
I see two dependencies hosting it, exiftool-vendored.exe and exiftool-vendored.pl, where can I find the mentioned config file?

[mceachen]

The path to a config file can be set up as described here:

#27 (comment)

[medikoo]

After some fiddling, I've found that it works without any extra settings if .ExifTool_config with following content is placed at current working directory, large file support will be enbled

%Image::ExifTool::UserDefined::Options = (
    LargeFileSupport => 1,
);

Otherwise, as pointed in above comment, path to config file can be passed as first argument:

new ExifTool({
	exiftoolArgs: [
		"-config", path.resolve(os.homedir(), ".ExifTool_config"), "-u", "-stay_open", "True", "-@",
		"-"
	]
});

[stephen304]

So it turns out the reason that ["-api", "largefilesupport=1", "-stay_open", "True", "-@", "-"] doesn't work is because stay_open mode resets the flags after each -execute, so lfs will only be enabled for the first operation. I was able to solve it without a config file by passing the flags to read instead of the constructor: (essentially sending the lfs flag with each operation into stdin with each option)

exiftool.read(path, undefined, {
    ...DefaultReadTaskOptions,
    // Enable exiftool LFS to parse metadata for files larger than 2GB.
    optionalArgs: ['-api', 'largefilesupport=1'],
});

Note that passing in args in the second parameter (optionalArgs) doesn't work because it gets overwritten when constructing the ReadTaskOptions here (at least when I move optionalArgs to the end of the object it gets passed through so I assume it's being clobbered by the other 2 entries):

exiftool-vendored.js/src/ExifTool.ts

Lines 292 to 296 in ece7bcd

	ReadTask.for(file, {
	optionalArgs,
	...pick(this.options, ...ReadTaskOptionFields),
	...options,
	})

[mceachen]

Oh 💩 ! I'll get this fixed, thanks for the research, @stephen304 !

Edit: 💩 💩 : I was about halfway through adding a new readArgs to ExifToolOptions before I realized ExifTool.read(file, optionalArgs) was already a thing: you want

exiftool.read(path, ['-api', 'largefilesupport=1'])

If this doesn't work, @stephen304 , holler.

[stephen304]

@mceachen I probably should have provided more context - I just now realized that the second parameter is probably only supposed to be used if you aren't using the third to pass in a ReadTaskOptions object.

The full code in immich is:

exiftool
      .read(path, undefined, {
        ...DefaultReadTaskOptions,

        defaultVideosToUTC: true,
        backfillTimezones: true,
        inferTimezoneFromDatestamps: true,
        useMWG: true,
        numericTags: [...DefaultReadTaskOptions.numericTags, 'FocalLength'],
        /* eslint unicorn/no-array-callback-reference: off, unicorn/no-array-method-this-argument: off */
        geoTz: (lat, lon) => geotz.find(lat, lon)[0],
      })

And since geoTz is a function, passing string array of options as the second parameter doesn't work - so that's why I guess they are passing in undefined as the second parameter and led to the confusion when I tried to combine your previous suggestion while still passing the rest of the arguments in as an object.

So moral of the story, exiftool.read(path, ['-api', 'largefilesupport=1']) should work and otherwise if using the third param, use

exiftool.read(path, undefined, {
    ...DefaultReadTaskOptions,
    optionalArgs: ['-api', 'largefilesupport=1'],
    <...>
})

[mceachen]

Given that the current API is a bit confusing, I'm actually going to move the readArgs into ExifToolOptions, and have now deprecated the method signature that took both a string[] and options hash:

https://photostructure.github.io/exiftool-vendored.js/classes/ExifTool.html#read