metadata options seem to be ignored?

[calestyo]

Hey.

I can't seem to get this working...

Having a config file like this:

{
        "metadata": {
                "directory": "metadata"
        },
        
        "extractor": {
                "deviantart": {
                        "include": ["gallery", "scraps"],
                        "metadata": true,
                        "extra": true,
                        "client-id": 0,
                        "client-secret": "012345",
                        "refresh-token": "012345"
                }
        }
}

When not specifying --write-medata on the command line, no metadata files are created at all. When specifying, they are written, but metadata.directory seems to be ignored.

Also, at least extractor.deviantart.metadata doesn't seem to have any effect - with and without the very same data is extracted (i.e. no difference in license, and so on, as the documentation implies).

Any ideas? Thanks,
Chris.

[mikf]

You need to put the metadata.directory value inside a post-processor object:

{
    "extractor": {
        "deviantart": {
            "postprocessors": [{
                "name": "metadata",
                "directory": "metadata"
            }],
            "include": ["gallery", "scraps"],
            "metadata": true,
            "extra": true,
            "client-id": 0,
            "client-secret": "012345",
            "refresh-token": "012345"
        }
    }
}

You also shouldn't use --write-medata when having this option in your config, because it's going to overwrite it (see #678)

Also, at least extractor.deviantart.metadata doesn't seem to have any effect - with and without the very same data is extracted (i.e. no difference in license, and so on, as the documentation implies).

extractor.deviantart.extra automatically enables extractor.deviantart.metadata

[calestyo]

Hi.

Thanks, the thing with the postprocessor did the trick.

The only remaining thing is now, that when downloading journals, it doesn't download linked sta.sh images in them. Is this expected to work, or am I making some mistake?

Thanks.

[Vrihub]

I got tricked too in the past by a similar misunderstanding.

Indeed configuration.rst sometimes fails at explaining where to put some of the configuration directives.

Suggested improvements:

• Link gallery-dl.conf and gallery-dl-example.conf in the intro of configuration.rst too.
• Add a global extractor.postprocessors snippet, showing the use of metadata, to gallery-dl-example.conf (instead of the current null value)

[calestyo]

@mikf Any idea about the thing with sta.sh images not downloaded for journals?

Also perhaps one can rededicate this ticket to improve documentation for how to use metadata:

Maybe as @Vrihub suggested, i.e. something like:

extractor.*.postprocessors
Contains all post-processors (like metadata, ...) to be applied on the respective extractor. Example:
"postprocessors": [{"name": "metadata", "directory": "someOtherDirectory"}, {...}] with name specifying the name of the postprocessor

And for the metadata section something like:
This needs to be specified as a post.processor for an extractor.

Thanks :-)

[mikf]

Any idea about the thing with sta.sh images not downloaded for journals?

Enabling extra only looks at the descriptions of Deviations (that's why it automatically enables metadata, to have access to said descriptions) Extending this functionality to also include Journal texts shouldn't be too much of a problem, I hope.

[calestyo]

Would sta.sh downloading work already for scraps? Since these seem to be just gallery items anyway?

[calestyo]

Extending this functionality to also include Journal texts shouldn't be too much of a problem, I hope.

Just wondered, whether this is actually on the roadmap? :-)

[mikf]

Sorry, I kind of forgot about this ... anyway: 41d0316

[calestyo]

Thanks a lot.

I guess this issue can be closed than.