Twitter "info" subcategory isn't configurable and isn't an option for "include"

[Twi-Hard]

When I try to set options for the "info" subcategory within the "twitter" section of the config it doesn't do anything. Here's what I have in the config:

"info": {
  "archive": "",
  "filename": "twitter_info_{date!T}.json",
  "cookies": ""
}

The json only downloads when I use -O archive= and it downloads with the name "twitter_None_00.json"

-E shows the custom settings but that's not what's being used when I try to download it.

❯ gdl https://x.com/mlp_twilight/info -E
[gallery-dl][debug] Starting InfoJob for 'https://x.com/mlp_twilight/info'
[twitter][debug] Using TwitterInfoExtractor for 'https://x.com/mlp_twilight/info'
Category / Subcategory
  "twitter" / "info"

Filename format (custom):
  "twitter_info_{date!T}.{extension!l}"
Filename format (default):
  "{tweet_id}_{num}.{extension}"

Directory format (custom):
  {"retweet_id": ["user", "{user[name]}", "retweets", "{author[name]}"], "quote_id": ["user", "{user[name]}", "quoted", "{author[name]}"], "reply_id": ["user", "{user[name]}", "replies", "{author[name]}"], "": ["user", "{user[name]}"]}
Directory format (default):
  ["{category}", "{user[name]}"]

Archive format (custom):
  "{tweet_id}, {retweet_id}, {quote_id}, {reply_id}, {search}, {num}"
Archive format (default):
  "{tweet_id}_{retweet_id}_{num}"

Here's the full "twitter" section of the config:

"twitter": {
  "archive": "$HOME/tools/gallery-dl/archive/twitter.com-media.db",
  "archive-prefix": "twitter.com, ",
  "archive-format": "{tweet_id}, {retweet_id}, {quote_id}, {reply_id}, {search}, {num}",
  "archive-pragma": [
    "journal_mode=WAL",
    "synchronous=NORMAL"
  ],
  "archive-event": [
    "file",
    "skip"
  ],
  "base-directory": "$OTHER/twitter.com/",
  "postprocessors": [
    {
      "name": "metadata",
      "mode": "json",
      "archive": "$HOME/tools/gallery-dl/archive/twitter.com-metadata.db",
      "archive-prefix": "twitter.com, ",
      "archive-format": "{tweet_id}, {retweet_id}, {quote_id}, {reply_id}, {search}, {num}",
      "archive-pragma": [
        "journal_mode=WAL",
        "synchronous=NORMAL"
      ],
      "archive-event": [
        "file",
        "skip"
      ],
      "directory": null,
      "filter": "subcategory != 'background' and subcategory != 'avatar'",
      "filename": "twitter_{tweet_id}_00.json",
      "extension": null,
      "extension-format": null,
      "event": "post",
      "mtime": true,
      "content-format": null,
      "whitelist": [
        "twitter"
      ],
      "blacklist": null
    },
    {
      "name": "metadata",
      "mode": "json",
      "directory": null,
      "filter": "subcategory == 'background'",
      "filename": "twitter_background_{date!T}.json",
      "extension": null,
      "extension-format": null,
      "event": "post",
      "mtime": true,
      "content-format": null,
      "whitelist": [
        "twitter"
      ],
      "blacklist": null
    },
    {
      "name": "metadata",
      "mode": "json",
      "directory": null,
      "filter": "subcategory == 'avatar'",
      "filename": "twitter_avatar_{date!T}.json",
      "extension": null,
      "extension-format": null,
      "event": "post",
      "mtime": true,
      "content-format": null,
      "whitelist": [
        "twitter"
      ],
      "blacklist": null
    },
    {
      "name": "exec",
      "async": false,
      "archive": "$HOME/tools/gallery-dl/archive/twitter.com-optimization-png.db",
      "archive-prefix": "twitter.com, ",
      "archive-format": "{tweet_id}, {retweet_id}, {quote_id}, {reply_id}, {search}, {num}, png",
      "archive-pragma": [
        "journal_mode=WAL",
        "synchronous=NORMAL"
      ],
      "archive-event": [
        "file",
        "skip"
      ],
      "command": "oxipng -o max --strip safe --force --preserve {}",
      "event": "after",
      "filter": "extension == 'png'",
      "mtime": true,
      "whitelist": [
        "#twitter"
      ]
    },
    {
      "name": "exec",
      "async": false,
      "archive": "$HOME/tools/gallery-dl/archive/twitter.com-optimization-jpg.db",
      "archive-prefix": "twitter.com, ",
      "archive-format": "{tweet_id}, {retweet_id}, {quote_id}, {reply_id}, {search}, {num}, jpg",
      "archive-pragma": [
        "journal_mode=WAL",
        "synchronous=NORMAL"
      ],
      "archive-event": [
        "file",
        "skip"
      ],
      "command": "exiftran -ai {} && jpegtran -copy none -perfect -optimize -outfile {} {}",
      "event": "after",
      "filter": "extension == 'jpg'",
      "mtime": true,
      "whitelist": [
        "#twitter"
      ]
    }
  ],
  "directory": {
    "retweet_id": [
      "user",
      "{user[name]}",
      "retweets",
      "{author[name]}"
    ],
    "quote_id": [
      "user",
      "{user[name]}",
      "quoted",
      "{author[name]}"
    ],
    "reply_id": [
      "user",
      "{user[name]}",
      "replies",
      "{author[name]}"
    ],
    "": [
      "user",
      "{user[name]}"
    ]
  },
  "search": {
    "archive": "$HOME/tools/gallery-dl/archive/twitter.com-search.db",
    "directory": {
      "retweet_id": [
        "search",
        "{search}",
        "{user[name]}",
        "retweets",
        "{author[name]}"
      ],
      "quote_id": [
        "search",
        "{search}",
        "{user[name]}",
        "quoted",
        "{author[name]}"
      ],
      "reply_id": [
        "search",
        "{search}",
        "{user[name]}",
        "replies",
        "{author[name]}"
      ],
      "": [
        "search",
        "{search}",
        "{user[name]}"
      ]
    }
  },
  "background": {
    "archive": "",
    "filename": "twitter_background_{date!T}.{extension!l}"
  },
  "avatar": {
    "archive": "",
    "filename": "twitter_avatar_{date!T}.{extension!l}"
  },
  "info": {
    "archive": "",
    "filename": "twitter_info_{date!T}.json",
    "cookies": ""
  },
  "filename": "twitter_{tweet_id}_{num:>02}.{extension!l}",
  "download": true,
  "username": null,
  "password": null,
  "cookies": null,
  "cards": false,
  "cards-blacklist": null,
  "conversations": false,
  "logout": false,
  "quoted": true,
  "replies": true,
  "retweets": true,
  "pinned": false,
  "text-tweets": true,
  "twitpic": false,
  "users": "timeline",
  "videos": true,
  "csrf": "auto",
  "expand": false,
  "size": [
    "orig",
    "4096x4096",
    "large",
    "medium",
    "small"
  ],
  "#syndication": "extended",
  "syndication": false,
  "timeline": {
    "strategy": "tweets"
  },
  "unique": true
}

It would be extremely useful if getting the info metadata could be an option for "include" if possible

[mikf]

The info extractor does not write any actual files, it only returns a bunch of metadata which can be viewed with -j or written to disk with an "event": "post" metadata post processor.

It would be extremely useful if getting the info metadata could be an option for "include" if possible

bd932b6

[mikf]

Right, you needed to be pinged to get proper notifications, didn't you? @Twi-Hard

I added the following post processor to your list, disabled info for the main metadata one, and added a directory setting to info since the metadata structure for it is a bit different ("directory": ["user", "{name}"],), and it seems to work as it should.

    {
      "name": "metadata",
      "filename": "twitter_info_{date!T}.json",
      "event": "post",
      "mtime": true,
      "whitelist": [":info"]
    }

[Twi-Hard]

It seems to work
Thank you :)

[a84r7a3rga76fg]

What does the subcategory info do?