Feature Request: Twitter - Download Only Replies to Self

[flaccidbagel]

Figure this might be a bit of a stretch, but at the same time figured I'd ask.

Is it possible to have twitter extractor download images from replies, but only if the reply comes from the same username as the originating tweet, but not replies to other users? Or possibly just a reply "whitelist" of sorts where it only downloads the file if a certain, specified username is present as one of the users the reply is targeting?

Noticed several artists would upload large sets as replies to themselves, but they also of course reply to other users with media.

[mikf]

It should be possible with --filter "reply_id and reply_to == author['name']"

Reply tweets have a nonzero reply_id value, which is the tweet id of the tweet it is replying to, and also a reply_to value, which is the screen name of the author of the replied-to tweet.

And you should also use search results rather than the regular or media timeline, i.e. https://twitter.com/search?q=from:USER instead of https://twitter.com/USER or https://twitter.com/USER/media. Maybe the media timeline results work as well, but you are better of doing a search.

[flaccidbagel]

That works great! I forgot about the -K tag as a whole, so that's definitely something to keep in mind for looking into filtering down other sites as well.

…

On Wed, Jan 20, 2021 at 4:02 PM Mike Fährmann ***@***.***> wrote: It should be possible with --filter "reply_id and reply_to == author['name']" Reply tweets have a nonzero reply_id value, which is the tweet id of the tweet it is replying to, and also a reply_to value, which is the screen name of the author of the replied-to tweet. And you should also use search results rather than the regular or media timeline, i.e. https://twitter.com/search?q=from:USER instead of https://twitter.com/USER or https://twitter.com/USER/media. Maybe the media timeline results work as well, but you are better of doing a search. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1254 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIJ5CZLMVWAMLMS6U24QR3TS25AH3ANCNFSM4WGYYEPA> .

-- Regards, Christopher Williams

[arisboch]

How do I put that into the .conf file?

[arisboch]

And --filter "reply_id and reply_to == author['name']" didn't work ;-(

[mikf]

How do I put that into the .conf file?

With image-filter:

        "twitter":
        {
            "image-filter": "reply_id and reply_to == author['name']"
        }

And --filter "reply_id and reply_to == author['name']" didn't work ;-(

As a command-line option or when writing this into a config file?

[arisboch]

@mikf I tried to put that line into my config file, but it didn't work ;-( For example [https://twitter.com/PrinceCanary/status/1424882701073453056](this post)

yielded that message (and no files downloaded):

 % gallery-dl https://twitter.com/PrinceCanary/status/1424882701073453056 --verbose
[gallery-dl][debug] Version 1.18.3-dev
[gallery-dl][debug] Python 3.9.5 - Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.33
[gallery-dl][debug] requests 2.25.1 - urllib3 1.26.2
[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/PrinceCanary/status/1424882701073453056'
[twitter][debug] Using TwitterTweetExtractor for 'https://twitter.com/PrinceCanary/status/1424882701073453056'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/2/timeline/conversation/1424882701073453056.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweet=true&count=100&ext=mediaStats%2ChighlightedLabel HTTP/1.1" 200 8138
[twitter][debug] Active postprocessor modules: [MetadataPP]

Is that an user error?

[mikf]

@arisboch
The filter expression from #1254 (comment) only matches self-replies and nothing else.
Use not reply_id or reply_to == author['name'] to allow both regular Tweets as well as replies to the same users.
Or you set the replies option to "self" once the next release comes out. (e5a93e1)

[arisboch]

@mikf I used both with the newest version of gallery-dl fresh off the repo, and it doesn't download the replies

Example: https://twitter.com/melatonezone/status/1425274586002862083

[mikf]

@arisboch I forgot to mention that you also need to enable the conversations option to get all tweets including replies from a URL to a specific tweet. Sorry about that.

[arisboch]

@mikf Thanks! 🥳

[arisboch]

But there's now another problem with some posts: When I try to download this post, it only downloads the post it is a comment to, not the post itself.

[mikf]

Well, that tweet is a reply to a different user and therefore gets filtered by the image-filter settings from #1254 (comment) as well as "replies": "self". I'm not sure how best to work around that. Maybe only have "replies": "self" without image-filter in your config and use -o replies=1 whenever you want to download from a reply tweet.

[arisboch]

@mikf The problem with that is that it'll download all replies, which would be a real problem with a sufficiently popular tweet (the "self" option works, though).

[mikf]

With 749802c and 8a70b94, user is now always the user an input URL points to (e.g. berdacs for https://twitter.com/berdacs/status/1425697321036115968). Use that to properly filter out unwanted Tweets instead of relying on "replies": "self".