[Deviantart] [Twitter] [Possibly other sites] Downloading a twitter account's main tab sometimes gets the same tweet twice and aborts

[Scripter17]

Sometimes when downloading a twitter account the same tweet will be grabbed twice, causing the extractor to end early. Just now I've seen this happen on a deviantart favourite list.
The only solution I can think of is keeping a list of all images the current gallery-dl command processed and, if it encounters any one of them again, don't abort

This has happened before on Twitter, but the only example whose URL I remember is NSFW and takes about 10GB to happen so you're going to have to take my word on this

[nisehime]

Most likely it's just a tweet with a quote retweet of the previous one. I've seen this too (here, for example (NSFW)) and it's not really a bug. Either disable quote retweets or increase the abort threshold.

[Doofy420]

"quoted": false will result in missing content, because it causes gallery-dl to also skip the actual tweets that have been quote RT'd at any point in the timeline. I forgot why it works this way but it's been discussed here before, I think?

On the flip side, "quoted": true may have gallery-dl downloading self QRTd images earlier than it should, causing abort <5 to trip prematurely when it gets to the original QRTd tweet.

TL;DR: I find that the only way to way to get --abort x to play nice with "quoted": true is to set it to 5.

edit: "quoted": false and abort 1 now works properly together after the update, thanks mikf!

[mikf]

The only solution I can think of is keeping a list of all images the current gallery-dl command processed and, if it encounters any one of them again, don't abort

That's actually already implemented via the image-unique option, but I concur with nisehime: just increase the abort threshold.

"quoted": false will result in missing content, because it causes gallery-dl to also skip the actual tweets that have been quote RT'd at any point in the timeline. I forgot why it works this way but it's been discussed here before, I think?.

No, that's an actual bug and shouldn't happen. Fixed in ae78d95.