Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Twitter extractor errors on replies to deleted posts #838

Closed
ntqr opened this issue Jun 17, 2020 · 2 comments
Closed

Twitter extractor errors on replies to deleted posts #838

ntqr opened this issue Jun 17, 2020 · 2 comments
Labels

Comments

@ntqr
Copy link

ntqr commented Jun 17, 2020

It seems gallery-dl fails to extract a tweet if it is a reply to a deleted tweet. I'm using version 1.14.1
Examples:
https://twitter.com/i/web/status/1170041925560258560
https://twitter.com/i/web/status/1245387866395435008

@mikf
Copy link
Owner

mikf commented Jun 18, 2020

Not for me:

$ gallery-dl --version
1.14.1
$ gallery-dl https://twitter.com/i/web/status/1170041925560258560
/tmp/twitter/OhishiMasaru/1170041925560258560_1.jpg
$ gallery-dl https://twitter.com/i/web/status/1245387866395435008
/tmp/twitter/OhishiMasaru/1245387866395435008_1.jpg

The verbose log even mentions it ignoring deleted Tweets:

[twitter][debug] Skipping unavailable Tweet 1245375392023601152

This was an issue during the Twitter rewrite phase but was fixed in 655c98c before 1.14.1 was released. Are you sure you are actually running 1.14.1 or does it say 1.14.1-dev?

@ntqr
Copy link
Author

ntqr commented Jun 19, 2020

Sorry, should've took a closer look at the error. It seems that the error happens only if I supply cookies to the config. And the error happens because the cookies somehow causes the dict key 'tweet' in entry['content']['item']['content']['tweet'] to not exist. Here's the log:

C:\Users\Me>gallery-dl --no-download --verbose https://twitter.com/i/web/status/1170041925560258560
[gallery-dl][debug] Version 1.14.1
[gallery-dl][debug] Python 3.8.1 - Windows-10-10.0.18362-SP0
[gallery-dl][debug] requests 2.23.0 - urllib3 1.25.9
[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/i/web/status/1170041925560258560'
[twitter][debug] Using TwitterTweetExtractor for 'https://twitter.com/i/web/status/1170041925560258560'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): api.twitter.com:443
[urllib3.connectionpool][debug] https://api.twitter.com:443 "GET /2/timeline/conversation/1170041925560258560.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_composer_source=true&include_ext_alt_text=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweet=true&count=100&ext=mediaStats%2ChighlightedLabel%2CcameraMoment&include_quote_count=true HTTP/1.1" 200 2783
[twitter][error] An unexpected error occurred: KeyError - 'tweet'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[twitter][debug]
Traceback (most recent call last):
  File "c:\users\Me\appdata\local\programs\python\python38-32\lib\site-packages\gallery_dl\job.py", line 61, in run
    for msg in self.extractor:
  File "c:\users\Me\appdata\local\programs\python\python38-32\lib\site-packages\gallery_dl\extractor\twitter.py", line 42, in items
    for tweet in self.tweets():
  File "c:\users\Me\appdata\local\programs\python\python38-32\lib\site-packages\gallery_dl\extractor\twitter.py", line 344, in tweets
    return TwitterAPI(self).tweet(self.tweet_id)
  File "c:\users\Me\appdata\local\programs\python\python38-32\lib\site-packages\gallery_dl\extractor\twitter.py", line 418, in tweet
    for tweet in self._pagination(endpoint):
  File "c:\users\Me\appdata\local\programs\python\python38-32\lib\site-packages\gallery_dl\extractor\twitter.py", line 482, in _pagination
    tid = entry["content"]["item"]["content"]["tweet"]["id"]
KeyError: 'tweet'

and without cookies:

C:\Users\Me>gallery-dl -o twitter.cookies= --no-download https://twitter.com/i/web/status/1170041925560258560
* .\gallery-dl\twitter\OhishiMasaru\1170041925560258560_1.jpg

@mikf mikf added bug and removed cant-reproduce labels Jun 19, 2020
@mikf mikf closed this as completed in d39eedd Jun 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants