Twitter KeyError - 'core' #2225

ghost · 2022-01-25T00:49:47Z

Using this command:

gallery-dl.exe --download-archive twitter.sqlite3 --cookies cookies-twitter-com.txt https://twitter.com/CENSORED --verbose

The following error occurs, when it didn't, in the past:

[twitter][error] An unexpected error occurred: KeyError - 'core'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[twitter][debug]
Traceback (most recent call last):
File "gallery_dl\job.pyc", line 79, in run
File "gallery_dl\extractor\twitter.pyc", line 62, in items
File "gallery_dl\extractor\twitter.pyc", line 1140, in _pagination_tweets
KeyError: 'core'

Any idea as to what might be happening here? I am using Version 1.20.2.

God-damnit-all · 2022-01-25T01:08:01Z

Looks like yet another backend change.

ermnick · 2022-01-25T01:12:34Z

I also get this error when it encounters a quote tweet where the quoted tweet has been deleted.

szero · 2022-01-25T01:34:54Z

[twitter][error] An unexpected error occurred: KeyError - 'core'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[twitter][debug]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/gallery_dl/job.py", line 79, in run
    for msg in extractor:
  File "/usr/local/lib/python3.9/dist-packages/gallery_dl/extractor/twitter.py", line 62, in items
    for tweet in self.tweets():
  File "/usr/local/lib/python3.9/dist-packages/gallery_dl/extractor/twitter.py", line 1140, in _pagination_tweets
    quoted["core"]["user_results"]["result"]
KeyError: 'core'

Can confirm, the verbose flag doesn't give on which url error has occured tho

Klonoa723 · 2022-01-25T02:03:32Z

Oh, I just shared about this in another thread, lol, guess I'll also share again my issue here, hoping it'll help figure why this happens.

[gallery-dl][debug] Version 1.20.2 - Executable
[gallery-dl][debug] Python 3.7.9 - Windows-10-10.0.19041
[gallery-dl][debug] requests 2.27.1 - urllib3 1.26.8
[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/xxx'
[twitter][debug] Using TwitterTimelineExtractor for 'https://twitter.com/xxx'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/7mjxD3-C6BxitPMVQ6w0-Q/UserByScreenName?variables=%7B%22screen_name%22%3A%22freesmellsinc%22%2C%22withSafetyModeUserFields%22%3Atrue%2C%22withSuperFollowsUserFields%22%3Atrue%7D HTTP/1.1" 200 946
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/DhQ8lYnLh5T5K8aVUgHVnQ/UserTweets?variables=%7B%22userId%22%3A%221086688842588086272%22%2C%22count%22%3A100%2C%22withQuickPromoteEligibilityTweetFields%22%3Atrue%2C%22includePromotedContent%22%3Afalse%2C%22withSuperFollowsUserFields%22%3Atrue%2C%22withBirdwatchPivots%22%3Afalse%2C%22withDownvotePerspective%22%3Afalse%2C%22withReactionsMetadata%22%3Afalse%2C%22withReactionsPerspective%22%3Afalse%2C%22withSuperFollowsTweetFields%22%3Atrue%2C%22withClientEventToken%22%3Afalse%2C%22withBirdwatchNotes%22%3Afalse%2C%22withVoice%22%3Atrue%2C%22withV2Timeline%22%3Afalse%2C%22__fs_interactive_text%22%3Afalse%2C%22__fs_dont_mention_me_view_api_enabled%22%3Afalse%7D HTTP/1.1" 200 88934

[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/DhQ8lYnLh5T5K8aVUgHVnQ/UserTweets?variables=%7B%22userId%22%3A%221086688842588086272%22%2C%22count%22%3A100%2C%22withQuickPromoteEligibilityTweetFields%22%3Atrue%2C%22includePromotedContent%22%3Afalse%2C%22withSuperFollowsUserFields%22%3Atrue%2C%22withBirdwatchPivots%22%3Afalse%2C%22withDownvotePerspective%22%3Afalse%2C%22withReactionsMetadata%22%3Afalse%2C%22withReactionsPerspective%22%3Afalse%2C%22withSuperFollowsTweetFields%22%3Atrue%2C%22withClientEventToken%22%3Afalse%2C%22withBirdwatchNotes%22%3Afalse%2C%22withVoice%22%3Atrue%2C%22withV2Timeline%22%3Afalse%2C%22__fs_interactive_text%22%3Afalse%2C%22__fs_dont_mention_me_view_api_enabled%22%3Afalse%2C%22cursor%22%3A%22HBaCwLiZ3ZTrjCkAAA%3D%3D%22%7D HTTP/1.1" 200 88477

[twitter][error] An unexpected error occurred: KeyError - 'core'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[twitter][debug]
Traceback (most recent call last):
  File "gallery_dl\job.pyc", line 79, in run
  File "gallery_dl\extractor\twitter.pyc", line 62, in items
  File "gallery_dl\extractor\twitter.pyc", line 1140, in _pagination_tweets
KeyError: 'core'

I'm not sure if maybe this could be caused by my config file maybe not being updated and such?
Since I've been only replacing the exe whenever a new version came out.

God-damnit-all · 2022-01-25T02:41:51Z

Oh, I just shared about this in another thread, lol, guess I'll also share again my issue here, hoping it'll help figure why this happens.

[gallery-dl][debug] Version 1.20.2 - Executable
[gallery-dl][debug] Python 3.7.9 - Windows-10-10.0.19041
[gallery-dl][debug] requests 2.27.1 - urllib3 1.26.8
[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/xxx'
[twitter][debug] Using TwitterTimelineExtractor for 'https://twitter.com/xxx'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/7mjxD3-C6BxitPMVQ6w0-Q/UserByScreenName?variables=%7B%22screen_name%22%3A%22freesmellsinc%22%2C%22withSafetyModeUserFields%22%3Atrue%2C%22withSuperFollowsUserFields%22%3Atrue%7D HTTP/1.1" 200 946
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/DhQ8lYnLh5T5K8aVUgHVnQ/UserTweets?variables=%7B%22userId%22%3A%221086688842588086272%22%2C%22count%22%3A100%2C%22withQuickPromoteEligibilityTweetFields%22%3Atrue%2C%22includePromotedContent%22%3Afalse%2C%22withSuperFollowsUserFields%22%3Atrue%2C%22withBirdwatchPivots%22%3Afalse%2C%22withDownvotePerspective%22%3Afalse%2C%22withReactionsMetadata%22%3Afalse%2C%22withReactionsPerspective%22%3Afalse%2C%22withSuperFollowsTweetFields%22%3Atrue%2C%22withClientEventToken%22%3Afalse%2C%22withBirdwatchNotes%22%3Afalse%2C%22withVoice%22%3Atrue%2C%22withV2Timeline%22%3Afalse%2C%22__fs_interactive_text%22%3Afalse%2C%22__fs_dont_mention_me_view_api_enabled%22%3Afalse%7D HTTP/1.1" 200 88934

[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/DhQ8lYnLh5T5K8aVUgHVnQ/UserTweets?variables=%7B%22userId%22%3A%221086688842588086272%22%2C%22count%22%3A100%2C%22withQuickPromoteEligibilityTweetFields%22%3Atrue%2C%22includePromotedContent%22%3Afalse%2C%22withSuperFollowsUserFields%22%3Atrue%2C%22withBirdwatchPivots%22%3Afalse%2C%22withDownvotePerspective%22%3Afalse%2C%22withReactionsMetadata%22%3Afalse%2C%22withReactionsPerspective%22%3Afalse%2C%22withSuperFollowsTweetFields%22%3Atrue%2C%22withClientEventToken%22%3Afalse%2C%22withBirdwatchNotes%22%3Afalse%2C%22withVoice%22%3Atrue%2C%22withV2Timeline%22%3Afalse%2C%22__fs_interactive_text%22%3Afalse%2C%22__fs_dont_mention_me_view_api_enabled%22%3Afalse%2C%22cursor%22%3A%22HBaCwLiZ3ZTrjCkAAA%3D%3D%22%7D HTTP/1.1" 200 88477

[twitter][error] An unexpected error occurred: KeyError - 'core'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[twitter][debug]
Traceback (most recent call last):
  File "gallery_dl\job.pyc", line 79, in run
  File "gallery_dl\extractor\twitter.pyc", line 62, in items
  File "gallery_dl\extractor\twitter.pyc", line 1140, in _pagination_tweets
KeyError: 'core'

I'm not sure if maybe this could be caused by my config file maybe not being updated and such? Since I've been only replacing the exe whenever a new version came out.

No, it's just that twitter updated their backend again shortly after the new version was released. Unfortunate timing.

- fix Tweets with deleted quotes - fix suspended Tweets without 'legacy' entry - fix unified_cards without 'type'

mikf · 2022-01-25T15:36:46Z

Fixed in 2bf554a among other things.
Has anyone found more, before I push the next release?

God-damnit-all · 2022-01-25T21:11:32Z

Fixed in 2bf554a among other things. Has anyone found more, before I push the next release?

I ended up getting KeyError - 'legacy', which seems to indicate that the fix for when legacy is missing broke handling it when it's actually present.

mikf · 2022-01-25T21:54:56Z

Which line? There are many places where this could potentially happen.

God-damnit-all · 2022-01-25T22:10:46Z

Which line? There are many places where this could potentially happen.

That's strange, running it again and it's giving me an error with core instead of legacy.

[twitter][error] An unexpected error occurred: KeyError - 'core'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[twitter][debug]
Traceback (most recent call last):
  File "gallery_dl\job.py", line 79, in run
  File "gallery_dl\extractor\twitter.py", line 62, in items
  File "gallery_dl\extractor\twitter.py", line 1142, in _pagination_tweets
KeyError: 'core'

sasami-i · 2022-01-25T22:23:27Z

Just made this account to submit this, I get this output when I run gallery-dl twitter.com/asteroid_ill/media --verbose
https://pastebin.com/jJN7JkyB
hopefully this can help

kattjevfel · 2022-01-25T22:34:35Z

@sasami-i It was fixed in the latest development version, you're running 1.20.2. Since I see you're running manjaro, you can simply just install gallery-dl-git from AUR.

mikf · 2022-01-25T23:03:49Z

@ImportTaste fixed in c180806
Your example must have been a Twitter error. The retweet entry for it is empty as a timeline result, but is normal when fetching it as a single Tweet.

  "retweeted_status_result": {
    "result": {
      "__typename": "Tweet",
      "rest_id": "122…096"
    }
  }

God-damnit-all · 2022-01-25T23:08:42Z

@ImportTaste fixed in c180806 Your example must have been a Twitter error. The retweet entry for it is empty as a timeline result, but is normal when fetching it as a single Tweet.
  "retweeted_status_result": {
    "result": {
      "__typename": "Tweet",
      "rest_id": "122…096"
    }
  }

Does the fix retrieve it as a single tweet, or does it just skip it altogether?

mikf · 2022-01-25T23:14:48Z

It skips the retweet since there is nothing there.
Twitter says there is a retweet, but doesn't deliver any data for it.

Klonoa723 · 2022-01-26T00:56:40Z

I can confirm on my end that the 'core' problem I had when downloading certain twitter galleries, well, it's fixed for me with the latest version.
Thanks a lot!

mikf added a commit that referenced this issue Jan 25, 2022

[twitter] fix several errors (#2212, #2216, #2225)

2bf554a

- fix Tweets with deleted quotes - fix suspended Tweets without 'legacy' entry - fix unified_cards without 'type'

mikf added a commit that referenced this issue Jan 25, 2022

[twitter] fix deleted/invalid retweets (#2225)

c180806

mikf closed this as completed Jan 30, 2022

mikf added the site:change label Jan 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Twitter KeyError - 'core' #2225

Twitter KeyError - 'core' #2225

ghost commented Jan 25, 2022

God-damnit-all commented Jan 25, 2022

ermnick commented Jan 25, 2022

szero commented Jan 25, 2022

Klonoa723 commented Jan 25, 2022

God-damnit-all commented Jan 25, 2022

mikf commented Jan 25, 2022

God-damnit-all commented Jan 25, 2022

mikf commented Jan 25, 2022

God-damnit-all commented Jan 25, 2022

sasami-i commented Jan 25, 2022

kattjevfel commented Jan 25, 2022 •

edited

Loading

mikf commented Jan 25, 2022

God-damnit-all commented Jan 25, 2022

mikf commented Jan 25, 2022

Klonoa723 commented Jan 26, 2022

Twitter KeyError - 'core' #2225

Twitter KeyError - 'core' #2225

Comments

ghost commented Jan 25, 2022

God-damnit-all commented Jan 25, 2022

ermnick commented Jan 25, 2022

szero commented Jan 25, 2022

Klonoa723 commented Jan 25, 2022

God-damnit-all commented Jan 25, 2022

mikf commented Jan 25, 2022

God-damnit-all commented Jan 25, 2022

mikf commented Jan 25, 2022

God-damnit-all commented Jan 25, 2022

sasami-i commented Jan 25, 2022

kattjevfel commented Jan 25, 2022 • edited Loading

mikf commented Jan 25, 2022

God-damnit-all commented Jan 25, 2022

mikf commented Jan 25, 2022

Klonoa723 commented Jan 26, 2022

kattjevfel commented Jan 25, 2022 •

edited

Loading