[twitter] Fix stop before real end #573
Conversation
Fix for #544. Makes sure that it really reached the end by checking that both "min_position" is null and "has_more_items" is false before stopping.
gallery_dl/extractor/twitter.py
Outdated
| @@ -231,7 +231,7 @@ def _tweets_from_api(self, url, max_position=None): | |||
| data["items_html"], '<div class="tweet ', '\n</li>'): | |||
| yield tweet | |||
|
|
|||
| if not data["has_more_items"]: | |||
| if not data["has_more_items"] and data["min_position"] == None: | |||
There was a problem hiding this comment.
data is not guaranteed to contain a min_position key (#544 (comment)), so either check with data.get("min_position") is None or ("min_position" in data and data["min_position"] is None), depending on whether there is a difference between min_position is set and None and min_position is not set.
(is is also preferred over == when testing for None)
There was a problem hiding this comment.
I didn't know that, whoops. The fix ended up being a bit more complicated then I thought. There is 6 possible responses from Twitter:
gallery-dl should continue:
{"has_more_items":True} (This happens on the first page)
{"has_more_items":True, "min_position":"123..."}
{"has_more_items":False, "min_position":"123..."}
gallery-dl should stop:
{"has_more_items":False}
{"has_more_items":False, "min_position":None}
{"has_more_items":True, "min_position":None}
Basically we want to stop whenever min_position is None or doesn't exist, except when there is more items and min_position doesn't exist.
Fix for #544. Makes sure that it really reached the end by checking that both "min_position" is null and "has_more_items" is false before stopping.