HTTP request failed: 418

[marlekal]

i'm using the --download-archive FILE command, so it starts checking the existing files but it stops halfway and it shows me this error:
[weibo][error] HTTP request failed: 418: for url: https://m.weibo.cn/api/container/getIndex

[wankio]

oh i dont know it supported weibo now, but when i'm testing with archive enabled, everything worked, no problem at all

[marlekal]

i used it yesterday and it worked fine, it started resuming downloads right after checking the exsiting ones, today the folder got over 5000 files, it feels like it's getting overwhelmed while checking??

[mikf]

The 418 HTTP status code is Weibo's way of saying "You are overusing our API". I don't know why they aren't using the default 429: Too Many Requests, but going through a user profile without actually downloading any files got me a 418 error as well after issuing something like 20-30 requests to https://m.weibo.cn/api/container/getIndex?page=...&containerid=... in only a few seconds. So if anything is overwhelmed, it's Weibo's API servers.

The problem now is resuming your download from were it left off. With the way things work at the moment, gallery-dl will always start at the beginning, skip over already downloaded files, and will eventually fail with the same error because it sent too many API requests in too short of a time while trying to get to where it previously was. The only way would be to manually modify the start page in the source code itself:

gallery-dl/gallery_dl/extractor/weibo.py

Line 99 in 20eb6c4

params = {"page": 1, "containerid": "107603" + self.user_id[-10:]}

Maybe a --start-page or --pages flag that works somewhat like --range would be a helpful addition ...

[wankio]

i'm using this https://github.com/nondanee/weiboPicDownloader with simultaneous download set to 2. maybe you can take a look.

[marlekal]

when i set a range value it shows me the error immidiatly without even checking for existing files, but i did an attempt with a lower value on range and it did skip and searched untill it crashed.
I couldn't get the --start-page or --pages commands to works neither changing the page value on weibo.py file, i guess i'm doing it wrong

i'm using this https://github.com/nondanee/weiboPicDownloader with simultaneous download set to 2. maybe you can take a look.

thanks will do

[marlekal]

here is how i made it work.
i made a lot of gaps by moving out about 100 files out of the folder in order to make gallery-dl download them again which will lower the amount of spam, that made it reach the point it stopped at yesterday and resumed the downloads.
thank you both for your time