Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] Some sites download image by sending POST #2433

Closed
chinggg opened this issue Mar 20, 2022 · 1 comment
Closed

[Enhancement] Some sites download image by sending POST #2433

chinggg opened this issue Mar 20, 2022 · 1 comment

Comments

@chinggg
Copy link
Contributor

chinggg commented Mar 20, 2022

Some sites, such as https://stocksnap.io download image by sending POST request to a fixed url (in this case https://stocksnap.io/photo/download/) with photoId and _csrf. The csrf token is hidden in the html of each individual site and can be extracted from <input type="hidden" name="_csrf" value="">.

How to implement SearchExtractor under the circumstance? Since job.run() will iterate over extractor.items() and dispatch each msg to handle_url(), which then called download(). But It seems the HttpDownloader is always sending GET request. (see https://github.com/mikf/gallery-dl/blob/master/gallery_dl/downloader/http.py#L122-L125)

mikf added a commit that referenced this issue Mar 21, 2022
by setting the '_http_method' metadata field for a file
mikf added a commit that referenced this issue Mar 24, 2022
by setting the '_http_data' metadata field for a file

needed in addition to be34927
to download files with POST requests
@mikf
Copy link
Owner

mikf commented Mar 24, 2022

This is now possible by setting _http_method and _http_data in a file's metadata dict.
(be34927, c0c1277)

For https://stocksnap.io/, it would be

file["_http_method"] = "POST"
file["_http_data"] = {
    "photoId": "…",
    "_csrf"  : "…",
}

@chinggg chinggg closed this as completed May 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants