Patreon Cloudflare CAPTCHA #508

TheOddball · 2019-12-07T19:11:33Z

I'm getting a cloudflare captcha on patreon requests. I updated the libraries, and installed pyopenssl.

mikf · 2019-12-07T22:06:51Z

It seems that pyOpenSSL is now actually causing those CAPTCHAs.
I've tested on Windows with Python 3.7.5 and 3.8.0 and it fails with pyOpenSSL installed, but starts working again when uninstalling it.

You are out of luck if you are using the .exe version, but otherwise update your Python interpreter to the latest version and uninstall pyOpenSSL.

The output of gallery-dl -v would also be helpful.

TheOddball · 2019-12-08T02:19:21Z

On my FreeNAS jail that I use to download:
[gallery-dl][debug] Version 1.11.1 [gallery-dl][debug] Python 3.6.9 - FreeBSD-11.3-RELEASE-amd64-64bit-ELF [gallery-dl][debug] requests 2.22.0 - urllib3 1.25.7

Also tested on my Windows workstation, same error.
[gallery-dl][debug] Version 1.11.1 [gallery-dl][debug] Python 3.8.0 - Windows-10-10.0.18362-SP0 [gallery-dl][debug] requests 2.22.0 - urllib3 1.25.7

GiovanH · 2019-12-08T08:12:10Z

Also experiencing this problem.
I don't understand how uninstalling pyOpenSSL solved the error for you. On my machine, uninstalling pyOpenSSL just throws the error I would expect it to:

[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /api/user/23297717 HTTP/1.1" 403 None
[patreon][warning] Cloudflare CAPTCHA - Install 'pyOpenSSL' and try again
[patreon][error] HttpError: '403 Forbidden' for 'https://www.patreon.com/api/user/23297717'

With pyOpenSSL installed and upgraded, here's output from -v, running off the latest git version and using cookies from a live session:

$ python3 -m gallery_dl https://www.patreon.com/homestuck --cookies cookies.txt -v
[gallery-dl][debug] Version 1.11.2-dev - Git HEAD: 5eb89b2
[gallery-dl][debug] Python 3.7.0b4 - Windows-10-10.0.17763-SP0
[gallery-dl][debug] requests 2.22.0 - urllib3 1.25.7
[gallery-dl][debug] Starting DownloadJob for 'https://www.patreon.com/homestuck'
[gallery-dl][debug] Updating urllib3 ciphers
[patreon][debug] Using PatreonCreatorExtractor for 'https://www.patreon.com/homestuck'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.patreon.com:443
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /homestuck HTTP/1.1" 302 None
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /homestuck/posts HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /api/posts?include=user,images,attachments,user_defined_tags,campaign,poll.choices,poll.current_user_responses.user,poll.current_user_responses.choice,poll.current_user_responses.poll,access_rules.tier.null&fields%5Bpost%5D=change_visibility_at,comment_count,content,current_user_can_delete,current_user_can_view,current_user_has_liked,embed,image,is_paid,like_count,min_cents_pledged_to_view,post_file,published_at,patron_count,patreon_url,post_type,pledge_url,thumbnail_url,teaser_text,title,upgrade_url,url,was_posted_by_campaign_owner&fields%5Buser%5D=image_url,full_name,url&fields%5Bcampaign%5D=avatar_photo_url,earnings_visibility,is_nsfw,is_monthly,name,url&fields%5Baccess_rule%5D=access_rule_type,amount_cents&sort=-published_at&filter%5Bis_draft%5D=false&filter%5Bcontains_exclusive_posts%5D=true&filter%5Bcampaign_id%5D=3063938&json-api-use-default-includes=false&json-api-version=1.0 HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /api/user/23297717 HTTP/1.1" 403 None
[patreon][warning] Cloudflare CAPTCHA
[patreon][error] HttpError: '403 Forbidden' for 'https://www.patreon.com/api/user/23297717'

(pyOpenSSL is now disabled by default)

GiovanH · 2019-12-09T03:28:05Z

@mikf gallery-dl still isn't working, with or without pyOpenSSL enabled, as previously detailed.

$ python3 -m gallery_dl https://www.patreon.com/homestuck --cookies cookies.txt -v -o pyopenssl=false
[gallery-dl][debug] Version 1.12.0 - Git HEAD: f451be4
[gallery-dl][debug] Python 3.7.0b4 - Windows-10-10.0.17763-SP0
[gallery-dl][debug] requests 2.22.0 - urllib3 1.25.7
[gallery-dl][debug] Starting DownloadJob for 'https://www.patreon.com/homestuck'
...
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /api/user/23297717 HTTP/1.1" 403 None
[patreon][warning] Cloudflare CAPTCHA
[patreon][error] HttpError: '403 Forbidden' for 'https://www.patreon.com/api/user/23297717'

TheOddball · 2019-12-09T03:48:01Z

Uninstalling pyOpenSSL fixed the error for me

GiovanH · 2019-12-09T04:04:40Z

Still busted on my end.

$ python3 -m gallery_dl https://www.patreon.com/giovan --cookies cookies.txt -v -o pyopenssl=true
[gallery-dl][debug] Version 1.12.0 - Git HEAD: f451be4
[gallery-dl][debug] Python 3.7.0b4 - Windows-10-10.0.17763-SP0
[gallery-dl][debug] requests 2.22.0 - urllib3 1.25.7
[gallery-dl][debug] Starting DownloadJob for 'https://www.patreon.com/giovan'
PYOPENSSL: True
[patreon][debug] Using PatreonCreatorExtractor for 'https://www.patreon.com/giovan'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.patreon.com:443
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /giovan HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /api/posts?include=user,images,attachments,user_defined_tags,campaign,poll.choices,poll.current_user_responses.user,poll.current_user_responses.choice,poll.current_user_responses.poll,access_rules.tier.null&fields%5Bpost%5D=change_visibility_at,comment_count,content,current_user_can_delete,current_user_can_view,current_user_has_liked,embed,image,is_paid,like_count,min_cents_pledged_to_view,post_file,published_at,patron_count,patreon_url,post_type,pledge_url,thumbnail_url,teaser_text,title,upgrade_url,url,was_posted_by_campaign_owner&fields%5Buser%5D=image_url,full_name,url&fields%5Bcampaign%5D=avatar_photo_url,earnings_visibility,is_nsfw,is_monthly,name,url&fields%5Baccess_rule%5D=access_rule_type,amount_cents&sort=-published_at&filter%5Bis_draft%5D=false&filter%5Bcontains_exclusive_posts%5D=true&filter%5Bcampaign_id%5D=1125369&json-api-use-default-includes=false&json-api-version=1.0 HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /api/user/4029845 HTTP/1.1" 403 None
[patreon][warning] Cloudflare CAPTCHA
[patreon][error] HttpError: '403 Forbidden' for 'https://www.patreon.com/api/user/4029845'

$ python3 -m gallery_dl https://www.patreon.com/giovan --cookies cookies.txt -v -o pyopenssl=false
[gallery-dl][debug] Version 1.12.0 - Git HEAD: f451be4
[gallery-dl][debug] Python 3.7.0b4 - Windows-10-10.0.17763-SP0
[gallery-dl][debug] requests 2.22.0 - urllib3 1.25.7
[gallery-dl][debug] Starting DownloadJob for 'https://www.patreon.com/giovan'
PYOPENSSL: False
[patreon][debug] Using PatreonCreatorExtractor for 'https://www.patreon.com/giovan'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.patreon.com:443
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /giovan HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /api/posts?include=user,images,attachments,user_defined_tags,campaign,poll.choices,poll.current_user_responses.user,poll.current_user_responses.choice,poll.current_user_responses.poll,access_rules.tier.null&fields%5Bpost%5D=change_visibility_at,comment_count,content,current_user_can_delete,current_user_can_view,current_user_has_liked,embed,image,is_paid,like_count,min_cents_pledged_to_view,post_file,published_at,patron_count,patreon_url,post_type,pledge_url,thumbnail_url,teaser_text,title,upgrade_url,url,was_posted_by_campaign_owner&fields%5Buser%5D=image_url,full_name,url&fields%5Bcampaign%5D=avatar_photo_url,earnings_visibility,is_nsfw,is_monthly,name,url&fields%5Baccess_rule%5D=access_rule_type,amount_cents&sort=-published_at&filter%5Bis_draft%5D=false&filter%5Bcontains_exclusive_posts%5D=true&filter%5Bcampaign_id%5D=1125369&json-api-use-default-includes=false&json-api-version=1.0 HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /api/user/4029845 HTTP/1.1" 403 None
[patreon][warning] Cloudflare CAPTCHA
[patreon][error] HttpError: '403 Forbidden' for 'https://www.patreon.com/api/user/4029845'

(For line 6, I just added a print statement in extractor/common.py to verify that the options set were registering)

@TheOddball What versions of python/requests/etc are you succeeding with?

GiovanH · 2019-12-09T04:13:33Z

I was able to solve the problem and download successfully by upgrading python to 3.8.0, though I'm not yet sure why this had any effect.

mikf · 2019-12-09T20:11:13Z

As far as I know, Cloudflare uses the parameters during SSL/TLS handshakes to determine whether it's a potential bot or a normal web browser. Python 3.7.4+ for Windows updated its OpenSSL version to 1.1.1, which Cloudflare seems to like and which is why updating to Python 3.8.0 solved the problem. pyOpenSSL also does something that triggers the CAPTCHA and is therefore now disabled by default (15f9bb3).

The CAPTCHA also only triggeres when accessing /api/user/<id>, so I've removed those API calls and use the user info included in the other /api/posts call (c9ef1b2)

To summarize:

Don't use pyOpenSSL
Upgrade to a Python interpreter build with OpenSSL version 1.1.1 (3.7.4+ for Windows and Mac)

… and fall back to the included data if an error occurs.

catlover1019 · 2020-02-02T01:35:59Z

I'm getting this problem on linux (specifically Ubuntu 19.10). The info here doesn't seem to exactly apply

mikf · 2020-02-02T16:02:18Z

What's the OpenSSL version used by your Python interpreter?
python3 -c 'import ssl; print(ssl.OPENSSL_VERSION)'

It works for me with version 1.1.1, but an older Python3.4 linked against OpenSSL 1.0.2 gets a Cloudflare CAPTCHA.

catlover1019 · 2020-02-03T00:29:47Z

adam@adam-ubuntu:~$ gallery-dl https://www.patreon.com/PuppyrooArt/posts --write-metadata [patreon][warning] Cloudflare CAPTCHA [patreon][error] HttpError: '403 Forbidden' for 'https://www.patreon.com/puppyrooart' adam@adam-ubuntu:~$ python3 -c 'import ssl; print(ssl.OPENSSL_VERSION)' OpenSSL 1.1.1c 28 May 2019 adam@adam-ubuntu:~$
Yeah, seems like my experince is different

catlover1019 · 2020-02-07T22:15:26Z

@mikf So, it seems like I have the right OpenSSL version, but it still doesn't work. Any ideas?

mikf · 2020-02-09T21:06:14Z

Maybe your OpenSSL was built with different build flags than the one on my machine and doesn't support the necessary ciphers?

$ openssl version -f
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -Wa,--noexecstack -D_FORTIFY_SOURCE=2 -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DNDEBUG -D_FORTIFY_SOURCE=2

You could try the standalone Linux binary. The libssl.so.1.1 packaged in there is the exact same as the one I'm using, so there is a chance that this might work.

catlover1019 · 2020-02-10T08:12:46Z

Yep, it works, thanks

mikf added a commit that referenced this issue Dec 7, 2019

don't install pyOpenSSL for Travis tests (#508)

5eb89b2

mikf added a commit that referenced this issue Dec 8, 2019

add option to disable pyOpenSSL usage (#508)

15f9bb3

(pyOpenSSL is now disabled by default)

mikf added a commit that referenced this issue Dec 13, 2019

[patreon] make retrieving user info nonfatal (#508)

9e63804

… and fall back to the included data if an error occurs.

mikf closed this as completed Dec 13, 2019

mikf added the cloudflare label Jan 22, 2020

mikf mentioned this issue Jan 22, 2020

Patreon Extractor Gets 403 Forbidden #342

Closed

mikf mentioned this issue Mar 1, 2020

Patreon Cloudfare CAPTCHA - Error 403 #629

Closed

biznizz mentioned this issue Mar 5, 2020

Patreon scrapper issue. #633

Closed

mikf mentioned this issue May 12, 2020

Patreon 403 forbidden #748

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Patreon Cloudflare CAPTCHA #508

Patreon Cloudflare CAPTCHA #508

TheOddball commented Dec 7, 2019

mikf commented Dec 7, 2019

TheOddball commented Dec 8, 2019 •

edited

Loading

GiovanH commented Dec 8, 2019 •

edited

Loading

GiovanH commented Dec 9, 2019 •

edited

Loading

TheOddball commented Dec 9, 2019

GiovanH commented Dec 9, 2019

GiovanH commented Dec 9, 2019

mikf commented Dec 9, 2019

catlover1019 commented Feb 2, 2020

mikf commented Feb 2, 2020

catlover1019 commented Feb 3, 2020

catlover1019 commented Feb 7, 2020

mikf commented Feb 9, 2020

catlover1019 commented Feb 10, 2020

Patreon Cloudflare CAPTCHA #508

Patreon Cloudflare CAPTCHA #508

Comments

TheOddball commented Dec 7, 2019

mikf commented Dec 7, 2019

TheOddball commented Dec 8, 2019 • edited Loading

GiovanH commented Dec 8, 2019 • edited Loading

GiovanH commented Dec 9, 2019 • edited Loading

TheOddball commented Dec 9, 2019

GiovanH commented Dec 9, 2019

GiovanH commented Dec 9, 2019

mikf commented Dec 9, 2019

catlover1019 commented Feb 2, 2020

mikf commented Feb 2, 2020

catlover1019 commented Feb 3, 2020

catlover1019 commented Feb 7, 2020

mikf commented Feb 9, 2020

catlover1019 commented Feb 10, 2020

TheOddball commented Dec 8, 2019 •

edited

Loading

GiovanH commented Dec 8, 2019 •

edited

Loading

GiovanH commented Dec 9, 2019 •

edited

Loading