[Patreon] Problem with Cloudflare #6140

NimbusCloudVoid · 2024-09-05T02:38:58Z

I have a problem with Patreon for a while now, the Cloudflare problem and i can't download anything.. I looked at other issues for solution like TLS, Ciphers, browser and header and none of them works. I use cookies and the cf clearence is in the cookies txt, i'm on the latest version of gallery-dl and there is nothing that works. I'm on VPN and tried multiple servers and none of them works and for some reason when i tried another program to download patreon stuff with cookies it works.

here is my usual config:

		"patreon": {
			"cookies": {
				"session_id": "****",
				"path": "C:\\Users\\* * * *\\gallery-dl\\Cookies\\cookies-patreon-com.txt",
				"user-agent": "Mozilla/5.0 (Windows NT 10.0; rv:129.0) Gecko/20100101 Firefox/129.0"

Hrxn · 2024-09-05T18:55:43Z

[..] I'm on VPN and tried multiple servers [..]

This might be the reason. IP ranges known for VPN usages or similar (VPS, whatever) are blocked entirely.

NimbusCloudVoid · 2024-09-05T21:12:34Z

Even after disabling the VPN it still doesn't works, i've even tried this program https://github.com/AlexCSDev/PatreonDownloader and while its scuffed i've managed to download the stuff i wanted, that's why its weird

mikf · 2024-09-06T12:07:10Z

This is not how cookies are supposed to be specified. Try these settings:

    "patreon": {
        "cookies": "C:\\Users\\* * * *\\gallery-dl\\Cookies\\cookies-patreon-com.txt",
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; rv:129.0) Gecko/20100101 Firefox/129.0",
        "browser": false
    }

NimbusCloudVoid · 2024-09-06T15:28:18Z

So, i've put your changes and i still get the cloudflare problem, i've also use the verbose option here:

		"patreon": 
		{
			"cookies": "C:\\Users\\****\\gallery-dl\\Cookies\\cookies-patreon-com.txt",
			"user-agent": "Mozilla/5.0 (Windows NT 10.0; rv:129.0) Gecko/20100101 Firefox/129.0",
			"browser": false
		},

C:\Users\ REDACTED>gallery-dl https://www.patreon.com/ REDACTED/posts --verbose
[gallery-dl][debug] Version 1.27.4
[gallery-dl][debug] Python 3.11.7 - Windows-10-10.0.19045-SP0
[gallery-dl][debug] requests 2.32.3 - urllib3 2.2.2
[gallery-dl][debug] Configuration Files ['%APPDATA%\gallery-dl\config.json']
[gallery-dl][debug] Starting DownloadJob for 'https://www.patreon.com/ REDACTED/posts'
[patreon][debug] Using PatreonCreatorExtractor for 'https://www.patreon.com/ REDACTED/posts'
[patreon][debug] TLS 1.2 disabled.
[patreon][debug] Loading cookies from 'C:\Users\ REDACTED\gallery-dl\Cookies\cookies-patreon-com.txt'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.patreon.com:443
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /****/posts HTTP/11" 403 None
[patreon][warning] Cloudflare challenge
[patreon][error] HttpError: '403 Forbidden' for 'https://www.patreon.com/ REDACTED/posts'

mikf · 2024-10-01T07:02:45Z

Could you try it with the Patreon mobile UA? #6241 (comment)

        "patreon": 
        {
            "cookies": "C:\\Users\\****\\gallery-dl\\Cookies\\cookies-patreon-com.txt",
            "user-agent": "Patreon/72.2.28 (Android; Android 14; Scale/2.10)",
            "browser": false
        },

NimbusCloudVoid · 2024-10-01T09:20:24Z

I tried it but for some reason i still get the Cloudflare error..

NimbusCloudVoid · 2024-10-01T19:18:01Z

I put the verbose result just in case but i seriously don't get it.. I put what is written but i still get the cloudflare error

NimbusCloudVoid · 2024-10-03T10:25:48Z

Update: if i use gallery-dl patreon.com/home it works i can download from Patreon but if i use gallery-dl https://www.patreon.com/(creator's name) or gallery-dl https://www.patreon.com/(creator's name)/posts i get the cloudflare problem.

All of these is weird, so my problem is that i can't download a specific creator due to cloudflare problem but i can download if i use gallery-dl patreon.com/home which i assume download everything of all your creators that your are subbed but the problem with that is that it's everyone which is annoying since he has to download everything (ofc it will skip the ones you've already downloaded but it still send the request to download it) and not just a specific creator.

I will post a new comment to notify if there was any problem with the gallery-dl patreon.com/home but as of now my problem with cloudflare seems to be "fixed", i hope.

NimbusCloudVoid · 2024-10-04T16:38:22Z

After using the command everything was downloaded, same with the attachements

prowlguru · 2024-10-12T17:15:21Z

@mikf I seem to have an issue with downloading Patreon videos. Images and other attachments seem to work.
This seems to happen when I use the direct url to a post and probably also with the creator's url, as in #6140 (comment) (but I can download posts without videos just fine using those urls). (I don't want to try downloading with patreon.com/home)

I've also tried using the mobile user agent and the browser's user agent. Downloading these videos worked quite well last month, I think. Is it some issue with yt_dlp or how the cookies are passed to it?

Traceback (most recent call last):
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/extractor/common.py", line 896, in _request_webpage
    return self._downloader.urlopen(self._create_request(url_or_request, data, headers, query, extensions))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/YoutubeDL.py", line 4165, in urlopen
    return self._request_director.send(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/networking/common.py", line 117, in send
    response = handler.send(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/networking/_helper.py", line 208, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/networking/common.py", line 340, in send
    return self._send(request)
           ^^^^^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/networking/_requests.py", line 365, in _send
    raise HTTPError(res, redirect_loop=max_redirects_exceeded)
yt_dlp.networking.exceptions.HTTPError: HTTP Error 401: Unauthorized

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/YoutubeDL.py", line 1626, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/YoutubeDL.py", line 1761, in __extract_info
    ie_result = ie.extract(url)
                ^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/extractor/common.py", line 740, in extract
    ie_result = self._real_extract(url)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/extractor/patreon.py", line 496, in _real_extract
    webpage = self._download_webpage(url, vanity, headers={'User-Agent': self.USER_AGENT})
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/extractor/common.py", line 1188, in _download_webpage
    return self.__download_webpage(url_or_request, video_id, note, errnote, None, fatal, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/extractor/common.py", line 1139, in download_content
    res = getattr(self, download_handle.__name__)(url_or_request, video_id, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/extractor/common.py", line 960, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/extractor/common.py", line 909, in _request_webpage
    raise ExtractorError(errmsg, cause=err)
yt_dlp.utils.ExtractorError: [PatreonCampaign] Unable to download webpage: HTTP Error 401: Unauthorized (caused by <HTTPError 401: Unauthorized>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "REDACTED/3.12.6/lib/python3.12/site-packages/gallery_dl/downloader/ytdl.py", line 66, in download
    info_dict = ytdl_instance.extract_info(url[5:], download=False)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/YoutubeDL.py", line 1615, in extract_info
    return self.__extract_info(url, self.get_info_extractor(key), download, extra_info, process)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/YoutubeDL.py", line 1644, in wrapper
    self.report_error(str(e), e.format_traceback())
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/YoutubeDL.py", line 1092, in report_error
    self.trouble(f'{self._format_err("ERROR:", self.Styles.ERROR)} {message}', *args, **kwargs)
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/YoutubeDL.py", line 1031, in trouble
    raise DownloadError(message, exc_info)
yt_dlp.utils.DownloadError: ERROR: [PatreonCampaign] Unable to download webpage: HTTP Error 401: Unauthorized (caused by <HTTPError 401: Unauthorized>)

mikf · 2024-10-12T18:06:10Z

@prowlguru Try enabling downloader.ytdl.forward-cookies and see if that helps.

prowlguru · 2024-10-12T19:25:39Z

@prowlguru Try enabling downloader.ytdl.forward-cookies and see if that helps.

That's interesting, I still get an error, but not an HTTP error. Might be a yt-dlp issue. Do you (or anyone else) have this issue?

Traceback (most recent call last):
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/YoutubeDL.py", line 1626, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/YoutubeDL.py", line 1761, in __extract_info
    ie_result = ie.extract(url)
                ^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/extractor/common.py", line 741, in extract
    ie_result = self._real_extract(url)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/extractor/patreon.py", line 530, in _real_extract
    campaign_id = self._search_nextjs_data(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/extractor/common.py", line 1772, in _search_nextjs_data
    return self._search_json(
           ^^^^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/extractor/common.py", line 1359, in _search_json
    json_string = self._search_regex(
                  ^^^^^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/extractor/common.py", line 1345, in _search_regex
    raise RegexNotFoundError(f'Unable to extract {_name}')
yt_dlp.utils.RegexNotFoundError: [PatreonCampaign] Unable to extract next.js data; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "REDACTED/3.12.6/lib/python3.12/site-packages/gallery_dl/downloader/ytdl.py", line 66, in download
    info_dict = ytdl_instance.extract_info(url[5:], download=False)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/YoutubeDL.py", line 1615, in extract_info
    return self.__extract_info(url, self.get_info_extractor(key), download, extra_info, process)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/YoutubeDL.py", line 1644, in wrapper
    self.report_error(str(e), e.format_traceback())
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/YoutubeDL.py", line 1092, in report_error
    self.trouble(f'{self._format_err("ERROR:", self.Styles.ERROR)} {message}', *args, **kwargs)
  File "REDACTED/3.12.6/lib/python3.12/site-packages/yt_dlp/YoutubeDL.py", line 1031, in trouble
    raise DownloadError(message, exc_info)
yt_dlp.utils.DownloadError: ERROR: [PatreonCampaign] Unable to extract next.js data; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U

yt-dlp -U
Latest version: [email protected] from yt-dlp/yt-dlp
yt-dlp is up to date ([email protected] from yt-dlp/yt-dlp)

prowlguru · 2024-10-12T20:05:03Z

I'm not entirely sure what url gallery-dl uses to download videos with yt-dlp, but I guess it is something like https://www.patreon.com/api/video/<id>/video.m3u8. I tried downloading that with yt-dlp and got the same error as with gallery-dl. However, when I simply supply the regular patreon post url to yt-dlp, it works just fine: https://www.patreon.com/posts/<post>.

prowlguru · 2024-10-12T21:25:29Z

It also seems to work if I don't use any configuration at all (example with public post):
gallery-dl -v https://www.patreon.com/posts/florida-man-1-31198726

Not sure what the default config is for using ytdl, if it uses some other url with ytdl? I've confirmed that https://www.patreon.com/api/video/<id>/video.m3u8 doesn't work with yt-dlp.

Note that this only works if you don't get 401 unauthorized and if you need to pass cookies with downloader.ytdl.forward-cookies it results in the same error.

prowlguru · 2024-10-13T00:37:26Z

OK, the issue has been fixed with yt-dlp/yt-dlp#11235

mikf added the cloudflare label Sep 6, 2024

mikf added a commit that referenced this issue Oct 1, 2024

[patreon] use mobile UA (#6241, #6239, #6140)

321161c

mikf closed this as completed Oct 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Patreon] Problem with Cloudflare #6140

[Patreon] Problem with Cloudflare #6140

NimbusCloudVoid commented Sep 5, 2024 •

edited by mikf

Loading

Hrxn commented Sep 5, 2024

NimbusCloudVoid commented Sep 5, 2024

mikf commented Sep 6, 2024

NimbusCloudVoid commented Sep 6, 2024 •

edited

Loading

mikf commented Oct 1, 2024

NimbusCloudVoid commented Oct 1, 2024

NimbusCloudVoid commented Oct 1, 2024

NimbusCloudVoid commented Oct 3, 2024 •

edited

Loading

NimbusCloudVoid commented Oct 4, 2024

prowlguru commented Oct 12, 2024 •

edited

Loading

mikf commented Oct 12, 2024

prowlguru commented Oct 12, 2024

prowlguru commented Oct 12, 2024 •

edited

Loading

prowlguru commented Oct 12, 2024

prowlguru commented Oct 13, 2024

[Patreon] Problem with Cloudflare #6140

[Patreon] Problem with Cloudflare #6140

Comments

NimbusCloudVoid commented Sep 5, 2024 • edited by mikf Loading

Hrxn commented Sep 5, 2024

NimbusCloudVoid commented Sep 5, 2024

mikf commented Sep 6, 2024

NimbusCloudVoid commented Sep 6, 2024 • edited Loading

mikf commented Oct 1, 2024

NimbusCloudVoid commented Oct 1, 2024

NimbusCloudVoid commented Oct 1, 2024

NimbusCloudVoid commented Oct 3, 2024 • edited Loading

NimbusCloudVoid commented Oct 4, 2024

prowlguru commented Oct 12, 2024 • edited Loading

mikf commented Oct 12, 2024

prowlguru commented Oct 12, 2024

prowlguru commented Oct 12, 2024 • edited Loading

prowlguru commented Oct 12, 2024

prowlguru commented Oct 13, 2024

NimbusCloudVoid commented Sep 5, 2024 •

edited by mikf

Loading

NimbusCloudVoid commented Sep 6, 2024 •

edited

Loading

NimbusCloudVoid commented Oct 3, 2024 •

edited

Loading

prowlguru commented Oct 12, 2024 •

edited

Loading

prowlguru commented Oct 12, 2024 •

edited

Loading