Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Cloudflare Challenge Page on Pixiv Fanbox #259

Closed
gentle-hen opened this issue Jul 1, 2024 · 11 comments
Closed

[Bug]: Cloudflare Challenge Page on Pixiv Fanbox #259

gentle-hen opened this issue Jul 1, 2024 · 11 comments
Assignees
@KJHJason
Labels
bug Something isn't working

Comments

@gentle-hen
Copy link

gentle-hen commented Jul 1, 2024 •
edited by KJHJason
Loading

Which program are you seeing this problem on?

Cultured Downloader (Go)

Program Version

5.1.0

Operating System

Windows 10

What error code did you receive? (If any)

1003 (Input Error)

Steps to reproduce the bug

  1. Go to "Sessions"
  2. Enter in my fanboxsessid
  3. "Error 1003: Pixiv Fanbox cookie is invalid" appears after clicking "Save"

Expected behaviour

I expected my fanboxsessid to work as I can see the paid content on chrome without problem.

Relevant log output (If any)

No response

Screenshots (Optional)

No response
@gentle-hen gentle-hen added the bug Something isn't working label Jul 1, 2024
@KJHJason
Copy link
Owner

KJHJason commented Jul 1, 2024

Looks like Pixiv Fanbox has enabled Cloudflare under attack mode.

Will take more time to test if using chromedp or go-rod will work...

@KJHJason
Copy link
Owner

KJHJason commented Jul 1, 2024 •
edited
Loading

You can refer to the workarounds found in Nandaka/PixivUtil2#1342 such as using a VPN to JP which should address the Cloudflare challenge page issue for now.

Not sure if this is temporary or permanent but probably be testing chromedp if it goes through the challenge page. Otherwise, I will have to add new inputs for Pixiv Fanbox for the Cloudflare clearance cookie.

However, with the hacky Cloudflare clearance cookie workaround, the __cf_bm cookie has a max-age of 15 minutes and is expected to only last for an hour so it's not an ideal UX for the user to re-enter the cookie value every hour.

@KJHJason
Copy link
Owner

KJHJason commented Jul 1, 2024 •
edited
Loading

Update: using go-rod with stealth works, but it uses the leakless binary to kill the automated browser which is commonly used by malware and gets wrongly identified by AV like bitdefender.

Edit: nvm, not sure what happened but now the challenge page is popping up.

@KJHJason
Copy link
Owner

KJHJason commented Jul 1, 2024 •
edited
Loading

However, with the hacky Cloudflare clearance cookie workaround, the __cf_bm cookie has a max-age of 15 minutes and is expected to only last for an hour so it's not an ideal UX for the user to re-enter the cookie value every hour.

How do you even do this workaround? I apparently couldn't properly parse the technical lingo used by the ones who discussed it in that thread, because my attempts to implement it aren't working. I even downloaded a zip of the master version and unpacked it.

Hello @MarqFJA87, I have not used PixivUtils2 but you need the cf_clearance and __cf_bm cookies that you can obtain from your browser. You can refer to Nandaka/PixivUtil2#1342 (comment).

Edit: Looks like you deleted your original comment

@KJHJason
Copy link
Owner

KJHJason commented Jul 1, 2024 •
edited
Loading

Update: using go-rod with stealth works, but it uses the leakless binary to kill the automated browser which is commonly used by malware and gets wrongly identified by AV like bitdefender.

Edit: nvm, not sure what happened but now the challenge page is popping up.

Managed to get it working via KJHJason/Cultured-Downloader-Logic@6e1997b though I'm not sure if it's consistent...

More testings will be required.

@MarqFJA87
Copy link

Hello @MarqFJA87, I have not used PixivUtils2 but you need the cf_clearance and __cf_bm cookies that you can obtain from your browser. You can refer to Nandaka/PixivUtil2#1342 (comment).

Edit: Looks like you deleted your original comment

Yeah, I just realized that the link I had clicked had taken me outside the PixivUtil2 repository and into another one.

And I did manage to find those cookies, but apparently I need something else to get it working, because I still keep getting the same error. Changing my user-agent just runs into an older issue with Pixiv that causes a login failure, which others have reported while trying the suggested fixes.

@gentle-hen
Copy link
Author

Update: using go-rod with stealth works, but it uses the leakless binary to kill the automated browser which is commonly used by malware and gets wrongly identified by AV like bitdefender.
Edit: nvm, not sure what happened but now the challenge page is popping up.

Managed to get it working via KJHJason/Cultured-Downloader-Logic@6e1997b though I'm not sure if it's consistent...

More testings will be required.

I'm sorry, but how do I use KJHJason/Cultured-Downloader-Logic@6e1997b to get it working? By the way, I tried using a VPN to switch to JP but did not work. Also, I forgot to ask in the initial post, is the executable version of your program the Python version?

Thank you.

@KJHJason
Copy link
Owner

KJHJason commented Jul 2, 2024 •
edited
Loading

Version 5 and above uses Go now with a GUI.

If you're using a CLI based program, it's the old Python program tho has been reported to no longer work due to webdriver manager.

With regards to VPN workaround, I will try test on my end first. Otherwise I will speed up the development of V5.2.0 👍

@KJHJason
Copy link
Owner

KJHJason commented Jul 7, 2024

Hello @gentle-hen, it seems like Pixiv Fanbox has relaxed their Challenge Page security level.
Hence, you should be able to set your Fanbox Session ID.

However, I will still work on implementing a Cloudflare solver into the program in the event this happens again 👍

@gentle-hen
Copy link
Author

Hello @gentle-hen, it seems like Pixiv Fanbox has relaxed their Challenge Page security level. Hence, you should be able to set your Fanbox Session ID.

However, I will still work on implementing a Cloudflare solver into the program in the event this happens again 👍

That's great to hear! Do you want me to close this, or keep it open?

Thank you for all the work!

@KJHJason
Copy link
Owner

KJHJason commented Jul 8, 2024

Just leave it open, thanks!

@KJHJason KJHJason changed the title [Bug]: "Error 1003: Pixiv Fanbox cookie is invalid" when using working fanbox sessid [Bug]: Cloudflare Challenge Page on Pixiv Fanbox Jul 8, 2024
@KJHJason KJHJason mentioned this issue Jul 18, 2024
Open
KJHJason added a commit to KJHJason/Cultured-Downloader-Logic that referenced this issue Jul 18, 2024
@KJHJason
should address KJHJason/Cultured-Downloader#259
3a02fa8
KJHJason added a commit to KJHJason/Cultured-Downloader-Logic that referenced this issue Jul 19, 2024
@KJHJason
Captcha, Metadata, Filters (#68)
108e747 
* prep for code cleanups and adding filters

* add progress to based;

* rename var

* filters mainly done for Fantia, Pixiv Fanbox, and Kemono

* add filter to kemono file field

* fixed bugs, optimise exec look ups, and adding metadata support

* metadata support wip

* CF testing

- somehow manage to get cf cookies no clue how it worked

* fix negative values for dl details

* cf test

* oops forgor to remove github copilot comment

* cf logic works but not bypassing it, might need to use drission on python instead

* gofmt

* update ua

* Integrating Python-based solution into Go

- Finished and tested the Python-based solution that uses DrissionPage mainly thanks to https://github.com/sarperavci/CloudflareBypassForScraping
- Left integeration with Go

* cf wip

* integrated cf python scripts into go

* remove line

* fix bug for unix when init py and wrapping xvfb wip

* fixing

* add concurrency limit to api wrapper

* dockler container still wip (BrowserConnectError)

* fixing

* changed location of failed bypass logging

* add orjson

* wrap cf cli in docker for xvfb support

* fix error

* improve UX

* fix

* touch up on the python codes

* touch up on docker

* fix bug

* improve readability for generating random time

* edit comment

* Change os.environ.get to os.getenv

* integrated InitFiles, left with pulling docker container

* gofmt

* fmt

* edit test commands

* touch up on python code

* touchup on python codes

* edit comments

* fix bug where __remove_navigator_js might be called twice

* python local done, docker wip

* remove unnecessary argument for for loop scoping which has been fixed in go 1.22

* switch buffered channels for thread-safe slices

* gofmt

* docker wip and improved python codes

* add pyautogui

* fix bug and removed pyautogui which didn't work in docker

* docker wip

* integrated docker with go

* captcha integration wip

* should address KJHJason/Cultured-Downloader#259

* touch on captcha handler and metadata re-work wip

* metadata completed

* fix syntax errors

* integrated BaseDl into Pixiv, filters for Pixiv wip

* remove cf py

* integrated filters and captcha for pixiv

* update .gitignore

* fixed bugs, update depends, improved cf

* fix potential bug with ugoira for filters
This was referenced Aug 8, 2024
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@KJHJason KJHJason

Labels
bug Something isn't working
Projects
Cultured Downloader
Status: Done
Milestone
No milestone
Development

No branches or pull requests
3 participants
@MarqFJA87 @KJHJason @gentle-hen