[Newgrounds] Adult works now behind log in requirement, cannot be downloaded without Login credentials

[biznizz]

As the title describes, it seems that Newgrounds has had a big site change, meaning that one will have to have an account and log in credentials to view A (Adult) pictures. Here are two examples, one M and one A from the same artist, without using a username and password.

https://www.newgrounds.com/art/view/mrvalentinex/do-it-inside-job-1 = Downloads with no problem

https://www.newgrounds.com/art/view/mrvalentinex/do-it-inside-job-2 - program returned with this error

←[1;33mnewgrounds: Unable to get download URL for 'https://www.newgrounds.com/art/view/mrvalentinex/do-it-inside-job-2'←[0m
←[1;37mnewgrounds: No results for https://www.newgrounds.com/art/view/mrvalentinex/do-it-inside-job-2←[0m

Highly recommend updating the Newgrounds info on the Supported Sites with Login Required to get all Adult works.

[mikf]

This site exists since 1995 and now they decide that it is necessary to require an account to view "adult"-rated content ... same with twitter, which is also nearly two decades old. Just a coincidence, I'm sure.
Want to bet which other sites will apply similar restrictions in the coming months?

Anyway, there's now a warning when posts cannot be downloaded because of this: a53cfc8.

[God-damnit-all]

I suggest that if gallery-dl detects that the user is logged in, it outputs the failed URL and tells you to navigate to it to input your birthdate. If the prompt is interactive, it could also pause and wait for the user to do this, and then retry when Enter is pressed.

I also suggest that if the prompt is not interactive, this should be an extractor-aborting error unless a new configuration option is toggled.

[GiovanH]

The actual image urls (in this case, https://art.ngfiles.com/images/2133000/2133630_mrvalentinex_do-it-inside-job-2.png) are not behind a login, only the portal page. gallery-dl should compute the url and use it directly.

[God-damnit-all]

The actual image urls (in this case, https://art.ngfiles.com/images/2133000/2133630_mrvalentinex_do-it-inside-job-2.png) are not behind a login, only the portal page. gallery-dl should compute the url and use it directly.

The descriptions are behind a login though, which often contain extra embeds.

[AlttiRi]

Also a hCaptcha page was added.

In this case gallery-dl shows:

[newgrounds][info] No results for ...

with -v you can see 503 error:

[newgrounds][debug] '503 Service Unavailable' for ...

Use NGBrowserValidation cookie. However, it expires fast.

[biznizz]

Can confirm that you need that captcha cookie. And yes, it expires after what looks like only 8 hours. I've moved all my Newgrounds cookies to my cookies.txt file

Here's my current extractor settings in config.

"newgrounds":
    {
		"cookies": "C:\\Users\\USER\\cookies.txt",
		"cookies-update": true,
        "flash": true,
        "include": "art"
    },

After some tests, cookies-update will be able to properly update the NGBrowserValidation cookie so it can be a set it and forget it, but if it ever does fail, I'll update you all.

[God-damnit-all]

The captcha isn't always active, just when the site is under high load. It's been that way for a while.

[biznizz]

Welp, it seems like when the captcha cookie expires, not even cookies-update is able to make a new valid one, so you can't set it and forget it for whenever the site is under load.

So it's something you'll have to export once in a while.

[biznizz]

I'm on the latest dev build and something's happened that's been giving me these errors.

C:\Users\USER>gallery-dl https://duskcy.newgrounds.com/art --verbose
←[0;37mgallery-dl: Version 1.21.1-dev←[0m
←[0;37mgallery-dl: Python 3.10.4 - Windows-10-10.0.19044-SP0←[0m
←[0;37mgallery-dl: requests 2.27.1 - urllib3 1.26.9←[0m
←[0;37mgallery-dl: Starting DownloadJob for 'https://duskcy.newgrounds.com/art'←[0m
←[1;33mnewgrounds: cookies: too many values to unpack (expected 7)←[0m
←[0;37mnewgrounds: Using NewgroundsArtExtractor for 'https://duskcy.newgrounds.com/art'←[0m
←[0;37murllib3.connectionpool: Starting new HTTPS connection (1): duskcy.newgrounds.com:443←[0m
←[0;37murllib3.connectionpool: https://duskcy.newgrounds.com:443 "GET /art/page/1 HTTP/1.1" 200 None←[0m
←[1;31mnewgrounds: An unexpected error occurred: KeyError - 'sequence'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .←[0m
←[0;37mnewgrounds: ←[0m
Traceback (most recent call last):
  File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\site-packages\gallery_dl\job.py", line 76, in run
    for msg in extractor:
  File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\site-packages\gallery_dl\extractor\newgrounds.py", line 43, in items
    for post_url in self.posts():
  File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\site-packages\gallery_dl\extractor\newgrounds.py", line 259, in _pagination
    for year in data["sequence"]:
KeyError: 'sequence'

[mikf]

@biznizz fixed in c176897

And at least at the moment, I can download adult stuff without login and without NGBrowserValidation cookie.

[biggestsonicfan]

I think more newgrounds shenanigans have been added, such as captcha.