Deviantart '401 Unauthorized' and switches to low-res "Fallback URL"

[spectrefps]

I am trying to download the full-res of this: (https://www.deviantart.com/aliens-of-star-wars/art/Star-Wars-Lord-Valenthyne-Farfalla-Harpy-warrior-327926322) and it full-size listed is 2480x3508. However, it fails to download it using the page's url even with the referrer (https://www.deviantart.com/) after producing a "401 Unauthorized" warning, and keeps resorting to a "Fallback URL" to download. Unfortunately, this Fallback URL downloads an image that is only 400x566...

I noticed that if I open the image in another tab, it doesn't have the typical "fullview" suffix at the end of the url. Instead of ending in "-fullview", the suffix at the end of the filename/url is "-375w-2x". I think it may be from before the "-fullview" suffix was the standard naming convention for their image URLs. Could that be throwing off the downloader? If so, would there be a way to make it parse that specific image URL so it could download it at it's full size?

UPDATE: Also, which set of cookies are needed in the config file? I see entries for "td", "userinfo", "auth", and "auth_secure", but I'm not sure which one I need to copy into the config file.

[kattjevfel]

isn't this a duplicate of #4548?

[spectrefps]

isn't this a duplicate of #4548?

I'm not sure I understand how they fixed it in that linked post. I saw they said that a"signed ALG=HS256" works, in their opening post? I'm afraid I don't have any idea what that means. Further down they talk about it not working for some since recently, so I'm confused. Was the problem 'solved' by the "ALG=HS256" bit up top, or do they mean it is still being investigated?

[kattjevfel]

It isn't solved, mikf knows about it and there doesn't seem to be a fix for now.

[spectrefps]

ah ok. As an update to this, I am able to do some crude altering of the URL (the "wixmp" url of the image, after I right-click on it and 'open in new tab') to grab it at the listed 'full' resolution. However, unlike when using gallery-dl, grabbing it this way doesn't match the filesize listed on the art's page, so I think that may not be the 'true' original image.

Could the format of the image's URL (specifically, the "-375w-2x" in place of the "-fullview") be causing problems? Also, for deviantart, which cookie should we have in our config file? I see a few different ones listed ("td", "userinfo", "auth", and "auth_secure"), and there was a similar question from May that didn't have an answer, so I'm not sure which is the correct one.

[Ironchest337]

Unfortunately with the way Deviantart is set up, simple link manipulation is not a viable strategy. At the end of all direct image links is a pesky "token" value. To keep things simple the token value tells you if you can access what you want to access. Reading and modifying the token to access the full resolution has never been the issue. The problem is that every token has a password built in and unless you know the password, your modified tokens will not work. Before we used something called a "none" algorithm to tell website the token never had a password to begin with, tricking it into just giving us what we needed and avoiding the entire password problem. Now it denies tokens created using said algorithm and we're back to square one.

[spectrefps]

Oh crap, so Deviantart might have just broken a part of gallery-dl's utility/use by changing something on their end recently?

[Ironchest337]

I assume so. This particular vulnerability only requires a few simple lines of code to fix in the first place. I think the bigger issue that prompted the change though is that this code actually allowed you to bypass certain items that were locked behind paid tiers or paid gallery access.

[oxi7589]

It's understandable that this JWT hack no longer works, but I'd expect the fallback to be the official free download (where available) and not the lowres "preview" variant. Currently the lowres preview is used even for images that can be downloaded in their original form using the "Free download" button on the site.

[mikf]

Could you post an example for were this happens?
I tried a few from DA's front page and they all download at the same resolution that you'd get with a browser. When there is a "download" button, gallery-dl will be using that. Or did you perhaps disable the original option?

[cywondering]

Could you post an example for were this happens? I tried a few from DA's front page and they all download at the same resolution that you'd get with a browser. When there is a "download" button, gallery-dl will be using that. Or did you perhaps disable the original option?

i also met this problem, in my account, i can download the orginal picture with clicking "free download" buttom below the picture on the website, but also met 401 Unauthorized problem in our gallery-dl, which can't download it in full-size. (with original option set with true)

example:
`PS gallery-dl -c ".......\gallery-dl\config.json" "https://www.deviantart.com/vdfvdvd/gallery"
[deviantart][info] Requesting public access token
[deviantart][info] Collecting folder information for 'vdfvdvd'
[downloader.http][warning] '401 Unauthorized' for 'https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/904e813e-4288-4401-8341-46cacc08f4c7/dg8kbhs-3d7735f7-086b-49ca-ab4d-6e8192ebfbab.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJzdWIiOiJ1cm46YXBwOiIsImlzcyI6InVybjphcHA6Iiwib2JqIjpbW3sicGF0aCI6Ii9mLzkwNGU4MTNlLTQyODgtNDQwMS04MzQxLTQ2Y2FjYzA4ZjRjNy9kZzhrYmhzLTNkNzczNWY3LTA4NmItNDljYS1hYjRkLTZlODE5MmViZmJhYi5wbmcifV1dLCJhdWQiOlsidXJuOnNlcnZpY2U6ZmlsZS5kb3dubG9hZCJdfQ.'
[download][info] Trying fallback URL #1

• H:\deviantart\vdfvdvd\deviantart_981843760_Dangerous Gambling [TG TF].jpg
  [downloader.http][warning] '401 Unauthorized' for 'https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/904e813e-4288-4401-8341-46cacc08f4c7/dg7oxte-366fd10a-1c91-40f6-9d4d-6867c7e1716a.png?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJzdWIiOiJ1cm46YXBwOiIsImlzcyI6InVybjphcHA6Iiwib2JqIjpbW3sicGF0aCI6Ii9mLzkwNGU4MTNlLTQyODgtNDQwMS04MzQxLTQ2Y2FjYzA4ZjRjNy9kZzdveHRlLTM2NmZkMTBhLTFjOTEtNDBmNi05ZDRkLTY4NjdjN2UxNzE2YS5wbmcifV1dLCJhdWQiOlsidXJuOnNlcnZpY2U6ZmlsZS5kb3dubG9hZCJdfQ.'`

[mikf]

Thanks for the example, I think I see were the problem is.

TLDR: Use -o public=0 with a refresh-token, but that'll cause lots of 429 errors after a while.

So I said "When there is a "download" button, gallery-dl will be using that", and that's still true, but for NSFW / mature-rated works there is no longer a download button unless one is logged in / is using a private access token via refresh token.

There is no longer any indicator that a "is_mature": true post is potentially "downloadable" or has a higher resolution version available, kind of like PDF downloads for which there's also no solution with the official API.

Always using private tokens is not an option as these have a very low rate limit and will cause 429 Too Many Requests errors after using them for a very short time, which makes mass downloading basically impossible.

I guess we could repeat API calls that fetch a bulk of posts when at least one of them has a "is_mature": true field, but that is still going to cause 429 errors when downloading from NSFW accounts.

[spectrefps]

Could you post an example for were this happens? I tried a few from DA's front page and they all download at the same resolution that you'd get with a browser. When there is a "download" button, gallery-dl will be using that. Or did you perhaps disable the original option?

I left the 'original' option as-is. This is the link in question from my first post that currently doesn't grab the full-sized image (2480x3508 and 997.66 KB per the image details below the description): https://www.deviantart.com/aliens-of-star-wars/art/Star-Wars-Lord-Valenthyne-Farfalla-Harpy-warrior-327926322

Also, there doesn't seem to be a download button for this art (quite a few arts on DA nowadays seem to lack that button, even if they aren't NSFW or private. Luckily gallery-dl was able to download them in full size/resolution matching the listed original image dimensions/filesize. This one in particular is proving tricky.

[spectrefps]

Thanks for the example, I think I see were the problem is.

TLDR: Use -o public=0 with a refresh-token, but that'll cause lots of 429 errors after a while.

So I said "When there is a "download" button, gallery-dl will be using that", and that's still true, but for NSFW / mature-rated works there is no longer a download button unless one is logged in / is using a private access token via refresh token.

There is no longer any indicator that a "is_mature": true post is potentially "downloadable" or has a higher resolution version available, kind of like PDF downloads for which there's also no solution with the official API.

Always using private tokens is not an option as these have a very low rate limit and will cause 429 Too Many Requests errors after using them for a very short time, which makes mass downloading basically impossible.

I guess we could repeat API calls that fetch a bulk of posts when at least one of them has a "is_mature": true field, but that is still going to cause 429 errors when downloading from NSFW accounts.

Also, I don't believe this link in question is NSFW or Mature rated either. I couldn't find anything on the page that it had any mature/NSFW rating or tag. I am logged in as well when I attempt to use gallery-dl.

Hmm, how would I implement this for this URL? Would it be:
gallery-dl -o public=0 "https://www.deviantart.com/aliens-of-star-wars/art/Star-Wars-Lord-Valenthyne-Farfalla-Harpy-warrior-327926322" ?