[deviantart] Deviantart images getting downloaded with a blur filter #4567

n00bmister69 · 2023-09-22T11:00:32Z

For the past two days, whenever I'm trying to download some images from Gallery-DL, it's getting downloaded with a blur filter. (I tried many things for these 2 days but nothing is working, and Gallery-DL was working fine for all images before 2 days, someone pls help, I'm kinda a coding noob myself). Now giving this error:

[downloader.http][warning] '401 Unauthorized' for

ClosedPort22 · 2023-09-22T12:13:41Z

it's getting downloaded with a blur filter.

Can't reproduce when not logged in. Make sure mature is set to true. This could also be due to your account's mature content setting

[downloader.http][warning] '401 Unauthorized' for

Duplicate of #4548

n00bmister69 · 2023-09-22T12:31:18Z

Can't reproduce when not logged in. Make sure mature is set to true. This could also be due to your account's mature content setting

Thank you for your reply! I was already logged in and mature content was set to true, but still the images are downloaded with a blur filter.

Ahmed0425 · 2023-09-23T00:21:51Z

Can't reproduce when not logged in. Make sure mature is set to true. This could also be due to your account's mature content setting

Thank you for your reply! I was already logged in and mature content was set to true, but still the images are downloaded with a blur filter.

You're trying to download a paid subscription image while you didn't subscribe right? this worked prior to 19/9/2023 and is a duplicate to the issue mentioned above

n00bmister69 · 2023-09-23T05:16:23Z

is a duplicate to the issue mentioned above

Oh ok!! Thanks for your reply! I'm not well versed with coding language and thus can't understand stuffs like tokens, etc. But did they kinda find any workaround? Someone wrote something involving "Hydrus". Can't understand that either.. Thank you for your time :)

Twi-Hard · 2023-09-23T06:28:10Z

Hydrus is a program to download/manage/view media. There is no a workaround currently and I really doubt there will be. The new commit seems to just add an option to skip trying to do the now-broken exploit that let us get full resolution images.

rewrib · 2023-09-23T23:07:25Z

You can try to build gallery-dl from source directly (or wait until the next release), the blur problem has been already fixed. Just make sure to set the "public" option in your deviantart config as false. I am not sure, if that fixes the "full resolution" issue though.

rautamiekka · 2023-09-24T01:43:20Z

It's Python, you don't need to build anything if you don't absolutely wanna have a single file representing the app.

left1000 · 2023-09-24T11:17:14Z

Can't reproduce when not logged in. Make sure mature is set to true. This could also be due to your account's mature content setting

Thank you for your reply! I was already logged in and mature content was set to true, but still the images are downloaded with a blur filter.

You're trying to download a paid subscription image while you didn't subscribe right? this worked prior to 19/9/2023 and is a duplicate to the issue mentioned above

To clarify this simply doesn't work at all anymore and there's nothing I can do, nor gallery-dl can change to fix this exploit? That seems to be what you're implying, but I'm not 100% sure since I can't find it explained in the other issue.

ClosedPort22 · 2023-09-24T11:18:58Z

To clarify this simply doesn't work at all anymore and there's nothing I can do, nor gallery-dl can change to fix this exploit?

Correct.

left1000 · 2023-09-25T15:09:40Z

Is there a way to use a command line flag to disable the downloading of the blurred images? I've just now realized I'm going to soon end up with 100s or 1000s of worthless blurry images :(

Progman2002 · 2023-09-29T09:53:23Z

I have the same issue and wrote a little hack in the extractor/deviantart.py file. In the DeviantartExtractor class inside the items() method (around line 93), there is a for deviation in self.deviations(): loop. I added the following check at the beginning:

    for deviation in self.deviations():
        if deviation["content"] and deviation["content"]["src"] and "blur_" in deviation["content"]["src"]:
            self.log.info("URL contains blur_ token, skip entry")
            continue
        if isinstance(deviation, tuple):
            url, data = deviation
            [...]

This will look for a string "blur_" inside the download URL (after the fallback URL is resolved), which indicates somehow that the image is blurry, so it skips it. This is obviously a temporary dirty workaround/hack until a real proper solution is found.

ClosedPort22 · 2023-09-29T10:50:48Z

I have the same issue and wrote a little hack in the extractor/deviantart.py file.

Can you post an example? I haven't seen blurred images outside of premium-only deviations, and gallery-dl will not download these if no refresh-token is configured, even though it should be possible to rewrite the JWT without fetching premium data at all.

Progman2002 · 2023-09-29T11:31:28Z

I'm using version 1.25.8 (latest available build on gentoo at this time), so I don't have other related fixes like #4548 or #4563 yet (nor have I checked it with the latest version in git). The deviant account I'm testing it with has premium content ((NSFW) https://www.deviantart.com/neotypical/gallery/all). I don't have a subscription, so I don't expect gallery-dl to download premium content but also not to download any blurry preview images.

My configuration file contains the following:

{
    "extractor" : {
        "deviantart" : {
            "client-id" : "** redacted **",
            "client-secret" : "** redacted **",
            "metadata": false
        }
    }
}

Downloading the gallery will show warnings like

[downloader.http][warning] '401 Unauthorized' for ...
[download][info] Trying fallback URL #1

At the time I was fiddling around in the source code to add the hack, I also tried setting the mature setting, but according to https://github.com/mikf/gallery-dl/blob/master/docs/configuration.rst#extractordeviantartmature it's set to true by default anyway.

n00bmister69 · 2023-10-02T05:49:55Z

this worked prior to 19/9/2023

Is it possible to download images posted prior to 19/9? I had previously downloaded images from a specific gallery but later deleted it, so is possible to use cookies/auth anything to atleast redownload the files I did previously?

left1000 · 2023-10-02T16:38:46Z

I'm worried about another knock on effect of this change....

If I download the blurred files, that'll get added to my archive.sqlite3 file? right?

So then later if I do subscribe to premium dA downloads in a gallery I'd have to nuke my archive to rip it?

That'll be highly inconvenient since one of the main reasons I use gallery-dl is so that I can pay for my monthly subscriptions once a year instead of every month and still have a full collection (that's sometimes out of date.)

kattjevfel · 2023-10-02T19:30:44Z

You could also just re-download that specific artist while using --no-skip, instead of wiping the whole archive.

left1000 · 2023-10-02T21:30:07Z

You could also just re-download that specific artist while using --no-skip, instead of wiping the whole archive.

How does "--no-skip" differ from "-o skip=false"?

Although wait, redownloading an entire rip could waste tons of GB, put I'm too lazy to copy paste 30 specific one page urls, hmm.

Hrxn · 2023-10-03T01:05:12Z

You could also just re-download that specific artist while using --no-skip, instead of wiping the whole archive.

How does "--no-skip" differ from "-o skip=false"?

Simple: It does not differ in any way.

Although wait, redownloading an entire rip could waste tons of GB, put I'm too lazy to copy paste 30 specific one page urls, hmm.

Not sure what you mean by wasted here, but if you download "full" subscription-only content, it obviously requires more disk capacity?

Hrxn · 2023-10-03T01:06:04Z

this worked prior to 19/9/2023

Is it possible to download images posted prior to 19/9? I had previously downloaded images from a specific gallery but later deleted it, so is possible to use cookies/auth anything to atleast redownload the files I did previously?

Do these images still exist on dA?
If not, then no.

left1000 · 2023-10-03T04:27:49Z

You could also just re-download that specific artist while using --no-skip, instead of wiping the whole archive.

How does "--no-skip" differ from "-o skip=false"?

Simple: It does not differ in any way.

Although wait, redownloading an entire rip could waste tons of GB, put I'm too lazy to copy paste 30 specific one page urls, hmm.

Not sure what you mean by wasted here, but if you download "full" subscription-only content, it obviously requires more disk capacity?

If a gallery has free content, and subscription content, and I rip it once a week, and pay one month out of every 12, I'm going to have to fully redownload the entire gallery rip once a year, even if I'm just adding 30 new files to a collection of 1000 files.

This is a problem that didn't exist before, because gallery-dl was simply ripping the paid content for free.

I guess what I'm saying is, not only would an ideal fix to this problem ignore and not download the blurred files (which is how it handles things in 1.25.8 but those files which attempt to feed gallery-dl a blurred version of an image instead... should ideally be marked as not downloaded in the archive sqlite file. At least imo. Then again this would probably have it's own drawbacks I am not think of right now.

Twi-Hard · 2023-10-03T08:12:54Z

Are you sure you need a subscription? It being blurred when the full resolution trick broke sounds like it was the trick was getting full res rather than being logged in? I could be wrong, I don't know

left1000 · 2023-10-03T20:30:22Z

Are you sure you need a subscription? It being blurred when the full resolution trick broke sounds like it was the trick was getting full res rather than being logged in? I could be wrong, I don't know

I'm sorry I do not know what you are asking about, are you saying that removing my cookies could bring the trick back? That doesn't make sense to me, because up here #4567 (comment) it was confirmed there is simply no more existence to the exploit at all, Regardless of any change a user or dev might make to their command or the code in gallery-dl...

Twi-Hard · 2023-10-04T10:00:06Z

The trick got fixed and a new release of gallery-dl just came out with it.

left1000 · 2023-10-04T10:19:32Z

The trick got fixed and a new release of gallery-dl just came out with it.

AWESOME HOORAY

also uh, so now I'm using a lot of skip=false but I wonder.... is there a flag for gallery-dl that would simply check to see if the file it's about to create already exists? and if it already exists don't overwrite it? That'd be useful today, and then never again (unless dA breaks)

edit: I am reading here, https://github.com/mikf/gallery-dl/blob/master/docs/configuration.rst#extractorskip but it feels like it would be nice if there was an option that would treat the archive id and file already existing separately, so I could do like -o skip=ignore-archive-but-dont-ignore-files-already-there

edit2: okay so I can just rename the archive, and run the command, but the ideal command I do not think exists....

What I need is the exact opposite of skip="abort:N" instead of aborting after N consecutive skips, I need to abort after N successfully not skipped downloads. That would do the job here. Although I guess I can also just run no options at all and get what I want for N=infinity.

edit3: I think I am gonna run -o skip="abort:50" on like almost all future commands, it'll end up making gallery-dl like ten times faster (how foolish of me to not have realized this before.) It should also be especially useful on twitter (where I am getting api throttled.)

n00bmister69 · 2023-10-09T11:14:46Z

The issue is resolved with the latest update. So I am closing this issue!

n00bmister69 · 2023-10-12T13:33:23Z

The issue is resolved with the latest update. So I am closing this issue!

The update isn't working for me anymore. Is it same for everyone?

kattjevfel · 2023-10-12T14:33:04Z

Yes.

huskyhi · 2023-10-13T23:09:51Z

I have a question? How can I tell if image is blurred vs be being only corrupted? I’ve notice images in some galleries now appear to only be half downloaded then turning into what appears to be a smear before the rest is black. Is this the blurred effect?

left1000 · 2023-10-21T20:55:24Z

This is still non-working right? there were so many fixes in today's updates I couldn't be sure (it's not working for me, but maybe I did something wrong?)

n00bmister69 · 2023-10-27T04:47:28Z

This is still non-working right? there were so many fixes in today's updates I couldn't be sure (it's not working for me, but maybe I did something wrong?)

Yea not for me atleast.

n00bmister69 · 2023-10-27T04:49:47Z

I have a question? How can I tell if image is blurred vs be being only corrupted? I’ve notice images in some galleries now appear to only be half downloaded then turning into what appears to be a smear before the rest is black. Is this the blurred effect?

Blur filter is what the name implies. the same images but hidden under an overall blur not black or anything else

n00bmister69 changed the title ~~Deviantart images getting downloaded with a blur filter~~ [deviantart] Deviantart images getting downloaded with a blur filter Sep 22, 2023

n00bmister69 closed this as completed Oct 9, 2023

n00bmister69 reopened this Oct 12, 2023

mikf added a commit that referenced this issue Feb 13, 2024

[deviantart] skip locked/blurred posts (#4567, #5193)

814ad93

mikf closed this as completed Feb 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[deviantart] Deviantart images getting downloaded with a blur filter #4567

[deviantart] Deviantart images getting downloaded with a blur filter #4567

n00bmister69 commented Sep 22, 2023

ClosedPort22 commented Sep 22, 2023

n00bmister69 commented Sep 22, 2023

Ahmed0425 commented Sep 23, 2023

n00bmister69 commented Sep 23, 2023 •

edited

Loading

Twi-Hard commented Sep 23, 2023

rewrib commented Sep 23, 2023 •

edited

Loading

rautamiekka commented Sep 24, 2023

left1000 commented Sep 24, 2023

ClosedPort22 commented Sep 24, 2023

left1000 commented Sep 25, 2023

Progman2002 commented Sep 29, 2023

ClosedPort22 commented Sep 29, 2023

Progman2002 commented Sep 29, 2023

n00bmister69 commented Oct 2, 2023

left1000 commented Oct 2, 2023

kattjevfel commented Oct 2, 2023

left1000 commented Oct 2, 2023 •

edited

Loading

Hrxn commented Oct 3, 2023

Hrxn commented Oct 3, 2023

left1000 commented Oct 3, 2023

Twi-Hard commented Oct 3, 2023

left1000 commented Oct 3, 2023 •

edited

Loading

Twi-Hard commented Oct 4, 2023

left1000 commented Oct 4, 2023 •

edited

Loading

n00bmister69 commented Oct 9, 2023

n00bmister69 commented Oct 12, 2023

kattjevfel commented Oct 12, 2023

huskyhi commented Oct 13, 2023 •

edited

Loading

left1000 commented Oct 21, 2023

n00bmister69 commented Oct 27, 2023 •

edited

Loading

n00bmister69 commented Oct 27, 2023

[deviantart] Deviantart images getting downloaded with a blur filter #4567

[deviantart] Deviantart images getting downloaded with a blur filter #4567

Comments

n00bmister69 commented Sep 22, 2023

ClosedPort22 commented Sep 22, 2023

n00bmister69 commented Sep 22, 2023

Ahmed0425 commented Sep 23, 2023

n00bmister69 commented Sep 23, 2023 • edited Loading

Twi-Hard commented Sep 23, 2023

rewrib commented Sep 23, 2023 • edited Loading

rautamiekka commented Sep 24, 2023

left1000 commented Sep 24, 2023

ClosedPort22 commented Sep 24, 2023

left1000 commented Sep 25, 2023

Progman2002 commented Sep 29, 2023

ClosedPort22 commented Sep 29, 2023

Progman2002 commented Sep 29, 2023

n00bmister69 commented Oct 2, 2023

left1000 commented Oct 2, 2023

kattjevfel commented Oct 2, 2023

left1000 commented Oct 2, 2023 • edited Loading

Hrxn commented Oct 3, 2023

Hrxn commented Oct 3, 2023

left1000 commented Oct 3, 2023

Twi-Hard commented Oct 3, 2023

left1000 commented Oct 3, 2023 • edited Loading

Twi-Hard commented Oct 4, 2023

left1000 commented Oct 4, 2023 • edited Loading

n00bmister69 commented Oct 9, 2023

n00bmister69 commented Oct 12, 2023

kattjevfel commented Oct 12, 2023

huskyhi commented Oct 13, 2023 • edited Loading

left1000 commented Oct 21, 2023

n00bmister69 commented Oct 27, 2023 • edited Loading

n00bmister69 commented Oct 27, 2023

n00bmister69 commented Sep 23, 2023 •

edited

Loading

rewrib commented Sep 23, 2023 •

edited

Loading

left1000 commented Oct 2, 2023 •

edited

Loading

left1000 commented Oct 3, 2023 •

edited

Loading

left1000 commented Oct 4, 2023 •

edited

Loading

huskyhi commented Oct 13, 2023 •

edited

Loading

n00bmister69 commented Oct 27, 2023 •

edited

Loading