DeviantArt extractor stopped working on some DA pages

[Ailothaen]

Hello,

I have been using gallery-dl in an automated process for months to backup profiles, galleries, journal... and I did not run into any issues.

However, I just noticed that since a recent update (I give more precisions below), the DeviantArt part of gallery-dl is failing to download galleries, but it working fine on other sections such as Journals.
Here is the output I am getting when I am trying to download my /gallery and my /journal section:

root@hestia:/opt/scripts/batch-da # /usr/local/bin/gallery-dl -c './gallery-dl-debug.conf' https://www.deviantart.com/ailothaen/journal -vjs
[gallery-dl][debug] Version 1.15.0
[gallery-dl][debug] Python 3.7.3 - Linux-4.19.118-v7+-armv7l-with-debian-10.4
[gallery-dl][debug] requests 2.21.0 - urllib3 1.24.1
[gallery-dl][debug] Starting SimulationJob for 'https://www.deviantart.com/ailothaen/journal'
[deviantart][debug] Using DeviantartJournalExtractor for 'https://www.deviantart.com/ailothaen/journal'
[deviantart][debug] Using custom API credentials (client-id 10734)
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/ailothaen HTTP/1.1" 200 1590
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/browse/user/journals?username=Ailothaen&offset=0&limit=50&mature_content=true&featured=false HTTP/1.1" 200 887
[deviantart][debug] Switching to private access token
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/browse/user/journals?username=Ailothaen&offset=0&limit=50&mature_content=true&featured=false HTTP/1.1" 200 887
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/metadata?deviationids[0]=DB94052F-4F82-CCF6-2090-90D78DF4A6A8&mature_content=true HTTP/1.1" 200 333
[deviantart][debug] Active postprocessor modules: [MetadataPP]
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/content?deviationid=DB94052F-4F82-CCF6-2090-90D78DF4A6A8 HTTP/1.1" 200 6256
deviantart_778485002_The making of the Ludara painting.htm

root@hestia:/opt/scripts/batch-da # /usr/local/bin/gallery-dl -c './gallery-dl-debug.conf' https://www.deviantart.com/ailothaen/gallery -vjs
[gallery-dl][debug] Version 1.15.0
[gallery-dl][debug] Python 3.7.3 - Linux-4.19.118-v7+-armv7l-with-debian-10.4
[gallery-dl][debug] requests 2.21.0 - urllib3 1.24.1
[gallery-dl][debug] Starting SimulationJob for 'https://www.deviantart.com/ailothaen/gallery'
[deviantart][debug] Using DeviantartGalleryExtractor for 'https://www.deviantart.com/ailothaen/gallery'
[deviantart][debug] Using custom API credentials (client-id 10734)
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/ailothaen HTTP/1.1" 200 1590
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/folders?username=Ailothaen&offset=0&limit=50&mature_content=true HTTP/1.1" 200 136
root@hestia:/opt/scripts/batch-da #

As you can see, gallery-dl downloads flawlessly the only element in my /journal section, but fails when I try to get it to download my /gallery section – without even an error message or a warning, it just exits.

I tested on another system with the same version and with a config file with only the tokens, and I noticed that it fails too with the latest version (1.15.0). However, everything is working fine with a previous version (1.14.4)

Thank you in advance for the help!

PS: I think adding a --trace option, where you able to see the content of every HTTP request, would be really useful in cases like this one...

[mikf]

Thank you for reporting this.

This is another unintended consequence of the new options/features implemented in 1.15.0, blacklist/whitelist in this case.

3ebb174 fixes this issue for Deviantart and all other affected sites, but you can get it working without this change by setting any non-empty blacklist for Deviantart:

"deviantart": {
    "blacklist": "foobar"
}

---

PS: I think adding a --trace option, where you able to see the content of every HTTP request, would be really useful in cases like this one...

  --write-pages             Write downloaded intermediary pages to files in
                            the current directory to debug problems

[mikf]

@Ailothaen 1.15.1 with the fix is out.