Deviantart Extractor will not download folders indicated in URL file.

[sledgehammer93]

I appear to be having issues regarding the Deviantart Extractor portion of this program. My list of URLs that I have configured for this program either download from the All folders or the scraps folders.

Example:
https://www.deviantart.com/someartisthere/gallery/?catpath=/ https://www.deviantart.com/anotherartisthere/gallery/?catpath=scraps

However, here recently, the program ignores these urls and defaults to the Featured Extractor.

Example:
https://www.deviantart.com/someartisthere/gallery/?catpath=/ #program decides to instead use the url at https://www.deviantart.com/someartisthere/gallery/?catpath=featured/

At first, I thought it might be an issue regarding the 429 errors from rate limiting. It appears to be an at random issue for me, though, with or without the 429 errors. Don't know if this is a reoccurring issue or not, so if this is already posted here, I apologize.

Thanks.

(My gallery-dl.conf file accounts for the rate limiting for the most part with sleep and wait functions, and I even have my own private access key, if that helps)

Verbose output:

[gallery-dl][debug] Version 1.10.5-dev
[gallery-dl][debug] Python 3.7.3 - Linux-5.0.0-29-generic-x86_64-with-Ubuntu-19.04-disco
[gallery-dl][debug] requests 2.21.0 - urllib3 1.24.1
[1/88] https://www.deviantart.com/someartisthere/gallery/?catpath=/
[gallery-dl][debug] Starting DownloadJob for 'https://www.deviantart.com/someartisthere/gallery/?catpath=/'
[gallery-dl][debug] Updating urllib3 ciphers
[deviantart][debug] Using DeviantartGalleryExtractor for 'https://www.deviantart.com/someartisthere/gallery/?catpath=/'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/someartisthere HTTP/1.1" 200 683
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/all?username=someartisthere&offset=0&limit=24&mature_content=true HTTP/1.1" 429 105
[deviantart][debug] {"error":"application_api_threshold","error_description":"Client request limit reached.","status":"error"}
[deviantart][warning] API responded with 429 Too Many Requests. Using 8s delay.
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/all?username=someartisthere&offset=0&limit=24&mature_content=true HTTP/1.1" 429 105
[deviantart][debug] {"error":"application_api_threshold","error_description":"Client request limit reached.","status":"error"}
[deviantart][warning] API responded with 429 Too Many Requests. Using 16s delay.
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/all?username=someartisthere&offset=0&limit=24&mature_content=true HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/metadata?deviationids[0]=D1F5FBE3-36D1-4DFC-1DCE-93F8BA935D29&deviationids[1]=B254C140-23DB-0DDA-F004-62C763686EA0&deviationids[2]=CA2F507B-433F-5F99-5A56-5B27710B6A85&deviationids[3]=10A37A27-B5F5-4708-2992-42AB594FE666&deviationids[4]=6F9C611C-4E4B-8324-3861-2BAAE6D195CF&deviationids[5]=0846628F-B69A-067B-3E00-58010A7E7A29&deviationids[6]=CADFF510-89E1-C7FF-E842-3E08D30E227D&deviationids[7]=BED07F32-1C18-DAEB-1557-7D80252BB16D&deviationids[8]=5A23BBCF-8057-7EE6-9E05-021586543C1E&deviationids[9]=F9EAC9BF-573F-3AB2-E3A1-454ACDAF7B0A&deviationids[10]=F1ECD0D2-004C-78F0-0B7E-931F02BD0AFC&deviationids[11]=0FFB7FA2-7060-2CBA-92E9-8CE3A748C698&deviationids[12]=8929C3EE-46AC-3271-95B2-2E02B5118F8F&deviationids[13]=09577F83-106A-2183-6409-04F8F2AD3E9F&deviationids[14]=53B80C99-9374-B555-CC4C-A8A16E103EC1&deviationids[15]=F0550128-009D-6EF6-AB2C-01374A79C09E&deviationids[16]=FD07CDEE-0CB9-2408-3032-5A6138324A3A&deviationids[17]=25D5337E-90AD-5352-435B-A97A9EF4ACD0&deviationids[18]=7D3AE01C-3807-ABFC-CEA7-87262E547102&deviationids[19]=773AA0E3-73E4-C33A-D8D4-1AEC9FF4D356&deviationids[20]=66C267EE-7A49-1958-BD20-34F7EB246098&deviationids[21]=98562875-ABAD-278B-4F31-5550C12C2273&deviationids[22]=218098B5-BEDD-A9A2-7974-C561F59930FF&deviationids[23]=F59C0BE7-1CC0-4A60-964B-E2AEDB7064F5&mature_content=true HTTP/1.1" 200 3478
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/D1F5FBE3-36D1-4DFC-1DCE-93F8BA935D29?mature_content=true HTTP/1.1" 200 1235
/home/william/Important/gallery-dl/deviantart/someartisthere/deviantart_813123016_deviantfilehere.jpg
[2/88] https://www.deviantart.com/someartisthere/gallery/?catpath=scraps
[gallery-dl][debug] Starting DownloadJob for 'https://www.deviantart.com/someartisthere/gallery/?catpath=scraps'
[deviantart][debug] Using DeviantartScrapsExtractor for 'https://www.deviantart.com/someartisthere/gallery/?catpath=scraps'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /_napi/da-user-profile/api/gallery/contents?username=someartisthere&offset=0&limit=24&scraps_folder=true HTTP/1.1" 200 None
[3/88] https://www.deviantart.com/anotherartisthere/gallery/?catpath=/       

 ##Note this URL. 

[gallery-dl][debug] Starting DownloadJob for 'https://www.deviantart.com/anotherartisthere/gallery/?catpath=/'
[deviantart][debug] Using DeviantartGalleryExtractor for 'https://www.deviantart.com/anotherartisthere/gallery/?catpath=/'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/anotherartisthere HTTP/1.1" 429 105
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/folders?username=anotherartisthere&offset=0&limit=50&mature_content=true HTTP/1.1" 200 184
[deviantart][debug] Using DeviantartFolderExtractor for 'https://www.deviantart.com/anotherartisthere/gallery/0/Featured'    

                                   
##Issue Starts around here. Begins downloading the Featured Folder, which is not what I requested. Does this regardless of the 429's, in my experience
```

[mikf]

The deviantart extractor tries to determine if anotherartisthere is an actual user or a group by trying to access its user profile: GET /api/v1/oauth2/user/profile/anotherartisthere. If it succeeds, its a user, otherwise a group, and since groups don't have an "all" folder, it falls back to requesting a list of all folders and downloading them.

In your case the user profile request failed because of a 429 Too Many Requests status code and gallery-dl wrongly assumed it's a group.

You should follow the instructions here to get your own client-id and -secret, so you won't get as many 429 errors, while I'll be working on fixing this.

[sledgehammer93]

Thank you for the prompt reply @mikf. I am already using my own client-id and -secret, but am still getting numerous 429 errors.

Could it be syntax related? Here is my gallery-dl.conf:

{
    "extractor":
    {
        "base-directory": "/home/william/Important/gallery-dl/",
        "postprocessors": null,
        "cookies": null,
        "proxy": null,
        "skip": true,
        "sleep": 2,
        "user-agent": "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0",
        "path-restrict": "\\\\|/<>:\"?*",

        "deviantart":
        {
            "extra": true,
            "flat": true,
            "folders": false,
            "journals": "html",
            "mature": true,
            "metadata": true,
            "original": true,
            "quality": 100,
	    "wait-min": 4,
	    "client_id": "10468",
	    "client_secret": "myid",
	    "refresh-token": "tokenhere"
        }
    },

    "downloader":
    {
        "part": true,
        "part-directory": null,

        "http":
        {
            "adjust-extensions": true,
            "mtime": true,
            "rate": "128k",
            "retries": 4,
            "timeout": 30.0,
            "verify": true
        },

        "ytdl":
        {
            "format": "bestvideo+bestaudio/best",
            "forward-cookies": true,
            "mtime": true,
            "rate": null,
            "retries": 4,
            "timeout": 30.0,
            "verify": true
        }
    },

    "output":
    {
        "mode": "auto",
        "progress": true,
        "shorten": true,
        "log": "[{name}][{levelname}] {message}",
        "logfile": null,
        "unsupportedfile": null
    },

    "netrc": false
}
`

[mikf]

It's client-id and client-secret with a - in between, not client_id and client_secret.
(You will also have to get a new refresh-token once you change the names and actually use your own API credentials)

[sledgehammer93]

Ahh, my mistake. Fixed that just now. The 429's appear to have settled down (for now). Thanks.