Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DeviantArt filename extension #621

Closed
LizeHelesta opened this issue Feb 23, 2020 · 4 comments
Closed

DeviantArt filename extension #621

LizeHelesta opened this issue Feb 23, 2020 · 4 comments

Comments

@LizeHelesta
Copy link 

[downloader.http][warning] No filename extension found for MIME type 'image/x-ms-bmp'

Sample

Besides, the single deviantart extractor will not use the cookies configuration even if i use --cookies to specific the path.

[deviantart][warning] No session cookies set: Disabling original file downloads.

But the gallery url is ok.
@kattjevfel
Copy link
Contributor

The URL you linked works fine here (also thanks for the beautiful art)

gallery-dl --verbose https://www.deviantart.com/nexus-4/art/Tamaki-Portrait-long-hair-829880882                                    130 ↵
[gallery-dl][debug] Version 1.13.1-dev
[gallery-dl][debug] Python 3.8.1 - Linux-5.5.5-zen1-1-zen-x86_64-with-glibc2.2.5
[gallery-dl][debug] requests 2.23.0 - urllib3 1.25.8
[gallery-dl][debug] Starting DownloadJob for 'https://www.deviantart.com/nexus-4/art/Tamaki-Portrait-long-hair-829880882'
[deviantart][debug] Using default API credentials (client-id 5388)
[deviantart][debug] Using DeviantartDeviationExtractor for 'https://www.deviantart.com/nexus-4/art/Tamaki-Portrait-long-hair-829880882'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /_napi/da-browse/shared_api/deviation/extended_fetch?deviationid=829880882&username=nexus-4&type=art&include_session=false HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /download/829880882/ddq386q-05116988-9b9c-40ec-a6be-5aa21e919eae.bmp?token=95c74b6324fb35875484cfc64aedac1d04ce017b&ts=1582463152 HTTP/1.1" 302 0
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): api-da.wixmp.com:443
[urllib3.connectionpool][debug] https://api-da.wixmp.com:443 "GET /_api/download/file?downloadToken=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsImV4cCI6MTU4MjQ2MzE1MiwiaWF0IjoxNTgyNDYyNTQyLCJqdGkiOiI1ZTUyNzY1ODg0ODI4Iiwib2JqIjpudWxsLCJhdWQiOlsidXJuOnNlcnZpY2U6ZmlsZS5kb3dubG9hZCJdLCJwYXlsb2FkIjp7InBhdGgiOiJcL2ZcL2FhMjgyZjFmLTVhMWMtNDk1Zi1hNGM4LTM5MGRmNTc1MDQ5NlwvZGRxMzg2cS0wNTExNjk4OC05YjljLTQwZWMtYTZiZS01YWEyMWU5MTllYWUuYm1wIiwiYXR0YWNobWVudCI6eyJmaWxlbmFtZSI6InRhbWFraV9wb3J0cmFpdF9fbG9uZ19oYWlyX19ieV9uZXh1c180X2RkcTM4NnEuYm1wIn19fQ.o2yek3uTDPxhmXE0mfRhHRBys7AjFygPboFItlsOyho HTTP/1.1" 200 None
/mnt/jupiter/Temp/gallery-dl/deviantart/Nexus-4/tamaki_portrait__long_hair__by_nexus_4-ddq386q.bmp

Are you sure you're using the latest version of gallery-dl?

My config for reference:

        "deviantart": {
            "filename": "{filename}.{extension}",
            "client_id": "<nope>",
            "client_secret": "<nope>",
            "refresh-token": "<nope>",
            "cookies": "/mnt/jupiter/Temp/gallery-dl/cookies-deviantart-com.txt",
            "cookies-update": true
        },

@LizeHelesta
Copy link
Author

LizeHelesta commented Feb 23, 2020 •
edited
Loading

Single Log 

$ gallery-dl -v -s https://www.deviantart.com/nexus-4/art/Tamaki-Portrait-long-hair-829880882
[gallery-dl][debug] Version 1.13.1-dev
[gallery-dl][debug] Python 3.7.1 - Windows-10-10.0.16299-SP0
[gallery-dl][debug] requests 2.21.0 - urllib3 1.24.1
[gallery-dl][debug] Starting SimulationJob for 'https://www.deviantart.com/nexus-4/art/Tamaki-Portrait-long-hair-829880882'
[deviantart][debug] Using custom API credentials (client-id ****)
[deviantart][debug] Using DeviantartDeviationExtractor for 'https://www.deviantart.com/nexus-4/art/Tamaki-Portrait-long-hair-829880882'
[deviantart][warning] No session cookies set: Disabling original file downloads.
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /_napi/da-browse/shared_api/deviation/extended_fetch?deviationid=829880882&username=nexus-4&type=art&include_session=false HTTP/1.1" 200 None
[deviantart][debug] Active postprocessor modules: [MetadataPP]
# deviantart_829880882_Tamaki Portrait (long hair).png

Gallery Log 

$ gallery-dl -v https://www.deviantart.com/nexus-4/ --range 1-2
[gallery-dl][debug] Version 1.13.1-dev
[gallery-dl][debug] Python 3.7.1 - Windows-10-10.0.16299-SP0
[gallery-dl][debug] requests 2.21.0 - urllib3 1.24.1
[gallery-dl][debug] Starting DownloadJob for 'https://www.deviantart.com/nexus-4/'
[deviantart][debug] Using custom API credentials (client-id ****)
[deviantart][debug] Using DeviantartUserExtractor for 'https://www.deviantart.com/nexus-4/'
[deviantart][debug] Using custom API credentials (client-id ****)
[deviantart][debug] Using DeviantartGalleryExtractor for 'https://www.deviantart.com/nexus-4/gallery'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/nexus-4 HTTP/1.1" 200 1696
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/all?username=Nexus-4&offset=0&limit=24&mature_content=true HTTP/1.1" 200 4747
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/metadata?deviationids[0]=3BAC8BF9-****-****-****-6BF65989263B&deviationids[1]=6970E540-****-****-****-1CA542B1D06D&deviationids[2]=7B42C542-****-****-****-D940A0B4A0A1&deviationids[3]=68A0C36F-****-****-****-27A9556E67E9&deviationids[4]=72C235E2-****-****-****-3840DDBF6A57&deviationids[5]=1D0EFFD4-****-****-****-4F07FB6C67AC&deviationids[6]=2448B1A2-****-****-****-A1273A4AD965&deviationids[7]=6920CEB5-****-****-****-568E6D4180A9&deviationids[8]=2FB678C1-****-****-****-43ED84BEA10B&deviationids[9]=BAE184B3-****-****-****-ADF8137AD188&deviationids[10]=79AB762D-****-****-****-BACA54BBF8EB&deviationids[11]=7DE907AE-****-****-****-61B9E4EB8CD8&deviationids[12]=C2C8EAB6-****-****-****-14DD3235138C&deviationids[13]=207D090C-****-****-****-F3078D82C983&deviationids[14]=B65880E5-****-****-****-699248B4497C&deviationids[15]=CFE3B67B-****-****-****-C663CB7B4A78&deviationids[16]=F8A63D6A-****-****-****-629799F302E4&deviationids[17]=3E196ADE-****-****-****-62883C6C8B3D&deviationids[18]=A2F72D46-****-****-****-166FB51CBAF1&deviationids[19]=ECC57065-****-****-****-FE0910A5FBDE&deviationids[20]=2DD0A838-****-****-****-6660A18D37CC&deviationids[21]=A9C06354-****-****-****-2C36D78E7859&deviationids[22]=77D4275D-****-****-****-CDF23621FBFA&deviationids[23]=BFAD1178-****-****-****-F1CE6C15E62E&mature_content=true HTTP/1.1" 200 5171
[deviantart][debug] Active postprocessor modules: [MetadataPP]
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/3BAC8BF9-****-****-****-6BF65989263B?mature_content=true HTTP/1.1" 200 501
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): api-da.wixmp.com:443
[urllib3.connectionpool][debug] https://api-da.wixmp.com:443 "GET /_api/download/file?downloadToken=**** HTTP/1.1" 200 None
[downloader.http][warning] No filename extension found for MIME type 'image/x-ms-bmp'
* .\gallery-dl\deviantart\Nexus-4\deviantart_831417767_Pole Dance Acrobatics (Kokoro-2).txt
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/6970E540-****-****-****-1CA542B1D06D?mature_content=true HTTP/1.1" 200 501
[urllib3.connectionpool][debug] https://api-da.wixmp.com:443 "GET /_api/download/file?downloadToken=**** HTTP/1.1" 200 None
[downloader.http][warning] No filename extension found for MIME type 'image/x-ms-bmp'
* .\gallery-dl\deviantart\Nexus-4\deviantart_831357697_Pole Dance Acrobatics (Amy Sorel).txt
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/7B42C542-****-****-****-D940A0B4A0A1?mature_content=true HTTP/1.1" 200 501

Config 

{
	"extractor": {
		"deviantart": {
			"client-id": "****",
			"client-secret": "****",
			"extra": true,
			"metadata": true,
			"postprocessors": [{"name": "metadata"}],
			"cookies": "E:/cookies.txt",
			"cookies-update": true
		}
	}
}

@LizeHelesta
Copy link
Author

LizeHelesta commented Feb 23, 2020 •
edited
Loading

After Python v3.7.5, mimetypes contain this MIME type, but gallery-dl just require v3.4+, so i think it need to be hard coded in MIMETYPE_MAP
DeviantartExtractor (the base of DeviantartGalleryExtractor) doesn't check cookies, but DeviantartExtractorV2 (the base of DeviantartDeviationExtractor) will.
And in my cookies, just contain auth_secure and userinfo no auth, so DeviantartExtractorV2 "Disabling original file downloads", but this cookies is enough for downloading original file so DeviantartExtractor work fine. (The cookies is updated by gallery-dl for a long time not me grab from browser just now)

mikf added a commit that referenced this issue Feb 28, 2020
@mikf
[downloader:http] add more MIME types for '.bmp' files (#621)
380b693
@mikf
Copy link
Owner

mikf commented Feb 28, 2020

i think it need to be hard coded in MIMETYPE_MAP

Done: 380b693

but this cookies is enough for downloading original file

It isn't, you definitely need all three cookies for downloading original scraps and single deviations Try re-exporting your DeviantArt cookies and see if that helps. Cookie handing in cookies.txt files was a bit iffy prior to v1.13.0, so maybe your auth cookie got lost back then, or you were browsing DeviantArt with the same login session as in your cookies.txt file. That apparently also doesn't work for long.

Also DeviantartGalleryExtractor and all the others that aren't derived from DeviantartExtractorV2 do not need cookies, it's only DeviantartDeviationExtractor and DeviantartScrapsExtractor that do. Everything else uses the OAuth API.

@mikf mikf closed this as completed Dec 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mikf @kattjevfel @LizeHelesta