Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Danbooru] HTTP request failed #406

Closed
DallasWhite opened this issue Aug 31, 2019 · 8 comments
Closed

[Danbooru] HTTP request failed #406

DallasWhite opened this issue Aug 31, 2019 · 8 comments
Labels
enhancement user:question

Comments

@DallasWhite
Copy link

I'm trying out the danbooru extractor, and I'm currently getting this error when I try to download a video post. Here's the debug:

[gallery-dl][debug] Version 1.10.1
[gallery-dl][debug] Python 3.7.2 - Windows-10-10.0.17763-SP0
[gallery-dl][debug] requests 2.21.0 - urllib3 1.24.1
[gallery-dl][debug] Starting DownloadJob for 'https://danbooru.donmai.us/posts/3501007'
[gallery-dl][debug] Updating urllib3 ciphers
[danbooru][debug] Using HTTP Basic Auth for user 'RiotRiot'
[danbooru][debug] Using DanbooruPostExtractor for 'https://danbooru.donmai.us/posts/3501007'
[danbooru][debug] Active postprocessor modules: [<gallery_dl.postprocessor.ugoira.UgoiraPP object at 0x03A014F0>]
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): danbooru.donmai.us:443
[urllib3.connectionpool][debug] https://danbooru.donmai.us:443 "GET /posts.json?tags=id%3A3501007&page=1 HTTP/1.1" 500 None
[danbooru][debug] 500: Internal Server Error for url: https://danbooru.donmai.us/posts.json (1/5)
[urllib3.connectionpool][debug] https://danbooru.donmai.us:443 "GET /posts.json?tags=id%3A3501007&page=1 HTTP/1.1" 500 None
[danbooru][debug] 500: Internal Server Error for url: https://danbooru.donmai.us/posts.json (2/5)
[urllib3.connectionpool][debug] https://danbooru.donmai.us:443 "GET /posts.json?tags=id%3A3501007&page=1 HTTP/1.1" 500 None
[danbooru][debug] 500: Internal Server Error for url: https://danbooru.donmai.us/posts.json (3/5)
[urllib3.connectionpool][debug] https://danbooru.donmai.us:443 "GET /posts.json?tags=id%3A3501007&page=1 HTTP/1.1" 500 None
[danbooru][debug] 500: Internal Server Error for url: https://danbooru.donmai.us/posts.json (4/5)
[urllib3.connectionpool][debug] https://danbooru.donmai.us:443 "GET /posts.json?tags=id%3A3501007&page=1 HTTP/1.1" 500 None
[danbooru][debug] 500: Internal Server Error for url: https://danbooru.donmai.us/posts.json (5/5)
[danbooru][error] HTTP request failed:  500: Internal Server Error for url: https://danbooru.donmai.us/posts.json
@mikf
Copy link
Owner

mikf commented Aug 31, 2019

This is the status code Danbooru sends when you've entered an invalid username + "password" combination. "password" in quotes because it is not your actual account password, but "the API key found in your user profile"

@DallasWhite
Copy link
Author

Ah sorry, I should have read that. However, I'm now getting this error:

[gallery-dl][debug] Version 1.10.1
[gallery-dl][debug] Python 3.7.2 - Windows-10-10.0.17763-SP0
[gallery-dl][debug] requests 2.21.0 - urllib3 1.24.1
[gallery-dl][debug] Starting DownloadJob for 'https://danbooru.donmai.us/posts/3488067'
[gallery-dl][debug] Updating urllib3 ciphers
[danbooru][debug] Using HTTP Basic Auth for user 'RiotRiot'
[danbooru][debug] Using DanbooruPostExtractor for 'https://danbooru.donmai.us/posts/3488067'
[danbooru][debug] Active postprocessor modules: [<gallery_dl.postprocessor.ugoira.UgoiraPP object at 0x041C13F0>]
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): danbooru.donmai.us:443
[urllib3.connectionpool][debug] https://danbooru.donmai.us:443 "GET /posts.json?tags=id%3A3488067&page=1 HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://danbooru.donmai.us:443 "GET /data/1d3f676166136fca5dc41dbc6cfcc6cb.zip HTTP/1.1" 416 None
[postprocessor][debug] ffmpeg args: ['ffmpeg', '-i', 'C:\\Users\\ABlack\\AppData\\Local\\Temp\\tmp6j_fefaf/ffconcat.txt', '-r', '1000/120', '-c:v', 'libvpx', '-crf', '4', '-b:v', '5000k', '-an']
[danbooru][error] Unable to download data:  FileNotFoundError: [WinError 2] The system cannot find the file specified
[danbooru][debug]
Traceback (most recent call last):
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\site-packages\gallery_dl\job.py", line 48, in run
    self.dispatch(msg)
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\site-packages\gallery_dl\job.py", line 93, in dispatch
    self.handle_url(url, kwds)
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\site-packages\gallery_dl\job.py", line 233, in handle_url
    pp.run(pathfmt)
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\site-packages\gallery_dl\postprocessor\ugoira.py", line 110, in run
    self._exec(args + ["1", "-y", os.devnull])
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\site-packages\gallery_dl\postprocessor\ugoira.py", line 123, in _exec
    return subprocess.Popen(args, stdout=out, stderr=out).wait()
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\subprocess.py", line 1178, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

@mikf
Copy link
Owner

mikf commented Aug 31, 2019 •
edited
Loading

You don't have FFmpeg installed or in your PATH.

For Ugoira videos on Danbooru, gallery-dl does the same conversion as it does on Pixiv, and that involves using FFmpeg.

I was planning on adding an option to simply download the pre-rendered video for those kinds of posts on Danbooru, but I never got to it. Guess now is as good a time as any.

@DallasWhite
Copy link
Author

Interesting, I have ffmpeg in my gallery-dl folder already, so I went to my config file to actually specify it's location

{
	"extractor":
	{
		"base-directory": "F:/Scripts/Gallery-dl/",
		"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36",
		"archive": "F:/Scripts/Gallery-dl/archive.sqlite3",
		
		"postprocessors": [
            {	
		"name": "ugoira",
		"whitelist": ["pixiv", "danbooru"],
                "ffmpeg-twopass": true,
                "ffmpeg-args": ["-c:v", "libvpx", "-crf", "4", "-b:v", "5000k", "-an", "2M"],
		"ffmpeg-location": "C:/Users/ABlack/gallery-dl/"
            }

but now I get this:

[gallery-dl][debug] Version 1.10.1
[gallery-dl][debug] Python 3.7.2 - Windows-10-10.0.17763-SP0
[gallery-dl][debug] requests 2.21.0 - urllib3 1.24.1
[gallery-dl][debug] Starting DownloadJob for 'https://danbooru.donmai.us/posts/3488067'
[gallery-dl][debug] Updating urllib3 ciphers
[danbooru][debug] Using HTTP Basic Auth for user 'RiotRiot'
[danbooru][debug] Using DanbooruPostExtractor for 'https://danbooru.donmai.us/posts/3488067'
[danbooru][debug] Active postprocessor modules: [<gallery_dl.postprocessor.ugoira.UgoiraPP object at 0x03A015F0>]
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): danbooru.donmai.us:443
[urllib3.connectionpool][debug] https://danbooru.donmai.us:443 "GET /posts.json?tags=id%3A3488067&page=1 HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://danbooru.donmai.us:443 "GET /data/1d3f676166136fca5dc41dbc6cfcc6cb.zip HTTP/1.1" 416 None
[postprocessor][debug] ffmpeg args: ['C:/Users/ABlack/gallery-dl/', '-i', 'C:\\Users\\ABlack\\AppData\\Local\\Temp\\tmpp_5p89d4/ffconcat.txt', '-r', '1000/120', '-c:v', 'libvpx', '-crf', '4', '-b:v', '5000k', '-an', '2M']
[danbooru][error] Unable to download data:  PermissionError: [WinError 5] Access is denied
[danbooru][debug]
Traceback (most recent call last):
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\site-packages\gallery_dl\job.py", line 48, in run
    self.dispatch(msg)
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\site-packages\gallery_dl\job.py", line 93, in dispatch
    self.handle_url(url, kwds)
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\site-packages\gallery_dl\job.py", line 233, in handle_url
    pp.run(pathfmt)
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\site-packages\gallery_dl\postprocessor\ugoira.py", line 110, in run
    self._exec(args + ["1", "-y", os.devnull])
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\site-packages\gallery_dl\postprocessor\ugoira.py", line 123, in _exec
    return subprocess.Popen(args, stdout=out, stderr=out).wait()
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "c:\users\ablack\appdata\local\programs\python\python37-32\lib\subprocess.py", line 1178, in _execute_child
    startupinfo)
PermissionError: [WinError 5] Access is denied

@mikf
Copy link
Owner

mikf commented Aug 31, 2019 •
edited
Loading

ffmpeg-location needs to point directly to the ffmpeg.exe file, just its directory doesn't work.

Edit: you should also remove the "2M" from the end of your ffmpeg-args. I don't think it works like that.

Edit 2: The next version (1.10.4) will have nicer error messages when invoking ffmpeg fails, and an option for Danbooru to download the pre-rendered video versions instead

mikf added a commit that referenced this issue Aug 31, 2019
@mikf
[danbooru] add 'ugoira' option (#406)
f02a768 
to choose between ZIP archives or converted video files
for Ugoira posts
mikf added a commit that referenced this issue Aug 31, 2019
@mikf
[postprocessor:ugoira] improve error messages (#406)
5fcebb6
@DallasWhite
Copy link
Author

Alright, it looks like the video downloads. Also, is this the output I should expect?

  F:\Scripts\Gallery-dl\danbooru\danbooru_3488067_1d3f676166136fca5dc41dbc6cfcc6cb.webmffmpeg version N-94324-g806ac7da69 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9.1.1 (GCC) 20190716
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt
  libavutil      56. 30.100 / 56. 30.100
  libavcodec     58. 53.101 / 58. 53.101
  libavformat    58. 28.102 / 58. 28.102
  libavdevice    58.  7.100 / 58.  7.100
  libavfilter     7. 56.101 /  7. 56.101
  libswscale      5.  4.101 /  5.  4.101
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100
Input #0, concat, from 'C:\Users\ABlack\AppData\Local\Temp\tmp81isfp_t/ffconcat.txt':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: mjpeg (Baseline), yuvj444p(pc, bt470bg/unknown/unknown), 560x504 [SAR 1:1 DAR 10:9], 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> vp8 (libvpx))
Press [q] to stop, [?] for help
[swscaler @ 0000025514c52e40] deprecated pixel format used, make sure you did set range correctly
[libvpx @ 00000255147c9940] v1.8.0-606-gd749bc7b3
Output #0, webm, to 'nul':
  Metadata:
    encoder         : Lavf58.28.102
    Stream #0:0: Video: vp8 (libvpx), yuv420p, 560x504 [SAR 1:1 DAR 10:9], q=-1--1, 5000 kb/s, 8.33 fps, 1k tbn, 8.33 tbc
    Metadata:
      encoder         : Lavc58.53.101 libvpx
    Side data:
      cpb: bitrate max/min/avg: 0/0/5000000 buffer size: 0 vbv_delay: -1
frame=   81 fps=0.0 q=0.0 Lsize=       1kB time=00:00:00.00 bitrate=N/A speed=   0x
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Output file is empty, nothing was encoded
ffmpeg version N-94324-g806ac7da69 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9.1.1 (GCC) 20190716
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt
  libavutil      56. 30.100 / 56. 30.100
  libavcodec     58. 53.101 / 58. 53.101
  libavformat    58. 28.102 / 58. 28.102
  libavdevice    58.  7.100 / 58.  7.100
  libavfilter     7. 56.101 /  7. 56.101
  libswscale      5.  4.101 /  5.  4.101
  libswresample   3.  4.100 /  3.  4.100
  libpostproc    55.  4.100 / 55.  4.100
Input #0, concat, from 'C:\Users\ABlack\AppData\Local\Temp\tmp81isfp_t/ffconcat.txt':
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: mjpeg (Baseline), yuvj444p(pc, bt470bg/unknown/unknown), 560x504 [SAR 1:1 DAR 10:9], 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> vp8 (libvpx))
Press [q] to stop, [?] for help
[swscaler @ 00000265dbfd3000] deprecated pixel format used, make sure you did set range correctly
[libvpx @ 00000265dba59980] v1.8.0-606-gd749bc7b3
Output #0, webm, to '\\?\F:\Scripts\Gallery-dl\danbooru\danbooru_3488067_1d3f676166136fca5dc41dbc6cfcc6cb.webm':
  Metadata:
    encoder         : Lavf58.28.102
    Stream #0:0: Video: vp8 (libvpx), yuv420p, 560x504 [SAR 1:1 DAR 10:9], q=-1--1, 5000 kb/s, 8.33 fps, 1k tbn, 8.33 tbc
    Metadata:
      encoder         : Lavc58.53.101 libvpx
    Side data:
      cpb: bitrate max/min/avg: 0/0/5000000 buffer size: 0 vbv_delay: -1
frame=   81 fps=0.0 q=0.0 Lsize=     721kB time=00:00:09.96 bitrate= 592.8kbits/s speed=11.5x
video:719kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.183917%
* F:\Scripts\Gallery-dl\danbooru\danbooru_3488067_1d3f676166136fca5dc41dbc6cfcc6cb.webm

@mikf
Copy link
Owner

mikf commented Aug 31, 2019

Yes, that's the output ffmpeg produces when it's running. You can disable it with ffmpeg-output if you have no need for it, but I find it quite useful as a progress indicator.

@DallasWhite
Copy link
Author

Okay great! Thanks for the help again; I'm learning a lot about how to use the program.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement user:question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mikf @DallasWhite