Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Weibo Unknown Error and archive issue #4221

Closed
AdamSaketume26 opened this issue Jun 22, 2023 · 4 comments
Closed

Weibo Unknown Error and archive issue #4221

AdamSaketume26 opened this issue Jun 22, 2023 · 4 comments

Comments

@AdamSaketume26
Copy link

version 1.25.6-dev
extractor: weibo
url: https://weibo.com/u/3540182725

command i run

gallery-dl --download-archive weibo_archive -o extractor.weibo.archive-prefix="[{category}] {user[status[user][screen_name]]} ({status[user][id]})" -o extractor.weibo.archive-format="{subcategory}_{status[mblogid]}_{filename}" -o extractor.weibo.skip="abort:10" -o extractor.weibo.include="home,feed,videos,newvideo,article,album" -o extractor.weibo.directory=[\"{subcategory}\"] -o extractor.weibo.sleep-request="0-1" --mtime-from-date --verbose https://weibo.com/u/3540182725

cmd

gallery-dl --download-archive weibo_archive -o extractor.weibo.archive-prefix="[{category}] {user[status[user][screen_name]]} ({status[user][id]})" -o extractor.weibo.archive-format="{subcategory}_{status[mblogid]}_{filename}" -o extractor.weibo.skip="abort:10" -o extractor.weibo.include="home,feed,videos,newvideo,article,album" -o extractor.weibo.directory=[\"{subcategory}\"] -o extractor.weibo.sleep-request="0-1" --mtime-from-date --verbose https://weibo.com/u/3540182725
[gallery-dl][debug] Version 1.25.6-dev - Executable
[gallery-dl][debug] Python 3.11.4 - Windows-10
[gallery-dl][debug] requests 2.31.0 - urllib3 2.0.3
[gallery-dl][debug] Configuration Files []
[gallery-dl][debug] Starting DownloadJob for 'https://weibo.com/u/3540182725'
[weibo][debug] Using WeiboUserExtractor for 'https://weibo.com/u/3540182725'
[weibo][debug] Using WeiboHomeExtractor for 'https://weibo.com/u/3540182725?tabtype=home'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): weibo.com:443
[urllib3.connectionpool][debug] https://weibo.com:443 "GET /ajax/profile/myhot?uid=3540182725&page=1&feature=2 HTTP/1.1" 200 2
[weibo][debug] b'{}'
[weibo][error] "unknown error"
[weibo][debug] Using WeiboFeedExtractor for 'https://weibo.com/u/3540182725?tabtype=feed'
[weibo][debug] Sleeping 0.70 seconds (request)
[urllib3.connectionpool][debug] https://weibo.com:443 "GET /ajax/statuses/mymblog?uid=3540182725&feature=0 HTTP/1.1" 200 None
[weibo][warning] Failed to open download archive at 'weibo_archive' ('ValueError: Only '.' or '[' may follow ']' in format field specifier')
[weibo][debug] Active postprocessor modules: [MtimePP]
# .\gallery-dl\feed\4891461072719599_01.mp4
# .\gallery-dl\feed\4909980135724607_01.jpg
# .\gallery-dl\feed\4907794376163399_01.jpg
# .\gallery-dl\feed\4907794376163399_02.jpg
# .\gallery-dl\feed\4907794376163399_03.jpg
# .\gallery-dl\feed\4907794376163399_04.jpg
# .\gallery-dl\feed\4900927921719690_01.jpg
# .\gallery-dl\feed\4900927921719690_02.jpg
# .\gallery-dl\feed\4900927921719690_03.jpg
# .\gallery-dl\feed\4900927921719690_04.jpg
[weibo][debug] Using WeiboVideosExtractor for 'https://weibo.com/u/3540182725?tabtype=video'
[weibo][debug] Sleeping 0.01 seconds (request)
[urllib3.connectionpool][debug] https://weibo.com:443 "GET /ajax/profile/getprofilevideolist?uid=3540182725 HTTP/1.1" 200 None
[weibo][warning] Failed to open download archive at 'weibo_archive' ('ValueError: Only '.' or '[' may follow ']' in format field specifier')
[weibo][debug] Active postprocessor modules: [MtimePP]
# .\gallery-dl\videos\4891461072719599_01.mp4
# .\gallery-dl\videos\4726575591852999_01.mp4
# .\gallery-dl\videos\4659906911670262_01.mp4
# .\gallery-dl\videos\4621850188189004_01.mp4
# .\gallery-dl\videos\4587087011256368_01.mp4
[weibo][debug] Sleeping 0.12 seconds (request)
[urllib3.connectionpool][debug] https://weibo.com:443 "GET /ajax/profile/getprofilevideolist?uid=3540182725&cursor=-1 HTTP/1.1" 200 44
[weibo][debug] Using WeiboNewvideoExtractor for 'https://weibo.com/u/3540182725?tabtype=newVideo'
[weibo][debug] Sleeping 0.72 seconds (request)
[urllib3.connectionpool][debug] https://weibo.com:443 "GET /ajax/profile/getWaterFallContent?uid=3540182725 HTTP/1.1" 200 None
[weibo][warning] Failed to open download archive at 'weibo_archive' ('ValueError: Only '.' or '[' may follow ']' in format field specifier')
[weibo][debug] Active postprocessor modules: [MtimePP]
# .\gallery-dl\newvideo\4891461072719599_01.mp4
# .\gallery-dl\newvideo\4860072381389020_01.mp4
# .\gallery-dl\newvideo\4855374034763817_01.mp4
# .\gallery-dl\newvideo\4854505982395567_01.mp4
# .\gallery-dl\newvideo\4852436559469072_01.mp4
# .\gallery-dl\newvideo\4818259856329281_01.mp4
# .\gallery-dl\newvideo\4777325641598115_01.mp4
# .\gallery-dl\newvideo\4767887167064567_01.mp4
# .\gallery-dl\newvideo\4767650519192263_01.mp4
# .\gallery-dl\newvideo\4727429660675118_01.mp4
[weibo][debug] Using WeiboAlbumExtractor for 'https://weibo.com/u/3540182725?tabtype=album'
[weibo][debug] Sleeping 0.10 seconds (request)
[urllib3.connectionpool][debug] https://weibo.com:443 "GET /ajax/profile/getImageWall?uid=3540182725 HTTP/1.1" 200 None
[weibo][debug] Sleeping 0.17 seconds (request)
[urllib3.connectionpool][debug] https://weibo.com:443 "GET /ajax/statuses/show?id=4909980135724607 HTTP/1.1" 200 None
[weibo][warning] Failed to open download archive at 'weibo_archive' ('ValueError: Only '.' or '[' may follow ']' in format field specifier')
[weibo][debug] Active postprocessor modules: [MtimePP]
# .\gallery-dl\album\4909980135724607_01.jpg
[weibo][debug] Sleeping 0.63 seconds (request)
[urllib3.connectionpool][debug] https://weibo.com:443 "GET /ajax/statuses/show?id=4907794376163399 HTTP/1.1" 200 None
# .\gallery-dl\album\4907794376163399_01.jpg
# .\gallery-dl\album\4907794376163399_02.jpg
# .\gallery-dl\album\4907794376163399_03.jpg
# .\gallery-dl\album\4907794376163399_04.jpg
[weibo][debug] Sleeping 0.12 seconds (request)
[urllib3.connectionpool][debug] https://weibo.com:443 "GET /ajax/statuses/show?id=4900927921719690 HTTP/1.1" 200 None
# .\gallery-dl\album\4900927921719690_01.jpg
# .\gallery-dl\album\4900927921719690_02.jpg
# .\gallery-dl\album\4900927921719690_03.jpg
# .\gallery-dl\album\4900927921719690_04.jpg
# .\gallery-dl\album\4900927921719690_05.jpg

I am getting two errors

  1. weibo extractor is failing with unknown error at first
  2. failing to open my archive
@AdamSaketume26
Copy link
Author

also the meta data for for the media is not working when i do --mtime-from-date it saves the date at the same time i downloaded them and it doesnt make it so it uses the date they were posted

@mikf
Copy link
Owner

mikf commented Jun 26, 2023

[weibo][debug] b'{}'
[weibo][error] "unknown error"

tabtype=home feeds are broken on Weibo. Just remove it from your include list.

archive-prefix="[{category}] {user[status[user][screen_name]]} ({status[user][id]})"

Try [{category}] {status[user][screen_name]} ({status[user][id]})

not working when i do --mtime-from-date

For weibo, --mtime-from-date doesn't work out of the box.
You need to set value to "{status[date]}", see #2721 and #2739.

--postprocessor-option value="{status[date]}"

@AdamSaketume26
Copy link
Author

removing home feed and changing archive prefix worked thank you
but using post processor still did not work and still makes both date created and modified to the date and time it downloaded

gallery-dl --download-archive weibo_archive -o extractor.weibo.archive-prefix="[{category}] {status[user][screen_name]} ({status[user][id]})" -o extractor.weibo.archive-format="{subcategory}_{status[mblogid]}_{filename}" -o extractor.weibo.skip="false" -o extractor.weibo.include="videos" -o extractor.weibo.directory=[\"{subcategory}\"] -o extractor.weibo.sleep-request="0-1" --mtime-from-date --postprocessor-option value="status[date]" --verbose https://weibo.com/u/3540182725

i only did newvideo feed to test to see if it works properly since there is only few media to download. it doesnt work

@AdamSaketume26
Copy link
Author

fixed by using {status[date]} for value

@mikf mikf closed this as completed Jun 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants