[youtube] premature download termination with --live-from-start #2898
Comments
|
For the second attempt, was the live ended at the middle of download (in other words, was count of fragments grown rapidly)? It's still strange though. memo to myself: run live for more than 1hs and start dl, then stop live after few mins of dl |
|
I use -N4 because I believed fragment downloads were throttled, is this not the case? I wanted the download to catch up before the stream ended. The last valid fragment should have been 4763. I'm not sure what you mean with "was count of fragments grown rapidly"? On disk, only up to fragment 2618 was downloaded for each format stream, for the second download. Based on ctimes, both format streams were caught up to the live after fragment 2235. As a side note, I'm curious as to why all the fragments on disk have the same mtime, as it seems like the fragment files on disk had their mtime set to the stream's start time. The final outfit file notably does not have the same mtime; it appears to be the time the download actually ended. |
|
I don't know about mtime thing. |
|
It may have been an untrapped URLError or OSError; with the updated HttpFD after #3065 the downloader caught an error from one of the stream downloads when my host roamed from an unresponsive router (AP): Presumably before this would just kill one of the streams and leave the other stream to download by itself if it didn't end up aborting the whole download. Initially I thought this would be caught by replacing HTTPError with URLError in fragment.py, but a closer look reveals it to be handled by HttpFD, so there may be nothing to fix anymore. |
|
@Lesmiscore You were to one who triaged this. Were you able to reproduce the issue? |
This comment was marked as outdated.
This comment was marked as outdated.
|
#3065 likely fixed this. Can be reopened if anyone can reproduce |
|
I can still reproduce this; in my case it was on a YouTube stream where the stream is lagging. I added some local debugging info to try and hunt down why, although I still lack a complete picture. First off, the streams "die" (stop generating) on their own volition, when When the fragment download fails, the MPD manifest is refetched via _extract_mpd_formats_and_subtitles. Upon doing so, the list of available formats can change:
There is a brief period (up to 10 seconds?) right when a livestream actually ends where all the formats may disappear before the stream no longer is live. Due to self-throttling for the manifest download this brief period could be missed. When the formats can be refreshed, new formats will appear:
I believe the throttling on refresh_manifest may be causing no_fragment_score to increment rapidly (beyond the 30 threshold) and causing the stream download to stop.
I likely made a few mistakes; feel free to point them out. |
|
Hello, I ran into a similar issue where the download resumes as normal after a transient error, however the final video is truncated. Reproduction steps:
|
|
I'm fairly certain this has to do with portrait vs landscape streams. portrait: landscape: landscape updated fragments a few times already. |
|
I highly doubt that to be the case, and if it is, then it's an unrelated issue. Perhaps the relative popularity of the stream is causing YT to hiccup. On this topic, I've recently discovered that if a stream is experiencing difficulties (say, OBS issues/crashes), then the timestamp can restart (as evidenced by the 't' of stats for nerds). Additionally, it's possible for formats to disappear and change (changed FPS? format? resolution?). Either of those cases will cause the current active download to fail to reload and eventually end the download. If you happen to have --keep-fragments, then restarting the stream download will erroneously reuse the downloaded fragments when it chases up to the current point again, which will result in missing part of the stream. This could theoretically be detectable and addressable, but it would be a lot of managing logic (complexity) for rather little gain. If the stream goes offline temporarily, obviously yt-dlp doesn't like that, but that is another way for it to end "prematurely". I haven't checked the download logic (or this issue) in ages so I don't really know what's happening anymore, but a retry sleep for everything helps for some of the more common issues (internet flickers, transient resolver issues). Other issues, you just have to keep an active eye on it. |
2024.02.18.2.mp4sorry for the low bit rate. I forgot to set the quality for desktop. I don't want to redo it again :S <3 maybe we're talking about two different issues here then I will open a separate issue, if issue worthy. at the start you can see the audio portion is at 673 then it jumps to 762 until it catches up to the live stream. that's two for two from my side already, no interruptions this time. it catches up to the audio part and it stops the audio, continues the video. I wouldn't be surprised if original poster's stream from #9097 is a portrait one too. |
|
https://gofile.io/d/ywl4VD bit rate again too low. thought it was going to be enough. is there any workaround for this? How do I rename the output file without overwriting the fragments? would I need to use batch for this somehow or can this be done within yt-dlp: to keep the fragments but keep overwriting the merge file in order to keep downloading it re-using the fragments. can't use |
frankly speaking, in bash this is what I do. I poll live_status after yt-dlp exits and if still live, try again, after I mv the old format files out of the way. On a minor note, if YT starts counting fragments from 1 again then using keep-fragments will be trouble, but honestly that case isn't very common. You could try the advice listed in the comment here: #9043 (comment) Your specific issue seems to be: It seems to occur when page reextraction is triggered, and from downloading the refreshed android player JSON, if I'm not mistaken. This does seem to be a new error. "Precondtion check" personally makes me think it may be related to cache or timestamping, likely internally on YouTube's end. If I'm not mistaken, format 135 is a 2K resolution? The download might be slowing the audio download (format 140) down. I noticed you didn't have any -N (parallels) flag in your second video. YouTube throttles DASH formats (a known issue), you may want to provide -N4 or greater to allow the downloads to catch up faster. |
ok sorry for noob question. I'm not following. how do I continue then my download if I already have a merged file aka .m4a/.mp4 (from the fragments, because it starts merging after the fail), I don't somehow delete it and don't keep fragments? or: how do I not start from frag 1 or what's the opposite of fragment 1 method? to my understanding ffmpeg won't even be able to create a file without --force-overwrites (which also rewrites fragments). That's why I'd like to plead to please look into portrait streams. This wouldn't be necessary if the download wasn't being discarded for whatever reason (if it's something you can fix on yt-dlp's end, considering landscape doesn't have this problem).
.... it was another portrait stream. Only happens in portrait. I invite you to find portrait streams with your mobile and try to download even one media stream (audio/video).
doesn't matter if single or dual media streams.
it's low. 480 or so. I guess the most feasible workaround for now is not to use I will try |
|
I have an interesting thing I just found out. I had/have the same problems. Since I have a bunch of old laptops laying around and a mac mini besides my Pc I decided to run some tests having them all run on the same stream with some different settings (all with --live-from-start) and see if I find something that works. long story short all Windows PCs constantly failed while on mac it worked and I have repeated it for 4 streams now. Successfully got the stream in a .mkv 1440p60 with audio and everything working everytime. |
|
@rattailgamer how did you install yt-dlp on each of the machines? could you show the output of |
|
@bashonly I have installed it via a gui to save time (Tartube) and their guide for each. While Tartube is gives out the full output including debugging mode unfortunately I can't enter commands manually or at least I haven't figured it out. But once I find time next week I will try to re do it all via ytdlp directly and send the logs. |
Checklist
Description
(URLs are no longer playable.)
Unfortunately I did not have an HLS download, but no playback issue was noticed in the browser.
Generally speaking, it seems possible for --live-from-start to prematurely terminating the download of each stream, resulting in a truncated or corrupted file.
First download, video lasted longer than the audio, 01:19:23.000 vs 00:52:58.998 (I believe the video is full-length). On playback in mpv, the following warning is logged, at the 53 minute mark:
Seeking beyond that point is considerably slower.
Second download, video and audio downloads terminated even earlier, at 00:43:38:00. The fragments after that point were never downloaded.
I suspect the reworking of --live-from-start may be at fault. It may also be another issue somewhere in the downloader as well, I do not know.
I do know, however, that based on the logs the interruption seems tied to a spontaneous webpage download in the middle of downloading the fragments. Presumably when this happens the stream download is permanently interrupted.
Verbose log
The text was updated successfully, but these errors were encountered: