more dl features by kapodamy · Pull Request #2679 · TeamNewPipe/NewPipe

kapodamy · 2019-10-01T17:32:30Z

This pull request does:

download opus (webm files) as ogg. close Repack OPUS to OGG container #2542 close Regenerate expired links #938
recover expired downloads, resolve Newpipe doesn't reload the expired links #2451 fix Allow resume of download #345
show a proper description on the error reporter when a download fails
possible fix for Cut off m4a downloads #2640 (cut-off audio in m4a). fixes Downloading large capacity videos will fail #2111
possible fixes for App crash when downloading: permission denied (0.17.2) #2634 and Unable to generate unique name. Location is not a directory #2429
calculate eta

required:

NPE change (use webm as format for opus audio streams NewPipeExtractor#199)
seeking is not available in webm (regresion from Downloader fixes #2149) fixes Videos in WebM and 3GP containers are not seekable #2676

[ ✔ ] I carefully read the contribution guidelines and agree to them.

opusforlife2 · 2019-10-02T10:57:44Z

Would it be possible to retain .opus for Ogg Opus files? Or is that a hard requirement for the fix?

It's not a deal breaker, but I find it convenient to tell apart between my Ogg Vorbis and Ogg Opus files by looking at the extensions rather than having to open MediaInfo or similar utilities.

Ogg Vorbis streams, if present, should certainly have the .ogg (or .oga) extension.

kapodamy · 2019-10-02T16:18:42Z

@opusforlife2 yes, is possible

TobiGr · 2019-10-10T12:03:05Z

@kapodamy The change list sounds interesting. Please ping me when you're done.

kapodamy · 2019-10-12T17:03:06Z

@TobiGr can you take a look on TeamNewPipe/NewPipeExtractor#199

TobiGr · 2019-11-24T12:34:04Z

@kapodamy I'll merge the extractor PR after a rebase. Can you rebase this PR, too? We won't wait for the other PRs because their authors do not respond / apply the requested changes.

kapodamy · 2019-11-24T14:57:23Z

@TobiGr ok

* used for opus audio stream * update WebMReader and WebMWriter * new post-processing algorithm

* reduce the number of iterations over the output file (less seeking) * fix audio samples with size of 255 do not handled correctly in the segment table (allows writing audio streams with 70kbps and 160kbps bitrate) * add support for VORBIS codec metadata * write packets based on the timestamp

* recovery infrastructure * bump serialVersionUID of DownloadMission * misc cleanup in DownloadMission.java * remove unused/redundant from strings.xml

* correct calculation of "co64" box and usage of 64bits offsets * generate one chunk for audio streams like ffmpeg does, attempt to fix cut-off audio * misc. cleanup

* OggFromWebMWriter: rewrite (again), reduce iterations over the input. Works as-is (video streams are not supported) * WebMReader: use int for SimpleBlock.dataSize instead of long * Download Recovery: allow recovering uninitialized downloads * check range-requests using HEAD method instead of GET * DownloadRunnableFallback: add workaround for 32kB/s issue, unknown issue origin, wont fix * reporting downloads errors now include the source url with the selected quality and format

* check if the directory pending_downloads was created

fix wrong cue generation

keep *.opus extension

* migrate few annotations to androidx * mission recovery: better error handling (except StreamExtractor.getErrorMessage() method always returns an error) * post-processing: more detailed progress [file specific changes] DownloadMission.java * remove redundant/boilerplate code (again) * make few variables volatile * better file "length" approximation * use "done" variable to count the amount of bytes downloaded (simplify percent calc in UI code) Postprocessing.java * if case of error use "ERROR_POSTPROCESSING" instead of "ERROR_UNKNOWN_EXCEPTION" * simplify source stream init DownloadManager.java * move all "service message sending" code to DownloadMission * remove not implemented method "notifyUserPendingDownloads()" also his unused strings DownloadManagerService.java * use START_STICKY instead of START_NOT_STICKY * simplify addMissionEventListener()/removeMissionEventListener() methods (always are called from the main thread) Deleter.java * better method definition MissionAdapter.java * better method definition * code cleanup * the UI is now refreshed every 750ms * simplify download progress calculation * indicates if the download is actually recovering * smooth download speed measure * show estimated remain time MainFragment.java: * check if viewPager is null (issued by "Apply changes" feature of Android Studio)

TobiGr · 2019-11-24T21:29:12Z

java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.Throwable java.lang.Throwable.getCause()' on a null object reference
        at us.shandian.giga.get.DownloadMission.notifyError(DownloadMission.java:319)
        at us.shandian.giga.get.DownloadMission.notifyError(DownloadMission.java:308)
        at us.shandian.giga.get.DownloadRunnable.run(DownloadRunnable.java:145)

This happened when I turned off wifi while downloading. I got the normal error screen. I might be better to modify the notification to promt the error. When taping on the notification or the unfinished item in the downloads view, more info should be displayed.

TobiGr · 2019-11-24T21:37:02Z

I know there is few space. But @ seems to be wrong here. Maybe something like 0:42 left/remaining

* used for opus audio stream * update WebMReader and WebMWriter * new post-processing algorithm

* reduce the number of iterations over the output file (less seeking) * fix audio samples with size of 255 do not handled correctly in the segment table (allows writing audio streams with 70kbps and 160kbps bitrate) * add support for VORBIS codec metadata * write packets based on the timestamp

* recovery infrastructure * bump serialVersionUID of DownloadMission * misc cleanup in DownloadMission.java * remove unused/redundant from strings.xml

* correct calculation of "co64" box and usage of 64bits offsets * generate one chunk for audio streams like ffmpeg does, attempt to fix cut-off audio * misc. cleanup

* OggFromWebMWriter: rewrite (again), reduce iterations over the input. Works as-is (video streams are not supported) * WebMReader: use int for SimpleBlock.dataSize instead of long * Download Recovery: allow recovering uninitialized downloads * check range-requests using HEAD method instead of GET * DownloadRunnableFallback: add workaround for 32kB/s issue, unknown issue origin, wont fix * reporting downloads errors now include the source url with the selected quality and format

* check if the directory pending_downloads was created

fix wrong cue generation

keep *.opus extension

* migrate few annotations to androidx * mission recovery: better error handling (except StreamExtractor.getErrorMessage() method always returns an error) * post-processing: more detailed progress [file specific changes] DownloadMission.java * remove redundant/boilerplate code (again) * make few variables volatile * better file "length" approximation * use "done" variable to count the amount of bytes downloaded (simplify percent calc in UI code) Postprocessing.java * if case of error use "ERROR_POSTPROCESSING" instead of "ERROR_UNKNOWN_EXCEPTION" * simplify source stream init DownloadManager.java * move all "service message sending" code to DownloadMission * remove not implemented method "notifyUserPendingDownloads()" also his unused strings DownloadManagerService.java * use START_STICKY instead of START_NOT_STICKY * simplify addMissionEventListener()/removeMissionEventListener() methods (always are called from the main thread) Deleter.java * better method definition MissionAdapter.java * better method definition * code cleanup * the UI is now refreshed every 750ms * simplify download progress calculation * indicates if the download is actually recovering * smooth download speed measure * show estimated remain time MainFragment.java: * check if viewPager is null (issued by "Apply changes" feature of Android Studio)

…into dl-last-features

kapodamy · 2019-11-26T14:21:50Z

@TobiGr ~~cannot test NPE is not working for me~~ nvm, npe doesnt work with lastest jsoup

kapodamy · 2019-11-26T16:37:32Z

now the ETA looks like

* rebase fixup, add null check * better ETA string * drop connection read timeout, for HSDPA networks * bump NPE version

TobiGr

The ETA looks good now 👍
There are some smaller things, but they are easy to fix

app/src/main/java/org/schabi/newpipe/fragments/MainFragment.java

app/src/main/java/us/shandian/giga/get/MissionRecoveryInfo.java

app/src/main/java/org/schabi/newpipe/streams/OggFromWebMWriter.java

app/src/main/java/org/schabi/newpipe/streams/WebMReader.java

TobiGr · 2019-12-06T23:44:21Z

Can you please check, why WebM with 720p downloads have no audio track? The same video with audio track is available when using 0.17.4, but not on your latest commit.

use +webm_opus instead of +opus

TobiGr

Thank you again for your ongoing work on the downloader

benymos · 2020-01-27T18:33:39Z

Je ne comprends rien, il n'y a pas un lien pour télécharger newpipe 0.18.2

Stypox · 2020-01-27T19:12:22Z

@benymos download it from here: https://github.com/TeamNewPipe/NewPipe/releases/tag/v0.18.2
Click on the NewPipe_v0.18.2.apk button in the Assets menu.

Translated into French using DeepL // Traduit en français à l'aide de DeepL:
@benymos le télécharger ici: https://github.com/TeamNewPipe/NewPipe/releases/tag/v0.18.2
Cliquez sur le bouton NewPipe_v0.18.2.apk dans le menu Assets.

kapodamy changed the title ~~more dl features~~ [WIP] more dl features Oct 1, 2019

TobiGr added the downloader Issue is related to the downloader label Oct 1, 2019

kapodamy changed the title ~~[WIP] more dl features~~ more dl features Oct 13, 2019

kapodamy added 11 commits November 24, 2019 12:21

implement webm to ogg demuxer

52a21e4

* used for opus audio stream * update WebMReader and WebMWriter * new post-processing algorithm

long-term downloads resume

c891f2f

* recovery infrastructure * bump serialVersionUID of DownloadMission * misc cleanup in DownloadMission.java * remove unused/redundant from strings.xml

Mp4FromDashWriter fixes

429ee7e

* correct calculation of "co64" box and usage of 64bits offsets * generate one chunk for audio streams like ffmpeg does, attempt to fix cut-off audio * misc. cleanup

fallback for pending downloads directory

d092e39

update DownloadManager.java

9339fc8

* check if the directory pending_downloads was created

update WebMWriter.java

94e2314

fix wrong cue generation

update DownloadDialog.java

844f80a

keep *.opus extension

Merge branch 'dev' into dl-last-features

ea1be11

kapodamy added 8 commits November 26, 2019 10:38

Merge branch 'dev' into dl-last-features

14de2f2

implement webm to ogg demuxer

773aa1e

* used for opus audio stream * update WebMReader and WebMWriter * new post-processing algorithm

long-term downloads resume

86dafdd

* recovery infrastructure * bump serialVersionUID of DownloadMission * misc cleanup in DownloadMission.java * remove unused/redundant from strings.xml

Mp4FromDashWriter fixes

5707381

* correct calculation of "co64" box and usage of 64bits offsets * generate one chunk for audio streams like ffmpeg does, attempt to fix cut-off audio * misc. cleanup

fallback for pending downloads directory

60d4c8a

update DownloadManager.java

da052df

* check if the directory pending_downloads was created

kapodamy added 5 commits November 26, 2019 10:46

update WebMWriter.java

8a992d4

fix wrong cue generation

update DownloadDialog.java

763995d

keep *.opus extension

Merge branch 'dev' into dl-last-features

3ca4614

Merge branch 'dl-last-features' of https://github.com/kapodamy/NewPipe …

0033843

…into dl-last-features

commit

84ec320

* rebase fixup, add null check * better ETA string * drop connection read timeout, for HSDPA networks * bump NPE version

TobiGr reviewed Dec 2, 2019

View reviewed changes

TobiGr mentioned this pull request Dec 2, 2019

Android TV support #2806

Merged

1 task

kapodamy added 2 commits December 5, 2019 14:04

remove unused imports

aae8865

remove netbeans editor-fold comments

5a2cd93

add missing change after updating NPE

0393955

use +webm_opus instead of +opus

TobiGr approved these changes Dec 7, 2019

View reviewed changes

TobiGr merged commit c2b1d45 into TeamNewPipe:dev Dec 7, 2019

This was referenced Dec 12, 2019

Release 0.18.0 #2861

Closed

Release 0.18.0 #2862

Merged

opusforlife2 mentioned this pull request Dec 28, 2019

[Bug] Downloaded Opus files still have the extension .webm instead of .opus #2904

Closed

opusforlife2 mentioned this pull request Jan 8, 2020

Cut off m4a downloads #2640

Closed

ksio89 mentioned this pull request Mar 8, 2020

Some WebM videos are still not seekable in 0.18.6 #3204

Closed

opusforlife2 mentioned this pull request Mar 25, 2020

Cannot seek in an opus file downloaded using Newpipe #2340

Closed

2 tasks

Uh oh!

Conversation

kapodamy commented Oct 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

opusforlife2 commented Oct 2, 2019

Uh oh!

kapodamy commented Oct 2, 2019

Uh oh!

TobiGr commented Oct 10, 2019

Uh oh!

kapodamy commented Oct 12, 2019

Uh oh!

TobiGr commented Nov 24, 2019

Uh oh!

kapodamy commented Nov 24, 2019

Uh oh!

TobiGr commented Nov 24, 2019

Uh oh!

TobiGr commented Nov 24, 2019

Uh oh!

kapodamy commented Nov 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kapodamy commented Nov 26, 2019

Uh oh!

TobiGr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TobiGr commented Dec 6, 2019

Uh oh!

TobiGr left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

benymos commented Jan 27, 2020

Uh oh!

Stypox commented Jan 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kapodamy commented Oct 1, 2019 •

edited

Loading

kapodamy commented Nov 26, 2019 •

edited

Loading

TobiGr left a comment •

edited

Loading