Downloader fixes

[kapodamy]

Main changes:

• use the "Error Reporter" for non-http errors
• implement "metered networks"
• add buttons in "Downloads" activity to start/pause all pending downloads, ignoring the queue flag or if the network is "metered"
• add workaround for VPN connections and/or network switching.
• fix missing offset for KaxCluster (WebMWriter.java), manifested as no thumbnails on file explorers
• fixes Downloads via cell network even though WiFi is available  #247 close [Suggestion]Tell me if I've already downloaded a video when I go to download it so I don't download it twice. #2063 (already implemented)

Download queue:

• remember queue status, unless the user pause the download (un-queue)
• semi-automatic downloads, between networks. Effective if the user create a new download or the downloads activity is starts
• allow enqueue failed downloads
• new option, queue limit, enabled by default. Used to allow one or multiple downloads at same time
• hopefully fixes the All downloads start as "Queued"  #2072 Downloads adding in pause #2290 issues

Miscellaneous:

• fix crash while selecting details/error menu (my mistake)
• allow overwrite paused downloads
• fix wrong icons for grid/list button in downloads. fixes Icons for Icon and List #2246
• add share option. close Share downloaded video file #1906
• check if the WiFi is metered. close Doesn't respect metered wifis #2006
• correct misspelled words in strings.xml (es) (cmn)
• fix crash during device shutdown
• declare downloads database scheme in the database doc
• fix double-quote escaping the download filename. fixes Error Downloading When Using Quotes in Filename #2229

Muxers:

• new mp4 muxer (from dash) with this, muxing on Android 7 is possible now!!! fixes [0.15.1] Downloading/Muxing files cause App crash #2087 fixes Code: 1007 Error post processing failed. #2279 fixes Length of downloaded audio not saved #274
• strip m4a DASH files to normal m4a format (for youtube) works on winamp 😃 . close M4A audio download and Dash header issues #2152
• download the audio stream first
• close [feature] Remuxing for video-only streams for downloading #1262

Storage changes:

• warn to the user if is choosing a "read only" download directory (for external SD Cards), useless is patched :)
• deal with "insufficient storage" error in downloads
• implement Storage Access Framework. resolves No support for the "SD Card Write Access" on Android 6.0 #2230
• SAF Tree API support on Android Lollipop or higher
• storage helpers: StoredDirectoryHelper & StoredFileHelper
• best effort to handle any kind of SAF errors/exceptions
• fixes WebM muxer fails for some videos #2114 fixes Video Download Glitch #2190 fixes Download location settings change requires app restart #1730 fixes Newpipe Stuck at post-processing donwloaded videos #2264

Downloader related issues that are already fixed from other PRs:

• close [FR] Download with NewPipe intent #281 close Wrong order of downloads after restarting the app #645 close On long press on video, add Download option. #1952 close Checksum bug newpipe #2077

Screenshots

[TobiGr]

@kapodamy Thanks for your work! Is there any specific reason for closing this PR?

[Flashwalker]

Downloads doesn't auto starts any more until call menu for each download and select start it. Is it bug or feature?

[kapodamy]

@TobiGr yes, needs to polish some things and i was very busy

[kapodamy]

implements #2230 (WIP)

[TobiGr]

That's good to hear! 👍
I am working on some the documentation about the database atm. @theScrabi created the EERM for the "normal" app database some time ago by (see #2136 for some info on the tool he used). Can you please also add the scheme for the download database to the assets directory once you have finished the PR? This would be time saving for me and I guess just five minutes work for you as you refactored the db in the lasts month.
Feel free to ping us once your done with this huge PR so we can start testing :)

[kapodamy]

ready for review and testing

[kapodamy]

@Flashwalker 50/50 the implementation is not well thought out

[TobiGr]

Nice work 👌 Tons of bugfixes, improvements and new features! Can you please provide an APK so other people can start testing, too?

I want to release 0.16.2 soon, so I will not review this in detail soon. But from taking a look at the screenshots and reading the description, I'd like to raise some points for discussion.

1. You posted the table which shows the differences between SAF-, SAF and Java IO. This is good for us to understand the advantages and new features. However, how can users understand what is best for them? Do you use an algorithm to detect whether SAF or IO is the best thing to use or is this completely dependent on the settings entry? If yes, how can users understand what to choose?

2. IIRC the clear button in the downloads wipes the database, but not the downloads from the phone. If this is still the case, we should change the description of the dialog.

[theScrabi]

@kapodamy @TobiGr well SAF is usefull if you want to save something on the SD card, while java IO can be used for saving something on the app internal storage as well as on the emulated sdcard (internal storage). As we do not need to save downloads to the app internal storage I suggest to move over to SAF completely. The user should not be getting in touch weather he wants to use Java IO or SAF.

[TobiGr]

The user should not be getting in touch weather he wants to use Java IO or SAF.

That's exactly what I wanted to say.

[kapodamy]

@TobiGr

1. I had not thought about it. SAF should be used by default and keep java IO for older devices

2. yes the button does not delete the files, only clear the database. maybe it was not understood why I do not speak english or if there is a guideline that says the opposite.

[theScrabi]

Well SAF was introduced with android 4.4 in theorie we can remove javaIO, the question is just, how much do we want to keep for the legacy app. Id be for removing, javaIO, the legacy app should then use an older version of the downloader. What do you think?

[kapodamy]

@theScrabi keep javaIO. SAF directory API is not supported on older devices.
Also, with lastest changes javaIO only will be used on older devices.

Kitkat:

Lollipop or newer (external sd card write is always available):

[kapodamy]

pr rebased (near dead) done

[TobiGr]

Thanks, I'll test and merge once 0.16.2 is released. So hopefully this weekend.

[TobiGr]

Thanks for enhancing the download process yet again!

[TobiGr]

About the second bug, I cannot reproduce it either (neither on Kitkat, Lollipop nor Oreo). I'd suggest to merge this and then bring out a release candidate for 0.17 soon, so we have a wider range of devices and android versions to test with.
@kapodamy What do you think?

[kapodamy]

ok, sound good

[theScrabi]

@kapodamy There is still abounty open that has the size of about 300$ concerning this toppic:
https://www.bountysource.com/issues/53420964-refactor-download-manager

Related issue: #962

If we don't pay it out the money is lost so... :D Your interested?

[TobiGr]

@theScrabi You are absolutely right. @kapodamy has done an anwesoe job 👍

[kapodamy]

@theScrabi yes

[theScrabi]

You may need to claim the money so we can say yes to the claim.