-
-
Notifications
You must be signed in to change notification settings - Fork 517
Add a proper support of other delivery methods than progressive HTTP #663
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
71 commits
Select commit
Hold shift + click to select a range
e72f207
Add support of other delivery methods than progressive HTTP
AudricV 83d7f94
Parse YouTube DASH manifest
AudricV 2998d73
Apply changes in extractor services
AudricV 78dd67d
Apply changes in DefaultStreamExtractorTest
AudricV 4017b4d
Update SoundcloudStreamExtractorTest and fix some warnings in some of…
AudricV 12be50a
Readd itags 134 and 136 and format the ItagItem class
AudricV 1efaf24
Add a baseUrl field to Stream classes
AudricV da07e31
Add POST_LIVE_STREAM and POST_LIVE_AUDIO_STREAM to StreamType enum
AudricV 0b1c787
Update livestream mocks and add instructions how to run YoutubeStream…
AudricV c69ecb9
[PeerTube] Return HLS variants of video streams
AudricV 2ec6a8a
Remove some unused code
AudricV c218899
[YouTube] Generate DASH manifests of OTF streams and extract subtitle…
AudricV 654d2b4
Fix a typo in SubtitlesStream
AudricV 00b78ea
Adress some requested changes + extract correctly PeerTube audio streams
AudricV c659476
Change a message in PeertubeStreamExtractorTest
AudricV 8a721c2
[YouTube] Generate DASH manifests for streams of ended livestreams (p…
AudricV 9b0802b
Fix some comments
AudricV 643f191
Add static constant UNKNOWN_BITRATE in AudioStream and use it in stre…
AudricV 87b270b
Add missing JavaDocs for Stream classes
AudricV a14a9fa
[YouTube] Use really to the last URL to prevent redirects when stream…
AudricV 9a90cd7
[YouTube] Return the POST_LIVE_STREAM stream type
AudricV 9517a0f
[PeerTube] Readd support of livestreams
AudricV e1bc190
Make the ItagItem class Serializable
AudricV 99ce0bf
Add a getItagItem method in Stream classes and store the ItagItem obj…
AudricV e0678c4
[YouTube] Use less requests when generating DASH manifests of OTF str…
AudricV 2a6f3dd
[YouTube] Don't generate DASH manifests for OTF streams when loading …
AudricV b0147f5
[YouTube] Cache the manifests generated with YoutubeDashManifestCreat…
AudricV 11ddc07
Add the missing Deprecated annotation on the getUrl method of the Str…
AudricV 0edaeba
[YouTube] Only fetch the Android mobile API for video streams
AudricV 4c0fd40
Rename the ContentAndItagItemAndIsUrl class to ItagInfo
AudricV e783c34
[YouTube] Don't generate manifests for post live streams when loading…
AudricV 6fcec47
Add STREAMING_DATA, SIGNATURE_CIPHER and CIPHER constants in YoutubeS…
AudricV 621a79f
Use the containsKey method instead of the get method in YoutubeDashMa…
AudricV a17f933
Improve the SEGMENT_DURATION_MS_PATTERN regular expression in YoutuDa…
AudricV 371ff75
[YouTube] Extract the audioChannels and the approxDurationMs values
AudricV fa2a30b
Adress requested changes in StreamInfo
AudricV 642e457
[Media.ccc.de] Improve stream parsing of livestreams
AudricV e8ea26b
Make the MediaFormat in Stream objects nullable + improve some code i…
AudricV 1be9160
Adress requested changes in YoutubeStreamExtractor
AudricV 20b393c
Add a constant for the string &sq=0 in YoutubeDashManifestCreator
AudricV da73de7
Write a better test for YoutubeDashManifestCreator
AudricV 0a8f78a
Fix some tests and a RESOLUTION_UNKNOWN constant string
AudricV 78146fa
Finish an unfinished JavaDoc in MediaCCCLiveStreamExtractor
AudricV 10d6854
Use braces in all if statements of DefaultStreamExtractorTest and Str…
AudricV c0bbec6
Remove an unused import in YoutubeStreamExtractor
AudricV ef8fdcf
[SoundCloud] Extract the downloadable file when available
AudricV d18e537
Fix failing tests + return empty lists instead of null
AudricV c34e0b1
[PeerTube] Improve code of stream extraction, fix some warnings and t…
AudricV 0ff33a9
Remove a useless attribute in DASH manifests generated
AudricV deeea85
[YouTube] Fix buffering when playing streams from the WEB client and …
AudricV c0bccbc
[YouTube] Spoof better official clients when generating and fetching …
AudricV 8a1a6a1
[YouTube] Always use again Android streams first
AudricV 91def33
[YouTube] Extract the content length field of streams when available
AudricV 9686aa7
[YouTube] Add ability to generate DASH manifests for YouTube progress…
AudricV eaa70a1
Apply requested changes and cache audio and video streams and HLS and…
AudricV 76f6dc5
[YouTube] Parse a long instead of an integer when setting the content…
AudricV 3a309a1
[YouTube] Apply documentation suggestions to YoutubeDashManifestCreator
AudricV 51c19f8
[YouTube] Fix a bug when using resetClientVersionAndKey method
AudricV 50b74c6
[YouTube] Fix StackOverflow error when generating manifests of OTF st…
AudricV ee673f7
Apply suggested changes on StreamInfo
AudricV c08b8dc
Adress requested changes, remove the use of StandardCharsets and impr…
AudricV 8357f6f
Override hashCode and equals methods on Stream classes
AudricV 7d1cafa
Add ability to set maximum size of caches
AudricV bdd9341
Update YoutubeDashManifestCreatorTest to Junit 5
AudricV f773190
Add basic tests for ManifestCreatorCache in UtilsTest
AudricV 4e59ea8
Add serialVersionUID to Serializable classes changed
AudricV f650808
Add the SmoothStreaming delivery method to the available delivery met…
AudricV 77026d9
Add documentation in the StreamType enum
AudricV 09bf96e
Don't remove public access of averageBitrate field in AudioStream and…
AudricV 3bec1f0
Apply some requested changes
AudricV bc33879
Adress new requested changes
AudricV File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -6,28 +6,29 @@ | |
| * Copyright (c) Christian Schabesberger <[email protected]> | ||
| * and Adam Howard <[email protected]> 2015 | ||
| * | ||
| * MediaFormat.java is part of NewPipe. | ||
| * MediaFormat.java is part of NewPipe Extractor. | ||
| * | ||
| * NewPipe is free software: you can redistribute it and/or modify | ||
| * NewPipe Extractor is free software: you can redistribute it and/or modify | ||
| * it under the terms of the GNU General Public License as published by | ||
| * the Free Software Foundation, either version 3 of the License, or | ||
| * (at your option) any later version. | ||
| * | ||
| * NewPipe is distributed in the hope that it will be useful, | ||
| * NewPipe Extractor is distributed in the hope that it will be useful, | ||
| * but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
| * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
| * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
| * GNU General Public License for more details. | ||
| * | ||
| * You should have received a copy of the GNU General Public License | ||
| * along with NewPipe. If not, see <http://www.gnu.org/licenses/>. | ||
| * along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>. | ||
| */ | ||
|
|
||
| import javax.annotation.Nullable; | ||
|
|
||
| /** | ||
| * Static data about various media formats support by NewPipe, eg mime type, extension | ||
| * Static data about various media formats support by NewPipe, e.g. mime type, extension. | ||
| */ | ||
|
|
||
| public enum MediaFormat { | ||
| //video and audio combined formats | ||
| // Video and audio combined formats | ||
| // id name suffix mime type | ||
| MPEG_4 (0x0, "MPEG-4", "mp4", "video/mp4"), | ||
| v3GPP (0x10, "3GPP", "3gp", "video/3gpp"), | ||
|
|
@@ -52,7 +53,7 @@ public enum MediaFormat { | |
| public final String suffix; | ||
| public final String mimeType; | ||
|
|
||
| MediaFormat(int id, String name, String suffix, String mimeType) { | ||
| MediaFormat(final int id, final String name, final String suffix, final String mimeType) { | ||
| this.id = id; | ||
| this.name = name; | ||
| this.suffix = suffix; | ||
|
|
@@ -66,9 +67,11 @@ public enum MediaFormat { | |
| * @return the friendly name of the MediaFormat associated with this ids, | ||
| * or an empty String if none match it. | ||
| */ | ||
| public static String getNameById(int ident) { | ||
| for (MediaFormat vf : MediaFormat.values()) { | ||
| if (vf.id == ident) return vf.name; | ||
| public static String getNameById(final int ident) { | ||
| for (final MediaFormat vf : MediaFormat.values()) { | ||
| if (vf.id == ident) { | ||
| return vf.name; | ||
| } | ||
| } | ||
| return ""; | ||
| } | ||
|
|
@@ -80,9 +83,11 @@ public static String getNameById(int ident) { | |
| * @return the file extension of the MediaFormat associated with this ids, | ||
| * or an empty String if none match it. | ||
| */ | ||
| public static String getSuffixById(int ident) { | ||
| for (MediaFormat vf : MediaFormat.values()) { | ||
| if (vf.id == ident) return vf.suffix; | ||
| public static String getSuffixById(final int ident) { | ||
| for (final MediaFormat vf : MediaFormat.values()) { | ||
| if (vf.id == ident) { | ||
| return vf.suffix; | ||
| } | ||
| } | ||
| return ""; | ||
| } | ||
|
|
@@ -94,9 +99,11 @@ public static String getSuffixById(int ident) { | |
| * @return the MIME type of the MediaFormat associated with this ids, | ||
| * or an empty String if none match it. | ||
| */ | ||
| public static String getMimeById(int ident) { | ||
| public static String getMimeById(final int ident) { | ||
| for (MediaFormat vf : MediaFormat.values()) { | ||
| if (vf.id == ident) return vf.mimeType; | ||
| if (vf.id == ident) { | ||
| return vf.mimeType; | ||
| } | ||
| } | ||
| return ""; | ||
| } | ||
|
|
@@ -107,9 +114,12 @@ public static String getMimeById(int ident) { | |
| * @return MediaFormat associated with this mime type, | ||
| * or null if none match it. | ||
| */ | ||
| public static MediaFormat getFromMimeType(String mimeType) { | ||
| for (MediaFormat vf : MediaFormat.values()) { | ||
| if (vf.mimeType.equals(mimeType)) return vf; | ||
| @Nullable | ||
| public static MediaFormat getFromMimeType(final String mimeType) { | ||
| for (final MediaFormat vf : MediaFormat.values()) { | ||
| if (vf.mimeType.equals(mimeType)) { | ||
| return vf; | ||
| } | ||
| } | ||
| return null; | ||
| } | ||
|
|
@@ -120,16 +130,22 @@ public static MediaFormat getFromMimeType(String mimeType) { | |
| * @param id the id | ||
| * @return the id of the media format or null. | ||
| */ | ||
| public static MediaFormat getFormatById(int id) { | ||
| @Nullable | ||
| public static MediaFormat getFormatById(final int id) { | ||
| for (MediaFormat vf : values()) { | ||
| if (vf.id == id) return vf; | ||
| if (vf.id == id) { | ||
| return vf; | ||
| } | ||
| } | ||
| return null; | ||
| } | ||
|
|
||
| public static MediaFormat getFromSuffix(String suffix) { | ||
| for (MediaFormat vf : values()) { | ||
| if (vf.suffix.equals(suffix)) return vf; | ||
| @Nullable | ||
| public static MediaFormat getFromSuffix(final String suffix) { | ||
| for (final MediaFormat vf : values()) { | ||
| if (vf.suffix.equals(suffix)) { | ||
| return vf; | ||
| } | ||
| } | ||
| return null; | ||
| } | ||
|
|
@@ -160,5 +176,4 @@ public String getSuffix() { | |
| public String getMimeType() { | ||
| return mimeType; | ||
| } | ||
|
|
||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -9,7 +9,6 @@ | |
| import org.jsoup.nodes.Element; | ||
| import org.jsoup.select.Elements; | ||
| import org.schabi.newpipe.extractor.MediaFormat; | ||
| import org.schabi.newpipe.extractor.MetaInfo; | ||
| import org.schabi.newpipe.extractor.StreamingService; | ||
| import org.schabi.newpipe.extractor.downloader.Downloader; | ||
| import org.schabi.newpipe.extractor.exceptions.ExtractionException; | ||
|
|
@@ -27,16 +26,17 @@ | |
| import java.util.ArrayList; | ||
| import java.util.Collections; | ||
| import java.util.List; | ||
| import java.util.Locale; | ||
|
|
||
| import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImageUrl; | ||
| import static org.schabi.newpipe.extractor.utils.Utils.*; | ||
|
|
||
| public class BandcampStreamExtractor extends StreamExtractor { | ||
|
|
||
| private JsonObject albumJson; | ||
| private JsonObject current; | ||
| private Document document; | ||
|
|
||
| private List<AudioStream> audioStreams = null; | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm still don't have a good feeling about caching here...
|
||
|
|
||
| public BandcampStreamExtractor(final StreamingService service, final LinkHandler linkHandler) { | ||
| super(service, linkHandler); | ||
| } | ||
|
|
@@ -83,13 +83,13 @@ public String getName() throws ParsingException { | |
| public String getUploaderUrl() throws ParsingException { | ||
| final String[] parts = getUrl().split("/"); | ||
| // https: (/) (/) * .bandcamp.com (/) and leave out the rest | ||
| return "https://" + parts[2] + "/"; | ||
| return HTTPS + parts[2] + "/"; | ||
| } | ||
|
|
||
| @Nonnull | ||
| @Override | ||
| public String getUrl() throws ParsingException { | ||
| return albumJson.getString("url").replace("http://", "https://"); | ||
| return albumJson.getString("url").replace(HTTP, HTTPS); | ||
| } | ||
|
|
||
| @Nonnull | ||
|
|
@@ -113,8 +113,11 @@ public DateWrapper getUploadDate() throws ParsingException { | |
| @Nonnull | ||
| @Override | ||
| public String getThumbnailUrl() throws ParsingException { | ||
| if (albumJson.isNull("art_id")) return ""; | ||
| else return getImageUrl(albumJson.getLong("art_id"), true); | ||
| if (albumJson.isNull("art_id")) { | ||
| return EMPTY_STRING; | ||
| } | ||
|
|
||
| return getImageUrl(albumJson.getLong("art_id"), true); | ||
| } | ||
|
|
||
| @Nonnull | ||
|
|
@@ -123,7 +126,7 @@ public String getUploaderAvatarUrl() { | |
| try { | ||
| return document.getElementsByClass("band-photo").first().attr("src"); | ||
| } catch (final NullPointerException e) { | ||
| return ""; | ||
| return EMPTY_STRING; | ||
| } | ||
| } | ||
|
|
||
|
|
@@ -132,7 +135,7 @@ public String getUploaderAvatarUrl() { | |
| public Description getDescription() { | ||
| final String s = Utils.nonEmptyAndNullJoin( | ||
| "\n\n", | ||
| new String[]{ | ||
| new String[] { | ||
| current.getString("about"), | ||
| current.getString("lyrics"), | ||
| current.getString("credits") | ||
|
|
@@ -143,13 +146,16 @@ public Description getDescription() { | |
|
|
||
| @Override | ||
| public List<AudioStream> getAudioStreams() { | ||
| final List<AudioStream> audioStreams = new ArrayList<>(); | ||
|
|
||
| audioStreams.add(new AudioStream( | ||
| albumJson.getArray("trackinfo").getObject(0) | ||
| .getObject("file").getString("mp3-128"), | ||
| MediaFormat.MP3, 128 | ||
| )); | ||
| if (audioStreams == null) { | ||
| audioStreams = new ArrayList<>(); | ||
| audioStreams.add(new AudioStream("mp3-128", | ||
| albumJson.getArray("trackinfo") | ||
| .getObject(0) | ||
| .getObject("file") | ||
| .getString("mp3-128"), | ||
| MediaFormat.MP3, | ||
| 128)); | ||
| } | ||
| return audioStreams; | ||
| } | ||
|
|
||
|
|
@@ -170,14 +176,11 @@ public StreamType getStreamType() { | |
|
|
||
| @Override | ||
| public PlaylistInfoItemsCollector getRelatedItems() { | ||
|
|
||
| PlaylistInfoItemsCollector collector = new PlaylistInfoItemsCollector(getServiceId()); | ||
|
|
||
| Elements recommendedAlbums = document.getElementsByClass("recommended-album"); | ||
|
|
||
| for (Element album : recommendedAlbums) { | ||
| collector.commit(new BandcampRelatedPlaylistInfoItemExtractor(album)); | ||
| } | ||
| final PlaylistInfoItemsCollector collector = | ||
| new PlaylistInfoItemsCollector(getServiceId()); | ||
| document.getElementsByClass("recommended-album").stream() | ||
| .map(BandcampRelatedPlaylistInfoItemExtractor::new) | ||
| .forEach(collector::commit); | ||
|
|
||
| return collector; | ||
| } | ||
|
|
@@ -194,8 +197,7 @@ public String getCategory() { | |
| @Nonnull | ||
| @Override | ||
| public String getLicence() { | ||
|
|
||
| int license = current.getInt("license_type"); | ||
| final int license = current.getInt("license_type"); | ||
|
|
||
| /* Tests resulted in this mapping of ints to licence: https://cloud.disroot.org/s/ZTWBxbQ9fKRmRWJ/preview | ||
| * (screenshot from a Bandcamp artist's account) | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.