Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
e72f207
Add support of other delivery methods than progressive HTTP
AudricV Jun 16, 2021
83d7f94
Parse YouTube DASH manifest
AudricV Jun 16, 2021
2998d73
Apply changes in extractor services
AudricV Jun 16, 2021
78dd67d
Apply changes in DefaultStreamExtractorTest
AudricV Jun 16, 2021
4017b4d
Update SoundcloudStreamExtractorTest and fix some warnings in some of…
AudricV Jun 16, 2021
12be50a
Readd itags 134 and 136 and format the ItagItem class
AudricV Jun 16, 2021
1efaf24
Add a baseUrl field to Stream classes
AudricV Jun 18, 2021
da07e31
Add POST_LIVE_STREAM and POST_LIVE_AUDIO_STREAM to StreamType enum
AudricV Jul 8, 2021
0b1c787
Update livestream mocks and add instructions how to run YoutubeStream…
AudricV Jul 8, 2021
c69ecb9
[PeerTube] Return HLS variants of video streams
AudricV Jul 11, 2021
2ec6a8a
Remove some unused code
AudricV Jul 13, 2021
c218899
[YouTube] Generate DASH manifests of OTF streams and extract subtitle…
AudricV Jul 29, 2021
654d2b4
Fix a typo in SubtitlesStream
AudricV Aug 7, 2021
00b78ea
Adress some requested changes + extract correctly PeerTube audio streams
AudricV Aug 7, 2021
c659476
Change a message in PeertubeStreamExtractorTest
AudricV Aug 7, 2021
8a721c2
[YouTube] Generate DASH manifests for streams of ended livestreams (p…
AudricV Aug 9, 2021
9b0802b
Fix some comments
AudricV Aug 9, 2021
643f191
Add static constant UNKNOWN_BITRATE in AudioStream and use it in stre…
AudricV Aug 9, 2021
87b270b
Add missing JavaDocs for Stream classes
AudricV Aug 9, 2021
a14a9fa
[YouTube] Use really to the last URL to prevent redirects when stream…
AudricV Aug 10, 2021
9a90cd7
[YouTube] Return the POST_LIVE_STREAM stream type
AudricV Aug 10, 2021
9517a0f
[PeerTube] Readd support of livestreams
AudricV Aug 10, 2021
e1bc190
Make the ItagItem class Serializable
AudricV Aug 14, 2021
99ce0bf
Add a getItagItem method in Stream classes and store the ItagItem obj…
AudricV Aug 14, 2021
e0678c4
[YouTube] Use less requests when generating DASH manifests of OTF str…
AudricV Aug 14, 2021
2a6f3dd
[YouTube] Don't generate DASH manifests for OTF streams when loading …
AudricV Aug 14, 2021
b0147f5
[YouTube] Cache the manifests generated with YoutubeDashManifestCreat…
AudricV Aug 15, 2021
11ddc07
Add the missing Deprecated annotation on the getUrl method of the Str…
AudricV Aug 15, 2021
0edaeba
[YouTube] Only fetch the Android mobile API for video streams
AudricV Aug 15, 2021
4c0fd40
Rename the ContentAndItagItemAndIsUrl class to ItagInfo
AudricV Aug 16, 2021
e783c34
[YouTube] Don't generate manifests for post live streams when loading…
AudricV Aug 16, 2021
6fcec47
Add STREAMING_DATA, SIGNATURE_CIPHER and CIPHER constants in YoutubeS…
AudricV Aug 18, 2021
621a79f
Use the containsKey method instead of the get method in YoutubeDashMa…
AudricV Aug 18, 2021
a17f933
Improve the SEGMENT_DURATION_MS_PATTERN regular expression in YoutuDa…
AudricV Aug 18, 2021
371ff75
[YouTube] Extract the audioChannels and the approxDurationMs values
AudricV Aug 19, 2021
fa2a30b
Adress requested changes in StreamInfo
AudricV Aug 24, 2021
642e457
[Media.ccc.de] Improve stream parsing of livestreams
AudricV Aug 24, 2021
e8ea26b
Make the MediaFormat in Stream objects nullable + improve some code i…
AudricV Aug 24, 2021
1be9160
Adress requested changes in YoutubeStreamExtractor
AudricV Aug 24, 2021
20b393c
Add a constant for the string &sq=0 in YoutubeDashManifestCreator
AudricV Aug 26, 2021
da73de7
Write a better test for YoutubeDashManifestCreator
AudricV Aug 26, 2021
0a8f78a
Fix some tests and a RESOLUTION_UNKNOWN constant string
AudricV Aug 30, 2021
78146fa
Finish an unfinished JavaDoc in MediaCCCLiveStreamExtractor
AudricV Aug 31, 2021
10d6854
Use braces in all if statements of DefaultStreamExtractorTest and Str…
AudricV Aug 31, 2021
c0bbec6
Remove an unused import in YoutubeStreamExtractor
AudricV Sep 10, 2021
ef8fdcf
[SoundCloud] Extract the downloadable file when available
AudricV Sep 10, 2021
d18e537
Fix failing tests + return empty lists instead of null
AudricV Sep 11, 2021
c34e0b1
[PeerTube] Improve code of stream extraction, fix some warnings and t…
AudricV Oct 2, 2021
0ff33a9
Remove a useless attribute in DASH manifests generated
AudricV Oct 24, 2021
deeea85
[YouTube] Fix buffering when playing streams from the WEB client and …
AudricV Nov 1, 2021
c0bccbc
[YouTube] Spoof better official clients when generating and fetching …
AudricV Nov 1, 2021
8a1a6a1
[YouTube] Always use again Android streams first
AudricV Nov 4, 2021
91def33
[YouTube] Extract the content length field of streams when available
AudricV Dec 6, 2021
9686aa7
[YouTube] Add ability to generate DASH manifests for YouTube progress…
AudricV Dec 6, 2021
eaa70a1
Apply requested changes and cache audio and video streams and HLS and…
AudricV Dec 12, 2021
76f6dc5
[YouTube] Parse a long instead of an integer when setting the content…
AudricV Dec 20, 2021
3a309a1
[YouTube] Apply documentation suggestions to YoutubeDashManifestCreator
AudricV Dec 20, 2021
51c19f8
[YouTube] Fix a bug when using resetClientVersionAndKey method
AudricV Dec 11, 2021
50b74c6
[YouTube] Fix StackOverflow error when generating manifests of OTF st…
AudricV Jan 1, 2022
ee673f7
Apply suggested changes on StreamInfo
AudricV Jan 1, 2022
c08b8dc
Adress requested changes, remove the use of StandardCharsets and impr…
AudricV Jan 8, 2022
8357f6f
Override hashCode and equals methods on Stream classes
AudricV Jan 26, 2022
7d1cafa
Add ability to set maximum size of caches
AudricV Jan 26, 2022
bdd9341
Update YoutubeDashManifestCreatorTest to Junit 5
AudricV Jan 30, 2022
f773190
Add basic tests for ManifestCreatorCache in UtilsTest
AudricV Jan 30, 2022
4e59ea8
Add serialVersionUID to Serializable classes changed
AudricV Feb 3, 2022
f650808
Add the SmoothStreaming delivery method to the available delivery met…
AudricV Feb 22, 2022
77026d9
Add documentation in the StreamType enum
AudricV Feb 22, 2022
09bf96e
Don't remove public access of averageBitrate field in AudioStream and…
AudricV Feb 22, 2022
3bec1f0
Apply some requested changes
AudricV Feb 22, 2022
bc33879
Adress new requested changes
AudricV Feb 26, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,6 @@ public String getId() throws ParsingException {
* @return the name
* @throws ParsingException if the name cannot be extracted
*/
@Nonnull
public abstract String getName() throws ParsingException;

@Nonnull
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,28 +6,29 @@
* Copyright (c) Christian Schabesberger <[email protected]>
* and Adam Howard <[email protected]> 2015
*
* MediaFormat.java is part of NewPipe.
* MediaFormat.java is part of NewPipe Extractor.
*
* NewPipe is free software: you can redistribute it and/or modify
* NewPipe Extractor is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* NewPipe is distributed in the hope that it will be useful,
* NewPipe Extractor is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with NewPipe. If not, see <http://www.gnu.org/licenses/>.
* along with NewPipe Extractor. If not, see <https://www.gnu.org/licenses/>.
*/

import javax.annotation.Nullable;

/**
* Static data about various media formats support by NewPipe, eg mime type, extension
* Static data about various media formats support by NewPipe, e.g. mime type, extension.
*/

public enum MediaFormat {
//video and audio combined formats
// Video and audio combined formats
// id name suffix mime type
MPEG_4 (0x0, "MPEG-4", "mp4", "video/mp4"),
v3GPP (0x10, "3GPP", "3gp", "video/3gpp"),
Expand All @@ -52,7 +53,7 @@ public enum MediaFormat {
public final String suffix;
public final String mimeType;

MediaFormat(int id, String name, String suffix, String mimeType) {
MediaFormat(final int id, final String name, final String suffix, final String mimeType) {
this.id = id;
this.name = name;
this.suffix = suffix;
Expand All @@ -66,9 +67,11 @@ public enum MediaFormat {
* @return the friendly name of the MediaFormat associated with this ids,
* or an empty String if none match it.
*/
public static String getNameById(int ident) {
for (MediaFormat vf : MediaFormat.values()) {
if (vf.id == ident) return vf.name;
public static String getNameById(final int ident) {
for (final MediaFormat vf : MediaFormat.values()) {
if (vf.id == ident) {
return vf.name;
}
}
return "";
}
Expand All @@ -80,9 +83,11 @@ public static String getNameById(int ident) {
* @return the file extension of the MediaFormat associated with this ids,
* or an empty String if none match it.
*/
public static String getSuffixById(int ident) {
for (MediaFormat vf : MediaFormat.values()) {
if (vf.id == ident) return vf.suffix;
public static String getSuffixById(final int ident) {
for (final MediaFormat vf : MediaFormat.values()) {
if (vf.id == ident) {
return vf.suffix;
}
}
return "";
}
Expand All @@ -94,9 +99,11 @@ public static String getSuffixById(int ident) {
* @return the MIME type of the MediaFormat associated with this ids,
* or an empty String if none match it.
*/
public static String getMimeById(int ident) {
public static String getMimeById(final int ident) {
for (MediaFormat vf : MediaFormat.values()) {
if (vf.id == ident) return vf.mimeType;
if (vf.id == ident) {
return vf.mimeType;
}
}
return "";
}
Expand All @@ -107,9 +114,12 @@ public static String getMimeById(int ident) {
* @return MediaFormat associated with this mime type,
* or null if none match it.
*/
public static MediaFormat getFromMimeType(String mimeType) {
for (MediaFormat vf : MediaFormat.values()) {
if (vf.mimeType.equals(mimeType)) return vf;
@Nullable
public static MediaFormat getFromMimeType(final String mimeType) {
for (final MediaFormat vf : MediaFormat.values()) {
if (vf.mimeType.equals(mimeType)) {
return vf;
}
}
return null;
}
Expand All @@ -120,16 +130,22 @@ public static MediaFormat getFromMimeType(String mimeType) {
* @param id the id
* @return the id of the media format or null.
*/
public static MediaFormat getFormatById(int id) {
@Nullable
public static MediaFormat getFormatById(final int id) {
for (MediaFormat vf : values()) {
if (vf.id == id) return vf;
if (vf.id == id) {
return vf;
}
}
return null;
}

public static MediaFormat getFromSuffix(String suffix) {
for (MediaFormat vf : values()) {
if (vf.suffix.equals(suffix)) return vf;
@Nullable
public static MediaFormat getFromSuffix(final String suffix) {
for (final MediaFormat vf : values()) {
if (vf.suffix.equals(suffix)) {
return vf;
}
}
return null;
}
Expand Down Expand Up @@ -160,5 +176,4 @@ public String getSuffix() {
public String getMimeType() {
return mimeType;
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -114,16 +114,12 @@ public List<AudioStream> getAudioStreams() {
final JsonObject streams = showInfo.getObject("audio_stream");

if (streams.has("opus-lo")) {
list.add(new AudioStream(
streams.getString("opus-lo"),
MediaFormat.OPUS, 100
));
list.add(new AudioStream("opus-lo", streams.getString("opus-lo"), MediaFormat.OPUS,
100));
}
if (streams.has("mp3-128")) {
list.add(new AudioStream(
streams.getString("mp3-128"),
MediaFormat.MP3, 128
));
list.add(new AudioStream("mp3-128", streams.getString("mp3-128"), MediaFormat.MP3,
128));
}

return list;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import org.schabi.newpipe.extractor.MediaFormat;
import org.schabi.newpipe.extractor.MetaInfo;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
import org.schabi.newpipe.extractor.exceptions.ExtractionException;
Expand All @@ -27,16 +26,17 @@
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Locale;

import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImageUrl;
import static org.schabi.newpipe.extractor.utils.Utils.*;

public class BandcampStreamExtractor extends StreamExtractor {

private JsonObject albumJson;
private JsonObject current;
private Document document;

private List<AudioStream> audioStreams = null;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still don't have a good feeling about caching here...

  1. As Stypox already said normally there are no repetitive calls because of StreamInfo
  2. This creates so much overhead. There are now a ton of null checks and fields in each extractor.


public BandcampStreamExtractor(final StreamingService service, final LinkHandler linkHandler) {
super(service, linkHandler);
}
Expand Down Expand Up @@ -83,13 +83,13 @@ public String getName() throws ParsingException {
public String getUploaderUrl() throws ParsingException {
final String[] parts = getUrl().split("/");
// https: (/) (/) * .bandcamp.com (/) and leave out the rest
return "https://" + parts[2] + "/";
return HTTPS + parts[2] + "/";
}

@Nonnull
@Override
public String getUrl() throws ParsingException {
return albumJson.getString("url").replace("http://", "https://");
return albumJson.getString("url").replace(HTTP, HTTPS);
}

@Nonnull
Expand All @@ -113,8 +113,11 @@ public DateWrapper getUploadDate() throws ParsingException {
@Nonnull
@Override
public String getThumbnailUrl() throws ParsingException {
if (albumJson.isNull("art_id")) return "";
else return getImageUrl(albumJson.getLong("art_id"), true);
if (albumJson.isNull("art_id")) {
return EMPTY_STRING;
}

return getImageUrl(albumJson.getLong("art_id"), true);
}

@Nonnull
Expand All @@ -123,7 +126,7 @@ public String getUploaderAvatarUrl() {
try {
return document.getElementsByClass("band-photo").first().attr("src");
} catch (final NullPointerException e) {
return "";
return EMPTY_STRING;
}
}

Expand All @@ -132,7 +135,7 @@ public String getUploaderAvatarUrl() {
public Description getDescription() {
final String s = Utils.nonEmptyAndNullJoin(
"\n\n",
new String[]{
new String[] {
current.getString("about"),
current.getString("lyrics"),
current.getString("credits")
Expand All @@ -143,13 +146,16 @@ public Description getDescription() {

@Override
public List<AudioStream> getAudioStreams() {
final List<AudioStream> audioStreams = new ArrayList<>();

audioStreams.add(new AudioStream(
albumJson.getArray("trackinfo").getObject(0)
.getObject("file").getString("mp3-128"),
MediaFormat.MP3, 128
));
if (audioStreams == null) {
audioStreams = new ArrayList<>();
audioStreams.add(new AudioStream("mp3-128",
albumJson.getArray("trackinfo")
.getObject(0)
.getObject("file")
.getString("mp3-128"),
MediaFormat.MP3,
128));
}
return audioStreams;
}

Expand All @@ -170,14 +176,11 @@ public StreamType getStreamType() {

@Override
public PlaylistInfoItemsCollector getRelatedItems() {

PlaylistInfoItemsCollector collector = new PlaylistInfoItemsCollector(getServiceId());

Elements recommendedAlbums = document.getElementsByClass("recommended-album");

for (Element album : recommendedAlbums) {
collector.commit(new BandcampRelatedPlaylistInfoItemExtractor(album));
}
final PlaylistInfoItemsCollector collector =
new PlaylistInfoItemsCollector(getServiceId());
document.getElementsByClass("recommended-album").stream()
.map(BandcampRelatedPlaylistInfoItemExtractor::new)
.forEach(collector::commit);

return collector;
}
Expand All @@ -194,8 +197,7 @@ public String getCategory() {
@Nonnull
@Override
public String getLicence() {

int license = current.getInt("license_type");
final int license = current.getInt("license_type");

/* Tests resulted in this mapping of ints to licence: https://cloud.disroot.org/s/ZTWBxbQ9fKRmRWJ/preview
* (screenshot from a Bandcamp artist's account)
Expand Down
Loading