[PeerTube] added metadata, fix descriptions, fix thumbnail, fix upload date, fix age limit, update tests. by B0pol · Pull Request #239 · TeamNewPipe/NewPipeExtractor

B0pol · 2020-01-20T11:35:31Z

Hi,
I fixed some issues with PeerTube that we have now.
Also, I extracted metadata: host, privacy, licence, language, tags.
For YouTube, someone could extract privacy, category (and maybe tags) after this PR, that would be a nice thing.

Fixes TeamNewPipe/NewPipe#2201 (comment) 1. 2. and 5.
3 is already fixed, and I plan to fix 4 (default instance only) and 6 with my future Frontend PR.
TeamNewPipe/NewPipe#2201 (comment) I aimed to do that.

Some in-app before/after screenshots:
Screenshot from PeerTube website
Screenshot from NewPipe (with my changes)

Thumbnail and description before
Thumbnail now
Description now

Btw, for the fronted I already handled the fact we are not getting info:
If no metadata is extracted, it stays as it was before
If only some metadata are extracted, no problem

I carefully read the contribution guidelines and agree to them.
I did test the API against NewPipe.
I agree to ASAP create a PULL request for NewPipe for making in compatible when I changed the api.

thumbnail: quality before: https://peertube.cpy.re/static/thumbnails/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg quality after: https://peertube.cpy.re/static/previews/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg description: we were getting about the first 260 characters, we now get full description (with fallback to first 260 chars if the get request for full description fails) test: updated tests to match description, also changed some test: it was assertEquals(extracted, expected), but the proper way to do it is assertEquals(expected, extracted) metadata: got host, privacy (public, private, unlisted), licence, language, tags

.../java/org/schabi/newpipe/extractor/services/peertube/extractors/PeertubeStreamExtractor.java

if the description length is above 254, and ends with ..., it means the description we got from the first request is shortened. why above 254: because in fact, shortened descriptions length are between 255 : https://framatube.org/videos/watch/24efbe1b-73c0-4d72-a3ff-77c8b32d3fcd https://framatube.org/videos/watch/1ca99f40-fb5b-4fa4-abe5-4d54325df7fc and 269: https://framatube.org/videos/watch/4d661d5f-a884-4510-bca8-15cb19aa3fe5 also fixed a typo in StreamExtractor.java

actually, the max description length is 250 after request with our extractor. during my tests, I made API requests with Firefox, copy/pasted into echo "insert description" | wc, and it was giving a wrong length, maybe due to the escapers, I have no idea anyway, it's now fixed

.../java/org/schabi/newpipe/extractor/services/peertube/extractors/PeertubeStreamExtractor.java

Co-Authored-By: Tobias Groza <TobiGr@users.noreply.github.com>

+ created getBoolean method in JsonUtils.java

on peertube.co.uk: https://i.imgur.com/8Pai1rb.png on newpipe, before this commit: https://i.imgur.com/NIRbs4Z.png on newpipe, with this commit: https://i.ibb.co/mhKKtRD/Screenshot-20200123-185422-New-Pipe-Debug.png

https://peertube.cpy.re/videos/watch/d2a5ec78-5f85-4090-8ec5-dc1102e022ea anonyme zirbeldrüse is his displayName, what was displayed in NewPipe. but on the website, it's shown 777@mastodon.xyz

TobiGr

Looks good so far. Just some minor comments.

extractor/src/main/java/org/schabi/newpipe/extractor/utils/JsonUtils.java

.../java/org/schabi/newpipe/extractor/services/peertube/extractors/PeertubeStreamExtractor.java

extractor/src/main/java/org/schabi/newpipe/extractor/stream/StreamExtractor.java

B0pol · 2020-01-24T19:45:01Z

About PR in NewPipe repository: I plan to make a first & quick PR right after this PR get merged, to update extractor version, and fix description formatting, because currently, all descriptions are formatted from HTML, but PeerTube gives plain Markdown, so I will change that, see this commit,
so that all fixes are in NewPipe fast, and a second PR, which will be longer to merge, so that we can discuss about UI, UX…

so that java can automatically translate with Locale.getDisplayLanguage(), instead of always having English name of the language

B0pol · 2020-02-02T10:31:22Z

Are there other changes needed?

TobiGr · 2020-02-02T12:59:04Z

Sorry, there are just too many PRs :D I forgot about this one.
Please open the PR in the app repository first so we can assure compatibility when merging this. I'll try to make a final review during the week.

description: - PeerTube: it's now full description (it cut at 250 characters before), and it displays ok (newlines are ok, but markdown isn't) - MediaCCC: descriptions are now displayed well (newlines added) - YouTube: timestamps in descriptions are clickable and work more PeerTube fixes: thumbnail is now high quality age limit is now handled upload date in «recently added» feed is good now (it was one hour delayed) all fixes come from TeamNewPipe/NewPipeExtractor#239, so it need to be merged before this PR

...ctor/src/main/java/org/schabi/newpipe/extractor/services/peertube/PeertubeParsingHelper.java

description: - PeerTube: it's now full description (it cut at 250 characters before), and it displays ok (newlines are ok, but markdown isn't) - MediaCCC: descriptions are now displayed well (newlines added) - YouTube: timestamps in descriptions are clickable and work more PeerTube fixes: thumbnail is now high quality age limit is now handled upload date in «recently added» feed is good now (it was one hour delayed) all fixes come from TeamNewPipe/NewPipeExtractor#239, so it need to be merged before this PR

extractor/src/main/java/org/schabi/newpipe/extractor/stream/Description.java

B0pol · 2020-02-08T19:38:44Z

Just FYI, I've succeed to extract some metadata for YouTube:

Privacy
Tags
Category

I will extract for YouTube

Licence: it won't be hard, I found it in the HTML file.

I won't extract for YouTube:

Host: it's always youtube.com
SupportInfo: there are no support info in YouTube videos page
LanguageInfo: the only way I see is with auto-generated subtitles, but it's not really accurate.

But this will be in another extractor PR.

TobiGr

Thanks again!

TobiGr added the peertube service, https://joinpeertube.org/ label Jan 20, 2020

TobiGr reviewed Jan 20, 2020

View reviewed changes

.../java/org/schabi/newpipe/extractor/services/peertube/extractors/PeertubeStreamExtractor.java Outdated Show resolved Hide resolved

B0pol added 2 commits January 20, 2020 14:36

TobiGr reviewed Jan 20, 2020

View reviewed changes

.../java/org/schabi/newpipe/extractor/services/peertube/extractors/PeertubeStreamExtractor.java Outdated Show resolved Hide resolved

B0pol and others added 6 commits January 20, 2020 21:28

Refactoring

f403490

Co-Authored-By: Tobias Groza <TobiGr@users.noreply.github.com>

Merge branch 'dev' into peertube

bcfe7be

agelimit now returns 18 if the video is marked as nsfw, 0 otherwise

1a15c0e

+ created getBoolean method in JsonUtils.java

add extraction for support info and rename getLanguageInfo function

74439f6

empty support returns "", same for empty description

20da475

fix upload date: there was a one hour delay

a691d6d

on peertube.co.uk: https://i.imgur.com/8Pai1rb.png on newpipe, before this commit: https://i.imgur.com/NIRbs4Z.png on newpipe, with this commit: https://i.ibb.co/mhKKtRD/Screenshot-20200123-185422-New-Pipe-Debug.png

B0pol changed the title ~~[PeerTube] added metadata, fix descriptions, fix thumbnail, update tests~~ [PeerTube] added metadata, fix descriptions, fix thumbnail, fix upload date, fix age limit, update tests. Jan 23, 2020

B0pol added 2 commits January 23, 2020 19:08

update test

c790261

authorName in comments now follow PeerTube website

812c8e0

https://peertube.cpy.re/videos/watch/d2a5ec78-5f85-4090-8ec5-dc1102e022ea anonyme zirbeldrüse is his displayName, what was displayed in NewPipe. but on the website, it's shown 777@mastodon.xyz

TobiGr reviewed Jan 24, 2020

View reviewed changes

B0pol added 2 commits January 24, 2020 20:10

Merge branch 'dev' into peertube

b816e48

reindenting (ctrl alt l) on JsonUtils and PeertubeStreamExtractor

341372c

getLanguageInfo returns Locale instead of String

e392b6c

so that java can automatically translate with Locale.getDisplayLanguage(), instead of always having English name of the language

B0pol force-pushed the peertube branch from 3833cd1 to e392b6c Compare January 25, 2020 13:13

Merge branch 'dev' into peertube

b671a4b

B0pol mentioned this pull request Feb 1, 2020

Fix timestamp links in video descriptions TeamNewPipe/NewPipe#2536

Closed

1 task

B0pol mentioned this pull request Feb 2, 2020

Description fix and some PeerTube fixes TeamNewPipe/NewPipe#3044

Merged

1 task

yausername suggested changes Feb 6, 2020

View reviewed changes

...ctor/src/main/java/org/schabi/newpipe/extractor/services/peertube/PeertubeParsingHelper.java Outdated Show resolved Hide resolved

B0pol force-pushed the peertube branch from 8074b09 to b671a4b Compare February 6, 2020 21:21

Use GMT as base time (actually fix upload date)

5756df8

Create class Description

26c65b2

yausername suggested changes Feb 7, 2020

View reviewed changes

extractor/src/main/java/org/schabi/newpipe/extractor/stream/Description.java Outdated Show resolved Hide resolved

B0pol added 3 commits February 7, 2020 13:28

Description: rm constructor by serviceId

70a40e7

fix testGetUploadDate for PeerTubeStreamExtractor

0f8a7f9

Description implements Serializable. fix NotSerializableException

11bcc78

yausername approved these changes Feb 8, 2020

View reviewed changes

B0pol force-pushed the peertube branch from 928787f to 11bcc78 Compare February 8, 2020 16:04

TobiGr approved these changes Feb 8, 2020

View reviewed changes

Fix SoundCloud description test

0e33249

TobiGr merged commit 3add704 into TeamNewPipe:dev Feb 8, 2020

B0pol deleted the peertube branch February 8, 2020 23:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[PeerTube] added metadata, fix descriptions, fix thumbnail, fix upload date, fix age limit, update tests.#239

[PeerTube] added metadata, fix descriptions, fix thumbnail, fix upload date, fix age limit, update tests.#239
TobiGr merged 21 commits intoTeamNewPipe:devfrom
B0pol:peertube

B0pol commented Jan 20, 2020 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

TobiGr left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

B0pol commented Jan 24, 2020

Uh oh!

B0pol commented Feb 2, 2020

Uh oh!

TobiGr commented Feb 2, 2020

Uh oh!

Uh oh!

Uh oh!

B0pol commented Feb 8, 2020

Uh oh!

TobiGr left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

B0pol commented Jan 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TobiGr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

B0pol commented Jan 24, 2020

Uh oh!

B0pol commented Feb 2, 2020

Uh oh!

TobiGr commented Feb 2, 2020

Uh oh!

Uh oh!

Uh oh!

B0pol commented Feb 8, 2020

Uh oh!

TobiGr left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

B0pol commented Jan 20, 2020 •

edited

Loading