Account/Cloud sync support

[yausername]

• I carefully read the contribution guidelines and agree to them.

This PR adds an option to create newpipe account on server and sync subscriptions, playlists, etc.
Backend code - https://github.com/yausername/newpipe-sync-server

• Login is optional, app will work as usual if one chooses to skip login.
• Latest data is fetched from server on login.
• Data can be updated using pull down to refresh in BookmarkFragment, SubscriptionFragment, LocalPlaylistFragment, StatisticsPlaylistFragment
• Data is encrypted before it is sent to server. So the server has no knowledge of your data. It only knows the ciphertext.
• There is no way to recover data is one forgets password.
• Changing password is hard, since all the existing data has to be re-encrypted.
• I will add more information here

Disclaimer: This is a draft implementation. This is expected to be buggy and might contain some security/crypto related bugs as well. Your local data (of debug app) will be erased if you choose to login.

[yausername]

Screenshots

[yausername]

Related issues - #1163 #1281 #2278

[TobiGr]

Hey @yausername,
that sounds awesome ❤️
We might need to add some information to the login screen for what that account is used, where the data is stored (one can host their own server), who can access the data, etc.
It would be nice of one could change the instance before creating an account, too.
@TheAssassin I guess these changes require updates to our privacy policy.
Is the data stored in a way, that FreeTube and Invidio.us can benefit from it and cross sync functionality can be added later?

[TheAssassin]

Wasn't the idea behind NewPipe always to not implement any form of authentication, whatever kind of?

We have to discuss if we want such a thing first of all, then how it should be hosted and then what privacy and data protection implications it will have.

[TheAssassin]

I see that there's some form of user data encryption. If we make this encryption part of our privacy policy (i.e., "we encrypt your data with your passphrase", ...), we'll have to carefully review this in order to be on the safe side. Implementing crypto is always a delicate task.

We're also not legal entity. Liability is a topic that needs to be discussed as well.

Spoiler: not a fan of this. Just because we can doesn't imply we should...

[yausername]

It would be nice of one could change the instance before creating an account, too.

Yes, makes sense. Currently you can skip the initial login screen, change instance from settings->account->server instance and then create account using settings->account->login

Is the data stored in a way, that FreeTube and Invidio.us can benefit from it and cross sync functionality can be added later?

The schema is similar to that of newpipe internal sqllite db. But since data is encrypted, other services will need to decrypt it before consuming.

[TheAssassin]

Is the data stored in a way, that FreeTube and Invidio.us can benefit from it and cross sync functionality can be added later?

The schema is similar to that of newpipe internal sqllite db. But since data is encrypted, other services will need to decrypt it before consuming.

That's not how this kind of encryption works. @yausername already pointed out in the initial post that the data is lost when the passwort is lost, i.e., there's no passphrase reset (which IMO is not so great, but perhaps we can find a possibility to implement this somehow later).

As said, this is a really delicate kind of service to implement and host in accordance to the laws.

I appreciate the suggestion to provide a self-hosting option.

By the way, do we really need custom service? Perhaps this can be integrated with existing services like {Own,Next}Cloud in some way?

[yausername]

Is the data stored in a way, that FreeTube and Invidio.us can benefit from it and cross sync functionality can be added later?

The schema is similar to that of newpipe internal sqllite db. But since data is encrypted, other services will need to decrypt it before consuming.

That's not how this kind of encryption works. @yausername already pointed out in the initial post that the data is lost when the passwort is lost, i.e., there's no passphrase reset (which IMO is not so great, but perhaps we can find a possibility to implement this somehow later).

Data can be decrypted with the same username, password combination by other clients like FreeTube, etc. So cross sync is still possible.

[yausername]

By the way, do we really need custom service? Perhaps this can be integrated with existing services like {Own,Next}Cloud in some way?

Sync across devices using flat files is not straightforward I think.

[omarroth]

there's no passphrase reset (which IMO is not so great, but perhaps we can find a possibility to implement this somehow later).

Generally you would create a master key that is actually used for encryption. The master can then be encrypted with the password to generate K1 and a random recovery passphrase to generate K2. You can extend this to generate K3, K4, etc.

As always with these things I would recommend finding some other resource that provides some more explanation. In case you haven't already seen it the OWASP cheatsheet should be useful, as well as this answer on SO.

[poperigby]

Why create a new account system if you can use Invidious? I think at least having the option to use that would be useful.

[TheAssassin]

Thanks @omarroth, highly appreciated. Of course the solution is a (or multiple) recovery passphrases, although that decreases the overall security again. It's logical that if you want multiple passphrases for a (symmetrically) encrypted storage, you need some sort of intermediate passphrase that is then encrypted with the real passphrases. This method is used widely e.g., by LUKS, which IIRC call that intermediate passphrase the master key (they have 8 slots for passphrases IIRC).
Another problem solved by this intermediate passphrase is you can simply add/remove new passphrases. Changing passphrases becomes adding the new one and removing the old one, basically, which is very convenient and much faster and more secure than re-encrypting.

Right now I'm not primarily interested into (web) security, but legal implications (and the need for proper security is a consequence from that).

@poperigby does Invidious provide sufficient functionality? I had a glance at their privacy policy, I think it'd need some additional love to be in accordance with EU laws (mainly GDPR related, it lacks some information regarding storage and treatment of data and also requires a few standard texts about users' rights), but we could fix that then and refer to them from our privacy policy.

[yausername]

I am using client-side encryption (correct me if this is not a good choice). The encryption key is derived from the username (salt), password (passphrase) combination. Here is the code for that.

    public SecretKey generateKey(char[] passphraseOrPin, byte[] salt) throws NoSuchAlgorithmException, InvalidKeySpecException {
        final int iterations = 1000;
        final int outputKeyLength = 512;
        SecretKeyFactory secretKeyFactory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1");
        KeySpec keySpec = new PBEKeySpec(passphraseOrPin, salt, iterations, outputKeyLength);
        SecretKey secretKey = secretKeyFactory.generateSecret(keySpec);
        return secretKey;
    }

A username, password combination will generate the same key every time. As @omarroth suggested we can generate a random master key and use that for encryption instead of directly using the password derived key. In this case we can have multiple encrypted versions of keys with different passphrases.

[yausername]

Why create a new account system if you can use Invidious? I think at least having the option to use that would be useful.

I haven't explored that option but my main reason to create a new service was client side data encryption.

[poperigby]

@poperigby does Invidious provide sufficient functionality? I had a glance at their privacy policy, I think it'd need some additional love to be in accordance with EU laws (mainly GDPR related, it lacks some information regarding storage and treatment of data and also requires a few standard texts about users' rights), but we could fix that then and refer to them from our privacy policy.

@omarroth do you need to do any GDPR stuff to make this work?

[dimqua]

does Invidious provide sufficient functionality? I had a glance at their privacy policy, I think it'd need some additional love to be in accordance with EU laws

It hasn't to be a specific server, there should be a possibility to specify any server.

[omarroth]

I had a glance at their privacy policy, I think it'd need some additional love to be in accordance with EU laws

As far as I'm aware all code should be GDPR-compliant, changes would only be required for the privacy policy. If there are any resources you'd recommend on this I'd be happy to take a look.

[TheAssassin]

I can't give legal advice, sorry. Key points I'm missing that you should add/revise are, as mentioned already, users' rights (you have to inform so-called "data subjects" (people using your service) that they can have their data corrected, deleted, ..., as long as you collected it via informed consent; with your service I doubt you have to collect anything (except for the server logs maybe, depending on your jurisdiction, I don't know where you live nor where the server is hosted), just make sure you make it sound like "you have a right to have it deleted/corrected as per GDPR Art. 16/17"), if you make backups refer to them as well, and, really important, to be GDPR compliant, you must provide information about who is responsible about the data collection and how they can be reached. I as a user cannot enforce any of my rights if I don't know who to talk to about. I know that's annoying, but like I can't have deleted what I don't know is collected I cannot send a GDPR Art. 17 deletion request if I don't know who to send it to.

Oh, and you support some Google authentication, right? I'm missing a link to Google's privacy policy for the service. Wouldn't hurt to refer to their privacy policy anyway as the videos are coming from YouTube as well.
Also I have big doubts you have a (valid!) contract about commissioned data processing with Google. You need such a thing in order to make use of third-party services. They basically agree on a level of data protection and become responsible for adhering to this. It also protects you a bit, if there's any legal issues with Google you as the service provider could be held responsible normally, but with that contract you can basically blame the third party as well.

Other than that, your privacy notice is pretty complete already. I like it. It's better than what I see most of the time. But it's not quite up to being GDPR compliant. And being compliant isn't too hard.

[theScrabi]

@omarroth Wouldn't syncing with invidious directly be "cleaner"?
Are you ok if we sync with our own server?
What if you or FreeTube and minirube would want to sync as well, would you hook onto this sync server?

[TheAssassin]

It appears to me that there's a need for an "inter-app/service video service preference protocol". Standardization ftw!

[FetchedAttached]

This sounds wonderful!
I know quite some people are hoping a very long time which lets you sync your history/subscriptions etc between NewPipe and Invidio!
Sometimes you want to watch on a desktop (invidio.us), sometimes on your smartdevice (NewPipe).
Completing this integration would be astounding!

[omarroth]

Wouldn't syncing with invidious directly be "cleaner"?

I think so, as I mentioned in #1163 probably the only thing lacking from the API would be proper support for playlists. I'm in the IRC as well if you'd like to chat there about anything that would need to be added/modified for something like sync.

Since this PR already exists though I think it makes sense to move forward with the proposed design if possible.

What if you or FreeTube and minirube would want to sync as well, would you hook onto this sync server?

It would be quite a bit of work as it is currently.

For syncing between two web services (newpipe-sync-server and an Invidious instance), it looks like it would be necessary to make calls fairly often (e.g. when viewing preferences, playlists, etc.) to the other service to ensure everything is up-to-date (this is what Invidious does for syncing Google accounts) which can carry a bit of overhead.

From a UI perspective, a user would have two different accounts, potentially depending on how they are registered with newpipe-sync-server (e.g. using same user and pass as Invidious). From a look through the code, my understanding is that Invidious would have to store the session token for newpipe-sync-server, and (potentially) a URL if using a custom instance, although I'm not sure if you're currently looking into this as a possibility.

Hopefully from the above comments I've made it clear I am not opposed to this design. It'll be the for the NewPipe team to decide what works best, and I will look into supporting it.

[TheAssassin]

For syncing between two web services (newpipe-sync-server and an Invidious instance), it looks like it would be necessary to make calls fairly often (e.g. when viewing preferences, playlists, etc.) to the other service to ensure everything is up-to-date (this is what Invidious does for syncing Google accounts) which can carry a bit of overhead.

Also hard to realize under the GDPR, since we'd have to create contracts between each other (German Vertrag zur Auftragsdatenverarbeitung, in English probably a contract for commissioned data processing). It's easier (and cheaper (less code, less inter-service interactions, easier to keep stuff e2e encrypted)) to just have the client device push to/pull from multiple services and resolve conflicts.
Also, this idea requires this kind-of standardized subscription data exchange format I mentioned earlier. I mean, we, NewPipe, could define such a format pretty freely. Anyone could then translate the data into their own system, and convert forth and back to it. However, without a standardization process and proper versioning, this is going to be a mess, and we're not guaranteed that everyone can make use of the data. And there's overhead.

Again, my main issue is that I'm not sure whether we want to offer a centralized sync server instance. Hosting a website with comments section is easy. This is a whole lot more complex, legally. We're not a public entity. If we were, we'd be safer on the legal side, since then there's the German Vereinsrecht that handles all the bits and makes sure only random private persons can be held liable. Also we can take donations and provide Spendenquittungen (which, to German donors, private as well as legal entities, is really important!) and we can be more transparent about what happens to your donations, fund meetings or help developers attend conferences, ...

<secret-plans-reveal>
We're planning to found some sort of association to run all the NewPipe related stuff on its own infrastructure, but due to lack of personnel this is going to take a little longer (the Satzung for a NewPipe Förderverein e.V. (a German registered association) is pretty much ready, but still needs some work to be on the safe side; and then, there will be a fight with the Finanzamt ("German IRS") to be acknowledged as not-for-profit organization for public benefit, other software projects have had issues about this, e.g., J and Beyond e.V. (see e.g., the article in the German Joomla blog for more information, but (spoiler!) they won eventually).

I think I'm going to open another issue about this. You can write your opinion about it there then. Let's please not bloat this one with off-topic discussions about these plans.
</secret-plans-reveal>

[Obyonekenooby]

Again, my main issue is that I'm not sure whether we want to offer a centralized sync server instance.

I get where you are coming from and a lot of this sounds challenging from a legal perspective to say the least.
I have been thinking about this, why not focus on a standardized format, but if the user wants to have his data
shared between multiple devices, he can do so by his own instance of owncloud/nextcloud?
So you won't have to deal with the legal aspect of having to store user data, you store the data of a user at
his/her own place. Such that the user is in full control whether they want to store data.
Maybe you could even let the user choose which data is stored, subscriptions, watched videos, etc.

Two things have to be kept in mind: a syncing service and a data format needs to be developed.
And, the apps/websites should not leak any information of the owncloud/nextcloud instances.
You just want these apps/websites to fetch and store the subscriptions/watched videos/.. of a user, and nothing more.

[poperigby]

If you just use Nextcloud, you can't sync playlists, history, etc. with Invidious, which I think is something that's very useful.

For example, people want to start a video on their desktop, and finish it on their phone.

[ushername]

A better implementation would be to use WebDAV, see #1163.
This gives the possibility to choose on which platform you want to host your data.
Also, you can watch videos and finish them on another device, since it is a possibility
to store at what point a video has been stopped.
Video_progress: {12:55}
or
Video_progress: {0:00}
The first gets triggered when the video hasn't completed, the second when one has completed
the video. This is a bad implementation, one could think of many solutions.

[poperigby]

I webDAV along with being able to sign in with Invidious would be a good idea.

[vinibali]

Great work, I subscribe :)

[trymeouteh]

#3198

Would like to suggest adding invidio.us account suppprt in NewPipe.

[poperigby]

What made you close the pull request?

[yausername]

I don't think this as a good idea any more. Storing user data, encrypted or not should be avoided as much as possible. An implementation using owncloud/nextcloud can be considered though.
I think #3157 implements backups in a simpler, more secure albeit not as feature-full way.

[BeatLink]

Im so sad to see this pull request closed as I was really looking forward to this feature. With the sync server being self hostable, I don't think this would have compromised user security any more than an implementation of owncloud or nextcloud. In any case, I believe it should be up to the user where their data is stored, and this pull request gives the that option.

[trymeouteh]

I don't think this as a good idea any more. Storing user data, encrypted or not should be avoided as much as possible. An implementation using owncloud/nextcloud can be considered though.
I think #3157 implements backups in a simpler, more secure albeit not as feature-full way.

Can you add an optional feature to login to one or more invidious instance which can sync your subscriptions, video feed and playlists? I use invidio.us on my desktop instead of YouTube and it will be cool to be able to access my invidious account in NewPipe.

Again this will be optional, not mandatory.

#3198