Subsonic API #2321

magne4000 · 2016-12-15T16:01:09Z

This PR adds Subsonic API support (version 1.10.1 of the protocol).
With it you can connect with any subsonic client as long as web plugin is enabled (if client ask for login/password, put anything, those are not read).

NB: encoding parameters for stream mode are not taken into account, file is streamed as-is.

Implemented methods

Category	Methods
System	`ping` `getLicense`
Browsing	`getMusicFolders` `getIndexes` `getMusicDirectory` `getArtists` `getArtist` `getAlbum` `getSong`
Album/song lists	`getAlbumList` `getAlbumList2` `getRandomSongs`
Searching	`search2` `search3`

Unimplemented methods:

All unimplemented methods generate a valid XML/JSON that should be interpreted by any client, so that they can show an appropriate error message without any crash.

Category	Methods
Browsing	`getGenres` `getVideos` `getVideoInfo` `getArtistInfo` `getArtistInfo2` `getAlbumInfo` `getAlbumInfo2` `getSimilarSongs` `getSimilarSongs2` `getTopSongs`
Album/song lists	`getSongsByGenre` `getNowPlaying` `getStarred` `getStarred2`
Searching	`search`
Playlists	`getPlaylists` `getPlaylist` `createPlaylist` `updatePlaylist` `deletePlaylist`
Media retrieval	`stream` `download` `hls` `getCaptions` `getCoverArt` `getLyrics` `getAvatar`
Media annotation	`star` `unstar` `setRating` `scrobble`
Sharing	`getShares` `createShare` `updateShare` `deleteShare`
Podcast	`getPodcasts` `getNewestPodcasts` `refreshPodcasts` `createPodcastChannel` `deletePodcastChannel` `deletePodcastEpisode` `downloadPodcastEpisode`
Jukebox	`jukeboxControl`
Internet radio	`getInternetRadioStations`
Chat	`getChatMessages` `addChatMessage`
User management	`getUser` `getUsers` `createUser` `updateUser` `deleteUser` `changePassword`
Bookmarks	`getBookmarks` `createBookmark` `deleteBookmark` `getPlayQueue` `savePlayQueue`

sampsyo · 2016-12-16T01:43:10Z

Interesting! Any comments about how this works in general? How broadly have you observed its compatibility?

Also, it looks like changes to a few other files got wrapped up in this PR.

magne4000 · 2016-12-16T08:24:34Z

I edited the first post.

Some comments on why I had to change others files (feel free to guide me on how I could have done it better):

beetsplug/random.py: Patched to be reuseable by subsonic plugin.
beetsplug/web/__init__.py: Patched to add subsonic parameter. It uses Flask blueprint.
beetsplug/web/subsonic.py: Code for the plugin.
docs/plugins/web.rst: Doc updated for new config parameter in web plugin.
setup.py and tox.ini: New dev dependency xmlunittest.
test/test_web_subsonic.py: Test cases.

Also, pagination is done entirely in python, which is rather inefficient. We'll need to to some work as stated in #750.

sampsyo · 2016-12-16T15:03:43Z

Got it; thanks for elaborating!

The one thing that looks like it might be a problem is the dependency on random.py. Cross-plugin dependencies can lead to plugins getting "accidentally" enabled when the user didn't request them. We may need a different tactic here.

And, to talk more about the long-term picture: I'm really interested in moving toward AURA, our new API that should replace our hacked-together initial API. There's even been some talk about constructing an AURA-to-Subsonic "adapter," more or less exactly like this, to connect to clients that support that API already. Would you be interested in exploring that direction?

magne4000 · 2016-12-19T09:46:53Z

I'll move reusable part of random.py outside the beetsplug structure, so it can be reused.

And regarding AURA, here are my remarks:

festival also implements its own API (that kinda look like AURA) , but with hindsight, It doesn't have any added value over subsonic (or other) API. So if I would have to change it I would stick to existent APIs (that way, you already have a lot of client to make your test and unit tests)
Will AURA API do more than subsonic API ?
AURA-to-subsonic API could be a way to do that, but is that necessary ? What are the advantages over this PR ? My opinion is that we should have a sane reusable base (python methods and classes) that we can use to EASILY implement any API, not an API to rule them all 💍.

So the real question is "Do you have a real need for a new API ?"

magne4000 · 2016-12-19T10:17:28Z

Also, to fix appveyor build, I would need to modify appveyor.yml file. xmlunittest package requires lxml package to be installed, and it seems to be kinda complicated under windows ... http://help.appveyor.com/discussions/problems/5330-pip-install-lxml-fails-with-missing-symbols

magne4000 · 2016-12-19T15:53:50Z

@sampsyo as you see I made some test trying to get lxml build working, without success...
I would like to try another one, but it would require including corresponding .whl files from http://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml into the repository (under test/prereq folder for example). Would you be okay with that ?

sampsyo · 2016-12-20T04:45:58Z

OK! Factoring out the random functionality sounds like a fine strategy if we can do it cleanly.

Here's my motivation for focusing on AURA in the future: interoperation. I'd like to be able to unify music software of all stripes, not just beets, so people don't have to invent their own APIs or use the cumbersome ones that already exist. In particular, it should be easy to get new tools—new players, new caching layers, etc.—up and running. That rules out XML in my opinion, as you're discovering with the lxml dependency hell. 😃

I know that's something of a dream that's low on practicality, so I don't object to adding a Subsonic API directly to beets for now. I'm just letting you know that, maybe someday, we might want to supplant it with the more grand vision.

Back to the matter at hand: I think we need to avoid checking other people's code into this repository… maybe we should just mark the tests as skippable if lxml isn't available? That's what we've done for the tests that require GStreamer, for example.

One other design point: might it make sense to make this its own plugin? It doesn't seem to share much with the existing web plugin, except that they both use Flask. It might be simpler to explain and use the different set of requirements for two different plugins.

magne4000 · 2016-12-20T08:24:01Z

Got it!
I'll try to remove subsonic test from AppVeyor CI.

We can move subsonic to it's own plugin, but that would make beets listen on another port. I think of the web plugin as something that should be extended with web functionalities, without bothering with redundant things as "What port should I use for my 34th web-ish plugin".

sampsyo · 2016-12-20T16:18:48Z

Yeah, we'd like to have an extensible server mode—see #718. That direction would allow plugins to extend the web functionality, instead of needing to roll all possible web-related functionality into a single monolithic plugin.

sampsyo · 2016-12-20T16:18:58Z

appveyor.yml

        - PYTHON: C:\Python35
-          TOX_ENV: py35-test


Is this change still needed?

Ah, I see now—you're using this flag to disable tests on the command line.

Instead would you mind looking into using @unittest.skipIf() or self.skipTest() in the tests themselves? This makes for a graceful degradation: developers can just type nosetests, regardless of whether they're on Windows or not, and have the appropriate set of tests skipped based on their current configuration. There are a few examples of this in the current test suite in case that's helpful.

That's what I wanted to do firstly, but that implies that we totally remove xmlunittest from tox deps. I fear that this test case will be forgotten if we do that.
Also, we still want this to run on travis, so do I just add the dependency in .travis.yml file ?

Ah, that's really tricky. So to summarize, the new tests have a new dependency from pip, so it makes sense to list that in the tox config. But that library has trouble installing on all platforms, because of the transitive dependency on lxml. 🙁

This is similar to (but not exactly the same as) the trouble we have with running tests against code that uses GStreamer on Travis. You can't install python-gobject from pip (who knows why), so you need to get it from the apt repositories. Then, to use that dependency in tests within tox's virtualenv, you have to use the "site packages" option, which lets all system-installed dependencies leak into the test environment. Clearly far from ideal, but it's the only way we've been able to make it work.

It looks like we're not the first to hit problems with lxml on AppVeyor:
http://help.appveyor.com/discussions/problems/2351-python-lxml-module
http://help.appveyor.com/discussions/problems/5330-pip-install-lxml-fails-with-missing-symbols

I don't, at the moment, see any good way around this. (Sometimes, CI seems like more effort than it's worth… 😢) I wish there were a way to make xmlunittest use the built-in ETree instead of lxml. I'll keep thinking about the right solution here.

We came to the same conclusions 😃
In fact I would just need a pure python lib that would implement C14N, but lxml is the only lib to do this for now.

Ideally, I think, we'd have the tox config add the dependency only on non-Windows platforms. It looks like platform-specific settings are being designed for tox, but aren't there yet? http://tox.readthedocs.io/en/latest/config-v2.html

That page does a pretty good job explaining why our use case is hard to accomplish in tox today. 👎

sampsyo · 2016-12-20T16:20:24Z

docs/plugins/web.rst

+It also implements a subset of `subsonic api`_ that allows any subsonic client
+to play music from beets.
+
+.. _subsonic api: http://www.subsonic.org/pages/api.jsp


The docs should probably list the required dependencies (whether we use a separate plugin or expand the current web plugin).

I'll modify the doc when we'll be happy with everything else .

magne4000 · 2016-12-21T09:07:37Z

I took a look at #718, the ideas there are really good. But for now we still just have the web plugin all by himself, and I think that, for now, just extending it for further APIs is the way to go (simpler to dev, simpler for users).
If we just add things like a single python file that creates a Flask Blueprint, for each new API, this is really easy to maintain.

sampsyo · 2016-12-21T17:01:20Z

OK, cool. I get the convenience argument, and I now see that this doesn't have any extra dependencies beyond what we require for the web plugin itself. (In part because you hand-rolled your own XML serialization! Wow!)

One small argument on the other side is that I imagine the Subsonic API would, in an ideal world, prefer to "live" on a different port (4040?) by default. Putting all the APIs under one roof means that they all get our funky joke port, 8337, by default. That's not the biggest deal in the world, but it wouldn't be the case for separate plugins.

magne4000 · 2016-12-21T17:07:33Z

Regarding subsonic case, I don't think there is a "default" port to use. Other APIs may indeed use such ports, and would require some other Flask App instead of Flask Blueprints.

And you're right, there is no new dependency for this API, only the ones already there for the web plugin.

cobra2 · 2016-12-23T19:29:05Z

I agree with the statement of there is not a 'default' port to use. Sure subsonic, libresonic, madsonic all point their ports to 4040 by default. However; most of the installs that I've seen and used change those ports to 80/443 or something personal to the user.

magne4000 · 2017-01-12T14:31:41Z

I moved random functionnalities to a dedicated util module.

magne4000 · 2017-01-12T15:34:29Z

I think that we can consider this PR as mergeable now, tests with real clients are conclusive.

sampsyo · 2017-01-12T18:22:39Z

OK; thank you! I'm still a little dissatisfied with the unfortunate impact the tests have on the Tox config, but I'll keep looking for a solution for this…

ghost · 2017-01-12T22:51:50Z

@magne4000 : I don't see the tests being skipped if lxml is missing as per #2321 (comment) Or did i miss it?

magne4000 · 2017-01-13T08:36:40Z

xmlunittest can't be installed without lxml previously installed, and testing xmlunittest is the real functionnality that we want to test for presence.

irskep · 2017-01-14T20:18:15Z

beetsplug/web/subsonic.py

+        if wrap or isinstance(d, dict):
+            xml = u"{}</{}>".format(xml, root)
+
+    return xml


Why all the string building instead of using the built-in XML builder in the standard library?

(I'm not a project maintainer, just jumping on here because I am also working with and potentially modifying the web API.)

That's something I based on some existing code, made some improvements.
I didn't want to completelly rewrite it, because it works and it's concise.

beetsplug/web/subsonic.py

irskep · 2017-01-14T20:21:16Z

beetsplug/web/__init__.py

@@ -309,6 +309,7 @@ def __init__(self):
            'host': u'127.0.0.1',
            'port': 8337,
            'cors': '',
+            'subsonic': True


If subsonic introduces a bunch of new dependencies, perhaps the default should be False?

No more deps than web plugin, so no risks on that side.

Oh, I misread the lxml comments, sorry.

irskep · 2017-01-16T21:24:08Z

To future-proof against future web plugin extensibility, perhaps it makes sense to namespace the subsonic API under /subsonic/? (It's possible that it already does this and I just misread, but if so, then the docs should probably explain what the new endpoints are.)

irskep · 2017-01-16T21:41:21Z

beetsplug/web/__init__.py

+            if self.config['subsonic']:
+                from .subsonic import subsonic_routes
+                app.register_blueprint(subsonic_routes)
+


It seems like this strategy could potentially be used to implement the extensible web plugin, #718!

Imagine implementing this as a separate plugin called subsonic. In that plugin's __init__.py, you could do something like this:

# beets/beetsplug/subsonic/__init__.py from beets.beetsplug.web import register_blueprint subsonic_routes = Blueprint('subsonic', 'subsonic') @subsonic_routes.route('/rest/getMusicFolders.view', methods=['GET', 'POST']) def v_music_folders(): pass # whatever register_blueprint(subsonic_routes)

On the web/__init__.py side, it would work like this:

# beets/beetsplug/web/__init__.py _BLUEPRINTS = [] def register_blueprint(blueprint): _BLUEPRINTS.append(blueprint) class WebPlugin(BeetsPlugin): # ... def commands(self): # ... def func(lib, opts, args): # ... for blueprint in _BLUEPRINTS: app.register_blueprint(blueprint)

This would require almost no changes to your patch, except that you would have to enable subsonic as an independent plugin. But when you run beet web, it will (I think?) include the subsonic routes.

@sampsyo what do you think? Am I wrong about how plugin imports and bootstrapping works?

And then eventually the AURA stuff could just be implemented as a separate plugin namespaced under /aura/.

Yes, this seems like it could work! The more typical strategy for beets would be to define a method on the plugin class to get a set of blueprints, which could also work here—the blueprint would still be defined at the module level, but it could be registered in a callback. That's worth exploring!

I'll find a proper way to do it 😃

magne4000 · 2017-01-17T08:25:38Z

@irskep subsonic API is defined to run under /rest/ namespace. Some clients won't be able to handle URIs such as /subsonic/rest/.

magne4000 · 2017-01-17T10:12:54Z

Moved to dedicated plugin (with corresponding doc).
@sampsyo Eventually, I just added a register_blueprint into web plugin, it's the simplest and most Flask way to do it I presume.

irskep · 2017-01-17T18:21:07Z

Wow, I don't know about you, but this seems so much better to me.

But reading the code, it occurred to me...why not just import beetsplug.web.app and call app.register_blueprint()? Then there is no more API surface area at all, except that app becomes part of the "public" API of the web plugin rather than just internal.

Sorry for not thinking of this earlier.

magne4000 · 2017-01-18T09:24:23Z

@irskep that's indeed simpler ! I'll document this in Web plugin then.

irskep · 2017-03-20T17:50:29Z

As a random person who happens to be commenting on your PR, this looks good to me. Nice separation of new plugin, existing web plugin, and core library changes.

muff1nman · 2017-04-22T21:59:23Z

@magne4000 which clients do not support arbitrary prefixes? As far as I am aware, you should be able to prefix the rest api at least one level down.

anarcat · 2017-06-29T15:59:09Z

wow, this is pretty awesome! i think the only thing that needs to be fixed here is to remove the conflicting file, but otherwise - is there a reason why this shouldn't be merged?

adamdmoss · 2017-06-29T18:41:30Z

Very excited by this so I thought I'd give it a spin!
Using play:Sub on iOS, to log in and get the artist+album count it does this:
GET /rest/ping.view?c=playSub&f=json&p=enc%3A626172&u=fooooo&v=1.9.0 HTTP/1.1
POST /rest/getGenres.view HTTP/1.1
POST /rest/getArtists.view HTTP/1.1

Unfortunately, both POST requests fail with the same traceback:

[2017-06-29 11:34:41,153] ERROR in app: Exception on /rest/getArtists.view [POST]
(or [2017-06-29 11:34:41,231] ERROR in app: Exception on /rest/getIndexes.view [POST])

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/pi/code/adambeets/beetsplug/subsonic.py", line 439, in v_indexes
    letter = artist[0].upper()
IndexError: string index out of range

ahstro · 2019-04-02T14:29:53Z

Whatever happened to this PR? How come it never got merged?

sampsyo · 2019-04-02T15:17:39Z

I'd still be interested! If anyone's excited to take it up, it would be great to split out all the components of this large change. There are changes to the random plugin, some CI testing changes, some core field additions, and then the plugin itself. It would be truly great to be able to review and tackle those pieces one at a time so we can be confident in the core changes.

magne4000 · 2019-04-03T13:45:28Z

I probably won't have time anytime soon to update this PR, so if someone wants to take over, feel free to do so.

sdrik · 2020-02-10T17:12:03Z

I've just came across this PR a few days after reviving another similar project (https://github.com/dangmai/beetsonic) which I've forked at https://github.com/sdrik/beetsonic.
My goal is to consume my Beets library from the DSub Android client. While quite a bit hackish, it is working well for my use case right now.

If there is still interest in updating/merging this PR, I'm willing to volunteer.

jtpavlock · 2020-07-10T03:32:59Z

@sdrik (or anyone else willing to continue this work) it seems the original repo for this PR no longer exists. You may be better off taking what code you can from here and opening your own PR.

Also, as @sampsyo suggested, it would be ideal to try and split this up a bit into separate PRs. Reviewing a ~1,300 line PR can be exhausting 😄

magne4000 mentioned this pull request Dec 15, 2016

Web audio #1002

Closed

sampsyo reviewed Dec 20, 2016

View reviewed changes

irskep reviewed Jan 14, 2017

View reviewed changes

beetsplug/web/subsonic.py Outdated Show resolved Hide resolved

irskep reviewed Jan 14, 2017

View reviewed changes

irskep reviewed Jan 16, 2017

View reviewed changes

Subsonic API

3e6efd4

magne4000 added 6 commits March 20, 2017 14:55

Also add xmlunittest to test coverage

ef5f921

Send comprehensible 404 when file is not found

a4697e4

skip subsonic tests if xmlunittest package is not found

181dc34

Moved subsonic into its own plugin

1aa8ae5

Document Flask Blueprint usage

8dcdfc6

Remove duplicate Flask URL

091333b

arcresu mentioned this pull request Apr 6, 2019

random: move functionality into beets.random #3206

Merged

jtpavlock closed this Jul 10, 2020

Subsonic API #2321

Subsonic API #2321

Conversation

magne4000 commented Dec 15, 2016 • edited Loading

Implemented methods

Unimplemented methods:

sampsyo commented Dec 16, 2016

magne4000 commented Dec 16, 2016 • edited Loading

sampsyo commented Dec 16, 2016

magne4000 commented Dec 19, 2016 • edited Loading

magne4000 commented Dec 19, 2016 • edited Loading

magne4000 commented Dec 19, 2016 • edited Loading

sampsyo commented Dec 20, 2016

magne4000 commented Dec 20, 2016

sampsyo commented Dec 20, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

magne4000 commented Dec 21, 2016

sampsyo commented Dec 21, 2016

magne4000 commented Dec 21, 2016 • edited Loading

cobra2 commented Dec 23, 2016

magne4000 commented Jan 12, 2017

magne4000 commented Jan 12, 2017

sampsyo commented Jan 12, 2017

ghost commented Jan 12, 2017

magne4000 commented Jan 13, 2017 • edited Loading

Choose a reason for hiding this comment

magne4000 Jan 16, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

irskep commented Jan 16, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

magne4000 Jan 17, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

magne4000 commented Jan 17, 2017

magne4000 commented Jan 17, 2017 • edited Loading

irskep commented Jan 17, 2017

magne4000 commented Jan 18, 2017

irskep commented Mar 20, 2017

muff1nman commented Apr 22, 2017

anarcat commented Jun 29, 2017

adamdmoss commented Jun 29, 2017 • edited Loading

ahstro commented Apr 2, 2019

sampsyo commented Apr 2, 2019

magne4000 commented Apr 3, 2019 • edited Loading

sdrik commented Feb 10, 2020

jtpavlock commented Jul 10, 2020 • edited Loading

magne4000 commented Dec 15, 2016 •

edited

Loading

magne4000 commented Dec 16, 2016 •

edited

Loading

magne4000 commented Dec 19, 2016 •

edited

Loading

magne4000 commented Dec 19, 2016 •

edited

Loading

magne4000 commented Dec 19, 2016 •

edited

Loading

magne4000 commented Dec 21, 2016 •

edited

Loading

magne4000 commented Jan 13, 2017 •

edited

Loading

magne4000 Jan 16, 2017 •

edited

Loading

irskep commented Jan 16, 2017 •

edited

Loading

magne4000 Jan 17, 2017 •

edited

Loading

magne4000 commented Jan 17, 2017 •

edited

Loading

adamdmoss commented Jun 29, 2017 •

edited

Loading

magne4000 commented Apr 3, 2019 •

edited

Loading

jtpavlock commented Jul 10, 2020 •

edited

Loading