Hypermedia Module

[pemrouz]

A little experimental thinking/coding, mostly to work out what a good API for this module would feel like. @mere, let me know if you have any thoughts on this..

Normally, you can register some array/object/function/whatever against a key:

ripple('key', { value })

From the tests, this module lets you load a remote resource if the body is a URL:

ripple('github', 'https://api.github.com')

// ripple('github') would then return the root resource

and it lets you follow deep links (from the nearest resolved link):

ripple('github.current_user_url.id')
ripple('github.current_user_url.repos_url.0')
ripple('github.current_user_url.repos_url.0.owner.login')

If it hits a property whose value is a URL (in the GitHub API, all those suffixed with *_url), it resolves and caches them as an intermediate resource. So accessing ripple('github.current_user_url.id'), would also populate the resource ripple('github.current_user_url').

You can pass any extra headers you want (in this case, basic auth) to be used in the request by setting the http header.

ripple('github', 'https://api.github.com', { http })

(note: hypothetical beyond this point..)

You can also alias deep paths:

ripple('repos', 'github.current_user_url.repos_url')

And use the same set syntax to send messages back:

ripple('github.current_user_url.repos_url', { new repo details })

Which would either pick up the method from the API, or use template expansion for GET, or default to 'POST', or allow explicit overrides in the headers (third parameter).

---

A few notes:

• I think this - i.e. the generic ability to follow links at any arbitrary point in the resource - is more valuable than a hypermedia client for just one or many of the media types (HAL, Siren, Collection, etc). Thankfully, there's the GitHub API to demonstrate what this vanilla links API would look like, although they need not suffix everything with _url, since the idea is that a link and an embedded resource could be interchangeable (i.e. repos: [..], or repos: https://..) with no impact to the consumer.
• This module is similar to Falcor, Relay etc, but uses standards like URLs over things like JSON Graph and Graph QL. It's not as network efficient (no batching), because it's just one end of the pipe, whereas the Netflix and Facebook model demand fullstack buy-in (client and server). If you are going to do that however, then you can do much better than those. You can choose to have an API that is RESTful over WebSockets, rather than HTTP. Everything gets "pushed" rather than "pulled", so everything is always up to date.
• Despite the sync module, there's still value in having this module however, since you can use this to consume resources on the server from an external API you don't have control over (e.g. GitHub), buffer in-memory, and then use sync between your client and server for realtime data synchronisation. Or if you are just writing a client, you can use this instead of the sync module.

//cc @mstade @sammyt @tomsugden would be interested to hear if you have any thoughts on this too..

[pemrouz]

Some awesome in-depth talks courtesy of @mamapitufo on Falcor, Relay and Om Next. Recommend checking out. I liked how JSON Graph is JSON with "symlinks" (represented by arrays), which is similar to the above but you can access a different resource via deep path or URL. Relay (and Om?) seem to unnecessarily require all fragments to be composed through your view tree. The Datomic Pull Syntax is simpler than defining the response shape, but not as simple as just using keys.

In general, it's a powerful paradigm to declare the data required on the view. You can abstract things like fetching and make it more efficient. The missing killer feature though in all of the above, is to have resources actually pushed to clients rather than pulled. Falcor talk mentions the high-latency of using REST, multiple network calls, and using finer-grained resources instead, but then only to pack up multiple small messages before sending it over the wire.

All those issues go away when you synchronise and stream smaller resources independent of one another over a single WebSockets connection.

[mstade]

I don't know man, I get all disappointed and sad when I see talks like that Falcor talk, where the guy basically dismisses decades of systems research and announces a format-that's-not-new-but-really-it's-new and then goes ahead and basically says coupling clients and servers is a good thing. There are some things that are good in there, such as declaratively describing what you're looking for, and having the server be smarter than static file serving. But there's no need for this to require proprietary formats and protocols, and moreover it's much too simplistic to just reduce applications and views to nicer formatting of hierarchical data.

Not all data is hierarchical, of course, which is why they ended up with JSON Graph in the first place. It's a neat hack to efficiently describe graphs in a JSON-like format (because it's JSON with semantics) but hierarchical data is not the only problem, and that's the entire point of Fielding's dissertation – semantics! It's not just traversing a path, it's what you do along that path as well, the semantics of those actions. It's what I tried to get at in my gist on hypermedia client blues. Compare the traversal compositions I present there with the paths the Falcor dude talks about. You could get the same kinds of optimizations that you get with something like Falcor, but without the need for inventing new-but-old-but-whatever-let's-do-it-again formats, and without the need for coupling clients and servers.

Whatever, I'm ranting; but it does sadden me that you have big companies like Netflix and Facebook completely miss the ball and dismiss all the good groundwork laid. I don't even know if it's in the name of propriety and not-invented-here-syndrome, it might just be ignorance. Or not seeing the forest for the trees or something. Whatever, I'm ranting...

[pemrouz]

Completely agree re: proprietary formats and the unnecessary coupling.

Just revisited your epic gist, a lot of good stuff in there. I'd be interested to know what you think of the above proposal as a follow up. The main difference is a more composable, declarative way to traverse hypermedia vs the imperative .to API, although you also touch on this under the "concise" heading (e.g. most-popular -> payment) and further link to Jon Moore's dot notation (which is surprisingly basically what I implemented). This syntax obviously works with JSON, but if configured with a different profile as he shows, it could also work with HTML and others. Really the key affordance seems to be the ability to follow a link, and I don't think we could express this anymore tersely than with just a .. The above also has the benefit of simply caching sub-resources against the path specified, and also works well with realtime/reactive resources (i.e. since to Ripple a sub-resource is just a resource, it could push partial updates to just a section of a larger resource). The dot notation could also be used in the markup without any API at all, e.g:

<!-- standard declaration -->
<twitter-feed data="tweets">

<!-- declaration referencing a sub resources -->
<repos-list data="github.repos">

[pemrouz]

More off-topic, but: some additional interesting comparative thoughts in this ClojureNYC talk. The one thing that looks a bit worrying is the tendency towards almost defining a new query syntax. Might as well use SQL if you go down that route! I must admit I've flirted with this a few times too for things like key, join, etc in utilise, but it's incredibly more flexible/composable to specify data transformations in JS rather than a new syntax. Besides that, I find it makes more sense for views to declare what they need (i.e. keeping co-location benefits), but not specify the definitions of those resources in the views. I use the to function for definitions (which is essentially a response handler, decoupled from a request). For example, some view specifies that it needs the list of staff:

<some-view data="staff">

The following (real world example) to function for the staff resource would send a subset of the list of users (those in the same hospital), plucking a few properties, and expand a foreign key. It would also send nothing if the user is not logged in. I'm not sure how you would achieve the same in the aforementioned frameworks. With their own respective query syntaxes, I doubt it would be anywhere near as elegant:

function to() {
  var me = ripple('user').whos(this) 

  return !me || !me.email
    ? [] 
    : ripple('users')
        .filter(by('hospital', me.hospital))
        .map(key(['id', 'name', 'speciality']))
        .map(join('speciality', 'specialities'))
}

And for reified mutations I use the from function (which is essentially a request handler, decoupled from a response). This same hook also pre-empts the need for RPC call's (code smell).

I look forward to seeing how all these other innovative approaches continue to evolve. One major question I have for the authors though, is whether they've also considered that if views now declare/define what they need, then as the intermediary between views and the server, you can push to them what they need, rather than pulling? (i.e. constant backpressure on views) //cc @swannodette @leeb @jhusain.

Anyways, as far as this hypermedia client is concerned, the dot notation seems to be a good flexible foundation. I just need to add profiles to support specific formats. Some of the general efficiency concerns, I've addressed in the backpressure module and Ripple v0.4.

[pemrouz]

moving this discussion to rijs/fullstack#6