data interface for interactive (updating) vis

[jacobhinkle]

I like the idea of plotting inside the web browser. However, in the current scheme it looks like the idea is to serve some html/js using d3 to create static plots sequentially in the web REPL. This is probably a nice simple way to get some basic plotting working, but it seems hard to extend so that one can "hold on" and put multiple plots on the same axes, or to update an existing plot to see time-varying behavior for example.

To me, it makes more sense in the long term for julia to include a text-based REPL through some network protocol maybe, as well as a separate data interface. Inside julia we could make data available to the outside world with something like publish(x,"some descriptive name for x")
whenever we have updated the x variable and are ready for those changes to be seen in a plot. Then a client can see the published data (shared via maybe JSON?) and plot it, or use the mtime to poll and update plots, etc. To keep the analogy with matlab, the plot() command can be a wrapper that combines the publishing with a network call to the client that suggests the graphical client should now plot y against x. Matlab xlabel, ylabel, title, etc can also be implemented as hints that are sent to the client regarding a suggested data view (created with a figure() command maybe).

This kind of data publishing interface of course could be useful for more than just visualization. It could provide a simple data sharing interface between different Julia sessions, a poor man's database but tied to computation instead of storage.

[JeffBezanson]

A better design for plotting stuff is absolutely a big need right now. The publishing idea sounds really good, since we want support for web-based collaboration too.
With the resources we have (technically only one full-time person...) we probably won't be able to do this unless somebody new is willing to own this part of the project. It can even be a separate project that julia integrates with. Clearly having graphics "built-in" to a programming language is old-fashioned, and I imagine it would be possible to use the publish/visualize system from multiple languages.
If you want to take the lead on this, feel free, and we'll do what it takes to adapt our part of it to your design.

[StefanKarpinski]

+1. We really need this. Having something language agnostic that can work with D3 as a frontend or other publication mechanisms would be very cool.

[JeffBezanson]

There is a huge void to fill here. Web-based, neutral-data-model plotting that's good enough to replace existing tightly-language-integrated systems is something whose time has come.

[jacobhinkle]

A standalone data server running either on your workstation or a single node of a cluster, etc, would be the way to go. It could accept per-session nd-arrays of the basic numeric types along with some metadata like a description and mtimes. The data would then be served to clients via JSON or other. It surprises me that nothing exists that really fits the bill (at least that I've heard of)

I'd love to see this happen, but as an abd phd student i have very little spare time to offer currently.

[StefanKarpinski]

Bah. I've been an ABD PhD for three years now :-P (Please don't emulate me, kids.)

[jacobhinkle]

I have a small group of people thinking about this issue now. we are hammering out a standard protocol for a middle-man data server that would accept and serve data to/from web clients as well as julia/matlab/R/python/C++ etc. When we have something at all useful I'll post back. Cheers.

[staticfloat]

@jacobhinkle, do you have anything share right now? I've been thinking about the same problem, and wanted to check in to see if you've gotten anywhere.

[ViralBShah]

With the work on ipython and gadfly, there is enough to warrant closing this and focus on having specific issues on this topic.