need standardized display function for non-text data (e.g. arbitrary MIME types)

[stevengj]

We need some kind of standardized display(data) API, where data is a Dict mapping MIME types to data, for Julia programs to request output of non-text data.

This is essential for good IPython support, but we can also support other display mechanisms (e.g. popping up Tk windows to display images).

@fperez, your experience would be helpful here.

[timholy]

Images already act like Dicts (they are "arrays with properties"), so we may be most of the way there. With more detail on the specific behavior you're looking for, I can discuss more concretely.

[stevengj]

The data might have several representations, keyed by MIME types. For example, data["text/plain"] might be a plain text representation, whereas data["image/png"] might be PNG data (not sure how we want to encode this as a Julia type).

[Keno]

It would be best if we also had some way to query what output is currently supported.

[stevengj]

Let me propose the following.

We add a new abstract type, Display, and three new functions: display, pushdisplay, and popdisplay.

• We internally maintain a stack of currently registered Displays. pushdisplay(d::Display) pushes d onto the stack, and popdisplay(d::Display) removes it. (If you push d multiple times, you need to pop it the same number of times to completely remove it.)
• display(d::Display, data::Dict) attempts to display data on d, returning true if d was able to display data and false if data contains no supported types. The keys of data are MIME-type strings, and data[mimetype] is the corresponding data in some format. string(data[mimetype]) should return the data in a String format (which should be the standard base64 MIME encoding in the case of binary data types like images).
• display(data::Dict) attempts to display data on at most one registered Display, trying the most recently added first and stopping when one of them succeeds. Returns true if some registered Display succeeds, and false if data was not supported by any registered Display.
• display(d::Display, mimetype::String) returns true if d supports mimetype and false otherwise. display(mimetype::String) returns true if any of the registered displays supports mimetype and false otherwise.

Also:

• If you want a type T to display non-text data when it is returned in the REPL, simply define a repl_show(io::IO, v::T) function that calls display. (For example, you could do display(...v data...) || show(io, v) to call show as a fallback if display failed to find a suitable Display, or you could call both display and show if you want to display both text and non-text representations.)

[dcjones]

+1

I have a crude version of this in compose. A dictionary maps mime types to functions, so how a graphic is displayed can change depending on the context. It would be great if this was standardized in julia.

One issue that might come up: there may be cases where a pushed display can display a particular mime type, but you would rather it didn't. Is there a nice way to push a display, while ignoring some mime types?

[timholy]

Suppose I start a fresh Julia session, and say display(img). It's going to need to pop open a new window for me. I'd like some kind of "handle" returned so I can manipulate the display (close it, resize it, change its properties, etc). How does this interact with your proposal to return success/failure information?

[stevengj]

@dcjones, it would be simple to write a Display type that wraps around another Display type and filters or transforms the data and MIME types in some way (e.g. to filter out some MIME types, or to transform one MIME type into another, e.g. to render SVG to PNG).

@timholy, in a fresh Julia session at the command line I'm not sure if we would have any display handlers registered, so display(img) would return false. In an IPython session, there would be a default display hander, but it would be for inline graphics, so you wouldn't expect to be able to resize it or change its properties. So, I think that if you want to have more control over how something is displayed you need to call a lower-level function in some specific display library like ImageView.

[stevengj]

We also probably want to make it easier to write base64-encoded strings, since I don't think we have a function for this now. Probably via a bio = Base64Stream(io::IOStream) so that print(bio, foo) prints a base64-encoded foo to io. Then on top of that you could write a base64(foo) function (ala sprint).

[timholy]

Suppose I've already said using ImageView, and that registers the handler.

My point is much more about the return value: having display(img) return true at the Julia repl doesn't seem terribly useful. How do I get the handle? Conversely, if I want to have ImageView.display(img) break your paradigm, then won't it be unusable from the ipython-notebook?

How about a counterproposal: it returns nothing if display fails, and anything else if it succeeds.

[stevengj]

@timholy, If you know you want to display using ImageView and use its facilities, why would you even use the Base.display function at all? Why not just call ImageView directly? And if you don't know what display backend is being used, then returning some opaque "handle" is useless because you have no idea what to do with it.

As long as you are running the Julia IPython kernel locally (on the same host/account as the notebook), then you are free to pop up windows and use event loops exactly like you are now; the display backend doesn't have to be inline, it is just that an inline handler might be pushed by default. (This is no different from using Python from the IPython notebook: it can pop up GUI windows with no problem as long as the kernel is local.) (And even if the kernel is not local, you could still pop GUI windows via X11 or similar.) So, I don't understand your concern about ImageView "breaking my paradigm".

I don't like your counterproposal because (a) I don't like losing type-stability and (b) returning "anything else" seems totally useless unless you know what display backend is being called, and if you know that then you might as well call that display backend directly.

[timholy]

OK, maybe I don't understand the scope of what you're proposing. This is not about a new generic display function in base/ that will force compatibility on all others? (In case you don't realize this, ImageView defines a display function and exports it.)

Basically, if it doesn't hinder or interact with ImageView in any way, then carry on :-). If there's some way you want it to interact with ImageView, we should keep talking.

[stevengj]

@timholy, ImageView is free to have a function ImageView.display(img::Images.AbstractImage) that returns whatever it wants, but that is simply multiple dispatch and is orthogonal to this discussion since Base.display will have a different method signature. (The only change to ImageView's source code is that you will have to import Base.display to add the new method signature.)

(And, if you want, ImageView can also define a Display type that acts as an opaque backend for Base.display of supported image types.)

[timholy]

Sounds as if we're fine. ImageView.display also works on plain Arrays, but it sounds as if you're only considering Dicts as arguments to your display function. So there's no conflict.

And sure, I'll be happy to add whatever for IPython notebook integration.

[stevengj]

I had a long discussion with the IPython guys (thanks @fperez), and they convinced me that my proposal is not really right. The display function(s) should be attached to the type, just like show. (And there is also some benefit in using the same spelling as IPython unless there is a good reason to do otherwise.) In particular, if we follow the IPython model (which seems to have worked pretty well for them, and would probably work even better in Julia due to our dispatch semantics), we would have the following kinds of functions:

• Analogous to the show(io, x) and repr(x) functions that provide a string representation of x, we would have formatter functions show_foo(io, x) for various media types (e.g. foo could be one of {html, svg, png, jpeg, latex, json, or javascript} following IPython, possibly with others added later) which write the corresponding data to a stream (and these would be called by convenience functions repr_foo(x) that return a String or Array{Uint8} of the data). You then just define whatever show_* functions you want for a given type to indicate which rich representations it can be rendered as.
• We have an abstract Display type representing a given display backend, e.g. IPython's inline rendering or a Tk GUI.
• For a given Display type D, one defines functions display_foo(d::D, x; raw=false, metadata...) (for foo in html, png, etc.) which call repr_foo(x) and display it in D. The keyword argument raw=true is used to indicate that x is already the raw data returned by repr_foo, and metadata could be arbitrary metadata (IPython uses this to indicate e.g. a desired image rescaling; there is the question of whether we use keywords for this or a Dict ... IPython uses a dict, but it seems like using keyword syntax would be nicer).
• We also have a function display(D, x; metadata...) that looks at what repr_foo methods are defined for x and picks a preferred format to display on D (usually the richest available format).
• We also have a function display(x; metadata...) that calls display(default_display, x; metadata...) for some default current output device, which could be changed by set_display(d::Display). (e.g. in the console REPL, this could default to just console output using plain show, whereas in an IPython front-end it would send several available representations of x to IPython as a mimetype=>data dictionary).

My feeling is that display(x) would be like print in that it would return nothing. On the other hand, display(d::D, x) could return whatever the implementer wants for a given type D <: Display (e.g. a display handle of some kind for ImageView.

(Note that to query whether a given Display type supports a given format foo, you can simply use method_exists on display_foo.)

[timholy]

A few comments:

• raw is confusing to me (it's not as raw as the original data I wanted plotted...). How about rendered = true? In general, I wonder if the word render should be considered as an alternative for display in some of these discussions. To me render is more abstract, display strongly tied to visuals. Maybe you're only intending to talk about visuals, in which case it's a fine choice of words.
• "...that looks at what repr_foo methods are defined for x and picks a preferred format..." Are you suggesting we call methods here? Or do multiple try/catches? Or is this just checking some dict? (The latter being probably the best of these, although it means that simply defining the functions won't be enough, you also have to register them.)

On the last point, I initially felt you got that backwards, but I find it growing on me. I guess the idea is that if you want mess with "handles," then you might as well have the user separate the initial display into two steps, display creation (which gives you one handle to the window/canvas/whatever) and rendering (which returns a handle to the actual object(s) being viewed in the display).

[stevengj]

@timholy, rendered does not seem like the right word here for raw, as I don't think "rendering" is the most common term for converting an object to a raw byte array in a given format. The only question is whether it is a high-level Julia object on which we need to call repl_foo(x), or whether it already a byte string/array. In any case, in questions like this that are mainly a matter of taste, my inclination would be to follow IPython precedent rather than introducing needless spelling changes in syntax that is extremely similar.

As to whether display uses method_exists or try ... catch, that is up to the implementation. But performance is not really critical here, as any kind of rich I/O will not be limited by the time to call method_exists.

Basically, the logic here is that the user wants a way to say display this object in the richest format supported by this device and this type, and the information about what formats can be displayed belongs with the device (via display methods) while the information about what formats can be produced belongs with the type (via show_foo methods).

[timholy]

On Thursday, July 25, 2013 03:57:45 PM Steven G. Johnson wrote:

The only question is whether it is a high-level Julia object on which we
need to call repl_foo(x), or whether it already a byte string/array. In
other words, the difference is not in the output of display, the
raw keyword refers to a difference in the input.

I understand that. My point is that one can think of "repl_foo" as doing the
rendering: it take what I think of as your "raw data" (which is why I think
the word "raw" is a bad choice here) and converts it into some visual
representation.

But whatever. Just trying to be helpful.

[stevengj]

Looks like I can avoid the need for keyword args entirely, so no longer an issue.

[stevengj]

I've refined these ideas and implemented them in our Julia IPython kernel. See:

• DataDisplay module: the abstract display functionality.
• IPythonDataDisplay module: an inline-graphics (etc.) display for the IPython front-end.

I made a few minor polishes to the above proposal. Following @JeffBezanson's suggestion, I use write_foo(io, x) instead of show_foo(). I also include a text/plain format and default write_text(io, x) functions which just use repl_show, as well as a default Display that writes this text representation to STDOUT.

There are no more raw or metadata arguments. However, if you call e.g. write_html (or write_latex or any other text-based format) with a String argument, it assumes that the string is already in that format and does no translation. Similarly if you call write_png (or write_jpeg or any other binary format) with a Vector{Uint8} argument it again assumes you are passing binary data in the requisite format. This allows you to do e.g. display_html("<b>bold</b> text") and it does what you expect. The whole metadata thing in IPython is a bit wonky and may need to be rethought (they barely use it now), so I'm leaving it out here.

[stevengj]

If people like this approach, I'm hoping to get something like this merged into Base before long, so that people can start writing write_foo functions to expose rich representations for their types (from equations to images to plots). This will make them immediately render beautifully in the IPython front-end, which already supports this API via our prototype kernel.

And packages like ImageView can begin to provide alternative Displays (e.g. if you just implement a Display type with display_png and display_jpeg methods, then setting your display to be the default will automatically display any type that can write those formats).

ping @StefanKarpinski

[stevengj]

I will probably put back in the notion of a default display stack, rather than a single default display, so that you can easily push a display that only handles a small number of types and rely on other displays for the rest.

My main other concern at this point is making it easier to add new MIME types, but I think this can be done by adding a lower-level interface without losing the simplicity of adding and using write_foo and display_foo methods for common types.