Roadmap: Galaxy Interactive Environments

[hexylena]

Issue for tracking planned IE enhancements

IEs on a remote host

• IEs should be able to be launched on remote hosts Add configurable Docker hostname #356
• Should use -P flag of docker. Per @jmchilton's comments and some discussion on IRC, it was obvious that this is the solution we've been looking for; offload port detection to docker, avoiding any race conditions, and allowing docker on a remote host to function 100% properly. Refactor GIE Launching to use -P #790
• It would be nice if we could somehow take advantage of the built-in job scheduling in Galaxy and the docker job scheduling available as part of that. Not useful with swarm & friends, and dedicated docker preferred by admins to keep images away from jobs.

General IE improvements

• IEs don't work with upstream proxying rules IEs don't work with upstream proxying rules #471
• ipython can be run insecurely and have port # removed from URLs Replace docker_port with proxy_prefix bgruening/docker-ipython-notebook#60
• rstudio can remove port # from URLs hexylena/docker-rstudio-notebook@a77f025
• javascript needs a refactor, remove some of the requirejs + login code for ipython and make it unsecured.
• Limit/Permit subsets of users to access GIEs, especially based on regex. (For "mixed" galaxy (pub + university users), cannot allow non-university people to run GIEs)
• s/IE/GIE/g
• need to consider providing a base IE image
  • nginx proxy, auto-kill script
• Better handling of upstream proxy. Currently I'm writing apache confs that look like:
• Replacement proxy

  • Current proxy requires NodeJS

  • Need to kill containers from proxy

    <Location /galaxy/ipython>
    ProxyPass http://localhost:8800/galaxy/ipython/
    ProxyPassReverse http://localhost:8800/galaxy/ipython/  
    </Location>

    I would like to be writing apache confs like:

    <Location /galaxy/proxy_ie>
    ProxyPass http://localhost:8800/galaxy/proxy_ie/
    ProxyPassReverse http://localhost:8800/galaxy/proxy_ie/  
    </Location>

• [ ] CRIU + persist to history
• [ ] Inform user if pulling container/provide some interaction saying "hey, we've detected the container isn't available, so we're pulling it, please be patient". xref Add BAM iobio visualisation as Interactive Environment #1131 Admin's job.

IEs with multiple images

This is more of a wishlist feature...

• it'd be nice if IEs could be launched from docker-compose.yml files, somehow. [WIP] GIE Golang Proxy and Docker Compose Support #852

e.g. I'm slowly working on an Apache Zeppelin IE, it'd be great if I could re-use existing Zeppelin images and link an Apache Spark image, rather than having to bake everything into one.

This might involve shipping a docker-compose.yml in the settings folder, and applying a config variable somewhere marking this as a multi-image container. I worry about networking+picking out the correct ports and running multiple containers. Maybe we'd create a temp dir, copy in docker-compose, launch. We could do docker-compose ps and grab ports there maybe. This means config would have to either specially name the proxy image/we'd provide a self-killing proxy/the config would allow specifying the name of the main/proxy image.

IE Image selection

• have a single viz which allows launching N different docker images Generic GIE Launcher, GIE Image Chooser, multiple datasets #1403

IEs as Tools

• [ ] Allow running IEs as tools OR converting them into .py + tool.xml files IE Provevance #1923 It would be a nice project, but a lot of work. If someone is motivated they will discover the issue and produce something, no need to keep this issue open just for that.

IE UX

• provide a clear 'shutdown' button for IEs (from IE UX improvements #1586)
• make the Jupyter tooltip readable (it is too long for default width of history panel) (Addressed in Generic GIE Launcher, GIE Image Chooser, multiple datasets #1403)
  
• provide a way to start Jupyter without interacting with a dataset (Addressed in Generic GIE Launcher, GIE Image Chooser, multiple datasets #1403)
• Provide a sharable link to a notebook that would start Jupyter with the NB loaded in.
• [ ] Provide an API to launching GIEs (?) Why? We need a better use case for this.

[hexylena]

@bgruening during our seminar, I noticed that we can get rid of a LOT of the port-in-url code, and the login code for IPython. The nodejs proxy uses the galaxy user's cookie for routing, so we can get rid of that.

[hexylena]

@jmchilton @bgruening open ended question/brainstorming:

We now have support for killing containers from the proxy. One problem that users will encounter nearly immediately will be the lack of a way to pause their work. Did they get interrupted for >2 minutes? Did their network drop out? Did they have to run to the loo? All of these situations will result in their container disappearing.

So, the obvious answer to that is that we should provide some "pause" functionality, or way to say "hey, I'm taking a break, please don't kill this container for at least N minutes, if I'm not back by then it's fair game".

I considered that we could implement it on the Galaxy side which got me to thinking, why not build up the full pause+restore functionality and maybe more within Galaxy? Guacamole (the VNC magic container) did this nicely with image previews for per-user VMs. Galaxy could have a section which is just dedicated to launching + using IEs? Don't know how I feel about that. It dovetails nicely with the thought that it would be nice if users could select an IE from a dropdown, and that some IEs are more tailored to taking in a whole history rather than just one dataset (IPython, Rstudio) whereas some IEs are more tailored to one dataset (iobio)

[hexylena]

Had some more thoughts on my drive today:

• can we build + submit "real" jobs without the API? Then we should move towards that. That would let us build a "real" job which:
  • contacts AMQP, sends an "I'm alive on HOST:PORT" message
  • GIE proxy picks this up and opens up routing
  • galaxy picks this up (maybe redundant with GIE proxy) and the user's "waiting..." page is listening/polling and receives that message, setting up the iframe.
• the container(s) run until done as reported by GIE proxy/traffic (I think it'll be more clear that this is just as effective as monitor_Traffic.sh going forward)
• once done, the top layers of the containers + image metadata gets tarred up + persisted as a history element
• making that re-runnable will be hella unpleasant but I think it's possible.

Will organise these into proper bullet points/plans for above later.

[mvdbeek]

@erasche regarding IEs as tools, do you have any plans on how to mark input/output files?
At least for ipython it's pretty straightforward to generate *.py.
In fact, I could make the toolfactory consume *.ipynb files ... .

[hexylena]

@mvdbeek we've had some thoughts about these in the past:

• we could add type/input annotations to the notebooks, a la this and specifically the image embedded in that page
• ...

@bgruening and I have been trying to keep our priorities straight with that one, we don't want to be a toolfactory:

• we're not building permanent tools
• we're building quick one-off analyses
• we'd like for some way for people to re-run those quick analyses and iterate/share.

I think full fledged tool building may even be an anti-goal? Dunno. If you have ideas then we'd be happy to hear 'em :)

[jmchilton]

@erasche I think it is fine that full fledged tool building isn't your goal, that is a lot of room for great, transparent, interactive analysis. IE-2-tool though I would consider my primary interest in this endeavor - that would be awesome because it would fit so well with workflows.

[mvdbeek]

@erasche I remember @bgruening mentioned this, but I don't see how to turn it into a tool, unless you're thinking more in terms of an interactive app?

Yeah, I'm not too positive about building full tools with the toolfactory, but it's great for running things in a workflow. That's how I use it primarily in research projects, when I don't know yet if a full tool is justified.
I'll see if I can hack together a demo.

[hexylena]

@mvdbeek I'd considered those style of annotations for turning jupyter notebooks into tool XML.

@jmchilton yes, for IE-2-tool as you call it, I've mostly had a mental picture of in-notebook annotations + a tool that's dynamic enough to let you run any notebook as a tool, even if it's a bit painful. (E.g. a repeat box with an "input type" selector followed by the values to pass.)

[mvdbeek]

@jmchilton yes, for IE-2-tool as you call it, I've mostly had a mental picture of in-notebook annotations + a tool that's dynamic enough to let you run any notebook as a tool, even if it's a bit painful. (E.g. a repeat box with an "input type" selector followed by the values to pass.)

well, except for the "annotations" being outside of the script, that's how the (admittedly not very ) interactive toolfactory works. give me the weekend, and i'll give you the demo ;)

[mvdbeek]

@mvdbeek I'd considered those style of annotations for turning jupyter notebooks into tool XML.

Ahh, OK, that makes sense.

[martenson]

Provide a sharable link to a notebook that would start Jupyter with the NB loaded in.

added to the list

[bgruening]

Oh, yes an API endpoint to start IEs! This would also help testing them automatically.

[hexylena]

@martenson added to main checklist. Should be dead easy to do since you can already launch an IE just by knowing the visualization URL to open.

[yvanlebras]

Hi everyone, not sure this is the acurate place where posting issues testing Jupyter IE on Galaxy but I'm testing some """advanced""" features and I face a first issue: When you select another kernel than Python 2, like Bash, the functionality "save the current notebook in Galaxy" did'nt work.... even if I go back to the Python 2 kernel. I continue testing it, Thank you so much for this beautiful "joujou" !!!

[bgruening]

@yvanlebras I will look at time. For the time being have a look at you PATH. There should be a get tool and a put tool that can be used by any kernel from the commandline.

[martenson]

I am going to close this issue, since the checklist seems pretty much complete and future will likely revolve around interactive tools (xref #7494)