Deployment of Julia-ipython for web usage

[ViralBShah]

This is perhaps a bit of an umbrella issue. I would like to have julia-ipython be ready for the classes this Fall. With stuff such notebookcloud, hosting such a solution for general use on tryjulia.org through EC2 seems like a possibility. As discussed on the mailing list, the earlier web-repl used to get about 50 trials a day and a few hundred a day whenever there was a mention on hackernews etc.

It would be nice to have a list of all the things that have to be done, so that they can be tracked for such a solution to be taken live in the next few months.

So far, I thought JuliaLang/julia#2775 was a bottleneck, but that has been resolved.

[stevengj]

@ViralBShah, the IJulia stuff is getting close to being usable (I would also like to use it this fall). Can you look into what we have to do to get this on a server?

[Keno]

I've been doing some work with Docker containers. Potentially useful here.

[ViralBShah]

Does ipython provide anything for user/session management, so that you can deploy it behind a apache or nginx, which forks off a new session for every new user?

[Carreau]

Does ipython provide anything for user/session management, so that you can deploy it behind a apache or nginx, which forks off a new session for every new user?

Not really, currently we make the assumption that the person that start the server wil own all the kernels. There is a simple https/password (no username) mechanisme to prevent other users from logging into others' notebook.

There are tools like hydra and other I forgot the names that spawn one server/user + setup auth on a per user basis.

Real development for multi-user loggin in planed for next year (if nobody outside of core team tackle it :-) )

[ViralBShah]

@yarapavan Would it be possible for you to try this out, and see how far we can get in providing a hosted IJulia based on hydra or notebookcloud?

[Carreau]

@aculich was it you that showed me another tool to deploy IPython notebook ? Can't remember the name.

[Carreau]

Of course 5 min later, I find it : https://github.com/UnataInc/ipydra. I also drafted that some time ago.

[yarapavan]

@ViralBShah Interesting. Let me try out this week.

[stevengj]

@yarapavan, any progress?

[yarapavan]

@stevengj Planned for weekend. will have an update by Monday

[yarapavan]

Folks,

I've tried 3 different approaches this weekend.

1. NotebookCloud
2. Docker
3. ipydra (ipython-hydra)
4. StarCluser (not tested)

Here is a brief summary:

1. NotebookCloud: Demo Google App Engine URL: https://ijulia-notebookcloud.appspot.com/. Users login using Google Authentication, inputs AWS EC2 Access ID, Secret Key, existing keypair and a password for the notebook cloud. After successful authentication, launches an EC2 instance (http://aws.amazon.com/ec2/instance-types/) using iJulia AMI (ami-487a3c21). The web url for iJulia will be: https://:8888, protected by his password (in the web portal). AMI bundles Julia 0.2 nightly build (as of Aug 10, 2013) + iPython 1.0.0 on Ubuntu 13.04. I seem to hit few issues with iPython kernel restarting few times (AMI issues?). Had issues with (I)Julia core dumping in couple of instance types ( openblas issues? do not try t1.micro for Julia)
2. Docker.io: LXC + AUFS + cgroups = Docker. This provides better isolation and performance but a minimal ubuntu-based ipython deployment takes about 1-1.5 GB disk space. One can easily do port redirection to expose container ports via host PUBLIC port. EC2 was too slow for me to try docker and had to use my Eucalyptus private cloud to launch couple of iPython instances. Still have to build Julia. Interesting project to note here: https://github.com/miha-stopar/IPyDocker. My docker file for iPython 1.0.0 deployment is here -https://gist.github.com/yarapavan/6200580
3. iPydra: Inspired from iPython-hydra project, uses Flask and SQLAlchemy (Sqlite3 backend) to spawn iPython notebook servers. When you hit the URL: http://:5555, asks for a user name, and then setups a session, redirecting to any of the ports >=9500. Tested with inbuilt Flask server, which fails to start if child processes are running. It is simple and still buggy.

[aculich]

Thanks for the summary of everything you've investigated!

I need to loop all of you into a much wider discussion that we've been
having here
about JiffyLab https://github.com/ptone/jiffylab that has also included
folks
from D-Lab, LBL, Rackspace, Mozilla, and more. There's a lot of momentum
around this, so let's combine forces to move forward in a coordinated way
so we're not continuing to re-invent wheels and also provide a solution that
works well for the IPython Notebook, but that is also easily generalized for
other tools that follow a similar pattern: browser access to process running
in an LXC running on a virtual container.

On Sun, Aug 11, 2013 at 9:12 AM, Pavan [email protected] wrote:

Folks,

I've tried 3 different approaches this weekend. Here is a brief summary:

1. NotebookCloud

2. Docker

3. ipydra (ipython-hydra)
   4.

   StarCluser (not tested)
   5.

   NotebookCloud: Demo Google App Engine URL:
   https://ijulia-notebookcloud.appspot.com/. Users login using Google
   Authentication, inputs AWS EC2 Access ID, Secret Key, existing keypair and
   a password for the notebook cloud. After successful authentication,
   launches an EC2 instance (http://aws.amazon.com/ec2/instance-types/)
   using iJulia AMI (ami-487a3c21). The web url for iJulia will be: https://:8888,
   protected by his password (in the web portal). AMI bundles Julia 0.2
   nightly build (as of Aug 10, 2013) + iPython 1.0.0 on Ubuntu 13.04. I seem
   to hit few issues with iPython kernel restarting few times (AMI issues?).
   Had issues with (I)Julia core dumping in couple of instance types (
   openblas issues? do not try t1.micro for Julia)
   6.

   Docker.io: LXC + AUFS + cgroups = Docker. This provides better
   isolation and performance but a minimal ubuntu-based ipython deployment
   takes about 1-1.5 GB disk space. One can easily do port redirection to
   expose container ports via host PUBLIC port. EC2 was too slow for me to try
   docker and had to use my Eucalyptus private cloud to launch couple of
   iPython instances. Still have to build Julia. Interesting project to note
   here: https://github.com/miha-stopar/IPyDocker. My docker file for
   iPython 1.0.0 deployment is here -
   https://gist.github.com/yarapavan/6200580
   7.

   iPydra: Inspired from iPython-hydra project, uses Flask and SQLAlchemy
   (Sqlite3 backend) to spawn iPython notebook servers. When you hit the URL:
   http://:5555, asks for a user name, and then setups a session,
   redirecting to any of the ports >=9500. Tested with inbuilt Flask server,
   which fails to start if child processes are running. It is simple and still
   buggy.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1#issuecomment-22460150
.

[stevengj]

I'd love to have something working for classes in the fall, ~100 students. What would be our best bet in the near term?

[ViralBShah]

I believe @alanedelman will also want something this fall for his classes. We really need to make this work.

Is JiffyLab likely to be ready in the next 2-3 weeks?

@yarapavan I think there are other issues open where micro instances are trouble for julia. Best to use a small or medium instance. If you need API keys, I have some for julia, and I can send those to you. Also, rather than compiling, why not use @staticfloat 's deb packages?

In your opinion of the options 1-3, which one is most likely that is likely to work in the near term?

[ViralBShah]

Also pinging @amitmurthy with whom I had discussed these ideas at one point.

[yarapavan]

@ViralBShah Thanks. Learnt the hard way, t1.micro doesn't work for Julia! The problem seems to be that the CPU availability varies dramatically and unpredictably (Ref: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts_micro_instances.html). Tried using @staticfloat 's packages but had issues occasionally in other instance types too. Perhaps, these issues are specific to VMs?

@stevengj, @ViralBShah IMO, NotebookCloud is easier to deploy if the backend environment is EC2 (uses python boto library) although Docker-based approach would be the right way to do in near future. Cannot say much about StarCluster (http://star.mit.edu/cluster/index.html) with out evaluating.

[ViralBShah]

The relevant issue was: JuliaLang/julia#3263

I believe @JeffBezanson (and perhaps even @stevengj) knows the guys who are doing StarCluster at MIT, and it may be worth investigating that.

[stevengj]

I don't know the StarCluster folks.

[yarapavan]

Did any one try this -> JQuery Terminal connecting to a 'real' shell running in a EC2/server host to parse/work over http/websockets, or have customized parsing of user commands on the client side.

[ViralBShah]

JQuery terminal seems really cool. However, IJulia offers graphical capabilities and among others, it is also good for teaching.

[jverzani]

Does anyone have a working solution here? I've been trying to get ipydra to work and am way over my head. After adjusting some environment variables through trial and error and copying a profile-julia to the right place, I can get an IJulia notebook to open, but can't do any work. The kernel dies and the apache log has:

ERROR: key not found: HOME
in getindex at env.jl:25
in path at pkg/dir.jl:9
in find_in_path at loading.jl:18
in find_in_node1_path at loading.jl:29
in require at loading.jl:45
in include at boot.jl:238
in include_from_node1 at loading.jl:96
in include at boot.jl:238
in include_from_node1 at loading.jl:96
in process_options at client.jl:274
in _start at client.jl:349
at /home/julia/.julia/IJulia/src/IJulia.jl:10
at /home/julia/.julia/IJulia/src/kernel.jl:1
2013-08-27 15:56:20.336 [NotebookApp] KernelRestarter: restarting kernel (1/5)
WARNING:root:kernel 4b541d33-c05a-45c0-90ce-031241c0a09d restarted

I guess I'm not quite sure how to set the HOME variable for the www-data user so that julia recognizes it (I'm using Ubuntu + apache2 + wsgi + ipydra)

[staticfloat]

Try editing /etc/apache2/envvars. You can put export HOME=<path> inside
of it, but ensure that whatever you set that to, www-data has permissions
to modify it.

HOME is typically set from /etc/passwd, (the second to last field, each
field separate by commas) and usually defaults to something like /var/www.
You could also change it from there, if you like, but the
/etc/apache2/envvars approach is more general for all environment variables.
-E

On Tue, Aug 27, 2013 at 4:00 PM, john verzani [email protected]:

Does anyone have a working solution here? I've been trying to get ipydra
to work and am way over my head. After adjusting some environment variables
through trial and error and copying a profile-julia to the right place, I
can get an IJulia notebook to open, but can't do any work. The kernel dies
and the apache log has:

ERROR: key not found: HOME
in getindex at env.jl:25
in path at pkg/dir.jl:9
in find_in_path at loading.jl:18
in find_in_node1_path at loading.jl:29
in require at loading.jl:45
in include at boot.jl:238
in include_from_node1 at loading.jl:96
in include at boot.jl:238
in include_from_node1 at loading.jl:96
in process_options at client.jl:274
in _start at client.jl:349
at /home/julia/.julia/IJulia/src/IJulia.jl:10
at /home/julia/.julia/IJulia/src/kernel.jl:1
2013-08-27 15:56:20.336 [NotebookApp] KernelRestarter: restarting kernel
(1/5)
WARNING:root:kernel 4b541d33-c05a-45c0-90ce-031241c0a09d restarted

I guess I'm not quite sure how to set the HOME variable for the www-data
user so that julia recognizes it (I'm using Ubuntu + apache2 + wsgi +
ipydra)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1#issuecomment-23365959
.

[staticfloat]

Oh, wait, you're using uwsgi. Do you have a separate uwsgi master process
running? If so, the /etc/apache2/envvars approach won't work, because the
uwsgi process runs independently from apache, you'll have to modify HOME
for whatever user uwsgi is running as, (possibly www-data, possibly
something else, I'm not sure).

On Tue, Aug 27, 2013 at 4:04 PM, Elliot Saba [email protected] wrote:

Try editing /etc/apache2/envvars. You can put export HOME=<path> inside
of it, but ensure that whatever you set that to, www-data has permissions
to modify it.

HOME is typically set from /etc/passwd, (the second to last field, each
field separate by commas) and usually defaults to something like /var/www.
You could also change it from there, if you like, but the
/etc/apache2/envvars approach is more general for all environment variables.
-E

On Tue, Aug 27, 2013 at 4:00 PM, john verzani [email protected]:

Does anyone have a working solution here? I've been trying to get ipydra
to work and am way over my head. After adjusting some environment variables
through trial and error and copying a profile-julia to the right place, I
can get an IJulia notebook to open, but can't do any work. The kernel dies
and the apache log has:

ERROR: key not found: HOME
in getindex at env.jl:25
in path at pkg/dir.jl:9
in find_in_path at loading.jl:18
in find_in_node1_path at loading.jl:29
in require at loading.jl:45
in include at boot.jl:238
in include_from_node1 at loading.jl:96
in include at boot.jl:238
in include_from_node1 at loading.jl:96
in process_options at client.jl:274
in _start at client.jl:349
at /home/julia/.julia/IJulia/src/IJulia.jl:10
at /home/julia/.julia/IJulia/src/kernel.jl:1
2013-08-27 15:56:20.336 [NotebookApp] KernelRestarter: restarting kernel
(1/5)
WARNING:root:kernel 4b541d33-c05a-45c0-90ce-031241c0a09d restarted

I guess I'm not quite sure how to set the HOME variable for the www-data
user so that julia recognizes it (I'm using Ubuntu + apache2 + wsgi +
ipydra)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1#issuecomment-23365959
.

[jverzani]

That HOME trick worked, though now it is dying for some other reason that
doesn't show up in the apache logs. (I'm not sure what the u is in uwsgi,
I'm using mod-wsgi with apache.) Thanks for the help.

On Tue, Aug 27, 2013 at 4:06 PM, Elliot Saba [email protected]:

Oh, wait, you're using uwsgi. Do you have a separate uwsgi master process
running? If so, the /etc/apache2/envvars approach won't work, because the
uwsgi process runs independently from apache, you'll have to modify HOME
for whatever user uwsgi is running as, (possibly www-data, possibly
something else, I'm not sure).

On Tue, Aug 27, 2013 at 4:04 PM, Elliot Saba [email protected]
wrote:

Try editing /etc/apache2/envvars. You can put export HOME=<path>
inside
of it, but ensure that whatever you set that to, www-data has
permissions
to modify it.

HOME is typically set from /etc/passwd, (the second to last field, each
field separate by commas) and usually defaults to something like
/var/www.
You could also change it from there, if you like, but the
/etc/apache2/envvars approach is more general for all environment
variables.
-E

On Tue, Aug 27, 2013 at 4:00 PM, john verzani [email protected]:

Does anyone have a working solution here? I've been trying to get
ipydra
to work and am way over my head. After adjusting some environment
variables
through trial and error and copying a profile-julia to the right place,
I
can get an IJulia notebook to open, but can't do any work. The kernel
dies
and the apache log has:

ERROR: key not found: HOME
in getindex at env.jl:25
in path at pkg/dir.jl:9
in find_in_path at loading.jl:18
in find_in_node1_path at loading.jl:29
in require at loading.jl:45
in include at boot.jl:238
in include_from_node1 at loading.jl:96
in include at boot.jl:238
in include_from_node1 at loading.jl:96
in process_options at client.jl:274
in _start at client.jl:349
at /home/julia/.julia/IJulia/src/IJulia.jl:10
at /home/julia/.julia/IJulia/src/kernel.jl:1
2013-08-27 15:56:20.336 [NotebookApp] KernelRestarter: restarting
kernel
(1/5)
WARNING:root:kernel 4b541d33-c05a-45c0-90ce-031241c0a09d restarted

I guess I'm not quite sure how to set the HOME variable for the
www-data
user so that julia recognizes it (I'm using Ubuntu + apache2 + wsgi +
ipydra)

—
Reply to this email directly or view it on GitHub<
https://github.com/JuliaLang/IJulia.jl/issues/1#issuecomment-23365959>
.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1#issuecomment-23366383
.

John Verzani
Chair, Department of Mathematics
College of Staten Island, CUNY
[email protected]

[jverzani]

Does anyone have a working solution here? Despite the help above, I still couldn't get ipydra + apache to work to show IJulia notebooks. Is it just me? I was hoping to have something in place for about 25 students by the end of next week.

[ViralBShah]

@yarapavan Were you able to get the jquery terminal to work? It would be great if we can get something to work for use in classes.

[Keno]

I'm working on something. Stay tuned.

[marksale]

Total Newby question:
Trying to get IJulia running, got the latest Julia (0.2.0, Aug 26), installed Anaconda Python 1.7, 64 bit, checked add to path and make default Python), installed the Ijulia package (Pkg.add("IJulia"). Launched the Ipython (ipython notebook --profile julia
It started (in a brower) just fine, but I don't seem to have any Julia functionality, e.g., sin(3) gives an error
NameError: name 'sin' is not defined
seems like it isn't running Julia
Any idea what I'm doing wrong?
thanks
Mark ([email protected])

[stevengj]

You are running Python, not Julia. See the troubleshooting section of my Tutorial notes. In any case, this issue is not the place to ask questions about installation.

[ViralBShah]

Relevant: https://github.com/amitmurthy/JDock

[stevengj]

Seems to be closed by JuliaBox.