Incorporating conda for package and env management

[echowhisky]

More of a proposal for an experimental branch than an "issue". And a place to discuss pros and cons.

Looking over the ipydeps code recently and the growing challenges in managing a set of install dependencies that combine apk and pip installed packages, it makes me wonder if using conda might be a better approach.

It looks like there have been some successes in getting conda and alpine to play nice lately. I was trying the show0k/alpine-minimal-notebook container last night and packages seem to load nice an cleanly. There were a couple of times where, when trying to run them I got errors because of the C libraries, but I don't think that container did the gcc inclusion the same way as this one, so combining some of the Dockerfile concepts might overcome that.

Introducing conda would be a bit of a shift.

• Instead of running containers as root, they could be run as a user account. I think this might be a good thing, but may also lead to some struggles
• operating out of a conda env instead of from the system install of python would make apk installs not work (or at least not be effective). Instead we'd have to make conda packages for things that needed a different installation capability (if they didn't exist already). This is not terribly different than having to add things to the dependencies.json file for ipydeps, except is a solution that has a larger community.
• !conda env update can take a yaml formatted input file to add additional packages to the current/root environment. I tested this last night having the first cell of a notebook use %%writefile magic to make a local file that specifies the packages to update. Then the second cell runs the update. It was pretty clean, and could be even cleaner if we created a %%conda magic wrapper.
  • that update file included installing things with pip instead of just from a conda package. And when it installs, whether via conda package or pip, the command !conda list shows everything installed and it's version (although the one installed by pip may just say "<pip>" and not show the version). Still, its there.
  • would essentially replace ipydeps
• would open the door for being able to have multiple python environments without package interference, and allow notebooks to self-create their own env. Not sure if this is a huge issue yet, but would certainly help some users, especially when testing specific package versions or combinations of installs. It might require the inclusion of the nb_conda plugin and figuring out a way to switch to a new kernel from inside a notebook. This should be do-able, though.
• other pro/cons?

Not sure if this is too much work for too little benefit, but is probably worth discussing further here or in Slack.