Support virtualenvs for python package building

[josegonzalez]

FPM currently requires separate packages for every piece of a given python package. Using the beaver repository, you would need to create the initial beaver package, as well as iterate over each line of the requirements file and create packages for those.

The above works great if you do not also distribute other fpm-based python packages on the server with conflicting requirements. If you do, then tough shit.

A better way would be to build a python virtualenv for the package. Thus we'd only have to install python-beaver to get everything up and running. This is contrary to the debian way -lawl - but is more immediately useful to those attempting to run services in production.

dh-virtualenv does something similar - though is python specific - and it doesn't create binaries for scripts like fpm does without need to learn more about debian packaging.

Issue #636 somewhat touches on this, though without any concrete outcome.

[jordansissel]

If you can, can you take a peek at the dh-virtualenv code and provide a list of the commands it runs in order to do its job?

[jordansissel]

(or anyone, for that matter, can do this) ;)

[josegonzalez]

Okay, so it seems straightforward, according to the bin. Basically, the debian rules file initializes the dh binary which then:

• Creating the virtualenv (with whatever options you enable)
• Install dependencies from the requirements.txt into that virtualenv
• Installs the package itself
• Optionally runs the tests which could be good for CI
• Fixes VIRTUAL_ENV path in activate script
• Fixes the shebangs within scripts to not point at global python with liberal use of grep and sed
• Fix any local symlinks. Not really sure what this is about, but there is linked commit that may explain

After that, it sort of goes back to debian. Seems pretty straightforward.

[brutasse]

FWIW we're currently packaging virtualenvs with fpm using the following recipe:

• create virtualenv
• install python packages
• delete .pyc and .pyo files
• update virtualenv location to /usr/share/python/<bundle name> with virtualenv-tools
• create deb with fpm -s dir

virtualenv-tools provides everything to update the location of a virtualenv so it'd be nice to rely on that rather than re-implement the functionality.

I'm not sure if this feature should be built into fpm directly or documented as a recipe. We do a bunch of extra stuff in the build process like creating init scripts, running django's collectstatic command, adding post-install and pre-remove files…

[brutasse]

@thedrow there's an example in a project of mine: https://github.com/brutasse/graphite-api/tree/master/fpm

[josegonzalez]

@thedrow you can make a PR that implements it.

[josegonzalez]

Thats what a PR is for. We can discuss implementation after the fact.

Would be nice to be able to specify a requirements.txt for the virtualenv, as well as config files to copy into the virtualenv.

[jordansissel]

@brutasse's script looks really straightforward: https://github.com/brutasse/graphite-api/blob/master/fpm/build-deb.sh

We can probably start with that as an fpm package type?

[josegonzalez]

This is what I did for circus, based on @brutasse's script. Some notes:

• Would be good to be able to replace lines 19, 25, and 26 with whatever manages deps within fpm
• A single virtualenv can and should be able to install multiple deps. I didn't use a requirements.txt file, but we should be able to support that.
• Remember to copy over any config files!

[jordansissel]

@josegonzalez As for interface, what would multiple deps look like? Specify on CLI or only do requirements.txt?

fpm -s virtualenv -t deb -n fancypants <what goes here>

dependencies? Directories to include in the package?G

[josegonzalez]

Shouldn't it be -t virtualenv ? I would figure we can keep an interface similar to dir types, and then you can specify multiple directories whose contents can be merged.

For instance, I might do:

fpm -t virtualenv -s folder-containing-etc -s folder-containing-var -n fancypants --requirements requirements.txt --normal-options-here

Requirements could be a single package or a path to a file that contains a list of reqs in requirements.txt format.

[jordansissel]

-t is for target, so the output of fpm. Is the goal to output a virtualenv named fancypants, in your example?

[josegonzalez]

My goal would be to make a debian package of a virtualenv. Ideally fpm manages the creation of that virtualenv - including any path modifications - so that I don't need to.

[jordansissel]

so then the source would be virtualenv, and target would be deb. And yes, I agree with your 'fpm manages' things - fpm would do all the hard work.

What inputs would you feed a virtualenv target? Dependencies? Directory to an app to package? Both?

[josegonzalez]

I think we can live with a dependency file as pip thinks of it. A single package could also work, though idk how difficult it would be to support both.

Ideally the virtualenv doesn't exist to begin with. I guess what I'm trying to do is a multi-stage fpm run:

• input is a dependency or set of dependencies
• transformed into a generic virtualenv
• virtualenv is then transformed into a debian package.

[jordansissel]

@josegonzalez Would you bundle an app along with its dependencies in a virtualenv? Or is it just using virtualenv to package a standalone python + some dependencies?

[josegonzalez]

What do you mean by "bundle an app"?

[jordansissel]

Let's say we write an app that uses django. It's sweet todo manager, because that's what all the cool webapp demos are these days.

You want to deploy this.

Do you:

• Deploy a packaged virtualenv that includes django and the todo manager code? (one .deb package)
• or, Deploy a packaged virtualenv that includes django but excludes the todo manager code, and deploy the todo manager code as a separate package? (two .deb packages, perhaps)

[brutasse]

Definitely a single package. The app + its dependencies, as a single, isolated virtualenv.

[josegonzalez]

1 Deb file.

[jordansissel]

woot, thanks for clarifying. I'll see what I can do to make this happen soon (assuming nobody else does). The scripting shown so far will make this very easy to implement.

[jordansissel]

<3

[neoice]

+1

I tried to package some Python webapps with virtualenvs in early 2013 and was not successful due to symlinks/PATHs within the virtualenvs. I'm revisiting the problem now and I'm bummed that it hasn't been resolved yet.

there's a number of issues on the FPM tracker that this feature would solve:

#262
#780
#636 (maybe)

I could also add to @thedrow's bounty (charity or BTC)

[evilsocket]

Pull request waiting :) #864

[jravetch]

+1 Been using dh-virtualenv as a substitute until fpm gets this feature.

[djha-skin]

If no one objects, I'll start on this feature based on the script found here --> https://github.com/brutasse/graphite-api/blob/master/fpm/build-deb.sh . It seems pretty straight forward; expect a pull request soon.

[djha-skin]

I have a nearly working prototype. I don't need to inherit from FPM::Package::Dir; if you look at FPM::Package::Tar's input function, it uses Dir without inheriting from it; my current solution does the same. You can take a look at what I have so far here, and in particular my first virtualenv commit.

Just take it with a grain of salt: It's a rough hewn stone at the moment. I copied from the python module so there are still errors. For example, I don't need to require the json module. I'll probably fix that tonight.

[djha-skin]

DOH facepalm I see now @evilsocket has a PR open and it's farther along than mine. For shame starting a new one 👎. Maybe I'll start from where he left off.

[evilsocket]

@djhaskin987 if u want to improve my stuff, that would be great :)

[djha-skin]

Thanks for your support, @evilsocket .

[djha-skin]

Submitted a PR! Many thanks again @evilsocket for your code and support. Comments and suggestions welcome! I don't think it's perfect, but I think it will cover most use cases and I think it's pretty good! 👍 Let me know if there are any deal breakers, everyone.

[jordansissel]

#930 is merged. Release coming soon.!

[evilsocket]

👯 👯 👯 👯

[rkiyanchuk]

Finally we have good solution for Python packaging! I think from Python developers point of view merge of #930 is the best that ever happened to fpm — it so much simplifies the packaging and solves the problems. Thank you! 👍 :)