pip module installation

[joshiemoore]

Currently the only part of the build process that happens outside of buildroot is the installation of pip modules in post-build.sh. This script executes the compiled python3 binary and has it install the pip modules specified in config/pip_modules.txt. This works fine when building for x86_64, but will not work when trying to cross-build for Raspberry Pi or other systems.

I have scoured the net and not found a good, modular, and extensible way to install external pip modules in buildroot. I found some tools online that seemed like they would do what we're looking for, but I wasn't able to make any of them work. If anyone has any ideas for this, please share them or submit a PR.

I don't want anyone to have to manually write a bunch of configs to add another pip module to the build. They should basically be able to just add the desired module to the list, and it gets installed in the build (like the behavior now).

Once we have networking all set, we could have it automatically install a list of external pip modules on first boot or something like that (this would keep distro images small), but I think I would prefer to just install them at build-time.

[Admicos]

On Tue Jun 2, 2020 at 6:29 PM +03, Josh Moore wrote: I have scoured the net and not found a good, modular, and extensible way to install external pip modules in buildroot. I found some tools online that seemed like they would do what we're looking for, but I wasn't able to make any of them work. If anyone has any ideas for this, please share them or submit a PR.

It's actually really easy. See my PR (#30) for an example of pygame_gui. Buildroot also has a script at `utils/scanpypi` to automatically package most pip packages.

[Admicos]

Oh, I missed something:

I don't want anyone to have to manually write a bunch of configs to add another pip module to the build.

That's fair. But for the small amount of pip i dealt with in Buildroot (pygame_gui and pytts3x, with the latter i accidentally deleted and have to re-package), it isn't that hard. You fill in the URL of the sources, save the hash and Buildroot does the rest.

The scanpypi script does all that for you most of the time, but for stuff like pytts3x it couldn't find the sources for some reason.

[karx1]

We can also have users install their own packages into the SNAKEUSER partition if they want and then add that directory to sys.path in the startup script.

[hungl6844]

but can't you import os and run commands like pip install in the python shell? I can use os.popen to run iwctl station wlan0 connect <network> --passphrase <passphrase>

[hungl6844]

Okay, so buildroot includes a tool for this called scanpypi. It basically creates a buildroot package based on the pypi package you specify. It has to be manually checked for errors, but it makes our lives a hell of a lot easier. It's 'buildroot/utils/scanpypi'. It hasn't worked out well for me, but it may work for you! Check it out.
EDIT: @Admicos has already mentioned this multiple times, and the issue isn't closed yet, so I suppose we'll see.