homeassistant: [WIP] init at 0.30.2

[peterhoeg]

Motivation for this change

Also includes a basic nixos module.

This is ridiculously painful due to how dependencies are handled (or I'm missing something completely).

The basic dependencies for the application itself are declared up front, but each sub-module has a REQUIREMENTS array where other python packages are listed. homeassistant will then try to load those dependencies if the module is activated and if failing to do so, will fall back to using pip to install them.

I honestly don't know what the best way to deal with this is. It's a rabbit hole of dependencies that reminds me of .rpm hell in the old RH days. Additionally, these sub modules will sometimes require specific versions which means multiple versions now need to be maintained and the maintenance cost just shoots up.

Any thoughts @FRidh?

Things done

• Tested using sandboxing
  (nix.useSandbox on NixOS,
  or option build-use-sandbox in nix.conf
  on non-NixOS)
• Built on platform(s)
  • NixOS
  • OS X
  • Linux
• Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Fits CONTRIBUTING.md.

---

[mention-bot]

@peterhoeg, thanks for your PR! By analyzing the history of the files in this pull request, we identified @edolstra, @joachifm and @offlinehacker to be potential reviewers.

[FRidh]

Some small changes should be made to the Python packages. I didn't check the module.

[FRidh]

this package is already in this file but unfortunately without the hyphen: pytestcov.

[FRidh]

and you've checked whether the tests run?

[FRidh]

An application that is build using Python shouldn't be here. This file is only for Python libraries.

[FRidh]

why do you specify outputs when you only have the default output? This can be removed.

[FRidh]

does that mean the package should only be available for Python >= 3.5? disabled = pythonOlder "3.5";

[FRidh]

why?

[FRidh]

remove

[FRidh]

remove

[FRidh]

I saw that zeroconf uses this. I've reported upstream that they should use environment markers instead in zeroconf
python-zeroconf/python-zeroconf#87

[FRidh]

if its an application and not a library then it shouldn't be in python-packages.nix. See below as well.

[schneefux]

I have worked with home-assistant before.

This is ridiculously painful due to how dependencies are handled (or I'm missing something completely).

requirements-all.txt includes all the packages you would possibly need. The dynamic pip installation is merely for convenience so you don't have to install packages you don't use.

[FRidh]

This is ridiculously painful due to how dependencies are handled (or I'm missing something completely).

Lots of dependencies -> lots of work. That's just how it is right now. Outside of Nixpkgs you could use pypi2nix.

[peterhoeg]

first of all, thank you @FRidh for all the feedback, much appreciated!

This is a WIP as I wasn't sure if my approach was correct, so I will get all the items cleaned up you mentioned. All tests will be enabled of course, I just wanted to see if I could get this thing up and running first.

But it offhand seems like there should be a better way. Coming from ruby where the combination of bundler and bundix makes it super easy, packaging a complex python app seems extremely work intensive.

[FRidh]

@peterhoeg you're welcome.

There should indeed be a better way, but unfortunately upstream doesn't provide any. Python packaging is quite a mess. People can put whatever code they want in setup.py. Therefore, the only way to find out what relations packages have, is to actually build them. This is the approach pypi2nix takes. But with extension types you need to override the results, and fixing test suites is still something you need to do yourself (if you're interested in the tests, that is).

As soon as Hydra allows network access during evaluation time, I will further work on #16005, which at least reduces the amount of work the packaging requires.

[FRidh]

(hmm, but that should actually work now since NixOS/nix@06bbfb6)

[peterhoeg]

So for the time being, what would you recommend for packaging home assistant?

The way I see it, we can do one of the following:

1. Package the bare minimum (ie the server without any modules and its requirements). Then let it download via PIP as it launches.

2. Item 1 + how ever many additional modules I can be bothered to manually write expressions for.

3. Use pypi2nix on the requirements_all.txt file and use it as an external source.

4. Wait for a more elegant solution.

2 seems a bit pointless as there will be modules that require PIP anyway and there is not much difference between having 70% and 97% of the packages dynamically installed.

3 requires additional work on the part of the person using home assistant which doesn't seem like a nice solution.

That really only leaves us with option 1.

[FRidh]

1. Package the bare minimum (ie the server without any modules and its requirements). Then let it download via PIP as it launches.

This won't work because it cannot write in the store. You would have to specify another directory to install the modules, however, the Nix Python won't be able to find those modules.

I suggest for now that you create a new repo in which you define the module, and where you create the expressions with pypi2nix. If one wants to use homeassistant, they can clone the repo, and import it. There are other modules like that, e.g. musnix. This saves you a lot of time, while keeping it fairly straightforward for others to use your module. Not everything needs to be in Nixpkgs.

[peterhoeg]

@FRidh assuming the home-assistant binary installs using --user, shouldn't both python modules in the store and local user modules be available?

In any case, thanks to @garbas, pypi2nix is now able to generate the expressions for home-assistant, so I will be sticking that in a separate repository and move the nixos bits there too in the same way as musnix.

Would you (the collective nix(os) maintainers be open to moving that repository under the main NixOS github organization if we handle changes similarly to nixpkgs (ie, only trusted project members can merge)? That should enable us to ship modules that cannot make it into the main repository but still be of the same high standard.

[FRidh]

@FRidh assuming the home-assistant binary installs using --user, shouldn't both python modules in the store and local user modules be available?

If you check sys.path of the interpreter, you'll see that it doesn't check any user site packages.

Would you (the collective nix(os) maintainers be open to moving that repository under the main NixOS github organization if we handle changes similarly to nixpkgs (ie, only trusted project members can merge)? That should enable us to ship modules that cannot make it into the main repository but still be of the same high standard.

I think at some point it might make sense to put such repo's together under one umbrella.

[peterhoeg]

For anyone else stumbling over this issue, @garbas has created a dedicated repos for python packages generated using pypi2nix which is where I'll be submitting the home-assistant stuff. https://github.com/garbas/nixpkgs-python

@FRidh, once again thank you for taking the time to explain things.