nixos/waydroid: Init module Waydroid

[CajuM]

Motivation for this change

Add an easy way to run waydroid on NixOS.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• 21.11 Release Notes (or backporting 21.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

Things not done

• waydroid-net.sh should be replaced with a dummy script or patched-out of Waydroid. Waydroid calls it to set-up container networking and it might be preferable to do so using NixOS modules.
• Also install a Waydroid container image. Right now the responsibility falls on the user to call waydroid init.
• It could be useful to pass a custom init script to the container, like the anbox module does.
• Currently a custom kernel or xanmod is required to run Waydroid due to missing kernel modules.

[Congee]

@samueldr ping 😉

[Mindavi]

Nice! I don't know much about systemd, but I'd think you want to tighten the unit up a little bit with some sandboxing/security features? There should be examples in other modules, I think.

Never got around to changing my kernel to the right kernel for testing waydroid out.

[deinferno]

Networking in waydroid is broken if firewall enabled in nixos, adding interface to trusted list seems to fix it.

networking.firewall.trustedInterfaces = [ "waydroid0" ];

Also there is missing getent package in PATH of waydroid-net.sh that breaks networking too.

[CajuM]

Nice! I don't know much about systemd, but I'd think you want to tighten the unit up a little bit with some sandboxing/security features? There should be examples in other modules, I think.

Never got around to changing my kernel to the right kernel for testing waydroid out.

systemd can make use of cgroups to limit resource usage or it can be configured to run a service as a different user. I'm not sure if waydroid can be run as a user other than root, specifically setting-up waydroid0 and loading kernel modules when necessary.

[Mindavi]

(probably related to waydroid packaging).

I did this:

• rebuild system with this module and xanmod kernel (5.14.9) (nixos-rebuild -I nixpkgs=https://github.com/cajum/nixpkgs/archive/waydroid-modules.tar.gz boot)
• sudo waydroid init (downloads lineageos from the internet)
• sudo systemctl start waydroid-container.service
• sudo waydroid log

In the log I see this:

waydroid log excerpt

modprobe: FATAL: Module binder_linux not found in directory /run/current-system/kernel-modules/lib/modules/5.14.7
(3168731) [23:12:55] Failed to load binder driver
(3168731) [23:12:55] modprobe: FATAL: Module binder_linux not found in directory /run/current-system/kernel-modules/lib/modules/5.14.7
(3168731) [23:12:55] ERROR: Binder node "binder" for waydroid not found
(3168731) [23:12:55] See also: <https://github.com/waydroid>
(3168731) [23:12:55] Traceback (most recent call last):
  File "/nix/store/61w69pjkqkciq2pn56sbmq199rmhs14c-waydroid-1.1.1/lib/python3.9/site-packages/tools/__init__.py", line 49, in main
    actions.init(args)
  File "/nix/store/61w69pjkqkciq2pn56sbmq199rmhs14c-waydroid-1.1.1/lib/python3.9/site-packages/tools/actions/initializer.py", line 85, in init
    setup_config(args)
  File "/nix/store/61w69pjkqkciq2pn56sbmq199rmhs14c-waydroid-1.1.1/lib/python3.9/site-packages/tools/actions/initializer.py", line 77, in setup_config
    helpers.drivers.setupBinderNodes(args)
  File "/nix/store/61w69pjkqkciq2pn56sbmq199rmhs14c-waydroid-1.1.1/lib/python3.9/site-packages/tools/helpers/drivers.py", line 129, in setupBinderNodes
    raise OSError('Binder node "binder" for waydroid not found')
OSError: Binder node "binder" for waydroid not found

It's unclear to me why it's trying to look in the folder 5.14.7, since I only have a folder 5.14.9-xanmod1-cacule.

Modprobing the module myself seems to work fine. Which makes it even weirder, since I would assume that the same modprobe is used on the same kernel...?

I need to read more documentation to get it up and running further, I guess.

[CajuM]

(probably related to waydroid packaging).

I did this:

* rebuild system with this module and xanmod kernel (5.14.9) (`nixos-rebuild -I nixpkgs=https://github.com/cajum/nixpkgs/archive/waydroid-modules.tar.gz boot`)

* sudo waydroid init (downloads lineageos from the internet)

* sudo systemctl start waydroid-container.service

* sudo waydroid log

In the log I see this:
waydroid log excerpt

It's unclear to me why it's trying to look in the folder 5.14.7, since I only have a folder 5.14.9-xanmod1-cacule.

Modprobing the module myself seems to work fine. Which makes it even weirder, since I would assume that the same modprobe is used on the same kernel...?

I need to read more documentation to get it up and running further, I guess.

Did you reboot into the Xanmod configuration?
It looks like Xanmod isn't necessary, your kernel might already have ashmem and binder builtin. I assumed that if it's not in pkgs/os-specific/linux/kernel/*.nix it would not be shipped... looks like we already had the proper modules and I didn't know about it :)

[Mindavi]

Yes, I did reboot into that configuration (I guess I'd have other folders there otherwise anyway). Somehow it seems like some kernel info might've leaked into the build output, not sure...

[KenMacD]

As a nice-to-have, pyclip seems required for clipboard support.

[Congee]

So, nobody else is gonna review this PR? :(

[mkg20001]

Did some formatting nits. Should be good now.

[dit7ya]

I tried this module with virtualisation.waydroid.enable = true; but this is what I get when I try to do sudo waydroid init

[15:12:23] Failed to load binder driver
[15:12:23] modprobe: FATAL: Module binder_linux not found in directory /run/current-system/kernel-modules/lib/modules/5.15.0
[15:12:23] ERROR: Binder node "binder" for waydroid not found
[15:12:23] See also: <https://github.com/waydroid>
Run 'waydroid log' for details.

And this is the result of sudo systemctl status waydroid-container.service

○ waydroid-container.service - Waydroid Container
     Loaded: loaded (/etc/systemd/system/waydroid-container.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
  Condition: start condition failed at Sun 2021-11-07 15:12:13 IST; 1min 55s ago
             └─ ConditionPathExists=/var/lib/waydroid/lxc/waydroid was not met

Nov 07 15:07:18 nixos systemd[1]: Condition check resulted in Waydroid Container being skipped.
Nov 07 15:12:13 nixos systemd[1]: Condition check resulted in Waydroid Container being skipped.

[CajuM]

I tried this module with virtualisation.waydroid.enable = true; but this is what I get when I try to do sudo waydroid init

[15:12:23] Failed to load binder driver
[15:12:23] modprobe: FATAL: Module binder_linux not found in directory /run/current-system/kernel-modules/lib/modules/5.15.0
[15:12:23] ERROR: Binder node "binder" for waydroid not found
[15:12:23] See also: <https://github.com/waydroid>
Run 'waydroid log' for details.

And this is the result of sudo systemctl status waydroid-container.service

○ waydroid-container.service - Waydroid Container
     Loaded: loaded (/etc/systemd/system/waydroid-container.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
  Condition: start condition failed at Sun 2021-11-07 15:12:13 IST; 1min 55s ago
             └─ ConditionPathExists=/var/lib/waydroid/lxc/waydroid was not met

Nov 07 15:07:18 nixos systemd[1]: Condition check resulted in Waydroid Container being skipped.
Nov 07 15:12:13 nixos systemd[1]: Condition check resulted in Waydroid Container being skipped.

The first error is caused by missing kernel modules, you can either install xanmod({ boot.kernelPackages = pkgs.linuxPackages_xanmod; } in configuration.nix) or build your own. The second is caused by waydroid init having failed previously.

[mkg20001]

For the kernel: Here is a PR that will enable binder,ashmem on the default NixOS Kernel so waydroid works out of the box #145768

[maralorn]

Hey there. I am also having trouble running this. I can seemingly reach the point where I run waydroid session start and then the log shows:

(852824) [04:22:42] Failed to add service waydroidhardware: -1
(875088) [04:22:43] Failed to add service waydroidusermonitor: -1
(875088) [04:22:43] Failed to add service waydroidclipboard: -1

Not really sure what’s going on there.

By the way, since waydroid session start is not supposed to be run as root I needed to give my user write permission for /var/lib/waydroid and /var/lib/waydroid/waydroid.log. Is that maybe something that could be made easier in the module?

[maralorn]

I had a look at the logcat output: Apparently the android is stuck in a boot loop. And there are tons of error messages concerning none existing files or directories under /data/user. So my guess is this a permission/fs problem. That does not surprise me a lot, seeing that now the data in /var/lib/waydroid is partly owned by root and partly by my user. But this is all quite foreign to me, and I am not sure how to fix it.

[CajuM]

There is no need to change the ownership of /var/lib/waydroid, if you suspect that it has been corrupted, you can stop the waydroid-container.service, delete /var/lib/waydroid and re-run waydroid init. If it is still an issue, consider filling a bug on nixpkgs with the following details: the nixpkgs version you are using, your configuration.nix, your GPU configuration and check you are running a Wayland session.

[andOrlando]

Hi, I'm having some pretty indecipherable issues. I have binder and ashmem, I've run waydroid init and I'm using sway, but when I run waydroid --details-to-stdout -v session start I just get this and literally nothing else.

(008822) [19:05:11] Save session config: /var/lib/waydroid/session.cfg
(008822) [19:05:11] UserMonitor service is not even started
(008822) [19:05:11] Clipboard service is not even started

and when I hit ctrl-c it just repeats itself

^C(008822) [19:05:37] UserMonitor service is not even started
(008822) [19:05:37] Clipboard service is not even started

running sudo waydroid --details-to-stdout container start prints

(020744) [19:35:39] % chmod 666 -R /dev/binder
(020744) [19:35:39] % chmod 666 -R /dev/vndbinder
(020744) [19:35:39] % chmod 666 -R /dev/hwbinder
(020744) [19:35:39] Container manager is waiting for session to load

and then it proceeds to wait forever for /var/lib/waydroid/session.cfg to exist, which it does not. (when terminating this, it tells me that lib/python3.9/site-packages/tools/actions/container_manager.py stops on line 113, which is just a time.sleep(1) where it checks for the existance of /var/lib/waydroid/session.cfg)

There are no log outputs on dmesg either

What could this possibly be? it's really not giving me much.

[CajuM]

Hello, your error sounds like you're not running waydroid container start and waydroid session start at the same time?

[andOrlando]

Hello, your error sounds like you're not running waydroid container start and waydroid session start at the same time?

You're totally right, my bad. Thanks for pointing this out...
It's still broken for me, but now at least it says something. I'll continue debugging and hopefully not have to bug you guys further...