Error when trying to run pihole container on UDM SE

[rpletka]

Describe the bug
When i follow the instructions for pihole I get the following error on step 5
root@UDMSE:/data/podman/cni# podman run -d
--network dns
--restart always
--name pihole
-e TZ="America/Chicago"
--cap-add=NET_ADMIN
-v "/data/etc-pihole/:/etc/pihole/"
-v "/data/pihole/etc-dnsmasq.d/:/etc/dnsmasq.d/"
--dns=127.0.0.1
--dns=1.1.1.1
--dns=8.8.8.8
--hostname pi.hole
-e VIRTUAL_HOST="pi.hole"
-e PROXY_LOCATION="pi.hole"
-e FTLCONF_REPLY_ADDR4="10.0.0.5"
-e IPv6="False"
pihole/pihole:latest
Error: OCI runtime error: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented

UDM Information

• Variant UniFi Dream Machine SE
• Firmware Version: 3.0.18
• Controller Version 7.3.83

[rpletka]

It seems that Ubiquiti has nerfed the ability to run containers via podman. The kernel seems to be compiled without the ability to leverage BPF syscalls.

root@UDMSE:/sys/fs/cgroup# zcat /proc/config.gz | grep -E 'CONFIG_BPF|CONFIG_FREEZER'
CONFIG_BPF=y

CONFIG_BPF_SYSCALL is not set

CONFIG_FREEZER=y

CONFIG_BPFILTER is not set

CONFIG_BPF_JIT=y

[boostchicken]

yeah I have the kernel source and we can make our own, someone was trying with @fabianishere.

@fabianishere any word?

[fabianishere]

@boostchicken Working on porting udm-kernel-tools to firmware v2.x/v3.x at the moment, but haven’t had much time to test the changes.

[ghost]

BPF SysCTL tweaks do not work on 2.4.X and 2.5.X for me either.

[peacey]

Well it's clear now that podman/docker won't work without a custom kernel that enables CONFIG_BPF_SYSCALL, but it's easy enough to create a container yourself without podman by using systemd-nspawn and install whatever you want within it (e.g. pihole). The process is a bit more involved, but it's pretty easy.

Basically, you just use debootstrap to create a directory under /data with a base debian system (this directory is your container), then you can boot and login to that container with systemd-nspawn or machinectl. After that, you can just install whatever you want in that container with apt-get or manually as any debian system (the container storage is persistent). This way your custom system changes don't affect the main Unifi system with regards to dependencies or conflicts with their files, and Unifi can't screw up your configuration with an update.

I've been running a debian unstable container on UDMP 3.x, with pihole, nginx, dnsdist, and some other programs. Works really well. You can even use pacstrap instead of debootstrap to create and run an Arch Linux container, or create a fedora, openSUSE, or other system container (examples).

Also, the debian base system is only like 300MB so the container size doesn't start off so high, but the container size can get to 1GB+ if you start installing a lot of large softwares (this applies to podman too, I guess).

Edit: Here's a guide I wrote on how to create and configure your own custom container on 3.x.

[boostchicken]

So I have not upgraded yet, but on 2.5x is overlayfs working, I expect it will work on 3 as well. So the disk space issue is now fixed @peacey

I am pretty sure I can get podman running on the UDMSE. See BPF is used for security lockdown on syscalls. We are always root anyways so it doesnt matter. I am going to drop seccomp and apparmor from the UDMP-UDMSE build and edit configs accordingly and I think that will get you in business.

Also, they new podman builds come with crun, if you want to use it. It's much better on resources. Uses less memory, and executes faster

Also, netavark is built and in the latest zips as well. This is a replacement for CNI which is now deprecated. It has cool things like macvlan dhcp working, but its not zero effort to migrate your networks, syntax of the files is quite different.

I would move to netavark ASAP. It makes containers much much much faster and much less latent on the network. It is written in Rust instead of Go (much like crun). I also included the dhcp-client-proxy if anyone wants to macvlan dhcp working.

I am not sure when I can move to 3. If someone would volunteer to test my new build for 3 build with the mods above I'll crank it out asap that would be awesome.

[boostchicken]

New builds are running that I think shoudl work on 3

seccomp.json removes permissions for bpf syscall, there is an update to the registries, cni is gone, replaced by netavark. Crun is also available if you want to.

If my IPFS server is being garbage use this link to get at the latest builds. Also you should use this link just to check if the builds are done. Even when they are done, it doesn't mean they are on IPFS. However, they all have versioned names now to make it clear which one you are getting.

https://github.com/unifi-utilities/unifios-utilities/actions

[peacey]

Sorry @boostchicken, I tried your new build and it still doesn't seem to work on 3.x. Still the same issue with BPF syscall with your new build and config files. I tried both crun and runc and also with and without --security-opt=seccomp=unconfined.

With runc I get,

Error: OCI runtime error: runc: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented

And with crun I get,

Error: crun: error opening file /proc/self/setgroups: No such file or directory: OCI runtime attempted to invoke a command that was not found

It seems /proc/self/setgroups is missing from the kernel interface on 3.x for processes.

So I don't think this will work unfortunately. Kernel is too locked down. But systemd-nspawn works well!

[sprocktech]

Just throwing this out there... There are other firewall vendors that embrace the community and actually give you instructions on how to run Pi-hole on their hardware. Why continue to try and fill the feature gaps that Ubiquiti continues to leave in their products when other products already have the features and they let you add your own stuff too? https://help.firewalla.com/hc/en-us/articles/360051625034-Guide-How-to-install-Pi-Hole-on-Gold-Purple-Beta-

[peacey]

@sprocktech I mean the whole point of a modding community is to mod the device. When someone comes here, they don't want to hear it can't be done. They want to hear how it can be done. For all of us that do like to use the UDM-series routers, it's nice to have this modding resource and community for all the extra bells and whistles.

And the UDMP/SE is a very capable, relatively low-cost router with a quad-core CPU! It would be a shame to let it go to waste. IMO, with its Debian architecture, it makes it very easy to mod for a Linux sysadmin.

[sprocktech]

@peacey I'm with you. I'm just saying that Ubiquiti is not with you. Actually, they are against you, as you know already from these recent firmware changes. Frankly, I've never seen anything like it before. It's an uphill battle my friend, and I'd rather put my money and effort elsewhere.

[pedropombeiro]

I can see the point behind Ubiquiti's decision. If they have a large percentage of customer support calls that end up being due to people having modded their UDM without a real understanding of what they're doing and how to solve problems that might arise, it is tempting to close the gates in order to keep the customer support costs (and quality) manageable.

It's still unfortunate, and I really love the fact that I don't have to run a Raspberry Pi on the side when the UDM can run the same software much more efficiently.

[xpherism]

@peacey Thanks for the tip regarding nspawn it actually seems good fit for the UDMP/SE. But I have some questions I hope you can answer :-)

In my case I've been testing Traefik using nspawn, at it works fine, either just using the traefik binary (you do need to have at least /usr/bin) defined in the rootfs otherwise nspawn won't start the "container" or using a complete rootfs (I tested using busybox:musl image).

But cannot seem figure out to make macvlan working with nspawn? I'm using the same network setup as I used with podman (using the found instruction on this site).

ip link set br5 promisc on
ip link add br5.mac link br5 type macvlan mode bridge
ip addr add 10.0.5.1/24 dev br5.mac noprefixroute
ip link set br5.mac promisc on
ip link set br5.mac up
ip route add 10.0.5.4/32 dev br5.mac

When I start the container with

$ systemd-nspawn --machine traefik --network-macvlan=br5 --directory /data/traefik/rootfs /bin/traefik

I need to figure out what to do inside the container to make things work?
nspawn adds a mv-br5 link inside the container, which I can assign an IP and bring up, then I can ping it from the host, but not from another client on the default network when testing. The container cannot ping addresses on the default network (network unreachable), or what else I have setup in the firewall (I haven't change any firewall, I worked as-is with podman on 2.x).

The information I can find on Internet suggest also using systemd inside container to configure container network (which I do not want to do).

I do not know what cni/ipam actually does when podman starts a container, but I'm guessing, it simply added necessary routes and assigns address to a virtual device, or am I missing something.

Any help is appreciated :-)

UPDATE!

If I add a default route inside the container it works, so all in all I do this inside the container

ip addr add 10.0.5.4/24 dev mv-br5
ip link set dev mv-br5 up
ip route add default via 10.0.5.1

Then I can reach the other networks just fine :-) It should be easy from there to setup nameservers.

[peacey]

Here's my guide on how to create and configure your own debian container on 3.x and install some services in it. It goes over how to isolate the container with a macvlan network too and gives an example on how to install pihole or adguard home in the container. Hopefully it helps those who are unfamiliar with systemd-nspawn.

Will eventually do a pull request to get this guide merged to this repository, but use that link for now.

[ghost]

@peacey I'm with you. I'm just saying that Ubiquiti is not with you. Actually, they are against you, as you know already from these recent firmware changes. Frankly, I've never seen anything like it before. It's an uphill battle my friend, and I'd rather put my money and effort elsewhere.

What is bad about recent UDM firmware changes?

[m-lan-at]

Sorry @boostchicken, I tried your new build and it still doesn't seem to work on 3.x. Still the same issue with BPF syscall with your new build and config files. I tried both crun and runc and also with and without --security-opt=seccomp=unconfined.

With runc I get,

Error: OCI runtime error: runc: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented

And with crun I get,

Error: crun: error opening file /proc/self/setgroups: No such file or directory: OCI runtime attempted to invoke a command that was not found

It seems /proc/self/setgroups is missing from the kernel interface on 3.x for processes.

So I don't think this will work unfortunately. Kernel is too locked down. But systemd-nspawn works well!

Same here with version 3.x it doesn't work, got the same errors with the newest builds...

[troian]

@peacey, was trying guide on UDM Pro with v2.5.17, and having this error thrown during install

root@Home:~# apt -y install systemd-container debootstrap
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 systemd-container : Depends: systemd (= 232-25+deb9u14) but 241-5~bpo9+1 is to be installed
                     Recommends: btrfs-progs but it is not going to be installed or
                                 btrfs-tools
                     Recommends: libnss-mymachines but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

edit: nvm, the guide is for Unifi os 3.x

[rloomans]

@peacey I've spun up pihole in an nspawn container using your guide, but UDP traffic to the host seems to be broken.

Inside the container DNS queries to the host time out:

root@pihole:~# dig github.com @192.168.10.1
;; communications error to 192.168.10.1#53: timed out
;; communications error to 192.168.10.1#53: timed out
;; communications error to 192.168.10.1#53: timed out

; <<>> DiG 9.18.13-1-Debian <<>> github.com @192.168.10.1
;; global options: +cmd
;; no servers could be reached

On the host, tcpdump complains about bad udp checksums:

root@udm:~# tcpdump -v -v -v -n -s200 -i br5 port 53 src 10.0.5.3 dst 192.168.10.1
tcpdump: can't parse filter expression: syntax error
root@udm:~# tcpdump -v -v -v -n -s200 -i br5 port 53 and src 10.0.5.3 and dst 192.168.10.1
tcpdump: listening on br5, link-type EN10MB (Ethernet), snapshot length 200 bytes
17:32:08.331622 IP (tos 0x0, ttl 64, id 39468, offset 0, flags [none], proto UDP (17), length 79)
    10.0.5.3.49485 > 192.168.10.1.53: [bad udp cksum 0xd9f8 -> 0x96a4!] 5668+ [1au] A? github.com. ar: . OPT UDPsize=1232 [COOKIE d360975bf0f637e1] (51)
17:32:13.332867 IP (tos 0x0, ttl 64, id 39511, offset 0, flags [none], proto UDP (17), length 79)
    10.0.5.3.44618 > 192.168.10.1.53: [bad udp cksum 0xd9f8 -> 0xa9a7!] 5668+ [1au] A? github.com. ar: . OPT UDPsize=1232 [COOKIE d360975bf0f637e1] (51)
17:32:18.336152 IP (tos 0x0, ttl 64, id 39680, offset 0, flags [none], proto UDP (17), length 79)
    10.0.5.3.43879 > 192.168.10.1.53: [bad udp cksum 0xd9f8 -> 0xac8a!] 5668+ [1au] A? github.com. ar: . OPT UDPsize=1232 [COOKIE d360975bf0f637e1] (51)
17:32:59.154746 IP (tos 0x0, ttl 64, id 42249, offset 0, flags [DF], proto UDP (17), length 74)
    10.0.5.3.55013 > 192.168.10.1.53: [bad udp cksum 0xd9f3 -> 0x1d32!] 37387+ [1au] A? pixelmon.zots.org. ar: . OPT UDPsize=1232 DO (46)
17:32:59.155002 IP (tos 0x0, ttl 64, id 42250, offset 0, flags [DF], proto UDP (17), length 74)
    10.0.5.3.33132 > 192.168.10.1.53: [bad udp cksum 0xd9f3 -> 0x4ab7!] 40703+ [1au] AAAA? pixelmon.zots.org. ar: . OPT UDPsize=1232 DO (46)
17:32:59.156003 IP (tos 0x0, ttl 64, id 42251, offset 0, flags [DF], proto UDP (17), length 77)
    10.0.5.3.36318 > 192.168.10.1.53: [bad udp cksum 0xd9f6 -> 0x5f27!] 26633+ [1au] AAAA? sammiecraft.zots.org. ar: . OPT UDPsize=1232 DO (49)
17:32:59.156208 IP (tos 0x0, ttl 64, id 42252, offset 0, flags [DF], proto UDP (17), length 77)
    10.0.5.3.55489 > 192.168.10.1.53: [bad udp cksum 0xd9f6 -> 0x823d!] 64042+ [1au] A? sammiecraft.zots.org. ar: . OPT UDPsize=1232 DO (49)

... and TCP is no better...

root@udm:~# tcpdump -v -v -v -n -s200 -i br5 port 53 and src 10.0.5.3 and dst 192.168.10.1
tcpdump: listening on br5, link-type EN10MB (Ethernet), snapshot length 200 bytes
17:34:21.687258 IP (tos 0x0, ttl 64, id 44028, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.5.3.37805 > 192.168.10.1.53: Flags [S], cksum 0xd9da (incorrect -> 0xca2d), seq 3907592455, win 42700, options [mss 1220,sackOK,TS val 1528431959 ecr 0,nop,wscale 12], length 0
17:34:21.687379 IP (tos 0x0, ttl 64, id 44029, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.5.3.37805 > 192.168.10.1.53: Flags [.], cksum 0xd9d2 (incorrect -> 0x8a78), seq 3907592456, ack 848062359, win 11, options [nop,nop,TS val 1528431959 ecr 2494424438], length 0
17:34:21.687487 IP (tos 0x0, ttl 64, id 44030, offset 0, flags [DF], proto TCP (6), length 105)
    10.0.5.3.37805 > 192.168.10.1.53: Flags [P.], cksum 0xda07 (incorrect -> 0x6e82), seq 0:53, ack 1, win 11, options [nop,nop,TS val 1528431959 ecr 2494424438], length 53 40345+ [1au] A? github.com. ar: . OPT UDPsize=1232 [COOKIE 2d9e1fac18f5991b] (51)
17:34:21.688531 IP (tos 0x0, ttl 64, id 58575, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.5.3.37121 > 192.168.10.1.53: Flags [S], cksum 0xd9da (incorrect -> 0x50f3), seq 1328690852, win 42700, options [mss 1220,sackOK,TS val 1528431960 ecr 0,nop,wscale 12], length 0
17:34:21.688625 IP (tos 0x0, ttl 64, id 58576, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.5.3.37121 > 192.168.10.1.53: Flags [.], cksum 0xd9d2 (incorrect -> 0x1d6b), seq 1328690853, ack 1506817061, win 11, options [nop,nop,TS val 1528431960 ecr 2494424439], length 0
17:34:21.688805 IP (tos 0x0, ttl 64, id 58577, offset 0, flags [DF], proto TCP (6), length 105)
    10.0.5.3.37121 > 192.168.10.1.53: Flags [P.], cksum 0xda07 (incorrect -> 0xf78a), seq 0:53, ack 1, win 11, options [nop,nop,TS val 1528431960 ecr 2494424439], length 53 42883+ [1au] A? github.com. ar: . OPT UDPsize=1232 [COOKIE 2d9e1fac18f5991b] (51)
17:34:21.689652 IP (tos 0x0, ttl 64, id 59204, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.5.3.41519 > 192.168.10.1.53: Flags [S], cksum 0xd9da (incorrect -> 0x45f0), seq 3473060007, win 42700, options [mss 1220,sackOK,TS val 1528431961 ecr 0,nop,wscale 12], length 0
17:34:21.689743 IP (tos 0x0, ttl 64, id 59205, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.5.3.41519 > 192.168.10.1.53: Flags [.], cksum 0xd9d2 (incorrect -> 0x8eda), seq 3473060008, ack 1346486592, win 11, options [nop,nop,TS val 1528431961 ecr 2494424440], length 0
17:34:21.689860 IP (tos 0x0, ttl 64, id 59206, offset 0, flags [DF], proto TCP (6), length 105)
    10.0.5.3.41519 > 192.168.10.1.53: Flags [P.], cksum 0xda07 (incorrect -> 0xd392), seq 0:53, ack 1, win 11, options [nop,nop,TS val 1528431961 ecr 2494424440], length 53 15595+ [1au] A? github.com. ar: . OPT UDPsize=1232 [COOKIE 2d9e1fac18f5991b] (51)

Any ideas?

[mtjoseph]

I'm having the same issue with DNS queries to host. Has this been resolved yet?

[joshuisken]

When trying local and reverse lookups in pi.hole (10.0.5.3) I have the same problem. When I disable "Use Conditional Forwarding" the timeout problem disappeared with the disadvantage that local lookups fail.
I tried to set an additional "Upstream DNS Servers" with my UDM-Pro (192.168.1.1) as workaround (the timeout problem does not show up) but that also did not fix my local lookups.

[gatesry]

@rloomans - The checksums won't match as these NICs use network offloading. You can check using ethtool --show-offload brXX

Edit: On 3.1.9, I do not have these DNS issues as well.

[peacey]

Everyone, if you're having issues with pihole via nspawn, please open your own separate issue and detail the problem. Trying to follow multiple conversations here makes things very confusing.

FYI, on 3.1.9, the nspawn approach works fine for me. And I don't have issues with DNS resolutions.

[emlynmac]

I too have successfully got the nspawn pi-hole working.
The key I had to ensure that the dns plan in unifi was a superset of the routable networks.
So, for the nspawn, it was configured with 10.0.5.3 and gw of 10.0.5.1.
The unifi DNS VLAN was set up as 10.0.0.0/16 with a udmpro GW of 10.0.6.1
I've had no routing issues with this setup, whereas with using the 10.0.5.1 on the UDM caused problems

[rloomans]

As it turns out, the solution appears to be to get dnsmasq to listen on the extra network interface: #550

…

On Fri, 26 May 2023 at 03:37, Ryan Gates ***@***.***> wrote: @rloomans <https://github.com/rloomans> - The checksums won't match as these NICs use network offloading. You can check using ethtool --show-offload brXX — Reply to this email directly, view it on GitHub <#510 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAWPZLPET2LU5XKCEDFVRLXH6KFVANCNFSM6AAAAAAVPHHHNU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[dvcrn]

@boostchicken sorry to jump in here but how did you get overlayfs working on 2.5?

So I have not upgraded yet, but on 2.5x is overlayfs working, I expect it will work on 3 as well. So the disk space issue is now fixed @peacey