Support for UDM SE #82

jeroenops · 2022-10-11T15:04:20Z

Hi,

I wanted to install a custom kernel for the UDMSE (version 2.5.11), however, it seems to fail:

root@FW:~# uname -a
Linux FW 4.19.152-ui-alpine #4.19.152 SMP Mon Aug 1 14:24:56 CST 2022 aarch64 GNU/Linux
root@FW:~#  cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@FW:~# udm-bootctl list
Version                                 	Initrd
4.19.152-edge4                          	yes
root@FW:~# udm-bootctl boot 4.19.152-edge4
/usr/bin/udm-bootctl: line 193: ssh-proxy: command not found

I assume this is due to the new 'major' UniFi-OS version, as the UDM and UDMP are at 1.12.30, while the UDMSE and UDR are at 2.5.11. However, I was unable to find any documentation on this - is there any?

Thank you!

The text was updated successfully, but these errors were encountered:

fabianishere · 2022-10-11T17:32:42Z

You are correct, the UDM SE and UDR are currently unsupported since they were released after this project was developed.

Porting this project to those devices requires building the kexec-mod project for the kernels running on these devices. However, I have no way of testing this since I don’t own a UDM SE.

Once the UDM (Pro) is on 2.x, I will start porting this project to the new firmware.

gtrabanco · 2022-10-16T12:28:31Z

If you only want wireguard support this project works great on UDM-SE:

https://github.com/tusc/wireguard-kmod

simkin · 2022-12-15T09:42:33Z

Hi @fabianishere wondering if this is still on your radar? :) (We now have moved to Unifi OS 3.x)

fabianishere · 2022-12-17T13:00:10Z

This is still on my ToDo list, but I don't have access to a UDM Pro at the moment. I'll probably start working on this start of next year.

ppkhoa · 2022-12-20T09:34:14Z

UDM/Pro is being updated to the same 3.x branch with UDM Pro SE 3.x, migration is already started for early access so this issue will probably apply to UDM/Pro soon once they go GA

foxcris · 2023-01-21T08:26:39Z

@fabianishere: I have got an udm se and am interested to help getting this project running. Could you point out what we have to achive to get it running? Some advise how to start?

fabianishere · 2023-01-21T20:06:36Z

@fabianishere: I have got an udm se and am interested to help getting this project running. Could you point out what we have to achive to get it running? Some advise how to start?

You'll have to retrieve the kernel sources from Ubiquiti and verify whether you can build kexec-mod for the kernel.

Then it is a matter of updating the udm-bootctl command to not make use of ssh-proxy, but invoke the commands on the system directly.

foxcris · 2023-01-22T12:04:19Z

Is there an specific adress at unifi to get the sources? Who did you turn to to get the sources? Am 21.01.2023 um 21:06 schrieb Fabian Mastenbroek:

…

@fabianishere <https://github.com/fabianishere>: I have got an udm se and am interested to help getting this project running. Could you point out what we have to achive to get it running? Some advise how to start? You'll have to retrieve the kernel sources from Ubiquiti and verify whether you can build kexec-mod <https://github.com/fabianishere/kexec-mod> for the kernel. Then it is a matter of updating the udm-bootctl <https://github.com/fabianishere/udm-kernel-tools/blob/master/udm-bootctl> command to not make use of |ssh-proxy|, but invoke the commands on the system directly. — Reply to this email directly, view it on GitHub <#82 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACYGU3HAXJI6U6Y7RVSLELWTQ6VPANCNFSM6AAAAAARCMKEGA>. You are receiving this because you commented.Message ID: ***@***.***>

fabianishere · 2023-01-22T18:41:05Z

You can mail Ubiquiti at [email protected] to request the GPL sources for the UDM SE.

foxcris · 2023-01-23T16:37:14Z

I have got the kernel sources and will take a look wether i can build kexec-mod.

foxcris · 2023-01-23T16:48:28Z

ok i get the following error:

make KDIR=../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine
make -C ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine M=/mnt/d/github/kexec-mod/kernel modules
make[1]: Entering directory '/mnt/d/github/kexec-mod/UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine'
mkdir -p /mnt/d/github/kexec-mod/kernel/orig
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec.c /mnt/d/github/kexec-mod/kernel/orig/kexec.c
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec_core.c /mnt/d/github/kexec-mod/kernel/orig/kexec_core.c
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec_internal.h /mnt/d/github/kexec-mod/kernel/orig/kexec_internal.h
scripts/Makefile.build:45: /mnt/d/github/kexec-mod/kernel/arch/x86/Makefile: No such file or directory
make[3]: *** No rule to make target '/mnt/d/github/kexec-mod/kernel/arch/x86/Makefile'.  Stop.
make[2]: *** [scripts/Makefile.build:544: /mnt/d/github/kexec-mod/kernel/arch/x86] Error 2
make[1]: *** [Makefile:1550: _module_/mnt/d/github/kexec-mod/kernel] Error 2
make[1]: Leaving directory '/mnt/d/github/kexec-mod/UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine'
make: *** [Makefile:15: module] Error 2

Did you build kexec-mod on the udm? I tried using the windows subsystem for linux. I will try using a plain linux later.

fabianishere · 2023-01-23T17:34:56Z

ok i get the following error:

make KDIR=../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine
make -C ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine M=/mnt/d/github/kexec-mod/kernel modules
make[1]: Entering directory '/mnt/d/github/kexec-mod/UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine'
mkdir -p /mnt/d/github/kexec-mod/kernel/orig
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec.c /mnt/d/github/kexec-mod/kernel/orig/kexec.c
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec_core.c /mnt/d/github/kexec-mod/kernel/orig/kexec_core.c
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec_internal.h /mnt/d/github/kexec-mod/kernel/orig/kexec_internal.h
scripts/Makefile.build:45: /mnt/d/github/kexec-mod/kernel/arch/x86/Makefile: No such file or directory
make[3]: *** No rule to make target '/mnt/d/github/kexec-mod/kernel/arch/x86/Makefile'.  Stop.
make[2]: *** [scripts/Makefile.build:544: /mnt/d/github/kexec-mod/kernel/arch/x86] Error 2
make[1]: *** [Makefile:1550: _module_/mnt/d/github/kexec-mod/kernel] Error 2
make[1]: Leaving directory '/mnt/d/github/kexec-mod/UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine'
make: *** [Makefile:15: module] Error 2

Did you build kexec-mod on the udm? I tried using the windows subsystem for linux. I will try using a plain linux later.

You need to cross-compile to ARM64, by setting ARCH=arm64 and CROSS_COMPILE=aarch64-linux-gnu- (make sure you have gcc-aarch64-linux-gnu installed)

boostchicken · 2023-01-24T10:37:23Z

I cam here to share I got sources but I have been defeated!

Well done! Its quite telling the UDMSE sources came quick but UDMPRO, nada!

foxcris · 2023-01-24T19:30:45Z

With cross compile i get a different error. Any hint for this?

export ARCH=arm64; export CROSS_COMPILE=aarch64-linux-gnu-; make KDIR=../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine
make -C ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine M=/mnt/d/github/kexec-mod/kernel modules
make[1]: Entering directory '/mnt/d/github/kexec-mod/UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine'
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec.c /mnt/d/github/kexec-mod/kernel/orig/kexec.c
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec_core.c /mnt/d/github/kexec-mod/kernel/orig/kexec_core.c
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec_internal.h /mnt/d/github/kexec-mod/kernel/orig/kexec_internal.h
mkdir -p /mnt/d/github/kexec-mod/kernel/arch/arm64/orig
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/arch/arm64/kernel/machine_kexec.c /mnt/d/github/kexec-mod/kernel/arch/arm64/orig/machine_kexec.c
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/arch/arm64/kernel/relocate_kernel.S /mnt/d/github/kexec-mod/kernel/arch/arm64/orig/relocate_kernel.S
  CC [M]  /mnt/d/github/kexec-mod/kernel/arch/arm64/machine_kexec_drv.o
  CC [M]  /mnt/d/github/kexec-mod/kernel/arch/arm64/machine_kexec_compat.o
  CC [M]  /mnt/d/github/kexec-mod/kernel/arch/arm64/idmap.o
  AS [M]  /mnt/d/github/kexec-mod/kernel/arch/arm64/cpu-reset.o
  AS [M]  /mnt/d/github/kexec-mod/kernel/arch/arm64/hyp-shim.o
  CC [M]  /mnt/d/github/kexec-mod/kernel/arch/arm64/machine_kexec.o
/mnt/d/github/kexec-mod/kernel/arch/arm64/machine_kexec.c:2:10: fatal error: orig/machine_kexec.c: No such file or directory
    2 | #include "orig/machine_kexec.c"
      |          ^~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[3]: *** [scripts/Makefile.build:304: /mnt/d/github/kexec-mod/kernel/arch/arm64/machine_kexec.o] Error 1
make[2]: *** [scripts/Makefile.build:544: /mnt/d/github/kexec-mod/kernel/arch/arm64] Error 2
make[1]: *** [Makefile:1550: _module_/mnt/d/github/kexec-mod/kernel] Error 2
make[1]: Leaving directory '/mnt/d/github/kexec-mod/UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine'
make: *** [Makefile:15: module] Error 2

foxcris · 2023-01-24T19:44:01Z

Ok got it working by using the absolut path for KDIR not a relative one. Build finished. Make of the User-space helper was also successfull. What are the next steps?

boostchicken · 2023-01-24T23:33:56Z

Build a custom kerneland make sure to add the configs for BPF needed for podman. Then you can follow the docs and boot into that kernel.

foxcris · 2023-01-28T15:55:16Z

Ok i am just building the kernel. How can i test it manually? Just by coping the kernel, kexex-module and redir.so to the udm and then run LD_PRELOAD=/root/redir.so kexec -l /path/to/custom/kernel --reuse-cmdline?

jimmy-ungerman · 2023-02-12T19:20:43Z

@foxcris just wanted to reach out and see if you were able to successfully boot into this kernel on your SE and run containers?

foxcris · 2023-02-19T09:59:21Z

Today i had some time to go on with kexec an a custom kernel. Currently i still didn't manage to boot a custom kernel. My problem is that i cannot load the kexec_mod module. First isse was the wrong version magic

kexec_mod: version magic '4.19.152+ SMP mod_unload aarch64' should be '4.19.152-ui-alpine SMP mod_unload aarch64'

I figured out how to fix this by setting the EXTRAVERSION. However it took me a while to "remove" the "+" sign. Wasn't really obvious and caused by git (you have to set LOCALVERSION manually during build).
Now the version magic missmatch is gone but i get different errors when i try to load the module with modprobe.

kexec_mod: Unknown symbol machine_kexec_prepare (err -2)
kexec_mod: Unknown symbol machine_crash_shutdown (err -2)
kexec_mod: Unknown symbol machine_kexec (err -2)
kexec_mod: Unknown symbol machine_kexec_cleanup (err -2)

@fabianishere: Any hints how to go on now? As soon as i can manually boot the custom kernel i will try to build/adapt your packages.

foxcris · 2023-02-19T10:10:53Z

Oh i just missed to load "kexec_mod_arm64.ko" first.
Kexec_mod is now loaded. I will try to boot a custom kernel as a next step.

foxcris · 2023-02-19T10:16:03Z

Kexec is still compaining about miss /proc/kcore.

root@UDM-SE:/ssd1/custom_data# LD_PRELOAD=/ssd1/custom_data/user/redir.so kexec -l /ssd1/custom_data/UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine/vmlinux --reuse-cmdline
ERROR: ld.so: object '/ssd1/custom_data/user/redir.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
Can't open (/proc/kcore).
Can't open (/proc/kcore).
kexec_load failed: Function not implemented
entry       = 0xcac690 flags = 0xb70000
nr_segments = 4
segment[0].buf   = 0x7f82b5a010
segment[0].bufsz = 0x8415f0
segment[0].mem   = 0x80000
segment[0].memsz = 0x842000
segment[1].buf   = 0x7f833aa010
segment[1].bufsz = 0x380200
segment[1].mem   = 0x8d0000
segment[1].memsz = 0x3cf000
segment[2].buf   = 0x556552b060
segment[2].bufsz = 0x68e7
segment[2].mem   = 0xca5000
segment[2].memsz = 0x7000
segment[3].buf   = 0x55655327a0
segment[3].bufsz = 0x3460
segment[3].mem   = 0xcac000
segment[3].memsz = 0x4000

foxcris · 2023-02-20T15:01:12Z

Ok found the error. I used the wrong path to redir.so. Now the kernel is loaded. Will try to boot it with kexec -e.

foxcris · 2023-02-20T15:51:05Z

After running

kexec -e

they system is hanning. The display on the router tells something about plugging in the wan cable (but its still plugged in). I needed to reboot it manually by switching it off and on again.
With

systemctl kexec

the system directly restarts and uses its own kernel again.

jimmy-ungerman · 2023-03-23T16:32:50Z

Curious if you've made any progress on this? Or if there's a different kernel that will work on the SE so that I can finally get multicast routing working?

fabianishere · 2023-03-23T17:09:24Z

Multicast routing should already be supported in the stock UDM SE kernels.

jimmy-ungerman · 2023-03-23T17:37:03Z

Doesn't seem to be working for me and all I saw was a bunch of posts pointing to here about Multicast routing...is there something I need to enable that I'm missing? Multicast DNS seems to work, but not routing

jimmy-ungerman · 2023-03-23T17:55:50Z

For example here:

I have a new BGP multicast route between my two nodes of 10.45.0.12, but it times out any time I try to connect

fabianishere · 2023-03-23T21:30:43Z

@jimmy-ungerman You can verify it on your device as follows:

$ zgrep "MROUTE" /proc/config.gz
CONFIG_IP_MROUTE_COMMON=y
CONFIG_IP_MROUTE=y
# CONFIG_IP_MROUTE_MULTIPLE_TABLES is not set
# CONFIG_IPV6_MROUTE is not set

Make sure you actually have the multicast routes installed into the kernel.

jimmy-ungerman · 2023-03-24T23:04:02Z

@fabianishere I'm sorry I realized I've been saying "Multicast" when I meant "MultiPATH". Are you aware if this is enabled or not?

fabianishere · 2023-03-26T15:55:34Z

@jimmy-ungerman No, multipath routing is not supported by the stock kernel. You’ll need to use a custom kernel for that (if you search in the repository, you’ll find others also using it for that feature).

jimmy-ungerman · 2023-03-28T16:22:34Z

Yeah that's what I thought. That's why I'm following this issue so closely, hoping to be able to get my UDM-Pro SE updated as soon as there is support

cftechwiz · 2023-06-05T03:44:27Z

Any progress on this? I am really in need of Multipath support also. Looks like some progress was made, any chance we have something for the SE anytime soon?

EddieRingle · 2023-06-08T07:34:56Z

After running
kexec -e
they system is hanning. The display on the router tells something about plugging in the wan cable (but its still plugged in). I needed to reboot it manually by switching it off and on again. With
systemctl kexec
the system directly restarts and uses its own kernel again.

Just wanted to add that I'm currently hitting this wall as well, on 3.0.13 (I've requested but still haven't received the GPL sources associated with 3.0.18). Working off of the v2.x branch, among other things, I had to update the udm-bootctl script to run kexec -e directly rather than systemctl kexec, because systemctl makes an extra syscall that causes the preloaded redir.so to abort. Even after all of that, the device seems to hang (routing connections stops, at least, so my SSH sessions hang) but eventually restarts itself using the stock kernel.

If there's a particular log I can check to get a better read on what's happening, please let me know. I'm also trying to simply enable multi-path routing, and it's increasingly looking like just overwriting the kernel in the boot image is the more feasible path than trying to get kexec hacks to work. (And as a side note, if anyone has resources handy for quickly modifying the U-Boot image specifically for UDM SE, that'd be appreciated.)

fabianishere · 2023-06-11T17:41:14Z

@EddieRingle Check the files in /sys/fs/pstore/. The kernel dumps the (partial) logs of the previous boot there (ramoops).

jimmy-ungerman · 2023-10-27T15:06:32Z

Are we safe to assume that work has stopped on this?

realies · 2024-06-09T12:07:29Z

has anyone been able to make this work on the udm se?

fabianishere changed the title ~~UDM-SE support?~~ Support for UDM SE Nov 16, 2022

fabianishere mentioned this issue Nov 16, 2022

Support for UXG-Pro #85

Open

simkin mentioned this issue Dec 5, 2022

Podman failing to start on UDM SE v3.0.10 unifi-utilities/unifios-utilities#428

Open

fabianishere added the kind/feature New feature or request label Dec 10, 2022

fabianishere linked a pull request Jun 11, 2023 that will close this issue

Add support for firmware 2.x/3.x #90

Open

Support for UDM SE #82

Support for UDM SE #82

Comments

jeroenops commented Oct 11, 2022 • edited Loading

fabianishere commented Oct 11, 2022

gtrabanco commented Oct 16, 2022

simkin commented Dec 15, 2022

fabianishere commented Dec 17, 2022

ppkhoa commented Dec 20, 2022

foxcris commented Jan 21, 2023

fabianishere commented Jan 21, 2023

foxcris commented Jan 22, 2023 via email

fabianishere commented Jan 22, 2023

foxcris commented Jan 23, 2023 • edited Loading

foxcris commented Jan 23, 2023

fabianishere commented Jan 23, 2023

boostchicken commented Jan 24, 2023

foxcris commented Jan 24, 2023

foxcris commented Jan 24, 2023

boostchicken commented Jan 24, 2023

foxcris commented Jan 28, 2023 • edited Loading

jimmy-ungerman commented Feb 12, 2023

foxcris commented Feb 19, 2023 • edited Loading

foxcris commented Feb 19, 2023

foxcris commented Feb 19, 2023

foxcris commented Feb 20, 2023

foxcris commented Feb 20, 2023

jimmy-ungerman commented Mar 23, 2023

fabianishere commented Mar 23, 2023

jimmy-ungerman commented Mar 23, 2023

jimmy-ungerman commented Mar 23, 2023

fabianishere commented Mar 23, 2023

jimmy-ungerman commented Mar 24, 2023

fabianishere commented Mar 26, 2023

jimmy-ungerman commented Mar 28, 2023

cftechwiz commented Jun 5, 2023

EddieRingle commented Jun 8, 2023

fabianishere commented Jun 11, 2023

jimmy-ungerman commented Oct 27, 2023

realies commented Jun 9, 2024

jeroenops commented Oct 11, 2022 •

edited

Loading

foxcris commented Jan 23, 2023 •

edited

Loading

foxcris commented Jan 28, 2023 •

edited

Loading

foxcris commented Feb 19, 2023 •

edited

Loading