wpa_supplicant autoconfigure removed in bookworm but not replaced with other mechanism, breaking network module

[guysoft]

In bookworm Raspberry Foundation/Ltd has removed wpa-suppilcant and switched to NetworkManger
In the network module we add a way to headless configure wpa-supplicant.

This is where its done in the network module:
https://github.com/guysoft/CustomPiOS/blob/devel/src/modules/network/start_chroot_script#L36

This is the issue on Rpi:
raspberrypi/bookworm-feedback#72

This is a related issue in FullPageOS:
guysoft/FullPageOS#570

What I think should be done is perhaps have an if statement depending on the distribution version, and use the solution here for bookworm, and perhaps add a symlnink from the old file. Open to other suggestions.
Might take me time to implement because I am busy theses days.

[guysoft]

Asked here NetworkManager if we need to generate a uuid or not:
https://lists.freedesktop.org/archives/networkmanager/2024-August/000333.html

[guysoft]

Confirmed, providing no uuid just generates one.

[guysoft]

At this point this works, the question is if to adopt what @KwadFan was working on in Mainsail, the wrote their own solution.
https://github.com/KwadFan/MainsailOS-dev/tree/chore%2Fbookworm/src%2Fmodules%2Fheadless_nm

[KwadFan]

I dont mind doing a PR if wished. My goal here was to keep it simple as possible, because the users we are aiming for, dont want to spend to much effort to get this thing connected.

But this method is odd, if you really know how to deal with .nmconnection files, you simply dont have the "power" to setup what ever you want.
Also want to mention, maybe it is a good idea to push your network module in a different direction for the future, since Infrastructure as Code is a hard buzzword, it could make sense to have something like netplan.io instead of a set of scripts.

For now, I would suggest, running both versions inside your repo and let maintainer decide what to use.

As an example for FullPageOS, honestly I hadnt a usecase for it yet but, I think some of your users dont want to deal with complicated network setup, the simply want to connect it to their WiFi networks. Maybe "our" solution is more handy in terms of "ease of use". But I can imaging there are human beeings with a deeper need to setup network connection in a advanced way.

[guysoft]

I think that most people use rpi-imager. The question is what are the current usecase for this configuration ATM. Its still relevant for non-rpi images.

I want to ask foosel tomorrow what she thinks tool.

And if anyone else wants to comment here that would really help us decide what is the best move for the future.

[KwadFan]

In this case, I wanted a solution that works with Armbian based and Rpi based images in the same way.
We always recommend using rpi-imager overall, but what to do if you move the printer and forget changing wifi while its live in its old location? Most of the time users where used to get the sdcard and doing the wpa_supplicant thingy.
Therefor I wanted to mimmick that behaviour.

But, yes, at this point "we" need some handy input from our communities/devs/endusers about the methods!
So, from my side I encurage you all to leave a comment about this.

Regards Kwad

[guysoft]

In this case, I wanted a solution that works with Armbian based and Rpi based images in the same way.

Hey mention now @foosel , hope you have time to review this, would value what you think about it.
Will also note that CustomPiOS v2 is close to release and will have support to build seamlessly for armbian, libre.computer and other devices too, so what @KwadFan is saying might be relevant too.

[foosel]

Tbh, at this point I'm fine with anything as long as the RPI Imager method (currently that's a firstrun.sh generated in the imager and run by a kernel commandline change) isn't destroyed by that. The thing is that the majority of users on a Pi will likely use this as they just want to get up running quickly and don't have any advanced wifi situations that require more control.

For the record, this is the current firstrun.sh template that works for provisioning on my headless testrig, and which is pretty much what the imager does. So as long as whatever you come up with here doesn't get into the way of this working out of the box, things should be fine. For reference, that calls /usr/lib/raspberrypi-sys-mods/imager_custom set_wlan 'SSID' 'PSK' 'COUNTRY' which in turn calls raspi-config nonint do_wifi_country "COUNTRY" and also creates a file /etc/NetworkManager/system-connections/preconfigured.nmconnection containing the wifi configuration, so pretty much the same as in the linked solution in the OP.

Personally I'd suggest going with something like the solution of @KwadFan, as that will keep existing stuff working and just act if required, from how I understand it. Though I would maybe go with a different name than headless_nm.txt, as that will be quite meaningless to people who are less experienced with the underlying systems, but who might still require advanced configuration options, or configuration changes.

[KwadFan]

@guysoft @foosel

So, I dont mind working on this, to get it suitable for all of us.
Here some key features I want to keep. First of all it should be stay written in sh/bash because its the safest option in a early boot stage, dont want to mess with python or alike which would handle the TOML style .nmconnection file way better, to be honest.

But I see, there is a need for some more advanced setup as in its current state, therefor I would learn more about the whole options we could set with it. That said I wouldnt change the service itself in it's current form because it works well, but I would enhance the underlying script for better options. Maybe it is also a good idea to transform the .txt with something like dos2unix instead of pushing users to have a specific text editor.

And as @foosel mentioned, yes, the name is pretty much what came to my mind because I am aware of the underlying system and was a obvious choice, but in "enduser" perspective thats really meaningless.

To come to an end here, what should be discussed?

1. Name sugesstions for the service name and with that inline for module name/config file.

2. Which features/setup possibilties should the config file provide? (Keeping in mind to much options might and will overwhelm some users)

3. Maybe having an option to copy over a specified .nmconnection file instead of getting Vars from the config file for advanced setups?

Please answer to those questions and I will work on this pretty quickly, because I am aware we need to update to bookworm asap. Bullseye is coming to an end and if I am not mistaken none of our projects is on bookworm right now.

In general, I would prefer to have that upstream, then using my version.

Regards Kwad

[foosel]

Maybe it is also a good idea to transform the .txt with something like dos2unix instead of pushing users to have a specific text editor.

A very good idea, we've seen a ton of issues in the past with this.

With the old network configuration approach that predates RPiOS' wpa_supplicant.conf by several years, we went with a .txt file that was a symlink already, an heavily commented, but that won't work here if we go just by file existence. So something like some tool support to create such a file if needed might also be a good idea (or just a README with examples on the FAT).

1. Name sugesstions for the service name and with that inline for module name/config file.

I think the service name is totally fine, it's more the config file I'm worried about. How about "network_config.txt"?

2. Which features/setup possibilties should the config file provide? (Keeping in my to much options might and will overwhelm some users)

3. Maybe having an option to copy over a specified .nmconnection file instead of getting Vars from the config file for advanced setups?

So, what's an issue with 3. is that this won't allow to set the wifi country as with NM that's now handled separately from the wifi config itself. I'm wondering if we should maybe just keep things ultra flexible by going with a config file like now, but also supporting a key NMCONFIG in there which then can contain a full heredoc of the connection file to write (untested example):

#### Headless Wifi configuration

# Wifi Country code
# See Wifi-README.txt for details!
REGDOMAIN="US"

# advanced config
read -r -d '' NMCONFIG <<'EOF'
[connection]
id=wifi
uuid=95b17351-297c-4d9c-90f9-669d3ab81464
type=wifi

[wifi]
mode=infrastructure
ssid=MySSID

[wifi-security]
key-mgmt=wpa-psk
psk=MyPassword
hidden=yes

[ipv4]
method=auto

[ipv6]
addr-gen-mode=default
method=auto

[proxy]
EOF

It would be interesting to see what happens when you put multiple connection files into the folder, basically leaving preconfigured.nmconnection alone and instead putting a headless_nm.connection there that's under control of that service. My expectation would be that NM would try to connect to either and go with whatever works, so maybe that's the cleanest solution to peacefully coexist with any other configuration approaches.

dont want to mess with python or alike which would handle the TOML style .nmconnection file way better, to be honest

How about something like yq? nmconnection appears to be ini though, not toml, and I guess some basic heredoc with variable replacement should work just fine for the simple cases.

[KwadFan]

@foosel
Sorry, didnt find time to reply in a adequate way.

Here are my thoughts about your suggestions.

Renaming the file to network_config.txt sounds good and I will change that. This is more obvious then my approach.

What makes me very anxious is your suggestions about the "inline nmconfig", especcially
read -r -d '' NMCONFIG <<'EOF'
if users somehow mess up that line, e.g. deleting a single char by accident, that will mess up the whole thing by it self.
To prevent that we have to filter out the contents in a better way.

Also, you suggested to not replace the preconfigured.nmconnection which, in my humble opinion, would interfere witth my thoughts about the usage overall, it is more like a wpa_supplicant.conf replacement mechanism then a "advanced configuration method", but if that is your idea behind, it should have advanced capabilties by having to setup additional parameter like for example SAVE_AS=myfooconnection instead.

That said, boils it down to what I had in mind, something like nmconnections.d in /boot for advanced setups.
Clearly there is the issue to have it documented properly with sources from networkmanager docs itself, but its just a matter of "more than two eyes" review. A simple Readme.txt with hyperlinks should be more tha sufficiant for that.

So, we (in that meaning you, Guy and me) should have a clear path how it should look like without dictating against each other. I am at will to work on it for sure, but I dont want to make work more than once. So, we really should decide together how it should be look like before I write a line of code.
I can do a POC if wished and we make our decisions based on that, let me know if there is an interesst for that.

Regards Kwad

[foosel]

What makes me very anxious is your suggestions about the "inline nmconfig", especcially
read -r -d '' NMCONFIG <<'EOF'
if users somehow mess up that line, e.g. deleting a single char by accident, that will mess up the whole thing by it self.
To prevent that we have to filter out the contents in a better way.

True, there's a risk here, and you are right, people will break everything. Forget that I suggested that 😅

Also, you suggested to not replace the preconfigured.nmconnection which, in my humble opinion, would interfere witth my thoughts about the usage overall, it is more like a wpa_supplicant.conf replacement mechanism then a "advanced configuration method"

It was just an idea to allow for easier advanced configuration and less interference with existing (RPi specific) stuff, I'm fine either way :) A dedicated nmconnections.d folder would also work fine!

My point of view here in general is really to allow for an easy mechanism that always works and doesn't require the use of the imager/firstrun.sh to get wifi configuration done in a headless setup, while not interfering with the initial bootstrap through the imager's firstrun.sh.

Now, with that being said, I'm wondering if we shouldn't just consider wifi configuration and country setting separate cases, that would make everything much easier. A single /boot/wifi_country.txt (or something along those lines) that makes sure the right country is set. And a separate /boot/network_config.d with nmconnection files for actually configuring the network. Possibly with a dedicated /boot/network_config.txt that just gets copied over as preconfigured.nmconnection.

The question this raises of course is whether we trust users to do configuration directly on simple nmconnection files. I'm honestly not sure myself. I'm frankly just trying to keep things as simple as possible here and not creating too much feature bloat and maintenance overhead in the resulting end product 😅

[KwadFan]

Just a start of solving it.

Todo:

• Build in backwards compatibility for bullseye (have to be tested! possibly also works on *buntu variants?!?)
• Convert file with dos2unix to avoid the Texteditor problematic
• Seperate simple configuration and setup for advanced users
• Allow setups using a networmanager.d in /boot for expert mode setups
  ( This will allow replace/adding new connections)
• Allow to setup REGDOMAINvia file instead of variable if advanced or expert setups are used.

I'll try to finish that as fast as possible.

For now please use test images from https://github.com/KwadFan/MainsailOS-dev/actions/runs/11212635550
I'll post new runs as soon I changed something

Regards Kwad

[guysoft]

I see you have a PR now. the checkboxs are done yet or perhaps better to move them there?
#230

[foosel]

Will be a while until I can take a look, I currently have my hands extremely full with other stuff.