Latest flatpak + appstream update breaks XIVLauncher with --parent-expose-pids --parent-share-pids

[yaomtc]

From Alicia on Discord:

On arch, I'm suddenly getting a really weird issue where the launcher fails to start when using the recommended launch options for steam.
Launch options: LD_PRELOAD="" XL_SECRET_PROVIDER=FILE %command% run --parent-expose-pids --parent-share-pids --parent-pid=1 --branch=stable --arch=x86_64 --command=xivlauncher dev.goats.xivlauncher
The only response in terminal under these launch options is bwrap: creation of new user namespaces was not disabled as requested
I've selectively removed launch options and found that removing --parent-expose-pids --parent-share-pids fixes the issue, but I have no idea why, and would like to be able to keep these options ofc
[...]
I restored my system with timeshift and I found out what action I did broke it
it looks like the latest flatpak + appstream update caused this break

I can confirm this. After updating (Arch), the game would no longer launch using the recommended options. The workaround above works.
I can post an issue to the Flatpak issue repository, if you think it's upstream. https://github.com/flatpak/flatpak/issues

[Blooym]

I believe this is an upstream issue. None of the documentation for those flags indicate that it was deprecated and we've not changed anything in a while with the Flatpak that might've caused it.

Will keep this open, if you could file this upstream that'd be great

[rankynbass]

I tried launching a few other flatpaks with the same pid options on arch. Same result. It's definitely either an arch packaging or flatpak upstream issue.

[rankynbass]

Temporary workaround for Arch users. This is (probably) caused by the program /usr/bin/bwrap not having the setuid bit. Do ls -l /usr/bin/bwrap. If it shows -rwxr-xr-x permissions, then you can probably fix by doing sudo chmod u+s /usr/bin/bwrap. Then the permissions should show -rwsr-xr-x, and xivlauncher should work.

[yaomtc]

That does fix things to some extent, the launcher does start. Thanks!

Upon closing the game, XIVLauncher does not close, and so Steam shows it and FFXIV as both running. Not a big issue though, telling Steam to close XIVLauncher works fine.

[smcv]

Hello, I am a Flatpak contributor. I would not recommend setting the setuid bit on /usr/bin/bwrap unless your Linux distribution requires it for basic functionality of Flatpak. It's needed on some obsolete versions of Debian and RHEL/CentOS, and on some non-default configurations of Arch and Gentoo, but should not be necessary in the default configuration of a modern Linux distribution.

Please see https://github.com/flatpak/flatpak/wiki/User-namespace-requirements for details of the two ways that bwrap can be set up: either setuid root, or unprivileged. Making it setuid root is not really recommended any more, and is known to break some Flatpak apps (notably, running Steam itself as a Flatpak). I'm also surprised that flatpak run --parent-expose-pids or --parent-share-pids will work at all with a setuid bwrap. (Perhaps it just silently doesn't?)

Why does the XIVLauncher FAQ recommend --parent-expose-pids --parent-share-pids on Deck? Using --parent-pid=1 is not how those options are intended to be used: the way they are intended to be used is for the --parent-pid to be a sandboxed Flatpak app itself, and not a process in the trusted computing base.

If you pass a non-sandboxed, non-Flatpak process ID to --parent-pid, that's opening up a significant "hole" in the sandbox through which the sandboxed app can attack non-sandboxed processes: at the very least, it can kill or pause the non-sandboxed processes, and it's possible that it can also cause arbitrary code execution in those processes by tracing them (a sandbox escape).

The workaround that I would recommend for this issue is removing --parent-pid=1, --parent-expose-pids and --parent-share-pids from the command-line.

The fact that launching fails with bwrap: creation of new user namespaces was not disabled as requested might still be a Flatpak bug, if it can be reproduced when using a --parent-pid that is a Flatpak app. In some brief testing I have not been able to reproduce it for a Flatpak app parent, though.

The source of that error message is that I added some security hardening in Flatpak 1.15.6 to make the sandbox more resilient against malicious/compromised apps, which I'm hoping will also allow for future improvements (adding more features, and/or disabling other security hardening mechanisms that have more performance impact). If you see that error message, it means that bwrap has detected that the security hardening was not effective in this case, and as a result it will "fail closed".

[rankynbass]

Why does the XIVLauncher FAQ recommend --parent-expose-pids --parent-share-pids on Deck? Using --parent-pid=1 is not how those options are intended to be used: the way they are intended to be used is for the --parent-pid to be a sandboxed Flatpak app itself, and not a process in the trusted computing base.

XIVLauncher is a flatpak app, unless I'm misunderstanding what you're saying. It's a third-party launcher for Final Fantasy XIV. But in order to get it working properly on the steam deck, we need to activate FFXIV in steam, otherwise the controller doesn't work properly. However, without the pid sharing commands, it will never detect when the game (and launcher) closes, and it will remain in the "Running" state until steam is closed. It can't be stopped manually, either; only shutting down steam will work.

[smcv]

XIVLauncher is a flatpak app, unless I'm misunderstanding what you're saying.

Sure, but the FAQ recommends Steam launch options for a flatpak run command that include --parent-pid=1, and process 1 is not a Flatpak app - it's the init system (systemd, on Steam Deck). That is not how the option is intended to be used.

Flatpak is designed to put each sandboxed app in its own new process ID namespace, but using --parent-pid=1 --parent-share-pids defeats that, and instead runs XIVLauncher in the "init namespace" at the root of the tree of namespaces.

It's a third-party launcher for Final Fantasy XIV. But in order to get it working properly on the steam deck, we need to activate FFXIV in steam, otherwise the controller doesn't work properly. However, without the pid sharing commands, it will never detect when the game (and launcher) closes, and it will remain in the "Running" state until steam is closed. It can't be stopped manually, either; only shutting down steam will work.

The Steam Overlay LD_PRELOAD module is, unfortunately, known to have some incompatibilities with non-trivial use of process ID namespaces. I've raised this with Valve, but it's an issue in the proprietary parts of Steam, so only Valve can resolve it.

The Steam Overlay is involved in how Steam Input works, so I suspect that's at least part of what you're seeing here.

[smcv]

we need to activate FFXIV in steam

What does this mean, precisely? There might be some other way to achieve what you want.

Does XIVLauncher do any launching of processes outside its sandbox (flatpak-spawn --host) or in a sub-sandbox (flatpak-spawn without --host), or anything else involving IPC to start a process "somewhere else", or otherwise rearranging the process tree?

However, without the pid sharing commands, it will never detect when the game (and launcher) closes, and it will remain in the "Running" state until steam is closed. It can't be stopped manually, either; only shutting down steam will work.

What do the various "it" refer to here?

Do you mean (my additions in []):

However, without the pid sharing commands, [Steam] will never detect when the game (and launcher) closes, and [the Steam library shortcut for XIVLauncher] will remain in the "Running" state until steam is closed. [The launcher and/or the game] can't be stopped manually [via the Steam UI], either; only shutting down steam will work.

Or something else?

[tommadness]

However, without the pid sharing commands, [Steam] will never detect when the game (and launcher) closes, and [the Steam library shortcut for XIVLauncher] will remain in the "Running" state until steam is closed. [The launcher and/or the game] can't be stopped manually [via the Steam UI], either; only shutting down steam will work.

This is the correct interpretation.

What does this mean, precisely? There might be some other way to achieve what you want.

XIVLauncher needs to initialize Steam API for authentication ticket and Steam Overlay purposes.

If we have some way to alleviate both of those issues without the use of pid flags, that would solve a common source of user troubles.

[smcv]

XIVLauncher needs to initialize Steam API for authentication ticket and Steam Overlay purposes.

What does this mean in terms of concrete API calls, or sockets, or pipes, or shared memory, or whatever? I might be able to bring this up with Steam-on-Linux developers, but I'm going to need specifics - ideally as concrete as possible.

As far as I know, the current implementation of Steam APIs in steamclient.so is based on shared memory in /dev/shm and/or /tmp, and possibly ~/.steam/steam.pipe; and the protocol used in those places involves processes identifying themselves in-band by their process IDs, so it is likely to have the same problem as the Steam Overlay when crossing a pid namespace boundary. In the past I've tried suggesting that a protocol with message framing over an AF_UNIX socket (like X11, Wayland, D-Bus, PulseAudio, Pipewire, etc.) would be more container-friendly, but I don't think there has been any progress towards providing one.

[marzent]

The problem is here is that unfortunately only part of the steam IPC mechanism breaks when launching a flatpak from Steam itself (with more breakage as I understand when Steam itself is run as flatpak).
XL.Core will register as FFXIV with Steam, which works, however the mechanism on Steams side to check if the game is still running does not work when crossing the pid namespace boundary, hence the workaround.
The problem is that users with multiple Steam installations end up in a situation where Steam can block launching the game then for other instances, which is especially egregious for family-shared accounts.
Not sure if this is helpful, but clicking the button to stop the running game (which sends signal 9 to the process registered with that steam app id) also does not work if the target process is run in a flatpak.

[Gryxx]

@smcv Small correction

the Steam library shortcut for XIVLauncher will remain in the "Running" state until steam is closed

Actually the shortcut for XIVLauncher will close, the FFXIV entry that is launched from inside the flatpak launcher will not.

[perroboc]

Got this error in Fedora 40. XIVLauncher options are set to LD_PRELOAD= XL_SECRET_PROVIDER=FILE %command% run --branch=stable --arch=x86_64 --command=xivlauncher dev.goats.xivlauncher, as stated as a workaround in previous posts, and the Launcher and Game run great.

The problem, as stated above, is that the game keeps running:

You can't kill the process (from what I've tried), and the only solution right now is to restart Steam.

[oz-orz]

You can kill the relevant processes via CLI using pkill or killall. On Fedora, with the Flatpak XIVLauncher, the relevant processes may include xivlauncher, ffxiv_dx11.exe, and others (Dalamud has one, with a name like Dalamud.Injectio something). Easy way to list processes is to run ps -e. On Arch, with an AUR installation of XIVLauncher, I believe the XIVLauncher process might be called xivlauncher-bin.

(Note: For pkill you will need to supply the process number, listed to the left of the process name in the output of ps -e; for killall, you can use the process name.)

[taylor85345]

I wanted to update that this issue now seems to be impacting the latest Preview build of SteamOS 3.6 on the Steam Deck.

[omor1]

An update that, as of today, it seems that the Beta client for the Steam Deck also installs an updated Flatpack 1.15.6 and causes this issue.

I'm not really sure why the Beta is installing OS components instead of just updating the Steam client. I suspect that Valve is getting ready to ship an updated SteamOS soon though, so it would be good to look into this—it's especially problematic on the Steam Deck, as there really isn't an easy way to kill XIVLauncher or FFXIV without restarting.

[Azure-Agst]

Not to necro this post, but I'm also running into this same bwrap: creation of new user namespaces was not disabled as requested error on the beta Steam Deck branch. Having to downgrade back to the stable branch to get the launcher to start.

The SetUID trick (although I understand it's ill advised) also doesn't work due to a R/O file system.