Drop xauth dependency / custom cookie creation

[mviereck]

Hello @eine ,

are you still using x11docker and runx?
Once we had the issue that MSYS2 does not provide package xauth so x11docker and runx could only run with option --no-auth.
These days I managed to create a cookie file without the need of xauth.
I'd like to implement this in runx and want to ask you if you could test it if it works.

[esaulenka]

I am using runx on MSYS2 (without auth, sure). Thanks for this app, running remote applications via SSH -X (or -Y in my case) without additional Linux virtual machines is very useful.
And I am able to do some tests if needed.

[mviereck]

Thank you for your offer to test!
I have commited a new version that creates a cookie without xauth. Please run a test without option --no-auth in MSYS2.

[esaulenka]

I run your latest script (this one #8 (comment) ), but bash on MSYS (and google too :) ) doesn't know, what is the strlenhex and makecookie.

$ ./runx.sh
telnet: Unable to connect to remote host: Connection refused
runx note: Windows firewall settings can forbid application access
  to the X server. If no application window appears, but no obvious error
  is shown, please check your firewall settings.
  Compare:   https://github.com/mviereck/x11docker/issues/108

runx note: If you get application error messages like 'Cannot open display'
  or 'Invalid MIT-MAGIC-COOKIE', the X authentication cookie might be broken.
  You can remove old cookies and stop running X servers with: runx --cleanup

DISPLAY=192.168.56.1:1 XAUTHORITY=/home/alexeye/runx_Xauthority
./runx.sh: line 456: strlenhex: command not found
./runx.sh: line 458: makecookie: command not found
awk: cmd. line:5: warning: no hex digits in `\x' escape sequence

[mviereck]

I run your latest script (this one #8 (comment) ), but bash on MSYS (and google too :) ) doesn't know, what is the strlenhex and makecookie.

Ooops, I forgot to include some subfunctions.
I've added them to the script, please update and try again.
I've taken the code from x11docker.

[mviereck]

It would be nice if you could test again.
I feel uncomfortable to have a possible broken version online.

[esaulenka]

Sorry, I was busy with my work tasks.

I did some tests, but unfortunately I could not get it to work.

First, I found issue with convertpath(). For some strange reason, MSYS uses C:\msys64\home\username instead of standard Windows C:\Users\username. You can use cygpath utility to convert (as I see, you already use it sometimes). It is always included in MSYS installation. So, in my case there is two copies of same runx_Xauthority: in ~ and in %HOME%.

Then, I found that your cookie have a strange form. As I see on stackoverflow, address field should contain hostname in ascii form, but you put there 4 bytes - IP in binary form.

Third, I found ready to use xauth in a VcXsrv. When I did alias xauth=/C/Program\ Files/VcXsrv/xauth.exe runx will run it. So, you can simply check for it and use when possible.
But even in this case xeyes on remote machine cannot find display...

[mviereck]

Thank you for the tests!
It seems introducing the custom cookie has more issues than I expected. At first I thought it would just work and only needs confirmation.

For now I've added a message that the custom cookie is experimental and one might use --no-auth if it fails.

First, I found issue with convertpath(). For some strange reason, MSYS uses C:\msys64\home\username instead of standard Windows C:\Users\username. You can use cygpath utility to convert (as I see, you already use it sometimes). It is always included in MSYS installation. So, in my case there is two copies of same runx_Xauthority: in ~ and in %HOME%.

x11docker creates two cookie files with the same content, one for the X server and one for the X client.
VcXsrv should use the cookie file in %HOME%, the client, in your case ssh -Y, should use the one in $HOME of MSYS2.
The --verbose output should show you the 2 cookie pathes, the one with $HOME is estimated with cygcheck.exe.

For some strange reason, MSYS uses C:\msys64\home\username

MSYS2 simulates a Linux environment, so this is consistent behaviour. And one might not want Linux config files in Windows' %HOME%.

Then, I found that your cookie have a strange form. As I see on stackoverflow, address field should contain hostname in ascii form, but you put there 4 bytes - IP in binary form.

I found this syntax by looking at cookie files with hexedit. I don't know a documentation for this behaviour.
xauth uses this syntax if DISPLAY has the form IP:DISPLAYNUMBER. If DISPLAY has the form :DISPLAYNUMBER, it stores the ascii hostname.
Example: xauth -f ~/mycookie add 127.0.0.1:100 MIT-MAGIC-COOKIE-1 $(mcookie) (4 byte IP)
Example: xauth -f ~/mycookie add :100 MIT-MAGIC-COOKIE-1 $(mcookie) (ascii hostname)

It might be worth a try to create a cookie $(hostname)/unix:DISPLAYNUMBER. I've changed the code accordingly. Maybe this helps. As it seems, only the final cookie code part itself and the display number are essential for authentication.

Third, I found ready to use xauth in a VcXsrv. When I did alias xauth=/C/Program\ Files/VcXsrv/xauth.exe runx will run it. So, you can simply check for it and use when possible.

I once tried xauth from VcXsrv and had a lot of issues with it. So I gave up to use it. Yet I don't remember the details, a related ticket is mviereck/x11docker#160.

But even in this case xeyes on remote machine cannot find display...

The exact error message is important at this point.

• Something like "Cannot open display" indicates that variable DISPLAY is wrong or the X server is down. (If you run runx with option --desktop, you can immediately see if the X server is running at all.)
• Something like "No protocol specified" indicates an authentication issue. Than it's either a wrong XAUTHORITY variable or a broken cookie file.

It might help to run runx --cleanup before every test run to stop all running instances of VcXsrv and Xwin and to delete the cookie files. Sometimes the cookie file is damaged for unknown reasons and causes startup failures.

[esaulenka]

oh, I completely confused :(

now I just did simple tests with your latest app:

$ source /usr/local/bin/runx --ip 127.0.0.1 --display 123 --verbose
....

$ echo $DISPLAY $XAUTHORITY
127.0.0.1:123 /home/alexeye/runx_Xauthority

$ hexdump -C $XAUTHORITY
00000000  ff ff 00 11 41 6c 65 78  65 79 45 2d 31 31 38 30  |....AlexeyE-1180|
00000010  2f 75 6e 69 78 00 03 31  32 33 00 12 4d 49 54 2d  |/unix..123..MIT-|
00000020  4d 41 47 49 43 2d 43 4f  4f 4b 49 45 2d 31 00 10  |MAGIC-COOKIE-1..|
00000030  e3 96 2a b4 ae fb d8 31  56 95 55 65 d0 04 78 d3  |..*....1V.Ue..x.|
00000040

$ ssh -XC ubuntu-vm xeyes
xxx's password:
Warning: untrusted X11 forwarding setup failed: xauth key data not generated
Error: Can't open display:

$ ssh -YC ubuntu-vm xeyes
xxx's password:
Warning: No xauth data; using fake authentication data for X11 forwarding.
Invalid MIT-MAGIC-COOKIE-1 keyError: Can't open display: localhost:11.0

so, it didn't working even with Y option in SSH.

$ source /usr/local/bin/runx --ip 127.0.0.1 --display 123 --verbose --no-auth
....


$ ssh -YC ubuntu-vm xeyes
xxx's password:
Warning: No xauth data; using fake authentication data for X11 forwarding.

but here it works fine

[mviereck]

oh, I completely confused :(

You have a close look on what is happening and I appreciate that!

$ echo $DISPLAY $XAUTHORITY
127.0.0.1:123 /home/alexeye/runx_Xauthority

$ hexdump -C $XAUTHORITY
00000000 ff ff 00 11 41 6c 65 78 65 79 45 2d 31 31 38 30 |....AlexeyE-1180|
00000010 2f 75 6e 69 78 00 03 31 32 33 00 12 4d 49 54 2d |/unix..123..MIT-|
00000020 4d 41 47 49 43 2d 43 4f 4f 4b 49 45 2d 31 00 10 |MAGIC-COOKIE-1..|
00000030 e3 96 2a b4 ae fb d8 31 56 95 55 65 d0 04 78 d3 |..*....1V.Ue..x.|
00000040

This looks well so far.
Could you compare with the cookie file in %HOME% if it shows any difference?

I have the impression that ssh does not find the cookie file, maybe due to path conversion. I can reproduce the error messages on Debian if I set XAUTHORITY to non-existing or empty files:

$ XAUTHORITY=/nonsense ssh -Y localhost -- xeyes
Warning: No xauth data; using fake authentication data for X11 forwarding.
Invalid MIT-MAGIC-COOKIE-1 keyError: Can't open display: localhost:10.0

$ touch nonsense

$ XAUTHORITY=~/nonsense ssh -Y localhost -- xeyes
Warning: No xauth data; using fake authentication data for X11 forwarding.
Invalid MIT-MAGIC-COOKIE-1 keyError: Can't open display: localhost:10.0

However, hexdump finds the file in variable XAUTHORITY, so why should ssh have an issue here?
I am a bit clueless now.

[mviereck]

Just an idea: Maybe IP 127.0.0.1 is an issue.
Try without setting option --ip and use the IP provided by runx.

[esaulenka]

Could you compare with the cookie file in %HOME% if it shows any difference?

no, they absolutely the same (only differs with several milliseconds in a timestamp)

I have the impression that ssh does not find the cookie file

I cannot find info about X-system forwarding in SSH debug output. Maybe, you will more lucky ? :)

use the IP provided by runx.

Same behavior. My host machine have several IPs (I run VirtualBox on it, and there is several virtual network adapters), but I checked that IP which runx was assigned to display, was available from linux machine.

Also I tried to use default filename .Xauthority (with simple hack to runx), but nothing changed.

BTW, I read man authx, but didn't understand, on which side this application needed. It seems SSH use it to transfer cookie, so it should be available on both sides.

[mviereck]

I cannot find info about X-system forwarding in SSH debug output. Maybe, you will more lucky ? :)

I've tried with an invalid cookie file and ssh option -vvv. I found that it tries to use /usr/bin/xauth but fails:

debug2: x11_get_proto: /usr/bin/xauth  list :0.0 2>/dev/null
Warning: No xauth data; using fake authentication data for X11 forwarding.
debug1: Requesting X11 forwarding with authentication spoofing.
debug2: channel 0: request x11-req confirm 1

So maybe runx creates a valid cookie, but ssh is not able to create one for X forwarding without xauth.

BTW, I read man authx, but didn't understand, on which side this application needed. It seems SSH use it to transfer cookie, so it should be available on both sides.

I'd say ssh -X/-Y needs xauth, but xauth is not available in MSYS2. (The remote side does not need xauth.)
Maybe we could fool ssh with a fake xauth that just returns the cookie created by runx.

[mviereck]

I've asked at MSYS2 to add a package xauth, but they don't want to because it is out of focus of their project.

Maybe we could have a look at xauth.exe of VcXsrv again.

Could you try alias xauth=/C/Program\ Files/VcXsrv/xauth.exe runx again and show me the hexdump of the cookie?

To avoid possible issues due to two different cookie files, I have changed runx to use one cookie file for server and client only. So if ssh creates a new cookie, it will be visible to X server and X client at the same time.

I've also made some path conversion changes to support xauth from VcXsrv.

I hope I did not break anything for Cygwin or WSL due to changed pathes; now the single cookie is located at %HOME%/Xauthority.runx.

Edit: Thinking about this, xauth.exe of VcXsrv will fail with ssh. runx sets XAUTHORITY to a Linux path within MSYS2 so MSYS2 applications can find it, but xauth.exe of VcXsrv expects a Windows path because it is not aware of MSYS2.
MSYS2 does some path conversion in some cases, but I don't know exactly how and when. Maybe it is possible to set XAUTHORITY to a path that is understood by MSYS2 applications as well as by Windows applications.

Edit2:
I had a look at some MSYS2 documentation and set the path in the form /c/.... This should be converted by MSYS2 in a way that is hopefully understood by ssh and xauth.exe as well.
Maybe you need to specify XAUTHORITY so the conversion takes place:

XAUTHORITY=$XAUTHORITY ssh -Y [...]

[esaulenka]

Could you try alias xauth=/C/Program\ Files/VcXsrv/xauth.exe runx again and show me the hexdump of the cookie?

(with updated runx):

$ source runx -v                                                                                                            
C:\msys64\usr\bin\bash: connect: Connection refused                                                                         
C:\msys64\usr\bin\bash: /dev/tcp/192.168.56.1/6739: Connection refused                                                      
DISPLAY=192.168.56.1:739 XAUTHORITY=/c/Users/alexeye/Xauthority.runx                                                        
Welcome to the VcXsrv X Server                                                                                              
Vendor: The VcXsrv Project                                                                                                  
Release: 1.20.14.0                                                                                                          
                                                                                                                            
OS: Windows NT 6.2 build 9200 (64-bit)                                                                                      
Contact: [email protected]                                                                                        
                                                                                                                            
_XSERVTransSocketOpenCOTSServer: Unable to open socket for hyperv                                                           
_XSERVTransOpen: transport open failed for hyperv/AlexeyE-1180:739                                                          
_XSERVTransMakeAllCOTSServerListeners: failed to open listener for hyperv                                                   
LoadPreferences: C:\msys64\home\alexeye\.XWinrc not found                                                                   
LoadPreferences: Loading C:\Program Files\VcXsrv\system.XWinrc                                                              
(II) IGLX: Loaded and initialized swrast                                                                                    
(II) GLX: Initialized DRISWRAST GL provider for screen 0                                                                    
winClipboardThreadProc - DISPLAY=127.0.0.1:739.0                                                                            
Using Composite redirection                                                                                                 
winClipboardProc - xcb_connect () returned and successfully opened the display.                                             
winMultiWindowWMProc - Error code: 8 (Match), ID: 0x00180000, Major opcode: 12 (ConfigureWindow), Minor opcode: 0 ((null))  

$ xauth list                                                                    
AlexeyE-1180/unix:739  MIT-MAGIC-COOKIE-1  ddb043870d1a05eff3733c0d0d0ae5e0     
                                                                                
$ hexdump -C $XAUTHORITY                                                        
00000000  01 00 00 0c 41 6c 65 78  65 79 45 2d 31 31 38 30  |....AlexeyE-1180|  
00000010  00 03 37 33 39 00 12 4d  49 54 2d 4d 41 47 49 43  |..739..MIT-MAGIC|  
00000020  2d 43 4f 4f 4b 49 45 2d  31 00 10 dd b0 43 87 0d  |-COOKIE-1....C..|  
00000030  1a 05 ef f3 73 3c 0d 0d  0a e5 e0 f2 ff ff 00 0c  |....s<..........|  
00000040  41 6c 65 78 65 79 45 2d  31 31 38 30 00 03 37 33  |AlexeyE-1180..73|  
00000050  39 00 12 4d 49 54 2d 4d  41 47 49 43 2d 43 4f 4f  |9..MIT-MAGIC-COO|  
00000060  4b 49 45 2d 31 00 10 dd  b0 43 87 0d 1a 05 ef f3  |KIE-1....C......|  
00000070  73 3c 0d 0d 0a e5 e0 f2                           |s<......|          
00000078

$ xauth --version
usage:  C:\Program Files\VcXsrv\xauth.exe [-options ...] [command arg ...]
.... (here is no version in the output)                                                                    

(I dont know, why file contains two records, but xauth show only one)

$ ssh -vvv -X user@machine
OpenSSH_8.9p1, OpenSSL 1.1.1q  5 Jul 2022
....
debug1: No xauth program.
Warning: untrusted X11 forwarding setup failed: xauth key data not generated

But when I did simple script /usr/X11R6/bin/xauth (this path hardcoded in my SSH. I didn't checked, can it searh for xauth anywhere else).

#! /bin/bash
/C/Program\ Files/VcXsrv/xauth.exe $@

I got in ssh output

debug2: client_x11_get_proto: xauth command: /usr/X11R6/bin/xauth -f /tmp/ssh-QZvF2LNiPOAP/xauthfile generate 192.168.56.1:739 MIT-MAGIC-COOKIE-1 untrusted timeout 1260 2>/dev/null
debug2: x11_get_proto: /usr/X11R6/bin/xauth -f /tmp/ssh-QZvF2LNiPOAP/xauthfile list 192.168.56.1:739 2>/dev/null
debug1: Requesting X11 forwarding with authentication spoofing.

and... yes!! currently I can run X-applications!!

I've asked at MSYS2 to add a package xauth, but they don't want to because it is out of focus of their project.

Yesterday I looked at xauth sources. It is very simple app, but it depends on X core things. Currently X-Window not supported on MSYS and I think maintainers don't want to do it.

[mviereck]

and... yes!! currently I can run X-applications!!

Great!
Thank you for your patience in testing this. :-)

I see two possible ways now:

• runx checks for xauth.exe if a native xauth is not available.
  • If yes, it uses it for cookie creation and furthermore creates your script /usr/X11R6/bin/xauth and makes sure that /usr/X11R6/bin is within PATH.
• Build xauth in MSYS2. Some Xorg dependencies might be dropped; some code requests the cookie directly from the X server instead of creating it itself. Essentially xauth should only need the structure of the cookie file, likely defined in some header, but no other X11 dependencies. Though, I am not familiar with C and would likely fail with this task.

I dont know, why file contains two records, but xauth show only one

runx creates a second entry with cookie family 'wild' / ffff. This allows docker container applications to use it.

.... (here is no version in the output)

You should get the xauth version with xauth -V.

/C/Program\ Files/VcXsrv/xauth.exe $@

Side note: You should always use "$@" instead of $@ to transfer arguments. This preserves possible whitespace in arguments. (Multiple options are still split because $@ is an array, not a string.)

/usr/X11R6/bin/xauth (this path hardcoded in my SSH)

Interestingly on Debian it seems to be hardcoded as /usr/bin/xauth.

[mviereck]

runx now uses xauth.exe from VcXsrv if no native xauth is available,
and also it creates a wrapper script in /usr/X11R6/bin.

If all is fine now, you should be able to use runx and ssh -X in MSYS2 without any manual setup.

[esaulenka]

Interestingly on Debian it seems to be hardcoded as /usr/bin/xauth.

Yes, I see the same. Maybe we need to check WSL and CygWin ?
latest MSYS2

$ strings /usr/bin/ssh.exe | grep xauth
/usr/X11R6/bin/xauth
xauthlocation
No xauth program.
Warning: No xauth data; using fake authentication data for X11 forwarding.
%s/xauthfile
xauthfile path too long
xauth command: %s
Warning: untrusted X11 forwarding setup failed: xauth key data not generated

very, very old Ubuntu:

$ strings /usr/bin/ssh | grep xauth
/usr/bin/xauth
xauthlocation
Warning: untrusted X11 forwarding setup failed: xauth key data not generated
Warning: No xauth data; using fake authentication data for X11 forwarding.
No xauth program.
%s/xauthfile
%s: xauthfile path too long

You should always use "$@" instead of $@

ok, thanks for explanation

Build xauth in MSYS2. ... Though, I am not familiar with C

I know C good enough (my current job - to support ancient applications running on small embedded systems), but I think it will be better to write new one from scratch (and sure, it should be interpreted language like python or bash - you shouldn't force your user to install make/gcc/etc).
But guys from VcXsrv already did this job. and we can use it.

[esaulenka]

runx now uses xauth.exe from VcXsrv if no native xauth is available,

Wow, thanks!

but there is a small error. You set Xauthbin to /c/Program Files/VcXsrv/xauth.exe, but space in it should be escaped.
So, I got (line numbers incorrect, I added some debug messages before it):

/usr/local/bin/runx: line 532: /c/Program: No such file or directory
/usr/local/bin/runx: line 534: /c/Program: No such file or directory
/usr/local/bin/runx: line 535: /c/Program: No such file or directory
/usr/local/bin/runx: line 536: /c/Program: No such file or directory

and I cannot understand, how you run an VcXsrv itself - this path also contain space...
UPD: $Vcxsrvexe have "raw" space, but Xcommand already have \ before it. I found, you already did an escapestring(), but forgot to use it for Xauthbin

UPD2: I just added Xauthbin="$(escapestring "$Xauthbin")" before creating wrapper (here

runx/runx

Line 511 in 2c69bfa

), but now it cannot find xauth.exe and therefore cannot create wrapper script.
Oh... What crazy man did this language? :-))
Yes, he is absolutely crazy, I cannot understand him.

$ [ -f "/c/Program\ Files/VcXsrv/xauth.exe" ] ; echo $?
1

$ [ -f "/c/Program Files/VcXsrv/xauth.exe" ] ; echo $?
0

$ [ -f /c/Program\ Files/VcXsrv/xauth.exe ] ; echo $?
0

$ [ -f /c/Program Files/VcXsrv/xauth.exe ] ; echo $?
C:\msys64\usr\bin\bash: [: /c/Program: binary operator expected
2

[mviereck]

but there is a small error. You set Xauthbin to /c/Program Files/VcXsrv/xauth.exe, but space in it should be escaped.
UPD: $Vcxsrvexe have "raw" space, but Xcommand already have \ before it. I found, you already did an escapestring(), but forgot to use it for Xauthbin

Good find! I've added quotes "$Xauthbin" to hopefully fix this. (It would need either escaping or quoting.)

Oh... What crazy man did this language? :-))
Yes, he is absolutely crazy, I cannot understand him.

And yet some other crazy man decided to add a space in folder Program Files. :-)
Originally, in old times of unix, pathes must not contain spaces. So shells were designed to split arguments at space. And the shells must keep backwards compatibility.
To cite man xauth:

Spaces are currently not allowed in the protocol name.  Quoting could be added  for  the truly perverse.

Yes, I see the same. Maybe we need to check WSL and CygWin ?

It would be great if you could check runx there, too. After that much code changes something might be broken accidently.
In Cygwin and WSL it is possible to install a native /usr/bin/xauth and I assume that ssh will be hardcoded for this. I don't think we need to add support for xauth.exe of VcXsrv there.

I know C good enough (my current job - to support ancient applications running on small embedded systems), but I think it will be better to write new one from scratch (and sure, it should be interpreted language like python or bash - you shouldn't force your user to install make/gcc/etc).
But guys from VcXsrv already did this job. and we can use it.

I agree. As long as xauth.exe from VcXsrv works fine, we don't need another xauth in MSYS2.
Years ago I had trouble with it, but now it seems to work well.
Btw, bash would be no good choice for a custom xauth implementation because it cannot handle nullbytes in a string, and it only knows strings. I already thought of implementing a rudimentary xauth myself, but that would require at least python. Since at least a year I want to start with python for another project, but I fear the beginner's hurdle.

and also it creates a wrapper script in /usr/X11R6/bin.

I've reverted this so runx does not create the script itself but prints a note how to set it up for ssh -X. I yet considered that secretly creating this file is no good practice.

  # describe xauth wrapper for 'ssh -X' in MSYS2, https://github.com/mviereck/runx/issues/7
  [ -e "$Xauthbin" ] && [ "$Winsubsystem" = "MSYS2" ] && [ ! -e /usr/X11R6/bin/xauth ] && note "If you want to use 'ssh -X' in MSYS2, you might need
  to create an executeable script /usr/X11R6/bin/xauth with the content:
#! /bin/bash
$(escapestring "$Xauthbin") \"$@\"
"

[esaulenka]

It would be great if you could check runx there, too.

OK, I will try to do it. I have WSL on my home laptop (but I never run linux GUI apps on it).

I want to start with python for another project

Wow! What different people one can meet on github! ;)

I yet considered that secretly creating this file is no good practice.

Yes, it is a good reason. BTW, here is a small bug: you should escape \"$@\" to \"\$\@\" when printing message.

Also, I think currently there is no need for cookiebaker() (and I am not sure that it provide correct cookies).

[mviereck]

you should escape "$@" to "$@" when printing message.

Good point, is fixed now.

Also, I think currently there is no need for cookiebaker()

But I am so proud of it and spent several hours to make it work ... * cry *. At least on Debian it works fine.
Maybe I'll leave it as a fallback; xauth might be missing in a Cygwin or WSL environment. (runx also supports Xwin from Cygwin to be used in Cygwin or WSL and prefers it over VcXsrv if available. In MSYS2 Xwin fails to start because the wrong cygwin.dll is used).

Wow! What different people one can meet on github! ;)

I quite appreciate our friendly and productive discussion across the world and across borders. :-)