"Command not found" error when running a batch script with space in path with an argument with space

[IngwiePhoenix]

Hello!

I am going up and down the docs, but I can not seem to make this work:

[windows/amd64] Ingwie Phoenix@bigboi ~# code '"'$runtime:effective-rc-path'"'
Der Befehl "C:\Users\Ingwie" ist entweder falsch geschrieben oder
konnte nicht gefunden werden.
Exception: code exited with 1
  [tty 9]:1:1-37: code '"'$runtime:effective-rc-path'"'

My username has a space, so the path to my rc.elv has one, naturally. But how do I quote, or escape, a variable? I tried re:quote as well but that didn't help.

Any idea?

[IngwiePhoenix]

code (echo $runtime:effective-rc-path | to-string) worked, but I am sure there is an easier method than that...

[krader1961]

You don't need to quote strings passed to other programs; at least on a Unix OS. You last example is essentially a no-op. Compare the following:

elvish> put $runtime:effective-rc-path
▶ /Users/krader/.config/elvish/rc.elv
elvish> put '"'$runtime:effective-rc-path'"'
▶ '"/Users/krader/.config/elvish/rc.elv"'
elvish> put (echo $runtime:effective-rc-path | to-string)

The last example outputs nothing because your subcommand outputs nothing. The to-string doesn't read from either the byte stream (which echo writes to) or the value stream (which put writes to). You really want to-lines in your last example but that is just a complicated way to use the var expansion directly:

elvish> put (put $runtime:effective-rc-path | to-lines)
▶ /Users/krader/.config/elvish/rc.elv
elvish> put $runtime:effective-rc-path
▶ /Users/krader/.config/elvish/rc.elv

Why do you think you need to quote the string? I know that historically Windows is rather odd in that programs historically received a single string that included all arguments passed to the program. That is, Windows cmd.exe did not parse the arguments and pass them separately to the program. Cmd.exe instead passed everything after the command name as a single string to the program and it was the responsibility of the program to parse that string into individual arguments. Is that still true? If it is why does shells like Elvish and Cygwin/MSYS2 Bash not seem to have any problem passing arguments to external programs? I'm guessing it's because the code for launching external programs on Windows implicitly escapes individual arguments (according to the usual Windows conventions) and concatenates the resulting strings before launching the external program. Which would mean there isn't any need for you to explicitly quote $runtime:effective-rc-path.

[krader1961]

Expanding on a point in my previous comment, you wrote:

code (echo $runtime:effective-rc-path | to-string) worked

I'm not sure what you mean by "worked". The subcommand produces no output because to-string does not read from either the byte or value stream (meaning the output of echo was discarded):

elvish>  put (echo $runtime:effective-rc-path | to-string)

So you were invoking the code command with no arguments which only means that your code program does something reasonable when invoked with no arguments. 😄

[IngwiePhoenix]

In all of your examples, none of your paths had a space.

On Windows, my user name has one, thus, anything below my home folder, has a space in the path. Passing that to an external program - like VSCode in this case - means that I must quote the path. Otherwise, it won't work. That is why I was trying to find a solution.

My suspicion - and I could be wrong - is that by passing it through |to-string, the entire string, with space, was now "one string". That said, I am not entirely sure on the semantics of it either...

All I know is that if I used code $runtime:effective-rc-path and my path had a space, it wouldn't work. :)

[iandol]

Right this must be a windows things, I tried to pass a file with a path with a space in it to vs code and it works fine on macOS.

[iandol]

I assume the simple two quote characters doesn't work, i.e.

var y = '/path with/space'
code ''$y''

For to-string you need to pass a value, it isn't designed for pipes as Kurtis mentioned above. But this should work:

var y = '/path with/space'
code (to-string $y)

EDIT:

The output is different between echo and put — I think you were using echo but I think put and to-string give the same output:

> var y = 'Library/Application Support/Auctavo β/649844.padl' 
> echo $y
Library/Application Support/Auctavo β/649844.padl
> to-string $y
▶ 'Library/Application Support/Auctavo β/649844.padl'
> put $y
▶ 'Library/Application Support/Auctavo β/649844.padl'

[xiaq]

This is a Windows thing 😞 I'm actually a bit surprised that it took so long for someone to file an issue :)

The gist of the issue is that:

• In Win32 API, argv is a string, not a string array
• APIs that take argv as a string array must serialize it to a string in some way
• APIs that give you the process's argv string array must deserialize it in some way
• There's no guarantee that the serialization and deserialization are consistent, because they depend on the program

Here's an article that goes into more details: The wild west of Windows command line parsing

---

On the one hand, this is not an Elvish problem. On the other hand, Elvish as the shell should obviously do something to help users handle this mess - one way I imagine is some builtin commands that help you do the quoting for certain subclasses of programs, which can then be used to create wrappers that behave as expected in argument passing behavior, like this:

fn code {|args|
  e:code (windows:quote-args-for-quirky-programs-1 $args)
}

This assumes that programs can be sorted into a finite number of "quirk classes" (and there would be windows:quote-args-for-quirky-programs-2, windows:quote-args-for-quirky-programs-3, and so on). I don't know whether that's true or not.

[xiaq]

Hmm OK, this is more complicated than I thought actually. When you pass an argument with spaces to code, the error message is actually complaining that the command could not be found, and this error comes from Windows's API, not from VS Code.

So adding an argument with spaces somehow causes the parsing of the command path to misbehave 🤔

[IngwiePhoenix]

I had been under the assumption that, even on Windows, argv is a string array; then again, even there, I used the regular int main(int argc, char** argv) entrypoint - I never looked too much into winMain(...).

Further, while reading over the replies, I remembered something; code doesn't neccessarily mean $PATH/code - Windows' shell accounts for extensions: %PATHEXT% in the classic cmd.exe shell. So, when sending off code abc to the API, which code is actually ran?

I checked on my system:

>where code
C:\Users\ingwi\AppData\Local\Programs\Microsoft VS Code\bin\code
C:\Users\ingwi\AppData\Local\Programs\Microsoft VS Code\bin\code.cmd

And then inspected that:

PS Z:\Work\Homelab> scp "C:\Users\ingwi\AppData\Local\Programs\Microsoft VS Code\bin\code" riscboi:
code                                                                                                                                             100% 2001     9.8KB/s   00:00
PS Z:\Work\Homelab> ssh riscboi file code
code: a sh script, ASCII text executable

So, it would probably run code.bat since it can not handle a shell script. Next, let's look at what that bat file does:

@echo off
setlocal
set VSCODE_DEV=
set ELECTRON_RUN_AS_NODE=1
"%~dp0..\Code.exe" "%~dp0..\resources\app\out\cli.js" %*
IF %ERRORLEVEL% NEQ 0 EXIT /b %ERRORLEVEL%
endlocal

My batch-fu is rusty as heck, but from what I remember, %* expands everything into one large string.

But how come my example from above then does work? I tried to reproduce that by prepending the actual call to code.exe with an echo statement but that obviously didn't give me a good result. So, I just wrote a tiny printer.

package main

import (
    "fmt"
    "os"
)

func main() {
    fmt.Println(len(os.Args), os.Args)
}

(Basic build with go build -o printer.exe ./printer.go)

Next, the reproduction:

@echo off
setlocal
"%~dp0.\printer.exe" %*
IF %ERRORLEVEL% NEQ 0 EXIT /b %ERRORLEVEL%
endlocal

And finally, the call:

~\Work\elvish-call-test> ./caller.bat a b c
4 [C:\Users\ingwi\Work\elvish-call-test\.\printer.exe a b c]
~\Work\elvish-call-test> ./caller.bat "a b" c
3 [C:\Users\ingwi\Work\elvish-call-test\.\printer.exe a b c]

So, clearly, at least in this Go example, the space of "a b" is properly counted. But the other variable, the JS script invoked by Code.exe, I have largely ignored.

Hopefuly this can contribute to finding out what's up here. =)

[xiaq]

Hmm, I believe the error comes before the ... %* line in the bat file. It happens as Windows is trying to execute the bat file and if both the command path and an argument has spaces.

This is my reproduction:

~> cmd /c md 'C:\a b'
~> echo 'echo foobar' > 'C:\a b\foobar.bat'
~> set @paths = 'C:\a b' $@paths
~> foobar

C:\Users\xiaqq>echo foobar
foobar
~> foobar 'a b'
'C:\a' 不是内部或外部命令，也不是可运行的程序
或批处理文件。
Exception: foobar exited with 1
  [tty 45]:1:1-12: foobar 'a b'

The last invocation foobar 'a b' fails because somewhere the command name C:\a b\foobar.bat got broken into C:\a and b\foobar.bat, and it only happens when there's an argument that contains a space.

Both PowerShell and cmd deal with foobar "a b" fine.

Also interestingly, this doesn't happen with binary executables:

~> echo (slurp < print-args.go)
package main

import (
        "fmt"
        "os"
)

func main() {
        fmt.Println(len(os.Args), "arguments")
        for i, arg := range os.Args {
                fmt.Printf("#%d: %q\n", i, arg)
        }
}

~> go build -o 'C:\a b\print-args.exe' print-args.go
~> print-args 'a b'
2 arguments
#0: "C:\\a b\\print-args.exe"
#1: "a b"

To recap, the issue seems to be that when all the following are true:

• A command is a bat file (which could have a .cmd extension instead; Windows allows that)
• The full path has a space
• At least one argument has a space

When we try to run the command, something (Elvish? Go stdlib? Windows?) causes the command path to be split, and this results in a "command not found" error.

[xiaq]

Hmm OK, it's golang/go#17149

[xiaq]

The issue is a bit long, here's my summary of what's relevant for Elvish:

• The root of the issue is that when you call StartProcess on Windows to run a bat file, Windows actually turns that into a command that looks like C:\Windows\system32\cmd.exe /c .... Process Monitor is a very useful tool to observe this.
• What comes after /c goes through interpretation by cmd, which has its own rules.
• There are two known workarounds:
  • os/exec: Cannot execute command with space in the name on Windows, when there are parameters golang/go#17149 (comment) suggests adding an additional pair of quotes around the command
  • os/exec: Cannot execute command with space in the name on Windows, when there are parameters golang/go#17149 (comment) suggests passing a NULL pointer to the lpApplicationName parameter (the first parameter). From what I observed on procmon, this actually has exactly the same effect of surrounding the command with an additional pair of quotes.

For simplicity, you don't need to go through Go's API to reproduce the issue, you can observe it by running some commands from cmd itself [1]:

REM this OK
cmd /c "C:\a b\foobar.bat" foo
REM this is NOT OK for some reason
cmd /c "C:\a b\foobar.bat" "foo bar"
REM but this is OK
cmd /c ""C:\a b\foobar.bat" "foo bar""

I'm not quite sure how the additional pair of quotes work; it doesn't seem to be covered in https://daviddeley.com/autohotkey/parameters/parameters.htm#WINCMDRULES. But maybe I should read this page a few more times.

Another issue is that cmd.exe /c also interpretes characters like ^, |, <, and so on. This means that these characters also don't get passed to batch scripts unchanged. The two workarounds offered in the Go issue don't address that, but it seems to be something Elvish should try to do.

[1] It might not be obvious how this works. Although cmd does some processing of the command line, importantly it doesn't remove any quotes, so if you avoid special characters, your code is used verbatim as the command line to CreateProcess.

[krader1961]

Note that the Go issue has been open for more than seven years. Elvish should try to correctly handle this situation if doing so can be accomplished at little cost, but it is preferable to rely on the Go stdlib being updated to handle this scenario. In other words, given all of the other improvements to Elvish that could be implemented my preference is that we simply wait for the Go stdlib to be improved to handle this particular Windows quirk. The limited Elvish developer resources are better spent on issues that are unique to Elvish rather than working around shortcomings of the packages, including the Go stdlib, Elvish depends on if those shortcomings affect very few users of Elvish.

[xiaq]

The limited Elvish developer resources are better spent on issues that are unique to Elvish rather than working around shortcomings of the packages, including the Go stdlib, Elvish depends on if those shortcomings affect very few users of Elvish.

I appreciate the feedback, but it's up to me or whoever is interested in contributing to decide what to do.

[IngwiePhoenix]

So, to condense it down: Basically, if windows requires it's very own "shell escaping" to work around the implicit cmd.exe /c call?
Hm... o.o Interesting.
I'll triage around a little bit; maybe i can come up with something like a "wrapper" that can work around this issue. But it is quite obvious that some kind of workaround is needed.

Thanks for all the additional info. This is such an interesting and weird issue. x)

[xiaq]

So, to condense it down: Basically, if windows requires it's very own "shell escaping" to work around the implicit cmd.exe /c call? Hm... o.o Interesting.

Yeah.

I'll triage around a little bit; maybe i can come up with something like a "wrapper" that can work around this issue. But it is quite obvious that some kind of workaround is needed.

You probably can't fix this from "userland" and need to change how Elvish calls into os.StartProcess (in pkg/eval/external_cmd.go).

The two suggested workarounds in the Go issue both require modifying the Go stdlib so they are both non-starters. Another workaround suggested multiple times is to manually supply the CmdLine passed to Windows (golang/go#17149 (comment)). This works if you have a single fixed command, but for Elvish it will require a bit of duplication of the Go stdlib. It's not as bad as it may sound because the function for escaping a single argument is exported (https://pkg.go.dev/syscall?GOOS=windows#EscapeArg).

However, none of these actually solve the problem of escaping metacharacters like ^ and | - so you still can't easily pass say an argument foo^bar to a batch script and expect the batch script to see the ^ unchanged. To fix that one would need to implement a different escaping algorithm that also quotes the argument if it contains any metacharacter. I don't think Go's stdlib implements that anywhere; again https://daviddeley.com/autohotkey/parameters/parameters.htm#WINCMDRULES seems to be the best source we have on this topic.

Interestingly, although PowerShell handles spaces correctly, it doesn't handle metacharacters correctly. It'd be nice and a bit hilarious if Elvish can support Windows better than PowerShell (at least in this one aspect).