Cannot correctly modify environment variable PATH for use in task commands

[i-am-david-fernandez]

• Task version: 2.5.1
• OS: Windows 10 Professional (64-bit)
• Example Taskfile showing the issue:

# https://taskfile.dev

version: '2'

tasks:

  a:
    cmds:
      - echo "Task a"
      - echo $PATH

  b:
    cmds:
      - echo "Task b"
      - echo "{{.PATH}}"

  c:
    env:
      PATH: tools
    cmds:
      - echo "Task c"
      - echo $PATH

  d:
    env:
      PATH: tools
    cmds:
      - echo "Task d"
      - echo "{{.PATH}}"

  e:
    env:
      PATH:
        sh: echo "tools;$PATH"
    cmds:
      - echo "Task e"
      - echo $PATH

  f:
    env:
      PATH:
        sh: echo "./tools;$PATH"
    cmds:
      - echo "Task f"
      - echo $PATH

  g:
    env:
      PATH:
        sh: echo "$(realpath ./tools);$PATH"
    cmds:
      - echo "Task g"
      - echo $PATH

Hello. Firstly, thank-you for Task. It has, overall, made my development life easier and more pleasant.

I don't know if the problem I have is a bug or if I simply don't understand the correct way to achieve my goal. I'd like to be able to modify the environment variable PATH (either prepending or appending a project-specific directory to the existing value), but I can't figure out how to do this using the set of commands available within Taskfiles.

To provide context, my Taskfile is for use in building go applications. Some applications (projects) require project-specific build tools/helpers that I place in a top-level tools subdirectory (i.e., <project name>/tools). Note that my Taskfile resides at <project name>/Taskfile.yml. I wish to use the go generate command, which effectively requires that external generation tools be accessible via PATH, so I must add <project name>/tools to PATH before running go generate in the task.

While I'm working in Windows, I'm making effective use of busybox-w32 to provide a fairly comprehensive set of what are otherwise Linux tools, so things like cp, echo and realpath are available. I've also tried this via WSL such that things are more like a real Linux environment, with the same results.

The output I get from the above Taskfile (when run under sh or bash via WSL) is as follows (results in native Windows are the same):

$ task a; task b; task c; task d; task e; task f; task g                                             
echo "Task a"                                                                                        
Task a                                                                                               
echo $PATH                                                                                           
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games             
echo "Task b"                                                                                        
Task b                                                                                               
echo "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"      
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games             
echo "Task c"                                                                                        
Task c                                                                                               
echo $PATH                                                                                           
tools                                                                                                
echo "Task d"                                                                                        
Task d                                                                                               
echo "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"      
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games             
echo "Task e"                                                                                        
Task e                                                                                               
echo $PATH                                                                                           
tools;/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games       
echo "Task f"                                                                                        
Task f                                                                                               
echo $PATH                                                                                           
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games             
echo "Task g"                                                                                        
Task g                                                                                               
echo $PATH                                                                                           
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

Tasks a, b, c and d are just simple tests; it's tasks e, f and g that illustrate the real issue. Task e works correctly, prepending tools to the current PATH, but neither f nor g have any effect. Task g is the closest to what I need in terms of operations (i.e., adding the absolute path of a project-specific directory).

I'd greatly appreciate any input you could provide on this issue. Thanks in advance.

[smyrman]

This at least works on Mac OS X:

Assuming file structure:

• test/Taskfile.yml
• test/bin/hello-world

Content of test/Taskfile.yml:

version: "2"

env:
  PATH: "{{.PWD}}/bin:{{.PATH}}"

tasks:
  foo:
    cmds:
    - hello-world

Content of test/bin/hello-world:

#!/bin/sh
echo "Hello World!"

Running it:

$ task foo
hello-world
Hello World!

Using task version:

$ task --version                                                                                                        :(
Task version: 2.5.0

[i-am-david-fernandez]

No such luck for me (under WSL):

/mnt/d/Projects/Taskfiles/scratch/test0$ tree
.
├── Taskfile.yml
└── bin
    └── hello-world

1 directory, 2 files
/mnt/d/Projects/Taskfiles/scratch/test0$ cat bin/hello-world
#!/bin/sh
echo "Hello World!"
/mnt/d/Projects/Taskfiles/scratch/test0$ ./bin/hello-world
Hello World!
/mnt/d/Projects/Taskfiles/scratch/test0$ cat Taskfile.yml
version: "2"

env:
  PATH: "{{.PWD}}/bin:{{.PATH}}"

tasks:
  foo:
    cmds:
    - hello-world
/mnt/d/Projects/Taskfiles/scratch/test0$ task foo
hello-world
"hello-world": executable file not found in $PATH
task: Failed to run task "foo": exit status 127

[smyrman]

What if you add - echo $PATH to the task? is .PWD by any chance set to <no value>?

version: "2"

env:
  PATH: "{{.PWD}}/bin:{{.PATH}}"

tasks:
  foo:
    cmds:
    - echo $PATH
    - hello-world

[i-am-david-fernandez]

Still no luck (extended to demonstrate more clearly that PWD was ok):

/mnt/d/Projects/Taskfiles/scratch/test0$ cat Taskfile.yml
version: "2"

env:
  TOOLS: "{{.PWD}}/bin"
  PATH: "{{.PWD}}/bin:{{.PATH}}"

tasks:
  foo:
    cmds:
    - echo $PATH
    - hello-world

  bar:
    cmds:
    - echo $TOOLS

/mnt/d/Projects/Taskfiles/scratch/test0$ task foo
echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
hello-world
"hello-world": executable file not found in $PATH
task: Failed to run task "foo": exit status 127
/mnt/d/Projects/Taskfiles/scratch/test0$ task bar
echo $TOOLS
/mnt/d/Projects/Taskfiles/scratch/test0/bin
/mnt/d/Projects/Taskfiles/scratch/test0$

Note, however, that this seems ok with other environment variables:

/mnt/d/Projects/Taskfiles/scratch/test0$ cat Taskfile.yml
version: "2"

env:
  TOOLS: "{{.PWD}}/bin"
  PATH: "{{.PWD}}/bin:{{.PATH}}"
  BLAH: "{{.PWD}}/bin:{{.PATH}}"

tasks:
  foo:
    cmds:
    - echo $PATH
    - hello-world

  bar:
    cmds:
    - echo $TOOLS

  blah:
    cmds:
    - echo $BLAH
/mnt/d/Projects/Taskfiles/scratch/test0$ task foo
echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
hello-world
"hello-world": executable file not found in $PATH
task: Failed to run task "foo": exit status 127
/mnt/d/Projects/Taskfiles/scratch/test0$ task bar
echo $TOOLS
/mnt/d/Projects/Taskfiles/scratch/test0/bin
/mnt/d/Projects/Taskfiles/scratch/test0$ task blah
echo $BLAH
/mnt/d/Projects/Taskfiles/scratch/test0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
/mnt/d/Projects/Taskfiles/scratch/test0$

That is, PATH and BLAH are identically specified, but only BLAH is parsed/interpreted/expanded correctly.

[andreynering]

Hi @i-am-david-fernandez,

Can you verify if your PATH environment variable on Windows is using a different case (Path, path, etc)?

[i-am-david-fernandez]

Natively (i.e., in a Powershell or cmd terminal), it appears as Path. In WSL (via bash or fish shells), it appears as PATH. Also of note, perhaps, is that the native Windows separator is ; (i.e., semicolon), compared to : (colon) in WSL (which if I recall is the same as a native Linux environment. Sadly it's been a while since I lived in Linux 😞).

To be fair I hadn't noticed the difference until now; I'd been using PATH in my Taskfile without checking more closely (mostly just blindly following the documentation/examples where all uppercase is used for environment variables).

I'd like to highlight part of the behaviour shown in the initial post, repeated here:

e:
    env:
      PATH:
        sh: echo "tools;$PATH"
    cmds:
      - echo "Task e"
      - echo $PATH

  f:
    env:
      PATH:
        sh: echo "./tools;$PATH"
    cmds:
      - echo "Task f"
      - echo $PATH

Output (`WSL):

echo "Task e"                                                                                        
Task e                                                                                               
echo $PATH                                                                                           
tools;/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games       
echo "Task f"                                                                                        
Task f                                                                                               
echo $PATH                                                                                           
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games             

Output (via cmd, for comparison):

d:\Projects\Taskfiles\scratch
$ task e
echo "Task e"
Task e
echo $PATH
tools:C:\PortablePrograms\ConEmu\ConEmu\Scripts:C:\PortablePrograms\ConEmu:C:\PortablePrograms\ConEmu\ConEmu:C:\WINDOWS\system32:C:\WINDOWS:C:\WINDOWS\System32\Wbem:C:\WINDOWS\System32\WindowsPowerShell\v1.0\:C:\WINDOWS\System32\OpenSSH\:C:\Program Files\Git\cmd:C:\Program Files (x86)\HashiCorp\Vagrant\bin:C:\Users\AppData\Local\Microsoft\WindowsApps:C:\Users\AppData\Local\Programs\Microsoft VS Code\bin:C:\Users\AppData\Local\Programs\Fiddler:C:\PortablePrograms\bin:C:\PortablePrograms\Busybox

d:\Projects\Taskfiles\scratch
$ task f
echo "Task f"
Task f
echo $PATH
C:\PortablePrograms\ConEmu\ConEmu\Scripts:C:\PortablePrograms\ConEmu:C:\PortablePrograms\ConEmu\ConEmu:C:\WINDOWS\system32:C:\WINDOWS:C:\WINDOWS\System32\Wbem:C:\WINDOWS\System32\WindowsPowerShell\v1.0\:C:\WINDOWS\System32\OpenSSH\:C:\Program Files\Git\cmd:C:\Program Files (x86)\HashiCorp\Vagrant\bin:C:\Users\AppData\Local\Microsoft\WindowsApps:C:\Users\AppData\Local\Programs\Microsoft VS Code\bin:C:\Users\AppData\Local\Programs\Fiddler:C:\PortablePrograms\bin:C:\PortablePrograms\Busybox

The only difference between these two tasks is the leading ./ in the fixed string that is being prepended to the existing PATH environmental variable. Without ./, the variable is correctly modified. With it, no change occurs.

These results suggest to me that, fundamentally, the environment variable is correctly being accessed, at least in terms of using its existing value initially.

I've just noticed that something funny is going on with regards to the separator in task e (the one that works). When run via cmd (which, as per my setup, potentially uses busybox-w32), despite the task specifying a semicolon separator, the output is changed to a colon. In WSL, however, this does not occur; the separator is as specified. It doesn't appear to be busybox-w32, as if I run echo "tools;$PATH" in the busybox-w32 shell, the output is as expected.

I've just looked more closely, and the output of task a (simply echoing PATH) shows the changed separator (i.e., the output of the task has colons instead of semicolons). I don't think this is busybox-w32 as the behaviour is the same when I effectively disable it.

Is something in mvdan/sh at play here?

[andreynering]

@i-am-david-fernandez Sorry, I got out of time in last days to help you with this.

I'll catch up once possible to try to understand the problem here.

[smyrman]

When I googled this earlier, I did find that WSL "inherits" the PATH from Windows (can be turned off by a register value apparently, but that sounds messy).

From what you are describing, it can look like it's "validating" the paths when doing this. It may look like it doesn't like paths starting with ./ (obvious local paths), or that it removes the semicolon separated path component for WSL maybe?

Probably it's a good idea to add only global paths to PATH. Does it work better if you prepend the result of "${PWD}/tools" or "$(pwd)/tools"?

Otherwise related to the colon v.s. semicolon, what is the output of a task with"echo {{OS}}" in Powershell v.s. WSL. Is it by any chance "linux" for WSL?

If it is, perhaps you can utilize the Taskfile_{{GOOS}}.yml feature to define the PATH with/without semicolon for WSL / Linux v.s. Powershell / Windows?

[i-am-david-fernandez]

Otherwise related to the colon v.s. semicolon, what is the output of a task with"echo {{OS}}" in Powershell v.s. WSL. Is it by any chance "linux" for WSL?

If it is, perhaps you can utilize the Taskfile_{{GOOS}}.yml feature to define the PATH with/without semicolon for WSL / Linux v.s. Powershell / Windows?

You are correct. The output is windows in Powershell and linux in WSL. Until (unless) task gets conditionals (which is feature I would love but is for another thread), using the OS-specific Taskfile is probably the simplest solution to the colon/semi-colon difference.

[i-am-david-fernandez]

I should note that I've been providing output produced via WSL under the assumption that its closeness to Linux makes it more useful from a debugging perspective. My working environment is Powershell in native Windows and that's where I'm most interested in having this work.

I apologise if the WSL stuff has become a distraction.

Here's the output from Powershell of the first Taskfile I posted (the opening post in this thread):

echo "Task a"
Task a
echo $PATH
C:\PortablePrograms\ConEmu\ConEmu\Scripts:C:\PortablePrograms\ConEmu:C:\PortablePrograms\ConEmu\ConEmu:C:\WINDOWS\system32:C:\WINDOWS:C:\WINDOWS\System32\Wbem:C:\WINDOWS\System32\WindowsPowerShell\v1.0\:C:\WINDOWS\System32\OpenSSH\:C:\Program Files\Git\cmd:C:\Program Files (x86)\HashiCorp\Vagrant\bin:C:\Users\AppData\Local\Microsoft\WindowsApps:C:\Users\AppData\Local\Programs\Microsoft VS Code\bin:C:\Users\AppData\Local\Programs\Fiddler:C:\PortablePrograms\bin:C:\PortablePrograms\Busybox
echo "Task b"
Task b
echo "<no value>"
<no value>
echo "Task c"
Task c
echo $PATH
tools
echo "Task d"
Task d
echo "<no value>"
<no value>
echo "Task e"
Task e
echo $PATH
tools:C:\PortablePrograms\ConEmu\ConEmu\Scripts:C:\PortablePrograms\ConEmu:C:\PortablePrograms\ConEmu\ConEmu:C:\WINDOWS\system32:C:\WINDOWS:C:\WINDOWS\System32\Wbem:C:\WINDOWS\System32\WindowsPowerShell\v1.0\:C:\WINDOWS\System32\OpenSSH\:C:\Program Files\Git\cmd:C:\Program Files (x86)\HashiCorp\Vagrant\bin:C:\Users\AppData\Local\Microsoft\WindowsApps:C:\Users\AppData\Local\Programs\Microsoft VS Code\bin:C:\Users\AppData\Local\Programs\Fiddler:C:\PortablePrograms\bin:C:\PortablePrograms\Busybox
echo "Task f"
Task f
echo $PATH
C:\PortablePrograms\ConEmu\ConEmu\Scripts:C:\PortablePrograms\ConEmu:C:\PortablePrograms\ConEmu\ConEmu:C:\WINDOWS\system32:C:\WINDOWS:C:\WINDOWS\System32\Wbem:C:\WINDOWS\System32\WindowsPowerShell\v1.0\:C:\WINDOWS\System32\OpenSSH\:C:\Program Files\Git\cmd:C:\Program Files (x86)\HashiCorp\Vagrant\bin:C:\Users\AppData\Local\Microsoft\WindowsApps:C:\Users\AppData\Local\Programs\Microsoft VS Code\bin:C:\Users\AppData\Local\Programs\Fiddler:C:\PortablePrograms\bin:C:\PortablePrograms\Busybox
echo "Task g"
Task g
echo $PATH
D:/Projects/Taskfiles/scratch/tools:C:\PortablePrograms\ConEmu\ConEmu\Scripts:C:\PortablePrograms\ConEmu:C:\PortablePrograms\ConEmu\ConEmu:C:\WINDOWS\system32:C:\WINDOWS:C:\WINDOWS\System32\Wbem:C:\WINDOWS\System32\WindowsPowerShell\v1.0\:C:\WINDOWS\System32\OpenSSH\:C:\Program Files\Git\cmd:C:\Program Files (x86)\HashiCorp\Vagrant\bin:C:\Users\AppData\Local\Microsoft\WindowsApps:C:\Users\AppData\Local\Programs\Microsoft VS Code\bin:C:\Users\AppData\Local\Programs\Fiddler:C:\PortablePrograms\bin:C:\PortablePrograms\Busybox

I clearly missed that task g partially works: the PATH as displayed has stuff prepended, but, whether because slashes are wrong or semi-colons have been swapped with colons, the end result isn't correct (I tried running something from that prepended path via the task and it fails as it can't find it in PATH).

For ease of reference, here is task g:

  g:
    env:
      PATH:
        sh: echo "$(realpath ./tools);$PATH"
    cmds:
      - echo "Task g"
      - echo $PATH

Changing the method of specifying the PATH from echo "$(realpath ./tools);$PATH" to sh: echo "$(pwd)\tools;$PATH" resolves the slash issue:

echo "Task g"
Task g
echo $PATH
D:\Projects\Taskfiles\scratch\tools:C:\PortablePrograms\ConEmu\ConEmu\Scripts:C:\PortablePrograms\ConEmu:C:\PortablePrograms\ConEmu\ConEmu:C:\WINDOWS\system32:C:\WINDOWS:C:\WINDOWS\System32\Wbem:C:\WINDOWS\System32\WindowsPowerShell\v1.0\:C:\WINDOWS\System32\OpenSSH\:C:\Program Files\Git\cmd:C:\Program Files (x86)\HashiCorp\Vagrant\bin:C:\Users\AppData\Local\Microsoft\WindowsApps:C:\Users\AppData\Local\Programs\Microsoft VS Code\bin:C:\Users\AppData\Local\Programs\Fiddler:C:\PortablePrograms\bin:C:\PortablePrograms\Busybox

But, perhaps because of the colons, still doesn't ultimately work.

[smyrman]

I think the information you have provided @i-am-david-fernandez, has been very clear and useful, and it's nice that you tried both WSL and PowerShell and highlighted the difference.

Obviously the colon replacement of PATH is problematic; If it where to be a converted PATH (to Liunx style), it would need to also replace or at least escape the Windows drove letters...

@mvdan, Do you have any input on how mvdan/sh deals with list separators and paths in general when run natively in Windows?

[andreynering]

@i-am-david-fernandez Sorry, I'll take long to debug this since my available time has being low recently, and I don't use Windows often nowadays. I do plan to debug and fix this, though.

---

Just a friendly reminder that @mvdan has no interest in Task, and likely won't digest a long thread like this to understand the problem. If we think we found a problem on mvdan/sh we should try to reproducing it in a small Go script (importing the sh's packages directly) and open a more specific issue there.

[andreynering]

Hi @i-am-david-fernandez,

I just did a lot of debugging on this bug on Windows, but I couldn't find the source of the problem. This doesn't seem trivial to fix, specially on Windows which handles some stuff differently than Linux/macOS.

My advice is to avoid overriding $PATH and just call the right path directly.

[i-am-david-fernandez]

@andreynering Thanks for the effort you've put into this, it's certainly appreciated.

Unfortunately, as best as I understand, the go generate framework doesn't allow for providing an executable path (though I'll dig deeper), so it seems I won't be able to use Task as broadly or elegantly as I'd have liked. That said, perhaps I can hack something with an intermediate wrapper script of some sort.

Again, thanks for the time and effort.

[andreynering]

OK @i-am-david-fernandez,

I'm going to close this issue then (this doesn't seem to be a common use case). But feel free to add more comments or open other issues if needed, though.

[twmb]

I'm running into this same thing, but on Linux. My use case: I'd like to append a repo-specific build path to PATH; if binaries exist on the host (local dev setup), then we use the binaries in the default PATH. If they do not exist on the host, we use the binaries in the repo-specific build path (which are installed through task). If binaries exist in that build path, I'd like to avoid re-downloading the binaries. This allows for easier command testing, and a more flexible local setup.

I've tried the (e) iteration at the top of this issue, but I can't get it to work for me. echo $PATH echoes the path I invoke task with.

[brutus]

Came here via Google, so for reference: Same here, running Linux (Fedora 37 wit go-task installed via DNF - reported version is "unknown", but RPM says 3.20.0). Installing v3.21.0 from releases worked though.

Or not :/ might have myt test cases mixed up, sorry…

version: '3'

vars:
  PYTHON_VERSION:
    sh: cat runtime.txt
  PATH:
    sh: echo $PATH
  PYTHON_PATH: '/usr/bin/python{{ trim .PYTHON_VERSION }}'
  VENV_PATH: '{{.PWD}}/.venv'
  NEW_PATH: "{{.VENV_PATH}}/bin:{{.PATH}}"


env:
  PATH: "{{.NEW_PATH}}"
  NEW_PATH: "{{.NEW_PATH}}"

tasks:
  py:
    cmds:
      - echo "$NEW_PATH"
      - echo "$PATH"
      - export PATH="$NEW_PATH"
      - echo "$PATH"
      - PATH="$NEW_PATH/bin:$PATH" echo $PATH
      - echo "({{.PYTHON_VERSION}}) {{.PYTHON_PATH}}"

% go-task py        
task: [py] echo "$NEW_PATH"
/tmp/test/.venv/bin:/usr/share/Modules/bin:/home/…
task: [py] echo "$PATH"
/usr/share/Modules/bin:/home/…
task: [py] export PATH="$NEW_PATH"
task: [py] echo "$PATH"
/usr/share/Modules/bin:/home/…
task: [py] PATH="$NEW_PATH/bin:$PATH" echo $PATH
/usr/share/Modules/bin:/home/…
task: [py] echo "(3.11) /usr/bin/python3.11"
(3.11) /usr/bin/python3.11

Probably b/c #482

[MatteoCalabro-TomTom]

For whoever lands here from searching, this behavior is (most likely) caused by #1038