${workspaceFolder} variable not replaced when debugging Python

[DonJayamanne]

Issue Type

Bug

This used to work in the previous version.

microsoft/vscode-python#691
#42787

Description

• The variable ${workspaceFolder} is no longer getting replaced with the valid value when debugging.

Steps to reproduce:

• Create a workspace and add any python file e.g. one.py with a print statement print(1)
• Create a virtual environment in this workspace using virtualenv as follows:
  virtualenv .env
  (instructions to install virtualenv can be found here)
• Reload VS Code and use the command Python: Select Interpreter (or click on the bottom status bar to bring up the list)
• From the list select the environment .env/bin/python
• The workspace settings should now have the following data:

{
	"python.pythonPath": "${workspaceFolder}/.env/bin/python"
}

• Attempt to debug the pyton file (create a launch.json if necessary) as follows:

{
	"version": "0.2.0",
	"configurations": [
		{
			"name": "Python: Current File",
			"type": "python",
			"request": "launch",
			"stopOnEntry": true,
			"pythonPath": "${config:python.pythonPath}",
			"program": "${file}",
			"cwd": "${workspaceFolder}",
			"env": {},
			"envFile": "${workspaceFolder}/.env",
			"debugOptions": [
				"RedirectOutput"
			]
		}
	]
}

• Debug the python file and it will complain about the path not being found.
• The problem is the variable ${workspaceFolder} hasn't been replaced when debugging (notice how the setting in settings.json is referenced in launch.json).

VS Code Info

VS Code version: Code 1.20.0 (c63189d, 2018-02-07T17:02:34.244Z)
OS version: Darwin x64 17.4.0

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz (8 x 2900)
Load (avg)	3, 3, 3
Memory (System)	16.00GB (0.17GB free)
Process Argv	/Applications/Visual Studio Code.app/Contents/MacOS/Electron
Screen Reader	no
VM	20%

Extensions (20)

Extension	Author (truncated)	Version
MagicPython	mag	1.0.12
python	ms-	2018.2.0-alpha

Reproduces only with extensions

[agelter]

Thank you for reporting this! I thought I was crazy, wondering what I'd accidentally changed somewhere to make things stop working.

+1 to ${workspaceFolder} disappearing when debugging

[DonJayamanne]

@agelter for the moment, please modify the setting python.pythonPath to contain the fully qualified path.

[shaofuzhang]

I met the same problem.
Modify the pythonPath property of the launch.json file, as shown below：
"pythonPath": "${workspaceFolder}/env/bin/python"
It works for me.

[pavlov-v]

same problem
Error: spawn ${workspaceFolder}.venv\Scripts\python.exe ENOENT

[bergmannf]

It seems to not be limited to the {workspaceFolder} variable.

I tried setting the path using ${env:HOME}, and it did not replace that variable either.

[cropr]

Same problem, have to hardcode the pythonPath setting with an absolute path to make it work. No issue in 1.19

[MightyGorilla]

I just updated to 1.20.0 and hit the same problem, but in case this helps at all, I had a similar experience as @shaofuzhang

It was not finding python when pythonPath was set to "${config:pythonPath}" in launch.json
and was fixed when I set pythonPath to "${workspaceFolder}/venv/bin/python", even though settings.json contains a python.pythonPath of "${workspaceFolder}/venv/bin/python".

Seemed more like global settings were what was not coming across.

[weinand]

Since variables like ${workspaceFolder} are not substituted in setting files, the value of python.pythonPath is "${workspaceFolder}/.env/bin/python".

If the setting python.pythonPath is used in a launch configuration like this:

"pythonPath": "${config:python.pythonPath}",

the value of pythonPath becomes "${workspaceFolder}/.env/bin/python".

In order to resolve this, another substitution pass is necessary.

Since you are seeing a regression here, it looks like VS Code did more than one substitution pass previously and in the latest version the second pass is now missing.

So not the general variable substitution is broken, but the multi-pass substitution behavior.

I don't like this (non documented) multi-pass behaviour, but if we did it in previous versions we have to continue doing it.

@isidorn Is this analysis correct?

[joshsleeper]

@weinand That's how I interpret the issue, yes.

Frankly, I'm not the biggest fan of the double-sub pass behavior either.

Any way you can think of that allows for scoping python.pythonPath to workspaceFolder without the double-sub? I think that's the important part, not the double-sub pass itself.

[weinand]

@DonJayamanne why do you require double pass substitution?
It is a bit confusing to see variables in the setting files that are not valid there, but get resolved later in the context where they are used. And if a user would use this pattern somewhere else, it might fail because double pass substitution is only available in launch.json and task.json.

I suggest that we restore the previous behaviour for this milestone but phase it out in the future.

[DonJayamanne]

It is a bit confusing to see variables in the setting files that are not valid there, but get resolved later in the context where they are used.

These are totally valid.
The value in the workspace setting is "python.pythonPath":"${workspaceFolder}/env/pyhton"
This setting points to the executable that is relative to the current workspace folder. This setting is used by the extension (for intellisense, and in other parts of the extension) as well as the debugger.

When debugging, the purpose of having "pythonPath": "${config:python.pythonPath}", is so that the user doesn't have to type the same thing in two different places. Previous versions users had to type this out in two places and was very error prone, then we introduced the above referencing mechanism.

I suggest that we restore the previous behaviour for this milestone but phase it out in the future.

I fail to understand the logic behind this.
The whole point of being able to reference workspace settings from within launch.json is lost.

If it's to be phased out do we have a solution that we could implement to work around this?

I can think of a long-winded solution (debugger passing custom messages and these getting intercepted by the extension). But then it gets complicated with multi-root workspaces.

[weinand]

Python is using variable syntax in the settings file, but it is relying on the fact that there is currently no general variable substitution available for settings. As a lucky consequence the variable syntax appears unmodified in the settings value.

If VS Code would add variable substitution for settings in the future, use of ${workspaceFolder} would be immediately invalid because a 'workspaceFolder' does not exist.
A better (because explicit) way of preserving the ${workspaceFolder} syntax in a value would be to escape the variable syntax.

The whole point of being able to reference workspace settings from within launch.json is lost.

No, referencing workspace settings from within launch.json is not lost.
What is lost is multi pass substitution.

In your case you are not only referencing a setting from the launch config (which is perfectly fine), but you expect that in the resulting value variables are substituted another time (and if this substitution results in more variables, then they are substituted again... ad infinitum).

I think we have two (future) options:

• eliminate support for multi pass substitution and find an alternative solution for the Python problem (I'm not aware of any other extension relying on this).
• implement multi pass substitution fully and correctly and document it.

[DonJayamanne]

, but it is relying on the fact that there is currently no general variable substitution available for settings.

Yes you're right, I have my own code in the extension to perform this substitution.

[DonJayamanne]

eliminate support for multi pass substitution and find an alternative solution for the Python problem (I'm not aware of any other extension relying on this).

If the Python extension is the only one using this capability, then I guess its up-to us (Python extension) to fix this at our end.

The solution I propose is simple, the adapter makes a request to the extension layer (via a custom event) to perform the translation. The extension replies back via a custom request message.

Note elegant, however based on the current API I don't see any other way around this. Feels wrong for the debug adapter to send requests to the extension in this manner, through an event and for the extension to send responses through requests!!

Unfortunately the above solution will not work in multi-root workspaces.
In a multi-root workspace, we'd need to know the workspace that the debug session is related to.
Currently I don't see this being provided in the API.

[isidorn]

@weinand yes your analysis is correct. We used to perform multi pass substitution. Since this was nowhere documented and implemented in a strange way I decided to remove it when refactoring the configurationResolverService. I was not aware that anyone is reliying on this. It was my mistake that I did not think of the python extension.
Bringing this back is very risky and I am definetley against that.

I propose the following fix. In the python extension you can implement resolving configurations via this API. That basically means that before each debug start your extension will get to additionaly "massage" a configration. You will also get passed a WorkspaceFolder so you can substitute it any way you want.
Example how node extension is doing that can be found here
More about DebugConfigurationProviders can be found in our docs
Let us know if you do not have cycles and we can contribute a PR to your extension that fixes this.

Closing this as there is no action needed on the vscode side.

@DonJayamanne for the future it would be great if you self host on vscode insiders so we catch changes like this earlier. This was pushed 3 weeks ago.

[weinand]

@isidorn Yep, that's the right approach.

@DonJayamanne in the Python extension's DebugConfigurationProvider.resolveDebugConfiguration you can just do this:

resolveDebugConfiguration(folder: WorkspaceFolder | undefined, config: DebugConfiguration, ...): ProviderResult<DebugConfiguration> {

	// if 'pythonPath' attribute is missing
	if (!config.pythonPath) {
		config.pythonPath = ... // get pythonPath value from python settings
	}

	return config;
}

In this case the config.pythonPath is allowed to contain variables: VS Code substitutes them upon return from resolveDebugConfiguration.

In general we recommend to simplify launch configs by making most attributes optional and filling in missing values in the resolveDebugConfiguration (which is simple because in contrast to the DA the extension has full access to the VS Code API).

[DonJayamanne]

Thanks guys, will make the necessary fix at our end.

@DonJayamanne for the future it would be great if you self host on vscode insiders so we catch changes like this earlier. This was pushed 3 weeks ago.

We did test and almost always test on the insiders, unfortunately it so happened that we weren't using relative paths (python installed somewhere in the workspace Folder) when testing.

In general we recommend to simplify launch configs by making most attributes optional and filling in missing values in the

Agreed

[DonJayamanne]

Let us know if you do not have cycles and we can contribute a PR to your extension that fixes this.

Thanks, I think we can manage.

[nbara]

Hi, I used to have this in my User settings (because I always use the same launch settings for all my projects/workspaces, so I don't want to have to copy and paste a launch.json file in all of them)

This used to work fine until 1.20.

    "launch": {
        "version": "0.2.0",
        "configurations": [
            {
                "name": "Python3",
                "type": "python",
                "request": "launch",
                "stopOnEntry": false,
                "pythonPath": "/Users/nicolas/anaconda3/envs/py3/bin/python",
                "program": "${file}",
                "cwd": "${workspaceFolder}",
                "env": {},
                "envFile": "${workspaceFolder}/.env",
                "debugOptions": [
                    "RedirectOutput"
                ]
            },
            {
                "name": "Python2",
                "type": "python",
                "request": "launch",
                "stopOnEntry": false,
                "pythonPath": "/Users/nicolas/anaconda3/envs/py2/bin/python",
                "program": "${file}",
                "cwd": "${workspaceFolder}",
                "env": {},
                "envFile": "${workspaceFolder}/.env",
                "debugOptions": [
                    "RedirectOutput"
                ]
            }
        ]
    }
}

But it doesn't work anymore. It seems like it's related to the issues discussed here but even specifying the full path to my python executable doesn't work

Is there a way to specify global python launch configurations in VSCode 1.20 ?

[breathe]

Will this issue be updated when the fix is released? In my case I use {env:HOME} in workspace settings as a low rent way to enforce that developers on my team are installing anaconda locally on their machine and following the proscribed development environment setup process — and this is currently broken ...

[DonJayamanne]

@breathe
This issue has been closed as the problem lies with the extension.
Please check here microsoft/vscode-python#691 (comment)

[sandy081]

@isidorn Anything to verify here?

[isidorn]

@sandy081 this is an issue with the extension, removed the bug label