Document use of current conda env

[thorbjornwolf]

• I have searched the issues of this repo and believe that this is not a duplicate.

Issue

Thank you all for your excellent work! And congratulations on cutting out a stable release! 🍾

My misunderstanding

Prior to reading the 1.0.0 release notes, my (wrong) mental model was that poetry always creates a new virtualenv for the project under consideration.
Now I've learned that it reuses the current virtualenv, if one is active, and that it now does the same for a current conda environment.

Current state

It is wonderful to see conda envs become a first-class citizen of poetry (#1432).
However, conspicuously, the PR's checkbox for "Updated documentation for changed code" remains unchecked 😁

And so, looking around in https://python-poetry.org/docs/, I fail to find mention of this behaviour. I guess I would put documentation in (at least) section "Managing environments". I would, however, also like to see it mentioned elsewhere.

Finally, when I am in an active conda environment and run poetry install, I see no indication that the current environment is being used/updated/modified. There is instead a lack of indication that a virtualenv is being created.

Desired state

This is a wishlist, for the consideration of the designers:

• I would like to see a command-line indication that the current (conda|virtual) env is being used when running commands relying on or modifying those. I think I prefer explicit state.
  • This is already the case for poetry shell, which helpfully says "Virtual environment already activated: "
• Documentation that "To achieve this, it will first check if it's currently running inside a virtual environment. If it is, it will use it directly without creating a new one." (from Managing environments) also counts for non-base conda environments

poetry debug info

Poetry
Version: 1.0.0
Python:  3.5.4

Virtualenv
Python:         3.5.4
Implementation: CPython
Path:           C:\Users\me\AppData\Local\Continuum\anaconda3\envs\py3_5_4
Valid:          True

System
Platform: win32
OS:       nt
Python:   C:\Users\me\AppData\Local\Continuum\anaconda3\envs\py3_5_4

Thank you for your efforts!

[dazza-codes]

I want to add to this - I would go so far as to revert this behavior to the poetry 0.12 behavior by default, where it creates a new virtualenv for the poetry project (it will be a derivative of the existing conda virtualenv that is already active, it should not pollute that conda env). To provide flexibility for folks who want it either way, add a config option to control how this behaves.

I have used conda to manage python versions and do so by creating a conda virtualenv for each python version on my system. I then activate a conda env with a given python version and then use poetry for a given project that needs a specific python version; then poetry 0.12 creates and manages a new virtualenv. The conda-envs are treated like base environments for spawning new virtualenvs. The new poetry 1.x behavior now pollutes the base conda-envs that manage python versions - arghhh.

What's the alternative way to use poetry 1.x without polluting an existing conda base-env? At present, it seems like poetry install has no option to demand that a new virtualenv is managed by poetry. So, it requires manually creating a new conda env with conda create -n {name} python={project-python-version}, using conda to activate it, and then calling poetry install. The poetry 0.12 behavior was easier than this.

Side note - I no longer trust pyenv shims, take your pick of issues about it

• https://github.com/pyenv/pyenv/issues?utf8=%E2%9C%93&q=is%3Aissue+shim+

[dazza-codes]

For other conda folks landing here who do not want poetry 1.x to pollute a base conda environment, I use this short bash function to spawn a new python 3.6 conda-env for my projects (based on the project root directory name, which is often the git-clone directory).

conda-venv () {
    # create and activate a conda environment with
    # the name of the current directory (often this
    # is a project name).  Default to python=3.6
    local py_ver="${1:-3.6}"
    wd=$(basename $(pwd))
    conda deactivate
    conda create -y -n "$wd" python="${py_ver}"
    conda activate "$wd"
}

For updates: https://github.com/dazza-codes/conda_container/blob/master/conda_venv.sh

I use it like this:

git clone [email protected]:sdispater/poetry.git
cd poetry
conda-venv
poetry install

It's not awesome by any means, but it helps.

[mforbes]

I strongly support @dazza-codes's suggestion to provide an option for poetry to not polute an active conda environment. I generally activate a work environment when I login, and am quite regularly frustrated when poetry corrupts this because I forgot to deactivate it before adding packages.

My current workflow is to create small conda environments with the various interpreters that I want, and symlink these somewhere accessible like ~/.local/bin/:

for py_ver in 2.7 3.6 3.7 3.8 3.9; do
   conda_env="py${py_ver}"
   conda create --strict-channel-priority -c defaults -y -n "${conda_env}" python="${py_ver}"
   conda activate "${conda_env}"
   ln -fs "$(type -p python${py_ver})" ~/.local/bin/
   conda deactivate
done

Then I can use poetry with poetry env use python3.8 etc. in each project.

$ cd project
$ poetry env info

Virtualenv
Python:         3.8.8
Implementation: CPython
Path:           /data/apps/conda/envs/work    # I don't want this corrupted!
...
$ poetry env use python3.8
$ poetry env info

Virtualenv
Python:         3.8.8
Implementation: CPython
Path:           /Users/mforbes/Library/Caches/pypoetry/virtualenvs/persist-ZlvMTnqX-py3.8
...

[rtbs-dev]

Just to add another voice here, I have migrated my dev team over to using poetry within conda by default.

We're a data-science/r&d team, so every project we build is usually accompanied with it's own env.yml file.
Any time we want to make a toolkit/package, we add a pyproject.toml, and add the relative package location to the env.yml (e.g. to use the mypackage code from within a parallel notebooks directory.
We make sure to activate the project-specific conda env (miniconda ftw) prior to running poetry commands. Poetry nicely notices conda's active environment and uses it for installing dependencies for the project.

Why do we do this?

1. due to proliferation of jupyter notebooks (whether for documentation of examples or reporting), we rely on jupytext and nb_conda_kernels. This is a great example of separation-of-concerns: conda's env.yml only needs to have an 'ipykernel' entry, and every running jupyter interface on our entire system can access/test the dev version of the package being made with poetry through the conda env it's being made in.
2. Our organization's internal gitlab runners have conda as the defacto env manager. So, we first tell the runner to activate/update the conda env, which gives us poetry, then use poetry to install the package for testing/building/docs. Here's a minimal example, mirrored on github. The alternative would require hard-installing poetry to the base runner, which also wouldn't work for e.g. apps using conda's nodejs for, say, react frontends.
3. While the plugin system isn't up yet, there are several plugin's that require "global" installation in the meantime (e.g. @mtkennerly makes poetry-dynamic-versioning. This would be a nightmare for ci/cd, except that poetry operating "inside" a conda environment believes that the plugin installation is global. This completely solved our issues with poetry+automatic git-tag-based semver.

So I suppose I view conda as a sufficiently general superset of what poetry is trying to achieve, so that poetry could happily work within conda when needed. Our use-case might be out-there, but having poetry automatically utilize the active conda environment has been a huge time-saver since the feature's introduction. Conda is a way more general environment manager, and is a widely adopted service in a ton of data science. Poetry is an elegant way to manage package creation, so having it work nicely inside conda makes sense from my perspective.

[dmcauslan]

It's also unclear how this should interact with the virtualenvs.in-project setting. My preference would be that a new virtual environment is always created in the project's directory when the setting is true.

In any case, better documentation around when virtual environments will (and will not) be created would be very helpful.

[joeld1]

I created a personal python package where I practice replacing bash scripts and improve my use of os.subprocess (actively being refactored and has been mainly tested on my Mac M1). Most of the actions are basically using conda and poetry together on a Mac M1.

In order to use a conda env as a poetry python interpreter I ensure the following local config settings are set forpoetry.toml .

Furthermore, before I execute poetry add or any other commands, I ensure that the conda env is activated.

Anyhow, please feel free to step through the code (apologies in advance for the lack of documentation and refactoring) and see if it fits your use-case!

Cheers!

Edit:
The workflow is as follows:

1. Create a conda env
2. Activate conda env before creating a new poetry project
3. Set the create a poetry.toml file to point to the conda env
4. Activate conda env before performing any poetry actions

[Green-li]

PDM can create a virtualenv in project directory, as I mentioned in: #4055 (comment)

[neersighted]

Duplicates #4055.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.