Support for .env files

[ptink]

• I have searched the issues of this repo and believe that this is not a duplicate.

Issue

First off I really like poetry- I'd actually only recently migrated over to pipenv but I already see the advantages of poetry and would ideally like to use it full time.

I primarily work with Django and other web frameworks that can require a large amount of configuration which- when following 12 factor app practices- involves lots of environment variables that necessitates the use of a .env file.

Is there a possibility of adding in .env file support, specifically for poetry's run command so that the resulting command runs both in the virtualenv and with the environment as configured in the .env file?

[sdispater]

I don't think this is in the scope of Poetry.

Since Poetry is not configured via environment variables there are no reasons for it to load a .env file.

If the libraries you use need it, it's their responsability to load it not Poetry's.

[jacebrowning]

I've always liked using direnv for this. It supports loading .env files automatically.

[ptink]

@jacebrowning direnv isn't cross platform unfortunately. When the environments I worked in were solely on Linux I used direnv in conjunction with vex to achieve functionality similar to pipenv run <command> for example. This workflow doesn't work on Windows which was a major reason for my discovery and subsequent switch to pipenv when I realised it could do the same thing.

I agree that poetry doesn't need .env files for it's own configuration (although it's not a bad idea to use them to allow for directory-specific configuration that may differ from the globals defined in the config file) but I don't think it's out of scope for poetry to load .env files for run and shell when you consider that they are both attempts to isolate the given command in an environment that is specific to the current project.

I understand the concerns about scope creep, especially when the PR adds a new dependency. If you're exclusively writing libraries then this particular use case may not occur very often but for 12 factor apps it's pretty vital to have tooling to support it- and it always seemed like a neat solution to me to have the virtualenvironment be in control of it.

[AlJohri]

I've used https://github.com/jezdez/envdir/ for something similar.

it uses an envs folder instead of a .env file.

https://github.com/toddbluhm/env-cmd is also good if you want to stick to the .env file

it should be simple to use an alias that chains the two commands together

[ptink]

Again, I'm aware other external libraries exist that do this but I'm of the belief that this particular functionality would be useful to have included in Poetry itself. It's a vital part of tooling required for 12 factor apps and it would be helpful to people who are moving towards adopting an environment variables-based approach in their project to have support for it in the tooling they (ideally) are already using for their project. The goal being to reduce the barrier to entry for managing python projects so that we don't end up back in the "oh just install pip and virtualenv and virtualenvwrapper, easy!" mess that we're finally moving away from :)

If @sdispater would prefer that this isn't included within the project then my next solution would be to modify the entry-point to my Django applications so that they read the required env vars using python-dotenv. That's obviously less ideal but still better than relying on more external libraries that drop in and out of maintanence.

[Pluckerpluck]

I'd like to voice my support of this. The ability to automatically load .env files when running poetry run or poetry shell makes a lot of sense to me. (There's a reason pipenv does it)

Environmental variables are generally designed to be read from the shell (i.e. not a file) and so it makes sense for this to be within the scope of poetry, given that it creates a sub shell for isolation.

In short: Environmental variables should be part of Poetry's virtual environment.

[pmav99]

What's the problem with having direnv or aactivator or whatever handle this? To be honest, I also use them to handle the virtualenv creation/activation but that's purely optional. I prefer those dedicated projects, because I can also use them on non-python projects (e.g. to set up credentials for databases, environmental variables etc).

I can understand that it could be convenient to have this implemented in poetry itself, but it feels like it is kind of out of scope + there are projects that do exactly this thing and IMHO quite well so.

[pawamoy]

Could be a candidate for a plugin once they are a thing.

[Pluckerpluck]

@pmav99 Me being on Windows is really the biggest problem here.

@pawamoy As long as plugins could work with this type of thing (likely, but never guaranteed) I'd be fine with that.

[pmav99]

The fact that setting up an environment on Widows is hard does not answer why setting up the environment becomes poetry's responsibility though.

[Pluckerpluck]

@pmav99 I've already stated why I think setting up the environment in poetry's virtual environment makes sense. Poetry uses virtual environments to provide isolation. That's the point of virtual environments. How does it not makes sense that within that environment, environmental variables could be set?

This is particularly true for the poetry shell command which literally spawns a sub shell for you to access directly. For example, I regularly use poetry shell followed by setting FLASK_ENV to development in order to run my HTTP endpoints locally while debugging.

You simply stated why I couldn't use other pieces of software, and I answered. Windows is not supported by them. Poetry, on the other hand, claims to support Windows.

[ptink]

If the upcoming plugin system has hooks for injecting env vars then I agree that this would be an ideal candidate for a plugin

[ekhaydarov]

oh boy, coming from pipenv, which has an array of issues, poetry seems to have issues which pipenv has solved. why dont they just marry already

[sdispater]

Like I said above this is beyond the scope of Poetry so I am closing this issue.

[marcosfelt]

So, I know this issue is closed, but can anyone give instructions on how to set the environmental variables on a poetry run command?

[orn688]

I use a custom shell function, so I can do withenv <env-file-name> <command-to-run>. This spawns a subshell, sources the .env file into that subshell, and then runs the command in the subshell.

For example, withenv .env poetry run python script.py.

Bash:

withenv () {
  env_file="$1"
  cmd="${@:2}"
  bash -c "source $env_file && $cmd"
}

Fish:

# Source: http://lewandowski.io/2016/10/fish-env/
function posix-source -d "Source a POSIX-formatted .env file"
  for i in (cat $argv)
    set arr (echo $i |tr = \n)
    set -gx $arr[1] $arr[2]
  end
end

function withenv -d "Run a command in a subshell with a .env file sourced"
  set env_file $argv[1]
  set cmd $argv[2..-1]
  fish -c "posix-source $env_file; and $cmd"
end

[Keda87]

For example, withenv .env poetry run python script.py.

Bash:

withenv () {
  env_file="$1"
  cmd="${@:2}"
  bash -c "source $env_file && $cmd"
}

Fish:

# Source: http://lewandowski.io/2016/10/fish-env/
function posix-source -d "Source a POSIX-formatted .env file"
  for i in (cat $argv)
    set arr (echo $i |tr = \n)
    set -gx $arr[1] $arr[2]
  end
end

function withenv -d "Run a command in a subshell with a .env file sourced"
  set env_file $argv[1]
  set cmd $argv[2..-1]
  fish -c "posix-source $env_file; and $cmd"
end

Hi @orn688 is it compatible with zsh ?

[orn688]

Hi @orn688 is it compatible with zsh ?

It should be, just substitute zsh -c for bash -c from the bash version.

[golyalpha]

I feel like the community is in strong disagreement about this issue...

[tumido]

I'll also voice my meaningless disagreement with this decision. I came to poetry from pipenv because I wanted to simplify the toolkit I use - to spend less time maintaining and more time creating. This issue goes against the flow and poetry actually creates a problem that was already solved with pipenv.

To paraphrase @sdispater:
Now we can't have single tool to manage my Python projects from start to finish. You wanted something reliable and intuitive that the community could use and enjoy. Sadly I'm not enjoying this anymore. And it's not intuitive anymore.

This issue was not closed because of a technical reason.

[rajasimon]

Coming from pipenv and uninstall poetry immediately after realizing it doesn't support .env 👋

[sdispater]

@tumido

This issue was not closed because of a technical reason.

You're right. The reason for closing it was not technical but because this feature does not align with the vision I have for Poetry. Believe me, I try to analyze every feature request made here to see if it matches the direction I want the project to go in and, if not, I reject the idea. It's as simple as that.

I understand that this feature could be useful for some people, I do, however it's not the purpose of a dependency and package manager to manage the .env file and environment variables.

That being said, my idea has always been for Poetry to be extensible that's why I am working on a plugin system so that people can tailor it to there needs if they feel like what Poetry is doing by default is not enough.

@rajasimon Sorry to see you go! I hope you'll change your mind in the future.

[flyte]

I am working on a plugin system so that people can tailor it to there needs

This would be great, thank you! I'm also of the opinion that if you're offering automated shell creation with poetry shell and poetry run then it makes sense to be able to set some environment vars in the shell you're creating. Of course I'll be happy with a plugin to do it though.

[mludvig]

however it's not the purpose of a dependency and package manager to manage the .env file and environment variables.

No one expects Poetry to manage the .env file. What we are calling for is a simple hook in poetry shell and poetry run that will source the .env file if it exists. Simple as that.

Having a vision is indeed a noble thing but also I assume that you are developing Poetry for people. When so many of us have a need for a particular feature there may be perhaps a genuine, logical reason?

[gormster]

@sdispater Were poetry shell, poetry run and poetry env part of your vision? If so, surely you understood that if you are taking on the responsibility of managing a virtual environment, you're also taking on the responsibility of modifying that environment?

[sdispater]

@gormster I am failing to see your point:

• poetry env is a command to create, delete and get info on virtual environments. This has nothing to do with environment variables.
• poetry run is a command to execute a command inside a virtual environment. This has nothing to do with environment variables.
• poetry shell is a commodity command to activate a virtual environment managed by Poetry. This has nothing to do with environment variables.

So, to be clear, I don't understand what virtual environments have to do with environment variables.

And to add to that, none of the package/dependency managers out there that I know of (beside pipenv, but then again Poetry won't replicate every feature of pipenv) do this automatic reading of environment variables because it's just not their role.

So, let me ask you a question: before Poetry (and pipenv) how did you manage your environment variables?

I will also reiterate what I said above:

If the libraries you use need it, it's their responsibility to load it not Poetry's.

If you want to go the 12-factor apps route, it's your responsibility to make your application load and use the environment variables it needs. Poetry has nothing to do with that.

[rajasimon]

When I'm deploying the application I don't work with Virtualenv so simply pip install -r will do the work. And I will add environment variable in systemd control.

But in local development I'm using this simple single line to update the environment before doning python app.py.

# This was before pipenv era
export $(grep -v '^#' .env | xargs -0)

[brianfinkelstein]

While working with one of my super clever coworkers this week, I happily learned how to work around this "issue" with very minimal overhead.

I decided to try and share my education with the community by putting together a quick boilerplate repo.

It is a really simple flask app setup with poetry and python-dotenv. (Be sure to check out the Justfile!)

[golyalpha]

@Vozf My issue is that this feature was already implemented, and the only reason it's not implemented as of yet in Poetry itself is because @sdispater doesn't want it.

[adonig]

If you want this feature for django, you can use django-environ. Otherwise you can also use direnv.

EDIT: Oh and python-dotenv looks really nice.

[kajuberdut]

If you want this feature for django, you can use django-environ. Otherwise you can also use direnv.

These, along with all of the other suggestions that I have seen here do not apply to the 50-80% of computer users who are on windows (https://en.wikipedia.org/wiki/Usage_share_of_operating_systems#Desktop_and_laptop_computers). From my research the only established library that makes loading .env painless and automatic for Windows users of Python is Pipenv.

[nikhilweee]

It's interesting to see that one of the oldest open issues out there (#208) also relates to using environment variables. Also interesting to note is that this issue (#337) is one of the most commented open issues. Moreover, this issue has been raised more than once (#3324). There's also a PR handy (#339). Clearly the community is in favour of manipulating custom environment variables using poetry.

@sdispater Is there a way where you'd be willing to reconsider this?
P.S. Please? 🥺

[systematicguy]

Ok, so I converted my dev team last year to use poetry. One of our biggest python project depends on cx_Oracle. We happen to work with MacOS Catalina 10.15.7 and lately it is not enough any more to just do a poetry install. Before poetry install we need to issue export ARCHFLAGS="-arch x86_64" otherwise upon installation we get error: architecture not supported

So it is not any more just for the design of the developed application itself which lacks this functionality (which is arguably not the scope of poetry), but the very freaking reason we use poetry: one-stop-shop for installing dependencies. This is now rendered incomplete. It is not any more a one-stop-shop.

[pizzapanther]

@kottalovag I got frustrated with the lack of dotenv support also but also wanted a better task runner features like poethepoet supports. So I made a wrapper that includes these features: https://pypi.org/project/trashy-poetry/

It's a drop in replacement, just use prose instead of poetry. Also if anyone has other suggestions for features please file an issue at: https://github.com/pizzapanther/prose

[askoretskiy]

I've developed a Rust application that reads .env file. See https://github.com/askoretskiy/readenv

[sdispater]

As you may already know, a plugin system for Poetry is currently being developed (see #3733) which will make it possible to build a plugin that will be in charge of loading .env files.

There is a naive example of what it could look like here: https://github.com/python-poetry/poetry/blob/plugin-system-revamped/docs/docs/plugins.md#event-handler

So I am hopeful that, once this lands in the 1.2 release, someone will build a plugin that satisfies the initial request.

[pizzapanther]

@sdispater how close is the plugin system? Weeks,months, year?

[sdispater]

@pizzapanther A first alpha release with the plugin system should be available by the end of March, if all goes well. Note that this is just a target but not a promise.

[kakarukeys]

For those interested in a solution that doesn't involve third party tools, on my zsh I overrode poetry to behave like pipenv.

function poetry() {
    # if POETRY_DONT_LOAD_ENV is *not* set, then load .env if it exists
    if [[ -z "$POETRY_DONT_LOAD_ENV" && -f .env ]]; then
        echo 'Loading .env environment variables…'
        export $(grep -v '^#' .env | tr -d ' ' | xargs)
        command poetry "$@"
        unset $(grep -v '^#' .env | sed -E 's/(.*)=.*/\1/' | xargs)
    else
        command poetry "$@"
    fi
}

New enhanced script here

I've enhanced @miguendes' script to load .env in parent directories, if it's not found in the current working directory. This is inline with Poetry's behavior of being able to locate pyproject.toml in parent directories.

[mdelz]

For the linux users out there dotenv allows to use poetry shell but it looks hideous.

# 1. install dotenv
poetry add "python-dotenv[cli]"

# 2. start your shell
poetry run dotenv run poetry shell

[mpeteuil]

There is a naive example of what it could look like here: https://github.com/python-poetry/poetry/blob/plugin-system-revamped/docs/docs/plugins.md#event-handler

So I am hopeful that, once this lands in the 1.2 release, someone will build a plugin that satisfies the initial request.

I've thrown together a plugin based on the naive example: mpeteuil/poetry-dotenv-plugin (also on PyPi).

You'll need a Poetry 1.2 pre-release in order to try it now though (which would currently be Poetry 1.2.0a1).

Poetry 1.2.0, which supports plugins, has been released as of August 31, 2022.

[ccrvlh]

I can't read my .env file, even with python-dotenv. Works fine when I pass the env_file in Docker, so I'm imagining it may have something to do with the way poetry deals with shell / venv?

I have pretty much the same setup in a Flask app using regular venv, never had any problem, so I imagine it may have something to do with the way poetry implements venv? Any other ideas?

[ghost]

for running django application with gunicorn in your wsgi.py remember to use loadenv

import os

from django.core.wsgi import get_wsgi_application
from dotenv import load_dotenv #dotenv is an external package https://github.com/theskumar/python-dotenv

load_dotenv(os.path.join(os.path.dirname(os.path.dirname(__file__)), '.env'))#load env

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "base.settings")
 
application = get_wsgi_application()

[ZetiMente]

@sdispater I've used Poetry since I found it months ago and only have had good things to say. Thank you, you are a hero.

But I thought to put my Google Cloud variables in a .env file because there are dev/qa/prod clouds. I've usually use poetry run python main.py to start my programs.

What would be the most clean solution here?

[mpeteuil]

@ZetiMente If you're on Poetry >= 1.2.0a1, then you can use poetry plugin add poetry-dotenv-plugin and it should "just work" 🤞. Otherwise use one of the other suggestions above until the Poetry 1.2.0 official release.

[Beaglefoot]

Probably the simplest solution with almost zero effort is this:

env $(cat .env) poetry shell

env is part of coreutils, so no additional dependency here.

[mconigliaro]

I'm just here to read the comments and this popcorn IS MAKING ME THIRSTY! 🍿😂

Thanks for the plugin, @mpeteuil.

[seekingThat]

It might be nice to mention that .env file should be present in the top directory of the package for it to be loaded. I didn't notice that in https://python-poetry.org/docs/managing-environments/ or in
https://python-poetry.org/docs/cli/#env

[seekingThat]

I had a different location for .env for the package. But for the purpose of poetry to facilitate poetry run, I created a .env in top directory as a link to the original .env file

[mpeteuil]

@seekingThat The reason you were unable to find any documentation about .env files is because it is not something that the Poetry organization/team supports and those are official Poetry docs. Those environment docs you're referencing are mostly talking about the virtual environment that Poetry manages. If you're using mpeteuil/poetry-dotenv-plugin, that is a plugin that is unaffiliated with the Poetry organization. Any issues that arise from using that plugin can be opened on the plugin's repo.

[kstech-hf]

This dude said "I don't understand what virtual environments have to do with environment variables."

[skanduru]

@seekingThat The reason you were unable to find any documentation about .env files is because it is not something that the Poetry organization/team supports and those are official Poetry docs. Those environment docs you're referencing are mostly talking about the virtual environment that Poetry manages. If you're using mpeteuil/poetry-dotenv-plugin, that is a plugin that is unaffiliated with the Poetry organization. Any issues that arise from using that plugin can be opened on the plugin's repo.

Thank you very much! Appreciate your clarification

[motin]

Note that poethepoet supports .env files since a while back: https://github.com/nat-n/poethepoet#environment-variables

[coltonbh]

If you are using pydantic in your app you can allow the BaseSettings object to correctly load environment variables from a .env file using the description found here: https://pydantic-docs.helpmanual.io/usage/settings/#dotenv-env-support

[WarpedPixel]

The downside of plugins (and various scripts to load .env files) is that they are local to the install and not captured in the package description in the pyproject.toml, so one more step to perform/document.

Note also that in the latest poetry 1.2, the command to install the plugin is

$ poetry self add poetry-dotenv-plugin