Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

install-poetry breaks with Poetry 1.2.0 #89

Closed
deronnax opened this issue Aug 31, 2022 · 29 comments
Closed

install-poetry breaks with Poetry 1.2.0 #89

deronnax opened this issue Aug 31, 2022 · 29 comments

Comments

@deronnax
Copy link

deronnax commented Aug 31, 2022

Poetry 1.2.0 changes a lot of things about how path and virtualenvs are handled, and it breaks install-poetry:

▶ Run snok/install-poetry@v1
  with:
    virtualenvs-create: true
    virtualenvs-in-project: true
    installer-parallel: true
    version: latest
    virtualenvs-path: {cache-dir}/virtualenvs
  env:
    DJANGO_ENV: LOCAL
    pythonLocation: /opt/hostedtoolcache/Python/3.6.15/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.6.15/x64/lib
Run $GITHUB_ACTION_PATH/main.sh
  $GITHUB_ACTION_PATH/main.sh
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    DJANGO_ENV: LOCAL
    pythonLocation: /opt/hostedtoolcache/Python/3.6.15/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.6.15/x64/lib
    VERSION: latest
    VIRTUALENVS_CREATE: true
    VIRTUALENVS_IN_PROJECT: true
    VIRTUALENVS_PATH: {cache-dir}/virtualenvs
    INSTALLER_PARALLEL: true
    INSTALLATION_ARGUMENTS: 

Setting Poetry installation path as /home/runner/.local/

Installing Poetry 👷

Retrieving Poetry metadata

# Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the `poetry` command to Poetry's bin directory, located at:

/home/runner/.local/bin

You can uninstall at any time by executing this script with the --uninstall option,
and these changes will be reverted.

Installing Poetry (1.[2](https://github.com/critizr/critizr-django-1.7/runs/8108219212?check_suite_focus=true#step:7:2).0)
Installing Poetry (1.2.0): Creating environment
Installing Poetry (1.2.0): Installing Poetry
Installing Poetry (1.2.0): Creating script
Installing Poetry (1.2.0): Done

Poetry (1.2.0) is installed now. Great!

You can test that everything is set up by executing:

`poetry --version`

/home/runner/work/_actions/snok/install-poetry/v1/main.sh: line [3](https://github.com/critizr/critizr-django-1.7/runs/8108219212?check_suite_focus=true#step:7:3)3: poetry: command not found
/home/runner/work/_actions/snok/install-poetry/v1/main.sh: line 3[4](https://github.com/critizr/critizr-django-1.7/runs/8108219212?check_suite_focus=true#step:7:4): poetry: command not found
/home/runner/work/_actions/snok/install-poetry/v1/main.sh: line 3[5](https://github.com/critizr/critizr-django-1.7/runs/8108219212?check_suite_focus=true#step:7:5): poetry: command not found
/home/runner/work/_actions/snok/install-poetry/v1/main.sh: line 3[7](https://github.com/critizr/critizr-django-1.7/runs/8108219212?check_suite_focus=true#step:7:7): poetry: command not found

Installation completed. Configuring settings 🛠

Done ✅

If you are creating a venv in your project, you can activate it by running 'source .venv/bin/activate'. If you're running this in an OS matrix, you can use 'source $VENV' instead, as an OS agnostic option

▶ Run poetry install --no-interaction --no-root
/home/runner/work/_temp/63f8c552-7bf4-4d02-bd7b-8b02d[9](https://github.com/critizr/critizr-django-1.7/runs/8109330298?check_suite_focus=true#step:9:10)c8b5c0.sh: line 1: poetry: command not found
Error: Process completed with exit code 127.
@sondrelg
Copy link
Member

Would you mind sharing your workflow configuration (just the install-poetry step)?

I don't think it breaks the action. AFAICT lockfiles generated by < 1.2 are incompatible with 1.2 and vice versa, so if your lockfile was generated with 1.1.X you need to specify a 1.1.X version in your action run.

    - uses: snok/install-poetry@v1
      with:
        version: 1.1.14

Should work, while

    - uses: snok/install-poetry@v1

Will not.

Does that make sense, or do you think it's broken for 1.2 with 1.2 lockfiles?

@deronnax
Copy link
Author

Hello :)
I pasted the output of the action (and then you have the action's configuration as well). It's the poetry command itself that is not found. See the original message.

@deronnax deronnax changed the title install-poetry broken with Poetry 1.2.0 install-poetry breaks with Poetry 1.2.0 Aug 31, 2022
@sondrelg
Copy link
Member

Ah yes. And I see you've specified latest as the version. Does it work if you change it to 1.1.14?

@deronnax
Copy link
Author

Yes

@sondrelg
Copy link
Member

sondrelg commented Aug 31, 2022

Ok great. If you want to use 1.2.0 in your github actions run you first need to upgrade Poetry to 1.2.0 locally, then run poetry update or poetry lock --no-update to make the file compatible with 1.2.0. After doing that, upgrading to 1.2.0 or latest in the action should work again.

@deronnax
Copy link
Author

Hello. You got it. Sorry I should not have cut the Github Actions output: the error we get is poetry: command not found. You can see this error (somewhat hidden) in GHA output. I think Poetry 1.2.0 changed the location where poetry is installed.

@sondrelg
Copy link
Member

I see. That's unfortunate if that's the case. I'm not able to look at this right now, but PRs are welcome 👍

@deronnax
Copy link
Author

No worries. I will make a PR if I manage to find out what broke. Let's not close this issue as I assume many people will come here to report the issue.

@fritz-astronomer
Copy link

Also seeing this issue 👍

@fritz-astronomer
Copy link

fritz-astronomer commented Aug 31, 2022

#90
Attempting a fix here, unsure how to test this in a Github Action

edit:// this seems to fix it - there was a trailing / in the path, by accident. I have a separate issue now of Poetry 1.2.0 dropping support for Python 3.6, but that's unrelated 😃

@sondrelg
Copy link
Member

I added two comments in the PR, but would be good to get your take too @deronnax. I tried updating two projects to Poetry version 1.2.0, and neither one had any problems running the action. Just to understand what is broken, what is different between these two workflows and yours?

@fritz-astronomer
Copy link

Discussion continued in #90

@deronnax
Copy link
Author

deronnax commented Sep 1, 2022

@sondrelg no clue, our workflow is pretty standard, my bet is that it's all related to the dropping of Python 3.6 from Poetry 1.2.0. We will pin the Poetry version and wait until we move to the next python version to see if it reproduces, which I bet it won't.

In between, thank you very much for your time

@ronlut
Copy link

ronlut commented Sep 1, 2022

To avoid the brownout, add the following to the poetry install step until you upgrade to 1.2.0

env:
  GET_POETRY_IGNORE_DEPRECATION: 1

@miigotu
Copy link
Collaborator

miigotu commented Sep 1, 2022

#90 Attempting a fix here, unsure how to test this in a Github Action

edit:// this seems to fix it - there was a trailing / in the path, by accident. I have a separate issue now of Poetry 1.2.0 dropping support for Python 3.6, but that's unrelated smiley

https://endoflife.date/python

Projects are dropping python3.6 because it is no longer considered secure as end of life was 8 months ago and it no longer receives security fixes. Tools and libraries dropping support forces OS distributions to release newer, supported versions of python on their platform. This is a good thing, because it is intended to result in a more secure internet and better safety for users' machines.

image

@amirhessam88
Copy link

I guess you gotta make sure to have

[build-system]
requires = ["poetry-core>=1.1.0"]

More on --> https://github.com/python-poetry/poetry/blob/master/pyproject.toml#L47

@Musaefendic
Copy link

Same issue here, but only with macos-latest
On the other hand no problem with windows-latest and ubuntu-latest

Here is the link to the yml file: https://github.com/Musaefendic/icao-emissions/blob/96d41895570a1aefe21cab13f32f906f44d6f6ab/.github/workflows/tests.yml

2022-09-29T16:05:45.3245480Z ##[group]Run poetry install --no-interaction
2022-09-29T16:05:45.3245930Z �[36;1mpoetry install --no-interaction�[0m
2022-09-29T16:05:45.3754580Z shell: /bin/bash --noprofile --norc -e -o pipefail {0}
2022-09-29T16:05:45.3754920Z env:
2022-09-29T16:05:45.3755490Z   pythonLocation: /Users/runner/hostedtoolcache/Python/3.9.10/x64
2022-09-29T16:05:45.3755920Z   PKG_CONFIG_PATH: /Users/runner/hostedtoolcache/Python/3.9.10/x64/lib/pkgconfig
2022-09-29T16:05:45.3756330Z   Python_ROOT_DIR: /Users/runner/hostedtoolcache/Python/3.9.10/x64
2022-09-29T16:05:45.3757220Z   Python2_ROOT_DIR: /Users/runner/hostedtoolcache/Python/3.9.10/x64
2022-09-29T16:05:45.3757620Z   Python3_ROOT_DIR: /Users/runner/hostedtoolcache/Python/3.9.10/x64
2022-09-29T16:05:45.3757940Z   VENV: .venv/bin/activate
2022-09-29T16:05:45.3758180Z ##[endgroup]
2022-09-29T16:05:45.4574200Z /Users/runner/work/_temp/7e911813-9a38-4bc5-8835-86f3f94d1643.sh: line 1: poetry: command not found
2022-09-29T16:05:45.4602330Z ##[error]Process completed with exit code 127.

did I set up my .yml wrong?

@amirhessam88
Copy link

@Musaefendic is it your intention not using poetry v1.2.1 ? I can see this https://github.com/Musaefendic/icao-emissions/blob/96d41895570a1aefe21cab13f32f906f44d6f6ab/.github/workflows/tests.yml#L48
You can drop the version and the action will automatically setup the latest version

@Musaefendic
Copy link

Musaefendic commented Sep 29, 2022

good catch! It is an error after several unsuccessful attempts.
I just removed the line version: 1.1.14 but the result is the same.

✅ windows-latest and ubuntu-latest
❌ macos-latest

i got the same error:

Run poetry install --no-interaction
[...]
/Users/runner/work/_temp/6c346071-2b8c-41b1-b4f2-75377cb405b1.sh: line 1: poetry: command not found
Error: Process completed with exit code 127.

@miigotu
Copy link
Collaborator

miigotu commented Sep 29, 2022

I wonder if your action is caching a really old version 1, and it's not using the newest version.

Maybe try uses: snok/[email protected]

You can also try activating it manually

@miigotu
Copy link
Collaborator

miigotu commented Sep 29, 2022

@Musaefendic also, restore venv cache BEFORE installing poetry.

@miigotu
Copy link
Collaborator

miigotu commented Sep 29, 2022

And if it still doesn't work I'd start with this after installing poetry

      - name: Check environment 
        run: |
          echo $PATH
          echo $VENV
          ls ./local/bin

@miigotu
Copy link
Collaborator

miigotu commented Sep 29, 2022

And I'm most sure snok/install-poetry@lint could work, because he added the path to the final commands.

@Musaefendic
Copy link

Musaefendic commented Sep 29, 2022

And if it still doesn't work I'd start with this after installing poetry

      - name: Check environment 
        run: |
          echo $PATH
          echo $VENV
          ls ./local/bin
/Users/runner/.local//bin:/Users/runner/hostedtoolcache/Python/3.9.10/x64/bin:/Users/runner/hostedtoolcache/Python/3.9.10/x64:/usr/local/lib/ruby/gems/2.7.0/bin:/usr/local/opt/[email protected]/bin:/usr/local/opt/pipx_bin:/Users/runner/.cargo/bin:/usr/local/opt/curl/bin:/usr/local/bin:/usr/local/sbin:/Users/runner/bin:/Users/runner/.yarn/bin:/Users/runner/Library/Android/sdk/tools:/Users/runner/Library/Android/sdk/platform-tools:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/usr/bin:/bin:/usr/sbin:/sbin:/Users/runner/.dotnet/tools:/Users/runner/.ghcup/bin:/Users/runner/hostedtoolcache/stack/2.9.1/x64

.venv/bin/activate

ls: ./local/bin: No such file or directory
Error: Process completed with exit code 1.

I wonder if your action is caching a really old version 1, and it's not using the newest version.

good intuition! if I delete Load cached poetry installation, all workflows pass ✅
any idea how to purge the cache?

@Musaefendic
Copy link

Musaefendic commented Sep 29, 2022

To clear the GH Actions cache, it's a new feature since June 22. See this SO post https://stackoverflow.com/a/64819132

Clearing the cache

curl \
    -H "Accept: application/vnd.github.v3+json" \
    -H "Authorization: token *********************" \
    https://api.github.com/repos/Musaefendic/icao-emissions/actions/caches
{
  "total_count": 0,
  "actions_caches": [

  ]
}

1st CI Run after clearing the cache.

Link to my tests.yml

  • ✅ ubuntu-latest
  • ✅ macos-latest
  • ✅ windows-latest

cache after this CI run

curl \
    -H "Accept: application/vnd.github.v3+json" \
    -H "Authorization: token ****************" \
    https://api.github.com/repos/Musaefendic/icao-emissions/actions/caches
{
  "total_count": 5,
  "actions_caches": [
    {
      "id": 21,
      "ref": "refs/heads/master",
      "key": "venv-macOS-3.9.10-7f3e4de67ec2fafad1743c550f0d8aaf4038b5f0f03ecfe5275d285833296cbf",
      "version": "25e65b70d167147605b4332eb57b6d34df8e0303b88c4de252efb18cf3ed148a",
      "last_accessed_at": "2022-09-29T21:31:08.286666700Z",
      "created_at": "2022-09-29T21:31:08.286666700Z",
      "size_in_bytes": 71408887
    },
    {
      "id": 20,
      "ref": "refs/heads/master",
      "key": "poetry-0",
      "version": "d6b93696bc837862e84764c8431f7c282393336a702ec638c820588088bfd7a1",
      "last_accessed_at": "2022-09-29T21:30:09.980000000Z",
      "created_at": "2022-09-29T21:30:09.980000000Z",
      "size_in_bytes": 30
    },
    {
      "id": 19,
      "ref": "refs/heads/master",
      "key": "venv-Windows-3.9.10-14a2a15fefb761fe78d797839a035d718998d5ff92247101add1124f1e5af187",
      "version": "519c93c80eb6dc7871185acd9844f0af47c509ad7b0399902b51e465753dba51",
      "last_accessed_at": "2022-09-29T21:29:51.203333300Z",
      "created_at": "2022-09-29T21:29:51.203333300Z",
      "size_in_bytes": 75356150
    },
    {
      "id": 18,
      "ref": "refs/heads/master",
      "key": "poetry-0",
      "version": "f4b5713a4657ae30a7c2804b2ad5dbc790b3736f71a74a904bcab1f478b0ed29",
      "last_accessed_at": "2022-09-29T21:27:04.993333300Z",
      "created_at": "2022-09-29T21:27:04.993333300Z",
      "size_in_bytes": 25628232
    },
    {
      "id": 17,
      "ref": "refs/heads/master",
      "key": "venv-Linux-3.9.10-7f3e4de67ec2fafad1743c550f0d8aaf4038b5f0f03ecfe5275d285833296cbf",
      "version": "25e65b70d167147605b4332eb57b6d34df8e0303b88c4de252efb18cf3ed148a",
      "last_accessed_at": "2022-09-29T21:27:00.680000000Z",
      "created_at": "2022-09-29T21:27:00.680000000Z",
      "size_in_bytes": 82591676
    }
  ]
}

2nd CI RUN

same tests.yml

  • ✅ ubuntu-latest
  • ❌ macos-latest -> Error: Process completed with exit code 127.
  • ✅ windows-latest

@sondrelg
Copy link
Member

sondrelg commented Oct 1, 2022

I'm on holidays almost without internet for a few days, but should have time to review a PR if anyone has time to investigate 👍 Otherwise, I'll be happy to look at this on Tuesday 🙂

@miigotu
Copy link
Collaborator

miigotu commented Oct 1, 2022

And if it still doesn't work I'd start with this after installing poetry

      - name: Check environment 
        run: |
          echo $PATH
          echo $VENV
          ls ./local/bin
/Users/runner/.local//bin:/Users/runner/hostedtoolcache/Python/3.9.10/x64/bin:/Users/runner/hostedtoolcache/Python/3.9.10/x64:/usr/local/lib/ruby/gems/2.7.0/bin:/usr/local/opt/[email protected]/bin:/usr/local/opt/pipx_bin:/Users/runner/.cargo/bin:/usr/local/opt/curl/bin:/usr/local/bin:/usr/local/sbin:/Users/runner/bin:/Users/runner/.yarn/bin:/Users/runner/Library/Android/sdk/tools:/Users/runner/Library/Android/sdk/platform-tools:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/usr/bin:/bin:/usr/sbin:/sbin:/Users/runner/.dotnet/tools:/Users/runner/.ghcup/bin:/Users/runner/hostedtoolcache/stack/2.9.1/x64

.venv/bin/activate

ls: ./local/bin: No such file or directory
Error: Process completed with exit code 1.

I wonder if your action is caching a really old version 1, and it's not using the newest version.

good intuition! if I delete Load cached poetry installation, all workflows pass ✅
any idea how to purge the cache?

You could always use a different cache key or add a restore key that's different than what it's using now. Poetry version changed but your pyproject.toml didn't, so the next change to your pyprotect.toml would have done it also.

@sTomerG
Copy link

sTomerG commented Oct 7, 2022

I'm having the exact same issue (exit code 127, only on macOS-latest) with the following workflow. Is there something I should change?

  push:
    branches: [main]
  pull_request:

jobs:
  build:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.10"]
        os: [ubuntu-latest, macOS-latest, windows-latest]
    env:
      POETRY_VIRTUALENVS_IN_PROJECT: true
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}

      - name: cache poetry install
        uses: actions/cache@v2
        with:
          path: ~/.local
          key: poetry-1.2.1-0

      - uses: snok/install-poetry@v1
        with:
          version: 1.2.1
          virtualenvs-create: true
          virtualenvs-in-project: true

      #  Cache your dependencies (i.e. all the stuff in your `pyproject.toml`)
      - name: cache venv
        uses: actions/cache@v2
        with:
          path: .venv
          key: venv-${{ runner.os }}-${{ matrix.python-version }}-${{ hashFiles('**/poetry.lock') }}
      - run: poetry install --no-interaction --no-root
        if: steps.cache-deps.outputs.cache-hit != 'true'
      - run: poetry install --no-interaction
      - run: poetry run flake8 src/ tests/
      - run: poetry run pytest```

@sTomerG
Copy link

sTomerG commented Oct 7, 2022

I'm having the exact same issue (exit code 127, only on macOS-latest) with the following workflow. Is there something I should change?

  push:
    branches: [main]
  pull_request:

jobs:
  build:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.10"]
        os: [ubuntu-latest, macOS-latest, windows-latest]
    env:
      POETRY_VIRTUALENVS_IN_PROJECT: true
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}

      - name: cache poetry install
        uses: actions/cache@v2
        with:
          path: ~/.local
          key: poetry-1.2.1-0

      - uses: snok/install-poetry@v1
        with:
          version: 1.2.1
          virtualenvs-create: true
          virtualenvs-in-project: true

      #  Cache your dependencies (i.e. all the stuff in your `pyproject.toml`)
      - name: cache venv
        uses: actions/cache@v2
        with:
          path: .venv
          key: venv-${{ runner.os }}-${{ matrix.python-version }}-${{ hashFiles('**/poetry.lock') }}
      - run: poetry install --no-interaction --no-root
        if: steps.cache-deps.outputs.cache-hit != 'true'
      - run: poetry install --no-interaction
      - run: poetry run flake8 src/ tests/
      - run: poetry run pytest```

fixed it by adding

env:
  POETRY_VERSION: 1.2.1

And replacing

key: poetry-1.2.1-0

with

key: poetry-cache-${{ runner.os }}-${{ matrix.python-version }}-${{ env.POETRY_VERSION }}

rgerkin pushed a commit to pyrfume/pyrfume that referenced this issue Oct 25, 2022
* refactor github action to perform linting before testing and use a typical poetry installation workflow

* fix syntax errors in github action config

* reformatted files with black

* reorganized imports with isort

* ignore swap files so we don't accidentally commit them

* create user group, add python executables to path, and upgrade pip.
I changed the default CMD to 'bash' because 'poetry shell' wasn't working and I couldn't figure out why...

* simplify github workflow into just one job
there's no reason to separate concerns between linting and testing now, plus I didn't want to have repetetitive jobs (although I could eventually templatize the setup steps)

* changed to an available python version on github runners

* workaround bug in snok/install-poetry action that gives 'poetry: command not found' for the latest version (1.2.0)

* went back to poetry 1.2.0 but added ~/.local/bin to the system path in the github action
the reason for this reversion is described in more detail here: python-poetry/poetry#4242
essentially there isn't backwards compatibility with 1.2.0 lock files and old poetry versions (like the 1.1.14 I switched it to before in the github action), and it couldn't find 'setuptools' because of this

* explicitly list directory, maybe because /home/tristan isn't resolved correctly?

* update poetry in dev dockerfile, lock updates with this new version, and upgrade the poetry-core version for building
these were recommended from the issue with the snok/install-poetry github action documented at snok/install-poetry#89

* add path logging so i can figure out what's going on in the github action

* add env variable to hopefully prevent issues with upgrade

* maybe this is caused by 3.6 being deprecated? comments in the previously linked issue suggested this could be why

* run linting and testing steps in the github action job in a poetry shell
otherwise, the job will fail because the tools are libraries aren't actually available in the environment

* poetry run wasn't accepted by github as valid for the 'shell' field so I added a bash -c to see if that works

* that didn't work either so calling poetry run on every command separately

* fix spacing in run command

* remove 3.10 runtime because I realized I didn't actually test this works

* take out pointless echos and exclude poetry .venv from flake8 linting because otherwise it will lint dependency source files we have no control over

* fix a bunch of linting errors

* reformat files and fix a couple remaining linting errors

* make isort compatible with black so they don't keep reformatting endlessly -- had to sacrifice some comments because I couldn't figure out how to get it to work without this (will try later)

* fix recursive import by importing tqdm modules from the library directly

* fix permissions on parent directory of the devcontainer's working directory to allow for the unit testing module to download some testing data from https://github.com/pyrfume/pyrfume-data

* add back the line that I thought was a pointless import but actually has relevant logic...

* fix last lingering linting error, left as a TODO

* drop support for python 3.6 (not even tested in actions and losing support anyways soon) to upgrade pandas to a compatible version so that tests fully pass

* remove macos runtime because for some reason it can't install public files from api.github.com/repos, even though I tested the same install locally on macOS with all of the failing python versions

* don't fail fast so we can see the results of runs on all python distributions. also I reincluded macos just to see how far they get
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants