Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix for git subdirectories #5172

Merged
merged 6 commits into from
Aug 10, 2022
Merged

Conversation

ashnair1
Copy link
Contributor

@ashnair1 ashnair1 commented Feb 9, 2022

Pull Request Check List

Resolves: #755

  • Added tests for changed code.

@ashnair1
Copy link
Contributor Author

ashnair1 commented Feb 9, 2022

Linked to python-poetry/poetry-core#288

@ashnair1
Copy link
Contributor Author

ashnair1 commented Mar 2, 2022

Pinging @sdispater @finswimmer

@ashnair1
Copy link
Contributor Author

ashnair1 commented Mar 2, 2022

Test requires poetry-core > 1.1.0 to pass since it needs ParsedUrl to extract subdirectory parameter correctly.

@ashnair1
Copy link
Contributor Author

@finswimmer Can this be included in the roadmap for 1.2?

@acaloiaro
Copy link

@ashnair1 Thanks for this!

It appears there are tests failing that'll need to be resolved before a maintainer can merge.

@ashnair1
Copy link
Contributor Author

ashnair1 commented Apr 5, 2022

Pinging @finswimmer. Python tests passed.

@ashnair1
Copy link
Contributor Author

ashnair1 commented Apr 12, 2022

Pinging @finswimmer @abn. Could someone run the remaining workflows?

@ashnair1
Copy link
Contributor Author

ashnair1 commented Apr 24, 2022

Updated pre-commit hook to use python 3.8 as default. This is required because flake8-type-checking is not supported for python <=3.7

@ashnair1
Copy link
Contributor Author

Pinging @abn @finswimmer

@andersk
Copy link
Contributor

andersk commented May 20, 2022

Some issues I’ve noticed:

The subdirectory is missing in the PEP 610 direct_url.json file.
$ poetry init -n
$ poetry add 'git+https://github.com/zulip/[email protected]#subdirectory=zulip'
$ poetry run sh -c 'python -mjson.tool < "$VIRTUAL_ENV"/lib/python3.9/site-packages/zulip-0.8.2.dist-info/direct_url.json'
{
    "url": "https://github.com/zulip/python-zulip-api.git",
    "vcs_info": {
        "vcs": "git",
        "requested_revision": "0.8.2",
        "commit_id": "d26416a1e79fabddf392bd4a528381305519fe52"
    }
}

Expected, based on installing the same package with Pip:

{
    "subdirectory": "zulip",
    "url": "https://github.com/zulip/python-zulip-api.git",
    "vcs_info": {
        "commit_id": "d26416a1e79fabddf392bd4a528381305519fe52",
        "requested_revision": "0.8.2",
        "vcs": "git"
    }
}
A package from a Git subdirectory is missing entirely from poetry export output.
$ poetry init -n
$ poetry add 'git+https://github.com/zulip/[email protected]#subdirectory=zulip'
$ poetry export --without-hashes
certifi==2022.5.18.1 ; python_version >= "3.9" and python_version < "4.0"
charset-normalizer==2.0.12 ; python_version >= "3.9" and python_version < "4.0"
click==8.1.3 ; python_version >= "3.9" and python_version < "4.0"
colorama==0.4.4 ; python_version >= "3.9" and python_version < "4.0" and platform_system == "Windows"
distro==1.7.0 ; python_version >= "3.9" and python_version < "4.0"
idna==3.3 ; python_version >= "3.9" and python_version < "4.0"
matrix-client==0.4.0 ; python_version >= "3.9" and python_version < "4.0"
requests==2.27.1 ; python_version >= "3.9" and python_version < "4.0"
typing-extensions==4.2.0 ; python_version >= "3.9" and python_version < "4.0"
urllib3==1.26.9 ; python_version >= "3.9" and python_version < "4"
$ poetry export --without-hashes | grep zulip
$
A subdirectory of a zip file still doesn’t work.
$ poetry init -n
$ poetry add 'https://github.com/zulip/python-zulip-api/archive/0.8.2.zip#subdirectory=zulip'

  ValueError

  Not a known archive format: /tmp/tmphx126w3m/0.8.2.zip

  at /nix/store/ybj5070jzjfi68rzrjv7qdhi2yvr2igk-python3.9-pkginfo-1.8.2/lib/python3.9/site-packages/pkginfo/bdist.py:25 in read
       21│             names = archive.namelist()
       22│             def read_file(name):
       23│                 return archive.read(name)
       24│         else:
    →  25│             raise ValueError('Not a known archive format: %s' % fqn)
       26│ 
       27│         try:
       28│             tuples = [x.split('/') for x in names if 'PKG-INFO' in x]
       29│             schwarz = sorted([(len(x), x) for x in tuples])

The following error occurred when trying to handle this error:


  PackageInfoError

  Unable to determine package info for path: /tmp/tmphx126w3m/0.8.2.zip
  
  Not a known archive format: /tmp/tmphx126w3m/0.8.2.zip

  at /nix/store/w4gas8kya80hphl1a3r9gnkvcwjqffyg-python3.9-poetry-1.2.0b1-177-gb8db5bd5/lib/python3.9/site-packages/poetry/inspection/info.py:542 in from_bdist
      538│ 
      539│         try:
      540│             return cls._from_distribution(pkginfo.BDist(str(path)))
      541│         except ValueError as e:
    → 542│             raise PackageInfoError(path, e)
      543│ 
      544│     @classmethod
      545│     def from_path(cls, path: Path) -> PackageInfo:
      546│         """

The following error occurred when trying to handle this error:


  CalledProcessError

  Command '['/tmp/tmp0lxiu8hj/.venv/bin/python', '-']' returned non-zero exit status 1.

  at /nix/store/wnrc4daqbd6v5ifqlxsj75ky8556zy0p-python3-3.9.12/lib/python3.9/subprocess.py:528 in run
       524│             # We don't call process.wait() as .__exit__ does that for us.
       525│             raise
       526│         retcode = process.poll()
       527│         if check and retcode:
    →  528│             raise CalledProcessError(retcode, process.args,
       529│                                      output=stdout, stderr=stderr)
       530│     return CompletedProcess(process.args, retcode, stdout, stderr)
       531│ 
       532│ 

The following error occurred when trying to handle this error:


  EnvCommandError

  Command ['/tmp/tmp0lxiu8hj/.venv/bin/python', '-'] errored with the following return code 1, and output: 
  Traceback (most recent call last):
    File "/tmp/tmp0lxiu8hj/.venv/lib/python3.9/site-packages/build/__init__.py", line 422, in _handle_backend
      yield
    File "/tmp/tmp0lxiu8hj/.venv/lib/python3.9/site-packages/build/__init__.py", line 310, in get_requires_for_build
      return set(get_requires(config_settings))
    File "/tmp/tmp0lxiu8hj/.venv/lib/python3.9/site-packages/pep517/wrappers.py", line 172, in get_requires_for_build_wheel
      return self._call_hook('get_requires_for_build_wheel', {
    File "/tmp/tmp0lxiu8hj/.venv/lib/python3.9/site-packages/pep517/wrappers.py", line 322, in _call_hook
      self._subprocess_runner(
    File "/tmp/tmp0lxiu8hj/.venv/lib/python3.9/site-packages/build/__init__.py", line 261, in _runner
      self._hook_runner(cmd, cwd, extra_environ)
    File "/tmp/tmp0lxiu8hj/.venv/lib/python3.9/site-packages/pep517/wrappers.py", line 75, in quiet_subprocess_runner
      check_output(cmd, cwd=cwd, env=env, stderr=STDOUT)
    File "/nix/store/wnrc4daqbd6v5ifqlxsj75ky8556zy0p-python3-3.9.12/lib/python3.9/subprocess.py", line 424, in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    File "/nix/store/wnrc4daqbd6v5ifqlxsj75ky8556zy0p-python3-3.9.12/lib/python3.9/subprocess.py", line 528, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['/tmp/build-env-k_zatkzo/bin/python', '/tmp/tmp0lxiu8hj/.venv/lib/python3.9/site-packages/pep517/in_process/_in_process.py', 'get_requires_for_build_wheel', '/tmp/tmpw7y1a_ew']' returned non-zero exit status 1.
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "<stdin>", line 16, in <module>
    File "/tmp/tmp0lxiu8hj/.venv/lib/python3.9/site-packages/build/__init__.py", line 310, in get_requires_for_build
      return set(get_requires(config_settings))
    File "/nix/store/wnrc4daqbd6v5ifqlxsj75ky8556zy0p-python3-3.9.12/lib/python3.9/contextlib.py", line 137, in __exit__
      self.gen.throw(typ, value, traceback)
    File "/tmp/tmp0lxiu8hj/.venv/lib/python3.9/site-packages/build/__init__.py", line 430, in _handle_backend
      raise BuildBackendException(exception, f'Backend subproccess exited when trying to invoke {hook}')
  build.BuildBackendException: Backend subproccess exited when trying to invoke get_requires_for_build_wheel
  input was : import build
  import build.env
  import pep517
  
  source = '/tmp/tmpcaz69rrj/python-zulip-api-0.8.2'
  dest = '/tmp/tmp0lxiu8hj/dist'
  
  with build.env.IsolatedEnvBuilder() as env:
      builder = build.ProjectBuilder(
          srcdir=source,
          scripts_dir=env.scripts_dir,
          python_executable=env.executable,
          runner=pep517.quiet_subprocess_runner,
      )
      env.install(builder.build_system_requires)
      env.install(builder.get_requires_for_build('wheel'))
      builder.metadata_path(dest)
  

  at /nix/store/w4gas8kya80hphl1a3r9gnkvcwjqffyg-python3.9-poetry-1.2.0b1-177-gb8db5bd5/lib/python3.9/site-packages/poetry/utils/env.py:1443 in _run
      1439│                 output = subprocess.check_output(
      1440│                     cmd, stderr=subprocess.STDOUT, env=env, **kwargs
      1441│                 )
      1442│         except CalledProcessError as e:
    → 1443│             raise EnvCommandError(e, input=input_)
      1444│ 
      1445│         return decode(output)
      1446│ 
      1447│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int | None:

The following error occurred when trying to handle this error:


  PackageInfoError

  Unable to determine package info for path: /tmp/tmpcaz69rrj/python-zulip-api-0.8.2
  
  Command ['/tmp/tmp0lxiu8hj/.venv/bin/python', '-'] errored with the following return code 1, and output: 
  Traceback (most recent call last):
    File "/tmp/tmp0lxiu8hj/.venv/lib/python3.9/site-packages/build/__init__.py", line 422, in _handle_backend
      yield
    File "/tmp/tmp0lxiu8hj/.venv/lib/python3.9/site-packages/build/__init__.py", line 310, in get_requires_for_build
      return set(get_requires(config_settings))
    File "/tmp/tmp0lxiu8hj/.venv/lib/python3.9/site-packages/pep517/wrappers.py", line 172, in get_requires_for_build_wheel
      return self._call_hook('get_requires_for_build_wheel', {
    File "/tmp/tmp0lxiu8hj/.venv/lib/python3.9/site-packages/pep517/wrappers.py", line 322, in _call_hook
      self._subprocess_runner(
    File "/tmp/tmp0lxiu8hj/.venv/lib/python3.9/site-packages/build/__init__.py", line 261, in _runner
      self._hook_runner(cmd, cwd, extra_environ)
    File "/tmp/tmp0lxiu8hj/.venv/lib/python3.9/site-packages/pep517/wrappers.py", line 75, in quiet_subprocess_runner
      check_output(cmd, cwd=cwd, env=env, stderr=STDOUT)
    File "/nix/store/wnrc4daqbd6v5ifqlxsj75ky8556zy0p-python3-3.9.12/lib/python3.9/subprocess.py", line 424, in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    File "/nix/store/wnrc4daqbd6v5ifqlxsj75ky8556zy0p-python3-3.9.12/lib/python3.9/subprocess.py", line 528, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['/tmp/build-env-k_zatkzo/bin/python', '/tmp/tmp0lxiu8hj/.venv/lib/python3.9/site-packages/pep517/in_process/_in_process.py', 'get_requires_for_build_wheel', '/tmp/tmpw7y1a_ew']' returned non-zero exit status 1.
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "<stdin>", line 16, in <module>
    File "/tmp/tmp0lxiu8hj/.venv/lib/python3.9/site-packages/build/__init__.py", line 310, in get_requires_for_build
      return set(get_requires(config_settings))
    File "/nix/store/wnrc4daqbd6v5ifqlxsj75ky8556zy0p-python3-3.9.12/lib/python3.9/contextlib.py", line 137, in __exit__
      self.gen.throw(typ, value, traceback)
    File "/tmp/tmp0lxiu8hj/.venv/lib/python3.9/site-packages/build/__init__.py", line 430, in _handle_backend
      raise BuildBackendException(exception, f'Backend subproccess exited when trying to invoke {hook}')
  build.BuildBackendException: Backend subproccess exited when trying to invoke get_requires_for_build_wheel
  input was : import build
  import build.env
  import pep517
  
  source = '/tmp/tmpcaz69rrj/python-zulip-api-0.8.2'
  dest = '/tmp/tmp0lxiu8hj/dist'
  
  with build.env.IsolatedEnvBuilder() as env:
      builder = build.ProjectBuilder(
          srcdir=source,
          scripts_dir=env.scripts_dir,
          python_executable=env.executable,
          runner=pep517.quiet_subprocess_runner,
      )
      env.install(builder.build_system_requires)
      env.install(builder.get_requires_for_build('wheel'))
      builder.metadata_path(dest)
  
  No fallback setup.py file was found to generate egg_info.

  at /nix/store/w4gas8kya80hphl1a3r9gnkvcwjqffyg-python3.9-poetry-1.2.0b1-177-gb8db5bd5/lib/python3.9/site-packages/poetry/inspection/info.py:602 in get_pep517_metadata
      598│             # fallback to egg_info if setup.py available
      599│             logger.debug("PEP517 build failed: %s", e)
      600│             setup_py = path / "setup.py"
      601│             if not setup_py.exists():
    → 602│                 raise PackageInfoError(
      603│                     path,
      604│                     e,
      605│                     "No fallback setup.py file was found to generate egg_info.",
      606│                 )

The following error occurred when trying to handle this error:


  RuntimeError

  Unable to determine package info from path: /tmp/tmphx126w3m/0.8.2.zip

  at /nix/store/w4gas8kya80hphl1a3r9gnkvcwjqffyg-python3.9-poetry-1.2.0b1-177-gb8db5bd5/lib/python3.9/site-packages/poetry/puzzle/provider.py:323 in get_package_from_file
      319│             package = PackageInfo.from_path(path=file_path).to_package(
      320│                 root_dir=file_path
      321│             )
      322│         except PackageInfoError:
    → 323│             raise RuntimeError(
      324│                 f"Unable to determine package info from path: {file_path}"
      325│             )
      326│ 
      327│         return package

@ashnair1
Copy link
Contributor Author

Seems #5648 has superseded this.

@andersk
Copy link
Contributor

andersk commented May 21, 2022

poetry add 'git+https://github.com/zulip/[email protected]#subdirectory=zulip' still doesn’t work on master at 80b03ec which has #5648 merged, while it does work with this PR (with the caveats in my previous comment).

Output on master
$ poetry init -n
$ poetry add 'git+https://github.com/zulip/python-zulip-api.git#subdirectory=zulip'

Updating dependencies
Resolving dependencies... (0.2s)

Writing lock file

Package operations: 0 installs, 1 update, 0 removals

  • Updating zulip (0.8.2 d26416a -> 0.8.2 d26416a): Failed

  CalledProcessError

  Command '['/home/anders/.cache/pypoetry/virtualenvs/q-jRydgEFu-py3.9/bin/python', '/nix/store/l6d4l1irq9p5cbxfjb211sycfygirdj6-python3.9-virtualenv-20.14.0/lib/python3.9/site-packages/virtualenv/seed/wheels/embed/pip-22.0.4-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/home/anders/.cache/pypoetry/virtualenvs/q-jRydgEFu-py3.9', '--upgrade', '--no-deps', '/home/anders/.cache/pypoetry/virtualenvs/q-jRydgEFu-py3.9/src/python-zulip-api']' returned non-zero exit status 1.

  at /nix/store/wnrc4daqbd6v5ifqlxsj75ky8556zy0p-python3-3.9.12/lib/python3.9/subprocess.py:528 in run
       524│             # We don't call process.wait() as .__exit__ does that for us.
       525│             raise
       526│         retcode = process.poll()
       527│         if check and retcode:
    →  528│             raise CalledProcessError(retcode, process.args,
       529│                                      output=stdout, stderr=stderr)
       530│     return CompletedProcess(process.args, retcode, stdout, stderr)
       531│ 
       532│ 

The following error occurred when trying to handle this error:


  EnvCommandError

  Command ['/home/anders/.cache/pypoetry/virtualenvs/q-jRydgEFu-py3.9/bin/python', '/nix/store/l6d4l1irq9p5cbxfjb211sycfygirdj6-python3.9-virtualenv-20.14.0/lib/python3.9/site-packages/virtualenv/seed/wheels/embed/pip-22.0.4-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/home/anders/.cache/pypoetry/virtualenvs/q-jRydgEFu-py3.9', '--upgrade', '--no-deps', '/home/anders/.cache/pypoetry/virtualenvs/q-jRydgEFu-py3.9/src/python-zulip-api'] errored with the following return code 1, and output: 
  Processing /home/anders/.cache/pypoetry/virtualenvs/q-jRydgEFu-py3.9/src/python-zulip-api
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'error'
    error: subprocess-exited-with-error
    
    × Getting requirements to build wheel did not run successfully.
    │ exit code: 1
    ╰─> [14 lines of output]
        error: Multiple top-level packages discovered in a flat-layout: ['stubs', 'zulip', 'zulip_bots', 'zulip_botserver', 'packaged_helloworld'].
        
        To avoid accidental inclusion of unwanted files or directories,
        setuptools will not proceed with this build.
        
        If you are trying to create a single distribution with multiple packages
        on purpose, you should not rely on automatic discovery.
        Instead, consider the following options:
        
        1. set up custom discovery (`find` directive with `include` or `exclude`)
        2. use a `src-layout`
        3. explicitly set `py_modules` or `packages` with a list of names
        
        To find more information, look for "package discovery" on setuptools docs.
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  

  at /nix/store/kav56g7mddb48s7sxh26g97sf5l1xvkj-python3.9-poetry-1.2.0b1-179-g80b03ece/lib/python3.9/site-packages/poetry/utils/env.py:1443 in _run
      1439│                 output = subprocess.check_output(
      1440│                     cmd, stderr=subprocess.STDOUT, env=env, **kwargs
      1441│                 )
      1442│         except CalledProcessError as e:
    → 1443│             raise EnvCommandError(e, input=input_)
      1444│ 
      1445│         return decode(output)
      1446│ 
      1447│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int | None:

The following error occurred when trying to handle this error:


  PoetryException

  Failed to install /home/anders/.cache/pypoetry/virtualenvs/q-jRydgEFu-py3.9/src/python-zulip-api

  at /nix/store/kav56g7mddb48s7sxh26g97sf5l1xvkj-python3.9-poetry-1.2.0b1-179-g80b03ece/lib/python3.9/site-packages/poetry/utils/pip.py:55 in pip_install
       51│ 
       52│     try:
       53│         return environment.run_pip(*args)
       54│     except EnvCommandError as e:
    →  55│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       56│ 

@ashnair1
Copy link
Contributor Author

Interesting. Thanks for the comments. I'll take a look.

@ghilesmeddour
Copy link

Any update on this?

@ashnair1
Copy link
Contributor Author

ashnair1 commented Jul 22, 2022

Pinging @neersighted @abn @finswimmer

@neersighted
Copy link
Member

neersighted commented Jul 22, 2022

Hey all, this is still a priority to get merged/a blocker for 1.2 -- and it's still at or near the top of list of priorities. However, most of the core team has been indisposed due to non-Poetry circumstances for the last couple weeks. On my end my health and free time are both looking up in the near future, so I hope to get to this in the next week barring someone else striking first.

Edit: Please don't repeatedly ping, posting on the issue will notify the interested maintainers.

@radoering
Copy link
Member

OK, I'm going to try to push this over the finish line. I tend not to squash all the commits so let's try to keep it clean.

One thing in advance, which probably helps to understand the following proposals: In the best case, each commit changing a source file should add a test that requires that change (at least in my opinion 😉).

I'd suggest proceeding as follows. At first please rebase on current master. Then, let's try to sort out the commits a bit:

  • Add tests for git urls with subdirectory
    • The change in test_pip_installer.py probably does not belong to this but to the next commit.
  • Add test for parsing dependency specification
  • Add tests for git urls with subdirectory
  • Add subdirectory reference to locker.py
    • This one needs a test in test_locker.py
  • Add subdirectory refs to installed_repository
    • This one needs a test in test_installed_repository.py
  • Add support for subdirectories in git urls
    • Probably, can be squashed with Add test for writing pep610 urls
    • There are two changes in this commit. It seems like the first change is reverted in Add test for writing pep610 urls. I suppose the second is required for the test in Add test for writing pep610 urls.

I hope this is not too much to ask. If it is, I can also offer my support.

minottic added a commit to paulscherrerinstitute/scilog that referenced this pull request Aug 8, 2022
minottic added a commit to paulscherrerinstitute/scilog that referenced this pull request Aug 8, 2022
minottic added a commit to paulscherrerinstitute/scilog that referenced this pull request Aug 8, 2022
@ashnair1
Copy link
Contributor Author

ashnair1 commented Aug 9, 2022

So to recap, the commit order should be as follows:

  • Add test for parsing dependency specification
  • Add tests for git urls with subdirectory (remove change in test_pip_installer.py)
  • Add subdirectory reference to locker.py + Add locker test
  • Add subdirectory refs to pip installer (add the change to test_pip_installer here)
  • Add subdirectory refs to installed_repository + Add installed repo test
  • Add support for subdirectories in git urls + Add test for writing pep610 urls

Note: Text in bold will be the commit message.

Is that correct?

@radoering
Copy link
Member

Correct. At first new tests that don't require any change in source and afterwards the rest in a sensible order. (A commit should not depend on a subsequent commit so that tests are succeeding all the time when stepping through the commits.)

@ashnair1
Copy link
Contributor Author

ashnair1 commented Aug 9, 2022

Done. Ready for review

Copy link
Member

@radoering radoering left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just added some test coverage. LGTM now.

Copy link

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 29, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Dependency rooted in a subfolder of a GIT repository
9 participants