Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Production dependencies not getting updated by Dependabot #964

Closed
StevenMaude opened this issue Jan 24, 2023 · 10 comments
Closed

Production dependencies not getting updated by Dependabot #964

StevenMaude opened this issue Jan 24, 2023 · 10 comments
Milestone

Comments

@StevenMaude
Copy link
Contributor

StevenMaude commented Jan 24, 2023

Relates to the past issue #579.

We currently have quite outdated dependencies.

Dependabot is currently failing on pyproject.toml:

updater | ERROR <job_585486053> Error processing pydocstyle[toml] (Dependabot::SharedHelpers::HelperSubprocessFailed)
updater | ERROR <job_585486053> Using indexes:
updater | <job_585486053>   https://pypi.org/simple
updater | <job_585486053> 
updater | <job_585486053>                           ROUND 1                           
updater | <job_585486053> /usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pip/_internal/req/req_install.py:866: PipDeprecationWarning: DEPRECATION: Constraints are only allowed to take the form of a package name and a version specifier. Other forms were originally permitted as an accident of the implementation, but were undocumented. The new implementation of the resolver no longer supports these forms. A possible replacement is replacing the constraint with a requirement. Discussion can be found at https://github.com/pypa/pip/issues/8210
updater | <job_585486053>   deprecated(
updater | <job_585486053> Traceback (most recent call last):
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/bin/pip-compile", line 8, in <module>
updater | <job_585486053>     sys.exit(cli())
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
updater | <job_585486053>     return self.main(*args, **kwargs)
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/click/core.py", line 1055, in main
updater | <job_585486053>     rv = self.invoke(ctx)
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
updater | <job_585486053>     return ctx.invoke(self.callback, **ctx.params)
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/click/core.py", line 760, in invoke
updater | <job_585486053>     return __callback(*args, **kwargs)
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
updater | <job_585486053>     return f(get_current_context(), *args, **kwargs)
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/piptools/scripts/compile.py", line 580, in cli
updater | <job_585486053>     results = resolver.resolve(max_rounds=max_rounds)
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/piptools/resolver.py", line 590, in resolve
updater | <job_585486053>     is_resolved = self._do_resolve(
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/piptools/resolver.py", line 622, in _do_resolve
updater | <job_585486053>     resolver.resolve(
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
updater | <job_585486053>     collected = self.factory.collect_root_requirements(root_reqs)
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 481, in collect_root_requirements
updater | <job_585486053>     raise InstallationError(problem)
updater | <job_585486053> pip._internal.exceptions.InstallationError: Constraints cannot have extras
updater | ERROR <job_585486053> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:224:in `run_command'
updater | ERROR <job_585486053> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:282:in `run_pip_compile_command'
updater | ERROR <job_585486053> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:82:in `block (3 levels) in fetch_latest_resolvable_version_string'
updater | ERROR <job_585486053> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:76:in `each'
updater | ERROR <job_585486053> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:76:in `block (2 levels) in fetch_latest_resolvable_version_string'
updater | ERROR <job_585486053> /home/dependabot/common/lib/dependabot/shared_helpers.rb:168:in `with_git_configured'
updater | ERROR <job_585486053> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:72:in `block in fetch_latest_resolvable_version_string'
updater | ERROR <job_585486053> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `block in in_a_temporary_directory'
updater | ERROR <job_585486053> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `chdir'
updater | ERROR <job_585486053> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `in_a_temporary_directory'
updater | ERROR <job_585486053> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:71:in `fetch_latest_resolvable_version_string'
updater | ERROR <job_585486053> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:48:in `latest_resolvable_version'
updater | ERROR <job_585486053> /home/dependabot/python/lib/dependabot/python/update_checker.rb:47:in `latest_resolvable_version'
updater | ERROR <job_585486053> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:74:in `preferred_resolvable_version'
updater | ERROR <job_585486053> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:260:in `preferred_version_resolvable_with_unlock?'
updater | ERROR <job_585486053> /home/dependabot/python/lib/dependabot/python/update_checker.rb:126:in `preferred_version_resolvable_with_unlock?'
updater | ERROR <job_585486053> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:252:in `numeric_version_can_update?'
updater | ERROR <job_585486053> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:202:in `version_can_update?'
updater | ERROR <job_585486053> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:44:in `can_update?'
updater | ERROR <job_585486053> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:486:in `requirements_to_unlock'
updater | ERROR <job_585486053> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:259:in `check_and_create_pull_request'
updater | ERROR <job_585486053> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:109:in `check_and_create_pr_with_error_handling'
updater | ERROR <job_585486053> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:82:in `block in run'
updater | ERROR <job_585486053> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:82:in `each'
updater | ERROR <job_585486053> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:82:in `run'
updater | ERROR <job_585486053> /home/dependabot/dependabot-updater/lib/dependabot/update_files_job.rb:17:in `perform_job'
updater | ERROR <job_585486053> /home/dependabot/dependabot-updater/lib/dependabot/base_job.rb:50:in `run'
updater | ERROR <job_585486053> bin/update_files.rb:23:in `<main>'

We have this dependency because we have a pydocstyle configuration in pyproject.toml.

Running pip-compile locally does work though, but maybe there's some difference in my local setup compared with Dependabot 🤷‍♂️

@StevenMaude
Copy link
Contributor Author

StevenMaude commented Jan 24, 2023

Simple workarounds:

  • Just use a .pydocstyle file instead, live with the additional configuration file and remove the extra specified dependency as it's no longer needed.
  • Add tomli as a separate optional dependency and remove the extra specified dependency — there might be issues if the required version by pydocstyle diverges in future from that which we pin.
  • Just remove toml as an optional dependency because tomli is added by other dependencies anyway, and the version in pydocstyle is currently unpinned.

Also worth noting that pydocstyle will use the standard library tomllib in Python 3.11.

More complicated workaround:

@StevenMaude StevenMaude changed the title Dependencies not getting updated by Dependabot Production dependencies not getting updated by Dependabot Jan 24, 2023
@StevenMaude StevenMaude changed the title Production dependencies not getting updated by Dependabot Dependencies not getting updated by Dependabot Jan 24, 2023
@StevenMaude StevenMaude changed the title Dependencies not getting updated by Dependabot Production dependencies not getting updated by Dependabot Jan 24, 2023
@StevenMaude
Copy link
Contributor Author

StevenMaude commented Jan 24, 2023

Other weird features of this issue:

  • It seems to prevent the requirements.prod.txt from being automatically updated. But the actual dependency causing an error is a dev requirement.
  • coverage[toml] doesn't seem to cause an error, but that's maybe because it's a transitive dependency.

@StevenMaude
Copy link
Contributor Author

StevenMaude commented Feb 16, 2023

I think this is a relevant issue: dependabot/dependabot-core#6550

And I think it also affects a few other of our repositories.

@StevenMaude
Copy link
Contributor Author

The linked Dependabot issue is fixed, so we don't get the:

pip._internal.exceptions.InstallationError: Constraints cannot have extras

any longer. Nor do we don't get associated errors in the panel that shows "last checked" for pyproject.toml.

But I'm still not convinced think it is entirely working, because we still don't get updates for the out-of-date dependencies.

StevenMaude added a commit that referenced this issue Mar 8, 2023
Dependabot keeps failing on this, for whatever reason.

Maybe this will unblock Dependabot from being able to upgrade production
dependencies? (See #964.)
StevenMaude added a commit that referenced this issue Mar 8, 2023
Dependabot keeps failing on this, for whatever reason, in the following
way:

```
updater | INFO <job_621493460> Latest version is 0.18.3
updater | INFO <job_621493460> Requirements to unlock own
updater | INFO <job_621493460> Requirements update strategy bump_versions
updater | INFO <job_621493460> Updating future from 0.18.2 to 0.18.3
…
updater | ERROR <job_621493460> Error processing future (RuntimeError)
updater | ERROR <job_621493460> No files have changed!
```

Maybe this will unblock Dependabot from being able to upgrade production
dependencies? (See #964.)
@StevenMaude
Copy link
Contributor Author

StevenMaude commented Mar 13, 2023

In fact, if you look at Dependabot's version update log, it doesn't even check the production requirements at all. There's no "No update needed" for any of the production requirements.

I think this Dependabot issue might be related, or it's something similar (Dependabot just picks one update ecosystem and use that; so it updates the requirements based on the requirements.dev.in file, and not pyproject.toml.)

It might also be worth trying to ditch the .in file and put the dev requirements into pyproject.toml. That's also more consistent. Here's a blog post with an example.

@evansd
Copy link
Contributor

evansd commented Mar 13, 2023

It might also be worth trying to ditch the .in file and put the dev requirements into pyproject.toml. That's also more consistent. Here's a blog post with an example.

Ooh, this looks nice.

@StevenMaude
Copy link
Contributor Author

Moving everything into pyproject.toml works.

dependabot-prs

@StevenMaude
Copy link
Contributor Author

Moving everything into pyproject.toml works.

That is, it works to get Dependabot back up and checking dependencies.

There's then an issue with transitive dependencies seemingly not being updated. So, the update of the virtualenv package fails because distlib (introduced via virtualenv) doesn't get bumped.

@StevenMaude
Copy link
Contributor Author

StevenMaude added a commit that referenced this issue Mar 16, 2023
Because Dependabot isn't quite working properly: see #964.
@StevenMaude
Copy link
Contributor Author

For now, I've suggested a manual update so at least things are back up to date again: #1156.

evansd added a commit that referenced this issue May 12, 2023
Dependabot still doesn't do this for us and I'm not sure we've yet
figured out why, see:
#964

We can do this manually with:

    pip-compile -U \
      --allow-unsafe --generate-hashes --output-file=requirements.prod.txt \
      pyproject.toml
evansd added a commit that referenced this issue Jun 16, 2023
Dependabot still doesn't do this for us and I'm not sure we've yet
figured out why, see:
#964

We can do this manually with:

    pip-compile -U \
      --allow-unsafe --generate-hashes --output-file=requirements.prod.txt \
      pyproject.toml
evansd added a commit that referenced this issue Oct 17, 2023
Dependabot still isn't doing this for us (see #964).

Closes #1531
@inglesp inglesp added this to the P3 milestone Oct 30, 2023
inglesp added a commit that referenced this issue Nov 2, 2023
Dependabot still isn't doing this for us (see #964).
evansd added a commit that referenced this issue Feb 27, 2024
Dependabot still isn't doing this for us (see #964).
evansd added a commit that referenced this issue Feb 27, 2024
Dependabot still isn't doing this for us (see #964).
@evansd evansd closed this as completed in 0b4ead7 May 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

No branches or pull requests

3 participants