[Python] [Pipenv] Extraneous markers cause dependencies to be ignored

[bsamuel-ui]

The relevant part of the Pipfile is:

[packages]
attrs = "*"
numpy = "*"
aiobotocore = {version="*", extras=['boto3']}

And what we're seeing is that Pipfile.lock is updated like so:

         "aiobotocore": {
+            "extras": [
+                "boto3"
+            ],
             "hashes": [ ... ]
         },
          ...
          "boto3": {
              "hashes": [...],
+            "markers": "extra == 'boto3'",
             "version": "==1.7.58"
         },

And similarly, the subdependencies for requests[security] get unnecessary markers added.

This causes in our build:

Installing dependencies from Pipfile.lock (9380c5)…
Ignoring boto3: markers 'extra == "boto3"' don't match your environment
Looking in indexes: https://pypi.python.org/simple, https://s3.amazonaws.com/(bucketname)
Ignoring boto3: markers 'extra == "boto3"' don't match your environment
Looking in indexes: https://pypi.python.org/simple, https://s3.amazonaws.com/(bucketname)
Ignoring cryptography: markers 'extra == "security"' don't match your environment
Looking in indexes: https://pypi.python.org/simple, https://s3.amazonaws.com/(bucketname)
Ignoring pyopenssl: markers 'extra == "security"' don't match your environment
Looking in indexes: https://pypi.python.org/simple, https://s3.amazonaws.com/(bucketname)

Once it's ignoring those dependencies, of course, the build fails after that. (I tried mentioning dependabot in a specific branch that was failing, for some reason it didn't pick up on that, though.)

[bsamuel-ui]

On further investigation, this may be an upstream problem, because I can reproduce the same behavior just by running pipenv lock -d...

[bsamuel-ui]

Looks like it's fixed in master; you might consider downgrading to 2018.10.9 until the next release is out.

[greysteil]

Thanks @bsamuel-ui (and apologies for the slow reply on this issue today).

I believe Dan is planning a Pipenv release for today or tomorrow, so we should be able to have this fixed ASAP. If we don't get that then I'll pin to a commit on master.

[bsamuel-ui]

No worries, you have high standards for responsiveness.

[greysteil]

FYI, I put together dependabot/dependabot-core#775 with the intention of switching Dependabot to the latest Pipenv commit, but found an issue in Pipenv. I've raised an issue for that bug here and will pin us to a commit once it's fixed. 🙂

[bsamuel-ui]

The latest release of Pipenv fixes this.

[greysteil]

Yes! Sorry, should have resolved this 🙂