pyup fails to handle --index-url correctly

[thanatos]

pyup appears to process --index-url lines in a requirements.txt file, in so much as it updates itself to point to that URL, instead of the default URL it has for PyPI. However, by default, pyup uses the JSON protocol to talk to PyPI; index URLs, however, are always for the simple protocol, since that's what pip uses.

pyup however, ends up attempting to use the given URL as if it were the URL for a JSON API. This doesn't work (the URL doesn't support that), and you get errors like,

DEBUG:urllib3.connectionpool:https://python-index.example.com:443 "GET /repository/pypi-all/simple/Envelopes HTTP/1.1" 200 1387
Traceback (most recent call last):
  File "/usr/local/bin/pyup", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pyup/cli.py", line 45, in main
    bot.update(branch=branch, initial=initial)
  File "/usr/local/lib/python2.7/dist-packages/pyup/bot.py", line 122, in update
    self.get_all_requirements()
  File "/usr/local/lib/python2.7/dist-packages/pyup/bot.py", line 499, in get_all_requirements
    self.add_requirement_file(req_file.path, sha=sha)
  File "/usr/local/lib/python2.7/dist-packages/pyup/bot.py", line 512, in add_requirement_file
    for other_file in req_file.other_files:
  File "/usr/local/lib/python2.7/dist-packages/pyup/requirements.py", line 121, in other_files
    self._parse()
  File "/usr/local/lib/python2.7/dist-packages/pyup/requirements.py", line 165, in _parse
    self._parse_requirements_txt()
  File "/usr/local/lib/python2.7/dist-packages/pyup/requirements.py", line 132, in _parse_requirements_txt
    self.parse_dependencies(filetypes.requirements_txt)
  File "/usr/local/lib/python2.7/dist-packages/pyup/requirements.py", line 193, in parse_dependencies
    if req.package:
  File "/usr/local/lib/python2.7/dist-packages/pyup/requirements.py", line 387, in package
    self._package = fetch_package(self.name, self.index_server)
  File "/usr/local/lib/python2.7/dist-packages/pyup/package.py", line 13, in fetch_package
    json = r.json()
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 896, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

This was slightly misreported in #303 (it doesn't matter if the private index does or does not support the JSON API; PyUp will crash all the same, since it's using the simple URL instead of the JSON one) and it was closed as there was a somewhat "clever"¹ fix attempted in #305, however, that fix was reverted in #335, so the issue persists.

_{¹in the sense that it attempts to sniff the body in order to figure out how to parse it instead of just using the right protocol for the right URLs. I also don't think the fix proposed would have worked, as it still constructs a URL that is only appropriate for the JSON API, not the simple API. (Specifically, it tacks /json on to the end.)}

[rafaelpivato]

Well reported. We have some issues related to how we do parse requirements and we are looking forward that. The idea is to bring pyupio/dparse more into the game-play and make it stronger on such requirements parsing.

[rafaelpivato]

After going through pull-requests #335, #303 and #305 in special, it is clear that this is not a bug or issue related to dparse but how do we communicate with some PyPi private stores or even some specific packages. This is a bug that must be fixed for sure.