Regression 2018-11-14 when using --pre flag with particular Pipfile syntax | TypeError: list indices must be integers or slices, not tuple

[kavdev]

Issue description

I started getting the following error when attempting to use pipenv versions 2018-11-14 (still exists in 2018-11-26) when installing with the --pre flag: pipenv install --pre.

Expected result

Pipenv installs packages.

Actual result

Pipenv fails with TypeError (this is with the --verbose flag)

Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/cli/command.py", line 254, in install
    editable_packages=state.installstate.editables,
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 1862, in do_install
    project.update_settings({"allow_prereleases": pre})
  File "/usr/local/lib/python3.6/site-packages/pipenv/project.py", line 583, in update_settings
    p["pipenv"] = settings
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/tomlkit/container.py", line 524, in __setitem__
    self.append(key, value)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/tomlkit/container.py", line 176, in append
    return self._insert_after(key_after, key, item)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/tomlkit/container.py", line 246, in _insert_after
    current_item = self._body[idx][1]
TypeError: list indices must be integers or slices, not tuple

Steps to replicate

1. Ensure pipenv version 2018-11-26 is installed.
2. Run pipenv install --pre
3. Observe TypeError

---

Support diagnostics file available upon request.

[frostming]

Hmm, that is probably an issue. Can you please provide the Pipfile content?

[kavdev]

Interesting. Turns out it's caused by pipenv syntax that seems to be no longer supported (the entire first packages section no longer gets locked). That might be another bug and could get annoying -- @pyupio uses this format when submitting automated PRs.

File that causes the bug to surface

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[packages]

babel = "==2.6.0"
boto3 = "==1.9.47"
celery = "==4.2.1"
colorama = "==0.4.0"
coreapi = "==2.3.3"
dj-database-url = "==0.5.0"
djangorestframework = "==3.9.0"
django = "==2.1.3"
django-axes = "==4.4.2"
django-celery-results = "==1.0.4"
django-clever-selects = "==0.8.2"
django-crispy-forms = "==1.7.2"
django-choices = "==1.6.1"
django-extra-views = "==0.11.0"
django-filter = "==2.0.0"
django-hijack = "==2.1.10"
django-hijack-admin = "==2.1.10"
django-js-reverse = "==0.8.2"
django-model-utils = "==3.1.2"
django-phonenumber-field = "==2.1.0"
django-polymorphic = "==2.0.3"
django-redis-cache = "==1.8.1"
django-role-permissions = "==2.2.0"
django-s3direct = "==1.0.4"
django-storages = "==1.7.1"
django-tables2 = "==1.21.2"
django-tagulous = "==0.13.2"
django-webpack-loader = "==0.6.0"
django-widget-tweaks = "==1.4.3"
facebook_business = "==3.2.3"
googleads = "==15.0.0"
markdown = "==3.0.1"
phonenumbers = "==8.10.0"
pillow = "==5.3.0"
psycopg2-binary = "==2.7.6.1"
pygments = "==2.2.0"
pyssim = "==0.4"
python-dotenv = "==0.9.1"
pytz = "==2018.7"
raven = "==6.9.0"
sendgrid-django = "==4.2.0"
slacker = "==0.9.65"
termcolor = "==1.1.0"
tqdm = "==4.28.1"
twitter-ads = "==3.0.0"
waitress = "==1.1.0"


[dev-packages]

coverage = "==4.5.2"
selenium = "==3.141.0"
tblib = "==1.3.2"
flake8 = "==3.6.0"
django-debug-toolbar = "==1.10.1"
django-extensions = "==2.1.3"
unittest-xml-reporting = "==2.2.0"


[requires]

python_version = "3.6"


[packages.django-static-precompiler]

extras = [ "libsass",]
version = "==1.8.2"


[packages.whitenoise]

extras = [ "brotli",]
version = "==4.0"

Reformatted file that works

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[requires]
python_version = "3.6"

[packages]
babel = "==2.6.0"
boto3 = "==1.9.52"
celery = "==4.2.1"
colorama = "==0.4.1"
coreapi = "==2.3.3"
dj-database-url = "==0.5.0"
djangorestframework = "==3.9.0"
django = "==2.1.3"
django-axes = "==4.4.2"
django-celery-results = "==1.0.4"
django-clever-selects = "==0.8.2"
django-crispy-forms = "==1.7.2"
django-choices = "==1.6.1"
django-extra-views = "==0.11.0"
django-filter = "==2.0.0"
django-hijack = "==2.1.10"
django-hijack-admin = "==2.1.10"
django-js-reverse = "==0.8.2"
django-model-utils = "==3.1.2"
django-phonenumber-field = "==2.1.0"
django-polymorphic = "==2.0.3"
django-redis-cache = "==1.8.1"
django-role-permissions = "==2.2.0"
django-s3direct = "==1.0.4"
django-static-precompiler = {version = "==1.8.2", extras = ["libsass"]}
django-storages = "==1.7.1"
django-tables2 = "==1.21.2"
django-tagulous = "==0.13.2"
django-webpack-loader = "==0.6.0"
django-widget-tweaks = "==1.4.3"
facebook_business = "==3.2.4"
googleads = "==15.0.0"
markdown = "==3.0.1"
phonenumbers = "==8.10.1"
pillow = "==5.3.0"
psycopg2-binary = "==2.7.6.1"
pygments = "==2.3.0"
pyssim = "==0.4"
python-dotenv = "==0.9.1"
pytz = "==2018.7"
raven = "==6.9.0"
sendgrid-django = "==4.2.0"
slacker = "==0.10.0"
termcolor = "==1.1.0"
tqdm = "==4.28.1"
twitter-ads = "==3.0.0"
waitress = "==1.1.0"
whitenoise = {version = "==4.0", extras = ["brotli"]}

[dev-packages]
coverage = "==4.5.2"
selenium = "==3.141.0"
tblib = "==1.3.2"
flake8 = "==3.6.0"
django-debug-toolbar = "==1.10.1"
django-extensions = "==2.1.4"
unittest-xml-reporting = "==2.2.0"

[frostming]

Aha, that it! Thanks for the details, it helps.

[kavdev]

@frostming no problem!

[frostming]

The root cause:

packages section is splitted in several places, in this case, appending a new table(pipenv section) will throw an error.

The workaround is to combine the packages section in one place, and I pushed a PR to the upstream python-poetry/tomlkit#34 to fix this.

[techalchemy]

btw is that valid toml? are you allowed to do that? toml confuses me

[techalchemy]

also @frostming can you let me know when this fix gets released

[frostming]

@techalchemy The fix has been in tomlkit's master but not released yet. Since this issue is corner case and has a workaround available, i'll update the vendor after the upstream is released.

BTW This toml is allowed by spec