Bundler 1.13 complains about corrupt lock when no lock exists

[indirect]

$ git clone https://github.com/chef/ffi-yajl
$ cd ffi-yajl
$ git checkout d343b78cc803636ecaa152a0517f4beccf2220db
$ bundle _1.13.1_ install --without development development_extras rbx
[...]
Your Gemfile.lock is corrupt. The following gem is missing from the DEPENDENCIES section: 'ffi'

This works just fine on Bundler 1.12.x, but on Bundler 1.13.x it's an error.

Environment

    Bundler   1.13.1
    Rubygems  2.5.1
    Ruby      2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin15]
    GEM_HOME  /Users/andre/.gem/ruby/2.3.1
    GEM_PATH  /Users/andre/.gem/ruby/2.3.1:/opt/rubies/2.3.1/lib/ruby/gems/2.3.0
    Git       2.7.3

Bundler settings

    gem.coc
      Set for the current user (/Users/andre/.bundle/config): "true"
    gem.mit
      Set for the current user (/Users/andre/.bundle/config): "true"
    gem.test
      Set for the current user (/Users/andre/.bundle/config): "rspec"
    build.eventmachine
      Set for the current user (/Users/andre/.bundle/config): "--with-cppflags=-I/usr/local/opt/openssl/include"
    local.rails
      Set for the current user (/Users/andre/.bundle/config): "/Users/andre/src/rails/rails"
    local.compact_index
      Set for the current user (/Users/andre/.bundle/config): "/Users/andre/src/bundler/compact_index"
    path.system
      Set for the current user (/Users/andre/.bundle/config): "true"
    local.lita-tweet
      Set for the current user (/Users/andre/.bundle/config): "/Users/andre/src/indirect/lita-tweet"
    build.nokogiri
      Set for the current user (/Users/andre/.bundle/config): "\"--use-system-libraries --with-xml2-include=/usr/include/libxml2"
    path
      Set for your local app (/Users/andre/src/bundler/testcases/4467/ffi-yajl/.bundle/config): ".bundle"
    disable_shared_gems
      Set for your local app (/Users/andre/src/bundler/testcases/4467/ffi-yajl/.bundle/config): "true"
    without
      Set for your local app (/Users/andre/src/bundler/testcases/4467/ffi-yajl/.bundle/config): "development:development_extras:rbx"
    jobs
      Set via BUNDLE_JOBS: "8"

Gemfile

    source "https://rubygems.org"

    gemspec name: "ffi-yajl"

    platforms :rbx do
      gem 'rubysl', '~> 2.0'
    end

    group :development do
      # for testing loading concurrently with yajl-ruby, not on jruby
      # gem 'yajl-ruby', platforms: [ :ruby, :mswin, :mingw ]
      gem 'github_changelog_generator'
    end

    group :development_extras do
      gem 'rubocop', '= 0.32.1'
      gem 'reek', '= 1.3.7'
      gem 'test-kitchen', '~> 1.2'
      gem 'kitchen-digitalocean'
      gem 'kitchen-ec2'
      gem 'kitchen-vagrant'
    end

Gemfile.lock

    <No /Users/andre/src/bundler/testcases/4467/ffi-yajl/Gemfile.lock found>

ffi-yajl.gemspec

    gemspec = eval(IO.read(File.expand_path(File.join(File.dirname(__FILE__), "ffi-yajl.gemspec.shared"))))

    gemspec.platform = Gem::Platform::RUBY
    gemspec.extensions = %w{ext/ffi_yajl/ext/encoder/extconf.rb ext/ffi_yajl/ext/parser/extconf.rb ext/ffi_yajl/ext/dlopen/extconf.rb}

    gemspec.add_development_dependency "ffi", "~> 1.5"

    gemspec

[indirect]

/cc @segiddins

[indirect]

@trevorturk since you're seeing this specific issue (of --without development causing issues with transitive development dependencies), it would be great if you could follow ISSUES and supply your Gemfile, lock, and steps to reproduce. That way we can make sure the fix also fixes your situation. 👍

[indirect]

Further information: as per information from @trevorturk in #4467, the issue this particular ticket is triggered specifically by --without development when some gems in the bundle have development dependencies.

[trevorturk]

Hmmmmmmmmmmmm I'm sorry, I'm having trouble making a repro script. I'll give it another try tomorrow and report back.

[segiddins]

I've reproduced this and have been digging in, it appears that having the multiple gem specs in the repo is related

[trevorturk]

Thanks, @segiddins! It occurred to me overnight that I personally ran into this issue only when I was changing ruby versions. So, I have a deployed app, then I change the ruby version, then I hit this issue when deploying again. I was wondering if the "ruby version" or "bundled with" lines in Gemfile.lock might cause a problem, but maybe that's silly.

Anyway, I'll try to work up a script that runs through the bundle --deployment, ruby version change, bundle --deployment again to see if I can get a reproducible issue. It could be there are multiple things going on. My attempt to make a repro script last night didn't work, but I'm not ready to give up just yet :)

[segiddins]

@trevorturk no need, I've already opened up #5008 -- just see if that fixes things for you

[trevorturk]

@segiddins I'm having trouble reproducing locally and I'm not sure how to get an unreleased gem onto my remote system. Any chance we could try your fix in #5008 via a --pre or --beta so I can test the fix on my known-to-be-failing remote system? Or do you know an easy-ish way for me to remotely install bundler with your fix included?

[segiddins]

git clone https://github.com/bundler/bundler /path/to/bundler
ruby -I/path/to/bundler/lib /path/to/bundler/exe/bundle $bundler_args

[trevorturk]

Thanks for that! I'm sorry to report, however, that my issue does not appear to be fixed. I'll spend some more time trying to reproduce with a local script, but please do let me know if you have any other ideas of what I might be able to do to help debug.

[trevorturk]

Here's the output from bundle env on the remote system, truncated to exclude private gems:

Environment    Bundler   1.13.1    Rubygems  2.6.6    Ruby      2.1.10p492 (2016-04-01 revision 54464) [x86_64-linux]    GEM_HOME  /usr/local/bundle    GEM_PATH      Git       2.1.4Bundler settings    frozen      Set for the current user (/root/.bundle/config): "1"    build.nokogiri      Set for the current user (/root/.bundle/config): "--use-system-libraries"    app_config      Set via BUNDLE_APP_CONFIG: "/usr/local/bundle"    bin      Set via BUNDLE_BIN: "/usr/local/bundle/bin"    path      Set via BUNDLE_PATH: "/usr/local/bundle"    silence_root_warning      Set via BUNDLE_SILENCE_ROOT_WARNING: "1"Gemfile    source 'https://rubygems.org'    ruby '2.1.10'        

... and I believe these are the relevant commands I'm running via a Dockerfile:

bundle config --global frozen 1
bundle config build.nokogiri --use-system-libraries
git clone https://github.com/bundler/bundler /usr/src/bundler
ruby -I/usr/src/bundler /usr/src/bundler/exe/bundle install --deployment --without 'development test' -j4

[trevorturk]

Ah, luckily I seem to be able to reproduce the issue with only a single gem now. Here are my Gemfile and Gemfile.lock and I'll include the difficult-to-read output from bundle env and the error I'm getting:

source 'https://rubygems.org'
ruby '2.1.10'

gem 'sinatra-sequel'

GEM
  remote: https://rubygems.org/
  specs:
    rack (1.6.0)
    rack-protection (1.5.3)
      rack
    sequel (4.38.0)
    sinatra (1.4.5)
      rack (~> 1.4)
      rack-protection (~> 1.4)
      tilt (~> 1.3, >= 1.3.4)
    sinatra-sequel (0.9.0)
      sequel (>= 3.2.0)
      sinatra (>= 0.9.4)
    tilt (1.4.1)

PLATFORMS
  ruby

DEPENDENCIES
  sinatra-sequel

RUBY VERSION
   ruby 2.1.10p492

BUNDLED WITH
   1.13.1

Environment    Bundler   1.13.1    Rubygems  2.6.6    Ruby      2.1.10p492 (2016-04-01 revision 54464) [x86_64-linux]    GEM_HOME  /usr/local/bundle    GEM_PATH      Git       2.1.4Bundler settings    frozen      Set for the current user (/root/.bundle/config): "1"    build.nokogiri      Set for the current user (/root/.bundle/config): "--use-system-libraries"    app_config      Set via BUNDLE_APP_CONFIG: "/usr/local/bundle"    bin      Set via BUNDLE_BIN: "/usr/local/bundle/bin"    path      Set via BUNDLE_PATH: "/usr/local/bundle"    silence_root_warning      Set via BUNDLE_SILENCE_ROOT_WARNING: "1"Gemfile    source 'https://rubygems.org'    ruby '2.1.10'        gem 'sinatra-sequel'Gemfile.lock    GEM      remote: https://rubygems.org/      specs:        rack (1.6.0)        rack-protection (1.5.3)          rack        sequel (4.38.0)        sinatra (1.4.5)          rack (~> 1.4)          rack-protection (~> 1.4)          tilt (~> 1.3, >= 1.3.4)        sinatra-sequel (0.9.0)          sequel (>= 3.2.0)          sinatra (>= 0.9.4)        tilt (1.4.1)        PLATFORMS      ruby        DEPENDENCIES      sinatra-sequel        RUBY VERSION       ruby 2.1.10p492        BUNDLED WITH       1.13.1

ruby -I/usr/src/bundler /usr/src/bundler/exe/bundle install --deployment --without 'development test' -j4
Fetching gem metadata from https://rubygems.org/
Fetching version metadata from https://rubygems.org/
Installing tilt 1.4.1
Installing rack 1.6.0
Installing sequel 4.38.0
Your Gemfile.lock is corrupt. The following gem is missing from the DEPENDENCIESsection: 'bacon'
The command '/bin/sh -c ruby -I/usr/src/bundler /usr/src/bundler/exe/bundle install --deployment --without 'development test' -j4' returned a non-zero code: 20

You can see a bit more detail in my comment here: #4467 (comment)

Apologies, I know this bug report is difficult to follow. I'll spend some more time trying to reproduce locally here, but I haven't had any luck so far, sadly. I'm more than happy to try a few different possible fixes and/or to change my settings etc if it'll help debug. Please do let me know.

In any case, thanks for your patience and attention, @segiddins -- it's very much appreciated, but no worries if you're reached the end of your patience. I have been able to find a workaround as mentioned in my comment on the other issue, so there's no pressure on my end, I'd just like to help if I can.

[segiddins]

You need to have -I/usr/src/bundler/lib

[trevorturk]

Ahh, sorry, I fat fingered that one. Sadly, fixing that doesn't seem to help:

ruby -I/usr/src/bundler/lib /usr/src/bundler/exe/bundle install --deployment --without 'development test' -j4
Your Gemfile.lock is corrupt. The following gem is missing from the DEPENDENCIESsection: 'bacon'
The command '/bin/sh -c ruby -I/usr/src/bundler/lib /usr/src/bundler/exe/bundle install --deployment --without 'development test' -j4' returned a non-zero code: 20

[segiddins]

OK, I'm really sorry, I forgot to mention the important step: git checkout seg-lazy-specification-materialize-platform

[trevorturk]

Ha! Sorry I should have noticed that, too. Sorry x2 I don't think that did the trick:

Step 11 : RUN git clone https://github.com/bundler/bundler /usr/src/bundler && cd /usr/src/bundler &&  git checkout seg-lazy-specification-materialize-platform
Cloning into '/usr/src/bundler'...
Switched to a new branch 'seg-lazy-specification-materialize-platform'
Branch seg-lazy-specification-materialize-platform set up to track remote branch seg-lazy-specification-materialize-platform from origin.
Step 12 : RUN ruby -I/usr/src/bundler/lib /usr/src/bundler/exe/bundle install --deployment --without 'development test' -j4
Fetching gem metadata from https://rubygems.org/
Fetching version metadata from https://rubygems.org/
Installing tilt 1.4.1
Installing rack 1.6.0
Installing sequel 4.38.0
Your Gemfile.lock is corrupt. The following gem is missing from the DEPENDENCIESsection: 'bacon'
The command '/bin/sh -c ruby -I/usr/src/bundler/lib /usr/src/bundler/exe/bundle install --deployment --without 'development test' -j4' returned a non-zero code: 20

[trevorturk]

I'll keep trying to repro locally -- I'd really like to have a script for you to test.

In any case I wanted to note again that this change to my Gemfile gets around the issue:

# gem 'sinatra-sequel'
gem 'sinatra-sequel', github: 'trevorturk/sinatra-sequel', branch: 'no-bacon'

This uses a branch rtomayko/sinatra-sequel@master...trevorturk:no-bacon where I comment out one line from the sinatra-sequel.gemspec file: s.add_development_dependency 'bacon' so the problem must be wrapped up in reading the gemspec somehow.

[salemine]

Is it possible that the problem here is the version of the gem command and not bundler ? I have found that rubygems version 2.5.1 produces an erroneous Gemfile.lock. Regenerating Gemfile.lock using gem version 2.5.0 or 2.5.2 would fix it. There was some work on the Rubygems dependency graph around that time but I did not look further.

I've added some gists to illustrate:
https://gist.github.com/salemine/5284ef41f1c152da26fc18406f41b459 (2.5.0)
https://gist.github.com/salemine/9010f0a97ebf398714b6a8d652fac3f8 (2.5.1) *broken
https://gist.github.com/salemine/1e9b30670c79d4f76e9c14c273f827d6 (2.5.2)

[trevorturk]

@salemine it doesn't seem to be an issue with the gem command in my case, at least. I was on gem --version 2.2.5 and upgraded to 2.5.2 and see no change. I'm still using that no-bacon fork which seems to be a reliable workaround.

Unfortunately, I still haven't been able to make a working reproduction script locally. I really sorry about that -- I'll keep trying, but I'm out of ideas for today. If anyone wants to team up I'm happy to screen share etc to try and debug where I can reproduce using Codeship CI easily.

[trevorturk]

@homu I don't believe #5008 fixes this issue -- I tested @segiddin's branch.

[segiddins]

@trevorturk please open a new issue

[trevorturk]

I'll report back to #4467 if I'm able to make a repro script, otherwise I'll watch for updates there. Thanks!

[lamont-granquist]

bundler-1.13.2 seems to have fixed the issue we were seeing:

% bundle install --without development
Fetching gem metadata from https://rubygems.org/........
Fetching version metadata from https://rubygems.org/..
Fetching dependency metadata from https://rubygems.org/.
Resolving dependencies...
Using libyajl2 1.2.0
Using bundler 1.13.2
Using ffi-yajl 2.3.0 from source at `.`
Bundle complete! 16 Gemfile dependencies, 3 gems now installed.
Gems in the groups development, development_extras and rbx were not installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.