Bundler 2 support by schneems · Pull Request #850 · heroku/heroku-buildpack-ruby

schneems · 2019-02-05T17:24:25Z

Detect bundler version

Problem 1

Bundler 2.x cannot be used with a Gemfile.lock that specifies bundler 1.x:

BUNDLED WITH
   1.16.4

If you try, then Bundler 2.x will look for an installed bundler 1.x version and try to use it. If no such version exists on the system then an error is thrown:

can't find gem bundler (>= 0.a) with executable bundle (Gem::GemNotFoundException)

Fix bundler rubygems binstub not properly looking for bundler ruby/rubygems#2426
[BudlerVersionFinder] set .filter! and .compatible? to match only on major versions ruby/rubygems#2515

Problem 2

Likewise a version of bundler 1.x cannot be used on a project where a BUNDLED WITH specifies 2.x:

$ bundle -v
1.17.3
$ cat Gemfile.lock | grep -A 1 BUNDLED WITH
BUNDLED WITH
  2.0.1
$ bundle
Traceback (most recent call last):
        2: from /Users/rschneeman/.gem/ruby/2.6.0/bin/bundle:23:in `<main>'
        1: from /Users/rschneeman/.rubies/ruby-2.6.0/lib/ruby/2.6.0/rubygems.rb:302:in `activate_bin_path'
/Users/rschneeman/.rubies/ruby-2.6.0/lib/ruby/2.6.0/rubygems.rb:283:in `find_spec_for_exe': Could not find 'bundler' (2.0.0) required by your /private/tmp/default_ruby/Gemfile.lock. (Gem::GemNotFoundException)
To update to the latest version installed on your system, run `bundle update --bundler`.
To install the missing version, run `gem install bundler:2.0.0`

This is due to a bug in Rubygems 3.x ruby/rubygems#2592

Proposed Solution

This PR implements a solution where we have a different "blessed" version of bundler for each major version.

The way this is implemented is to read in a Gemfile.lock and to read in the BUNDLED WITH value. Then the major version is pulled out and converted into a "blessed" version.

This allows for multiple major versions of bundler to be used on Heroku without sacrificing stability.

## Problem 1 Bundler 2.x cannot be used with a Gemfile.lock that specifies bundler 1.x: ``` BUNDLED WITH 1.16.4 ``` If you try, then Bundler 2.x will look for an installed bundler 1.x version and try to use it. If no such version exists on the system then an error is thrown: ``` can't find gem bundler (>= 0.a) with executable bundle (Gem::GemNotFoundException) ``` - ruby/rubygems#2426 - ruby/rubygems#2515 ## Problem 2 Likewise a version of bundler 1.x cannot be used on a project where a `BUNDLED WITH` specifies 2.x: ``` $ bundle -v 1.17.3 $ cat Gemfile.lock | grep -A 1 BUNDLED WITH BUNDLED WITH 2.0.1 $ bundle Traceback (most recent call last): 2: from /Users/rschneeman/.gem/ruby/2.6.0/bin/bundle:23:in `<main>' 1: from /Users/rschneeman/.rubies/ruby-2.6.0/lib/ruby/2.6.0/rubygems.rb:302:in `activate_bin_path' /Users/rschneeman/.rubies/ruby-2.6.0/lib/ruby/2.6.0/rubygems.rb:283:in `find_spec_for_exe': Could not find 'bundler' (2.0.0) required by your /private/tmp/default_ruby/Gemfile.lock. (Gem::GemNotFoundException) To update to the latest version installed on your system, run `bundle update --bundler`. To install the missing version, run `gem install bundler:2.0.0` ``` This is due to a bug in Rubygems 3.x ruby/rubygems#2592 ## Proposed Solution This PR implements a solution where we have a different "blessed" version of bundler for each major version. The way this is implemented is to read in a Gemfile.lock and to read in the `BUNDLED WITH` value. Then the major version is pulled out and converted into a "blessed" version. This allows for multiple major versions of bundler to be used on Heroku without sacrificing stability.

Usually this would take a LONG time as the test suite when run serially takes about an hour. The other thing that is tough about this, is it only fails on CI and not locally. On a hunch I guessed the failing behavior came only from `spec/helpers` which is all unit tested and therefore pretty fast. This allowed me to actually find the failing order. Via `$ heroku ci:debug` First I ran: ``` $ bundle exec rspec spec/helpers --bisect --seed 1 ``` The `--seed` argument is used because otherwise the order is randomized on every run. It doesn't matter what value is used as long as it's consistent. This produce a "minimal failing example" of running: ``` $ rspec './spec/helpers/fetcher_spec.rb[1:1]' './spec/helpers/rails_runner_spec.rb[1:3,1:5]' './spec/helpers/rake_runner_spec.rb[1:1,1:2]' --seed 1 ``` Sure enough this command fails 100% of the time. I knew the culprit had to be mutating something global. Previously I've been focused on files on disk but considering every example is now run inside of it's own temp directory then it is likely something else. The other culprit could be an environment variable. This lead me to look into the bundler wrapper which sure enoug mutates the env var `BUNDLE_GEMFILE`.

hone · 2019-02-06T23:18:55Z

lib/language_pack/helpers/bundler_wrapper.rb

+      msg << "\n"
+      msg << "```\n"
+      msg << "BUNDLED WITH\n"
+      msg << "   #{version_hash["1"]}\n"


why index 1 instead of 0?

This is the major version. So this will be "1.15.2"

hone · 2019-02-06T23:20:40Z

lib/language_pack/helpers/bundler_wrapper.rb

+    @bundler_tmp.rmtree if @bundler_tmp.directory?

-    if LanguagePack::Ruby::BUNDLER_VERSION  == "1.7.12"
+    if version  == "1.7.12"


should we extract this into a constant somewhere?

Based on this code I don't think it's possible to ever actually trigger this case. Since we hard code a version and we're not using "1.7.12" it won't ever fire.

Let me know if you agree and I can remove it.

hone · 2019-02-06T23:22:37Z

lib/language_pack/helpers/bundler_wrapper.rb

+
+  def major_bundler_version
+    # https://rubular.com/r/uuRpai9IheL68d
+    bundler_version_match = @gemfile_lock_path.read.match(/^BUNDLED WITH$(\r?\n)   (?<major>\d*)\.\d*\.\d*/m)


Should these be + and not *?

Also, scraping BUNDLED_WITH seems awful :(

Should these be +

Great call.

Also, scraping BUNDLED_WITH seems awful :(

Agreed, though It's the least awful thing I could come up with :(

hone · 2019-02-06T23:23:50Z

lib/language_pack/helpers/bundler_wrapper.rb

+    if bundler_version_match
+      bundler_version_match[:major]
+    else
+      "1"


Should we actually fail instead of defaulting?

People can still deploy without having a BUNDLED WITH in their Gemfile.lock. For example in some of our hatchet test apps https://github.com/sharpstone/default_ruby/blob/master/Gemfile.lock.

hone

Overall looks good if scary. I'm not sure what other ramifications we need to take into account for bundler 2 "support".

Since we control the version, this code will never execute unless we regress to a REALLY old version. I'm deleting it.

schneems force-pushed the schneems/bundler-multiple-major-versions branch 11 times, most recently from 9ade75b to 141c176 Compare February 5, 2019 20:35

schneems changed the title ~~WIP Bundler 2 support~~ Bundler 2 support Feb 5, 2019

schneems force-pushed the schneems/bundler-multiple-major-versions branch 3 times, most recently from 82303e4 to edd9334 Compare February 6, 2019 17:28

schneems added 2 commits February 6, 2019 11:33

De-couple BundlerWrapper from LanguagePack::Ruby

45981c0

schneems force-pushed the schneems/bundler-multiple-major-versions branch from edd9334 to a382e1d Compare February 6, 2019 17:33

schneems force-pushed the schneems/bundler-multiple-major-versions branch from a382e1d to 80f3d88 Compare February 6, 2019 17:34

schneems added the languages-health-squad-review label Feb 6, 2019

hone reviewed Feb 6, 2019

View reviewed changes

hone approved these changes Feb 6, 2019

View reviewed changes

schneems added 2 commits February 6, 2019 20:05

Ensure at least 1 digit in version

c1fd23e

Remove code that will never get called

7108542

Since we control the version, this code will never execute unless we regress to a REALLY old version. I'm deleting it.

schneems mentioned this pull request Feb 8, 2019

You must use Bundler 2 or greater with this lockfile rubygems/bundler#6784

Closed

schneems merged commit 1bfeda9 into master Feb 11, 2019

This was referenced Feb 15, 2019

Releases fail with bundler version error rubygems/heroku-buildpack-bundler2#5

Closed

Bundler 2 fails on release command #856

Closed

schneems mentioned this pull request Feb 19, 2019

Warn that this buildpack is not needed rubygems/heroku-buildpack-bundler2#7

Merged

edmorley deleted the schneems/bundler-multiple-major-versions branch August 27, 2020 06:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bundler 2 support#850

Bundler 2 support#850
schneems merged 5 commits intomasterfrom
schneems/bundler-multiple-major-versions

schneems commented Feb 5, 2019 •

edited

Loading

Uh oh!

hone Feb 6, 2019

Uh oh!

schneems Feb 7, 2019

Uh oh!

hone Feb 6, 2019

Uh oh!

schneems Feb 7, 2019

Uh oh!

hone Feb 6, 2019

Uh oh!

hone Feb 6, 2019

Uh oh!

schneems Feb 7, 2019

Uh oh!

hone Feb 6, 2019

Uh oh!

schneems Feb 7, 2019

Uh oh!

hone left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

schneems commented Feb 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem 1

Problem 2

Proposed Solution

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hone left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

schneems commented Feb 5, 2019 •

edited

Loading