Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to bundle install latest psych 5.0.0 on github actions #409

Closed
2 tasks done
danielwestendorf opened this issue Dec 5, 2022 · 31 comments
Closed
2 tasks done

Comments

@danielwestendorf
Copy link

Ensure the following before filing this issue

  • I verified it reproduces with the latest version with - uses: ruby/setup-ruby@v1 (see Versioning policy)

  • I tried to reproduce the issue locally by following the workflow steps (including all commands done by ruby/setup-ruby, except for Downloading Ruby & Extracting Ruby),
    and it did not reproduce locally (if it does reproduce locally, it's not a ruby/setup-ruby issue)

Are you running on a GitHub-hosted runner or a self-hosted runner?

GitHub-hosted runner

The workflow code or a link to the workflow file

Generated via GH web UI Actions > Ruby > Configure and commenting out the test line.

https://github.com/danielwestendorf/gh-actions-ruby-psych-5/blob/main/.github/workflows/ruby.yml

Link to the log of a failed workflow job, or to a gist with the output

https://github.com/danielwestendorf/gh-actions-ruby-psych-5/actions/runs/3622724366/jobs/6107778167#step:3:44

The command and output of the failing step

bundle install
  /opt/hostedtoolcache/Ruby/2.7.6/x64/bin/bundle config --local path /home/runner/work/gh-actions-ruby-psych-5/gh-actions-ruby-psych-5/vendor/bundle
  /opt/hostedtoolcache/Ruby/2.7.6/x64/bin/bundle lock
  Fetching gem metadata from https://rubygems.org/........
  Resolving dependencies...
  Writing lockfile to /home/runner/work/gh-actions-ruby-psych-5/gh-actions-ruby-psych-5/Gemfile.lock
  Cache key: setup-ruby-bundler-cache-v4-ubuntu-20.04-ruby-2.7.6-Gemfile.lock-32b0ae944f0f9b2d8f3c61f68f6d3cb975db47fe2ae2efbf5bf6fa7b2f6a4dbe
  /opt/hostedtoolcache/Ruby/2.7.6/x64/bin/bundle install --jobs 4
  Fetching gem metadata from https://rubygems.org/........
  Using bundler 2.3.26
  Fetching stringio 3.0.2
  Installing stringio 3.0.2 with native extensions
  Fetching psych 5.0.0
  Installing psych 5.0.0 with native extensions
  Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
  
  current directory:
  /home/runner/work/gh-actions-ruby-psych-5/gh-actions-ruby-psych-5/vendor/bundle/ruby/2.7.0/gems/psych-5.0.0/ext/psych
  /opt/hostedtoolcache/Ruby/2.7.6/x64/bin/ruby -I
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0 -r
  ./siteconf20221205-1835-1muibuj.rb extconf.rb
  checking for yaml.h... no
  yaml.h not found
  *** extconf.rb failed ***
  Could not create Makefile due to some reason, probably lack of necessary
  libraries and/or headers.  Check the mkmf.log file for more details.  You may
  need configuration options.
  
  Provided configuration options:
  	--with-opt-dir
  	--without-opt-dir
  	--with-opt-include
  	--without-opt-include=${opt-dir}/include
  	--with-opt-lib
  	--without-opt-lib=${opt-dir}/lib
  	--with-make-prog
  	--without-make-prog
  	--srcdir=.
  	--curdir
  	--ruby=/opt/hostedtoolcache/Ruby/2.7.6/x64/bin/$(RUBY_BASE_NAME)
  	--with-libyaml-source-dir
  	--without-libyaml-source-dir
  	--with-yaml-0.1-config
  	--without-yaml-0.1-config
  	--with-pkg-config
  	--without-pkg-config
  	--with-libyaml-dir
  	--without-libyaml-dir
  	--with-libyaml-include
  	--without-libyaml-include=${libyaml-dir}/include
  	--with-libyaml-lib
  	--without-libyaml-lib=${libyaml-dir}/lib
  
  To see why this extension failed to compile, please check the mkmf.log which can
  be found here:
  
  /home/runner/work/gh-actions-ruby-psych-5/gh-actions-ruby-psych-5/vendor/bundle/ruby/2.7.0/extensions/x86_64-linux/2.7.0/psych-5.0.0/mkmf.log
  
  extconf failed, exit code 1
  
  Gem files will remain installed in
  /home/runner/work/gh-actions-ruby-psych-5/gh-actions-ruby-psych-5/vendor/bundle/ruby/2.7.0/gems/psych-5.0.0
  for inspection.
  Results logged to
  /home/runner/work/gh-actions-ruby-psych-5/gh-actions-ruby-psych-5/vendor/bundle/ruby/2.7.0/extensions/x86_64-linux/2.7.0/psych-5.0.0/gem_make.out
  
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/rubygems/ext/builder.rb:99:in
  `run'
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/rubygems/ext/ext_conf_builder.rb:48:in
  `block in build'
    /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/tempfile.rb:291:in `open'
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/rubygems/ext/ext_conf_builder.rb:30:in
  `build'
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/rubygems/ext/builder.rb:169:in
  `block in build_extension'
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/rubygems/ext/builder.rb:165:in
  `synchronize'
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/rubygems/ext/builder.rb:165:in
  `build_extension'
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/rubygems/ext/builder.rb:210:in
  `block in build_extensions'
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/rubygems/ext/builder.rb:207:in
  `each'
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/rubygems/ext/builder.rb:207:in
  `build_extensions'
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/2.7.0/rubygems/installer.rb:844:in
  `build_extensions'
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/rubygems_gem_installer.rb:72:in
  `build_extensions'
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/rubygems_gem_installer.rb:28:in
  `install'
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/source/rubygems.rb:207:in
  `install'
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/installer/gem_installer.rb:54:in
  `install'
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/installer/gem_installer.rb:16:in
  `install_from_spec'
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/installer/parallel_installer.rb:186:in
  `do_install'
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/installer/parallel_installer.rb:177:in
  `block in worker_pool'
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/worker.rb:62:in
  `apply_func'
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/worker.rb:57:in
  `block in process_queue'
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/worker.rb:54:in
  `loop'
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/worker.rb:54:in
  `process_queue'
  /opt/hostedtoolcache/Ruby/2.7.6/x64/lib/ruby/gems/2.7.0/gems/bundler-2.3.26/lib/bundler/worker.rb:91:in
  `block (2 levels) in create_threads'
  
  An error occurred while installing psych (5.0.0), and Bundler cannot continue.
  
  In Gemfile:
    psych
  Took   5.42 seconds
Error: Error: The process '/opt/hostedtoolcache/Ruby/2.7.6/x64/bin/bundle' failed with exit code 5
    at ExecState._setResult (/home/runner/work/_actions/ruby/setup-ruby/0a29871fe2b0200a17a4497bae54fe5df0d973aa/dist/index.js:6023:25)
    at ExecState.CheckComplete (/home/runner/work/_actions/ruby/setup-ruby/0a29871fe2b0200a17a4497bae54fe5df0d973aa/dist/index.js:6006:18)
    at ChildProcess.<anonymous> (/home/runner/work/_actions/ruby/setup-ruby/0a29871fe2b0200a17a4497bae54fe5df0d973aa/dist/index.js:5900:27)
    at ChildProcess.emit (node:events:390:28)
    at maybeClose (node:internal/child_process:1064:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)

Any other notes?

No response

@MSP-Greg
Copy link
Collaborator

MSP-Greg commented Dec 5, 2022

A good place to check are the Actions workflows for the repo, in this case, ruby/psych. It looks like libyaml-dev is not installed?

In general, when building Ruby or any extension gems (like psych), there are often build dependencies. They may not be included with the 'base' Actions image.

@eregon
Copy link
Member

eregon commented Dec 5, 2022

As it says:

  checking for yaml.h... no
  yaml.h not found
  *** extconf.rb failed ***
  Could not create Makefile due to some reason, probably lack of necessary
  libraries and/or headers.  Check the mkmf.log file for more details.  You may
  need configuration options.

So you'll need to install libyaml-dev or so, or use --libyaml-source-dir.
cc @hsbt --libyaml-source-dir should be documented in https://github.com/ruby/psych/blob/master/README.md probably. But even then it's not really convenient to get libyaml.

@eregon eregon closed this as not planned Won't fix, can't repro, duplicate, stale Dec 5, 2022
@dentarg
Copy link

dentarg commented Dec 5, 2022

Shouldn't installing Pysch 5.0.0 on Ruby 2.7.6 just work? Thanks to ruby/psych#550 or am I missing something?

@MSP-Greg
Copy link
Collaborator

MSP-Greg commented Dec 5, 2022

@dentarg
Copy link

dentarg commented Dec 5, 2022

Oh I thought Ruby itself bundled libyaml

@MSP-Greg
Copy link
Collaborator

MSP-Greg commented Dec 5, 2022

3.1 does, master does not

@dentarg
Copy link

dentarg commented Dec 5, 2022

but Pysch 5.0.0 can't use it?

@MSP-Greg
Copy link
Collaborator

MSP-Greg commented Dec 5, 2022

I suppose it could, never tried. Vendoring c source for dependencies has pros and cons, and other default extension gems do not vendor, like Ruby's OpenSSL.

@eregon
Copy link
Member

eregon commented Dec 5, 2022

Oh I thought Ruby itself bundled libyaml

It was just psych itself vendoring libyaml.
So this is an expected breaking change for Psych >= 5, that it needs a system libyaml-dev installed (like openssl indeed).

@hsbt
Copy link
Member

hsbt commented Dec 6, 2022

👋

Can we install libyaml-dev into https://github.com/actions/runner-images by ruby/setup-ruby layer?

I'm not sure how handle the additional packages like libffi, libgmp and libyaml in ruby/setup-ruby.

@MSP-Greg
Copy link
Collaborator

MSP-Greg commented Dec 6, 2022

Yes. Similar code is in setup-ruby-pkgs, but the packages are user defined. Windows builds already have all packages installed by setup-ruby. By packages, I mean Ruby dependencies, but you knew that.

@MSP-Greg
Copy link
Collaborator

MSP-Greg commented Dec 6, 2022

Briefly looked at ruby/ruby CI, libncurses5-dev & libyaml-dev are needed with Ubuntu, but not macOS?

@eregon
Copy link
Member

eregon commented Dec 6, 2022

Can we install libyaml-dev into https://github.com/actions/runner-images by ruby/setup-ruby layer?

I don't think setup-ruby should install any system package, for various reasons including time (it's a non-trivial overhead and in many cases not needed) and confusing side effects of setup-ruby (i.e. some extra system packages installed after the action).
If Ruby would need a new system package to even run then IMHO we should ask to preinstall that package in https://github.com/actions/runner-images rather than installing that package on every usage of setup-ruby.

Just like one needs to install a system package when using mysql2 or pg gem, now one needs to sudo apt-get install libyaml-dev before installing psych 5+.

@flavorjones
Copy link

flavorjones commented Dec 6, 2022

I want to strongly encourage this group to please push this new library requirement down the stack (to runner-images if that's the right place) and not up the stack to the individuals trying to maintain CI.

Asking users of psych to add a step to every pipeline (or alternatively to use setup-ruby-pkgs) is a considerable cost in terms of human time, and a collectively large amount of clock time and cpu time to install it. Especially for indirect users of psych (e.g., a project requires rdoc which then transitively requires psych).

I'm not complaining, but as a data point from someone who is familiar with the problem and the workarounds, I've already spent a few hours trying to get just one project's pipelines green again.

@eregon
Copy link
Member

eregon commented Dec 6, 2022

Is it common to have an unbounded dependency on psych though? Because otherwise it's not a problem.
Regarding rdoc I think we should fix https://github.com/ruby/rdoc/blob/c5e7ee03ac3035b001d58e12564c1ea79540c3a8/rdoc.gemspec#L232, I think either it should not specify the dependency and use the default gem, or it should pin to a specific major version (e.g., 4) to avoid such incompatible changes, cc @aycabta for ruby/rdoc@ebe185c.

I think it's worth a try to create an issue at https://github.com/actions/runner-images to ask to preinstall libyaml-dev. It's probably very small in size and I believe the libyaml package is already included. I would like someone else to create that issue though (I don't personally have the need for it).

@MSP-Greg
Copy link
Collaborator

MSP-Greg commented Dec 6, 2022

See 'Ubuntu : Update the libyaml version to 0.2.5 (latest)' in actions/runner-images

@ioquatix
Copy link
Member

ioquatix commented Dec 8, 2022

Because of this change, the logger CI is now failing.

https://github.com/ruby/logger

This probably won't be the only case.

@ioquatix
Copy link
Member

ioquatix commented Dec 8, 2022

We fixed it by removing rdoc from the development dependency list.

@MSP-Greg
Copy link
Collaborator

MSP-Greg commented Dec 8, 2022

I'll be devil's advocate here, undecided whether setup-ruby should handle this. This type of issue may happen more in the future.

  1. What should be installed, 0.2.2 (standard) or 0.2.5 (available from kinetic)?

  2. I suspect that caching the packages in a GitHub release might be a good idea, not sure. If they are cached, more code somewhere.

Off-topic: Puma has a bit of 'app reload' logic, and upgrading bundled extension gems in the main process causes issues. Our yaml needs were trivial, so we replaced Psych with pure Ruby code. Maybe rdoc can do the same?

@eregon
Copy link
Member

eregon commented Dec 8, 2022

FWIW I measured how long sudo apt install libyaml-dev takes and it's 5-6 seconds.
That's way too much to add by default for setup-ruby.

Also from @MSP-Greg's last reply it comforts me in my position that we certainly don't want additional complexity and risks here (using a package from another Ubuntu version feels very risky, especially if the existing /usr/lib/libyaml.so is of a different version).
My reasoning is if people can install system packages of sqlite3/mysql2/pg/etc, they can for psych too.
And that has other advantages including more flexibility and e.g. caching via this action I noticed today.

I understand it's annoying and possibly more frequent but this is not the place to solve it, this action sets up Ruby it doesn't guess system dependencies or do anything about them.

As I said in #409 (comment), I think the only way to make it nicer is to ask to add libyaml-dev in https://github.com/actions/runner-images.

@eregon
Copy link
Member

eregon commented Dec 8, 2022

Oh and it looks like I was very lucky with thet 5-6 second run, because in 3 other runs it's more like 15 seconds:
https://github.com/eregon/setup-ruby/actions/runs/3649371803/jobs/6163957728
https://github.com/eregon/setup-ruby/actions/runs/3649348737/jobs/6163903717
https://github.com/ruby/rdoc/actions/runs/3648958399/jobs/6163011270

(FWIW that's a big part of why I don't like apt, dnf is so much faster/better)
Seems an additional reason to add it to runner images.

@MSP-Greg
Copy link
Collaborator

MSP-Greg commented Dec 8, 2022

using a package from another Ubuntu version feels very risky, especially if the existing /usr/lib/libyaml.so is of a different version

Just to clarify, if we used kinetic, we'd need to install both libyaml & libyaml-dev...

I suspect runner-images may not want to add libyaml-dev, not sure. I'll see...

@MSP-Greg
Copy link
Collaborator

MSP-Greg commented Dec 8, 2022

Help - can anyone verify whether libyaml & libyaml-dev are a normal part of a macOS install? Looking at the macOS workflow in ruby/ruby, can't see any mention of it? Maybe I need more coffee...

Or, one can use apt list --installed on Ubuntu. What's the equivalent on macOS?

@eregon
Copy link
Member

eregon commented Dec 8, 2022

Actually runner images provide some versions of Ruby, so that guarantees libyaml (i.e., the .so but not the headers) will be available on runner images, and the same for all required dependencies of ruby (or if one is missing it's an obvious issue and I'd think that would be quickly fixed in runner-images).

@eregon
Copy link
Member

eregon commented Dec 8, 2022

@MSP-Greg brew list (see https://apple.stackexchange.com/questions/101090/list-of-all-packages-installed-using-homebrew)
The macOS system might or not have libyaml by default, I have no idea.

@flavorjones
Copy link

flavorjones commented Dec 8, 2022

On my dev machine:

$ brew list libyaml -v
/opt/homebrew/Cellar/libyaml/0.2.5/INSTALL_RECEIPT.json
/opt/homebrew/Cellar/libyaml/0.2.5/License
/opt/homebrew/Cellar/libyaml/0.2.5/.brew/libyaml.rb
/opt/homebrew/Cellar/libyaml/0.2.5/Changes
/opt/homebrew/Cellar/libyaml/0.2.5/include/yaml.h
/opt/homebrew/Cellar/libyaml/0.2.5/ReadMe.md
/opt/homebrew/Cellar/libyaml/0.2.5/lib/pkgconfig/yaml-0.1.pc
/opt/homebrew/Cellar/libyaml/0.2.5/lib/libyaml.dylib
/opt/homebrew/Cellar/libyaml/0.2.5/lib/libyaml.a
/opt/homebrew/Cellar/libyaml/0.2.5/lib/libyaml-0.2.dylib

So this is the equivalent of both libyaml and libyaml-dev

@MSP-Greg
Copy link
Collaborator

MSP-Greg commented Dec 8, 2022

@flavorjones

I suspected as much, thanks for verifying.

@flavorjones
Copy link

I don't see a libyaml installation on the core macOS system (but I may not be looking in all the places):

$ find /usr /System/Library /System/Developer -name 'yaml.h' 2> /dev/null
(no output)

$ find /usr /System/Library /System/Developer -name 'libyaml*' 2> /dev/null
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/share/ri/2.6.0/system/Psych/libyaml_version-c.ri

so I'm guessing it's a brew-installed package.

@MSP-Greg
Copy link
Collaborator

MSP-Greg commented Dec 8, 2022

@flavorjones

Interesting. You mentioned some CI that failed. Did it fail on Ubuntu and pass on macOS? I assume GitHub Actions?

I opened an issue, actions/runner-images#6725, and I hope 'parity' between Ubuntu & macOS is a valid reason...

@flavorjones
Copy link

@MSP-Greg Correct, the failures have only been on Ubuntu images.

@flavorjones
Copy link

Closing the loop here: libyaml-dev should be on runner images sometime this week:

dentarg added a commit to dentarg/sinatra that referenced this issue Dec 12, 2022
Needed to be able to install psych >=5.
Need to do it before setup-ruby runs as that runs `bundle install`.

Related to:
- ruby/psych#541
- ruby/setup-ruby#409
- actions/runner-images#6725

Yes, libyaml-dev will be added to GitHub runner images (this week they
say) but opening this PR just in case anyone encounters failing builds.
I don't think it hurts having this in the repo even after libyaml-dev
have been added to the images.

Added some blank lines to make the workflow easier to read.
jkowens pushed a commit to sinatra/sinatra that referenced this issue Dec 15, 2022
Needed to be able to install psych >=5.
Need to do it before setup-ruby runs as that runs `bundle install`.

Related to:
- ruby/psych#541
- ruby/setup-ruby#409
- actions/runner-images#6725

Yes, libyaml-dev will be added to GitHub runner images (this week they
say) but opening this PR just in case anyone encounters failing builds.
I don't think it hurts having this in the repo even after libyaml-dev
have been added to the images.

Added some blank lines to make the workflow easier to read.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

7 participants