Update Ruby to 3.2 #425

ashie · 2023-01-16T06:56:20Z

Since Ruby 2.7 is close to EOL, we have to update Ruby.
The next major version of td-agent will bundle Ruby 3.2.

Bundle Ruby 3.2 and remove older Ruby sources
Add libyaml-devel and libffi-devel package since Ruby 3.2 no longer bundles these sources
Add Rust to build YJIT of Ruby
Remove obsolete distributions from default build targets
Update nokogiri to 1.14.0 to make it buildable on Windows
Bump up the version number to 5.0.0 to indicate that the master branch is going to develop next major version

FYI: https://www.ruby-lang.org/en/news/2022/12/25/ruby-3-2-0-released/

TODO:

Add missing libs for macOS (but I won't address it in this PR, I'll track it at td-agent 4.4.2 for macOS x86_64 lacks libyaml #421)
~~Fix CI failure on Windows~~ Done

Since Ruby 2.7 is close to EOL, we are going to switch bundled Ruby to 3.2. We'll ship it as td-agent 5.0. TODO: Install Rust to build environment to build YJIT. Signed-off-by: Takuro Ashie <[email protected]>

Because they are already or close to EOL, we'll drop supporting them from the next release. Signed-off-by: Takuro Ashie <[email protected]>

Signed-off-by: Takuro Ashie <[email protected]>

The source code of libffi is no longer bundled as of Ruby 3.2. Signed-off-by: Takuro Ashie <[email protected]>

The mater branch is now going to develop v5.0. Signed-off-by: Takuro Ashie <[email protected]>

* libffi-dev: Need to build fiddle as of Ruby 3.2 * libreadline-dev: Originally it was not built for Debian based distros but is enabled on Red Hat based distros. Signed-off-by: Takuro Ashie <[email protected]>

Signed-off-by: Takuro Ashie <[email protected]>

ashie · 2023-01-18T03:57:04Z

Hmm, bundle install stalls after completing to install all gems on Ruby 3.2 on Windows:

...
Fetching fluent-plugin-rewrite-tag-filter 2.4.0
Fetching fluent-plugin-calyptia-monitoring 0.1.3
Installing fluent-plugin-rewrite-tag-filter 2.4.0
Installing fluent-plugin-calyptia-monitoring 0.1.3
Fetching fluent-plugin-prometheus_pushgateway 0.1.0
Installing fluent-plugin-prometheus_pushgateway 0.1.0
(no output then timed out)

Updating bundler to 2.4.4 doesn't take effect.

It doesn't reproduce on Ruby 3.1:

...
Installing fluent-plugin-prometheus_pushgateway 0.1.0
Bundle complete! 63 Gemfile dependencies, 110 gems now installed.
...
(complete building)

ashie · 2023-01-18T06:06:49Z

Run rake msi:selfbuild and break by C-c after stall:

C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/worker.rb:108:in `block in add_interrupt_handler'
<internal:thread_sync>:18:in `pop'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/worker.rb:42:in `deq'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/installer/parallel_installer.rb:173:in `process_specs'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/installer/parallel_installer.rb:133:in `install_with_worker'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/installer/parallel_installer.rb:89:in `call'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/installer/parallel_installer.rb:67:in `call'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/installer.rb:244:in `install_in_parallel'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/installer.rb:201:in `install'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/installer.rb:89:in `block in run'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/process_lock.rb:12:in `block in lock'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/process_lock.rb:9:in `open'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/process_lock.rb:9:in `lock'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/installer.rb:71:in `run'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/installer.rb:23:in `install'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/cli/install.rb:62:in `run'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/cli.rb:260:in `block in install'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/settings.rb:131:in `temporary'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/cli.rb:259:in `install'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/cli.rb:34:in `dispatch'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/cli.rb:28:in `start'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/exe/bundle:45:in `block in <top (required)>'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
C:/Users/aho/Projects/td-agent-builder/td-agent/staging/lib/ruby/gems/3.2.0/gems/bundler-2.4.4/exe/bundle:33:in `<top (required)>'

ashie · 2023-01-18T08:27:43Z

When I run bundler install with --jobs=1, it stalls on installing nokogiri:

Fetching nokogiri 1.13.9
Installing nokogiri 1.13.9 with native extensions

(It reproduces on msi:build but doesn't reproduce on msi:selfbuild.)

It might be a same issue we experienced in the past.

ashie · 2023-01-18T23:56:01Z

I've confirmed on my local environment that nokogiri 1.14.0 is buildable on Ruby 3.2 for Windows.

* msys2 20221216.0.0 * Ruby 3.1.3.1 Signed-off-by: Takuro Ashie <[email protected]>

Signed-off-by: Takuro Ashie <[email protected]>

cosmo0920 · 2023-01-21T01:00:10Z

It might be a same issue we experienced in the past.

This could be occurred with fat gem mechanism. Maybe, nokogiri 1.13 or older does not include a binary for Ruby3.2, right?

ashie · 2023-01-21T01:38:06Z

This could be occurred with fat gem mechanism. Maybe, nokogiri 1.13 or older does not include a binary for Ruby3.2, right?

Yes. Now I succeeded to build it also on CI by nokogiri 1.14.0.
The next problem is that installation test is failed on CI.
But the built package is installable on my & @daipom's local environment.
Probably it fails to launch the fluentwinsvc service on CI.
I'm suspecting that required runtime library is lacked in CI env (it's implicitly installed on our environment?).

cosmo0920 · 2023-01-21T06:37:51Z

I'm suspecting that required runtime library is lacked in CI env (it's implicitly installed on our environment?).

I recommend to use Windows Sandbox for testing msi package or zip package. This is provided from Windows Professional Edition or higher. However, this isolated and refreshed in every time environment quite useful to test implicit dependencies. Or, using Dependencies to testify dependencies of Windows executables: https://github.com/lucasg/Dependencies

cosmo0920 · 2023-01-21T12:09:29Z

This is because YJIT will introduce MSVC runtime dlls.
Ruby's YJIT does not link CRT as static.

https://github.com/ruby/ruby/blob/master/yjit/Cargo.toml

ashie · 2023-01-26T00:52:32Z

I recommend to use Windows Sandbox for testing msi package or zip package.

Thanks for your information. I forgot it so far.
But even I tried on Windows Sandbox, it succeeded to install...

ashie · 2023-01-26T00:56:18Z

This is because YJIT will introduce MSVC runtime dlls. Ruby's YJIT does not link CRT as static.

In my understanding, Ruby on Windows doesn't support YJIT.
https://www.ruby-lang.org/ja/news/2022/12/25/ruby-3-2-0-released/

In actual, RubyInstaller 3.2 doesn't provide --yjit option.

C:\opt\td-agent>ruby --version
ruby 3.2.0 (2022-12-25 revision a528908271) [x64-mingw-ucrt]

C:\opt\td-agent>ruby --yjit
ruby: warning: Ruby was built without YJIT support. You may need to install rustc to build Ruby with YJIT.

cosmo0920 · 2023-01-26T01:21:54Z

Got it. Interesting...

ashie · 2023-01-27T08:48:22Z

I completely misunderstood the error, it's not installation error:

  1) Package "td-agent v5.0.0" is expected to be installed
     Failure/Error: it { should be_installed }
     NoMethodError:
       undefined method `exists?' for File:Class

     # ./serverspec/windows/td-agent.rb:12:in `block (2 levels) in <top (required)>'

Something uses File.exists? which is removed as of Ruby 3.2:
https://www.ruby-lang.org/ja/news/2022/12/25/ruby-3-2-0-released/

One of suspected module is serverspec but it's already fixed in 2.42.1:
mizzy/serverspec@d75dbb4

daipom · 2023-01-27T08:52:11Z

this? https://github.com/fluent/fluent-package-builder/blob/master/td-agent/Rakefile#L87

daipom · 2023-01-27T08:53:01Z

Oh, sorry, this is not related to serverspec test...

ashie · 2023-01-27T09:57:35Z

I got it. It's specinfra's issue:

  48) gem files Package "ffi" is expected to be installed by "gem" with version "1.15.5"
      Failure/Error: it { should be_installed.by("gem").with_version(gem.version) }
      NoMethodError:
        undefined method `exists?' for File:Class

      # C:/Users/aho/.gem/ruby/3.2.0/gems/specinfra-2.84.0/lib/specinfra/backend/cmd.rb:73:in `block in find_powershell'
      # C:/Users/aho/.gem/ruby/3.2.0/gems/specinfra-2.84.0/lib/specinfra/backend/cmd.rb:73:in `each'
      # C:/Users/aho/.gem/ruby/3.2.0/gems/specinfra-2.84.0/lib/specinfra/backend/cmd.rb:73:in `find'
      # C:/Users/aho/.gem/ruby/3.2.0/gems/specinfra-2.84.0/lib/specinfra/backend/cmd.rb:73:in `find_powershell'
      # C:/Users/aho/.gem/ruby/3.2.0/gems/specinfra-2.84.0/lib/specinfra/backend/cmd.rb:65:in `x64_powershell'
      # C:/Users/aho/.gem/ruby/3.2.0/gems/specinfra-2.84.0/lib/specinfra/backend/cmd.rb:59:in `powershell'
      # C:/Users/aho/.gem/ruby/3.2.0/gems/specinfra-2.84.0/lib/specinfra/backend/cmd.rb:14:in `run_command'
      # C:/Users/aho/.gem/ruby/3.2.0/gems/specinfra-2.84.0/lib/specinfra/runner.rb:27:in `run'
      # C:/Users/aho/.gem/ruby/3.2.0/gems/specinfra-2.84.0/lib/specinfra/runner.rb:19:in `method_missing'
      # C:/Users/aho/.gem/ruby/3.2.0/gems/serverspec-2.42.1/lib/serverspec/type/package.rb:9:in `installed?'
      # C:/Users/aho/.gem/ruby/3.2.0/gems/serverspec-2.42.1/lib/serverspec/matcher/be_installed.rb:6:in `block (2 levels) in <top (required)>'
      # ./serverspec/windows/td-agent.rb:26:in `block (4 levels) in <top (required)>'

ashie · 2023-01-27T10:07:35Z

Fixed in specinfra-2.84.1

daipom · 2023-01-27T10:27:22Z

Wow! Super quick!

cosmo0920

Almost of all really good to me. However, the latest version of msys2 is not same for currently used in this PR. Could you confirm whether we can use the latest version of msys2 on this PR or not?

td-agent/msi/Dockerfile

daipom · 2023-01-31T05:36:48Z

Sorry, please let me see this tomorrow.

kenhys · 2023-01-31T08:04:57Z

If it doesn't cause a regression, adopting latest openssl 1.1.1s may be better.
https://mta.openssl.org/pipermail/openssl-announce/2022-November/000242.html

NOTE: openssl 1.1.1r has a regression. so announced of withdrawal
https://mta.openssl.org/pipermail/openssl-announce/2022-October/000237.html

kenhys

LGTM.

td-agent/config.rb

ashie · 2023-01-31T09:32:48Z

If it doesn't cause a regression, adopting latest openssl 1.1.1s may be better. https://mta.openssl.org/pipermail/openssl-announce/2022-November/000242.html

NOTE: openssl 1.1.1r has a regression. so announced of withdrawal https://mta.openssl.org/pipermail/openssl-announce/2022-October/000237.html

This PR is focused on updating Ruby.
So I'll update OpenSSL for macOS in another PR.
For windows, OpenSSL 3.0 bundled with RubyInstaller 3.2 will be used.

Signed-off-by: Takuro Ashie <[email protected]>

daipom

Sorry for being late.
I have checked the fixes.
In addition, I have checked that the --yjit option of the embedded Ruby works in my M1 MacBook.

LGTM.

ashie · 2023-02-01T08:02:39Z

Thanks for your review!
If some issues concerned with this are remained, I'll address them in other PR.

ashie added 2 commits January 16, 2023 13:48

Switch bundled ruby to 3.2

b5b2718

Since Ruby 2.7 is close to EOL, we are going to switch bundled Ruby to 3.2. We'll ship it as td-agent 5.0. TODO: Install Rust to build environment to build YJIT. Signed-off-by: Takuro Ashie <[email protected]>

Remove debian-buster & ubuntu-bionic from default targets

4314ccb

Because they are already or close to EOL, we'll drop supporting them from the next release. Signed-off-by: Takuro Ashie <[email protected]>

ashie force-pushed the ruby-3.2 branch from add0723 to 297a896 Compare January 16, 2023 07:01

Use devtoolset-11 for CentOS 7 to build Ruby 3.2

b78adbb

Signed-off-by: Takuro Ashie <[email protected]>

ashie force-pushed the ruby-3.2 branch 2 times, most recently from 632888c to db92e83 Compare January 17, 2023 04:20

ashie added 5 commits January 17, 2023 15:01

yum: Install Rust to build YJIT of Ruby

243da62

Signed-off-by: Takuro Ashie <[email protected]>

yum: Require libffi-devel to build Fiddle

0a4e61a

The source code of libffi is no longer bundled as of Ruby 3.2. Signed-off-by: Takuro Ashie <[email protected]>

Bump up the package version to 5.0

b7ddd36

The mater branch is now going to develop v5.0. Signed-off-by: Takuro Ashie <[email protected]>

apt: Add missing dev pacakges

6c37458

* libffi-dev: Need to build fiddle as of Ruby 3.2 * libreadline-dev: Originally it was not built for Debian based distros but is enabled on Red Hat based distros. Signed-off-by: Takuro Ashie <[email protected]>

apt: Install Rust to build YJIT of Ruby

ee80f48

Signed-off-by: Takuro Ashie <[email protected]>

ashie force-pushed the ruby-3.2 branch from 7874186 to ee80f48 Compare January 17, 2023 06:03

ashie added 2 commits January 17, 2023 17:47

macOS: Enable Ruby's YJIT

f526f0f

Signed-off-by: Takuro Ashie <[email protected]>

Relax package size check to introduce Ruby 3.2

53076e7

Signed-off-by: Takuro Ashie <[email protected]>

ashie force-pushed the ruby-3.2 branch 2 times, most recently from a367c02 to 6e9be98 Compare January 19, 2023 09:31

msi: Update build env

450652d

* msys2 20221216.0.0 * Ruby 3.1.3.1 Signed-off-by: Takuro Ashie <[email protected]>

ashie force-pushed the ruby-3.2 branch from 6e9be98 to a01c2ca Compare January 19, 2023 09:46

Add nokogiri 1.14.0

6c40b9a

Signed-off-by: Takuro Ashie <[email protected]>

ashie force-pushed the ruby-3.2 branch from a01c2ca to 6c40b9a Compare January 19, 2023 13:22

ashie added 2 commits January 20, 2023 21:20

msi: Stop replacing OpenSSL in RubyInstaller

a1efa04

Signed-off-by: Takuro Ashie <[email protected]>

msi: Remove needless patch entries for RubyInstaller

07050a4

Signed-off-by: Takuro Ashie <[email protected]>

ashie force-pushed the ruby-3.2 branch from 7f40841 to 6961d11 Compare January 21, 2023 00:36

ashie force-pushed the ruby-3.2 branch from 6961d11 to 07050a4 Compare January 21, 2023 01:20

ashie mentioned this pull request Jan 27, 2023

Replace obsolete File.exists? with File.exist? mizzy/specinfra#739

Merged

ashie marked this pull request as ready for review January 27, 2023 11:48

ashie requested review from kenhys, cosmo0920 and daipom January 31, 2023 00:05

cosmo0920 reviewed Jan 31, 2023

View reviewed changes

td-agent/msi/Dockerfile Outdated Show resolved Hide resolved

kenhys approved these changes Jan 31, 2023

View reviewed changes

cosmo0920 reviewed Jan 31, 2023

View reviewed changes

td-agent/config.rb Show resolved Hide resolved

Update to msys2 20230127.0.0 to build msi

2ce445c

Signed-off-by: Takuro Ashie <[email protected]>

cosmo0920 approved these changes Jan 31, 2023

View reviewed changes

daipom approved these changes Feb 1, 2023

View reviewed changes

ashie merged commit d526cb0 into master Feb 1, 2023

ashie deleted the ruby-3.2 branch February 1, 2023 08:02

This was referenced Feb 8, 2023

Remaining issues to support Ruby 3.2 #429

Closed

Remaining tasks to ready for Ruby 3.1 or later #380

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Ruby to 3.2 #425

Update Ruby to 3.2 #425

ashie commented Jan 16, 2023 •

edited

Loading

ashie commented Jan 18, 2023 •

edited

Loading

ashie commented Jan 18, 2023

ashie commented Jan 18, 2023

ashie commented Jan 18, 2023

cosmo0920 commented Jan 21, 2023

ashie commented Jan 21, 2023 •

edited

Loading

cosmo0920 commented Jan 21, 2023 •

edited

Loading

cosmo0920 commented Jan 21, 2023

ashie commented Jan 26, 2023

ashie commented Jan 26, 2023

cosmo0920 commented Jan 26, 2023

ashie commented Jan 27, 2023

daipom commented Jan 27, 2023

daipom commented Jan 27, 2023

ashie commented Jan 27, 2023

ashie commented Jan 27, 2023

daipom commented Jan 27, 2023

cosmo0920 left a comment •

edited

Loading

daipom commented Jan 31, 2023

kenhys commented Jan 31, 2023

kenhys left a comment

ashie commented Jan 31, 2023

daipom left a comment

ashie commented Feb 1, 2023

Update Ruby to 3.2 #425

Update Ruby to 3.2 #425

Conversation

ashie commented Jan 16, 2023 • edited Loading

ashie commented Jan 18, 2023 • edited Loading

ashie commented Jan 18, 2023

ashie commented Jan 18, 2023

ashie commented Jan 18, 2023

cosmo0920 commented Jan 21, 2023

ashie commented Jan 21, 2023 • edited Loading

cosmo0920 commented Jan 21, 2023 • edited Loading

cosmo0920 commented Jan 21, 2023

ashie commented Jan 26, 2023

ashie commented Jan 26, 2023

cosmo0920 commented Jan 26, 2023

ashie commented Jan 27, 2023

daipom commented Jan 27, 2023

daipom commented Jan 27, 2023

ashie commented Jan 27, 2023

ashie commented Jan 27, 2023

daipom commented Jan 27, 2023

cosmo0920 left a comment • edited Loading

Choose a reason for hiding this comment

daipom commented Jan 31, 2023

kenhys commented Jan 31, 2023

kenhys left a comment

Choose a reason for hiding this comment

ashie commented Jan 31, 2023

daipom left a comment

Choose a reason for hiding this comment

ashie commented Feb 1, 2023

ashie commented Jan 16, 2023 •

edited

Loading

ashie commented Jan 18, 2023 •

edited

Loading

ashie commented Jan 21, 2023 •

edited

Loading

cosmo0920 commented Jan 21, 2023 •

edited

Loading

cosmo0920 left a comment •

edited

Loading