Submit CFP to RubyKaigi Takeout 2021

[fujimotos]

Problem Description

• RubyKaigi Takeout 2021 is calling for speakers.
• We are palnning to apply for the conference as a speaker.
• For that purpose, we need to submit a CFP before the proposal deadline (2021/6/30 23:59).

Assginee

Me (fujimotos)

Timeline

• I will write a CFP draft and post the draft to this ticket next Monday.
• I'm thinking to talk something like "Fluentd: Ship a Ruby Appliction to Windows".
  • td-agent-builder, Windows event support etc.
  • Basically the stuffs we've done for Windows last months.
• Review & revise the draft, and submit it to rubykaigi.org on Tue/Wed.

[github-actions]

@fujimotos this issue was automatically closed because it did not follow the issue template

[fujimotos]

@ashie @kenhys @kou This is the CFP draft for Rubykaigi Takeout 2021.

Please comment here if you have any suggestions (preferably by Tuesday
as the submission close is next Wed).

# Title

Fluentd: How to ship a Ruby Application to Windows

# Abstract

Shipping a Ruby application to Windows is a challenging task. For starters, we
cannot even expect that Ruby is handily pre-installed there!

We (Fluentd team) maintain a set of Windows packages to make it easier to
deploy and run Fluentd on Windows. In this talk, we discuss the general problems
of shipping Ruby applications to Windows, and explain how we addressed them.

# Details

The goal of my talk is to let attendants obtain the general knowledge of how to
distribute Ruby applications on Windows.

Here is the (planned) outline of my talk:

* What is Fluentd? (intro)
* Windows and Ruby
  * Myth: "Ruby is cross-platform. My app should just work fine!"
  * Not all platform incosistencies are abstracted away.
  * Not all gems are Windows-ready.
* Detailed know-hows
  * Porting your app to Windows.
  * How to create a Windows installer (WIX/NSIS...)
  * How to include a copy of Ruby in your package.
  * How to manage gems for your app on Windows.
  * How to run your Windows app as Windows Service.
  * ...
* Open problems
  * Unix culture vs Windows culture
* Conclusion

# Pitch

Ruby developers demographic is heavily leaned toward Unix (Linux/BSD/macOS).
Generally speaking, there is much less community knowledge and expertise
regarding Windows.

However Windows is the most widely used desktop operating system, especially
in the business space. There are indeed a lot of demands for Ruby apps to support
Windows. I see here a certain gap waiting to be filled between Ruby devs (Unix)
and general user demographic (mostly Windows).

Fluentd project is maintaining the Windows distribution for years, and I think
we can provide a unique perspective on this topic.

[kenhys]

One more topic for "Detailed know-hows"

• How to apply a fix(patch) for a copy of Ruby (and gems)

[kou]

RubyKaigi is a deeply technical conference for Ruby itself.

http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/104408

we're looking for hardcore tech talks from Ruby hackers!

It seems that this talk focuses on packaging mainly. Can we focus on deeply technical topic related to Ruby as the main topic?

[ashie]

I'm not sure they are suitable for RubyKaigi though, following topics are Ruby on Windows related things in Fluentd I know.

• How to treat Windows spefic problems
  • Cannot use WSAGetLastError on Ruby 3.0: Unexpected error code with WSAGetLastError even though the previous function calll was failed ruby/fiddle#72, WinSock: Ensure to retrieve correct error code on Ruby 3.0 treasure-data/serverengine#112
  • Cannot use UNIX signals, we replace them with pipe communication or Win32 Event
  • System error code: errno vs GetLastError (can be mapped by rb_w32_map_errno)
  • winsock
    • linger related issues: Encryption support on out_forwarder plugin in Windows (Fluentd 1.0.2) #1968 out_forward: do not use SO_LINGER on Windows #2398 out_forward: windows: Permit to specify linger_timeout #2868 windows: Disable specifying linger_timeout again #2959
  • file handle
• How to get Windows EventLog (on Ruby)
  • https://github.com/fluent-plugins-nursery/winevt_c/
  • https://github.com/fluent/fluent-plugin-windows-eventlog
  • https://www.clear-code.com/blog/2019/7/25.html
• How to use TLS certifications in Windows Certificate Store (on Ruby)
  • https://github.com/fluent-plugins-nursery/certstore_c

  • fluentd/lib/fluent/plugin_helper/socket.rb

    Lines 115 to 126 in da162e1

    	if enable_system_cert_store
    	if Fluent.windows? && cert_logical_store_name
    	log.trace "loading Windows system certificate store"
    	loader = Certstore::OpenSSL::Loader.new(log, cert_store, cert_logical_store_name,
    	enterprise: cert_use_enterprise_store)
    	loader.load_cert_store
    	cert_store = loader.cert_store
    	context.cert = loader.get_certificate(cert_thumbprint) if cert_thumbprint
    	end
    	log.trace "loading system default certificate store"
    	cert_store.set_default_paths
    	end

• How to run as Windows Service (on Ruby)
  • https://github.com/fluent/fluentd/blob/master/lib/fluent/winsvc.rb

  • fluentd/lib/fluent/command/fluentd.rb

    Lines 270 to 301 in da162e1

    	case winsvcinstmode
    	when 'i'
    	binary_path = File.join(File.dirname(__FILE__), "..")
    	ruby_path = "\0" * 256
    	GetModuleFileName.call(0,ruby_path,256)
    	ruby_path = ruby_path.rstrip.gsub(/\\/, '/')
    	start_type = Service::DEMAND_START
    	if opts[:regwinsvcautostart]
    	start_type = Service::AUTO_START
    	start_service = true
    	end
    	Service.create(
    	service_name: opts[:winsvc_name],
    	host: nil,
    	service_type: Service::WIN32_OWN_PROCESS,
    	description: opts[:winsvc_desc],
    	start_type: start_type,
    	error_control: Service::ERROR_NORMAL,
    	binary_path_name: "\"#{ruby_path}\" -C \"#{binary_path}\" winsvc.rb --service-name #{opts[:winsvc_name]}",
    	load_order_group: "",
    	dependencies: [""],
    	display_name: opts[:winsvc_display_name]
    	)
    	if opts[:regwinsvcdelaystart]
    	Service.configure(
    	service_name: opts[:winsvc_name],
    	delayed_start: true
    	)
    	end

• Actual usage examples
  • https://aws.amazon.com/jp/blogs/news/streaming-logs-from-amazon-eks-windows-pods-to-amazon-cloudwatch-logs-using-fluentd/

[kou]

• How to get Windows EventLog (on Ruby)
• How to use TLS certifications in Windows Certificate Store (on Ruby)
• How to run as Windows Service (on Ruby)

Could you provide links of their implementations?

[ashie]

Could you provide links of their implementations?

I've added them to the comment.

[kou]

Thanks.
I think that describing how to implement XXXs on Ruby deeply is more suitable topic than packaging for RubyKaigi's talk.

[fujimotos]

@kou @ashie I created a new CFP draft. The presentation topic is Ractor and Fluentd.

• I'm thinking to add a Ractor-enabled version of the following function:
  • https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/parser_ltsv.rb#L38-L48
  • https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/in_tail.rb#L601-L607
• Add an experimental support and take some benchmark.
• LTSV parsing should be fairly easily Ractor-able.

RubyKaigi Takout 201 CFP (draft v2)

# Title

Speed up data parsing in Fluentd using Ractor

# Abstract

Prallelization is becoming the norm in moden programming. As more cores
are added to CPUs, programmers are expected to utilize these additional cores.

Ractor is the new parallelism mechanism available since Ruby 3.0. Recently,
Fluentd team experimented with this relatively-new feature to make text parsing
faster. In this talk, we introduce the general concept of Ractor and discuss our
experience with it.

# Details

The goal of my talk is to let attendants obtain the general knowledge of Ractor,
the bleeding-edge multi-thereading feature

Here is the (planned) outline of my talk:

* What is Ractor? (Introduction)
  * Ractor = safe threading with easier interface
  * Feature: Parallel processing (with multiple Ractors)
  * Feature: Message passing
  * Restriction: Cannot share data
  * Restriction: Most gems do not support it (yet)
* Make Fluentd faster to parse texts using Ractor
  * Basic: How Fluentd parser works
  * Making LTSV parser faster (1)
  * Making LTSV parser faster (2)
  * Making LTSV parser faster (3)
  * Experiment
  * Result 
* Open issues
  * Ractor: still "experiment" feature
  * We need to make more gems Ractor-ready
* Conclusion

# Pitch

Parallelism becomes a very hot topic recently. As CPU vendors add more and more
cores to their products, more programing languages are starting to give the special
consideration to parallel processing.

Ruby was historically not very good at concurrent programming (e.g. GIL issue).
However, the status has rapidly changed in the last few years. Especially
the introduction of Ractor was a big factor.

In this talk, we're going to introduce Ractor to general audience and show how
it can be made used of to speed up data processing.

[fujimotos]

Here is a PoC implementation of Ractor-enabled in_tail.

fujimotos@7fdb403

Using the following configuration:

<source>
  @type tail
  tag test.log
  path test.log
  read_from_head true
  <parse>
    @type ltsv
  </parse>
  <buffer>
    @type memory
    flush_interval 1s
  </buffer>
</source>

<match test.**>
  @type flowcounter_simple
  unit second
</match>

I can confirm that it actually utilizes 8 threads (cores):

A caveat is: actually the current implementation is a little slower than
the single-threaded implementation. (Maybe due to Ractor's overhead?)

But anyway, I think we can polish the PoC up and make a talk on it.

[kou]

It seems that this is a good starting point.
If we can improve it, we can talk about how to use Ractor for performance.

BTW, is it faster than Thread based implementation?

[kou]

Abound 30% CPU usage per thread may be low.
Each Ractor may be more idle than expected.

[fujimotos]

@kou We don't have any thread implementation on in_tail.
in_tail basically uses a "one thread for one file" model.

Here is some performance stats:

Mode	Speed [lines/s]
single-thread	238,934
Ractor (2 workers)	203,435
Ractor (8 workers)	211,627

Note that the test machine has 8 CPUs (Azure F8s with Xeon 8272CL)
Ruby processes maxed out at around 30-60%. Never goes 100%.

[kou]

Is in_tail a CPU bound process?
If it's an IO bound process, it may not be suitable for Ractor.

[fujimotos]

I experimened with Ractor using another part of Fluentd last night.

https://github.com/fujimotos/fluentd/blob/aa5c1626/lib/fluent/plugin/filter_grep_ractor.rb#L81-L104

It didn't work. Ractor was actually slower than a single-thread implementation.
Other people seem to report similar results too:

• More benchmarks, Ractors vs Forks + Redis + Marshal
  
• Ractors in Ruby 3

So, apparently, Ractor seem not to be eligible for production usage (yet)...

[kou]

Ractor seem not to be eligible for production usage (yet)...

Yes. Ractor is still an experimental feature.

[fujimotos]

RubyKaigi Takout 201 CFP (draft v3)

# Title

Ractor: Experimenting with the new parallelization feature

# Abstract

Parallelization is becoming the norm in model programming. As more cores are
added to CPUs, programmers are expected to utilize these additional cores.

Ractor is the new parallelism mechanism available since Ruby 3.0. Recently,
Fluentd team experimented with this relatively-new feature to see if it can
accelarete data processing.

In this talk, we explain our experience with Ractor and discuss the current
status of this technology.

# Details

The goal of my talk is to let attendants obtain the general knowledge of Ractor,
the bleeding-edge multi-thereading model.

Here is the (planned) outline of my talk:

* What is Ractor? (Introduction)
  * Ractor = safe threading with easier interface
  * Feature: Parallel processing (with multiple Ractors)
  * Restriction: Data sharing
  * Restriction: Gem support
* Practical performance of Ractor
  * Basic: How Fluentd works
  * Make Fluentd Plugin Ractor-ready (1)
  * Make Fluentd Plugin Ractor-ready (2)
  * Make Fluentd Plugin Ractor-ready (3)
  * Experiment & Result 
* Open problems
* Conclusion

# Pitch

Parallelism becomes a hot topic recently. As multi-CPU architecture becomes
popular, programming languages are starting to give the special consideration
to parallel processing.

Ruby was historically not very good at concurrent programming (e.g. GIL vs Thread).
However, the status has rapidly changed in the last few years. Especially
the introduction of Ractor in Ruby 3 was deemed as a big step ahead.

In this presentation, we want to talk about this breeding-edge technology from
application developers' perspective

[fujimotos]

RubyKaigi Taketout 2021 CFP draft (v4)

# Title

Ractor: Experimenting with the new parallelization approach

# Abstract

Parallelism has become the norm in modern programming. As more cores are
added to CPU chips, developers are expected to integrate multi-threading
into programs to utilize these additional cores.

Ractor is the new multi-threading mechanism introduced in Ruby 3.0.0.
Recently, Fluentd team experimented with this new technique to see if it
can bring higher data throughputs.

In this talk, we explain our experience with Ractor and discuss the
current state of the technology.

# Details

The goal of this talk is to show how to use Ractor, the breeding-edge
multithreading feature, with practical use cases.

Here is the (planned) outline of my talk:

* Introduction: What is Ractor?
  * Ractor = Actor-based multi-threading
  * Ractor can be executed in parallel (vs Threads)
  * Ractor is easier and safer to use
  * Caveats (1) Limited object sharing
  * Caveats (2) Not all gems support it yet
* Practical performance of Ractor
  * Basics: How Fluentd works
  * Make Fluentd Plugin Ractor-ready (1)
  * Make Fluentd Plugin Ractor-ready (2)
  * Make Fluentd Plugin Ractor-ready (3)
  * Experiment 
  * Result 
* Open problems
  * Some cases Ractor cannot handle well
  * On Gem ecosystem and Ractor
  * The experimental side of Ractor
* Conclusion

# Pitch

Parallelism is a hot topic right now. More and more programming languages
are integrating multi-threading technologies into their design (e.g. Go's
goroutines), and some are getting real traction.

Ruby was historically heavily restricted to a single thread. However, the
status has been rapidly changing in recent years. In particular, the
introduction of Ractor in 3.0.0 was regarded as a big step ahead.

Still, this technology is relatively new, and not much experience is
accumulated on community. I think our presentation will act as a good
introduction for general Ruby developers.

[fujimotos]

@kou @ashie @kenhys CFP submitted at Jun 30 23:30.
The actual submitted version is #3435 (comment).

I close this issue as done now.