Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory Use exploded in 2.6 #812

Closed
schneems opened this issue Oct 14, 2014 · 12 comments
Closed

Memory Use exploded in 2.6 #812

schneems opened this issue Oct 14, 2014 · 12 comments

Comments

@schneems
Copy link

Rails 4.2 is moving to Mail 2.6.1 and i'm seeing a huge increase in memory usage in my running application. After some digging, it looks like the culprit was the Mail version bump.

To demonstrate the issue, here's a script to reproduce without rails:

Build Gemfiles

$ mkdir 2.6.1
$ cat <<  EOF > 2.6.1/Gemfile
  source 'https://rubygems.org'
  gem 'mail', '2.6.1'
  gem 'get_process_mem'
EOF

$ mkdir 2.5.4
$ cat <<  EOF > 2.5.4/Gemfile
  source 'https://rubygems.org'
  gem 'mail', '2.5.4'
  gem 'get_process_mem'
EOF

Install

$ env BUNDLE_GEMFILE=2.6.1/gemfile bundle install
$ env BUNDLE_GEMFILE=2.5.4/gemfile bundle install

Test script

$ cat <<  EOF > script.rb
  require 'get_process_mem'
  mem = GetProcessMem.new
  before = mem.mb
  require 'mail'
  after = mem.mb
  puts after - before
EOF

Results

$ env BUNDLE_GEMFILE=2.6.1/gemfile bundle exec ruby script.rb
# => 41.44140625 # mb
$ env BUNDLE_GEMFILE=2.5.4/gemfile bundle exec ruby script.rb
# => 11.41015625 # mb

EWAT

Mail 2.6.1 takes up almost 4 times more memory at Require time 0_o. Initial guess is that this has to do with the switch from treetop to ragel via @bpot

cc/@matthewd
@schneems
Copy link
Author

Here's a breakdown of libraries inside of mail and their RSS cost

mail: 22.93 mb
  mail/parsers: 21.82 mb
    mail/parsers/ragel: 21.82 mb
      mail/parsers/ragel/ruby: 21.8 mb
        mail/parsers/ragel/ruby/machines/address_lists_machine: 17.27 mb
        mail/parsers/ragel/ruby/machines/received_machine: 2.75 mb
        mail/parsers/ragel/ruby/machines/date_time_machine: 0.69 mb
        mail/parsers/ragel/ruby/machines/envelope_from_machine: 0.36 mb
        mail/parsers/ragel/ruby/machines/message_ids_machine: 0.34 mb
        mail/parsers/ragel/ruby/machines/phrase_lists_machine: 0.32 mb
        mail/parsers/ragel/ruby/machines/content_location_machine: 0.02 mb
        mail/parsers/ragel/ruby/machines/mime_version_machine: 0.01 mb
        mail/parsers/ragel/ruby/machines/content_disposition_machine: 0.0 mb
        mail/parsers/ragel/ruby/machines/content_transfer_encoding_machine: 0.0 mb
        mail/parsers/ragel/ruby/machines/content_type_machine: 0.0 mb
      mail/parsers/ragel/parser_info: 0.0 mb
    mail/parsers/message_ids_parser: 0.0 mb
    mail/parsers/address_lists_parser: 0.0 mb
    mail/parsers/phrase_lists_parser: 0.0 mb
    mail/parsers/date_time_parser: 0.0 mb
    mail/parsers/received_parser: 0.0 mb
    mail/parsers/ragel/parser_info: 0.0 mb
    mail/parsers/envelope_from_parser: 0.0 mb
    mail/parsers/mime_version_parser: 0.0 mb
    mail/parsers/content_type_parser: 0.0 mb
    mail/parsers/content_disposition_parser: 0.0 mb
    mail/parsers/content_transfer_encoding_parser: 0.0 mb
    mail/parsers/content_location_parser: 0.0 mb
  mail/field: 0.46 mb
    mail/fields: 0.0 mb
    mail/fields/common/common_address: 0.0 mb
    mail/fields/common/common_field: 0.0 mb
    mail/fields/common/common_message_id: 0.0 mb
    mail/fields/common/common_date: 0.0 mb
    mail/fields/common/parameter_hash: 0.0 mb
  mail/message: 0.24 mb
    yaml: 0.0 mb
  net/smtp: 0.17 mb
    net/protocol: 0.0 mb
    digest/md5: 0.0 mb
    timeout: 0.0 mb
    openssl: 0.0 mb
  mail/body: 0.04 mb
  mail/header: 0.03 mb
  mail/attachments_list: 0.02 mb
  mail/indifferent_hash: 0.02 mb
  mail/core_extensions/string: 0.02 mb
  mail/mail: 0.02 mb
  mail/parts_list: 0.02 mb
  mail/network: 0.02 mb
    mail/network/retriever_methods/base: 0.02 mb
  mail/utilities: 0.01 mb
  mail/part: 0.01 mb
  mail/core_extensions/nil: 0.01 mb
  mail/version: 0.01 mb
  mail/encodings/base64: 0.0 mb
    mail/encodings/7bit: 0.0 mb
      mail/encodings/8bit: 0.0 mb
        mail/encodings/binary: 0.0 mb
          mail/encodings/transfer_encoding: 0.0 mb
  mime/types: 0.0 mb
  mail/core_extensions/object: 0.0 mb
  mail/patterns: 0.0 mb
  mail/version_specific/ruby_1_9: 0.0 mb
  mail/configuration: 0.0 mb
    singleton: 0.0 mb
  uri: 0.0 mb
  mail/field_list: 0.0 mb
  mail/envelope: 0.0 mb
  shellwords: 0.0 mb
  mail/elements: 0.0 mb
  mail/encodings: 0.0 mb
  active_support/inflector: 0.0 mb
  mail/encodings/quoted_printable: 0.0 mb
    mail/encodings/7bit: 0.0 mb
  mail/matchers/has_sent_mail: 0.0 mb
  date: 0.0 mb

Also the mime/types gem takes 21.16 mb.

@bf4
Copy link
Collaborator

bf4 commented Oct 21, 2014

yegawds! Thanks @schneems I don't know anything about the cost/benefit of memory vs. performance of the parser switch. Is this something you might look into further? Would it help if @mikel adds you as a collab? @jeremy thoughts?

I was thinking we're about ready for a 2.7.0 release and a fix for this would be good to get into it.

@jeremy
Copy link
Collaborator

jeremy commented Oct 21, 2014

Possible to defer the parser requires until we need to parse a message?

@bpot
Copy link
Contributor

bpot commented Oct 21, 2014

I think it should be possible to defer the requires.

Ragel also has a number of different ways to generate state machines, it may be possible to generate one that has less memory overhead. There may also be room to tweak the parser definition to take up less memory while still being correct. It looks like it's mostly the address lists parser that is the problem.

I'll try to play around with this today.

@bf4
Copy link
Collaborator

bf4 commented Oct 21, 2014

re: requires: maybe use Module#autoload? It's threadsafe in Ruby 2.0+, rbx-2, and jruby

@schneems
Copy link
Author

If possible delaying the require here would be the best, then apps that only parse mail need to have the extra memory usage. I think that re-writing some of the Ragel state machines to reduce size is a potential for a future project. If we delay the require that will be a great first step.

Btw, thanks for all your work. I've not done benchmarking on speed, but i hear that it's much faster ❤️ ❤️

@bpot
Copy link
Contributor

bpot commented Oct 21, 2014

Cool. For future reference, I tried changing the state machine generation mode (-F1 to -T0) and was able to reduce memory usage from 14MB to 3MB but at the cost of ~3x reduction in performance.

@bf4 bf4 mentioned this issue Oct 22, 2014
Merged
@grosser grosser mentioned this issue Oct 22, 2014
Closed
@grosser
Copy link
Contributor

grosser commented Oct 22, 2014

does this help: #815 ?

@jeremy jeremy closed this as completed Oct 23, 2014
@bf4 bf4 mentioned this issue Oct 23, 2014
Merged
@vishnurkrishnan
Copy link

vishnurkrishnan commented Aug 2, 2017 •
edited
Loading

How can we do this with rails 3.2.22 and ruby 2.0?, because i can't upgrade the mail gem to 2.6, it seems that is not compatible with action mailer, i was following this https://www.schneems.com/2014/11/07/i-ram-what-i-ram.html

@grosser
Copy link
Contributor

grosser commented Aug 2, 2017 via email

@vishnurkrishnan
Copy link

vishnurkrishnan commented Aug 3, 2017 •
edited
Loading

Yes, but here in this https://www.schneems.com/2014/11/07/i-ram-what-i-ram.html says need to upgrade to 2.6.3 or higher, because this 2.6.1 version is using high memory and that has fixed in 2.6.3, so will it be possible with this patch '261-as-259'?

@grosser
Copy link
Contributor

grosser commented Aug 3, 2017

nope, then do the same patch again but from 2.6.3 and it should be good

@jeremy jeremy mentioned this issue May 30, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@jeremy @bpot @grosser @schneems @bf4 @vishnurkrishnan