auto_instruments breaks coverage

[jasdeepgosal]

We are currently trying to use https://github.com/danmayer/coverband to measure production code usage, and that gem relies on https://ruby-doc.org/stdlib-2.6.6/libdoc/coverage/rdoc/Coverage.html to actually measure when code is executed.

Unfortunately, when I enable SCOUT_AUTO_INSTRUMENTS, it stops reporting any coverage and I think it's because this: https://github.com/scoutapp/scout_apm_ruby/blob/master/lib/scout_apm/auto_instrument/instruction_sequence.rb#L19 rewrites the code, thus preventing it from being measured by Coverage?

Any advice?

Thanks!

scout_apm_ruby version: 2.6.8
rails: 6.0.3.2
ruby: 2.6.6

[danmayer]

let me know if anyone at scout looks at this or if there is a way we could make the lib play more nicely with Coverage.

[AshAngell]

I too am having this problem...Can anyone at Scout please advise?

[danmayer]

Hey @scoutapm folks, since I get tagged in this. If you find what you think is a good solution for other libraries or users of Coverage, just let me know and I can try to implement it or give something a shot... at the moment, not sure of a good work around for auto instrumented code.

[dlanderson]

Thanks @danmayer - I'm not familiar with how Coverband detects which code has been executed. Can you give me a pointer to how it works?

Does coverband work when https://github.com/Shopify/bootsnap is enabled? IIRC we use the same compile_file.

[danmayer]

So Coverband is just a wrapper around Ruby's StdLib Coverage module. It adds some helpers mostly to collect and view the data... The issues users have reported or found between scout and Coverage get reported on Coverband sometimes, but really are just about how Coverage works.

https://ruby-doc.org/stdlib-2.5.3/libdoc/coverage/rdoc/Coverage.html

The important part is that Coverage has to start before various Ruby files are required, and then it instruments those files as they are parsed into "Ruby machine code"... As I understand it scout tries to rewrite some of the files vs just patching them... I am thinking that somehow the re-written instruction sequences end up not having Coverage's expected instrumentation applied.

[danmayer]

I haven't seen any issues with bootsnap, I could try to test it out a bit more specifically, but we use bootsnap on most of our work projects and I have never run into issues.

[yskkin]

We are experiencing same problem with Rails 5.1.4 and Ruby 2.6.6.
So bootsnap should be irrelevant.

It works fine on Rails 5.1.4 and Ruby 2.5.9.

[amkisko]

Still relevant, had to turn it off in test environment, but previously it was useful to track issues with decorators usage.

[ksweetie]

@jrothrock Hey, sorry to bother you. Do you know if there's any workaround or path forward for this? We needed to uninstall Coverband, but would really love to turn it back on. Thank you!

[jrothrock]

@ksweetie The only real workaround at this time is to disable / set auto_instruments: false in the test environment:
https://scoutapm.com/apps/new_ruby_application_configuration

This can also be done with an environment variable: SCOUT_AUTO_INSTRUMENTS=false

One thing to note is that if a specific configuration key-value pair is being set in both the scout_apm.yml as well as environment variables, we will use the value from the environment variable

[ksweetie]

@jrothrock Thanks for the fast response!

Coverband is meant for production environments, so we weren't using it in test. We used it to spot dead code paths that hadn't been executed in months.

It sounds like we can't run both Scout and Coverband in production. Is the intention to fix that, or are they fundamentally incompatible?

[danmayer]

My understanding is the way that auto instrument rewrites code isn't compatible with the Ruby standard lib Coverage approach... So Coverband which relies on standard lib coverage isn't compatible with auto instrument.

[jrothrock]

I brought this up with our product team. We would like to get it worked in, but I'm not exactly certain on timelines

[ksweetie]

@jrothrock thank you, I appreciate it!

[danmayer]

very cool. If you all find anything that could help even if it is just better documenting the issue, let me know.