Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

auto_instruments breaks coverage #343

Open
jasdeepgosal opened this issue Jul 13, 2020 · 15 comments
Open

auto_instruments breaks coverage #343

jasdeepgosal opened this issue Jul 13, 2020 · 15 comments

Comments

@jasdeepgosal
Copy link

We are currently trying to use https://github.com/danmayer/coverband to measure production code usage, and that gem relies on https://ruby-doc.org/stdlib-2.6.6/libdoc/coverage/rdoc/Coverage.html to actually measure when code is executed.

Unfortunately, when I enable SCOUT_AUTO_INSTRUMENTS, it stops reporting any coverage and I think it's because this: https://github.com/scoutapp/scout_apm_ruby/blob/master/lib/scout_apm/auto_instrument/instruction_sequence.rb#L19 rewrites the code, thus preventing it from being measured by Coverage?

Any advice?

Thanks!

scout_apm_ruby version: 2.6.8
rails: 6.0.3.2
ruby: 2.6.6

@danmayer
Copy link

let me know if anyone at scout looks at this or if there is a way we could make the lib play more nicely with Coverage.

@AshAngell
Copy link

I too am having this problem...Can anyone at Scout please advise?

@danmayer
Copy link

Hey @scoutapm folks, since I get tagged in this. If you find what you think is a good solution for other libraries or users of Coverage, just let me know and I can try to implement it or give something a shot... at the moment, not sure of a good work around for auto instrumented code.

@dlanderson
Copy link
Contributor

Thanks @danmayer - I'm not familiar with how Coverband detects which code has been executed. Can you give me a pointer to how it works?

Does coverband work when https://github.com/Shopify/bootsnap is enabled? IIRC we use the same compile_file.

@danmayer
Copy link

So Coverband is just a wrapper around Ruby's StdLib Coverage module. It adds some helpers mostly to collect and view the data... The issues users have reported or found between scout and Coverage get reported on Coverband sometimes, but really are just about how Coverage works.

https://ruby-doc.org/stdlib-2.5.3/libdoc/coverage/rdoc/Coverage.html

The important part is that Coverage has to start before various Ruby files are required, and then it instruments those files as they are parsed into "Ruby machine code"... As I understand it scout tries to rewrite some of the files vs just patching them... I am thinking that somehow the re-written instruction sequences end up not having Coverage's expected instrumentation applied.

@danmayer
Copy link

I haven't seen any issues with bootsnap, I could try to test it out a bit more specifically, but we use bootsnap on most of our work projects and I have never run into issues.

@yskkin
Copy link

yskkin commented May 28, 2021

We are experiencing same problem with Rails 5.1.4 and Ruby 2.6.6.
So bootsnap should be irrelevant.

It works fine on Rails 5.1.4 and Ruby 2.5.9.

@amkisko
Copy link

amkisko commented Oct 26, 2023

Still relevant, had to turn it off in test environment, but previously it was useful to track issues with decorators usage.

@ksweetie
Copy link

ksweetie commented Mar 1, 2024

@jrothrock Hey, sorry to bother you. Do you know if there's any workaround or path forward for this? We needed to uninstall Coverband, but would really love to turn it back on. Thank you!

@jrothrock
Copy link
Contributor

@ksweetie The only real workaround at this time is to disable / set auto_instruments: false in the test environment:
https://scoutapm.com/apps/new_ruby_application_configuration

This can also be done with an environment variable: SCOUT_AUTO_INSTRUMENTS=false

One thing to note is that if a specific configuration key-value pair is being set in both the scout_apm.yml as well as environment variables, we will use the value from the environment variable

@ksweetie
Copy link

ksweetie commented Mar 4, 2024

@jrothrock Thanks for the fast response!

Coverband is meant for production environments, so we weren't using it in test. We used it to spot dead code paths that hadn't been executed in months.

It sounds like we can't run both Scout and Coverband in production. Is the intention to fix that, or are they fundamentally incompatible?

@danmayer
Copy link

danmayer commented Mar 4, 2024

My understanding is the way that auto instrument rewrites code isn't compatible with the Ruby standard lib Coverage approach... So Coverband which relies on standard lib coverage isn't compatible with auto instrument.

@jrothrock
Copy link
Contributor

I brought this up with our product team. We would like to get it worked in, but I'm not exactly certain on timelines

@ksweetie
Copy link

ksweetie commented Mar 6, 2024

@jrothrock thank you, I appreciate it!

@danmayer
Copy link

danmayer commented Mar 6, 2024

very cool. If you all find anything that could help even if it is just better documenting the issue, let me know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants