Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TODO] nim-dustmite source code minimization tool for automated bug reduction #8276

Closed
timotheecour opened this issue Jul 11, 2018 · 13 comments

Comments

@timotheecour
Copy link
Member

timotheecour commented Jul 11, 2018

As I was trying to reduce a compiler error that only triggers in my full project and is hard/tedious to minimize manually, it occurred to me we could try to develop a source code minimization tool to automate that.

in D, that tool exists, see https://github.com/CyberShadow/DustMite and docs https://github.com/CyberShadow/DustMite/wiki ; developped by @CyberShadow
It's used extensively by the community for investigating bugs; it's especially invaluable for those that trigger in your whole project but are hard to isolate to a minimal bug (which is the 1st step to solving it). I've used it myself a lot for great benefit.

It turns out dustmite is not super tied to D language (doesn't need to know about D's semantics) ; in fact it can reduce C++ code too as it only operates at much shallower level.

  • First step could be adapting/forking dustmite to work with Nim code

@CyberShadow any help / pointers in this thread would be most welcome!

  • second step (optional, not needed for that purpose...) could be porting that forked dustmite to nim

links

@Araq
Copy link
Member

Araq commented Jul 11, 2018

IIRC @zielmicha had a tool that does this.

@Araq
Copy link
Member

Araq commented Jul 11, 2018

I'm not too happy these tools as they tend to produce "too minimal" test cases that then get fixed but in the real, larger example the problem remains. Minimal test cases cause an "overfit".

@CyberShadow
Copy link

CyberShadow commented Jul 11, 2018

@CyberShadow any help / pointers in this thread would be most welcome!

DustMite has, more or less, pluggable splitters:

https://github.com/CyberShadow/DustMite/blob/7403f0d053b96b2c21950df038923b5d335d557f/splitter.d#L75-L82

There is already support for reducing files in a second "syntax" (patch files). D mode should, to some extent, work with any curly-brace languages already though.

I'm not too happy these tools as they tend to produce "too minimal" test cases that then get fixed but in the real, larger example the problem remains. Minimal test cases cause an "overfit".

In our experience this happens very rarely (less than 1% of test cases). In any case, this mistake can be done by humans as well, who are not aware of the exact internal mechanics that cause the bug.

What happens more commonly is that, along the way of reducing the intended bug, DustMite generates a test case that triggers a different bug with similar symptoms (e.g. a compiler segfault), and then gets sidetracked into reducing that bug. In such cases, a more strict test script (e.g. which checks the segfault stack trace) is the solution.

https://github.com/CyberShadow/DustMite/wiki/Detecting-a-specific-segfault

@krux02 krux02 changed the title [RFC] source code minimization tool for automated bug reduction (port D's dustmite or adapt it to work with Nim) source code minimization tool for automated bug reduction Oct 28, 2018
@krux02
Copy link
Contributor

krux02 commented Oct 28, 2018

What is a "compiler ICE"?

@krux02
Copy link
Contributor

krux02 commented Oct 29, 2018

I added the Won't fix label, because that is how I understand the status quo.

@Araq
Copy link
Member

Araq commented Oct 29, 2018

@timotheecour Feel free to submit one, it's not a bad idea, but we don't intent to work on one.

@timotheecour
Copy link
Member Author

timotheecour commented Oct 30, 2018

What is a "compiler ICE"?

internal compiler error; looks like nim doesn't use this terminology used in some other languages; I rephrasd it as just 'compiler error'

  • changed from wontfix to postponed

@timotheecour timotheecour changed the title source code minimization tool for automated bug reduction nim-dustmite source code minimization tool for automated bug reduction Nov 20, 2020
@timotheecour timotheecour changed the title nim-dustmite source code minimization tool for automated bug reduction [TODO] nim-dustmite source code minimization tool for automated bug reduction Nov 20, 2020
@timotheecour
Copy link
Member Author

see recent project: disruptek/dust: DUST is Unattended Syntax Truncation

@CyberShadow
Copy link

CyberShadow commented Nov 21, 2020

I am obviously extremely biased, but at this point I would recommend implementing similar tools as an add-on to DustMite. I spent some time earlier this year working on some algorithmic improvements to DustMite which made it significantly faster than earlier versions.

If implementing a Nim parser/lexer in D is undesirable and a Nim parser/lexer is already available as a Nim library that could be used for this purpose, then what we can do is make the Nim program produce an intermediary format with a pre-split reducible tree that DustMite can consume, and invoke DustMite with it. I'll be happy to help with the D/DustMite side if there's interest.

CC @disruptek

@disruptek
Copy link
Contributor

Sounds great; gimme a spec and I'll write the Nim side. Thank you!

@CyberShadow
Copy link

@disruptek
Copy link
Contributor

Sorry I missed this. Looks pretty easy; I will reach out in the repo if I have any trouble. Thanks again! Looking forward to seeing this thing work. 🙂

@timotheecour
Copy link
Member Author

timotheecour commented Dec 6, 2020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants