Start a fuzzing suite to test for consistency of lints

[MichaelChirico]

Part of #2191. Progress on #2737. Continuation of #2190.

For this initial PR, I've only added a rule about FUNCTION (function(...) { ... }) and OP-LAMBDA (\(...) { ... }) equivalency.

Hopefully my approach here is extensible for other rules (I tried for that). My plan is for this to only run on main, though it's certainly fast enough to run on every PR.

The upshot is, it smoked out three linters with inconsistencies already!

[MichaelChirico]

@DavisVaughan / @lionel-, I'm curious if any of your recent work on {air}/{treesitter.r} & friends could be re-used here.

The idea is we want to make random injections/edits to the R AST (and then do stuff), my approach here felt quite manual/labor-intensive.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.27%. Comparing base (89e29d8) to head (800ba3f).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2818   +/-   ##
=======================================
  Coverage   99.27%   99.27%           
=======================================
  Files         128      128           
  Lines        7164     7165    +1     
=======================================
+ Hits         7112     7113    +1     
  Misses         52       52           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[DavisVaughan]

The closest thing I can think of is parser_reparse() in {treesitter}
https://davisvaughan.github.io/r-tree-sitter/reference/parser-parse.html

It performs an incremental reparse and hands you back a new tree with the edit you made. It's quite verbose in terms of what you have to provide (bytes and points) but this is a tree-sitter limitation (and I'm sure they had some reason for it).

Note that text here in parser_reparse() is the entire original source document with a single contiguous edit already applied to it (it's not the incremental bit of text)

For Air, I don't think Rowan trees are really intended to be edited. At least, there doesn't seem to be any easy to use API for that (and plus Rowan isn't exposed in any R package right now anyways)

[MichaelChirico]

Interesting, yea the incremental re-parse is very neat, but more useful in an IDE setting. Here the latency of re-parse is not much concern as the edits are only a tiny fraction of the action run time.

For Air, I don't think Rowan trees are really intended to be edited. At least, there doesn't seem to be any easy to use API for that (and plus Rowan isn't exposed in any R package right now anyways)

Hmm, I guess in my mind something like {styler} / {air} is well-suited here if it exposes a friendly API for transformers, i.e. instead of the usual helpful rule like 'make this line <80 characters', we have a more adversarial rule like "add comments in random but syntactically valid places". I not be understanding where in the stack behind the format-on-save tool such an analog would sit.

[lionel-]

For Air, I don't think Rowan trees are really intended to be edited. At least, there doesn't seem to be any easy to use API for that (and plus Rowan isn't exposed in any R package right now anyways)

I may be misunderstanding what you mean but AFAIK the red trees in Rowan are modifiable. I believe this is used for refactorings in rust-analyzer.

[MichaelChirico]

Bumping for review :)

[AshesITR]

This could use a comment explaining that this is a fuzzer entrypoint.

[MichaelChirico]

Here, or in ast_fuzz_test.R? The latter seems more natural to me. But maybe I'm missing your point.

[AshesITR]

Here. The control flow does two back-to-back null-checks on file, which has no obvious reason without knowing about the fuzzer.

[MichaelChirico]

Well, the second is.null() check is inside maybe_write_content(), so I wouldn't look at this code in a vacuum and say "that doesn't look right".

it's two steps because it's not easy to set an on.exit() hook in a parent frame (well, we could: https://yihui.org/en/2017/12/on-exit-parent/ but I'm not sure it's worth the complexity)

[AshesITR]

To me I'd at least wonder why the null check wasn't removed from the call and the call moved into the if branch.

I was thinking something like

# this line is replaced by the fuzzer in .dev/ast_fuzz_test.R

[MichaelChirico]

We might be able to have it both ways if you can suggest an alternative -- IIRC my original thinking was this was an independently OK way to rewrite the code here. For now I have added the comment.

[MichaelChirico]

@AshesITR anything else left to fix in this branch before proceeding?

[AshesITR]

Looks good to me. I haven't been able to test it on my machine for time reasons, though. Feel free to merge anyway.

[MichaelChirico]

NP, the GHA-only parts are not so crucial since they're not user-facing -- only matters insofar as it generates correct fixes in the R code. We can fine-tune the dev-facing stuff at our leisure.