Add --diff to report only linter messages introduced by a git diff #472

MasseGuillaume · 2017-12-01T18:53:37Z

No description provided.

This implement the --diff command to use scalafix in an existing codebase. It run the git diff algoritm via JGit and only lint messages where the code is modified. > I figured out the JGit api calls by looking at DiffCommand.java https://github.com/eclipse/jgit/blob/v4.9.1.201712030800-r/org.eclipse.jgit/src/org/eclipse/jgit/api/DiffCommand.java#L113 > to print the git diff from jgit import org.eclipse.jgit.api.Git val git = new Git(repository) git .diff() .setOldTree(oldTree2) .setNewTree(newTree2) .setContextLines(0) .setOutputStream(System.out) .call() > DiffFormatter.setProgressMonitor Will only show when the rename algorithm is used. Not really useful.

olafurpg

jgit looks great, nice call! Only important comment is the git tests, i wonder if we can reuse the same infrastructure as the other cli tests with only diffs added to the mix.

olafurpg · 2017-12-04T12:13:25Z

scalafix-cli/src/main/scala/scalafix/internal/jgit/JGitDiff.scala

+    iterator(repository, _.parseCommit(repository.exactRef(ref).getObjectId()))
+
+  // could be handy in future
+  private def commit(


If this is unused then we should remove it.

olafurpg · 2017-12-04T12:16:36Z

scalafix-cli/src/main/scala/scalafix/internal/jgit/JGitDiff.scala

+
+    def edits(file: FileHeader): Unit = {
+      val changes = List.newBuilder[GitChange]
+      file.toEditList.asScala.foreach { edit =>


Could this be turned into a map? Builders are great when they're needed but they should be avoided if they can be replaced with a single map (optionally to iterator before and to list after)

olafurpg · 2017-12-04T12:17:31Z

scalafix-cli/src/main/scala/scalafix/internal/jgit/JGitDiff.scala

+      diffs += ModifiedFile(path(file.getNewPath), changes.result())
+    }
+
+    getDiff(repository, oldTree1, newTree1).foreach { file =>


Same here, could this be a flatmap?

olafurpg · 2017-12-04T12:18:10Z

scalafix-tests/unit/src/test/scala/scalafix/tests/cli/CliGitDiffTests.scala

+          |  ^
+          |""".stripMargin
+
+    println("-------")


No println in tests :)

olafurpg · 2017-12-04T12:20:39Z

scalafix-tests/unit/src/test/scala/scalafix/tests/cli/CliGitDiffTests.scala

+import scalafix.testkit.DiffAssertions
+
+class CliGitDiffTests() extends FunSuite with DiffAssertions {
+  gitTest("it should handle addition") { (fs, git, cli) =>


Name this test "addition"? The convention in scalafix tests is to try keep test names as concise as possible

with utest it's Possible to run a single test by name, which makes sentences annoying to work with

olafurpg · 2017-12-04T12:22:20Z

scalafix-tests/unit/src/test/scala/scalafix/tests/cli/CliGitDiffTests.scala

+    assertNoDiff(obtained, expected)
+  }
+
+  gitTest("it should handle modification") { (fs, git, cli) =>


olafurpg · 2017-12-04T12:23:56Z

scalafix-tests/unit/src/test/scala/scalafix/tests/cli/CliGitDiffTests.scala

+  }
+
+  private class Fs() {
+    val workingDirectory: Path =


Can we replace this utility with jgit?

Commented too early you are using jgit! 👍

olafurpg · 2017-12-04T12:26:25Z

scalafix-tests/unit/src/test/scala/scalafix/tests/cli/CliGitDiffTests.scala

+  private class Git(workingDirectory: Path) {
+    import org.eclipse.jgit.api.{Git => JGit}
+
+    private val git = JGit.init().setDirectory(workingDirectory.toFile).call()


Is it possible to apply a unified diff string? I wonder if we can refactor the tests into a "check" method that takes a name, a StringFS string and expected linter output? Similar to how BaseCliTest supports

nope, I don't think there is a good opportunity for refactoring. If we add a lot of stuff in BaseCliTest it's going to be unnecessarily complex.

olafurpg · 2017-12-04T12:37:35Z

scalafix-tests/unit/src/test/scala/scalafix/tests/cli/CliGitDiffTests.scala

+    assertNoDiff(obtained, expected)
+  }
+
+  gitTest("it should handle rename") { (fs, git, cli) =>


Can we add a test for unstaged/staged files? From my experience in scalafmt, we should treat those as new files. It's convenient to be able to run scalafix on changes before commit

git diff will only pick staged files with git diff --cached is this what you want?

MasseGuillaume · 2017-12-04T12:46:49Z

Ready for round 2.

olafurpg

Running the cli now results in new log entries from SLF4J

$ scalafix-nightly -r DisableSyntax --config-str="DisableSyntax.keywords=[def]" --diff
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

Can we avoid that entry? I'm fine with adding a dependency on logback.

It seems we are not using --diff to decide which files to run on.

$ scalafix-nightly -r DisableSyntax --config-str="DisableSyntax.keywords=[def]" --diff-branch drop-in-jgit
Running DisableSyntax
  100.0% [##########] 249 source files fixed

That diff is empty so I expected it to run scalafix on 0 files instead of 249 files. We use DiffDisable here to filter our mismatching files from result

scalafix/scalafix-cli/src/main/scala/scalafix/cli/CliRunner.scala

Line 413 in 9a8451e

} else Ok(result)

It seems a NPE is thrown when we pass jgit a tag instead of a branch

scalafix-nightly -r DisableSyntax --config-str="DisableSyntax.keywords=[def]" --diff-branch=v0.5.6
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
error: java.lang.NullPointerException
scalafix.internal.jgit.JGitDiff$.$anonfun$ref$1(JGitDiff.scala:57)
scalafix.internal.jgit.JGitDiff$.iterator(JGitDiff.scala:63)
scalafix.internal.jgit.JGitDiff$.ref(JGitDiff.scala:57)
scalafix.internal.jgit.JGitDiff$.apply(JGitDiff.scala:32)
scalafix.internal.jgit.DiffDisable$.apply(DiffDisable.scala:17)

MasseGuillaume · 2017-12-05T17:26:00Z

All comments addressed, ready for round 3.

olafurpg · 2017-12-06T08:37:02Z

scalafix-cli/src/main/scala/scalafix/cli/CliRunner.scala

+    val diffDisable: Option[DiffDisable] = {
+      if (cli.diff || cli.diffBranch.nonEmpty) {
+        val baseBranch = cli.diffBranch.getOrElse("master")
+        Some(DiffDisable(common.workingPath.toNIO, baseBranch))


What happens when DiffDisable fails? For example a non-existent branch is used. We can convert diffDisable into a Configured[Option[DiffDisable]] and (resolvedPathMatcher |@| diffDisable).map

olafurpg · 2017-12-06T08:56:44Z

scalafix-cli/src/main/scala/scalafix/internal/jgit/JGitDiff.scala

+import scala.collection.JavaConverters._
+
+object JGitDiff {
+  def apply(workingDir: Path, baseBranch: String): List[GitDiff] = {


Can we make this return Configured[List[GitDiff]] and use Configured.error("message") instead of throwing exceptions?

olafurpg · 2017-12-06T08:57:23Z

scalafix-cli/src/main/scala/scalafix/internal/jgit/JGitDiff.scala

+    val git = new Git(repository)
+    val branches = git.branchList().call()
+    if (branches.isEmpty) {
+      throw new Exception(s"$workingDir is not a git repository")


If we use exceptions we should try to create custom exception types instead of java.lang.Exception so that these can be caught without string fiddling.

olafurpg · 2017-12-06T09:00:01Z

scalafix-cli/src/main/scala/scalafix/internal/jgit/JGitDiff.scala

+    }
+    val baseBranchExists = branches.asScala.exists(_.getName() == baseBranchRef)
+    if (!baseBranchExists) {
+      throw new Exception(s"Cannot find git branch $baseBranch")


This error is reported for tags, can we treat tags like branches?

scalafix-nightly -r DisableSyntax --config-str="DisableSyntax.keywords=[def]" --diff-branch=v0.5.6  8   drop-in-jgit error: java.lang.Exception: Cannot find git branch v0.5.6 scalafix.internal.jgit.JGitDiff$.apply(JGitDiff.scala:40)

I will rename --diff-branch= to --diff-base= and allow more options (hash and tags)

olafurpg

It seems some branches are not picked up

 ollie  ~  dev  scalafix  scalafix-nightly -r DisableSyntax --config-str="DisableSyntax.keywords=[def]" --diff-branch=vararg-pattern
error: Cannot find git branch vararg-pattern
 ollie  ~  dev  scalafix  git checkout vararg-pattern                                        8   drop-in-jgit
Branch vararg-pattern set up to track remote branch vararg-pattern from gosubpl.
Switched to a new branch 'vararg-pattern'

This is not urgent to fix, master is picked up. We can open an issue to fix this separately.

olafurpg · 2017-12-06T10:50:03Z

scalafix-cli/src/main/scala/scalafix/cli/CliRunner.scala

      }
    }

-    val fixFiles: Configured[Seq[FixFile]] =
+    val fixFiles: Configured[Seq[FixFile]] = diffDisable.andThen(diffDisable0 =>
      resolvedPathMatcher.andThen { pathMatcher =>


You can use diffDisable.product(resolvedPathMatcher) or |@| instead of product

olafurpg · 2017-12-06T10:51:07Z

scalafix-cli/src/main/scala/scalafix/internal/jgit/JGitDiff.scala


-    val oldTree1 = ref(repository, baseBranchRef)
-    val newTree1 = new FileTreeIterator(repository)
+    if (!branches.isEmpty) {


It might be more readable to have the Configured.error branch come first

olafurpg · 2017-12-06T10:53:48Z

scalafix-core/shared/src/main/scala/scalafix/internal/util/IntervalSet.scala

+
+  private def fromRange(xs: List[(Int, Int)]): BitSet = {
+    val n = 64
+    val max = xs.maxBy(_._2)._2


We should handle the case when xs is empty

scalafix-nightly -r DisableSyntax --config-str="DisableSyntax.keywords=[def]" --diff-branch=master error: java.lang.UnsupportedOperationException: empty.maxBy scala.collection.TraversableOnce.maxBy(TraversableOnce.scala:236)

MasseGuillaume · 2017-12-06T13:10:42Z

ollie  ~  dev  scalafix  scalafix-nightly -r DisableSyntax --config-str="DisableSyntax.keywords=[def]" > --diff-branch=vararg-pattern
error: Cannot find git branch vararg-pattern
ollie  ~  dev  scalafix  git checkout vararg-pattern  8   drop-in-jgit
Branch vararg-pattern set up to track remote branch vararg-pattern from gosubpl.
Switched to a new branch 'vararg-pattern'

This works now, even with remote branch:

./scalafix-nightly -r DisableSyntax --config-str="DisableSyntax.keywords=[def]" --diff-base=olaf/2.12.4
/home/gui/scalafix/scalafix-core/shared/src/main/scala/scalafix/rule/RuleCtx.scala:55:21: error: [DisableSyntax.keywords.def] keywords.def is disabled 5.0% [          ] 1 source files fixed
  private[scalafix] def filterLintMessage(
                    ^
/home/gui/scalafix/project/Dependencies.scala:15:3: error: [DisableSyntax.keywords.def] keywords.def is disabled
  def scala212 = "2.12.3"ource files fixed
  ^
/home/gui/scalafix/project/Dependencies.scala:19:3: error: [DisableSyntax.keywords.def] keywords.def is disabled
  def sbt1 = "1.0.3" // blocked by https://github.com/scalameta/scalameta/issues/1157
  ^
/home/gui/scalafix/scalafix-tests/unit/src/test/scala/scalafix/tests/cli/CliGitDiffTests.scala:154:11: error: [DisableSyntax.keywords.def] keywords.def is disabled    ] 8 source files fixed
  private def runDiff(cli: Cli, args: String*): String =
          ^
/home/gui/scalafix/scalafix-tests/unit/src/test/scala/scalafix/tests/cli/CliGitDiffTests.scala:158:11: error: [DisableSyntax.keywords.def] keywords.def is disabled
  private def addConf(fs: Fs, git: Git): Unit = {
          ^
...

MasseGuillaume · 2017-12-06T13:16:03Z

Ready for round 4!

olafurpg

You outdid yourself @MasseGuillaume! Very nice job, I tested locally it works great, LGTM 👍

olafurpg · 2017-12-06T17:31:03Z

scalafix-cli/src/main/scala/scalafix/internal/jgit/JGitDiff.scala

+    try {
+      Option(repo.resolve(revstr)) match {
+        case Some(id) => Right(id)
+        case None => Left(s"cannot resolve $revstr")


cannot resolve --diff-base $revstr

MasseGuillaume changed the title ~~Drop in jgit~~ WIP: Drop in jgit Dec 1, 2017

MasseGuillaume force-pushed the drop-in-jgit branch 2 times, most recently from a69b514 to c894738 Compare December 1, 2017 19:00

MasseGuillaume changed the title ~~WIP: Drop in jgit~~ Drop-in jgit Dec 4, 2017

MasseGuillaume force-pushed the drop-in-jgit branch 4 times, most recently from 2bb15e3 to e006831 Compare December 4, 2017 11:36

MasseGuillaume force-pushed the drop-in-jgit branch from e006831 to f4e0e3c Compare December 4, 2017 12:25

MasseGuillaume added the ready label Dec 4, 2017

olafurpg suggested changes Dec 4, 2017

View reviewed changes

olafurpg reviewed Dec 4, 2017

View reviewed changes

scalfix --diff address PR review

eadcbc3

olafurpg suggested changes Dec 5, 2017

View reviewed changes

MasseGuillaume added 3 commits December 5, 2017 18:19

Add logback

aff91ad

Only fix modfied and added files when using --diff

6b44166

--diff report when non in a git repo or when a branch is not found

fd22482

MasseGuillaume added 2 commits December 5, 2017 19:36

Optimize IntervalSet to avoid lots of intermediate collections

c375c83

fmt

63e49d7

olafurpg suggested changes Dec 6, 2017

View reviewed changes

--diff use Configured to avoid throwing exceptions

21801fd

olafurpg reviewed Dec 6, 2017

View reviewed changes

MasseGuillaume added 3 commits December 6, 2017 12:52

--diff resolve any reference, not just branches

37b99f8

fmt

127aaa5

empty interval set

b5ba2c1

add handy scalafix-nightly script

0ee0adf

olafurpg approved these changes Dec 6, 2017

View reviewed changes

olafurpg merged commit 76ed12d into scalacenter:master Dec 7, 2017

MasseGuillaume deleted the drop-in-jgit branch December 7, 2017 12:21

olafurpg mentioned this pull request Dec 7, 2017

drop-in feature #462

Closed

olafurpg added this to the v0.5.7 milestone Dec 7, 2017

olafurpg changed the title ~~Drop-in jgit~~ Add --diff to report only linter messages introduced by a git diff Dec 7, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add --diff to report only linter messages introduced by a git diff #472

Add --diff to report only linter messages introduced by a git diff #472

MasseGuillaume commented Dec 1, 2017

olafurpg left a comment

olafurpg Dec 4, 2017

olafurpg Dec 4, 2017

olafurpg Dec 4, 2017

olafurpg Dec 4, 2017

olafurpg Dec 4, 2017

olafurpg Dec 4, 2017

olafurpg Dec 4, 2017

olafurpg Dec 4, 2017

olafurpg Dec 4, 2017

olafurpg Dec 4, 2017

MasseGuillaume Dec 4, 2017

olafurpg Dec 4, 2017

MasseGuillaume Dec 4, 2017

MasseGuillaume commented Dec 4, 2017

olafurpg left a comment •

edited

Loading

MasseGuillaume commented Dec 5, 2017

olafurpg Dec 6, 2017

olafurpg Dec 6, 2017

olafurpg Dec 6, 2017

olafurpg Dec 6, 2017

MasseGuillaume Dec 6, 2017

olafurpg left a comment

olafurpg Dec 6, 2017

olafurpg Dec 6, 2017

olafurpg Dec 6, 2017

MasseGuillaume commented Dec 6, 2017 •

edited

Loading

MasseGuillaume commented Dec 6, 2017

olafurpg left a comment

olafurpg Dec 6, 2017

Add --diff to report only linter messages introduced by a git diff #472

Add --diff to report only linter messages introduced by a git diff #472

Conversation

MasseGuillaume commented Dec 1, 2017

olafurpg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MasseGuillaume commented Dec 4, 2017

olafurpg left a comment • edited Loading

Choose a reason for hiding this comment

MasseGuillaume commented Dec 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olafurpg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MasseGuillaume commented Dec 6, 2017 • edited Loading

MasseGuillaume commented Dec 6, 2017

olafurpg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olafurpg left a comment •

edited

Loading

MasseGuillaume commented Dec 6, 2017 •

edited

Loading