Skip to content

Commit

Permalink
Exclude folders bsp (#3329)
Browse files Browse the repository at this point in the history
fixes #3015

There are several limitations implied by `intellijBSP` client which led
to this implementation:
mainly:
1. There is no way to exclude a directory which is not inside a module
`content root` aka `baseDirectory` (at least I didnt find a way -
excluding `.bsp` is hardcoded in `intellijBSP`).
2. `Intellij` won't include content root of module if it has no sources
defined (this might be a bug in `intellijBSP`).

This implies that there should be at least one module which content root
is equal to project root.

In some sitiations such module is defined by user in `build.sc`, but not
always.
In other cases, a `SyntheticRootBspBuildTargetData` with
`bspTarget.baseDirectory` == `topLevelProjectRoot` would be created to
handle exclusions.

I went for a solution that is very local to `bsp` module.

Alternative solutions I though about:
- make RootModule extend BSPModule
- create synthetic `RootModule with BspModule` either during bootstrap
if there is no module with `BspTarget.baseDirectory` =
`topLevelProjectDirectory`
-  create synthetic `RootModule with BspModule` in bspWorker `State`

However personally I think that alternative solutions seem overkill for
achieving such simple effect as excluding directories.


How it works:

1. `MillBuildServer.State` contains additional field -
`Option[SyntheticRootBspBuildTargetData]`
`SyntheticRootBspBuildTargetData` contains minimal data required for
`BuildTarget` to be reported by`def workspaceBuildTargets` (forced by
limitation 1).
2. this field will be `Some` only if there is no normal module with
`BspBuildTarget.contentRoot `== `topLevelProjectRoot`
     aka. module which can exclude topLevel folders
3. if `SyntheticRootBspBuildTargetData` is created, it is reported in
`def workspaceBuildTargets` together with other modules, but filtered
out from other bspClient requests (exceptions: `buildOutputPaths`, and
`buildTargetSources`)
4. `def buildTargetOutputPaths` - for a module which has
`BspBuildTarget.contentRoot `== `topLevelProjectRoot` (so potentially
`SyntheticRootBspBuildTargetData`) i `os.walk` from
`topLevelProjectRoot` looking for `build.sc` files and in their folders
I exclude ".idea",".bsp","out",".bloop". - os.walk ensures exclusion of
folders in nested mill projects
  note: following paths are excluded from walk
    ```scala
    def ignore(path: os.Path): Boolean = {
          path.last.startsWith(".") ||
          path.endsWith(os.RelPath("out")) ||
          path.endsWith(os.RelPath("target")) ||
          path.endsWith(os.RelPath("docs")) ||
          path.endsWith(os.RelPath("mill-build"))
        }
    ```
Actually if there was no limitations I mentioned, this would be the only
required step.
5. Because of `limitation 2` i had to report some source for
`SyntheticRootBspBuildTargetData` - chosen "src", but this can be any,
even non existing path.

Tested manually:

- Navigation of sources
- Navigation of meta-builds
- Navigation of build.sc
- excluded folders appear in IDE
- solution does not collide with user-made RootModule in build.sc
(actually it has to be a RootModule with JavaModule to be seen by BSP)

---------

Co-authored-by: Li Haoyi <[email protected]>
  • Loading branch information
pawelsadlo and lihaoyi authored Aug 22, 2024
1 parent 6c49d7f commit 5041301
Show file tree
Hide file tree
Showing 7 changed files with 179 additions and 112 deletions.
1 change: 1 addition & 0 deletions bsp/src/mill/bsp/BspContext.scala
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ private[mill] class BspContext(
BspWorker(mill.api.WorkspaceRoot.workspaceRoot, home, log).flatMap { worker =>
os.makeDir.all(home / Constants.bspDir)
worker.startBspServer(
mill.api.WorkspaceRoot.workspaceRoot,
streams,
logStream.getOrElse(streams.err),
home / Constants.bspDir,
Expand Down
1 change: 1 addition & 0 deletions bsp/src/mill/bsp/BspWorker.scala
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import java.net.URL

private trait BspWorker {
def startBspServer(
topLevelBuildRoot: os.Path,
streams: SystemStreams,
logStream: PrintStream,
logDir: os.Path,
Expand Down
2 changes: 2 additions & 0 deletions bsp/worker/src/mill/bsp/worker/BspWorkerImpl.scala
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import scala.concurrent.{Await, CancellationException, Promise}
private class BspWorkerImpl() extends BspWorker {

override def startBspServer(
topLevelBuildRoot: os.Path,
streams: SystemStreams,
logStream: PrintStream,
logDir: os.Path,
Expand All @@ -23,6 +24,7 @@ private class BspWorkerImpl() extends BspWorker {

val millServer =
new MillBuildServer(
topLevelProjectRoot = topLevelBuildRoot,
bspVersion = Constants.bspProtocolVersion,
serverVersion = BuildInfo.millVersion,
serverName = Constants.serverName,
Expand Down
165 changes: 56 additions & 109 deletions bsp/worker/src/mill/bsp/worker/MillBuildServer.scala
Original file line number Diff line number Diff line change
@@ -1,87 +1,31 @@
package mill.bsp.worker

import ch.epfl.scala.bsp4j.{
BuildClient,
BuildServer,
BuildServerCapabilities,
BuildTarget,
BuildTargetCapabilities,
BuildTargetIdentifier,
CleanCacheParams,
CleanCacheResult,
CompileParams,
CompileProvider,
CompileResult,
DebugProvider,
DebugSessionAddress,
DebugSessionParams,
DependencyModule,
DependencyModulesItem,
DependencyModulesParams,
DependencyModulesResult,
DependencySourcesItem,
DependencySourcesParams,
DependencySourcesResult,
InitializeBuildParams,
InitializeBuildResult,
InverseSourcesParams,
InverseSourcesResult,
LogMessageParams,
MessageType,
OutputPathItem,
OutputPathItemKind,
OutputPathsItem,
OutputPathsParams,
OutputPathsResult,
ReadParams,
ResourcesItem,
ResourcesParams,
ResourcesResult,
RunParams,
RunProvider,
RunResult,
SourceItem,
SourceItemKind,
SourcesItem,
SourcesParams,
SourcesResult,
StatusCode,
TaskFinishDataKind,
TaskFinishParams,
TaskId,
TaskStartDataKind,
TaskStartParams,
TestParams,
TestProvider,
TestResult,
TestTask,
WorkspaceBuildTargetsResult
}
import ch.epfl.scala.bsp4j
import ch.epfl.scala.bsp4j._
import com.google.gson.JsonObject
import mill.T
import mill.api.{DummyTestReporter, Result, Strict}
import mill.bsp.BspServerResult
import mill.bsp.worker.Utils.{makeBuildTarget, outputPaths, sanitizeUri}
import mill.define.Segment.Label
import mill.define.{Args, Discover, ExternalModule, Task}
import mill.eval.Evaluator
import mill.eval.Evaluator.TaskResult
import mill.main.MainModule
import mill.scalalib.{JavaModule, SemanticDbJavaModule, TestModule}
import mill.scalalib.bsp.{BspModule, JvmBuildTarget, ScalaBuildTarget}
import mill.runner.MillBuildRootModule
import mill.scalalib.bsp.{BspModule, JvmBuildTarget, ScalaBuildTarget}
import mill.scalalib.{JavaModule, SemanticDbJavaModule, TestModule}

import java.io.PrintStream
import java.util.concurrent.CompletableFuture
import scala.concurrent.Promise
import scala.jdk.CollectionConverters._
import scala.reflect.ClassTag
import scala.util.{Failure, Success, Try}
import Utils.sanitizeUri
import mill.bsp.BspServerResult
import mill.eval.Evaluator.TaskResult

import scala.util.chaining.scalaUtilChainingOps
import scala.util.control.NonFatal
import scala.util.{Failure, Success, Try}
private class MillBuildServer(
topLevelProjectRoot: os.Path,
bspVersion: String,
serverVersion: String,
serverName: String,
Expand Down Expand Up @@ -110,7 +54,7 @@ private class MillBuildServer(
if (statePromise.isCompleted) statePromise = Promise[State]() // replace the promise
evaluatorsOpt.foreach { evaluators =>
statePromise.success(
new State(evaluators, debug)
new State(topLevelProjectRoot, evaluators, debug)
)
}
}
Expand Down Expand Up @@ -209,7 +153,7 @@ private class MillBuildServer(
}

override def workspaceBuildTargets(): CompletableFuture[WorkspaceBuildTargetsResult] =
completableTasks(
completableTasksWithState(
"workspaceBuildTargets",
targetIds = _.bspModulesById.keySet.toSeq,
tasks = { case m: BspModule => m.bspBuildTargetData }
Expand Down Expand Up @@ -246,30 +190,13 @@ private class MillBuildServer(
}

val bt = m.bspBuildTarget
val buildTarget = new BuildTarget(
id,
bt.tags.asJava,
bt.languageIds.asJava,
depsIds.asJava,
new BuildTargetCapabilities().tap { it =>
it.setCanCompile(bt.canCompile)
it.setCanTest(bt.canTest)
it.setCanRun(bt.canRun)
it.setCanDebug(bt.canDebug)
}
)

bt.displayName.foreach(buildTarget.setDisplayName)
bt.baseDirectory.foreach(p => buildTarget.setBaseDirectory(sanitizeUri(p)))

for ((dataKind, data) <- data) {
buildTarget.setDataKind(dataKind)
buildTarget.setData(data)
}

buildTarget
makeBuildTarget(id, depsIds, bt, data)

}(new WorkspaceBuildTargetsResult(_))
} { (targets, state) =>
new WorkspaceBuildTargetsResult(
(targets.asScala ++ state.syntheticRootBspBuildTarget.map(_.target)).asJava
)
}

override def workspaceReload(): CompletableFuture[Object] =
completableNoState("workspaceReload", false) {
Expand Down Expand Up @@ -298,7 +225,7 @@ private class MillBuildServer(
generated
)

completableTasks(
completableTasksWithState(
hint = s"buildTargetSources ${sourcesParams}",
targetIds = _ => sourcesParams.getTargets.asScala.toSeq,
tasks = {
Expand All @@ -316,8 +243,10 @@ private class MillBuildServer(
}
) {
case (ev, state, id, module, items) => new SourcesItem(id, items.asJava)
} {
new SourcesResult(_)
} { (sourceItems, state) =>
new SourcesResult(
(sourceItems.asScala.toSeq ++ state.syntheticRootBspBuildTarget.map(_.synthSources)).asJava
)
}

}
Expand Down Expand Up @@ -451,6 +380,7 @@ private class MillBuildServer(
// already has some from the build file, what to do?
override def buildTargetCompile(p: CompileParams): CompletableFuture[CompileResult] =
completable(s"buildTargetCompile ${p}") { state =>
p.setTargets(state.filterNonSynthetic(p.getTargets))
val params = TaskParameters.fromCompileParams(p)
val taskId = params.hashCode()
val compileTasksEvs = params.getTargets.distinct.map(state.bspModulesById).map {
Expand Down Expand Up @@ -483,29 +413,29 @@ private class MillBuildServer(
override def buildTargetOutputPaths(params: OutputPathsParams)
: CompletableFuture[OutputPathsResult] =
completable(s"buildTargetOutputPaths ${params}") { state =>
val synthOutpaths = for {
synthTarget <- state.syntheticRootBspBuildTarget
if params.getTargets.contains(synthTarget.id)
baseDir <- synthTarget.bt.baseDirectory
} yield new OutputPathsItem(synthTarget.id, outputPaths(baseDir, topLevelProjectRoot).asJava)

val items = for {
target <- params.getTargets.asScala
(module, ev) <- state.bspModulesById.get(target)
} yield {
val items =
if (module.millOuterCtx.external) List(
new OutputPathItem(
// Spec says, a directory must end with a forward slash
sanitizeUri(ev.externalOutPath) + "/",
OutputPathItemKind.DIRECTORY
)
)
else List(
new OutputPathItem(
// Spec says, a directory must end with a forward slash
sanitizeUri(ev.outPath) + "/",
OutputPathItemKind.DIRECTORY
if (module.millOuterCtx.external)
outputPaths(
module.bspBuildTarget.baseDirectory.get,
topLevelProjectRoot
)
)
else
outputPaths(module.bspBuildTarget.baseDirectory.get, topLevelProjectRoot)

new OutputPathsItem(target, items.asJava)
}

new OutputPathsResult(items.asJava)
new OutputPathsResult((items ++ synthOutpaths).asJava)
}

override def buildTargetRun(runParams: RunParams): CompletableFuture[RunResult] =
Expand Down Expand Up @@ -536,6 +466,7 @@ private class MillBuildServer(

override def buildTargetTest(testParams: TestParams): CompletableFuture[TestResult] =
completable(s"buildTargetTest ${testParams}") { state =>
testParams.setTargets(state.filterNonSynthetic(testParams.getTargets))
val millBuildTargetIds = state
.rootModules
.map { case m: BspModule => state.bspIdByModule(m) }
Expand Down Expand Up @@ -627,6 +558,7 @@ private class MillBuildServer(
override def buildTargetCleanCache(cleanCacheParams: CleanCacheParams)
: CompletableFuture[CleanCacheResult] =
completable(s"buildTargetCleanCache ${cleanCacheParams}") { state =>
cleanCacheParams.setTargets(state.filterNonSynthetic(cleanCacheParams.getTargets))
val (msg, cleaned) =
cleanCacheParams.getTargets.asScala.foldLeft((
"",
Expand Down Expand Up @@ -675,6 +607,7 @@ private class MillBuildServer(
override def debugSessionStart(debugParams: DebugSessionParams)
: CompletableFuture[DebugSessionAddress] =
completable(s"debugSessionStart ${debugParams}") { state =>
debugParams.setTargets(state.filterNonSynthetic(debugParams.getTargets))
throw new NotImplementedError("debugSessionStart endpoint is not implemented")
}

Expand All @@ -687,10 +620,24 @@ private class MillBuildServer(
targetIds: State => Seq[BuildTargetIdentifier],
tasks: PartialFunction[BspModule, Task[W]]
)(f: (Evaluator, State, BuildTargetIdentifier, BspModule, W) => T)(agg: java.util.List[T] => V)
: CompletableFuture[V] = {
: CompletableFuture[V] =
completableTasksWithState[T, V, W](hint, targetIds, tasks)(f)((l, _) => agg(l))

/**
* @params tasks A partial function
* @param f The function must accept the same modules as the partial function given by `tasks`.
*/
def completableTasksWithState[T, V, W: ClassTag](
hint: String,
targetIds: State => Seq[BuildTargetIdentifier],
tasks: PartialFunction[BspModule, Task[W]]
)(f: (Evaluator, State, BuildTargetIdentifier, BspModule, W) => T)(agg: (
java.util.List[T],
State
) => V): CompletableFuture[V] = {
val prefix = hint.split(" ").head
completable(hint) { state: State =>
val ids = targetIds(state)
val ids = state.filterNonSynthetic(targetIds(state).asJava).asScala
val tasksSeq = ids.flatMap { id =>
val (m, ev) = state.bspModulesById(id)
tasks.lift.apply(m).map(ts => (ts, (ev, id)))
Expand Down Expand Up @@ -729,7 +676,7 @@ private class MillBuildServer(
}
}

agg(evaluated.flatten.toSeq.asJava)
agg(evaluated.flatten.toSeq.asJava, state)
}
}

Expand Down
10 changes: 9 additions & 1 deletion bsp/worker/src/mill/bsp/worker/State.scala
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import mill.scalalib.internal.JavaModuleUtils
import mill.define.Module
import mill.eval.Evaluator

private class State(evaluators: Seq[Evaluator], debug: String => Unit) {
private class State(workspaceDir: os.Path, evaluators: Seq[Evaluator], debug: String => Unit) {
lazy val bspModulesById: Map[BuildTargetIdentifier, (BspModule, Evaluator)] = {
val modules: Seq[(Module, Seq[Module], Evaluator)] = evaluators
.flatMap(ev => ev.rootModules.map(rm => (rm, JavaModuleUtils.transitiveModules(rm), ev)))
Expand Down Expand Up @@ -34,4 +34,12 @@ private class State(evaluators: Seq[Evaluator], debug: String => Unit) {

lazy val bspIdByModule: Map[BspModule, BuildTargetIdentifier] =
bspModulesById.view.mapValues(_._1).map(_.swap).toMap
lazy val syntheticRootBspBuildTarget: Option[SyntheticRootBspBuildTargetData] =
SyntheticRootBspBuildTargetData.makeIfNeeded(bspModulesById.values.map(_._1), workspaceDir)

def filterNonSynthetic(input: java.util.List[BuildTargetIdentifier])
: java.util.List[BuildTargetIdentifier] = {
import collection.JavaConverters._
input.asScala.filterNot(syntheticRootBspBuildTarget.map(_.id).contains).toList.asJava
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
package mill.bsp.worker

import ch.epfl.scala.bsp4j.{BuildTargetIdentifier, SourceItem, SourceItemKind, SourcesItem}
import mill.bsp.worker.Utils.{makeBuildTarget, sanitizeUri}
import mill.scalalib.bsp.{BspBuildTarget, BspModule}
import mill.scalalib.bsp.BspModule.Tag

import java.util.UUID
import scala.jdk.CollectionConverters._
import ch.epfl.scala.bsp4j.BuildTarget

/**
* Synthesised [[BspBuildTarget]] to handle exclusions.
* Intellij-Bsp doesn't provide a way to exclude files outside of module,so if there is no module having content root of [[topLevelProjectRoot]], [[SyntheticRootBspBuildTargetData]] will be created
*/
class SyntheticRootBspBuildTargetData(topLevelProjectRoot: os.Path) {
val id: BuildTargetIdentifier = new BuildTargetIdentifier(
Utils.sanitizeUri(topLevelProjectRoot / s"synth-build-target-${UUID.randomUUID()}")
)

val bt: BspBuildTarget = BspBuildTarget(
displayName = Some(topLevelProjectRoot.last + "-root"),
baseDirectory = Some(topLevelProjectRoot),
tags = Seq(Tag.Manual),
languageIds = Seq.empty,
canCompile = false,
canTest = false,
canRun = false,
canDebug = false
)

val target: BuildTarget = makeBuildTarget(id, Seq.empty, bt, None)
private val sourcePath = topLevelProjectRoot / "src"
def synthSources = new SourcesItem(
id,
Seq(new SourceItem(sanitizeUri(sourcePath), SourceItemKind.DIRECTORY, false)).asJava
) // intellijBSP does not create contentRootData for module with only outputPaths (this is probably a bug)
}
object SyntheticRootBspBuildTargetData {
def makeIfNeeded(
existingModules: Iterable[BspModule],
workspaceDir: os.Path
): Option[SyntheticRootBspBuildTargetData] = {
def containsWorkspaceDir(path: Option[os.Path]) = path.exists(workspaceDir.startsWith)
if (existingModules.exists { m => containsWorkspaceDir(m.bspBuildTarget.baseDirectory) }) None
else Some(new SyntheticRootBspBuildTargetData(workspaceDir))
}
}
Loading

0 comments on commit 5041301

Please sign in to comment.