multi-module dependency resolving broken (duplicates)

[francisdb]

I seem to be getting 2 versions of the same dependency in the classpath, should mill not be taking care of these?

mill show foo.compileClasspath
...
"path": "/Users/francisdb/.coursier/cache/v1/https/repo1.maven.org/maven2/com/typesafe/akka/akka-actor_2.12/2.5.3/akka-actor_2.12-2.5.3.jar"
...
"path": "/Users/francisdb/.coursier/cache/v1/https/repo1.maven.org/maven2/com/typesafe/akka/akka-actor_2.12/2.5.11/akka-actor_2.12-2.5.11.jar"

simple test case available here:
https://github.com/francisdb/milltest

[lihaoyi]

maybe; what's your build?

[francisdb]

My build is a big 20+ module project, where some modules bring in akka through a transitive dependency from it's own dependency

Anything more specific you want me to mention?
Can you also clarify maybe, in what case should mill allow this?

[lihaoyi]

can you poke around the build using describe/inspect and show to see where the various versions come from? Using describe tells you what the parents of a task are and show tells you what the value is. It should be pretty quick to figure out where the various copies of Akka are coming from

[francisdb]

I made it even worse by adding more dependencies to other versions of akka

On the modules level
logging depends on core

On the ivy level
core has a direct dependency on akka-actor_2.12-2.5.4
core has a transitive dependency on akka-actor_2.12-2.5.3
logging has a dependecny on akka-actor_2.12-2.5.11

but when I do the following

logging.compileClasspath().foreach(println)
...
PathRef(/Users/francisdb/.coursier/cache/v1/https/repo1.maven.org/maven2/com/typesafe/akka/akka-actor_2.12/2.5.3/akka-actor_2.12-2.5.3.jar,true,426226507)
...
PathRef(/Users/francisdb/.coursier/cache/v1/https/repo1.maven.org/maven2/com/typesafe/akka/akka-actor_2.12/2.5.4/akka-actor_2.12-2.5.4.jar,true,-2030674072)
...
PathRef(/Users/francisdb/.coursier/cache/v1/https/repo1.maven.org/maven2/com/typesafe/akka/akka-actor_2.12/2.5.11/akka-actor_2.12-2.5.11.jar,true,2056048751)
...

Using sbt akka-actor_2.12-2.5.3 and akka-actor_2.12-2.5.4 are evicted in favor of akka-actor_2.12-2.5.11

[lihaoyi]

I don't really know any reason mill should allow it, but there might be bugs, hence maybe.

So you have a big 20+ module project; just delete everything you can while reproducing this bug. You shouldn't even need any source files present, just the build.sc, and it should be pretty quick to bisect.

Once the project is minimized, let me know and maybe the problem will be clearer. Perhaps you may even spot the issue yourself

[francisdb]

I created a very simple reproducer (just the build file needed) here: https://github.com/francisdb/milltest

Direct dependencies are deduplicated, module-transitive ones are not

[francisdb]

the main issue seems to be here
https://github.com/lihaoyi/mill/blob/cb30f1c42c818b2a2b1e216823ca09c8430da5f9/scalalib/src/mill/scalalib/ScalaModule.scala#L175

You can't just use the upstreamRunClasspath, it needs to be re-resolved together with the current module dependencies to make sure there are no duplicates.

[lihaoyi]

@francisdb nice find, yeah we probably should restrict upstreamRunClasspath to compile resources and unmanagedDependencies only, since ivyDeps and friends will all get aggregated separately and resolved using coursier. Can you send a PR?

[francisdb]

I fear I don't feel enough at home with all these scopes and dependencies to be able to fix this.

[francisdb]

@lihaoyi I can confirm this to be fixed!