Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Overhaul build file management with new
build.mill/package.mill f…
…ormat (#3426) This PR overhauls how build files are handled, especially in subfolders. The goal is to come up with an approach that is (1) scalable to large builds and (2) works intuitively with how people expect things to work and (3) plays well with IDEs (4) converges with the Scala language to reduce the special casing we have to do. There's a bunch of hackiness in the implementation, but the end result is that when I load the updated `10-multi-file-build` project into IntelliJ, navigation around the multi-file project works seamlessly (after adding the `.mill` file association): <img width="976" alt="Screenshot 2024-08-31 at 4 35 16 PM" src="https://github.com/user-attachments/assets/b387a99a-eeb7-44f0-9a6e-6e6e864fb27f"> VSCode works as well, though it only understands `.scala` extensions and will likely need to be patched to work with `.mill` <img width="976" alt="Screenshot 2024-08-31 at 4 49 46 PM" src="https://github.com/user-attachments/assets/c26c5eef-58d7-4b4d-8d51-c45b2f2cf4c2"> This means that to begin with, all the IDE tooling around Mill just works, with the IDE being blissfully unaware of the nasty code transformations and other things Mill does. From that base, we can slowly look into removing boilerplate in an IDE/language-compliant way, and removing the backend hackiness while keeping the user experience mostly unchanged Major user-facing changes: 1. `build.sc` is now `build.mill`, sub-folders can define modules in `package.mill` files. Helper libraries live in files like `foo/bar.mill` * Long term Mill needs our own extension because `.sc` is overloaded to work with Ammonite/Scala-CLI scripts. That means IDEs like IntelliJ treat them specially, in ways that are incompatible with Mill (e.g. IDEs don't understand importing between script files). `.sc` is still supported for migration/compatibility reasons. Editors don't currently associate `.mill` with Mill Scala code, but it's a few clicks for the end user and a trivial PR to send to IntelliJ and Metals to support it * `.scala` as an extension could possibly work from a technical level, but there will still be user-facing confusion if we want Mill to be able to target non-Scala developers, and possibly technical confusion for tools that can't differentiate between Mill build files and application source files 2. All `.mill` files need a `package` declaration at the top, e.g. `build.mill` needs `package build`, `foo/bar/qux.sc` needs `package build.foo.bar`. I left it optional for the root `build.mill` for migration purposes * This is necessary for full IDE support, as editors do not understand the "things in subfolders are automatically in packages" thing that Mill and Ammonite did in the past. We had hacks around `import $file`, but they never were super robust, and `import $file` is itself a hack we've discussed getting rid of * We can look into making package declarations optional again in future, but for now this gives us working IDE support with minimal effort, converging with vanilla Scala, at the cost of a little boilerplate per-file. 3. No more `import $file`: any files with `.mill` (or `.sc`) extensions adjacent to a `build.mill` or `package.mill` are treated as Mill files * This simplifies file discovery considerably, v.s. the previous approach that required multiple-passes of parsing and traversing all imported scripts as a graph. It also aligns things more with how Scala and other JVM projects work, where files are typically picked up by folder regardless of imports Major internal changes: 1. We move the handling of `RootModule` from runtime reflection/classpath-scanning/custom-resolve-logic to code-generation logic that "unpacks" any `object module`, `object build`, or `object foo extends RootModule` into the enclosing wrapper class * We can do this as a compiler plugin, which would help preserve line numbers and avoid showing generated code, but it also makes things terribly hard to debug when things go wrong * This allows us to ensure `RootModule`s are handled uniformly from Scala code and from the command line, where previously Scala code would need to write `bar.qux.module.mytask` suffix while the CLI would just write `bar.qux.mytask` * By making the unpacking name-based, rather than type-based, this removes the "what should we name the root module?" degree of freedom that really shouldn't be necessary and ensures everyone names them consistently 2. Replace the `package object`s previously generated for root modules with normal `object`s named `build` or `module` (reflecting the file names) * `package object`s are on their way out in Scala 3 https://docs.scala-lang.org/scala3/reference/dropped-features/package-objects.html. No concrete timeline, but good to stop using them. They also are a common source of bugs around compilation, incremental compilation, etc. since even their present-day implementation isn't terribly robust * This also removes the need to refer to `build` as `build.package` since it is now a normal object 3. Generate `final def` aliases in the generated `object`s in order to make code references and module discovery work * These aliases allow both Scala-code references to modules without needing this `.build` or `.module` suffix, as well as allow the `Resolve.scala` logic to work * We can thus eliminate a whole bunch of gnarly code plumbing multiple-root-modules all over the Mill codebase and gnarly classpath-scanning code to try and identify the sub-folder root modules and wire them up. * These aliases should more reliably generate conflicts than overlapping `package object`/`object`s, which seemed pretty flaky Limitations: * Partially migrating subfolders out of a parent `build.sc` into a child `module.sc` no longer works. The aliases we generate for subfolder `module.sc` files always conflicts at compile time with any locally-defined modules * References to helper files e.g. `foo/bar.sc` with `def qux` would get referenced via `build_.foo.bar.qux` is kind of awkward. I expect this to go away once we get Scala 3 support, and we can use `export` clauses to allow referencing them via `build.foo.qux`, which is more in line with how Scala code normally works where different files in the same folder are all part of the same combined namespace. * To work around `package object` weirdness e.g. needing to reference things via `foo.bar.package`, we build our own parallel object hierarchy via codegen and provide that to the user. This is awkward, but it looks similar enough people shouldn't notice it, and we can look into somehow fixing the `package object` issues upstream in scala/scala3 in future * The `package build` prefix is kind of verbose, but it's necessary so we have some way of reliably referencing other files by their fully qualified names. e.g. a file in `util/package.mill` can be referenced by `build.util.*`, but cannot be referenced by `util.*` because `import mill._` pulls in `mill.util` which shadows it.
- Loading branch information
