Add example Typescript toolchain

docs/modules/ROOT/nav.adoc

@@ -85,6 +85,7 @@
 * xref:extending/running-jvm-code.adoc[]
 * xref:extending/writing-plugins.adoc[]
 * xref:extending/meta-build.adoc[]
+* xref:extending/new-language.adoc[]
 
 // This section focuses on diving into deeper, more advanced topics for Mill.
 // These are things that most Mill developers would not encounter day to day,

docs/modules/ROOT/pages/extending/new-language.adoc

@@ -0,0 +1,37 @@
+= Support for New Languages
+
+This section walks through the process of adding support for a new programming
+language to Mill. We will be adding a small `trait TypeScriptModule` with the
+ability to resolve dependencies, typecheck local code, and optimize a final
+bundle.
+
+The TypeScript integration here is not intended for production usage, but is
+instead meant for illustration purposes of the techniques typically used in
+implementing language toolchains.
+
+== Basic TypeScript Pipeline
+
+include::partial$example/extending/newlang/1-hello-typescript.adoc[]
+
+== Re-usable TypeScriptModule
+
+include::partial$example/extending/newlang/2-typescript-modules.adoc[]
+
+== TypeScriptModule `moduleDeps`
+
+include::partial$example/extending/newlang/3-module-deps.adoc[]
+
+== NPM dependencies and bundling
+
+include::partial$example/extending/newlang/4-npm-deps-bundle.adoc[]
+
+
+
+
+As mentioned earlier, the `TypeScriptModule` examples on this page are meant for
+demo purposes: to show what it looks like to add support in Mill for a new
+programming language toolchain. It would take significantly more work to flesh out
+the featureset and performance of `TypeScriptModule` to be usable in a real world
+build. But this should be enough to get you started working with Mill to add support
+to any language you need: whether it's TypeScript or some other language, most programming
+language toolchains have similar concepts of `compile`, `run`, `bundle`, etc.

example/extending/newlang/1-hello-typescript/build.mill

@@ -0,0 +1,139 @@
+// This example demonstrates basic integration of https://www.typescriptlang.org/[Typescript]
+// compilation into a Mill build to compile https://nodejs.org/en[Node.js] apps. Mill
+// does not come bundled with a Typescript integration, so here we begin setting
+// one up from first principles using the https://www.npmjs.com/[NPM] command line
+// tool and package repository
+
+// === Installing TypeScript
+//
+// First, we need to use the `npm` CLI tool to install typescript and the `@types/node`
+// library necessary for accessing Node.js APIs:
+
+
+package build
+import mill._
+
+def npmInstall = Task{
+  os.call(("npm", "install", "--save-dev", "[email protected]", "@types/[email protected]"))
+  PathRef(Task.dest)
+}
+
+// The `npmInstall` task runs `npm install` to install TypeScript locally, following
+// the https://www.typescriptlang.org/download/[TypeScript installation instructions].
+// The `os.call` above by default runs inside the `npmInstall` task's unique `Task.dest`
+// output directory due to xref:depth/sandboxing.adoc[task sandboxing]. Note that we
+// use an explicit version on each of the modules to ensure the `Task` is reproducible.
+// We then return a `PathRef` to the `Task.dest` so downstream tasks can make use of it.
+//
+// Note that as written, the `npmInstall` task will never invalidate unless you change its
+// code. This is what we should expect, since `npmInstall` has no upstream tasks it depends
+// on and the versions of `typescript` and `@types/node` are fully specified in the task.
+// This assumes that the `npm` package repository always returns the same artifacts for
+// the given name and version, which is a reasonable assumption for most package repositories.
+
+// === Defining our sources
+
+// Next, we define the `sources` of our Typescript build using a
+// xref:fundamentals/tasks.adoc#_sources[source task]. Here `sources` refers to the
+// source folder, and the subsequent `allSources` walks that folder and picks up all
+// the individual typescript files within. This is a common pattern to give flexibility,
+// e.g. someone can later easily override `allSources` to add additional filtering
+// on exactly which files within the source root they wish to pick up.
+
+def sources = Task.Source(millSourcePath / "src")
+def allSources = Task{
+  os.walk(sources().path).filter(_.ext == "ts").map(PathRef(_))
+}
+
+// === Compilation
+//
+// Next, we define our `compile` task. This is again a relatively straightforward subprocess
+// call invoking the `typescript/bin/tsc` executable within the `node_modules` folder from the
+// upstream `npmInstall` task, passing it the sources, `--outDir`, `--types`, and `--typeRoots`
+// Again we return a `PathRef` to the `Task.dest` folder we output the compiled JS files to
+
+def compile = Task{
+  os.call(
+    (
+      npmInstall().path / "node_modules/typescript/bin/tsc",
+      allSources().map(_.path),
+      "--outDir", Task.dest,
+      "--typeRoots", npmInstall().path / "node_modules/@types"
+    )
+  )
+  PathRef(Task.dest)
+}
+// At this point, we have a minimal working build, with a build graph that looks like this:
+//
+// ```graphviz
+// digraph G {
+//   rankdir=LR
+//   node [shape=box width=0 height=0 style=filled fillcolor=white]
+//   npmInstall -> compile
+//   sources -> allSources -> compile
+// }
+// ```
+//
+// Given an input file below, we can run
+// `mill compile` and demonstrate it is installing typescript locally and using it to compile
+// the `.ts` files in out `src/` folder:
+
+/** See Also: src/hello.ts */
+
+/** Usage
+
+> mill compile
+error: .../src/hello.ts(...): error ... Property 'name' does not exist on type...
+
+> sed -i.bak 's/user.name/user.firstName/g' src/hello.ts
+
+> mill compile
+
+> cat out/compile.dest/hello.js # Output is stripped of types and converted to javascript
+var user = {
+    firstName: process.argv[2],
+    lastName: process.argv[3],
+    role: "Professor",
+};
+console.log("Hello " + user.firstName + " " + user.lastName);
+
+*/
+
+// === Running
+
+// The last step here is to allow the ability to run our compiled JavaScript file.
+// To do this, we need a `mainFileName` task to tell Mill which file should be used
+// as the program entrypoint, and a `run` command taking arguments that get used to
+// call `node` along with the main Javascript file:
+
+def mainFileName = Task{ "hello.js" }
+def run(args: mill.define.Args) = Task.Command{
+  os.call(
+    ("node", compile().path / mainFileName(), args.value),
+    stdout = os.Inherit
+  )
+}
+// Note that we use `stdout = os.Inherit` since we want to display any output to
+// the user, rather than capturing it for use in our command.
+//
+// ```graphviz
+// digraph G {
+//   rankdir=LR
+//   node [shape=box width=0 height=0 style=filled fillcolor=white]
+//   npmInstall -> compile
+//   sources -> allSources -> compile -> run
+//   mainFileName -> run
+//   mainFileName [color=green, penwidth=3]
+//   run [color=green, penwidth=3]
+// }
+// ```
+
+/** Usage
+> mill run James Bond
+Hello James Bond
+
+*/
+
+// So that's a minimal example of implementing a single TypeScript to JavaScript build
+// pipeline locally. Next, we will look at turning it into a `TypeScriptModule` that
+// can be re-used

example/extending/newlang/1-hello-typescript/src/hello.ts

@@ -0,0 +1,13 @@
+interface User {
+    firstName: string
+    lastName: string
+    role: string
+}
+
+const user: User = {
+    firstName: process.argv[2],
+    lastName: process.argv[3],
+    role: "Professor",
+}
+
+console.log("Hello " + user.name + " " + user.lastName)

example/extending/newlang/2-typescript-modules/build.mill

@@ -0,0 +1,93 @@
+// In this example, we will explore how to take the one-off typescript build pipeline
+// we wrote above, and turn it into a re-usable `TypeScriptModule`.
+//
+// To do this, we take all the code we wrote earlier and surround it with
+// `trait TypeScriptModule extends Module` wrapper:
+
+package build
+import mill._
+
+trait TypeScriptModule extends Module{
+  def npmInstall = Task{
+    os.call(("npm", "install", "--save-dev", "[email protected]", "@types/[email protected]"))
+    PathRef(Task.dest)
+  }
+
+  def sources = Task.Source(millSourcePath / "src")
+  def allSources = Task{
+    os.walk(sources().path).filter(_.ext == "ts").map(PathRef(_))
+  }
+
+  def compile = Task{
+    val tsc = npmInstall().path / "node_modules/typescript/bin/tsc"
+    val types = npmInstall().path / "node_modules/@types"
+    os.call((tsc, allSources().map(_.path), "--outDir", Task.dest, "--typeRoots", types))
+    PathRef(Task.dest)
+  }
+
+  def mainFileName = Task{ s"${millSourcePath.last}.js" }
+  def run(args: mill.define.Args) = Task.Command{
+    val mainFile = compile().path / mainFileName()
+    os.call(("node", mainFile, args.value), stdout = os.Inherit)
+  }
+}
+
+// We can then instantiate the module three times. Module can be adjacent or nested,
+// as shown belo:
+
+object foo extends TypeScriptModule{
+  object bar extends TypeScriptModule
+}
+object qux extends TypeScriptModule
+
+/** See Also: foo/src/foo.ts */
+/** See Also: foo/bar/src/bar.ts */
+/** See Also: qux/src/qux.ts */
+
+// And then invoke the `.run` method on each module from the command line:
+
+/** Usage
+> mill foo.run James
+Hello James Foo
+
+> mill foo.bar.run James
+Hello James Bar
+
+> mill qux.run James
+Hello James Qux
+
+*/
+
+// At this point, we have multiple ``TypeScriptModule``s, with `bar` nested inside `foo`,
+// but they are each independent and do not depend on one another.
+
+// ```graphviz
+// digraph G {
+//   rankdir=LR
+//   node [shape=box width=0 height=0 style=filled fillcolor=white]
+//   subgraph cluster_3 {
+//     style=dashed
+//     label=qux
+//     "qux.npmInstall" -> "qux.compile"
+//     "qux.sources" -> "qux.allSources" -> "qux.compile" -> "qux.run"
+//     "qux.mainFileName" -> "qux.run"
+//   }
+//   subgraph cluster_1 {
+//     subgraph cluster_2 {
+//       style=dashed
+//       label=bar
+//       "bar.npmInstall" -> "bar.compile"
+//       "bar.sources" -> "bar.allSources" -> "bar.compile" -> "bar.run"
+//       "bar.mainFileName" -> "bar.run"
+//     }
+//     style=dashed
+//     label=foo
+//     "foo.npmInstall" -> "foo.compile"
+//     "foo.sources" -> "foo.allSources" -> "foo.compile" -> "foo.run"
+//     "foo.mainFileName" -> "foo.run"
+//   }
+// }
+// ```
+//
+// Next, we will look at how to wire them up using
+// `moduleDeps`.

example/extending/newlang/2-typescript-modules/foo/bar/src/bar.ts

@@ -0,0 +1,5 @@
+interface User { firstName: string }
+
+const user: User = { firstName: process.argv[2] }
+
+console.log("Hello " + user.firstName + " Bar")

example/extending/newlang/2-typescript-modules/foo/src/foo.ts

@@ -0,0 +1,5 @@
+interface User { firstName: string }
+
+const user: User = { firstName: process.argv[2] }
+
+console.log("Hello " + user.firstName + " Foo")

example/extending/newlang/2-typescript-modules/qux/src/qux.ts

@@ -0,0 +1,5 @@
+interface User { firstName: string }
+
+const user: User = { firstName: process.argv[2] }
+
+console.log("Hello " + user.firstName + " Qux")