Skip to content

Commit

Permalink
Implement publishing to Bintray (#77)
Browse files Browse the repository at this point in the history
This introduces two user-facing features: the ability to publish and
document modules. The changes resulted in the stabilisation of the
artefact resolution and progress bar logic. Furthermore, the
`import` keyword is now available in the global Seed configuration
file.

With the new CLI command `publish`, modules can be published to a
repository. It builds, packages and uploads the specified modules.
At the moment, only Maven-style Bintray repositories are supported.
For each supplied module, JAR artefacts along with a POM file are
created. These artefacts contain the compiled classes, source files
and documentation. Publishing of sources and documentation is
optional and can be disabled. Progress bars are shown when
publishing modules. Unless specified, Seed will attempt to read the
version from the project's Git repository. Internally, the feature
uses the Apache HTTP client, wrapped as a ZIO task, and sends
requests to the [Bintray REST API](https://bintray.com/docs/api/).

The newly-introduced CLI command `doc` runs Scaladoc on the supplied
modules and generates an HTML documentation for them. Seed runs the
Scala compiler directly, bypassing Bloop. This results in an
additional compilation pass. Compiler bridges are provided for Scala
2.11, 2.12 and 2.13. However, the actual resolution of the
`scala-compiler` artefact occurs during runtime. Therefore,
documentation generation is compatible with alternative compilers
such as Typelevel Scala.

The aforementioned features required changes to the artefact
resolution. Previously, all dependencies were resolved at once.
However, modular projects are likely to contain libraries with
diverging versions. A resolution pass in Coursier merges these
duplicate libraries, only retaining the latest version. This may
lead to unexpected compile- and runtime behaviour. The new
resolution logic solves this limitation by performing a separate
resolution pass for each module. Test modules are resolved
separately too. As a consequence of these changes, the resolution is
slightly slower, but still acceptable on larger projects.

A further change is that the standard library's organisation and
version are forcibly set during dependency resolution by using
Coursier's `ResolutionParams` feature. This avoids having to patch
artefacts later on, e.g. during generation.

The progress bar logic was refactored and several bugs were fixed.
Notably, as build targets may run asynchronously, they can produce
output while the project is still compiling. In this case, the
progress bar output would get corrupted. The graphical glitch was
resolved by using the console output's logger in
`seed.cli.BuildTarget.buildTargets()`.

Finally, the `import` keyword, previously only available in build
files, can now be used within the global Seed configuration. Since
Bintray repository credentials are part of this file, users may want
to move the credentials to a custom file and import it instead.
tindzk authored Feb 23, 2020
1 parent 5512473 commit ee2e274
Showing 45 changed files with 3,139 additions and 684 deletions.
2 changes: 1 addition & 1 deletion .drone.yml
Original file line number Diff line number Diff line change
@@ -18,7 +18,7 @@ steps:
- export PATH=$(pwd)/bloop:$PATH
- blp-server &
- ./build.sh
- COURSIER_SBT_LAUNCHER_ADD_PLUGIN=true ./csbt test
- COURSIER_SBT_LAUNCHER_ADD_PLUGIN=true ./csbt "; scaladoc211/publishLocal; scaladoc212/publishLocal; scaladoc213/publishLocal; test"
- name: publish_prerelease
image: plugins/docker
environment:
122 changes: 117 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -150,7 +150,10 @@ This compiles the module to `build/` and runs it.
* Unicode characters
* Progress bars
* Project creation wizard
* Packaging support
* Generate module documentation
* Publish artefacts to Bintray
* Read version from Git repository
* Package modules
* Copy over dependencies
* Server mode
* Expose a WebSocket server
@@ -528,7 +531,7 @@ For two equivalent examples of using code generation, please refer to these link
* [Command target](test/custom-command-target/)

### Compiler plug-ins
Project and modules can add Scala plug-ins with `compilerDeps`. The setting behaves like `scalaDeps`, but also adds the `-Xplugin` parameter to the Scala compiler when modules are compiled. For example:
Scala plug-ins can be included with the `compilerDeps` setting. It behaves similar to `scalaDeps`, but adds the `-Xplugin` parameter to the Scala compiler. `compilerDeps` is available on the project as well as module level:

```toml
[project]
@@ -542,8 +545,9 @@ compilerDeps = [
]
```

Note that project-level plug-ins are inherited by all modules defined under the project, as well as any dependent modules such that `compilerDeps` only needs to be defined on the base project or module.
In the example above, `module.macros.js` inherits the semanticdb plug-in from the *project* and adds a separate dependency on the macro paradise plug-in.
Note that project-level plug-ins are inherited by all modules defined in the project file. When module `a` depends on module `b` which defines compiler plug-ins, these are inherited by `a`. Thus, in order to avoid duplication, shared compiler dependencies can be defined on base projects and modules.

In the example above, `module.macros.js` inherits the SemanticDB plug-in from the project and adds a separate dependency for the Macro Paradise plug-in.

For a complete cross-compiled Macro Paradise example, please refer to [this project](test/example-paradise/).

@@ -613,6 +617,52 @@ ivy = [
]
```

### Package
When publishing artefacts, a [Maven POM](http://maven.apache.org/pom.html) file is created. To configure its contents, define a `package` section:

```toml
[package]
# Project name (`name` field)
# name = "example"
name = ""

# Package organisation (`groupId` field)
# organisation = "com.smith"
organisation = ""

# List of developers (ID, name, e-mail)
# developers = [{ id = "joesmith", name = "Joe Smith", email = "joe@smith.com" }]
# developers = [["joesmith", "Joe Smith", "joe@smith.com"]]
developers = []

# Project URL
url = ""

# List of project licences
# Available values:
# gpl:2.0, gpl:3.0, lgpl:2.1, lgpl:3.0, cddl:1.0, cddl+gpl, apache:2.0, bsd:2,
# bsd:3, mit, epl:1.0, ecl:1.0, mpl:2.0
# licences = ["apache:2.0"]
licences = []

# Source code information
# `developerConnection` is optional and if omitted, has the same value as `connection`
#scm = {
# url = "https://github.com/joesmith/example",
# connection = "scm:git:git@github.com:joesmith/joesmith.git",
# developerConnection = "scm:git:git@github.com:joesmith/joesmith.git"
#}
scm = { url = "", connection = "", developerConnection = "" }

# Publish sources
# Alternatively, use --skip-sources to override this setting
sources = true

# Generate and publish documentation
# Alternatively, use --skip-docs to override this setting
docs = true
```

### Sample configurations
You can take some inspiration from the following projects:

@@ -683,6 +733,20 @@ progress = true

The default values are indicated.

### Publishing settings
```toml
# Bintray repository credentials
[repository.bintray]
user = ""
apiKey = ""
```

If you maintain your configuration files in a public Git repository, it is advisable to move the Bintray section to a separate untracked file (e.g. `seed-credentials.toml`) and import it from the main configuration:

```toml
import = ["seed-credentials.toml"]
```

## Git
### .gitignore
For a Seed project, `.gitignore` only needs to contain these four directories:
@@ -825,6 +889,15 @@ There are two related issues tracking the testing problems: [issue SCL-8972](htt

As a workaround, you can open a terminal within IntelliJ and use Bloop, for example: `bloop test <module>-js`

## Generate documentation
Seed can generate an HTML documentation for your modules with the `doc` command:

```shell
seed doc example:jvm example:js
```

The functionality is based on Scaladoc. For each specified module, it uses the corresponding Scala compiler. Scaladoc bypasses Bloop and performs a separate non-incremental compilation pass.

## Packaging
In order to distribute your project, you may want to package the compiled sources. The approach chosen in Seed is to bundle them as a JAR file.

@@ -872,6 +945,34 @@ If you would like to use pre-release versions, you can also pass in this paramet
seed update --pre-releases
```

## Publishing
Seed can publish modules to Maven-style Bintray repositories. First, populate the build file's `package` section. Then run the following command:

```shell
seed publish --version=1.0 bintray:joesmith/maven/releases demo:js demo:jvm
```

The credentials must be configured in the global Seed configuration:

```toml
[repository.bintray]
user = "joesmith"
apiKey = "<API key>"
```

Alternatively, Seed reads the environment variables `BINTRAY_USER` and `BINTRAY_API_KEY`.

Seed publishes sources and generates the documentation, which slows down the publishing process. If any of these artefacts are unneeded, you can permanently change the behaviour via `package.sources` and `package.docs` in the build file, or temporarily by passing `--skip-sources` and `--skip-docs` to the CLI.

Finally, to depend on the published library in another project, add the Bintray resolver there:

```toml
[resolvers]
maven = ["https://repo1.maven.org/maven2", "https://dl.bintray.com/joesmith/maven"]
```

The `--version` parameter is optional. By default, Seed reads the version number from the Git repository via `git describe --tags`. This allows publishing artefacts for every single commit as part of CI executions.

## Performance
On average, Bloop project generation and compilation are roughly 3x faster in Seed compared to sbt in non-interactive mode. Seed's startup is 10x faster than sbt's.

@@ -911,7 +1012,7 @@ Seed does not offer any capability for writing plug-ins. If you would like to re
If some settings of the build are dynamic, you could write a script to generate TOML files from a template. A use case would be to cross-compile your modules for different Scala versions. Cross-compilation between Scala versions may require code changes. It is thinkable to have the `build.toml` point to the latest supported Scala version and have scripts that downgrade the sources, e.g. using a tool like [scalafix](https://scalacenter.github.io/scalafix/).

### Publishing
Publishing libraries is not possible yet, but Bintray/Sonatype support is planned for future versions.
At the moment, artefacts can only be published to Bintray. Sonatype support is planned for future versions.

## Design Goals
The three overarching design goals were usability, simplicity and speed. The objective for Seed is to offer a narrow, but well-designed feature set that covers most common use cases for library and application development. This means leaving out features such as plug-ins, shell, tasks and code execution that would have a high footprint on the design.
@@ -932,6 +1033,17 @@ The output could use colours and bold/italic/underlined text. Also, components s
[error] Trace: module → targets
```

## Development
To run all test cases, the Scaladoc bridges must be published locally after each commit:

```shell
$ sbt
scaladoc211/publishLocal
scaladoc212/publishLocal
scaladoc213/publishLocal
test
```

## Credits
Seed achieves its simplicity by delegating much of the heavy lifting to external tools and libraries, notably Bloop and Coursier. The decision of using TOML for configuration and the build schema are influenced by [Cargo](https://github.com/rust-lang/cargo).

75 changes: 67 additions & 8 deletions build.sbt
Original file line number Diff line number Diff line change
@@ -12,13 +12,17 @@ def parseVersion(file: Path): Option[String] =
.find(_.nonEmpty)
.map(_.trim)

def seedVersion =
val seedOrganisation = "tindzk"

// Should be `val` instead of `def`, otherwise it is necessary to run
// scaladoc*/publishLocal after every commit
val seedVersion =
parseVersion(Paths.get("SEED")) // CI
.getOrElse(Seq("git", "describe", "--tags").!!.trim) // Local development
def bloopVersion = parseVersion(Paths.get("BLOOP")).get
def bloopCoursierVersion = parseVersion(Paths.get("COURSIER")).get
val bloopVersion = parseVersion(Paths.get("BLOOP")).get
val bloopCoursierVersion = parseVersion(Paths.get("COURSIER")).get

organization := "tindzk"
organization := seedOrganisation
version := seedVersion
scalaVersion := "2.12.4-bin-typelevel-4"
scalaOrganization := "org.typelevel"
@@ -31,9 +35,10 @@ Compile / sourceGenerators += Def.task {
s"""package seed
|
|object BuildInfo {
| val Version = "$seedVersion"
| val Bloop = "$bloopVersion"
| val Coursier = "$bloopCoursierVersion"
| val Organisation = "$seedOrganisation"
| val Version = "$seedVersion"
| val Bloop = "$bloopVersion"
| val Coursier = "$bloopCoursierVersion"
|}""".stripMargin
)

@@ -51,6 +56,7 @@ libraryDependencies ++= Seq(
"ch.epfl.scala" % "directory-watcher" % "0.8.0+6-f651bd93",
"com.joefkelley" %% "argyle" % "1.0.0",
"org.scalaj" %% "scalaj-http" % "2.4.2",
"org.apache.httpcomponents" % "httpasyncclient" % "4.1.4",
"dev.zio" %% "zio" % "1.0.0-RC14",
"dev.zio" %% "zio-streams" % "1.0.0-RC14",
"io.circe" %% "circe-core" % "0.11.1",
@@ -71,7 +77,60 @@ Global / cancelable := true
testFrameworks += new TestFramework("minitest.runner.Framework")

licenses += ("Apache-2.0", url("http://opensource.org/licenses/Apache-2.0"))
bintrayVcsUrl := Some("git@github.com:tindzk/seed.git")
bintrayVcsUrl := Some(s"git@github.com:$seedOrganisation/seed.git")

publishArtifact in (Compile, packageDoc) := false
publishArtifact in (Compile, packageSrc) := false

lazy val scaladoc211 = project
.in(file("scaladoc211"))
.settings(
organization := seedOrganisation,
version := seedVersion,
name := "seed-scaladoc",
scalaVersion := "2.11.12",
libraryDependencies += "org.scala-lang" % "scala-compiler" % "2.11.12" % Provided,
// Publish artefact without the standard library. The correct version of
// scala-library will be resolved during runtime
pomPostProcess := dropScalaLibraries,
Compile / unmanagedSourceDirectories += baseDirectory.value / ".." / "scaladoc" / "src" / "main" / "scala"
)

lazy val scaladoc212 = project
.in(file("scaladoc212"))
.settings(
organization := seedOrganisation,
version := seedVersion,
name := "seed-scaladoc",
scalaVersion := "2.12.10",
libraryDependencies += "org.scala-lang" % "scala-compiler" % "2.12.10" % Provided,
pomPostProcess := dropScalaLibraries,
Compile / unmanagedSourceDirectories += baseDirectory.value / ".." / "scaladoc" / "src" / "main" / "scala"
)

lazy val scaladoc213 = project
.in(file("scaladoc213"))
.settings(
organization := seedOrganisation,
version := seedVersion,
name := "seed-scaladoc",
scalaVersion := "2.13.1",
libraryDependencies += "org.scala-lang" % "scala-compiler" % "2.13.1" % Provided,
pomPostProcess := dropScalaLibraries,
Compile / unmanagedSourceDirectories += baseDirectory.value / ".." / "scaladoc" / "src" / "main" / "scala"
)

// From https://stackoverflow.com/questions/27835740/sbt-exclude-certain-dependency-only-during-publish
import scala.xml.{Node => XmlNode, NodeSeq => XmlNodeSeq, _}
import scala.xml.transform.{RewriteRule, RuleTransformer}
def dropScalaLibraries(node: XmlNode): XmlNode =
new RuleTransformer(new RewriteRule {
override def transform(node: XmlNode): XmlNodeSeq = node match {
case e: Elem
if e.label == "dependency" && e.child.exists(
child => child.label == "groupId" && child.text == "org.scala-lang"
) =>
XmlNodeSeq.Empty
case _ => node
}
}).transform(node).head
3 changes: 3 additions & 0 deletions release.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
#!/bin/sh
set -e
set -x

version=$1

/usr/bin/git tag -f $version
Loading

0 comments on commit ee2e274

Please sign in to comment.