Add build instructions for developers

.gitignore

@@ -82,8 +82,6 @@ deploy/dockerephemeral/build/smtp/
 /libs/libzauth/bzauth-c/deb/usr
 
 # Generated by "make hie.yaml"
-hie.yaml
-hie.orig.yaml
 stack-dev.yaml
 
 # HIE db files (e.g. generated for stan)

Makefile

@@ -53,9 +53,14 @@ endif
 .PHONY: delete-cache-on-linker-errors
 delete-cache-on-linker-errors:
 	rm -rf ~/.cache/hie-bios
-	rm -rf ~/.cabal
+	rm -rf ~/.cabal/store
 	rm -rf ./dist-newstyle
 
+.PHONY: cabal.project.local
+cabal.project.local:
+	echo "optimization: False" > ./cabal.project.local
+	./hack/bin/cabal-project-local-template.sh "ghc-options: -O0" >> ./cabal.project.local
+
 # Build all Haskell services and executables with -O0, run unit tests
 .PHONY: fast
 fast: init

README.md

@@ -18,22 +18,6 @@ For documentation on how to self host your own Wire-Server see [this section](#h
 
 See more in "[Open sourcing Wire server code](https://medium.com/@wireapp/open-sourcing-wire-server-code-ef7866a731d5)".
 
-## Table of contents
-
-<!-- vim-markdown-toc GFM -->
-
-* [Contents of this repository](#contents-of-this-repository)
-* [Architecture Overview](#architecture-overview)
-* [Development setup](#development-setup)
-    * [How to build `wire-server` binaries](#how-to-build-wire-server-binaries)
-        * [1. Compile sources natively.](#1-compile-sources-natively)
-        * [2. Use docker](#2-use-docker)
-    * [How to run integration tests](#how-to-run-integration-tests)
-    * [when you need more fine-grained control over your build-test loops](#when-you-need-more-fine-grained-control-over-your-build-test-loops)
-* [How to install and run `wire-server`](#how-to-install-and-run-wire-server)
-
-<!-- vim-markdown-toc -->
-
 ## Contents of this repository
 
 This repository contains the following source code:
@@ -82,29 +66,7 @@ private network.
 
 There are two options:
 
-#### 1. Compile sources natively.
-
-This requires a range of dependencies that depend on your platform/OS, such as:
-
-- Haskell & Rust compiler and package managers
-- Some package dependencies (libsodium, openssl, protobuf, icu, geoip, snappy, [cryptobox-c](https://github.com/wireapp/cryptobox-c), ...) that depend on your platform/OS
-
-See [docs/developer/dependencies.md](docs/legacy/developer/dependencies.md) for details.
-
-Once all dependencies are set up, the following should succeed:
-
-```bash
-# build all haskell services
-make
-# build one haskell service, e.g. brig:
-cd services/brig && make
-```
-
-The default make target (`fast`) compiles unoptimized (faster compilation time, slower binaries), which should be fine for development purposes. Use `make install` to get optimized binaries.
-
-For building nginz, see [services/nginz/README.md](services/nginz/README.md)
-
-#### 2. Use docker
+#### 1. Use docker
 
 *If you don't wish to build all docker images from scratch (e.g. the `ubuntu20-builder` takes a very long time), ready-built images can be downloaded from [here](https://quay.io/organization/wire).*
 
@@ -123,54 +85,9 @@ will, eventually, have built a range of docker images. Make sure to [give Docker
 
 See the `Makefile`s and `Dockerfile`s, as well as [build/ubuntu/README.md](build/ubuntu/README.md) for details.
 
-### How to run integration tests
-
-Integration tests require all of the haskell services (brig, galley, cannon, gundeck, proxy, cargohold, spar) to be correctly configured and running, before being able to execute e.g. the `brig-integration` binary. The test for brig also starts nginz, so make sure it has been built before.
-These services require most of the deployment dependencies as seen in the architecture diagram to also be available:
-
-- Required internal dependencies:
-    - cassandra (with the correct schema)
-    - elasticsearch (with the correct schema)
-    - redis
-- Required external dependencies are the following configured AWS services (or "fake" replacements providing the same API):
-    - SES
-    - SQS
-    - SNS
-    - S3
-    - DynamoDB
-- Required additional software:
-    - netcat (in order to allow the services being tested to talk to the dependencies above)
-
-Setting up these real, but in-memory internal and "fake" external dependencies is done easiest using [`docker-compose`](https://docs.docker.com/compose/install/). Run the following in a separate terminal (it will block that terminal, C-c to shut all these docker images down again):
-
-```
-deploy/dockerephemeral/run.sh
-```
-
-Then, to run all integration tests:
-
-```bash
-make integration
-```
-
-Or, alternatively, `make` on the top-level directory (to produce all the service's binaries) followed by e.g `cd services/brig && make integration` to run one service's integration tests only.
-
-### when you need more fine-grained control over your build-test loops
-
-You can use `$WIRE_STACK_OPTIONS` to pass arguments to stack through the `Makefile`s.  This is useful to e.g. pass arguments to a unit test suite or temporarily disable `-Werror` without the risk of accidentally committing anything, like this:
-
-```bash
-WIRE_STACK_OPTIONS='--ghc-options=-Wwarn --test-arguments="--quickcheck-tests=19919 --quickcheck-replay=651712"' make -C services/gundeck
-```
-
-Integration tests are run via `/services/integration.sh`, which does not know about stack or `$WIRE_STACK_OPTIONS`.  Here you can use `$WIRE_INTEGRATION_TEST_OPTIONS`:
-
-```bash
-cd services/spar
-WIRE_INTEGRATION_TEST_OPTIONS="--match='POST /identity-providers'" make i
-```
+#### 2. Use nix-provided build environment 
 
-Alternatively, you can use [tasty's support for passing arguments vie shell variables directly](https://github.com/feuerbach/tasty#runtime).  Or, in the case of spar, the [hspec equivalent](https://hspec.github.io/options.html#specifying-options-through-an-environment-variable), which [is less helpful at times](https://github.com/hspec/hspec/issues/335).
+This is suitable only for local development and testing. See [build instructions](./docs/developer/building.md) in the developer documentation.
 
 ## How to install and run `wire-server`
 

changelog.d/5-internal/building-docs

@@ -0,0 +1 @@
+Add build instructions for developers

docs/developer/building.md

@@ -0,0 +1,84 @@
+# How to build wire-server
+
+As a prerequisiste install the [nix package manager](https://nixos.org/) and [direnv](https://direnv.net/).
+
+All following commands expect that you've entered the nix-provided build-environment by running `direnv allow`.
+
+
+1. Create a `.envrc.local` file with these contents
+
+    ```
+    export COMPILE_NGINX_USING_NIX=1
+    export WIRE_BUILD_WITH_CABAL=1
+    ```
+
+   and reload the direnv via `direnv reload`
+
+2. Create a `cabal.project.local`. This file is not included in wire-server because it disables optimization.
+
+   ```
+   make cabal.project.local
+   ```
+
+   This should be re-run whenver a new local cabal package is added to the cabal project.
+
+Then the following Makefile targets can be used to compile and test wire-server locally:
+
+```
+# to compile all binaries to ./dist run
+make
+
+# to build and install all of galley's executables
+make c package=galley
+
+# also run galley's unit tests
+make c package=galley test=1
+```
+
+## Troubleshooting
+
+### Linker errors while compiling
+
+Linker errors can occur if the nix-provided build environment (see `nix/` directory) changes. Since cabal is not aware of the changed environment the cached build artifacts in `./dist-newstyle` and `~/.cabal/store/` from previous builds may be invalid causing the linker errors.
+
+Haskell Language Server stores its build artifacts in `~/.cache/hie-bios` (equivalent to the `./dist-newstyle` directory) which become invalid for the same reason.
+
+The easiest course of action is to to remove these directories via: 
+
+```
+make delete-cache-on-linker-errors
+```
+
+# How to run integration tests
+
+Integration tests require all of the haskell services (brig, galley, cannon, gundeck, proxy, cargohold, spar) to be correctly configured and running, before being able to execute e.g. the `brig-integration` binary. The test for brig also starts nginz, so make sure it has been built before.
+These services require most of the deployment dependencies as seen in the architecture diagram to also be available:
+
+- Required internal dependencies:
+    - cassandra (with the correct schema)
+    - elasticsearch (with the correct schema)
+    - redis
+- Required external dependencies are the following configured AWS services (or "fake" replacements providing the same API):
+    - SES
+    - SQS
+    - SNS
+    - S3
+    - DynamoDB
+- Required additional software:
+    - netcat (in order to allow the services being tested to talk to the dependencies above)
+
+Setting up these real, but in-memory internal and "fake" external dependencies is done easiest using [`docker-compose`](https://docs.docker.com/compose/install/). Run the following in a separate terminal (it will block that terminal, C-c to shut all these docker images down again):
+
+```
+deploy/dockerephemeral/run.sh
+```
+
+After all containers are up you can use these Makefile targets to run the tests locally:
+
+```
+# build and run galley's integration tests
+make ci package=galley 
+
+# run galley's integration tests that match a pattern
+TASTY_PATTERN="/MLS/" make ci package=galley
+```

hie.yaml

@@ -0,0 +1,2 @@
+cradle:
+  cabal: {}

tools/convert-to-cabal/README.md

@@ -1,86 +1,13 @@
 # How to convert the project to cabal
 
-1. Run
+Run
 
-   ```bash
-   ./tools/convert-to-cabal/generate.sh
-   ```
-
-   This will generate these files
-   - `cabal.project.freeze`
-   - `cabal.project`
-
-2. Create a `cabal.project.local` file with
-
-   ``` 
-   optimization: False
-   ```
-
-   This configures that local builds fast without optimization.
-
-   To make sure Haskell Language Server also builds all projects without optimization run this:
-
-   ```bash
-   echo "optimization: False" > ./cabal.project.local
-   ./hack/bin/cabal-project-local-template.sh "ghc-options: -O0" >> ./cabal.project.local
-   ```
-
-   Note: cabal v2-repl (which is run by hie-bios (HLS)) seem to be ignoring "optimization" flag for local dependencies, this is why we need to specify `ghc-options` explicitely.
-
-
-# How to use the project with cabal
-
-1. Update your environment.
-   ```bash
-   cabal update
-   ```
-
-   Add this to your .envrc.local
-   ```bash
-   export WIRE_BUILD_WITH_CABAL=1
-   ```
-
-   You should be able to build wire-server with cabal now:
-
-   ```bash
-   make install # using cabal
-   make c package=brig # to build and install all of brig's executables
-   make c package=brig test=1 # also run unit tests
-   make ci package=brig pattern="delete" # build and run brig's integration tests
-   ```
-
-2. For Haskell Language Server change `hie.yaml` to use cabal
-   ```bash
-   WIRE_BUILD_WITH_CABAL=1 make hie.yaml
-   ```
-
-
-
-## Notes
-
-- `cabal v2-repl` (used by hie-bios) seem to be ignoring "optimization" flag for local dependencies. However it respects ghc-options
-
-```
-package foo
-  ghc-options: -O0
+```bash
+./tools/convert-to-cabal/generate.sh
 ```
 
-- With new cabal build there doesn't seem to be any way of running tests as part of a build. You have to run the tests manually.
-      https://github.com/haskell/cabal/issues/7267
-
-- Nix integration (`nix: True` in `~/.cabal/config`) is not supported in new-build.
-  https://github.com/haskell/cabal/issues/4646
-  That's why you have to enter the environment defined by .envrc to use cabal.
+This will generate these files
+- `cabal.project.freeze`
+- `cabal.project`
 
-- cabal oddity? Specifying `--ghc-options` twice yields different result
-
-    if run
-    ```
-    cabal build --ghc-options "-O0"  exe:brig
-    ```
-
-    and then
-    ```
-    cabal build --ghc-options "-O0" --ghc-options "-O0"  exe:brig
-    ```
-    Cabal will retry to build brig and _all_ of its dependencies
+   Note: cabal v2-repl (which is run by hie-bios (HLS)) seem to be ignoring "optimization" flag for local dependencies, this is why we need to specify `ghc-options` explicitely.