diff --git a/.gitignore b/.gitignore index 9e15d9491d0..47112aa3707 100644 --- a/.gitignore +++ b/.gitignore @@ -82,8 +82,6 @@ deploy/dockerephemeral/build/smtp/ /libs/libzauth/bzauth-c/deb/usr # Generated by "make hie.yaml" -hie.yaml -hie.orig.yaml stack-dev.yaml # HIE db files (e.g. generated for stan) diff --git a/Makefile b/Makefile index 663e449c58b..51a258b816a 100644 --- a/Makefile +++ b/Makefile @@ -53,9 +53,14 @@ endif .PHONY: delete-cache-on-linker-errors delete-cache-on-linker-errors: rm -rf ~/.cache/hie-bios - rm -rf ~/.cabal + rm -rf ~/.cabal/store rm -rf ./dist-newstyle +.PHONY: cabal.project.local +cabal.project.local: + echo "optimization: False" > ./cabal.project.local + ./hack/bin/cabal-project-local-template.sh "ghc-options: -O0" >> ./cabal.project.local + # Build all Haskell services and executables with -O0, run unit tests .PHONY: fast fast: init diff --git a/README.md b/README.md index 7c277e24111..fba5b3a2e49 100644 --- a/README.md +++ b/README.md @@ -18,22 +18,6 @@ For documentation on how to self host your own Wire-Server see [this section](#h See more in "[Open sourcing Wire server code](https://medium.com/@wireapp/open-sourcing-wire-server-code-ef7866a731d5)". -## Table of contents - - - -* [Contents of this repository](#contents-of-this-repository) -* [Architecture Overview](#architecture-overview) -* [Development setup](#development-setup) - * [How to build `wire-server` binaries](#how-to-build-wire-server-binaries) - * [1. Compile sources natively.](#1-compile-sources-natively) - * [2. Use docker](#2-use-docker) - * [How to run integration tests](#how-to-run-integration-tests) - * [when you need more fine-grained control over your build-test loops](#when-you-need-more-fine-grained-control-over-your-build-test-loops) -* [How to install and run `wire-server`](#how-to-install-and-run-wire-server) - - - ## Contents of this repository This repository contains the following source code: @@ -82,29 +66,7 @@ private network. There are two options: -#### 1. Compile sources natively. - -This requires a range of dependencies that depend on your platform/OS, such as: - -- Haskell & Rust compiler and package managers -- Some package dependencies (libsodium, openssl, protobuf, icu, geoip, snappy, [cryptobox-c](https://github.com/wireapp/cryptobox-c), ...) that depend on your platform/OS - -See [docs/developer/dependencies.md](docs/legacy/developer/dependencies.md) for details. - -Once all dependencies are set up, the following should succeed: - -```bash -# build all haskell services -make -# build one haskell service, e.g. brig: -cd services/brig && make -``` - -The default make target (`fast`) compiles unoptimized (faster compilation time, slower binaries), which should be fine for development purposes. Use `make install` to get optimized binaries. - -For building nginz, see [services/nginz/README.md](services/nginz/README.md) - -#### 2. Use docker +#### 1. Use docker *If you don't wish to build all docker images from scratch (e.g. the `ubuntu20-builder` takes a very long time), ready-built images can be downloaded from [here](https://quay.io/organization/wire).* @@ -123,54 +85,9 @@ will, eventually, have built a range of docker images. Make sure to [give Docker See the `Makefile`s and `Dockerfile`s, as well as [build/ubuntu/README.md](build/ubuntu/README.md) for details. -### How to run integration tests - -Integration tests require all of the haskell services (brig, galley, cannon, gundeck, proxy, cargohold, spar) to be correctly configured and running, before being able to execute e.g. the `brig-integration` binary. The test for brig also starts nginz, so make sure it has been built before. -These services require most of the deployment dependencies as seen in the architecture diagram to also be available: - -- Required internal dependencies: - - cassandra (with the correct schema) - - elasticsearch (with the correct schema) - - redis -- Required external dependencies are the following configured AWS services (or "fake" replacements providing the same API): - - SES - - SQS - - SNS - - S3 - - DynamoDB -- Required additional software: - - netcat (in order to allow the services being tested to talk to the dependencies above) - -Setting up these real, but in-memory internal and "fake" external dependencies is done easiest using [`docker-compose`](https://docs.docker.com/compose/install/). Run the following in a separate terminal (it will block that terminal, C-c to shut all these docker images down again): - -``` -deploy/dockerephemeral/run.sh -``` - -Then, to run all integration tests: - -```bash -make integration -``` - -Or, alternatively, `make` on the top-level directory (to produce all the service's binaries) followed by e.g `cd services/brig && make integration` to run one service's integration tests only. - -### when you need more fine-grained control over your build-test loops - -You can use `$WIRE_STACK_OPTIONS` to pass arguments to stack through the `Makefile`s. This is useful to e.g. pass arguments to a unit test suite or temporarily disable `-Werror` without the risk of accidentally committing anything, like this: - -```bash -WIRE_STACK_OPTIONS='--ghc-options=-Wwarn --test-arguments="--quickcheck-tests=19919 --quickcheck-replay=651712"' make -C services/gundeck -``` - -Integration tests are run via `/services/integration.sh`, which does not know about stack or `$WIRE_STACK_OPTIONS`. Here you can use `$WIRE_INTEGRATION_TEST_OPTIONS`: - -```bash -cd services/spar -WIRE_INTEGRATION_TEST_OPTIONS="--match='POST /identity-providers'" make i -``` +#### 2. Use nix-provided build environment -Alternatively, you can use [tasty's support for passing arguments vie shell variables directly](https://github.com/feuerbach/tasty#runtime). Or, in the case of spar, the [hspec equivalent](https://hspec.github.io/options.html#specifying-options-through-an-environment-variable), which [is less helpful at times](https://github.com/hspec/hspec/issues/335). +This is suitable only for local development and testing. See [build instructions](./docs/developer/building.md) in the developer documentation. ## How to install and run `wire-server` diff --git a/changelog.d/5-internal/building-docs b/changelog.d/5-internal/building-docs new file mode 100644 index 00000000000..bb36e0a7929 --- /dev/null +++ b/changelog.d/5-internal/building-docs @@ -0,0 +1 @@ +Add build instructions for developers diff --git a/docs/developer/building.md b/docs/developer/building.md new file mode 100644 index 00000000000..9ed0d365b7f --- /dev/null +++ b/docs/developer/building.md @@ -0,0 +1,84 @@ +# How to build wire-server + +As a prerequisiste install the [nix package manager](https://nixos.org/) and [direnv](https://direnv.net/). + +All following commands expect that you've entered the nix-provided build-environment by running `direnv allow`. + + +1. Create a `.envrc.local` file with these contents + + ``` + export COMPILE_NGINX_USING_NIX=1 + export WIRE_BUILD_WITH_CABAL=1 + ``` + + and reload the direnv via `direnv reload` + +2. Create a `cabal.project.local`. This file is not included in wire-server because it disables optimization. + + ``` + make cabal.project.local + ``` + + This should be re-run whenver a new local cabal package is added to the cabal project. + +Then the following Makefile targets can be used to compile and test wire-server locally: + +``` +# to compile all binaries to ./dist run +make + +# to build and install all of galley's executables +make c package=galley + +# also run galley's unit tests +make c package=galley test=1 +``` + +## Troubleshooting + +### Linker errors while compiling + +Linker errors can occur if the nix-provided build environment (see `nix/` directory) changes. Since cabal is not aware of the changed environment the cached build artifacts in `./dist-newstyle` and `~/.cabal/store/` from previous builds may be invalid causing the linker errors. + +Haskell Language Server stores its build artifacts in `~/.cache/hie-bios` (equivalent to the `./dist-newstyle` directory) which become invalid for the same reason. + +The easiest course of action is to to remove these directories via: + +``` +make delete-cache-on-linker-errors +``` + +# How to run integration tests + +Integration tests require all of the haskell services (brig, galley, cannon, gundeck, proxy, cargohold, spar) to be correctly configured and running, before being able to execute e.g. the `brig-integration` binary. The test for brig also starts nginz, so make sure it has been built before. +These services require most of the deployment dependencies as seen in the architecture diagram to also be available: + +- Required internal dependencies: + - cassandra (with the correct schema) + - elasticsearch (with the correct schema) + - redis +- Required external dependencies are the following configured AWS services (or "fake" replacements providing the same API): + - SES + - SQS + - SNS + - S3 + - DynamoDB +- Required additional software: + - netcat (in order to allow the services being tested to talk to the dependencies above) + +Setting up these real, but in-memory internal and "fake" external dependencies is done easiest using [`docker-compose`](https://docs.docker.com/compose/install/). Run the following in a separate terminal (it will block that terminal, C-c to shut all these docker images down again): + +``` +deploy/dockerephemeral/run.sh +``` + +After all containers are up you can use these Makefile targets to run the tests locally: + +``` +# build and run galley's integration tests +make ci package=galley + +# run galley's integration tests that match a pattern +TASTY_PATTERN="/MLS/" make ci package=galley +``` diff --git a/hie.yaml b/hie.yaml new file mode 100644 index 00000000000..daf97e6f4ab --- /dev/null +++ b/hie.yaml @@ -0,0 +1,2 @@ +cradle: + cabal: {} diff --git a/tools/convert-to-cabal/README.md b/tools/convert-to-cabal/README.md index 52e78d1695c..1514458cb07 100644 --- a/tools/convert-to-cabal/README.md +++ b/tools/convert-to-cabal/README.md @@ -1,86 +1,13 @@ # How to convert the project to cabal -1. Run +Run - ```bash - ./tools/convert-to-cabal/generate.sh - ``` - - This will generate these files - - `cabal.project.freeze` - - `cabal.project` - -2. Create a `cabal.project.local` file with - - ``` - optimization: False - ``` - - This configures that local builds fast without optimization. - - To make sure Haskell Language Server also builds all projects without optimization run this: - - ```bash - echo "optimization: False" > ./cabal.project.local - ./hack/bin/cabal-project-local-template.sh "ghc-options: -O0" >> ./cabal.project.local - ``` - - Note: cabal v2-repl (which is run by hie-bios (HLS)) seem to be ignoring "optimization" flag for local dependencies, this is why we need to specify `ghc-options` explicitely. - - -# How to use the project with cabal - -1. Update your environment. - ```bash - cabal update - ``` - - Add this to your .envrc.local - ```bash - export WIRE_BUILD_WITH_CABAL=1 - ``` - - You should be able to build wire-server with cabal now: - - ```bash - make install # using cabal - make c package=brig # to build and install all of brig's executables - make c package=brig test=1 # also run unit tests - make ci package=brig pattern="delete" # build and run brig's integration tests - ``` - -2. For Haskell Language Server change `hie.yaml` to use cabal - ```bash - WIRE_BUILD_WITH_CABAL=1 make hie.yaml - ``` - - - -## Notes - -- `cabal v2-repl` (used by hie-bios) seem to be ignoring "optimization" flag for local dependencies. However it respects ghc-options - -``` -package foo - ghc-options: -O0 +```bash +./tools/convert-to-cabal/generate.sh ``` -- With new cabal build there doesn't seem to be any way of running tests as part of a build. You have to run the tests manually. - https://github.com/haskell/cabal/issues/7267 - -- Nix integration (`nix: True` in `~/.cabal/config`) is not supported in new-build. - https://github.com/haskell/cabal/issues/4646 - That's why you have to enter the environment defined by .envrc to use cabal. +This will generate these files +- `cabal.project.freeze` +- `cabal.project` -- cabal oddity? Specifying `--ghc-options` twice yields different result - - if run - ``` - cabal build --ghc-options "-O0" exe:brig - ``` - - and then - ``` - cabal build --ghc-options "-O0" --ghc-options "-O0" exe:brig - ``` - Cabal will retry to build brig and _all_ of its dependencies + Note: cabal v2-repl (which is run by hie-bios (HLS)) seem to be ignoring "optimization" flag for local dependencies, this is why we need to specify `ghc-options` explicitely.