Release 2022-01-18

.dockerignore

@@ -1,5 +1,6 @@
 **/.*
 **/dist
+**/dist-buildah
 **/target
 **/*.aci
 **/*.tgz

.gitignore

@@ -85,6 +85,9 @@ hie.yaml
 hie.orig.yaml
 stack-dev.yaml
 
+# HIE db files (e.g. generated for stan)
+*.hie
+
 # generated files under .local
 .local
 
@@ -102,4 +105,4 @@ telepresence.log
 
 # local config
 .envrc.local
-cabal.project.local
+cabal.project.local

CHANGELOG.md

@@ -1,5 +1,83 @@
 <!-- if you're not the release manager, do your edits to changelog under CHANGELOG.d/ -->
 
+# [2022-01-18]
+
+## Release notes
+
+* This release introduces a mandatory `federationDomain` configuration setting to cargohold. Please update your `values/wire-server/values.yaml` to set `cargohold.settings.federationDomain` to the same value as the corresponding option in galley (and brig). (#1990)
+* The brig server config option `setDefaultLocale` has been replaced by `setDefaultUserLocale` and `setDefaultTemplateLocale` (see docs/reference/config-options.md for details) (#2028)
+* From this release onwards, the images for haskell components (brig, galley,
+  cargohold, etc.) will be using Ubuntu 20.04 as the base. The images are about
+  30-35 MB larger than the previous alpine based images. (#1852)
+* Wire cloud operators: Make sure [#35](https://github.com/wireapp/ansible-sft/pull/35) is applied to all SFT servers before deploying. (#2030)
+
+## API changes
+
+* The deprecated endpoint `GET /teams` now ignores query parameters `ids`, `start` (#2027)
+* Add qualified v4 endpoints for downloading and deleting assets. The upload API is still on the same path, but the asset object it returns now contains a `domain` field. Note that federated behaviour is still not implemented. (#2002)
+* Enable downloading assets from a remote (federated) cargohold instance via the v4 API.
+  The content of remote assets is returned as stream with content type
+  `application/octet-stream`.
+  Please refer to the Swagger API documentation for more details. (#2004)
+* Remove resumable upload API (#1998)
+
+## Features
+
+* Allow configuring setDefaultLocale in brig using helm chart (#2025)
+* If the guest links team feature is disabled guest links will be revoked. (#1976)
+* Revoke guest links if feature is disabled. If the guest links team feature is disabled `get /conversations/join`, `post /conversations/:cnv/code`, and `get /conversations/:cnv/code` will return an error. (#1980)
+* Specialize `setDefaultLocale` to distinguish between default user locale and default template locale if the user's locale is n/a. (#2028)
+
+## Bug fixes and other updates
+
+* Fix an issue with remote asset streaming (#2037, #2038)
+
+## Documentation
+
+* Annotate a first batch of integration and unit tests to map them to externally-facing documentation (#1869)
+* Add the description to several test cases (#1991)
+* Improve documentation for stern tool and helm chart (#2032)
+
+## Internal changes
+
+* Replace servant-generic in Galley with a custom `Named` combinator (#2022)
+* The Swagger documentation module is not regenerated anymore if its content is unchanged (#2018)
+* cabal-run-integration.sh - remove Makefile indirection (#2044)
+* Fix test runner for global cabal make target (#1987)
+* The `cabal-install-artefacts.sh` script now creates the `dist` directory if it does not exist (#2007)
+* Set `purge: false` in fake-s3 chart (#1981)
+* Add missing backendTwo.carghold in integration.yaml (#2039)
+* Use GHC 8.10.7 and stack 2.7.3 for builds (#1852)
+* Fix non-controversial HLint issues in federator to improve code quality (#2011)
+* Added laws for DefaultSsoCode, Now, IdP and ScimExternalIdStore (#1940)
+* Moved specifications for Spar effects out of the test suite and into the library (#2005)
+* Tag integration tests for security audit. (#2000)
+* Upgrade nixpkgs pin used to provision developement dependencies (#1852)
+* Servantify Galley Teams API. (#2008, #2010, #2027)
+* When sending an activation code, the blocked domains are checked before the whitelist. This only affects the wire SaaS staging environment (there is no whitelist configuration in prod, and blocked domains are not applicable to on-prem installations). (#2023)
+* Add a helm chart that deploys [restund](https://docs.wire.com/understand/restund.html) (#2003)
+* Publish restund helm chart (#2036)
+* Improve optional field API in schema-profunctor (#1988)
+* Migrate the public API of Cannon to Servant. (There is an internal API that is not yet migrated.) (#2024)
+* sftd chart: Add multiSFT option, remove additionalArgs option (#1992)
+* sftd chart: Fix quoted args for multiSFT option (#1999)
+* `rangedSchema` does not need to be passed singletons explicitly anymore (#2017)
+* Split cannon benchmarks and tests (#1986)
+* Tag integration tests for certification. (#1985)
+* Tag integration tests for certification. (#2001)
+* New internal endpoint to configure the guest links team feature. (#1993)
+
+## Federation changes
+
+* Make federator capable of streaming responses (#1966)
+* Use `Named` routes for the federation API (#2033)
+* Fix Brig's configmap for SFT lookups (#2015)
+* SFTD chart: provide a /sft_servers_all.json url that can be used by brig to populate /calls/config/v2 (#2019)
+* Allow making HTTP-only requests to SFTs via an IPv4 address (#2026)
+* Replace IPv4-HTTP-only Approach to SFT Server Lookup with /sft_servers_all.json (#2030)
+* Extend GET /calls/config/v2 to include all SFT servers in federation (#2012)
+* Improve Brig's configuration for SFTs and fix a call to SFT servers (#2014)
+
 # [2021-12-10]
 
 ## Release notes

Makefile

@@ -7,14 +7,14 @@ NAMESPACE             ?= test-$(USER)
 DOCKER_TAG            ?= $(USER)
 # default helm chart version must be 0.0.42 for local development (because 42 is the answer to the universe and everything)
 HELM_SEMVER           ?= 0.0.42
-# The list of helm charts needed for integration tests on kubernetes
-CHARTS_INTEGRATION    := wire-server databases-ephemeral fake-aws nginx-ingress-controller nginx-ingress-services wire-server-metrics fluent-bit kibana
+# The list of helm charts needed on internal kubernetes testing environments
+CHARTS_INTEGRATION    := wire-server databases-ephemeral fake-aws nginx-ingress-controller nginx-ingress-services wire-server-metrics fluent-bit kibana sftd restund
 # The list of helm charts to publish on S3
 # FUTUREWORK: after we "inline local subcharts",
 # (e.g. move charts/brig to charts/wire-server/brig)
 # this list could be generated from the folder names under ./charts/ like so:
 # CHARTS_RELEASE := $(shell find charts/ -maxdepth 1 -type d | xargs -n 1 basename | grep -v charts)
-CHARTS_RELEASE        := wire-server redis-ephemeral databases-ephemeral fake-aws fake-aws-s3 fake-aws-sqs aws-ingress  fluent-bit kibana backoffice calling-test demo-smtp elasticsearch-curator elasticsearch-external elasticsearch-ephemeral minio-external cassandra-external nginx-ingress-controller nginx-ingress-services reaper wire-server-metrics sftd
+CHARTS_RELEASE        := wire-server redis-ephemeral databases-ephemeral fake-aws fake-aws-s3 fake-aws-sqs aws-ingress  fluent-bit kibana backoffice calling-test demo-smtp elasticsearch-curator elasticsearch-external elasticsearch-ephemeral minio-external cassandra-external nginx-ingress-controller nginx-ingress-services reaper wire-server-metrics sftd restund
 BUILDAH_PUSH          ?= 0
 KIND_CLUSTER_NAME     := wire-server
 BUILDAH_KIND_LOAD     ?= 1
@@ -70,9 +70,10 @@ endif
 
 # ci here doesn't refer to continuous integration, but to cabal-integration
 # Usage: make ci package=brig test=1
+# If you want to pass arguments to the test-suite call the script directly.
 .PHONY: ci
 ci: c
-	./hack/bin/cabal-run-integration.sh $(package) $(pattern)
+	./hack/bin/cabal-run-integration.sh $(package)
 
 # reset db using cabal
 .PHONY: db-reset-package
@@ -189,39 +190,39 @@ i-%:
 .PHONY: docker-prebuilder
 docker-prebuilder:
 	# `docker-prebuilder` needs to be built or pulled only once (unless native dependencies change)
-	$(MAKE) -C build/alpine prebuilder
+	$(MAKE) -C build/ubuntu prebuilder
 
 .PHONY: docker-deps
 docker-deps:
 	# `docker-deps` needs to be built or pulled only once (unless native dependencies change)
-	$(MAKE) -C build/alpine deps
+	$(MAKE) -C build/ubuntu deps
 
 .PHONY: docker-builder
 docker-builder:
 	# `docker-builder` needs to be built or pulled only once (unless native dependencies change)
-	$(MAKE) -C build/alpine builder
+	$(MAKE) -C build/ubuntu builder
 
 .PHONY: docker-intermediate
 docker-intermediate:
 	# `docker-intermediate` needs to be built whenever code changes - this essentially runs `stack clean && stack install` on the whole repo
-	docker build -t $(DOCKER_USER)/alpine-intermediate:$(DOCKER_TAG) -f build/alpine/Dockerfile.intermediate --build-arg builder=$(DOCKER_USER)/alpine-builder:develop --build-arg deps=$(DOCKER_USER)/alpine-deps:develop .;
-	docker tag $(DOCKER_USER)/alpine-intermediate:$(DOCKER_TAG) $(DOCKER_USER)/alpine-intermediate:latest;
-	if test -n "$$DOCKER_PUSH"; then docker login -u $(DOCKER_USERNAME) -p $(DOCKER_PASSWORD); docker push $(DOCKER_USER)/alpine-intermediate:$(DOCKER_TAG); docker push $(DOCKER_USER)/alpine-intermediate:latest; fi;
+	docker build -t $(DOCKER_USER)/ubuntu20-intermediate:$(DOCKER_TAG) -f build/ubuntu/Dockerfile.intermediate --build-arg builder=$(DOCKER_USER)/ubuntu20-builder:develop --build-arg deps=$(DOCKER_USER)/ubuntu20-deps:develop .;
+	docker tag $(DOCKER_USER)/ubuntu20-intermediate:$(DOCKER_TAG) $(DOCKER_USER)/ubuntu20-intermediate:latest;
+	if test -n "$$DOCKER_PUSH"; then docker login -u $(DOCKER_USERNAME) -p $(DOCKER_PASSWORD); docker push $(DOCKER_USER)/ubuntu20-intermediate:$(DOCKER_TAG); docker push $(DOCKER_USER)/ubuntu20-intermediate:latest; fi;
 
 .PHONY: docker-exe-%
 docker-exe-%:
-	docker image ls | grep $(DOCKER_USER)/alpine-deps > /dev/null || (echo "'make docker-deps' required.", exit 1)
-	docker image ls | grep $(DOCKER_USER)/alpine-intermediate > /dev/null || (echo "'make docker-intermediate' required."; exit 1)
-	docker build -t $(DOCKER_USER)/"$*":$(DOCKER_TAG) -f build/alpine/Dockerfile.executable --build-arg executable="$*" --build-arg intermediate=$(DOCKER_USER)/alpine-intermediate --build-arg deps=$(DOCKER_USER)/alpine-deps .
+	docker image ls | grep $(DOCKER_USER)/ubuntu20-deps > /dev/null || (echo "'make docker-deps' required.", exit 1)
+	docker image ls | grep $(DOCKER_USER)/ubuntu20-intermediate > /dev/null || (echo "'make docker-intermediate' required."; exit 1)
+	docker build -t $(DOCKER_USER)/"$*":$(DOCKER_TAG) -f build/ubuntu/Dockerfile.executable --build-arg executable="$*" --build-arg intermediate=$(DOCKER_USER)/ubuntu20-intermediate --build-arg deps=$(DOCKER_USER)/ubuntu20-deps .
 	docker tag $(DOCKER_USER)/"$*":$(DOCKER_TAG) $(DOCKER_USER)/"$*":latest
 	if test -n "$$DOCKER_PUSH"; then docker login -u $(DOCKER_USERNAME) -p $(DOCKER_PASSWORD); docker push $(DOCKER_USER)/"$*":$(DOCKER_TAG); docker push $(DOCKER_USER)/"$*":latest; fi;
 
 .PHONY: docker-services
 docker-services:
 	# make docker-services doesn't compile, only makes small images out of the `docker-intermediate` image
 	# to recompile, run `docker-intermediate` first.
-	docker image ls | grep $(DOCKER_USER)/alpine-deps > /dev/null || (echo "'make docker-deps' required.", exit 1)
-	docker image ls | grep $(DOCKER_USER)/alpine-intermediate > /dev/null || (echo "'make docker-intermediate' required."; exit 1)
+	docker image ls | grep $(DOCKER_USER)/ubuntu20-deps > /dev/null || (echo "'make docker-deps' required.", exit 1)
+	docker image ls | grep $(DOCKER_USER)/ubuntu20-intermediate > /dev/null || (echo "'make docker-intermediate' required."; exit 1)
 	# `make -C services/brig docker` == `make docker-exe-brig docker-exe-brig-integration docker-exe-brig-schema docker-exe-brig-index`
 	$(MAKE) -C services/brig docker
 	$(MAKE) -C services/gundeck docker
@@ -235,10 +236,10 @@ docker-services:
 
 DOCKER_DEV_NETWORK := --net=host
 DOCKER_DEV_VOLUMES := -v `pwd`:/wire-server
-DOCKER_DEV_IMAGE   := quay.io/wire/alpine-builder:$(DOCKER_TAG)
+DOCKER_DEV_IMAGE   := quay.io/wire/ubuntu20-builder:$(DOCKER_TAG)
 .PHONY: run-docker-builder
 run-docker-builder:
-	@echo "if this does not work, consider 'docker pull', 'docker tag', or 'make -C build-alpine builder'."
+	@echo "if this does not work, consider 'docker pull', 'docker tag', or 'make -C build/ubuntu builder'."
 	docker run --workdir /wire-server -it $(DOCKER_DEV_NETWORK) $(DOCKER_DEV_VOLUMES) --rm $(DOCKER_DEV_IMAGE) /bin/bash
 
 .PHONY: git-add-cassandra-schema

README.md

@@ -108,22 +108,22 @@ For building nginz, see [services/nginz/README.md](services/nginz/README.md)
 
 #### 2. Use docker
 
-*If you don't wish to build all docker images from scratch (e.g. the `alpine-builder` takes a very long time), ready-built images can be downloaded from [here](https://quay.io/organization/wire).*
+*If you don't wish to build all docker images from scratch (e.g. the `ubuntu20-builder` takes a very long time), ready-built images can be downloaded from [here](https://quay.io/organization/wire).*
 
 If you wish to build your own docker images, you need [docker version >= 17.05](https://www.docker.com/) and [`make`](https://www.gnu.org/software/make/). Then,
 
 ```bash
 # optionally:
-# make docker-builder # if you don't run this, it pulls the alpine-builder image from quay.io
+# make docker-builder # if you don't run this, it pulls the ubuntu20-builder image from quay.io
 make docker-deps docker-intermediate docker-services
 
 # subsequent times, after changing code, if you wish to re-create docker images, it's sufficient to
 make docker-intermediate docker-services
 ```
 
-will, eventually, have built a range of docker images. Make sure to [give Docker enough RAM](https://github.com/wireapp/wire-server/issues/562); if you see `make: *** [builder] Error 137`, it might be a sign that the build ran out of memory. You can also mix and match – e.g. pull the [`alpine-builder`](https://quay.io/repository/wire/alpine-builder?tab=tags) image and build the rest locally.
+will, eventually, have built a range of docker images. Make sure to [give Docker enough RAM](https://github.com/wireapp/wire-server/issues/562); if you see `make: *** [builder] Error 137`, it might be a sign that the build ran out of memory. You can also mix and match – e.g. pull the [`ubuntu20-builder`](https://quay.io/repository/wire/ubuntu20-builder?tab=tags) image and build the rest locally.
 
-See the `Makefile`s and `Dockerfile`s, as well as [build/alpine/README.md](build/alpine/README.md) for details.
+See the `Makefile`s and `Dockerfile`s, as well as [build/ubuntu/README.md](build/ubuntu/README.md) for details.
 
 ### How to run integration tests