diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index a0a014d6e2f..dc8abfb10dc 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,15 +1,5 @@ ## Checklist - - [ ] The **PR Title** explains the impact of the change. - - [ ] The **PR description** provides context as to why the change should occur and what the code contributes to that effect. This could also be a link to a JIRA ticket or a Github issue, if there is one. - - [ ] If this PR changes development workflow or dependencies, they have been A) automated and B) documented under docs/developer/. All efforts have been taken to minimize development setup breakage or slowdown for co-workers. - - [ ] If HTTP endpoint paths have been added or renamed, or feature configs have changed, the **endpoint / config-flag checklist** (see Wire-employee only backend [wiki page](https://github.com/zinfra/backend-wiki/wiki/Checklists)) has been followed. - - [ ] If a cassandra schema migration has been added, I ran **`make git-add-cassandra-schema`** to update the cassandra schema documentation. - - [ ] **changelog.d** contains the following bits of information ([details](https://github.com/wireapp/wire-server/blob/develop/docs/developer/changelog.md)): - - [ ] A file with the changelog entry in one or more suitable sub-sections. The sub-sections are marked by directories inside `changelog.d`. - - [ ] If new config options introduced: added usage description under docs/reference/config-options.md - - [ ] If new config options introduced: recommended measures to be taken by on-premise instance operators. - - [ ] If a cassandra schema migration is backwards incompatible (see also [these docs](https://github.com/wireapp/wire-server/blob/develop/docs/developer/cassandra-interaction.md#cassandra-schema-migrations)), measures to be taken by on-premise instance operators are explained. - - [ ] If a data migration (not schema migration) introduced: measures to be taken by on-premise instance operators. - - [ ] If public end-points have been changed or added: does nginz need un upgrade? - - [ ] If internal end-points have been added or changed: which services have to be deployed in a specific order? + - [ ] Add a new entry in an appropriate subdirectory of `changelog.d` + - [ ] Read and follow the +[PR guidelines](https://github.com/wireapp/wire-server/blob/develop/docs/developer/pr-guidelines.md) diff --git a/.gitignore b/.gitignore index 9e15d9491d0..47112aa3707 100644 --- a/.gitignore +++ b/.gitignore @@ -82,8 +82,6 @@ deploy/dockerephemeral/build/smtp/ /libs/libzauth/bzauth-c/deb/usr # Generated by "make hie.yaml" -hie.yaml -hie.orig.yaml stack-dev.yaml # HIE db files (e.g. generated for stan) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8a317848f30..ed7e44be1dc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,95 @@ +# [2022-09-01] (Chart Release 4.23.0) + +## Release notes + + +* The internal endpoint `GET i/mls/clients` has been changed, and it now returns a list of `ClientInfo` instead of a list of `ClientId`. (#2631) + + +## API changes + + +* Fix key package error description (#2651) + +* Expose MLS public keys in a new endpoint `GET /mls/public-keys`. (#2602) + + +## Features + + +* The coturn chart now supports exposing the control port over TLS. (#2620) + +* Forward all MLS default proposal types (#2628) + +* New endpoints `HEAD` and `GET /nonce/clients` to request new nonces for client certificate requests (coming up soon). (#2641, #2655) + +## Bug fixes and other updates + + +* Fix cql-io bug where restarting whole cassandra cluster could cause downtime. Upstream changes in https://gitlab.com/twittner/cql-io/-/merge_requests/20 (#2640) + +* Improve client check when adding clients to MLS conversations (#2631) + + +## Documentation + + +* Move developer docs onto docs.wire.com (instead of exposing them on github only) (#2622, #2649) + +* Add build instructions for developers (#2621) + +* Make target audience explicit on docs.wire.com (#2662) + + +## Internal changes + + +* Support for external Add proposals (#2567) + +* Add additional checks on incoming MLS messages: + * if the sender matches the authenticated user + * if the sender of message to a remote conversation is a member + * if the group ID of a remote conversation matches the local mapping (#2618) + +* Apply changes introduced by cabal-fmt. (#2624) + +* Remove some redudant constraints in brig (#2638) + +* Brig Polysemy: Port UserPendingActivationStore to polysemy (#2636) + + +* Add make target `delete-cache-on-linker-errors` to delete all Haskell compilation related caches. This is useful in cases where the development environment gets into an inconsistent state. (#2623) + + +* Move Paging effect from galley into polysemy-wire-zoo (#2648) + +* Fix broken hls-hlint-plugin in nix env (#2629) + +* Adjust developer PR template and document config and API procedures in-tree. (#2617) + +* Add mls-test-cli to builder image (#2626) + +* Add mls-test-cli to deps image (#2630) + +* mls-test-cli: Use Cargo.lock file when building (#2634) + +* Move common Arbitrary instances to types-common package for compilation speed (#2658) + +* `LoginId` migrated to schema-profunctor (#2633, #2645) + +* Improve cleaning rules in Makefile. (#2639) + +* Fix typos, dangling reference in source code haddocs, etc. (#2586) + +* Update the Elastic Search version used for running integration tests to the one that is delivered by wire-server-deploy. (#2656) + + +## Federation changes + + +* Add mlsPrivateKeyPaths setting to galley (#2602) + + # [2022-08-16] (Chart Release 4.22.0) ## API changes diff --git a/Makefile b/Makefile index cd43c3ee29f..90c9a95208f 100644 --- a/Makefile +++ b/Makefile @@ -50,6 +50,38 @@ else stack install --pedantic --test --bench --no-run-benchmarks --local-bin-path=dist endif +# Clean +.PHONY: full-clean +full-clean: clean + rm -rf ~/.cache/hie-bios +ifdef CABAL_DIR + rm -rf $(CABAL_DIR)/store +else + rm -rf ~/.cabal/store +endif + +.PHONY: clean +clean: +ifeq ($(WIRE_BUILD_WITH_CABAL), 1) + cabal clean +else + stack clean +endif + $(MAKE) -C services/nginz clean + -rm -rf dist + -rm -f .metadata + +.PHONY: clean-hint +clean-hint: + @echo -e "\n\n\n>>> PSA: if you get errors that are hard to explain," + @echo -e ">>> try 'make full-clean' and run your command again." + @echo -e ">>> see https://github.com/wireapp/wire-server/blob/develop/docs/developer/building.md#linker-errors-while-compiling\n\n\n" + +.PHONY: cabal.project.local +cabal.project.local: + echo "optimization: False" > ./cabal.project.local + ./hack/bin/cabal-project-local-template.sh "ghc-options: -O0" >> ./cabal.project.local + # Build all Haskell services and executables with -O0, run unit tests .PHONY: fast fast: init @@ -62,7 +94,7 @@ endif # Usage: make c package=brig test=1 .PHONY: c c: cabal-fmt - cabal build $(WIRE_CABAL_BUILD_OPTIONS) $(package) + cabal build $(WIRE_CABAL_BUILD_OPTIONS) $(package) || ( make clean-hint; false ) ifeq ($(test), 1) ./hack/bin/cabal-run-tests.sh $(package) $(testargs) endif @@ -141,18 +173,6 @@ add-license: shellcheck: ./hack/bin/shellcheck.sh -# Clean -.PHONY: clean -clean: -ifeq ($(WIRE_BUILD_WITH_CABAL), 1) - cabal clean -else - stack clean -endif - $(MAKE) -C services/nginz clean - -rm -rf dist - -rm -f .metadata - ################################# ## running integration tests diff --git a/README.md b/README.md index 7c277e24111..fba5b3a2e49 100644 --- a/README.md +++ b/README.md @@ -18,22 +18,6 @@ For documentation on how to self host your own Wire-Server see [this section](#h See more in "[Open sourcing Wire server code](https://medium.com/@wireapp/open-sourcing-wire-server-code-ef7866a731d5)". -## Table of contents - - - -* [Contents of this repository](#contents-of-this-repository) -* [Architecture Overview](#architecture-overview) -* [Development setup](#development-setup) - * [How to build `wire-server` binaries](#how-to-build-wire-server-binaries) - * [1. Compile sources natively.](#1-compile-sources-natively) - * [2. Use docker](#2-use-docker) - * [How to run integration tests](#how-to-run-integration-tests) - * [when you need more fine-grained control over your build-test loops](#when-you-need-more-fine-grained-control-over-your-build-test-loops) -* [How to install and run `wire-server`](#how-to-install-and-run-wire-server) - - - ## Contents of this repository This repository contains the following source code: @@ -82,29 +66,7 @@ private network. There are two options: -#### 1. Compile sources natively. - -This requires a range of dependencies that depend on your platform/OS, such as: - -- Haskell & Rust compiler and package managers -- Some package dependencies (libsodium, openssl, protobuf, icu, geoip, snappy, [cryptobox-c](https://github.com/wireapp/cryptobox-c), ...) that depend on your platform/OS - -See [docs/developer/dependencies.md](docs/legacy/developer/dependencies.md) for details. - -Once all dependencies are set up, the following should succeed: - -```bash -# build all haskell services -make -# build one haskell service, e.g. brig: -cd services/brig && make -``` - -The default make target (`fast`) compiles unoptimized (faster compilation time, slower binaries), which should be fine for development purposes. Use `make install` to get optimized binaries. - -For building nginz, see [services/nginz/README.md](services/nginz/README.md) - -#### 2. Use docker +#### 1. Use docker *If you don't wish to build all docker images from scratch (e.g. the `ubuntu20-builder` takes a very long time), ready-built images can be downloaded from [here](https://quay.io/organization/wire).* @@ -123,54 +85,9 @@ will, eventually, have built a range of docker images. Make sure to [give Docker See the `Makefile`s and `Dockerfile`s, as well as [build/ubuntu/README.md](build/ubuntu/README.md) for details. -### How to run integration tests - -Integration tests require all of the haskell services (brig, galley, cannon, gundeck, proxy, cargohold, spar) to be correctly configured and running, before being able to execute e.g. the `brig-integration` binary. The test for brig also starts nginz, so make sure it has been built before. -These services require most of the deployment dependencies as seen in the architecture diagram to also be available: - -- Required internal dependencies: - - cassandra (with the correct schema) - - elasticsearch (with the correct schema) - - redis -- Required external dependencies are the following configured AWS services (or "fake" replacements providing the same API): - - SES - - SQS - - SNS - - S3 - - DynamoDB -- Required additional software: - - netcat (in order to allow the services being tested to talk to the dependencies above) - -Setting up these real, but in-memory internal and "fake" external dependencies is done easiest using [`docker-compose`](https://docs.docker.com/compose/install/). Run the following in a separate terminal (it will block that terminal, C-c to shut all these docker images down again): - -``` -deploy/dockerephemeral/run.sh -``` - -Then, to run all integration tests: - -```bash -make integration -``` - -Or, alternatively, `make` on the top-level directory (to produce all the service's binaries) followed by e.g `cd services/brig && make integration` to run one service's integration tests only. - -### when you need more fine-grained control over your build-test loops - -You can use `$WIRE_STACK_OPTIONS` to pass arguments to stack through the `Makefile`s. This is useful to e.g. pass arguments to a unit test suite or temporarily disable `-Werror` without the risk of accidentally committing anything, like this: - -```bash -WIRE_STACK_OPTIONS='--ghc-options=-Wwarn --test-arguments="--quickcheck-tests=19919 --quickcheck-replay=651712"' make -C services/gundeck -``` - -Integration tests are run via `/services/integration.sh`, which does not know about stack or `$WIRE_STACK_OPTIONS`. Here you can use `$WIRE_INTEGRATION_TEST_OPTIONS`: - -```bash -cd services/spar -WIRE_INTEGRATION_TEST_OPTIONS="--match='POST /identity-providers'" make i -``` +#### 2. Use nix-provided build environment -Alternatively, you can use [tasty's support for passing arguments vie shell variables directly](https://github.com/feuerbach/tasty#runtime). Or, in the case of spar, the [hspec equivalent](https://hspec.github.io/options.html#specifying-options-through-an-environment-variable), which [is less helpful at times](https://github.com/hspec/hspec/issues/335). +This is suitable only for local development and testing. See [build instructions](./docs/developer/building.md) in the developer documentation. ## How to install and run `wire-server` diff --git a/build/ubuntu/Dockerfile.builder b/build/ubuntu/Dockerfile.builder index 92b1c8ca573..df52ce9bc4e 100644 --- a/build/ubuntu/Dockerfile.builder +++ b/build/ubuntu/Dockerfile.builder @@ -1,6 +1,19 @@ ARG prebuilder=quay.io/wire/ubuntu20-prebuilder +FROM rust:1.63 as mls-test-cli-builder + +# compile mls-test-cli tool +RUN cd /tmp && \ + git clone https://github.com/wireapp/mls-test-cli && \ + cd mls-test-cli && \ + git rev-parse HEAD + +RUN cd /tmp/mls-test-cli && RUSTFLAGS='-C target-feature=+crt-static' cargo build --release --target x86_64-unknown-linux-gnu + FROM ${prebuilder} + +COPY --from=mls-test-cli-builder /tmp/mls-test-cli/target/x86_64-unknown-linux-gnu/release/mls-test-cli /usr/bin/mls-test-cli + WORKDIR / # Download stack indices and compile/cache dependencies to speed up subsequent diff --git a/build/ubuntu/Dockerfile.deps b/build/ubuntu/Dockerfile.deps index cedf528210d..7b356804b42 100644 --- a/build/ubuntu/Dockerfile.deps +++ b/build/ubuntu/Dockerfile.deps @@ -1,12 +1,3 @@ -FROM rust:1.63 as mls-test-cli-builder - -# compile mls-test-cli tool -RUN cd /tmp && \ - git clone https://github.com/wireapp/mls-test-cli && \ - cd mls-test-cli && \ - cargo build --release - - FROM ubuntu:20.04 as cryptobox-builder # compile cryptobox-c @@ -19,15 +10,21 @@ RUN export DEBIAN_FRONTEND=noninteractive && \ export SODIUM_USE_PKG_CONFIG=1 && \ cargo build --release +FROM rust:1.63 as mls-test-cli-builder + +# compile mls-test-cli tool +RUN cd /tmp && \ + git clone https://github.com/wireapp/mls-test-cli && \ + cd mls-test-cli && \ + git rev-parse HEAD + +RUN cd /tmp/mls-test-cli && RUSTFLAGS='-C target-feature=+crt-static' cargo build --release --target x86_64-unknown-linux-gnu # Minimal dependencies for ubuntu-compiled, dynamically linked wire-server Haskell services FROM ubuntu:20.04 COPY --from=cryptobox-builder /tmp/cryptobox-c/target/release/libcryptobox.so /usr/lib - -# FUTUREWORK: only copy mls-test-cli executables if we are building an -# integration test image -COPY --from=mls-test-cli-builder /tmp/mls-test-cli/target/release/mls-test-cli /usr/bin +COPY --from=mls-test-cli-builder /tmp/mls-test-cli/target/x86_64-unknown-linux-gnu/release/mls-test-cli /usr/bin/mls-test-cli RUN export DEBIAN_FRONTEND=noninteractive && \ apt-get update && \ diff --git a/cabal.project b/cabal.project index 81df343b755..c0c9d989abb 100644 --- a/cabal.project +++ b/cabal.project @@ -142,6 +142,11 @@ source-repository-package location: https://github.com/wireapp/saml2-web-sso tag: 74371cd775cb98d6cf85f6e182244a3c4fd48702 +source-repository-package + type: git + location: https://gitlab.com/axeman/cql-io + tag: c2b6aa995b5817ed7c78c53f72d5aa586ef87c36 + source-repository-package type: git location: https://gitlab.com/axeman/swagger diff --git a/cabal.project.freeze b/cabal.project.freeze index a26406ae156..da0e31f39ce 100644 --- a/cabal.project.freeze +++ b/cabal.project.freeze @@ -575,7 +575,6 @@ constraints: any.AC-Angle ==1.0, any.cpu ==0.1.2, any.cpuinfo ==0.1.0.2, any.cql ==4.0.3, - any.cql-io ==1.1.1, any.cql-io-tinylog ==0.1.0, any.crackNum ==3.1, any.crc32c ==0.0.0, @@ -1153,6 +1152,8 @@ constraints: any.AC-Angle ==1.0, any.hourglass ==0.2.12, any.hourglass-orphans ==0.1.0.0, any.hp2pretty ==0.10, + any.hpack ==0.34.5, + any.hpack-dhall ==0.5.3, any.hpc-codecov ==0.3.0.0, any.hpc-lcov ==1.0.1, any.hprotoc ==2.4.17, @@ -2457,6 +2458,7 @@ constraints: any.AC-Angle ==1.0, any.time-units ==1.0.0, any.timeit ==2.0, any.timelens ==0.2.0.2, + any.timeout ==0.1.1, any.timer-wheel ==0.3.0, any.timerep ==2.0.1.0, any.timezone-olson ==0.2.0, diff --git a/cassandra-schema.cql b/cassandra-schema.cql index 8ef6fcfc8dc..c4666a1071b 100644 --- a/cassandra-schema.cql +++ b/cassandra-schema.cql @@ -1092,6 +1092,27 @@ CREATE TABLE brig_test.invitee_info ( AND read_repair_chance = 0.0 AND speculative_retry = '99PERCENTILE'; +CREATE TABLE brig_test.nonce ( + user uuid, + key text, + nonce uuid, + PRIMARY KEY (user, key) +) WITH CLUSTERING ORDER BY (key ASC) + AND bloom_filter_fp_chance = 0.01 + AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} + AND comment = '' + AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'} + AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'} + AND crc_check_chance = 1.0 + AND dclocal_read_repair_chance = 0.1 + AND default_time_to_live = 300 + AND gc_grace_seconds = 864000 + AND max_index_interval = 2048 + AND memtable_flush_period_in_ms = 0 + AND min_index_interval = 128 + AND read_repair_chance = 0.0 + AND speculative_retry = '99PERCENTILE'; + CREATE TABLE brig_test.provider_keys ( key text PRIMARY KEY, provider uuid diff --git a/charts/brig/templates/configmap.yaml b/charts/brig/templates/configmap.yaml index 5ba1ea5e0fe..196b978a362 100644 --- a/charts/brig/templates/configmap.yaml +++ b/charts/brig/templates/configmap.yaml @@ -284,5 +284,8 @@ data: {{- if .set2FACodeGenerationDelaySecs }} set2FACodeGenerationDelaySecs: {{ .set2FACodeGenerationDelaySecs }} {{- end }} + {{- if .setNonceTtlSecs }} + setNonceTtlSecs: {{ .setNonceTtlSecs }} + {{- end }} {{- end }} {{- end }} diff --git a/charts/brig/values.yaml b/charts/brig/values.yaml index 26dd629905d..ced8902dcfc 100644 --- a/charts/brig/values.yaml +++ b/charts/brig/values.yaml @@ -85,6 +85,7 @@ config: # domainsBlockedForRegistration: # - example.com set2FACodeGenerationDelaySecs: 300 # 5 minutes + setNonceTtlSecs: 300 # 5 minutes smtp: passwordFile: /etc/wire/brig/secrets/smtp-password.txt proxy: {} diff --git a/charts/coturn/Chart.yaml b/charts/coturn/Chart.yaml index 1893104adec..e42ecb81497 100644 --- a/charts/coturn/Chart.yaml +++ b/charts/coturn/Chart.yaml @@ -11,4 +11,4 @@ version: 0.0.42 # This is the version number of the application being deployed. This version number should be # incremented each time you make changes to the application. Versions are not expected to # follow Semantic Versioning. They should reflect the version the application is using. -appVersion: 4.5.2-wireapp.6 +appVersion: 4.5.2-wireapp.8 diff --git a/charts/coturn/README.md b/charts/coturn/README.md index 56124a6194b..85f8765d78c 100644 --- a/charts/coturn/README.md +++ b/charts/coturn/README.md @@ -1,10 +1,13 @@ **Warning**: this chart is currently considered alpha. Use at your own risk! -This chart deploys [coturn](https://github.com/coturn/coturn), a STUN and TURN -server. +This chart deploys Wire's fork of [coturn](https://github.com/coturn/coturn), +a STUN and TURN server, with some additional features developed by Wire (see +[here](https://github.com/wireapp/coturn/tree/wireapp)) to support our calling +services. -You need to supply the zrestSecret at key `secrets.zrestSecret`. Make sure this -matches `secrets.turn.secret` of the brig chart. +You need to supply a list of one or more zrest secrets at the key +`secrets.zrestSecrets`. The secret provided to the brig chart in +`secrets.turn.secret` must be included in this list. Note that coturn pods are deployed with `hostNetwork: true`, as they need to listen on a wide range of UDP ports. Additionally, some TCP ports need to be @@ -16,3 +19,7 @@ therefore recommended to run coturn on a separate Kubernetes cluster from the rest of the Wire services. Further details may be found in Wire's documentation for Restund, another TURN implementation, on [this](https://docs.wire.com/understand/restund.html#network) page. + +coturn can optionally be configured to expose a TLS control port. The TLS +private key and certificates should be provided in a `Secret` whose name is +given in `tls.secretRef`. diff --git a/charts/coturn/templates/configmap-coturn-conf-template.yaml b/charts/coturn/templates/configmap-coturn-conf-template.yaml index 6d10d7718bb..76e0f95605d 100644 --- a/charts/coturn/templates/configmap-coturn-conf-template.yaml +++ b/charts/coturn/templates/configmap-coturn-conf-template.yaml @@ -7,11 +7,18 @@ metadata: data: coturn.conf.template: | - ## disable (d)tls control plane; don't permit relaying tcp connections. - no-tls + ## disable dtls control plane; don't permit relaying tcp connections. no-dtls no-tcp-relay + ## tls handling + {{- if .Values.tls.enabled }} + cert=/secrets-tls/tls.crt + pkey=/secrets-tls/tls.key + {{- else }} + no-tls + {{- end }} + ## don't turn on coturn's cli. no-cli @@ -28,7 +35,6 @@ data: ## prometheus metrics prometheus-ip=__COTURN_POD_IP__ prometheus-port={{ .Values.coturnMetricsListenPort }} - prometheus-no-username-labels ## logs log-file=stdout diff --git a/charts/coturn/templates/secret.yaml b/charts/coturn/templates/secret.yaml index f85a13153ac..af6a8563cf3 100644 --- a/charts/coturn/templates/secret.yaml +++ b/charts/coturn/templates/secret.yaml @@ -1,5 +1,5 @@ {{- if or (not .Values.secrets) (not .Values.secrets.zrestSecrets) }} -{{- fail "Secrets are not defined" }} +{{- fail "TURN authentication secrets are not defined in .Values.secrets.zrestSecrets" }} {{- else if eq (len .Values.secrets.zrestSecrets) 0 }} {{- fail "At least one authentication secret must be defined" }} {{- else }} diff --git a/charts/coturn/templates/service.yaml b/charts/coturn/templates/service.yaml index b671439da76..a5f8f15bd5c 100644 --- a/charts/coturn/templates/service.yaml +++ b/charts/coturn/templates/service.yaml @@ -17,5 +17,10 @@ spec: port: {{ .Values.coturnTurnListenPort }} targetPort: coturn-udp protocol: UDP + {{- if .Values.tls.enabled }} + - name: coturn-tls + port: {{ .Values.coturnTurnTlsListenPort }} + targetPort: coturn-tls + {{- end }} selector: {{- include "coturn.selectorLabels" . | nindent 4 }} diff --git a/charts/coturn/templates/statefulset.yaml b/charts/coturn/templates/statefulset.yaml index a2a4fe11980..daf90ace402 100644 --- a/charts/coturn/templates/statefulset.yaml +++ b/charts/coturn/templates/statefulset.yaml @@ -28,6 +28,10 @@ spec: spec: securityContext: {{- toYaml .Values.podSecurityContext | nindent 8 }} + {{- if .Values.tls.enabled }} + # Needed for automatic certificate reload handling + shareProcessNamespace: true + {{- end }} hostNetwork: true serviceAccountName: {{ include "coturn.fullname" . }} volumes: @@ -41,6 +45,11 @@ spec: - name: secrets secret: secretName: coturn + {{- if .Values.tls.enabled }} + - name: secrets-tls + secret: + secretName: {{ .Values.tls.secretRef }} + {{- end }} initContainers: - name: get-external-ip image: bitnami/kubectl:1.19.7 @@ -84,6 +93,11 @@ spec: - name: secrets mountPath: /secrets/ readOnly: true + {{- if .Values.tls.enabled }} + - name: secrets-tls + mountPath: /secrets-tls/ + readOnly: true + {{- end }} command: - /bin/sh - -c @@ -110,6 +124,11 @@ spec: - name: coturn-udp containerPort: {{ .Values.coturnTurnListenPort }} protocol: UDP + {{- if .Values.tls.enabled }} + - name: coturn-tls + containerPort: {{ .Values.coturnTurnTlsListenPort }} + protocol: TCP + {{- end }} - name: status-http containerPort: {{ .Values.coturnMetricsListenPort }} protocol: TCP @@ -126,6 +145,24 @@ spec: resources: {{- toYaml .Values.resources | nindent 12 }} + + {{- if .Values.tls.enabled }} + - name: {{ .Chart.Name }}-cert-reloader + image: "{{ .Values.tls.reloaderImage.repository }}:{{ .Values.tls.reloaderImage.tag }}" + imagePullPolicy: {{ .Values.tls.reloaderImage.pullPolicy }} + env: + - name: CONFIG_DIR + value: /secrets-tls + - name: PROCESS_NAME + value: /usr/bin/turnserver + - name: RELOAD_SIGNAL + value: SIGUSR2 + volumeMounts: + - name: secrets-tls + mountPath: /secrets-tls/ + readOnly: true + {{- end }} + {{- if .Values.coturnGracefulTermination }} terminationGracePeriodSeconds: {{ .Values.coturnGracePeriodSeconds }} {{- end }} diff --git a/charts/coturn/values.yaml b/charts/coturn/values.yaml index 964b816a4b0..1504bbcdcad 100644 --- a/charts/coturn/values.yaml +++ b/charts/coturn/values.yaml @@ -24,6 +24,17 @@ securityContext: coturnTurnListenPort: 3478 coturnMetricsListenPort: 9641 +coturnTurnTlsListenPort: 5349 + +tls: + enabled: false + secretRef: + reloaderImage: + # container image containing https://github.com/Pluies/config-reloader-sidecar + # for handling runtime certificate reloads. + repository: quay.io/wire/config-reloader-sidecar + pullPolicy: IfNotPresent + tag: 1aa6cbbf2ce3a5182ec47e3579bbcb8f47e22fdc # This chart optionally supports waiting for traffic to drain from coturn # before pods are terminated. Warning: coturn does not have any way to steer diff --git a/charts/elasticsearch-ephemeral/values.yaml b/charts/elasticsearch-ephemeral/values.yaml index 13100a3b40b..9d0c5cae8ab 100644 --- a/charts/elasticsearch-ephemeral/values.yaml +++ b/charts/elasticsearch-ephemeral/values.yaml @@ -1,6 +1,7 @@ image: repository: elasticsearch - tag: 6.8.18 + # Keep this aligned with the Elastic Search version in wire-server-deploy! + tag: 6.8.23 service: httpPort: 9200 diff --git a/charts/galley/templates/secret.yaml b/charts/galley/templates/aws-secret.yaml similarity index 95% rename from charts/galley/templates/secret.yaml rename to charts/galley/templates/aws-secret.yaml index 449be3903f9..a72862ee5a4 100644 --- a/charts/galley/templates/secret.yaml +++ b/charts/galley/templates/aws-secret.yaml @@ -2,7 +2,7 @@ apiVersion: v1 kind: Secret metadata: - name: galley + name: galley-aws labels: app: galley chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" diff --git a/charts/galley/templates/configmap.yaml b/charts/galley/templates/configmap.yaml index 664868ef21c..9a877ab26ff 100644 --- a/charts/galley/templates/configmap.yaml +++ b/charts/galley/templates/configmap.yaml @@ -59,6 +59,13 @@ data: enableIndexedBillingTeamMembers: {{ .settings.enableIndexedBillingTeamMembers }} {{- end }} federationDomain: {{ .settings.federationDomain }} + {{- if $.Values.secrets.mlsPrivateKeys }} + mlsPrivateKeyPaths: + {{- if $.Values.secrets.mlsPrivateKeys.removal.ed25519 }} + removal: + ed25519: "/etc/wire/galley/secrets/removal_ed25519.pem" + {{- end }} + {{- end -}} {{- if .settings.featureFlags }} featureFlags: sso: {{ .settings.featureFlags.sso }} diff --git a/charts/galley/templates/deployment.yaml b/charts/galley/templates/deployment.yaml index ca23d999674..c5f1b9ee258 100644 --- a/charts/galley/templates/deployment.yaml +++ b/charts/galley/templates/deployment.yaml @@ -25,13 +25,17 @@ spec: annotations: # An annotation of the configmap checksum ensures changes to the configmap cause a redeployment upon `helm upgrade` checksum/configmap: {{ include (print .Template.BasePath "/configmap.yaml") . | sha256sum }} - checksum/secret: {{ include (print .Template.BasePath "/secret.yaml") . | sha256sum }} + checksum/aws-secret: {{ include (print .Template.BasePath "/aws-secret.yaml") . | sha256sum }} + checksum/mls-secret: {{ include (print .Template.BasePath "/mls-secret.yaml") . | sha256sum }} spec: serviceAccountName: {{ .Values.serviceAccount.name }} volumes: - name: "galley-config" configMap: name: "galley" + - name: "galley-secrets" + secret: + secretName: "galley-mls" containers: - name: galley image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" @@ -39,17 +43,19 @@ spec: volumeMounts: - name: "galley-config" mountPath: "/etc/wire/galley/conf" + - name: "galley-secrets" + mountPath: "/etc/wire/galley/secrets" env: {{- if hasKey .Values.secrets "awsKeyId" }} - name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: - name: galley + name: galley-aws key: awsKeyId - name: AWS_SECRET_ACCESS_KEY valueFrom: secretKeyRef: - name: galley + name: galley-aws key: awsSecretKey {{- end }} - name: AWS_REGION diff --git a/charts/galley/templates/mls-secret.yaml b/charts/galley/templates/mls-secret.yaml new file mode 100644 index 00000000000..1b77c325a9a --- /dev/null +++ b/charts/galley/templates/mls-secret.yaml @@ -0,0 +1,16 @@ +apiVersion: v1 +kind: Secret +metadata: + name: galley-mls + labels: + app: galley + chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" + release: "{{ .Release.Name }}" + heritage: "{{ .Release.Service }}" +type: Opaque +data: + {{- if .Values.secrets.mlsPrivateKeys }} + {{- if .Values.secrets.mlsPrivateKeys.removal.ed25519 }} + removal_ed25519.pem: {{ .Values.secrets.mlsPrivateKeys.removal.ed25519 | b64enc | quote }} + {{- end -}} + {{- end -}} diff --git a/charts/galley/templates/tests/galley-integration.yaml b/charts/galley/templates/tests/galley-integration.yaml index a688764dfe4..883f57e2d9d 100644 --- a/charts/galley/templates/tests/galley-integration.yaml +++ b/charts/galley/templates/tests/galley-integration.yaml @@ -36,6 +36,9 @@ spec: - name: "galley-integration-secrets" configMap: name: "galley-integration-secrets" + - name: "galley-secrets" + secret: + secretName: "galley-mls" containers: - name: integration image: "{{ .Values.image.repository }}-integration:{{ .Values.image.tag }}" @@ -47,6 +50,8 @@ spec: - name: "galley-integration-secrets" # TODO: see corresp. TODO in brig. mountPath: "/etc/wire/integration-secrets" + - name: "galley-secrets" + mountPath: "/etc/wire/galley/secrets" env: # these dummy values are necessary for Amazonka's "Discover" - name: AWS_ACCESS_KEY_ID diff --git a/charts/nginz/values.yaml b/charts/nginz/values.yaml index 66952a5d231..3cd3e6d2b5a 100644 --- a/charts/nginz/values.yaml +++ b/charts/nginz/values.yaml @@ -476,6 +476,9 @@ nginx_conf: - path: /mls/messages envs: - all + - path: /nonce/clients + envs: + - all gundeck: - path: /push/api-docs$ envs: diff --git a/deploy/services-demo/conf/brig.demo-docker.yaml b/deploy/services-demo/conf/brig.demo-docker.yaml index ff9c1417d13..a0ca16fadc3 100644 --- a/deploy/services-demo/conf/brig.demo-docker.yaml +++ b/deploy/services-demo/conf/brig.demo-docker.yaml @@ -115,6 +115,7 @@ optSettings: setMaxConvSize: 128 setEmailVisibility: visible_to_self setFederationDomain: example.com + setNonceTtlSecs: 300 # 5 minutes logLevel: Debug logNetStrings: false diff --git a/deploy/services-demo/conf/brig.demo.yaml b/deploy/services-demo/conf/brig.demo.yaml index 409ad813cec..437de1417f6 100644 --- a/deploy/services-demo/conf/brig.demo.yaml +++ b/deploy/services-demo/conf/brig.demo.yaml @@ -116,6 +116,7 @@ optSettings: setMaxConvSize: 128 setEmailVisibility: visible_to_self setFederationDomain: example.com + setNonceTtlSecs: 300 # 5 minutes logLevel: Debug logNetStrings: false diff --git a/deploy/services-demo/conf/ed25519.pem b/deploy/services-demo/conf/ed25519.pem new file mode 100644 index 00000000000..4e87cf573cf --- /dev/null +++ b/deploy/services-demo/conf/ed25519.pem @@ -0,0 +1,3 @@ +-----BEGIN PRIVATE KEY----- +MC4CAQAwBQYDK2VwBCIEIAocCDXsKIAjb65gOUn5vEF0RIKnVJkKR4ebQzuZ709c +-----END PRIVATE KEY----- diff --git a/deploy/services-demo/conf/galley.demo.yaml b/deploy/services-demo/conf/galley.demo.yaml index 628e0d22b22..9e9150ce5ca 100644 --- a/deploy/services-demo/conf/galley.demo.yaml +++ b/deploy/services-demo/conf/galley.demo.yaml @@ -28,6 +28,9 @@ settings: conversationCodeURI: https://127.0.0.1/conversation-join/ concurrentDeletionEvents: 1024 deleteConvThrottleMillis: 0 + mlsPrivateKeyPaths: + removal: + ed25519: conf/ed25519.pem featureFlags: # see #RefConfigOptions in `/docs/reference` sso: disabled-by-default diff --git a/deploy/services-demo/conf/nginz/nginx.conf b/deploy/services-demo/conf/nginz/nginx.conf index 1195f450954..36ae5cd9023 100644 --- a/deploy/services-demo/conf/nginz/nginx.conf +++ b/deploy/services-demo/conf/nginz/nginx.conf @@ -286,6 +286,11 @@ http { proxy_pass http://brig; } + location /nonce/clients { + include common_response_with_zauth.conf; + proxy_pass http://brig; + } + # Cargohold Endpoints rewrite ^/api-docs/assets /assets/api-docs?base_url=http://127.0.0.1:8080/ break; diff --git a/docs/README.origial.md b/docs/README.origial.md deleted file mode 120000 index 5888983db9f..00000000000 --- a/docs/README.origial.md +++ /dev/null @@ -1 +0,0 @@ -legacy/README.md \ No newline at end of file diff --git a/docs/developer/api-versioning.md b/docs/developer/api-versioning.md deleted file mode 100644 index 741d4dcc232..00000000000 --- a/docs/developer/api-versioning.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/developer/api-versioning.md) diff --git a/docs/developer/architecture/wire-arch-2.png b/docs/developer/architecture/wire-arch-2.png deleted file mode 100644 index 55b2ed6ab4b..00000000000 --- a/docs/developer/architecture/wire-arch-2.png +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/developer/architecture/wire-arch-2.png) diff --git a/docs/developer/architecture/wire-arch-2.xml b/docs/developer/architecture/wire-arch-2.xml deleted file mode 100644 index ace4d60fbcc..00000000000 --- a/docs/developer/architecture/wire-arch-2.xml +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/developer/architecture/wire-arch-2.xml) diff --git a/docs/developer/cassandra-interaction.md b/docs/developer/cassandra-interaction.md deleted file mode 100644 index 63af6207cd7..00000000000 --- a/docs/developer/cassandra-interaction.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/developer/cassandra-interaction.md) diff --git a/docs/developer/changelog.md b/docs/developer/changelog.md deleted file mode 100644 index 508079c863b..00000000000 --- a/docs/developer/changelog.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/developer/changelog.md) diff --git a/docs/developer/convert-to-cabal.md b/docs/developer/convert-to-cabal.md deleted file mode 120000 index 05eb4322d55..00000000000 --- a/docs/developer/convert-to-cabal.md +++ /dev/null @@ -1 +0,0 @@ -../../tools/convert-to-cabal/README.md \ No newline at end of file diff --git a/docs/developer/dependencies.md b/docs/developer/dependencies.md deleted file mode 100644 index 392d1d14c90..00000000000 --- a/docs/developer/dependencies.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/developer/dependencies.md) diff --git a/docs/developer/editor-setup.md b/docs/developer/editor-setup.md deleted file mode 100644 index b3fd30b7ee1..00000000000 --- a/docs/developer/editor-setup.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/developer/editor-setup.md) diff --git a/docs/developer/features.md b/docs/developer/features.md deleted file mode 100644 index fc5fed6c1fd..00000000000 --- a/docs/developer/features.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/developer/features.md) diff --git a/docs/developer/federation-api-conventions.md b/docs/developer/federation-api-conventions.md deleted file mode 100644 index 257a6bae26f..00000000000 --- a/docs/developer/federation-api-conventions.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/developer/federation-api-conventions.md) diff --git a/docs/developer/how-to.md b/docs/developer/how-to.md deleted file mode 100644 index e9179580fbe..00000000000 --- a/docs/developer/how-to.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/developer/how-to.md) diff --git a/docs/developer/linting.md b/docs/developer/linting.md deleted file mode 100644 index 49d740d3c47..00000000000 --- a/docs/developer/linting.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/developer/linting.md) diff --git a/docs/developer/processes.md b/docs/developer/processes.md deleted file mode 100644 index bda71f73e02..00000000000 --- a/docs/developer/processes.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/developer/processes.md) diff --git a/docs/developer/scim/storage.md b/docs/developer/scim/storage.md deleted file mode 100644 index a070cdb42f5..00000000000 --- a/docs/developer/scim/storage.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/developer/scim/storage.md) diff --git a/docs/developer/servant.md b/docs/developer/servant.md deleted file mode 100644 index de2b8599d52..00000000000 --- a/docs/developer/servant.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/developer/servant.md) diff --git a/docs/developer/stern.md b/docs/developer/stern.md deleted file mode 120000 index f1a37ac1b5f..00000000000 --- a/docs/developer/stern.md +++ /dev/null @@ -1 +0,0 @@ -../../tools/stern/README.md \ No newline at end of file diff --git a/docs/developer/testing.md b/docs/developer/testing.md deleted file mode 100644 index ae5e8568bda..00000000000 --- a/docs/developer/testing.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/developer/testing.md) diff --git a/docs/diagrams/README.md b/docs/diagrams/README.md index e82e3fe35ab..ed60a1b3ff4 100644 --- a/docs/diagrams/README.md +++ b/docs/diagrams/README.md @@ -1,6 +1,6 @@ # Diagrams with Kroki / Mermaid -This is a "diagrams playground" folder and you don't have to use this. Instead, you can create diagrams in a .rst file inside wire-docs/src with this example kroki/mermaid syntax and use the normal development setup described in wire-docs/README.md to get live previews: +This is a "diagrams playground" folder and you don't have to use this. Instead, you can create diagrams in a .rst file inside ..//src with this example kroki/mermaid syntax and use the normal development setup described in ../README.md to get live previews: ```rst .. kroki:: diff --git a/docs/legacy/README.md b/docs/legacy/README.md deleted file mode 100644 index 650b2b0baae..00000000000 --- a/docs/legacy/README.md +++ /dev/null @@ -1,63 +0,0 @@ -# Reference documentation - -What you need to know as a user of the Wire backend: concepts, features, and API. We strive to keep these up to date. - -## Users - -User lifecycle: - -* [User registration](reference/user/registration.md) `{#RefRegistration}` -* [User activation](reference/user/activation.md) `{#RefActivation}` - -User profiles and metadata: - -* [Connections between users](reference/user/connection.md) `{#RefConnection}` -* [Rich info](reference/user/rich-info.md) `{#RefRichInfo}` - -TODO. - -## Teams - -TODO. - -## Messaging - -TODO. - -## Single sign-on - -TODO. - -## SCIM provisioning - -We have support for provisioning users via SCIM ([RFC 7664][], [RFC 7643][]). It's in the beta stage. - -[RFC 7664]: https://tools.ietf.org/html/rfc7664 -[RFC 7643]: https://tools.ietf.org/html/rfc7643 - -* [Using the SCIM API with curl](reference/provisioning/scim-via-curl.md) `{#RefScimViaCurl}` -* [Authentication via SCIM tokens](reference/provisioning/scim-token.md) `{#RefScimToken}` - -# Developer documentation - -Internal documentation detailing what you need to know as a Wire backend developer. All of these documents can and should be referenced in the code. - -If you're not a member of the Wire backend team, you might still find these documents useful, but keep in mind that they are a work in progress. - -* [Development setup](developer/dependencies.md) `{#DevDeps}` -* [Editor setup](developer/editor-setup.md) `{#DevEditor}` -* [Storing SCIM-related data](developer/scim/storage.md) `{#DevScimStorage}` -* TODO - -## Cassandra - -We use [Cassandra](http://cassandra.apache.org/) as the primary data store. It is scalable, has very fast reads and writes, and is conceptually simple (or at least simpler than SQL databases). - -Some helpful links: - -* [Query syntax](https://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlReferenceTOC.html) - -* How deletes work in Cassandra: - - - [Understanding Deletes](https://medium.com/@foundev/domain-modeling-around-deletes-1cc9b6da0d24) - - [Cassandra Compaction and Tombstone Behavior](http://engblog.polyvore.com/2015/03/cassandra-compaction-and-tombstone.html) diff --git a/docs/legacy/reference/provisioning/scim-via-curl.md b/docs/legacy/reference/provisioning/scim-via-curl.md deleted file mode 100644 index aaa2a9eea56..00000000000 --- a/docs/legacy/reference/provisioning/scim-via-curl.md +++ /dev/null @@ -1 +0,0 @@ -# This page [has gone here](https://docs.wire.com/understand/single-sign-on/main.html#using-scim-via-curl). diff --git a/docs/legacy/reference/provisioning/wire_scim_token.py b/docs/legacy/reference/provisioning/wire_scim_token.py deleted file mode 100755 index 2823a7bbca8..00000000000 --- a/docs/legacy/reference/provisioning/wire_scim_token.py +++ /dev/null @@ -1,107 +0,0 @@ -#!/usr/bin/env python3 -# -# This file is part of the Wire Server implementation. -# -# Copyright (C) 2020 Wire Swiss GmbH -# -# This program is free software: you can redistribute it and/or modify it under -# the terms of the GNU Affero General Public License as published by the Free -# Software Foundation, either version 3 of the License, or (at your option) any -# later version. -# -# This program is distributed in the hope that it will be useful, but WITHOUT -# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS -# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more -# details. -# -# You should have received a copy of the GNU Affero General Public License along -# with this program. If not, see . - -from __future__ import print_function - -# NOTE: This python script requires the "requests" library to be installed. - -# Change this if you are running your own instance of Wire. -BACKEND_URL='https://prod-nginz-https.wire.com' - -import sys -import getpass -from requests import Request, Session -import requests -import json -import datetime - -session = None - -def init_session(): - global session - session = Session() - session.headers.update({'User-Agent': "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36"}) - -def post(url): - return Request('POST', url) - -def get(url): - return Request('GET', url) - -def set_bearer_token(request, token): - request.headers['Authorization'] = 'Bearer ' + token - -def backend(path): - return BACKEND_URL + path - -def has_json_content(response): - content_type = response.headers.get('Content-Type') - if content_type is not None: - return (content_type.startswith('application/json') - or content_type == 'application/scim+json;charset=utf-8') - else: - return False - -def send_request(session, request): - response = session.send(session.prepare_request(request)) - if 200 <= response.status_code and response.status_code < 300: - if has_json_content(response): - return response.json() - else: - return response - else: - print(f"Request failed {request.url}", file=sys.stderr) - if has_json_content(response): - tpl = response, response.json() - else: - tpl = response, response.content - print(tpl, file=sys.stderr) - exit(1) - -def create_bearer(email, password): - r = post(backend('/login?persist=false')) - r.headers['Accept'] = 'application/json' - r.json = {'email': email, 'password': password} - return send_request(session, r) - -def create_scim_token(admin_password, token): - r = post(backend('/scim/auth-tokens')) - set_bearer_token(r, token) - r.json = {'description': 'token generated at ' + datetime.datetime.now().isoformat(), - 'password': admin_password - } - return send_request(session, r) - -def exit_fail(msg): - print(msg, file=sys.stderr) - exit(1) - -def main(): - init_session() - print('This script generates a token that authorizes calls to Wire\'s SCIM endpoints.\n') - print('Please enter the login credentials of a user that has role "owner" or "admin".') - ADMIN_EMAIL=input("Email: ") or exit_fail('Please provide an email.') - ADMIN_PASSWORD=getpass.getpass('Password: ') or exit_fail('Please provide password.') - bearer_token = create_bearer(ADMIN_EMAIL, ADMIN_PASSWORD) - scim_token = create_scim_token(ADMIN_PASSWORD, bearer_token['access_token']) - print('Wire SCIM Token: ' + scim_token['token']) - print('The token will be valid until you generate a new token for this user.') - -if __name__ == '__main__': - main() diff --git a/docs/reference/cassandra-schema.cql b/docs/reference/cassandra-schema.cql deleted file mode 120000 index 77e9ef36a84..00000000000 --- a/docs/reference/cassandra-schema.cql +++ /dev/null @@ -1 +0,0 @@ -../../cassandra-schema.cql \ No newline at end of file diff --git a/docs/reference/config-options.md b/docs/reference/config-options.md deleted file mode 100644 index 2a6e342a549..00000000000 --- a/docs/reference/config-options.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/reference/config-options.md) diff --git a/docs/reference/conversation.md b/docs/reference/conversation.md deleted file mode 100644 index 23342da490f..00000000000 --- a/docs/reference/conversation.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/reference/conversation.md) diff --git a/docs/reference/elastic-search.md b/docs/reference/elastic-search.md deleted file mode 100644 index fbdf61469b9..00000000000 --- a/docs/reference/elastic-search.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/reference/elastic-search.md) diff --git a/docs/reference/elasticsearch-migration-2021-02-16.md b/docs/reference/elasticsearch-migration-2021-02-16.md deleted file mode 100644 index 8e17b0b9fdf..00000000000 --- a/docs/reference/elasticsearch-migration-2021-02-16.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/reference/elasticsearch-migration-2021-02-16.md) diff --git a/docs/reference/make-docker-and-qemu.md b/docs/reference/make-docker-and-qemu.md deleted file mode 100644 index 3c15d68bec0..00000000000 --- a/docs/reference/make-docker-and-qemu.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/reference/make-docker-and-qemu.md) diff --git a/docs/reference/provisioning/scim-token.md b/docs/reference/provisioning/scim-token.md deleted file mode 100644 index b0cf7e2ecbd..00000000000 --- a/docs/reference/provisioning/scim-token.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../../legacy/reference/provisioning/scim-token.md) diff --git a/docs/reference/provisioning/scim-via-curl.md b/docs/reference/provisioning/scim-via-curl.md deleted file mode 100644 index 6eeebaa5438..00000000000 --- a/docs/reference/provisioning/scim-via-curl.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../../legacy/reference/provisioning/scim-via-curl.md) diff --git a/docs/reference/provisioning/wire_scim_token.py b/docs/reference/provisioning/wire_scim_token.py deleted file mode 120000 index 7be5401a613..00000000000 --- a/docs/reference/provisioning/wire_scim_token.py +++ /dev/null @@ -1 +0,0 @@ -../../legacy/reference/provisioning/wire_scim_token.py \ No newline at end of file diff --git a/docs/reference/spar-braindump.md b/docs/reference/spar-braindump.md deleted file mode 100644 index d3e2b1f3ffd..00000000000 --- a/docs/reference/spar-braindump.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/reference/spar-braindump.md) diff --git a/docs/reference/user/activation.md b/docs/reference/user/activation.md deleted file mode 100644 index 63bf6c00f1c..00000000000 --- a/docs/reference/user/activation.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/reference/user/activation.md) diff --git a/docs/reference/user/connection-transitions.png b/docs/reference/user/connection-transitions.png deleted file mode 100644 index 6722ff359bf..00000000000 --- a/docs/reference/user/connection-transitions.png +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/reference/user/connection-transitions.png) diff --git a/docs/reference/user/connection-transitions.xml b/docs/reference/user/connection-transitions.xml deleted file mode 100644 index b78f17675bd..00000000000 --- a/docs/reference/user/connection-transitions.xml +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/reference/user/connection-transitions.xml) diff --git a/docs/reference/user/connection.md b/docs/reference/user/connection.md deleted file mode 100644 index 08c35e1049d..00000000000 --- a/docs/reference/user/connection.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/reference/user/connection.md) diff --git a/docs/reference/user/connections-flow-1-backend.png b/docs/reference/user/connections-flow-1-backend.png deleted file mode 100644 index e2166d82f99..00000000000 --- a/docs/reference/user/connections-flow-1-backend.png +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/reference/user/connections-flow-1-backend.png) diff --git a/docs/reference/user/registration.md b/docs/reference/user/registration.md deleted file mode 100644 index 1ebf9383d5b..00000000000 --- a/docs/reference/user/registration.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../../legacy/reference/user/registration.md) diff --git a/docs/reference/user/rich-info.md b/docs/reference/user/rich-info.md deleted file mode 100644 index c195e062fe0..00000000000 --- a/docs/reference/user/rich-info.md +++ /dev/null @@ -1 +0,0 @@ -file has moved [here](../legacy/reference/user/rich-info.md) diff --git a/docs/src/conf.py b/docs/src/conf.py index 777df273f2e..cd6e8343d19 100644 --- a/docs/src/conf.py +++ b/docs/src/conf.py @@ -111,3 +111,6 @@ smv_outputdir_format = 'versions/{ref.name}' smv_prefer_remote_refs = True + +# As per https://myst-parser.readthedocs.io/en/latest/syntax/optional.html?highlight=anchor#auto-generated-header-anchors +myst_heading_anchors = 4 diff --git a/docs/legacy/developer/api-versioning.md b/docs/src/developer/developer/api-versioning.md similarity index 99% rename from docs/legacy/developer/api-versioning.md rename to docs/src/developer/developer/api-versioning.md index 1bae9bfe46a..6e1f3d75b31 100644 --- a/docs/legacy/developer/api-versioning.md +++ b/docs/src/developer/developer/api-versioning.md @@ -16,7 +16,7 @@ A backend advertises a set of *supported* API versions, divided into a set of be discovered via the `GET /api-version` endpoint, which returns a JSON object of the form: -```json +``` { "supported": [0, 1, 2, 3, 4], "development": [4], ... diff --git a/docs/legacy/developer/architecture/wire-arch-2.png b/docs/src/developer/developer/architecture/wire-arch-2.png similarity index 100% rename from docs/legacy/developer/architecture/wire-arch-2.png rename to docs/src/developer/developer/architecture/wire-arch-2.png diff --git a/docs/legacy/developer/architecture/wire-arch-2.xml b/docs/src/developer/developer/architecture/wire-arch-2.xml similarity index 100% rename from docs/legacy/developer/architecture/wire-arch-2.xml rename to docs/src/developer/developer/architecture/wire-arch-2.xml diff --git a/docs/src/developer/developer/building.md b/docs/src/developer/developer/building.md new file mode 100644 index 00000000000..2457128bfe4 --- /dev/null +++ b/docs/src/developer/developer/building.md @@ -0,0 +1,84 @@ +# How to build wire-server + +As a prerequisiste install the [nix package manager](https://nixos.org/) and [direnv](https://direnv.net/). + +All following commands expect that you've entered the nix-provided build-environment by running `direnv allow`. + + +1. Create a `.envrc.local` file with these contents + +``` + export COMPILE_NGINX_USING_NIX=1 + export WIRE_BUILD_WITH_CABAL=1 +``` + + and reload the direnv via `direnv reload` + +2. Create a `cabal.project.local`. This file is not included in wire-server because it disables optimization. + + + make cabal.project.local + + + This should be re-run whenver a new local cabal package is added to the cabal project. + +Then the following Makefile targets can be used to compile and test wire-server locally: + + + # to compile all binaries to ./dist run + make + + # to build and install all of galley's executables + make c package=galley + + # also run galley's unit tests + make c package=galley test=1 + + +## Troubleshooting + +### Linker errors while compiling + +Linker errors can occur if the nix-provided build environment (see `nix/` directory) changes. Since cabal is not aware of the changed environment the cached build artifacts in `./dist-newstyle` and `~/.cabal/store/` from previous builds may be invalid causing the linker errors. + +Haskell Language Server stores its build artifacts in `~/.cache/hie-bios` (equivalent to the `./dist-newstyle` directory) which become invalid for the same reason. + +The easiest course of action is to to remove these directories via: + +``` +make full-clean +``` + +## How to run integration tests + +Integration tests require all of the haskell services (brig, galley, cannon, gundeck, proxy, cargohold, spar) to be correctly configured and running, before being able to execute e.g. the `brig-integration` binary. The test for brig also starts nginz, so make sure it has been built before. +These services require most of the deployment dependencies as seen in the architecture diagram to also be available: + +- Required internal dependencies: + - cassandra (with the correct schema) + - elasticsearch (with the correct schema) + - redis +- Required external dependencies are the following configured AWS services (or "fake" replacements providing the same API): + - SES + - SQS + - SNS + - S3 + - DynamoDB +- Required additional software: + - netcat (in order to allow the services being tested to talk to the dependencies above) + +Setting up these real, but in-memory internal and "fake" external dependencies is done easiest using [`docker-compose`](https://docs.docker.com/compose/install/). Run the following in a separate terminal (it will block that terminal, C-c to shut all these docker images down again): + +``` +deploy/dockerephemeral/run.sh +``` + +After all containers are up you can use these Makefile targets to run the tests locally: + +``` +# build and run galley's integration tests +make ci package=galley + +# run galley's integration tests that match a pattern +TASTY_PATTERN="/MLS/" make ci package=galley +``` diff --git a/docs/legacy/developer/cassandra-interaction.md b/docs/src/developer/developer/cassandra-interaction.md similarity index 100% rename from docs/legacy/developer/cassandra-interaction.md rename to docs/src/developer/developer/cassandra-interaction.md diff --git a/docs/legacy/developer/changelog.md b/docs/src/developer/developer/changelog.md similarity index 99% rename from docs/legacy/developer/changelog.md rename to docs/src/developer/developer/changelog.md index 886df37964a..6d1640ed85b 100644 --- a/docs/legacy/developer/changelog.md +++ b/docs/src/developer/developer/changelog.md @@ -1,3 +1,5 @@ +# Changelog + The wire-server repo has a process for changelog editing that prevents merge conflicts and enforces a consistent structure to the release notes. diff --git a/docs/legacy/developer/dependencies.md b/docs/src/developer/developer/dependencies.md similarity index 87% rename from docs/legacy/developer/dependencies.md rename to docs/src/developer/developer/dependencies.md index 27322255d0c..15896f6d75d 100644 --- a/docs/legacy/developer/dependencies.md +++ b/docs/src/developer/developer/dependencies.md @@ -1,4 +1,6 @@ -# Dependencies {#DevDeps} +# Dependencies + +Reference: {#DevDeps} This page documents how to install necessary dependencies to work with the wire-server code base. @@ -78,11 +80,11 @@ If `openssl-dev` does not work for you, try `libssl-dev`. ### Arch: -``` -# You might also need 'sudo pacman -S base-devel' if you haven't -# installed the base-devel group already. -sudo pacman -S geoip snappy icu openssl ncurses-compat-libs -``` + ``` + # You might also need 'sudo pacman -S base-devel' if you haven't + # installed the base-devel group already. + sudo pacman -S geoip snappy icu openssl ncurses-compat-libs + ``` ### macOS: @@ -112,7 +114,7 @@ sudo installer -pkg /Library/Developer/CommandLineTools/Packages/macOS_SDK_heade Please refer to [Stack's installation instructions](https://docs.haskellstack.org/en/stable/README/#how-to-install). -When you're done, ensure `stack --version` is the same as `STACK_VERSION` in [`build/ubuntu/Dockerfile.prebuilder`](../../build/ubuntu/Dockerfile.prebuilder). +When you're done, ensure `stack --version` is the same as `STACK_VERSION` in [`build/ubuntu/Dockerfile.prebuilder`](https://github.com/wireapp/wire-server/blob/develop/build/ubuntu/Dockerfile.prebuilder). If you have to, you can downgrade stack with this command: @@ -129,11 +131,11 @@ sudo apt install haskell-stack -y ### Generic -```bash -curl -sSL https://get.haskellstack.org/ | sh -# or -wget -qO- https://get.haskellstack.org/ | sh -``` + ```bash + curl -sSL https://get.haskellstack.org/ | sh + # or + wget -qO- https://get.haskellstack.org/ | sh + ``` ## Rust @@ -168,18 +170,18 @@ dpkg -i target/release/cryptobox*.deb ``` ### Generic -```bash -export TARGET_LIB="$HOME/.wire-dev/lib" -export TARGET_INCLUDE="$HOME/.wire-dev/include" -mkdir -p "$TARGET_LIB" -mkdir -p "$TARGET_INCLUDE" -git clone https://github.com/wireapp/cryptobox-c && cd cryptobox-c -make install - -# Add cryptobox-c to ldconfig -sudo bash -c "echo \"${TARGET_LIB}\" > /etc/ld.so.conf.d/cryptobox.conf" -sudo ldconfig -``` + ```bash + export TARGET_LIB="$HOME/.wire-dev/lib" + export TARGET_INCLUDE="$HOME/.wire-dev/include" + mkdir -p "$TARGET_LIB" + mkdir -p "$TARGET_INCLUDE" + git clone https://github.com/wireapp/cryptobox-c && cd cryptobox-c + make install + + # Add cryptobox-c to ldconfig + sudo bash -c "echo \"${TARGET_LIB}\" > /etc/ld.so.conf.d/cryptobox.conf" + sudo ldconfig + ``` Make sure stack knows where to find it. In `~/.stack/config.yaml` add: @@ -228,22 +230,22 @@ Requirements: ### Telepresence example usage: -``` -# terminal 1 -telepresence --namespace "$NAMESPACE" --also-proxy cassandra-ephemeral -``` + ``` + # terminal 1 + telepresence --namespace "$NAMESPACE" --also-proxy cassandra-ephemeral + ``` -``` -# terminal 2 -curl http://elasticsearch-ephemeral:9200 -``` + ``` + # terminal 2 + curl http://elasticsearch-ephemeral:9200 + ``` ### Telepresence example usage 2: -``` -# just one terminal -telepresence --namespace "$NAMESPACE" --also-proxy cassandra-ephemeral --run bash -c "curl http://elasticsearch-ephemeral:9200" -``` + ``` + # just one terminal + telepresence --namespace "$NAMESPACE" --also-proxy cassandra-ephemeral --run bash -c "curl http://elasticsearch-ephemeral:9200" + ``` ### Telepresence usage discussion: diff --git a/docs/legacy/developer/editor-setup.md b/docs/src/developer/developer/editor-setup.md similarity index 98% rename from docs/legacy/developer/editor-setup.md rename to docs/src/developer/developer/editor-setup.md index 4e030f88b08..7b39f8a463f 100644 --- a/docs/legacy/developer/editor-setup.md +++ b/docs/src/developer/developer/editor-setup.md @@ -1,4 +1,6 @@ -# Editor setup {#DevEditor} +# Editor setup + +Reference: {#DevEditor} This page provides tips for setting up editors to work with the Wire codebase. @@ -61,7 +63,7 @@ Install the [projectile][] package for Emacs and do `M-x projectile-add-known-pr To use HLS bundled in direnv setup, here is a sample `.dir-locals.el` that can be put in the root directory of the project: -```el +``` ((haskell-mode . ((haskell-completion-backend . lsp) (lsp-haskell-server-path . "/home/haskeller/code/wire-server/hack/bin/nix-hls.sh") ))) @@ -114,4 +116,4 @@ Setup steps: An alternative way to make these dependencies accessible to VSCode is to start it in the `direnv` environment. I.e. from a shell that's current working directory is in the project. The drawbacks of this approach are -that it only works locally (not on a remote connection) and one VSCode process needs to be started per project. \ No newline at end of file +that it only works locally (not on a remote connection) and one VSCode process needs to be started per project. diff --git a/docs/legacy/developer/features.md b/docs/src/developer/developer/features.md similarity index 99% rename from docs/legacy/developer/features.md rename to docs/src/developer/developer/features.md index a07f505d4d3..d560b441626 100644 --- a/docs/legacy/developer/features.md +++ b/docs/src/developer/developer/features.md @@ -1,4 +1,4 @@ -## Features +# Features Wire has multiple features (or feature flags) that affect the behaviour of backend and clients, e.g. `sso`, `searchVisibility`, `digitalSignatures`, diff --git a/docs/legacy/developer/federation-api-conventions.md b/docs/src/developer/developer/federation-api-conventions.md similarity index 91% rename from docs/legacy/developer/federation-api-conventions.md rename to docs/src/developer/developer/federation-api-conventions.md index 84b73a8a6c0..612a4eb67ed 100644 --- a/docs/legacy/developer/federation-api-conventions.md +++ b/docs/src/developer/developer/federation-api-conventions.md @@ -1,5 +1,3 @@ - - # Federation API Conventions - All endpoints must start with `/federation/` diff --git a/docs/legacy/developer/how-to.md b/docs/src/developer/developer/how-to.md similarity index 98% rename from docs/legacy/developer/how-to.md rename to docs/src/developer/developer/how-to.md index aa40ccdd2ae..87c60e278c5 100644 --- a/docs/legacy/developer/how-to.md +++ b/docs/src/developer/developer/how-to.md @@ -9,7 +9,7 @@ Terminal 1: Terminal 2: * Compile all services: `make services` - * Note that you have to [import the public signing keys for nginx](../../services/nginz/README.md#common-problems-while-compiling) to be able to build nginz + * Note that you have to [import the public signing keys for nginx](https://github.com/wireapp/wire-server/blob/develop/services/nginz/README.md#common-problems-while-compiling) to be able to build nginz * Run services including nginz: `export INTEGRATION_USE_NGINZ=1; ./services/start-services-only.sh` Open your browser at: diff --git a/docs/src/developer/developer/index.rst b/docs/src/developer/developer/index.rst new file mode 100644 index 00000000000..a8fefaa7706 --- /dev/null +++ b/docs/src/developer/developer/index.rst @@ -0,0 +1,10 @@ +Developer +========= + +.. toctree:: + :titlesonly: + :numbered: + :caption: Contents: + :glob: + + ** diff --git a/docs/legacy/developer/linting.md b/docs/src/developer/developer/linting.md similarity index 98% rename from docs/legacy/developer/linting.md rename to docs/src/developer/developer/linting.md index 5dcbf75ab0f..3a1bcbecb6a 100644 --- a/docs/legacy/developer/linting.md +++ b/docs/src/developer/developer/linting.md @@ -1,6 +1,6 @@ # Linting -# HLint +## HLint To run [HLint](https://github.com/ndmitchell/hlint) you need it's binary, e.g. by executing: @@ -21,7 +21,7 @@ To run it on a sub-project: hlint services/federator ``` -# Stan +## Stan To run [Stan](https://github.com/kowainik/stan), you need it's binary compiled by the same GHC version as used in the project. diff --git a/docs/src/developer/developer/pr-guidelines.md b/docs/src/developer/developer/pr-guidelines.md new file mode 100644 index 00000000000..81842fba401 --- /dev/null +++ b/docs/src/developer/developer/pr-guidelines.md @@ -0,0 +1,104 @@ +# PR Guidelines + +This document outlines the steps that need to be taken before merging any PR. In most cases, the only required action is creating a changelog entry (see below). However, when a PR affects the database schema, or the API, or service configuration, extra steps are required, and those are detailed below. + +The recommended way to use this document is to copy the relevant checklists below into the PR description, when appropriate, and make sure they are all checked before the PR is merged. + +## Changelog entries + +Every PR should add a new file in the appropriate subdirectory of `changelog.d`, containing just the text of the corresponding changelog entry. There is no need to explicitly write a PR number, because the `mk-changelog.sh` script (used on release) will add it automatically at the end. The name of the file does not matter, but it should be unique to avoid unnecessary conflicts (e.g. use the branch name). + +It is still possible to write the PR number manually if so desired, which is useful in case the entry is shared by multiple PRs, or if the PR is merged with a merge commit rather than by squashing. In that case, the script would leave the PR number reference intact, as long as it is at the very end of the entry, with no period afterwards, in brackets, and preceded by a `#` symbol (e.g. #2646). + +As long as the PR is merged by squashing, it is also possible to use the pattern `##` to refer to the current PR number. This will be replaced throughout. + +Multiline entries are supported, and are handled correctly by the script. Again, the PR reference should either be omitted or put at the very end. If multiple entries for a single PR are desired, there should be a different file for each of them. + +See `docs/legacy/developer/changelog.md` for more information. + +## Schema migrations + +If a cassandra schema migration has been added then + + - [ ] Run **`make git-add-cassandra-schema`** to update the cassandra schema documentation + +### Incompatible schema migrations and data migrations + +If the PR contains a cassandra *schema* migration which is backwards incompatible, a changelog entry should be added to the release notes. See [notes on Cassandra](https://github.com/wireapp/wire-server/blob/develop/docs/developer/cassandra-interaction.md#cassandra-schema-migrations) for more details on how to implement such schema changes. A similar entry should be added if the PR contains a *data* migration. + + - [ ] Add a changelog entry in `0-release-notes` detailing measures to be taken by instance operators + +## Adding new public endpoints + +When adding new endpoints in the Haskell code in wire-server, correct routing needs to be applied at the nginz level. + +NB: The nginz paths are interpreted as *prefixes*. If you add a new end-point that is identical to an existing one except for the path of the latter being a proper prefix of the former, and if the nginz configuration of the two paths should be the same, nothing needs to be done. Exception: if you see a path like `/self$`, you know it doesn't match `/self/sub/what`. + +The following needs to be done, as part of a PR adding endpoints or changing endpoint paths. + + - [ ] Update nginz config in helm: `charts/nginz/values.yaml` + - [ ] Update nginz config in the demo: `deploy/services-demo/conf/nginz/nginx.conf` + +### Helm configuration + +For internal endpoints for QA access on staging environments, copy a block with `/i/` containing +``` +zauth: false +basic_auth: true +whitelisted_envs: ['staging'] +``` + +For customer support access to an internal endpoint, instead update code in [stern](https://github.com/wireapp/wire-server/tree/develop/tools/stern) as part of your PR. There is no need to add that endpoint to nginz. + +### Demo nginz configuration + +New entris should include `common_response_no_zauth.conf;` for public endpoints without authentication and `common_response_with_zauth.conf;` for regular (authenticated) endpoints. Browse the file to see examples. + +### Example + +If the following endpoints are added to galley: + +``` +GET /new/endpoint +POST /turtles +PUT /turtles//name +``` + +Add to `charts/nginz/values.yaml`, under the `galley` section: + +``` +- path: /new/endpoint +- path: ^/turtles(.*) +``` + +## Adding new configuration flags in wire-server + +If a PR adds new configuration options for say brig, the following files need to be edited: + +* [ ] The parser under `services/brig/src/Brig/Options.hs` +* [ ] The integration test config: `services/brig/brig.integration.yaml` +* [ ] The demo config: `deploy/services-demo/conf/brig.demo.yaml` and `deploy/services-demo/conf/brig.demo.yaml` +* [ ] The charts: `charts/brig/templates/configmap.yaml` +* [ ] The default values: `charts/brig/values.yaml` +* [ ] The values files for CI: `hack/helm_vars/wire-server/values.yaml` +* [ ] The configuration docs: `docs/legacy/reference/config-options.md` + +If any new configuration value is required and has no default, then: + +* [ ] Write a changelog entry in `0-release-notes` advertising the new configuration value +* [ ] Update all the relevant environments + +For wire Cloud, look into all the relevant environments (look for `helm_vars/wire-server/values.yaml.gotmpl` files in cailleach). Ideally, these configuration updates should be merged **before** merging the corresponding wire-server PR. + +### Removing configuration flags + +Remove them with the PR from wire-server `./charts` folder, as charts are linked to code and go hand-in hand. Possibly notify all operators (through `./changelog.d/0-release-notes/`) that some overrides may not have any effect anymore. + +### Renaming configuration flags + +Avoid doing this. If you must, see Removing/adding sections above. But please note that all people who have an installation of wire also may have overridden any of the configuration option you may wish to change the name of. As this is not type checked, it's very error prone and people may find themselves with default configuration values being used instead of their intended configuration settings. Guideline: only rename for good reasons, not for aesthetics; or be prepared to spend a significant +amount on documenting and communication about this change. + +## Changes to developer workflow + +If a PR changes development workflow or dependencies, they should be automated and documented under `docs/developer/`. All efforts should be taken to minimize development setup breakage or slowdown for co-workers. diff --git a/docs/legacy/developer/processes.md b/docs/src/developer/developer/processes.md similarity index 100% rename from docs/legacy/developer/processes.md rename to docs/src/developer/developer/processes.md diff --git a/docs/legacy/developer/scim/storage.md b/docs/src/developer/developer/scim/storage.md similarity index 96% rename from docs/legacy/developer/scim/storage.md rename to docs/src/developer/developer/scim/storage.md index bcf64c64154..6fdfcd5ffe8 100644 --- a/docs/legacy/developer/scim/storage.md +++ b/docs/src/developer/developer/scim/storage.md @@ -1,4 +1,6 @@ -# Storing SCIM-related data {#DevScimStorage} +# Storing SCIM-related data + +Reference: {#DevScimStorage} _Author: Artyom Kazak, Matthias Fischmann_ diff --git a/docs/legacy/developer/servant.md b/docs/src/developer/developer/servant.md similarity index 87% rename from docs/legacy/developer/servant.md rename to docs/src/developer/developer/servant.md index 0b577f7f2cf..4bc2ea858e8 100644 --- a/docs/legacy/developer/servant.md +++ b/docs/src/developer/developer/servant.md @@ -1,16 +1,16 @@ -# Introduction +# Servant We currently use Servant for the public (i.e. client-facing) API in brig, galley and spar, as well as for their federation (i.e. server-to-server) and internal API. Client-facing APIs are defined in `Wire.API.Routes.Public.{Brig,Galley}`. Internal APIs are all over the place at the moment. Federation APIs are in `Wire.API.Federation.API.{Brig,Galley}`. -Our APIs are able to generate Swagger documentation semi-automatically using `servant-swagger2`. The `schema-profunctor` library (see [`README.md`](/libs/schema-profunctor/README.md) in `libs/schema-profunctor`) is used to create "schemas" for the input and output types used in the Servant APIs. A schema contains all the information needed to serialise/deserialise JSON values, as well as the documentation and metadata needed to generate Swagger. +Our APIs are able to generate Swagger documentation semi-automatically using `servant-swagger2`. The `schema-profunctor` library (see [`README.md`](https://github.com/wireapp/wire-server/blob/develop/libs/schema-profunctor/README.md) in `libs/schema-profunctor`) is used to create "schemas" for the input and output types used in the Servant APIs. A schema contains all the information needed to serialise/deserialise JSON values, as well as the documentation and metadata needed to generate Swagger. -# Combinators +## Combinators We have employed a few custom combinators to try to keep HTTP concerns and vocabulary out of the API handlers that actually implement the functionality of the API. -## `ZAuth` +### `ZAuth` This is a family of combinators to handle the headers that nginx adds to requests. We currently have: @@ -19,21 +19,21 @@ This is a family of combinators to handle the headers that nginx adds to request - `ZConn`: extracts the `ConnId` in the `Z-Connection` header. - `ZConversation`: extracts the `ConvId` in the `Z-Conversation` header. -## `MultiVerb` +### `MultiVerb` This is an alternative to `UVerb`, designed to prevent any HTTP-specific information from leaking into the type of the handler. Use this for endpoints that can return multiple responses. -## `CanThrow` +### `CanThrow` This can be used to add an error response to the Swagger documentation. In services that use polysemy for error handling (currently only Galley), it also adds a corresponding error effect to the type of the handler. The argument of `CanThrow` can be of a custom kind, usually a service-specific error kind (such as `GalleyError`, `BrigError`, etc...), but kind `*` can also be used. Note that error types can also be turned into `MultiVerb` responses using the `ErrorResponse` combinator. This is useful for handlers that can return errors as part of their return type, instead of simply throwing them as IO exceptions or using polysemy. If an error is part of `MultiVerb`, there is no need to also report it with `CanThrow`. -## `QualifiedCapture` +### `QualifiedCapture` This is a capture combinator for a path that looks like `/:domain/:value`, where `value` is of some arbitrary type `a`. The value is returned as a value of type `Qualified a`, which can then be used in federation-aware endpoints. -# Error handling +## Error handling Several layers of error handling are involved when serving a request. A handler in service code (e.g. Brig or Galley) can: diff --git a/docs/legacy/developer/testing.md b/docs/src/developer/developer/testing.md similarity index 94% rename from docs/legacy/developer/testing.md rename to docs/src/developer/developer/testing.md index 0e41f8920bf..d36be1808b5 100644 --- a/docs/legacy/developer/testing.md +++ b/docs/src/developer/developer/testing.md @@ -1,4 +1,4 @@ -## Testing the wire-server Haskell code base +# Testing the wire-server Haskell code base Every package in this repository should have one or more directories named `test` or `test*`. Ideally, there are at least unit tests @@ -6,19 +6,19 @@ named `test` or `test*`. Ideally, there are at least unit tests integration tests (eg. `test/integration`) for packages with executables (`/services`). -### General rule +## General rule Write as much pure code as possible. If you write a function that has an effect only in a small part of its implementation, write a pure function instead, and call if from an effectful function. -### Unit tests +## Unit tests All data types that are serialized ([`ToByteString`](https://hackage.haskell.org/package/amazonka-core-1.6.1/docs/Network-AWS-Data-ByteString.html#t:ToByteString), [`ToByteString`](https://hackage.haskell.org/package/aeson/docs/Data-Aeson.html#t:ToJSON), swagger, cassandra, ...) should have roundtrip quickcheck tests like [here](https://github.com/wireapp/wire-server/blob/develop/libs/wire-api/test/unit/Test/Wire/API/Roundtrip/Aeson.hs#L235). All pure functions `f` that do something interesting should have a couple of tests of the form `shouldBe (f ) ` to cover corner cases. If code is refactored, all the effects should be mocked with polysemy or mtl, the old implementation should be moved to the test suite, and there should be quickcheck tests running the new implementation against the old and comparing results ([example](https://github.com/wireapp/wire-server/blob/develop/services/gundeck/test/unit/MockGundeck.hs)). -### Integration tests +## Integration tests - All new rest API end-points need to be called a few times to cover as much of the corner cases as feasible. - We have machinery to set up scaffolding teams and modify context such as server configuration files where needed. Look through the existing code for inspiration. diff --git a/docs/src/developer/index.rst b/docs/src/developer/index.rst new file mode 100644 index 00000000000..b48dbecae0a --- /dev/null +++ b/docs/src/developer/index.rst @@ -0,0 +1,19 @@ +Notes for developers +==================== + +If you are an on-premise operator (administrating your own self-hosted installation of wire-server), you may want to go back to `docs.wire.com `_ and ignore this section of the docs. + +If you are a wire end-user, please check out our `support pages `_. + +What you need to know as a user of the Wire backend: concepts, features, +and API. We want to keep these up to date. They could benefit from some +re-ordering, and they are far from complete, but we hope they will still +help you. + +.. toctree:: + :titlesonly: + :caption: Contents: + :glob: + + developer/index.rst + reference/index.rst diff --git a/docs/legacy/reference/cassandra-schema.cql b/docs/src/developer/reference/cassandra-schema.cql similarity index 100% rename from docs/legacy/reference/cassandra-schema.cql rename to docs/src/developer/reference/cassandra-schema.cql diff --git a/docs/legacy/reference/config-options.md b/docs/src/developer/reference/config-options.md similarity index 96% rename from docs/legacy/reference/config-options.md rename to docs/src/developer/reference/config-options.md index 33d8eb04445..8a74338aa70 100644 --- a/docs/legacy/reference/config-options.md +++ b/docs/src/developer/reference/config-options.md @@ -1,4 +1,6 @@ -# Config Options {#RefConfigOptions} +# Config Options + +Reference: {#RefConfigOptions} Fragment. @@ -26,6 +28,30 @@ Even when the flag is `disabled`, galley will keep writing to the been added in order to deploy new code and backfill data in production. +### MLS private key paths + +The `mlsPrivateKeyPaths` field should contain a mapping from *purposes* and +signature schemes to file paths of corresponding x509 private keys in PEM +format. + +At the moment, the only purpose is `removal`, meaning that the key will be used +to sign external remove proposals. + +For example: + +``` + mlsPrivateKeyPaths: + removal: + ed25519: /etc/secrets/ed25519.pem +``` + +A simple way to generate an ed25519 private key, discarding the corresponding +certificate, is to run the following command: + +``` +openssl req -nodes -newkey ed25519 -keyout ed25519.pem -out /dev/null -subj / +``` + ## Feature flags > Also see [Wire docs](https://docs.wire.com/how-to/install/team-feature-settings.html) where some of the feature flags are documented from an operations point of view. @@ -258,7 +284,7 @@ mls: ``` -This default configuration can be overriden on a per-team basis through the [feature config API](./features.md) +This default configuration can be overriden on a per-team basis through the [feature config API](../developer/features.md) ### Federation Domain diff --git a/docs/legacy/reference/conversation.md b/docs/src/developer/reference/conversation.md similarity index 98% rename from docs/legacy/reference/conversation.md rename to docs/src/developer/reference/conversation.md index 1d4af8569e6..eedbfa7ce0b 100644 --- a/docs/legacy/reference/conversation.md +++ b/docs/src/developer/reference/conversation.md @@ -1,4 +1,6 @@ -# Creating and populating conversations {#RefCreateAndPopulateConvs} +# Creating and populating conversations + +Reference: {#RefCreateAndPopulateConvs} _Author: Matthias Fischmann_ diff --git a/docs/legacy/reference/elastic-search.md b/docs/src/developer/reference/elastic-search.md similarity index 98% rename from docs/legacy/reference/elastic-search.md rename to docs/src/developer/reference/elastic-search.md index 246988ec014..82b061fbfb4 100644 --- a/docs/legacy/reference/elastic-search.md +++ b/docs/src/developer/reference/elastic-search.md @@ -123,7 +123,7 @@ Now you can delete the old index. **NOTE**: There is a bug hidden when using this way. Sometimes a user won't get deleted from the index. Attempts at reproducing this issue in a simpler environment have failed. As a workaround, there is a tool in -[tools/db/find-undead](../../tools/db/find-undead) which can be used to find the +[tools/db/find-undead](https://github.com/wireapp/wire-server/tree/develop/tools/db/find-undead) which can be used to find the undead users right after the migration. If they exist, please run refill the ES documents from cassandra as described [above](#refill-es-documents-from-cassandra) diff --git a/docs/legacy/reference/elasticsearch-migration-2021-02-16.md b/docs/src/developer/reference/elasticsearch-migration-2021-02-16.md similarity index 100% rename from docs/legacy/reference/elasticsearch-migration-2021-02-16.md rename to docs/src/developer/reference/elasticsearch-migration-2021-02-16.md diff --git a/docs/src/developer/reference/index.rst b/docs/src/developer/reference/index.rst new file mode 100644 index 00000000000..1eb9feedbaa --- /dev/null +++ b/docs/src/developer/reference/index.rst @@ -0,0 +1,10 @@ +Reference +========= + +.. toctree:: + :titlesonly: + :numbered: + :caption: Contents: + :glob: + + ** diff --git a/docs/legacy/reference/make-docker-and-qemu.md b/docs/src/developer/reference/make-docker-and-qemu.md similarity index 97% rename from docs/legacy/reference/make-docker-and-qemu.md rename to docs/src/developer/reference/make-docker-and-qemu.md index 3088eda1a0d..a7c10f06cfc 100644 --- a/docs/legacy/reference/make-docker-and-qemu.md +++ b/docs/src/developer/reference/make-docker-and-qemu.md @@ -1,9 +1,9 @@ -# About this document: +# Make docker and QEMU This document is written with the goal of explaining https://github.com/wireapp/wire-server/pull/622 well enough that someone can honestly review it. :) In this document, we're going to rapidly bounce back and forth between GNU make, bash, GNU sed, Docker, and QEMU. -# What does this Makefile do? Why was it created? +## What does this Makefile do? Why was it created? To answer that, we're going to have to go back to Wire-Server, specifically, our integration tests. Integration tests are run locally on all of our machines, in order to ensure that changes we make to the Wire backend do not break currently existing functionality. In order to simulate the components that wire's backend depends on (s3, cassandra, redis, etc..), we use a series of docker images. These docker images are downloaded from dockerhub, are maintained (or not maintained) by outside parties, and are built by those parties. @@ -13,13 +13,13 @@ This Makefile contains rules that allow our Mac users to build all of the docker It builds non-AMD64 images on linux by using QEMU, a system emulator, to allow docker to run images that are not built for the architecture the system is currently running on. This is full system emulation, like many video game engines you're probably familiar with. You know how you have to throw gobs of hardware at a machine, to play a game written for a gaming system 20 years ago? This is similarly slow. To work around this, the Makefile is written in a manner that allows us to build many docker images at once, to take advantage of the fact that most of us have many processor cores lying around doing not-all-much. -# What does this get us? +## What does this get us? To start with, the resulting docker images allow us to tune the JVM settings on cassandra and elasticsearch, resulting in lower memory consumption, and faster integration tests that don't impact our systems as much. Additionally, it allows us more control of the docker images we're depending on, so that another leftpad incident on docker doesn't impact us. As things stand, any of the developers of these docker images can upload a new docker image that does Very Bad Things(tm), and we'll gladly download and run it many times a day. Building these images ourselves from known good GIT revisions prevents this. Additionally, the multi-architecture approach allows us to be one step closer to running the backend on more esoteric systems, like a Raspberry pi, or an AWS A instance, both of which are built on the ARM architecture. Or, if rumour is to be believed, the next release of MacBook Pros. :) -# Breaking it down: +## Breaking it down: -## Docker: +### Docker: to start with, we're going to have to get a bit into some docker architecture. We all have used docker, and pretty much understand the following workflow: @@ -27,19 +27,19 @@ I build a docker image from a Dockerfile and maybe some additions, I upload it t While this workflow works well for working with a single architecture, we're going to have to introduce some new concepts in order to support the multiple architecture way of building docker files. -### Manifest files. +#### Manifest files. Manifest files are agenerated by docker and contain references to multiple docker images, one for each architecture a given docker image has been built for. Each image in the manifest file is tagged with the architecture that the image is built for. Docker contains just enough built-in logic to interpret a manifest file on dockerhub, and download an image that matches the architecture that docker was built for. When using a manifest file, this is how docker determines what image to download. -### A Manifest centric Workflow: +#### A Manifest centric Workflow: If you're building a docker image for multiple architectures, you want a Manifest, so that docker automatically grabs the right image for the user's machine. This changes our workflow from earlier quite a bit: I build a docker image from a Dockerfile, and I build other images from slightly different versions of this Dockerfile (more on this later). I tag these images with a suffix, so that I can tell them apart. I upload the images to dockerhub, retaining the tags that differentiate the diffenent versions from each other. I create a manifest file, referring to the images that have been pushed to DockerHub, and upload the manifest file to DockerHub. People can download and use the image from dockerhub by refering to the tag of the manifest file. I can share the Dockerfile and additions via git, on dockerhub, and others can build their own images from it. -#### What does this look like? +##### What does this look like? All of us on the team are using AMD64 based machines, so in this example, we're going to build one image for AMD64, and one for it's predecessor architecture, I386. We're going to build the SMTP server image we depend on, from https://hub.docker.com/r/namshi/smtp. We're going to use a known safe git revision, and use some minor GNU sed to generate architecture dependent Dockerfiles from the Dockerfile in git. Everyone should be able to do this on your laptops. @@ -76,13 +76,13 @@ $ That wasn't so bad, was it? -##### what was that SED all about? +###### what was that SED all about? The sed commands used above accomplished two things. One, they changed out the MAINTAINER line in the Dockerfile, to indicate that I am the maintainer of this docker image. Two, for the 386 image, it specified that Docker was to start by using the i386 version of debian to base the image off of, not the AMD64 version. we did not need to make that change to the AMD64 version of the Dockerfile, because Docker on our local machine automatically downloads AMD64 images, since our copies of docker were built on AMD64 machines. -##### OK, what was the --amend on the docker manifest create line? +###### OK, what was the --amend on the docker manifest create line? Docker creates manifest files, and stores them in your local docker. I haven't found a good way to remove them, so instead, I add --amend, so that if one already exists, docker overwrites the locally stored file, instead of just telling you one already exists, and exiting with an error. -##### What does a manifest file look like? +###### What does a manifest file look like? to look at a manifest file (local or remote), use 'docker manifest inspect'. for example, here's the original namshi/smtp manifest. ```bash @@ -160,32 +160,32 @@ This is very different. instead of showing layers, it has the SHASUMs of the ima That's it as far as the docker parts of this. Simple, right? :) -### Limits of Manifest files: +#### Limits of Manifest files: I can't figure out how to delete local manifest files. I haven't figured out how to point to local images in a manifest file. this means if we use the name of a manifest in our Docker compose configuration, docker will go out to dockerhub for the image, rather than using a new image we just built, and we have to build a manifest file AFTER the push do dockerhub has been completed. -## QEMU + BinFmt Support: +### QEMU + BinFmt Support: The previous section has shown us how docker handles multiple architectures, this section is going to cover abusing the BinFmt linux kernel extensions and the QEMU system emulator to allow us to build docker images for non-native architectures, like arm, arm64, ppl64le, etc. -### About QEMU: +#### About QEMU: QEMU is a full system emulator, and a userspace emulator. QEMU's system emulation means that you can start qemu with disk images, ISOs, etc, for a supported architecture, and it will emulate a whole system around it, showing it's display in a window on your system. We're not going to be using this, and instead use it's userspace emulation. QEMU's userspace emulation allows you to download a program written for a different processor, and assuming you have the appropriate libraries, DLLs, etc... you can just run it locally. Typically this involves either having a program with no dependencies, or installing a set of system libraries on your machine for the target architecture alongside your current set of libraries. -### About BinFmt Support: +#### About BinFmt Support: BinFmt support is a set of extensions to the linux kernel that allow you to specify an interpreter for binaries of a certain pattern (magic number, ELF header, .EXE header, etc), so that when you attempt to execute them, the kernel will launch the interpreter of your choice, passing it the path to the binary you tried to execute. On my debian machine, it is used for python by default. Many people use this support for executing windows executables on linux using the WINE package, which contains a re-implementation of the Windows system libraries. The Linux kernel's BinFmt module can be set to look for an interpreter at run time, or to load the interpreter into memory when it is configured. The packages we're going to set up and exercise in this stage use the "load when you configure" approach. This is useful, so than when you're operating in a docker container, you don't have to place the system emulator in the docker container itsself for the kernel to find it. -### Installing QEMU with BinFmt support: +#### Installing QEMU with BinFmt support: Debian's qemu-user-static package sets this all up for you. as root: ```bash -# apt install qemu-user-static +## apt install qemu-user-static Reading package lists... Done Building dependency tree Reading state information... Done @@ -200,11 +200,11 @@ The following NEW packages will be installed: Unpacking qemu-user-static (1:3.1+dfsg-4) ... Setting up qemu-user-static (1:3.1+dfsg-4) ... Processing triggers for man-db (2.8.5-2) ... -# +## ``` -### Verifying it's configuration: +#### Verifying it's configuration: The linux kernel's BinFmt support is configured through a series of 'fake' files, in the /proc/sys/fs/binfmt_misc directory. by default, this directory contains the following: @@ -274,7 +274,7 @@ $ the 'F' in the flags field of that file indicates we're loading the emulators into ram. note that this file specifies the interpreter by it's full path, and a magic and mask field. these are the values the kernel looks for when executing a file, and if it finds them, launches the interpreter. -### Putting BinFmt and QEMU to work +#### Putting BinFmt and QEMU to work To test all of this, let's try executing something that's not built for our machine. We're going to try to launch a copy of SASH, the Staticly linked Almquist Shell. Because this is statically linked, we are not going to need to install any system libraries to launch it. We're going to launch an ARM copy, which normally wouldn't think of running on our machines. First, we'll grab a copy with wget. @@ -328,7 +328,7 @@ demo > ``` -## QEMU, BinFmt, and Docker (Oh my!) +### QEMU, BinFmt, and Docker (Oh my!) After following the directions in the last two sections, you've created two docker images (one for i386, one for AMD64), created a manifest referring to them, set up for linux to load qemu and use it, and launched a binary for another architecture. @@ -336,7 +336,7 @@ Creating non-native docker images can now be done very similar to how i386 was d Because you are using a system emulator, your docker builds for non-x86 will be slower. additionally, the emulators are not perfect, so some images won't build. finally, code is just less tested on machines that are not an AMD64 machine, so there are generally more bugs. -### Arm Complications: +#### Arm Complications: The 32 bit version of arm is actually divided into versions, and not all linux distributions are available for all versions. arm32v5 and arm32v7 are supported by debian, while arm32v6 is supported by alpine. This variant must be specified during manifest construction, so to continue with our current example, these are the commands for tagging the docker images for our arm32v5 and arm32v7 builds of smtp: ```bash $ docker manifest annotate julialongtin/smtp:0.0.9 julialongtin/smtp:0.0.9-arm32v5 --arch arm --variant 5 @@ -344,14 +344,14 @@ $ docker manifest annotate julialongtin/smtp:0.0.9 julialongtin/smtp:0.0.9-arm32 ``` -# Into the GNU Make Abyss +## Into the GNU Make Abyss Now that we've done all of the above, we should be capable of working with docker images independent of the architecture we're targeting. Now, into the rabit hole we go, automating everything with GNU Make -## Why Make? +### Why Make? GNU make is designed to build targets by looking at the environment it's in, and executing a number of rules depending on what it sees, and what it has been requested to do. The Makefile we're going to look through does all of the above, along with making some minor changes to the docker images. It does this in parallel, calling as many of the commands at once as possible, in order to take advantage of idle cores. -## Using the Makefile +### Using the Makefile Before we take the Makefile apart, let's go over using it. @@ -401,8 +401,8 @@ If we want to use these images in our docker compose, we can edit the docker com ports: - "127.0.0.1:9042:9042" environment: -# what's present in the jvm.options file by default. -# - "CS_JAVA_OPTIONS=-Xmx1024M -Xms1024M -Xmn200M" +## what's present in the jvm.options file by default. +## - "CS_JAVA_OPTIONS=-Xmx1024M -Xms1024M -Xmn200M" - "CS_JVM_OPTIONS=-Xmx128M -Xms128M -Xmn50M" networks: - demo_wire @@ -410,15 +410,15 @@ If we want to use these images in our docker compose, we can edit the docker com To remove all of the git repositories containing the Dockerfiles we download to build these images, we can run `make clean`. There is also the option to run `make cleandocker` to REMOVE ALL OF THE DOCKER IMAGES ON YOUR MACHINE. careful with that one. Note that docker makes good use of caching, so running 'make clean' and the same make command you used to build the images will complete really fast, as docker does not actually need to rebuild the images. -## Reading through the Makefile +### Reading through the Makefile OK, now that we have a handle on what it does, and how to use it, let's get into the Makefile itsself. A Makefile is a series of rules for performing tasks, variables used when creating those tasks, and some minimal functions and conditional structures. Rules are implemented as groups of bash commands, where each line is handled by a new bash interpreter. Personally, I think it 'feels functiony', only without a type system and with lots of side effects. Like if bash tried to be functional. -### Variables +#### Variables -#### Overrideable Variables +##### Overrideable Variables the make language has multiple types of variables and variable assignments. To begin with, let's look at the variables we used in the last step. ```bash $ cat Makefile | grep "?=" @@ -451,7 +451,7 @@ LOCALARCH and the assignments for ARCHES and NAMES are a bit different. LOCALARC Note the block of COMMIT IDs. This is in case we want to experiment with newer releases of each of the docker images we're using. Fixing what we're using to a commit ID makes it much harder for an upstream source to send us malicious code. -#### Non-Overrideable Variables +##### Non-Overrideable Variables The following group of variables use a different assignment operator, that tells make not to look in the environment first. ```bash $ cat Makefile | grep ":=" @@ -489,7 +489,7 @@ LOCALDEBARCH is a variable set by executing a small snippet of bash. The snippet NOMANIFEST lists images that need a work-around for fetching image dependencies for specific architectures. You know how we added the name of the architecture BEFORE the image name in the dockerfiles? well, in the case of the dependencies of the images listed here, dockerhub isn't supporting that. DockerHub is supporting that form only for 'official' docker images, like alpine, debian, etc. as a result, in order to fetch an architecture specific version of the dependencies of these images, we need to add a - suffix. like -386 -arm32v7, etc. -### Conditionals +#### Conditionals We don't make much use of conditionals, but there are three total uses in this Makefile. let's take a look at them. In order to look at our conditionals (and many other sections of this Makefile later), we're going to abuse sed. If you're not comfortable with the sed shown here, or are having problems getting it to work, you can instead just open the Makefile in your favorite text editor, and search around. I abuse sed here for both brevity, and to encourage the reader to understand complicated sed commands, for when we are using them later IN the Makefile. @@ -524,28 +524,28 @@ Now, back to our sed abuse. SED is a stream editor, and quite a powerful one. In this case, we're using it for a multi-line search. we're supplying the -n option, which squashes all output, except what sed is told specificly to print something with a command. Let's look at each of the commands in that statement seperately. ```sed -# find a line that has 'ifeq' in it. +## find a line that has 'ifeq' in it. /ifeq/ -# begin a block of commands. every command in the block should be seperated by a semicolon. +## begin a block of commands. every command in the block should be seperated by a semicolon. { -# create an anchor, that is to say, a point that can be branched to. +## create an anchor, that is to say, a point that can be branched to. :n; -# Append the next line into the parameter space. so now, for the first block, the hold parameter space would include "ifeq ($(LOCALARCH),)\n $(error LOCALARCH is empty, you may need to supply it.)". +## Append the next line into the parameter space. so now, for the first block, the hold parameter space would include "ifeq ($(LOCALARCH),)\n $(error LOCALARCH is empty, you may need to supply it.)". N; -# Replace the two spaces in the parameter space with one space. +## Replace the two spaces in the parameter space with one space. s/\n /\n /; -# If the previous 's' command found something, and changed something, go to our label. +## If the previous 's' command found something, and changed something, go to our label. tn; -# print the contents of the parameter space. +## print the contents of the parameter space. p -# close the block of commands. +## close the block of commands. } ``` ... Simple, right? note that the contents above can be stored to a file, and run with sed's "-f" command, for more complicated sed scripts. Sed is turing complete, so... things like tetris have been written in it. My longest sed scripts do things like sanity check OS install procedures, or change binaryish protocols into xmlish forms. -### Functions +#### Functions Make has a concept of functions, and the first two functions we use are a bit haskell inspired. SED ABUSE: @@ -607,7 +607,7 @@ $ cat Makefile | sed -n '/^[a-z]*=/p' ``` Again, we're going to use sed in '-n' mode, supressing all output except the output we are searching for. /PATTERN/ searches the lines of the input for a pattern, and if it's found, the command afterward is executed, which is a 'p' for print, in this case. the patern given is '^[a-z]*='. The '^' at the beginning means 'look for this patern at the beginning of the line, and the '=' at the end is the equal sign we were looking for. '[a-z]*' is us using a character class. character classes are sedspeak for sets of characters. they can be individually listed, or in this case, be a character range. the '*' after the character class just means "look for these characters any number of times". technically, that means a line starting in '=' would work (since zero is any number of times), but luckily, our file doesn't contain lines starting with =, as this is not valid make syntax. -### Rules. +#### Rules. Traditionally, makefiles are pretty simple. they are used to build a piece of software on your local machine, so you don't have to memorize all of the steps, and can type 'make', and have it just done. A simple Makefile looks like the following: ```make @@ -641,7 +641,7 @@ target: prerequisites The commands to build a thing (recipe lines) are prefaced with a tab character, and not spaces. Each line is executed in a seperate shell instance. -#### The roots of the trees +##### The roots of the trees In the section where we showed you how to use our Makefile, we were calling 'make' with an option, such as push-all, build-smtp, names, or clean. We're now going to show you the rules that implement these options. @@ -711,7 +711,7 @@ $ cat Makefile | sed -n -E '/^(.SECOND)/{:n;N;s/\n\t/\n /;tn;p}' | less build-% also uses the same 'fake recipe' trick as push-%, that is, having a recipe that does nothing, to trick make into letting you run this. -#### One Level Deeper +##### One Level Deeper The rules you've seen so far were intended for user interaction. they are all rules that the end user of this Makefile picks between, when deciding what they want this makefile to do. Let's look at the rules that these depend on. @@ -737,7 +737,7 @@ manifest-create-%: $$(foreach arch,$$(call goodarches,%), upload-$$(arch)-$$*) manifest-push depends on manifest-annotate, which depends on manifest-create, that depends on upload-... so when make tries to push a manifest, it makes sure an image has been uploaded, then creates a manifest, then annotates the manifest. We're basically writing rules for each step of our manifest, only backwards. continuing this pattern, the last thing we will depend on will be the rules that actually download the dockerfiles from git. -#### Dependency Resolving +##### Dependency Resolving We've covered the entry points of this Makefile, and the chained dependencies that create, annotate, and upload a manifest file. now, we get into two seriously complicated sets of rules, the upload rules and the create rules. These accomplish their tasks of uploading and building docker containers, but at the same time, they accomplish our dependency resolution. Let's take a look. @@ -788,7 +788,7 @@ create-%, depend-create-%, and depend-subcreate-% work similarly to the upload r It's worth noting that for all of these *create* and *upload* rules, we pipe the output of docker to cat, which causes docker to stop trying to draw progress bars. This seriously cleans up the -#### Building Dockerfiles. +##### Building Dockerfiles. There are two rules for creating Dockerfiles, and we decide in the *create* rules which of these to use by looking at the NOMANIFEST variable, and adding -NOMANIFEST in the name of the rule we depend on for dockerfile creation. @@ -825,13 +825,13 @@ The substitution section of this sed command uses the \1 and \2 variable referen Because we are using that sed command in a Makefile, we have to double up the "$" symbol, to prevent make from interpreting it as a variable. In the first sed command in these rules, we're also doing some minor escaping, adding a '\' in front of some quotes, so that our substitution of the maintainer has quotes around the email address. -#### Downloading Dockerfiles +##### Downloading Dockerfiles Finally, we are at the bottom of our dependency tree. We've followed this is reverse order, but when we actually ask for things to be pushed, or to be built, these rules are the first ones run. There are a lot of these, of various complexities, so let's start with the simple ones first. -##### Simple Checkout +###### Simple Checkout ```bash $ cat Makefile | sed -n -E '/^(smtp|dynamo|minio)/{:n;N;s/\n\t/\n /;tn;p}' @@ -851,7 +851,7 @@ minio/Dockerfile: These rules are simple. They git clone a repo, then reset the repo to a known good revision. This isolates us from potential breakage from upstreams, and prevents someone from stealing git credentials for our upstreams, and using those credentials to make a malignant version. -##### Checkout with Modifications +###### Checkout with Modifications A bit more complex rule is localstack/Dockerfile: ```bash @@ -889,7 +889,7 @@ SED ABUSE: To disable the installation of docker here, we do something a bit hacky. we find the line with 'install Docker' on it, we pull the next 5 lines into the pattern buffer, then delete them. This is effectively just a multiline delete. we use the -i.bak command line, just like the last sed abuse. neat and simple. -##### Checkout, Copy, Modify +###### Checkout, Copy, Modify Some of the git repositories that we depend on do not store the Dockerfile in the root of the repository. instead, they have one large repository, with many directories containing many docker images. In these cases, we use git to check out the repository into a directory with the name of the image followed by '-all', then copy the directory we want out of the tree. @@ -980,7 +980,7 @@ Structurally, the first, second, and third sed command are all pretty standard t Note that when we wrote our 'clean' rule, we added these '-all' directories manually, to make sure they would get deleted. -##### Checkout, Copy, Modify Multiline +###### Checkout, Copy, Modify Multiline elasticsearch and cassandra's checkouts are complicated, as they do a bit of injection of code into the docker entrypoint script. The entrypoint script is the script that is launched when you run a docker image. It's responsible for reading in environment variables, setting up the service that the docker image is supposed to run, and then running the service. For both elasticsearch and cassandra, we do a multiline insert, and we do it with multiple chained commands. @@ -1063,10 +1063,10 @@ This substitution command uses slashes as its separators. it starts by anchoring The cassandra/Dockerfile rule is almost identical to this last rule, only substituting out the name of the variable we expect from docker to CS_JVM_OPTIONS, and changing the path to the jvm.options file. -# Pitfalls I fell into writing this. +## Pitfalls I fell into writing this. The first large mistake I made when writing this, is that the root of the makefile's dependency tree contained both images that had dependencies, and the dependent images themselves. This had me writing methods to keep the image build process from stepping on itsself. what was happening is that, in the case of the airdock-* and localstack images, when trying to build all of the images at once, make would race all the way down to the git clone steps, and run the git clone multiple times at the same time, where it just needs to be run once. The second was that I didn't really understand that manifest files refer to dockerhub only, not to the local machine. This was giving me similar race conditions, where an image build for architecture A would complete, and try to build the manifest when architecture B was still building. -The third was writing really complicated SED and BASH and MAKE. ;p \ No newline at end of file +The third was writing really complicated SED and BASH and MAKE. ;p diff --git a/docs/legacy/reference/provisioning/scim-token.md b/docs/src/developer/reference/provisioning/scim-token.md similarity index 98% rename from docs/legacy/reference/provisioning/scim-token.md rename to docs/src/developer/reference/provisioning/scim-token.md index b7ae891de53..165e635d0c2 100644 --- a/docs/legacy/reference/provisioning/scim-token.md +++ b/docs/src/developer/reference/provisioning/scim-token.md @@ -1,4 +1,6 @@ -# SCIM tokens {#RefScimToken} +# SCIM tokens + +Reference: {#RefScimToken} _Author: Artyom Kazak_ diff --git a/docs/legacy/reference/spar-braindump.md b/docs/src/developer/reference/spar-braindump.md similarity index 98% rename from docs/legacy/reference/spar-braindump.md rename to docs/src/developer/reference/spar-braindump.md index ecb6ccda3ef..94d9b557970 100644 --- a/docs/legacy/reference/spar-braindump.md +++ b/docs/src/developer/reference/spar-braindump.md @@ -1,11 +1,11 @@ -# Spar braindump {#SparBrainDump} +# Spar braindump + +Reference: {#SparBrainDump} _Author: Matthias Fischmann_ --- -# the spar service for user provisioning (scim) and authentication (saml) - a brain dump - this is a mix of information on inmplementation details, architecture, and operation. it should probably be sorted into different places in the future, but if you can't find any more well-structured @@ -113,8 +113,7 @@ export IDP_ID=... Copy the new metadata file to one of your spar instances. -Ssh into it. If you can't, [the scim -docs](provisioning/scim-via-curl.md) explain how you can create a +Ssh into it. If you can't, [the sso docs](../../understand/single-sign-on/main.rst) explain how you can create a bearer token if you have the admin's login credentials. If you follow that approach, you need to replace all mentions of `-H'Z-User ...'` with `-H'Authorization: Bearer ...'` in the following, and you won't need @@ -169,7 +168,7 @@ Effects: and active IdPs. (Internal details: https://github.com/wireapp/wire-team-settings/issues/3465). -```shell +``` curl -v \ -XPOST http://localhost:8080/identity-providers'?replaces='${IDP_ID} \ -H"Z-User: ${ADMIN_ID}" \ @@ -183,7 +182,7 @@ curl -v \ Read the beginning of the last section up to "Option 1". You need `ADMIN_ID` (or `BEARER`) and `IDP_ID`, but not `METADATA_FILE`. -```shell +``` curl -v -XDELETE http://localhost:8080/identity-providers/${IDP_ID} \ -H"Z-User: ${ADMIN_ID}" \ @@ -195,7 +194,7 @@ with this IdP, you will get an error. You can either move these users elsewhere, delete them manually, or purge them implicitly during deletion of the IdP: -```shell +``` curl -v -XDELETE http://localhost:8080/identity-providers/${IDP_ID}?purge=true \ -H"Z-User: ${ADMIN_ID}" \ diff --git a/docs/legacy/reference/team/legalhold.md b/docs/src/developer/reference/team/legalhold.md similarity index 100% rename from docs/legacy/reference/team/legalhold.md rename to docs/src/developer/reference/team/legalhold.md diff --git a/docs/legacy/reference/user/activation.md b/docs/src/developer/reference/user/activation.md similarity index 76% rename from docs/legacy/reference/user/activation.md rename to docs/src/developer/reference/user/activation.md index 60868cd5811..723457ca7e9 100644 --- a/docs/legacy/reference/user/activation.md +++ b/docs/src/developer/reference/user/activation.md @@ -1,4 +1,6 @@ -# Activation {#RefActivation} +# User Activation + +Reference: {#RefActivation} _Author: Artyom Kazak_ @@ -8,19 +10,22 @@ A user is called _activated_ they have a verified identity -- e.g. a phone numbe A user that has been provisioned via single sign-on is always considered to be activated. -## Activated vs. non-activated users {#RefActivationBenefits} +## Activated vs. non-activated users +(RefActivationBenefits)= Non-activated users can not [connect](connection.md) to others, nor can connection requests be made to anonymous accounts from verified accounts. As a result: * A non-activated user cannot add other users to conversations. The only way to participate in a conversation is to either create a new conversation with link access or to use a link provided by another user. -The only flow where it makes sense for non-activated users to exist is the [wireless flow](registration.md#RefRegistrationWireless) used for [guest rooms](https://wire.com/en/features/encrypted-guest-rooms/) +The only flow where it makes sense for non-activated users to exist is the [wireless flow](RefRegistrationWireless) used for [guest rooms](https://wire.com/en/features/encrypted-guest-rooms/) -## API {#RefActivationApi} +## API +(RefActivationApi)= -### Requesting an activation code {#RefActivationRequest} +### Requesting an activation code +(RefActivationRequest)= -During the [standard registration flow](registration.md#RefRegistrationStandard), the user submits an email address or phone number by making a request to `POST /activate/send`. A six-digit activation code will be sent to that email address / phone number. Sample request and response: +During the [standard registration flow](RefRegistrationStandard), the user submits an email address or phone number by making a request to `POST /activate/send`. A six-digit activation code will be sent to that email address / phone number. Sample request and response: ``` POST /activate/send @@ -39,9 +44,10 @@ The user can submit the activation code during registration to prove that they o The same `POST /activate/send` endpoint can be used to re-request an activation code. Please use this ability sparingly! To avoid unnecessary activation code requests, users should be warned that it might take up to a few minutes for an email or text message to arrive. -### Activating an existing account {#RefActivationSubmit} +### Activating an existing account +(RefActivationSubmit)= -If the account [has not been activated during verification](registration.md#RefRegistrationNoPreverification), it can be activated afterwards by submitting an activation code to `POST /activate`. Sample request and response: +If the account [has not been activated during verification](RefRegistrationNoPreverification), it can be activated afterwards by submitting an activation code to `POST /activate`. Sample request and response: ``` POST /activate @@ -74,20 +80,22 @@ If the email or phone has been verified already, `POST /activate` will return st There is a maximum of 3 activation attempts per activation code. On the third failed attempt the code is invalidated and a new one must be requested. -### Activation event {#RefActivationEvent} +### Activation event +(RefActivationEvent)= When the user becomes activated, they receive an event: -```json +``` { "type": "user.activate", "user": } ``` -### Detecting activation in the self profile {#RefActivationProfile} +### Detecting activation in the self profile +(RefActivationProfile)= -In addition to the [activation event](#RefActivationEvent), activation can be detected by polling the self profile: +In addition to the [activation event](RefActivationEvent), activation can be detected by polling the self profile: ``` GET /self @@ -106,7 +114,8 @@ GET /self If the profile includes `"email"` or `"phone"`, the account is activated. -## Automating activation via email {#RefActivationEmailHeaders} +## Automating activation via email +(RefActivationEmailHeaders)= Our email verification messages contain headers that can be used to automate the activation process. @@ -125,7 +134,8 @@ X-Zeta-Key: ... X-Zeta-Code: 123456 ``` -## Phone/email whitelist {#RefActivationWhitelist} +## Phone/email whitelist +(RefActivationWhitelist)= The backend can be configured to only allow specific phone numbers or email addresses to register. The following options have to be set in `brig.yaml`: diff --git a/docs/legacy/reference/user/connection-transitions.png b/docs/src/developer/reference/user/connection-transitions.png similarity index 100% rename from docs/legacy/reference/user/connection-transitions.png rename to docs/src/developer/reference/user/connection-transitions.png diff --git a/docs/legacy/reference/user/connection-transitions.xml b/docs/src/developer/reference/user/connection-transitions.xml similarity index 100% rename from docs/legacy/reference/user/connection-transitions.xml rename to docs/src/developer/reference/user/connection-transitions.xml diff --git a/docs/legacy/reference/user/connection.md b/docs/src/developer/reference/user/connection.md similarity index 90% rename from docs/legacy/reference/user/connection.md rename to docs/src/developer/reference/user/connection.md index 910c7452881..aee2506e22e 100644 --- a/docs/legacy/reference/user/connection.md +++ b/docs/src/developer/reference/user/connection.md @@ -1,4 +1,6 @@ -# Connection {#RefConnection} +# Connection + +Reference: {#RefConnection} Two users can be _connected_ or not. If the users are connected, each of them can: @@ -8,25 +10,29 @@ Two users can be _connected_ or not. If the users are connected, each of them ca By default users with personal accounts are not connected. A user can send another user a _connection request_, which can be ignored or accepted by the other user. A user can also block an existing connection. -Members of the same team are always considered connected, see [Connections between team members](#RefConnectionTeam). +Members of the same team are always considered connected, see [Connections between team members](RefConnectionTeam). -Internally, connection status is a _directed_ edge from one user to another that is attributed with a relation state and some meta information. If a user has a connection to another user, it can be in one of the six [connection states](#RefConnectionStates). +Internally, connection status is a _directed_ edge from one user to another that is attributed with a relation state and some meta information. If a user has a connection to another user, it can be in one of the six [connection states](RefConnectionStates). -## Connection states {#RefConnectionStates} +## Connection states +(RefConnectionStates)= -### Sent {#RefConnectionSent} +### Sent +(RefConnectionSent)= In order for two users to become connected, one of them performs a _connection request_ and the other one accepts it. Initiating a new connection results in a pending 1-1 conversation to be created with the sender as the sole member. When the connection is accepted, the other user joins the conversation. The creator of a new connection (i.e. the sender of the connection request) ends up in this state. From the point of view of the creator, it indicates that a connection request has been sent but not accepted (it might be blocked or ignored). -### Pending {#RefConnectionPending} +### Pending +(RefConnectionPending)= The recipient of a connection request automatically ends up in this state. From his point of view, the state indicates that the connection is pending and awaiting further action (i.e. through accepting, ignoring or blocking it). -### Blocked {#RefConnectionBlocked} +### Blocked +(RefConnectionBlocked)= When a connection is in this state it indicates that the user does not want to be bothered by the other user, e.g. by receiving messages, calls or being added to conversations. @@ -34,27 +40,32 @@ Blocking a user does not prevent receiving further messages of that user in exis When user A blocks user B, the connection restrictions apply to both users -- e.g. A can not add B to conversations, even though it's A who blocked B and not vice-versa. -### Ignored {#RefConnectionIgnored} +### Ignored +(RefConnectionIgnored)= The recipient of a connection request may decide to explicitly "ignore" the request In this state the sender can continue to send further connection attempts. The recipient can change their mind and accept the request later. -### Cancelled {#RefConnectionCancelled} +### Cancelled +(RefConnectionCancelled)= This is a state that the sender can change to if the connection has not yet been accepted. The state will also change for the recipient, unless blocked. -### Accepted {#RefConnectionAccepted} +### Accepted +(RefConnectionAccepted)= A connection in this state is fully accepted by a user. The user thus allows the user at the other end of the connection to add him to conversations. For two users to be considered "connected", both A->B and B->A connections have to be in the "Accepted" state. -## Transitions between connection states {#RefConnectionTransitions} +## Transitions between connection states +(RefConnectionTransitions)= ![Connection state transitions](connection-transitions.png) (To edit this diagram, open [connection-transitions.xml](connection-transitions.xml) with .) -## Connections between team members {#RefConnectionTeam} +## Connections between team members +(RefConnectionTeam)= Users belonging to the same team are always implicitly treated as connected, to make it easier for team members to see each other's profiles, create conversations, etc. diff --git a/docs/legacy/reference/user/connections-flow-1-backend.png b/docs/src/developer/reference/user/connections-flow-1-backend.png similarity index 100% rename from docs/legacy/reference/user/connections-flow-1-backend.png rename to docs/src/developer/reference/user/connections-flow-1-backend.png diff --git a/docs/legacy/reference/user/registration.md b/docs/src/developer/reference/user/registration.md similarity index 90% rename from docs/legacy/reference/user/registration.md rename to docs/src/developer/reference/user/registration.md index fee8c310227..90fb353d583 100644 --- a/docs/legacy/reference/user/registration.md +++ b/docs/src/developer/reference/user/registration.md @@ -1,4 +1,5 @@ -# Registration {#RefRegistration} +# User Registration +(RefRegistration)= _Authors: Artyom Kazak, Matthias Fischmann_ @@ -6,15 +7,17 @@ _Authors: Artyom Kazak, Matthias Fischmann_ This page describes the "normal" user registration flow. Autoprovisioning is covered separately. -## Summary {#RefRegistrationSummary} +## Summary +(RefRegistrationSummary)= The vast majority of our API is only available to Wire users. Unless a user is autoprovisioned, they have to register an account by calling the `POST /register` endpoint. -Most users also go through [activation](activation.md) -- sharing and verifying an email address and/or phone number with Wire. This can happen either before or after registration. [Certain functionality](activation.md#RefActivationBenefits) is only available to activated users. +Most users also go through [activation](activation.md) -- sharing and verifying an email address and/or phone number with Wire. This can happen either before or after registration. [Certain functionality](RefActivationBenefits) is only available to activated users. -## Standard registration flow {#RefRegistrationStandard} +## Standard registration flow +(RefRegistrationStandard)= -During the standard registration flow, the user first calls [`POST /activate/send`](activation.md#RefActivationRequest) to pre-verify their email address or phone number. Phone numbers must be in [E.164][] format. +During the standard registration flow, the user first calls [`POST /activate/send`](RefActivationRequest) to pre-verify their email address or phone number. Phone numbers must be in [E.164][] format. [E.164]: https://en.wikipedia.org/wiki/E.164 @@ -68,11 +71,12 @@ If the code is incorrect or if an incorrect code has been tried enough times, th } ``` -## Registration without pre-verification {#RefRegistrationNoPreverification} +## Registration without pre-verification +(RefRegistrationNoPreverification)= _NOTE: This flow is currently not used by any clients. At least this was the state on 2020-05-28_ -It is also possible to call `POST /register` without verifying the email address or phone number, in which case the account will have to be activated later by calling [`POST /activate`](activation.md#RefActivationSubmit). Sample API request and response: +It is also possible to call `POST /register` without verifying the email address or phone number, in which case the account will have to be activated later by calling [`POST /activate`](RefActivationSubmit). Sample API request and response: ``` POST /register @@ -107,7 +111,9 @@ Set-Cookie: zuid=... A verification email will be sent to the email address (if provided), and a verification text message will be sent to the phone number (also, if provided). -## Anonymous registration, aka "Wireless" {#RefRegistrationWireless} +## Anonymous registration, aka "Wireless" +(RefRegistrationWireless)= + A user can be created without either email or phone number, in which case only `"name"` is required. The `"name"` does not have to be unique. This feature is used for [guest rooms](https://wire.com/en/features/encrypted-guest-rooms/). @@ -192,7 +198,7 @@ The rest of the unauthorized end-points is safe: - `~* ^/teams/invitations/info$`: only `GET`; requires invitation code. - `~* ^/teams/invitations/by-email$`: only `HEAD`. - `/invitations/info`: discontinued feature, can be removed from nginz config. -- `/conversations/code-check`: link validatoin for ephemeral/guest users. +- `/conversations/code-check`: link validation for ephemeral/guest users. - `/provider/*`: bots need to be registered to a team before becoming active. so if an attacker does not get access to a team, they cannot deploy a bot. - `~* ^/custom-backend/by-domain/([^/]*)$`: only `GET`; only exposes a list of domains that has is maintained through an internal end-point. used to redirect stock clients from the cloud instance to on-prem instances. - `~* ^/teams/api-docs`: only `GET`; swagger for part of the rest API. safe: it is trivial to identify the software that is running on the instance, and from there it is trivial to get to the source on github, where this can be obtained easily, and more. diff --git a/docs/legacy/reference/user/rich-info.md b/docs/src/developer/reference/user/rich-info.md similarity index 98% rename from docs/legacy/reference/user/rich-info.md rename to docs/src/developer/reference/user/rich-info.md index e6ecb60c297..6f8c6b666cd 100644 --- a/docs/legacy/reference/user/rich-info.md +++ b/docs/src/developer/reference/user/rich-info.md @@ -1,4 +1,6 @@ -# Rich info {#RefRichInfo} +# User Rich info + +Reference: {#RefRichInfo} _Author: Artyom Kazak_ @@ -86,7 +88,7 @@ PUT /scim/v2/Users/:id Here is an example for `PATCH`: -```json +``` PATCH /scim/v2/Users/:id { diff --git a/docs/src/how-to/administrate/users.rst b/docs/src/how-to/administrate/users.rst index 5c21fe44480..8aa3493860e 100644 --- a/docs/src/how-to/administrate/users.rst +++ b/docs/src/how-to/administrate/users.rst @@ -88,6 +88,29 @@ Afterwards, the previous command (to search for a user in cassandra) should retu When done, on terminal 1, ctrl+c to cancel the port-forwarding. +Searching and deleting users with no team +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you require users to be part of a team, or for some other reason you need to delete all users who are not part of a team, you need to first find all such users, and then delete them. + +To find users that are not part of a team, first you need to connect via SSH to the machine where cassandra is running, and then run the following command: + +.. code:: sh + + cqlsh 9042 -e "select team, handle, id from brig.user" | grep -E "^\s+null" + +This will give you a list of handles and IDs with no team associated: + +.. code:: sh + + null | null | bc22119f-ce11-4402-aa70-307a58fb22ec + null | tom | 8ecee3d0-47a4-43ff-977b-40a4fc350fed + null | alice | 2a4c3468-c1e6-422f-bc4d-4aeff47941ac + null | null | 1b5ca44a-aeb4-4a68-861b-48612438c4cc + null | bob | 701b4eab-6df2-476d-a818-90dc93e8446e + +You can then `delete each user with these instructions <./users.html#deleting-a-user-which-is-not-a-team-user>`__. + Manual search on elasticsearch (via brig, recommended) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/src/how-to/install/team-feature-settings.md b/docs/src/how-to/install/team-feature-settings.md index 3697b9b7ef4..ff1670f3449 100644 --- a/docs/src/how-to/install/team-feature-settings.md +++ b/docs/src/how-to/install/team-feature-settings.md @@ -67,3 +67,17 @@ galley: status: disabled lockStatus: locked ``` + +### TTL for nonces + +Nonces that can be retrieved e.g. by calling `HEAD /nonce/clients` have a default time-to-live of 5 minutes. To change this setting add the following to your Helm overrides in `values/wire-server/values.yaml`: + +```yaml +brig: + # ... + config: + # ... + optSettings: + # ... + setNonceTtlSecs: 360 # 6 minutes +``` diff --git a/docs/src/index.rst b/docs/src/index.rst index 8750a458584..a45427919b4 100644 --- a/docs/src/index.rst +++ b/docs/src/index.rst @@ -3,13 +3,18 @@ You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. -Welcome to Wire's (self-hosting) documentation! +Welcome to Wire's documentation! =============================================== +If you are a Wire end-user, please check out our `support pages `_. + The targeted audience of this documentation is: -* people wanting to understand how the server components of Wire work -* people wishing to self-host Wire on their own datacentres or cloud +* the curious power-user (people who want to understand how the server components of Wire work) +* on-premise operators/administrators (people who want to self-host Wire-Server on their own datacentres or cloud) +* developers (people who are working with the wire-server source code) + +If you are a developer, you may want to check out the "Notes for developers" first. This documentation may be expanded in the future to cover other aspects of Wire. @@ -26,6 +31,7 @@ This documentation may be expanded in the future to cover other aspects of Wire. How to set up user provisioning with LDAP or SCIM Client API documentation Security responses + Notes for developers .. Overview diff --git a/hack/helm_vars/wire-server/values.yaml.gotmpl b/hack/helm_vars/wire-server/values.yaml.gotmpl index b1c5191414d..fa1da9a7722 100644 --- a/hack/helm_vars/wire-server/values.yaml.gotmpl +++ b/hack/helm_vars/wire-server/values.yaml.gotmpl @@ -82,6 +82,7 @@ brig: - domain: federation-test-helper.{{ .Release.Namespace }}.svc.cluster.local search_policy: full_search set2FACodeGenerationDelaySecs: 5 + setNonceTtlSecs: 300 aws: sesEndpoint: http://fake-aws-ses:4569 sqsEndpoint: http://fake-aws-sqs:4568 @@ -167,6 +168,13 @@ galley: secrets: awsKeyId: dummykey awsSecretKey: dummysecret + mlsPrivateKeys: + removal: + ed25519: | + -----BEGIN PRIVATE KEY----- + MC4CAQAwBQYDK2VwBCIEIAocCDXsKIAjb65gOUn5vEF0RIKnVJkKR4ebQzuZ709c + -----END PRIVATE KEY----- + gundeck: replicaCount: 1 imagePullPolicy: {{ .Values.imagePullPolicy }} diff --git a/hie.yaml b/hie.yaml new file mode 100644 index 00000000000..daf97e6f4ab --- /dev/null +++ b/hie.yaml @@ -0,0 +1,2 @@ +cradle: + cabal: {} diff --git a/libs/brig-types/src/Brig/Types/Connection.hs b/libs/brig-types/src/Brig/Types/Connection.hs index c69ae043701..8bf4afcc5e6 100644 --- a/libs/brig-types/src/Brig/Types/Connection.hs +++ b/libs/brig-types/src/Brig/Types/Connection.hs @@ -32,7 +32,7 @@ import Data.Aeson import Data.Id (UserId) import Data.Qualified import Imports -import Wire.API.Arbitrary +import Wire.Arbitrary -- | Response type for endpoints returning lists of users with a specific connection state. -- E.g. 'getContactList' returns a 'UserIds' containing the list of connections in an diff --git a/libs/brig-types/src/Brig/Types/Test/Arbitrary.hs b/libs/brig-types/src/Brig/Types/Test/Arbitrary.hs index 936abe501cd..b4f19a89eb6 100644 --- a/libs/brig-types/src/Brig/Types/Test/Arbitrary.hs +++ b/libs/brig-types/src/Brig/Types/Test/Arbitrary.hs @@ -18,7 +18,7 @@ -- with this program. If not, see . module Brig.Types.Test.Arbitrary - ( module Wire.API.Arbitrary, + ( module Wire.Arbitrary, ) where @@ -27,7 +27,7 @@ import Brig.Types.Team.LegalHold import Data.String.Conversions (cs) import Imports import Test.QuickCheck -import Wire.API.Arbitrary +import Wire.Arbitrary instance Arbitrary ExcludedPrefix where arbitrary = ExcludedPrefix <$> arbitrary <*> arbitrary diff --git a/libs/dns-util/dns-util.cabal b/libs/dns-util/dns-util.cabal index 571b8901f74..9b03100539e 100644 --- a/libs/dns-util/dns-util.cabal +++ b/libs/dns-util/dns-util.cabal @@ -126,7 +126,7 @@ test-suite spec -Wpartial-fields -fwarn-tabs -optP-Wno-nonportable-include-path -threaded -rtsopts -with-rtsopts=-N - build-tool-depends: hspec-discover:hspec-discover -any + build-tool-depends: hspec-discover:hspec-discover build-depends: base >=4.6 && <5.0 , dns diff --git a/libs/extended/extended.cabal b/libs/extended/extended.cabal index d368bdfc386..9d64c15b6cc 100644 --- a/libs/extended/extended.cabal +++ b/libs/extended/extended.cabal @@ -143,7 +143,7 @@ test-suite extended-tests -Wpartial-fields -fwarn-tabs -optP-Wno-nonportable-include-path -threaded -with-rtsopts=-N - build-tool-depends: hspec-discover:hspec-discover -any + build-tool-depends: hspec-discover:hspec-discover build-depends: aeson , base diff --git a/libs/galley-types/src/Galley/Types/Teams.hs b/libs/galley-types/src/Galley/Types/Teams.hs index ce1f686034e..a06c1b0ea0e 100644 --- a/libs/galley-types/src/Galley/Types/Teams.hs +++ b/libs/galley-types/src/Galley/Types/Teams.hs @@ -178,8 +178,6 @@ data FeatureLegalHold deriving (Eq, Ord, Show, Enum, Bounded, Generic) -- | Default value for all teams that have not enabled or disabled this feature explicitly. --- See also 'Wire.API.Team.SearchVisibility.TeamSearchVisibilityEnabled', --- 'Wire.API.Team.SearchVisibility.TeamSearchVisibility'. data FeatureTeamSearchVisibilityAvailability = FeatureTeamSearchVisibilityAvailableByDefault | FeatureTeamSearchVisibilityUnavailableByDefault diff --git a/libs/galley-types/src/Galley/Types/Teams/Intra.hs b/libs/galley-types/src/Galley/Types/Teams/Intra.hs index 44b1607978b..1df8dd665fe 100644 --- a/libs/galley-types/src/Galley/Types/Teams/Intra.hs +++ b/libs/galley-types/src/Galley/Types/Teams/Intra.hs @@ -34,10 +34,10 @@ import Data.Json.Util import Data.Time (UTCTime) import Imports import Test.QuickCheck.Arbitrary (Arbitrary) -import Wire.API.Arbitrary (GenericUniform (..)) import Wire.API.Message (UserClients) import Wire.API.Team (Team) import Wire.API.Team.LegalHold (LegalholdProtectee) +import Wire.Arbitrary (GenericUniform (..)) data TeamStatus = Active diff --git a/libs/hscim/hscim.cabal b/libs/hscim/hscim.cabal index ef5bdc9eed1..185972c50f9 100644 --- a/libs/hscim/hscim.cabal +++ b/libs/hscim/hscim.cabal @@ -230,7 +230,7 @@ test-suite spec TypeSynonymInstances ghc-options: -Wall -Werror -threaded -rtsopts -with-rtsopts=-N - build-tool-depends: hspec-discover:hspec-discover -any + build-tool-depends: hspec-discover:hspec-discover build-depends: aeson >=2 , aeson-qq >=0.8.2 && <0.9 diff --git a/libs/metrics-wai/metrics-wai.cabal b/libs/metrics-wai/metrics-wai.cabal index 030bc71556a..f5e340b24aa 100644 --- a/libs/metrics-wai/metrics-wai.cabal +++ b/libs/metrics-wai/metrics-wai.cabal @@ -137,7 +137,7 @@ test-suite unit -Wpartial-fields -fwarn-tabs -optP-Wno-nonportable-include-path -threaded -with-rtsopts=-N - build-tool-depends: hspec-discover:hspec-discover -any + build-tool-depends: hspec-discover:hspec-discover build-depends: base >=4 && <5 , bytestring >=0.10 diff --git a/libs/polysemy-wire-zoo/polysemy-wire-zoo.cabal b/libs/polysemy-wire-zoo/polysemy-wire-zoo.cabal index 9c992d764b2..f239bef1903 100644 --- a/libs/polysemy-wire-zoo/polysemy-wire-zoo.cabal +++ b/libs/polysemy-wire-zoo/polysemy-wire-zoo.cabal @@ -21,6 +21,8 @@ library Wire.Sem.Now.Input Wire.Sem.Now.IO Wire.Sem.Now.Spec + Wire.Sem.Paging + Wire.Sem.Paging.Cassandra Wire.Sem.Random Wire.Sem.Random.IO @@ -72,6 +74,7 @@ library build-depends: base >=4.6 && <5.0 + , cassandra-util , HsOpenSSL , hspec , imports @@ -84,5 +87,6 @@ library , tinylog , types-common , uuid + , wire-api default-language: Haskell2010 diff --git a/services/galley/src/Galley/Effects/Paging.hs b/libs/polysemy-wire-zoo/src/Wire/Sem/Paging.hs similarity index 98% rename from services/galley/src/Galley/Effects/Paging.hs rename to libs/polysemy-wire-zoo/src/Wire/Sem/Paging.hs index 8df8e9393af..2636eae7a96 100644 --- a/services/galley/src/Galley/Effects/Paging.hs +++ b/libs/polysemy-wire-zoo/src/Wire/Sem/Paging.hs @@ -15,7 +15,7 @@ -- You should have received a copy of the GNU Affero General Public License along -- with this program. If not, see . -module Galley.Effects.Paging +module Wire.Sem.Paging ( -- * General paging types Page, PagingState, diff --git a/services/galley/src/Galley/Cassandra/Paging.hs b/libs/polysemy-wire-zoo/src/Wire/Sem/Paging/Cassandra.hs similarity index 72% rename from services/galley/src/Galley/Cassandra/Paging.hs rename to libs/polysemy-wire-zoo/src/Wire/Sem/Paging/Cassandra.hs index 139a42df587..ec3efee4fdb 100644 --- a/services/galley/src/Galley/Cassandra/Paging.hs +++ b/libs/polysemy-wire-zoo/src/Wire/Sem/Paging/Cassandra.hs @@ -15,7 +15,7 @@ -- You should have received a copy of the GNU Affero General Public License along -- with this program. If not, see . -module Galley.Cassandra.Paging +module Wire.Sem.Paging.Cassandra ( CassandraPaging, LegacyPaging, InternalPaging, @@ -23,9 +23,8 @@ module Galley.Cassandra.Paging InternalPagingState (..), mkInternalPage, ipNext, - - -- * Re-exports ResultSet, + mkResultSet, resultSetResult, resultSetType, ResultSetType (..), @@ -36,10 +35,9 @@ import Cassandra import Data.Id import Data.Qualified import Data.Range -import Galley.Cassandra.ResultSet -import qualified Galley.Effects.Paging as E import Imports import Wire.API.Team.Member (HardTruncationLimit, TeamMember) +import qualified Wire.Sem.Paging as E -- | This paging system uses Cassandra's 'PagingState' to keep track of state, -- and does not rely on ordering. This is the preferred way of paging across @@ -101,3 +99,33 @@ instance E.Paging InternalPaging where pageItems (InternalPage (_, _, items)) = items pageHasMore (InternalPage (p, _, _)) = hasMore p pageState (InternalPage (p, f, _)) = InternalPagingState (p, f) + +-- We use this newtype to highlight the fact that the 'Page' wrapped in here +-- can not reliably used for paging. +-- +-- The reason for this is that Cassandra returns 'hasMore' as true if the +-- page size requested is equal to result size. To work around this we +-- actually request for one additional element and drop the last value if +-- necessary. This means however that 'nextPage' does not work properly as +-- we would miss a value on every page size. +-- Thus, and since we don't want to expose the ResultSet constructor +-- because it gives access to `nextPage`, we give accessors to the results +-- and a more typed `hasMore` (ResultSetComplete | ResultSetTruncated) +data ResultSet a = ResultSet + { resultSetResult :: [a], + resultSetType :: ResultSetType + } + deriving stock (Show, Functor, Foldable, Traversable) + +-- | A more descriptive type than using a simple bool to represent `hasMore` +data ResultSetType + = ResultSetComplete + | ResultSetTruncated + deriving stock (Eq, Show) + +mkResultSet :: Page a -> ResultSet a +mkResultSet page = ResultSet (result page) typ + where + typ + | hasMore page = ResultSetTruncated + | otherwise = ResultSetComplete diff --git a/libs/sodium-crypto-sign/sodium-crypto-sign.cabal b/libs/sodium-crypto-sign/sodium-crypto-sign.cabal index e028e4d5de7..d4b44e350c3 100644 --- a/libs/sodium-crypto-sign/sodium-crypto-sign.cabal +++ b/libs/sodium-crypto-sign/sodium-crypto-sign.cabal @@ -61,7 +61,7 @@ library -O2 -Wall -Wincomplete-uni-patterns -Wincomplete-record-updates -Wpartial-fields -fwarn-tabs -optP-Wno-nonportable-include-path - pkgconfig-depends: libsodium ==0.4.5 || >0.4.5 + pkgconfig-depends: libsodium >=0.4.5 build-depends: base >=4.6 && <5 , base64-bytestring >=1.0 diff --git a/libs/types-common-journal/types-common-journal.cabal b/libs/types-common-journal/types-common-journal.cabal index 99f1dbc9e5a..bda3351280c 100644 --- a/libs/types-common-journal/types-common-journal.cabal +++ b/libs/types-common-journal/types-common-journal.cabal @@ -78,7 +78,7 @@ library -fno-warn-redundant-constraints ghc-prof-options: -fprof-auto-exported - build-tool-depends: proto-lens-protoc:proto-lens-protoc -any + build-tool-depends: proto-lens-protoc:proto-lens-protoc build-depends: base >=4 && <5 , bytestring diff --git a/libs/types-common/src/Data/Nonce.hs b/libs/types-common/src/Data/Nonce.hs new file mode 100644 index 00000000000..26e4073a5cf --- /dev/null +++ b/libs/types-common/src/Data/Nonce.hs @@ -0,0 +1,100 @@ +{-# LANGUAGE DataKinds #-} +{-# LANGUAGE GeneralizedNewtypeDeriving #-} + +-- This file is part of the Wire Server implementation. +-- +-- Copyright (C) 2022 Wire Swiss GmbH +-- +-- This program is free software: you can redistribute it and/or modify it under +-- the terms of the GNU Affero General Public License as published by the Free +-- Software Foundation, either version 3 of the License, or (at your option) any +-- later version. +-- +-- This program is distributed in the hope that it will be useful, but WITHOUT +-- ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +-- FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more +-- details. +-- +-- You should have received a copy of the GNU Affero General Public License along +-- with this program. If not, see . + +module Data.Nonce + ( Nonce (..), + NonceTtlSecs (..), + randomNonce, + isValidBase64UrlEncodedUUID, + ) +where + +import Cassandra hiding (Value) +import qualified Data.Aeson as A +import qualified Data.ByteString.Base64.URL as Base64 +import Data.ByteString.Conversion +import Data.ByteString.Lazy (fromStrict, toStrict) +import Data.Proxy (Proxy (Proxy)) +import Data.Schema +import Data.String.Conversions (cs) +import qualified Data.Swagger as S +import Data.Swagger.ParamSchema +import Data.UUID as UUID (UUID, fromByteString, toByteString) +import Data.UUID.V4 (nextRandom) +import Imports +import Servant (FromHttpApiData (..), ToHttpApiData (..)) +import Test.QuickCheck (Arbitrary) +import Test.QuickCheck.Instances.UUID () + +newtype Nonce = Nonce {unNonce :: UUID} + deriving (Eq, Show) + deriving newtype (A.FromJSON, A.ToJSON, S.ToSchema, Arbitrary) + +instance ToByteString Nonce where + builder = builder . Base64.encode . toStrict . UUID.toByteString . unNonce + +instance FromByteString Nonce where + parser = do + a <- parser + maybe + (fail "invalid base64url encoded uuidv4") + (pure . Nonce) + (either (const Nothing) (UUID.fromByteString . fromStrict) (Base64.decode a)) + +instance ToParamSchema Nonce where + toParamSchema _ = toParamSchema (Proxy @Text) + +instance ToHttpApiData Nonce where + toQueryParam = cs . toByteString' + +instance FromHttpApiData Nonce where + parseQueryParam = maybe (Left "Invalid Nonce") Right . fromByteString' . cs + +randomNonce :: (Functor m, MonadIO m) => m Nonce +randomNonce = Nonce <$> liftIO nextRandom + +isValidBase64UrlEncodedUUID :: ByteString -> Bool +isValidBase64UrlEncodedUUID = isJust . fromByteString' @Nonce . cs + +instance Cql Nonce where + ctype = Tagged UuidColumn + toCql = toCql . unNonce + fromCql v = Nonce <$> fromCql v + +newtype NonceTtlSecs = NonceTtlSecs {unNonceTtlSecs :: Word32} + deriving (Eq, Show, Generic) + deriving (A.FromJSON, A.ToJSON, S.ToSchema) via (Schema NonceTtlSecs) + +-- | Convert 'Word32' to 'Int32', with clipping if it doesn't fit. +word32ToInt32 :: Word32 -> Int32 +word32ToInt32 = fromIntegral . min (fromIntegral (maxBound @Int32)) + +-- | Convert 'Int32' to 'Word32', rounding negative values to 0. +int32ToWord32 :: Int32 -> Word32 +int32ToWord32 = fromIntegral . max 0 + +instance ToSchema NonceTtlSecs where + schema = NonceTtlSecs . int32ToWord32 <$> (word32ToInt32 . unNonceTtlSecs) .= schema + +instance Cql NonceTtlSecs where + ctype = Tagged IntColumn + toCql = CqlInt . (word32ToInt32 . unNonceTtlSecs) + fromCql (CqlInt i) = pure $ NonceTtlSecs $ int32ToWord32 i + fromCql _ = Left "fromCql: NonceTtlSecs expects CqlInt" diff --git a/libs/wire-api/src/Wire/API/Arbitrary.hs b/libs/types-common/src/Wire/Arbitrary.hs similarity index 99% rename from libs/wire-api/src/Wire/API/Arbitrary.hs rename to libs/types-common/src/Wire/Arbitrary.hs index 9974d5f90bd..ed591cca132 100644 --- a/libs/wire-api/src/Wire/API/Arbitrary.hs +++ b/libs/types-common/src/Wire/Arbitrary.hs @@ -20,7 +20,7 @@ -- You should have received a copy of the GNU Affero General Public License along -- with this program. If not, see . -module Wire.API.Arbitrary +module Wire.Arbitrary ( Arbitrary (..), GenericUniform (..), listOf', diff --git a/libs/types-common/test/Test/Properties.hs b/libs/types-common/test/Test/Properties.hs index 9b5ba7880e3..31a3eef1e58 100644 --- a/libs/types-common/test/Test/Properties.hs +++ b/libs/types-common/test/Test/Properties.hs @@ -35,6 +35,7 @@ import Data.Domain (Domain) import Data.Handle (Handle) import Data.Id import qualified Data.Json.Util as Util +import Data.Nonce (Nonce) import Data.ProtocolBuffers.Internal import Data.Serialize import Data.String.Conversions (cs) @@ -209,9 +210,17 @@ tests = "Domain" [ jsonRoundtrip @Domain, jsonKeyRoundtrip @Domain + ], + testGroup + "Nonce" + [ testProperty "decode . encode = id" $ + \(x :: Nonce) -> bsRoundtrip x ] ] +bsRoundtrip :: (Eq a, Show a, FromByteString a, ToByteString a) => a -> Property +bsRoundtrip a = fromByteString' (cs $ toByteString' a) === Just a + roundtrip :: (EncodeWire a, DecodeWire a) => Tag' -> a -> Either String a roundtrip (Tag' t) = runGet (getWireField >>= decodeWire) . runPut . encodeWire t diff --git a/libs/types-common/types-common.cabal b/libs/types-common/types-common.cabal index beffd1bd551..9be5949ebc2 100644 --- a/libs/types-common/types-common.cabal +++ b/libs/types-common/types-common.cabal @@ -23,6 +23,7 @@ library Data.LegalHold Data.List1 Data.Misc + Data.Nonce Data.Qualified Data.Range Data.RetryAfter @@ -34,6 +35,7 @@ library Util.Options Util.Options.Common Util.Test + Wire.Arbitrary Wire.Swagger other-modules: Paths_types_common @@ -87,7 +89,7 @@ library aeson >=2.0.1.0 , attoparsec >=0.11 , attoparsec-iso8601 - , base >=4 && <5 + , base >=4 && <5 , base16-bytestring >=0.1 , base64-bytestring >=1.0 , binary @@ -98,13 +100,18 @@ library , cryptohash-md5 >=0.11.7.2 , cryptohash-sha1 >=0.11.7.2 , cryptonite >=0.26 + , currency-codes >=3.0.0.1 , data-default >=0.5 + , generic-random >=1.4.0.0 , hashable >=1.2 , http-api-data , imports , iproute >=1.5 + , iso3166-country-codes >=0.20140203.8 + , iso639 >=0.1.0.3 , lens >=4.10 , lens-datetime >=0.3 + , mime >=0.4.0.2 , optparse-applicative >=0.10 , protobuf >=0.2 , QuickCheck >=2.9 diff --git a/libs/wire-api-federation/src/Wire/API/Federation/API/Brig.hs b/libs/wire-api-federation/src/Wire/API/Federation/API/Brig.hs index 5b83c228fff..f9c36367bd0 100644 --- a/libs/wire-api-federation/src/Wire/API/Federation/API/Brig.hs +++ b/libs/wire-api-federation/src/Wire/API/Federation/API/Brig.hs @@ -24,19 +24,18 @@ import Data.Range import Imports import Servant.API import Test.QuickCheck (Arbitrary) -import Wire.API.Arbitrary (GenericUniform (..)) import Wire.API.Federation.API.Common import Wire.API.Federation.Endpoint import Wire.API.Federation.Version import Wire.API.MLS.Credential import Wire.API.MLS.KeyPackage -import Wire.API.Message (UserClients) import Wire.API.User (UserProfile) -import Wire.API.User.Client (PubClient, UserClientPrekeyMap) +import Wire.API.User.Client import Wire.API.User.Client.Prekey (ClientPrekey, PrekeyBundle) import Wire.API.User.Search import Wire.API.UserMap (UserMap) import Wire.API.Util.Aeson (CustomEncoded (..)) +import Wire.Arbitrary (GenericUniform (..)) newtype SearchRequest = SearchRequest {term :: Text} deriving (Show, Eq, Generic, Typeable) @@ -68,7 +67,7 @@ type BrigApi = -- (handles can be up to 256 chars currently) :<|> FedEndpoint "search-users" SearchRequest SearchResponse :<|> FedEndpoint "get-user-clients" GetUserClients (UserMap (Set PubClient)) - :<|> FedEndpoint "get-mls-clients" MLSClientsRequest (Set ClientId) + :<|> FedEndpoint "get-mls-clients" MLSClientsRequest (Set ClientInfo) :<|> FedEndpoint "send-connection-action" NewConnectionRequest NewConnectionResponse :<|> FedEndpoint "on-user-deleted-connections" UserDeletedConnectionsNotification EmptyResponse :<|> FedEndpoint "claim-key-packages" ClaimKeyPackageRequest (Maybe KeyPackageBundle) diff --git a/libs/wire-api-federation/src/Wire/API/Federation/API/Cargohold.hs b/libs/wire-api-federation/src/Wire/API/Federation/API/Cargohold.hs index f4da5333664..bcff826a17b 100644 --- a/libs/wire-api-federation/src/Wire/API/Federation/API/Cargohold.hs +++ b/libs/wire-api-federation/src/Wire/API/Federation/API/Cargohold.hs @@ -21,11 +21,11 @@ import Data.Aeson (FromJSON (..), ToJSON (..)) import Data.Id import Imports import Servant.API -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) import Wire.API.Asset import Wire.API.Federation.Endpoint import Wire.API.Routes.AssetBody import Wire.API.Util.Aeson +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) data GetAsset = GetAsset { -- | User requesting the asset. Implictly qualified with the source domain. diff --git a/libs/wire-api-federation/src/Wire/API/Federation/API/Common.hs b/libs/wire-api-federation/src/Wire/API/Federation/API/Common.hs index ebc038f58f3..265a40f7fd0 100644 --- a/libs/wire-api-federation/src/Wire/API/Federation/API/Common.hs +++ b/libs/wire-api-federation/src/Wire/API/Federation/API/Common.hs @@ -20,7 +20,7 @@ module Wire.API.Federation.API.Common where import Data.Aeson import Imports import Test.QuickCheck -import Wire.API.Arbitrary +import Wire.Arbitrary -- | This is equivalent to '()', but JSONifies to an empty object instead of an -- empty array. diff --git a/libs/wire-api-federation/src/Wire/API/Federation/API/Galley.hs b/libs/wire-api-federation/src/Wire/API/Federation/API/Galley.hs index e93c1f8f65a..b654911291b 100644 --- a/libs/wire-api-federation/src/Wire/API/Federation/API/Galley.hs +++ b/libs/wire-api-federation/src/Wire/API/Federation/API/Galley.hs @@ -27,7 +27,6 @@ import Data.Time.Clock (UTCTime) import Imports import qualified Network.Wai.Utilities.Error as Wai import Servant.API -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) import Wire.API.Conversation import Wire.API.Conversation.Action import Wire.API.Conversation.Protocol @@ -38,6 +37,7 @@ import Wire.API.Federation.Endpoint import Wire.API.Message import Wire.API.Routes.Public.Galley import Wire.API.Util.Aeson (CustomEncoded (..)) +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) -- FUTUREWORK: data types, json instances, more endpoints. See -- https://wearezeta.atlassian.net/wiki/spaces/CORE/pages/356090113/Federation+Galley+Conversation+API diff --git a/libs/wire-api-federation/src/Wire/API/Federation/Component.hs b/libs/wire-api-federation/src/Wire/API/Federation/Component.hs index 075a5aa9007..908f3b01c47 100644 --- a/libs/wire-api-federation/src/Wire/API/Federation/Component.hs +++ b/libs/wire-api-federation/src/Wire/API/Federation/Component.hs @@ -19,7 +19,7 @@ module Wire.API.Federation.Component where import Imports import Test.QuickCheck (Arbitrary) -import Wire.API.Arbitrary (GenericUniform (..)) +import Wire.Arbitrary (GenericUniform (..)) data Component = Brig diff --git a/libs/wire-api-federation/wire-api-federation.cabal b/libs/wire-api-federation/wire-api-federation.cabal index 1396a594d20..81eae46d309 100644 --- a/libs/wire-api-federation/wire-api-federation.cabal +++ b/libs/wire-api-federation/wire-api-federation.cabal @@ -181,7 +181,7 @@ test-suite spec -Wpartial-fields -fwarn-tabs -optP-Wno-nonportable-include-path -threaded -rtsopts -with-rtsopts=-N - build-tool-depends: hspec-discover:hspec-discover -any + build-tool-depends: hspec-discover:hspec-discover build-depends: aeson >=2.0.1.0 , aeson-pretty diff --git a/libs/wire-api/src/Wire/API/Asset.hs b/libs/wire-api/src/Wire/API/Asset.hs index 8f2bf9c6ac7..04f157ae7ba 100644 --- a/libs/wire-api/src/Wire/API/Asset.hs +++ b/libs/wire-api/src/Wire/API/Asset.hs @@ -88,9 +88,9 @@ import qualified Data.UUID as UUID import Imports import Servant import URI.ByteString -import Wire.API.Arbitrary (Arbitrary (..), GenericUniform (..)) import Wire.API.Error import Wire.API.Routes.MultiVerb +import Wire.Arbitrary (Arbitrary (..), GenericUniform (..)) -------------------------------------------------------------------------------- -- Asset diff --git a/libs/wire-api/src/Wire/API/Call/Config.hs b/libs/wire-api/src/Wire/API/Call/Config.hs index 8cc1e729a07..3777bdcbb77 100644 --- a/libs/wire-api/src/Wire/API/Call/Config.hs +++ b/libs/wire-api/src/Wire/API/Call/Config.hs @@ -91,7 +91,7 @@ import Data.Time.Clock.POSIX import Imports import qualified Test.QuickCheck as QC import Text.Hostname (validHostname) -import Wire.API.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) +import Wire.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) -------------------------------------------------------------------------------- -- RTCConfiguration diff --git a/libs/wire-api/src/Wire/API/Connection.hs b/libs/wire-api/src/Wire/API/Connection.hs index d9437e0663c..fa3fc2af11e 100644 --- a/libs/wire-api/src/Wire/API/Connection.hs +++ b/libs/wire-api/src/Wire/API/Connection.hs @@ -58,8 +58,8 @@ import qualified Data.Swagger.Build.Api as Doc import Data.Text as Text import Imports import Servant.API -import Wire.API.Arbitrary (Arbitrary (..), GenericUniform (..)) import Wire.API.Routes.MultiTablePaging +import Wire.Arbitrary (Arbitrary (..), GenericUniform (..)) -------------------------------------------------------------------------------- -- UserConnectionList diff --git a/libs/wire-api/src/Wire/API/Conversation.hs b/libs/wire-api/src/Wire/API/Conversation.hs index 2c8e6f9613a..f9ad1b122ea 100644 --- a/libs/wire-api/src/Wire/API/Conversation.hs +++ b/libs/wire-api/src/Wire/API/Conversation.hs @@ -112,12 +112,12 @@ import qualified Data.Swagger as S import qualified Data.Swagger.Build.Api as Doc import Imports import System.Random (randomRIO) -import Wire.API.Arbitrary import Wire.API.Conversation.Member import Wire.API.Conversation.Protocol import Wire.API.Conversation.Role (RoleName, roleNameWireAdmin) import Wire.API.MLS.Group import Wire.API.Routes.MultiTablePaging +import Wire.Arbitrary -------------------------------------------------------------------------------- -- Conversation diff --git a/libs/wire-api/src/Wire/API/Conversation/Action.hs b/libs/wire-api/src/Wire/API/Conversation/Action.hs index 15a0ec0879d..bd0700a521a 100644 --- a/libs/wire-api/src/Wire/API/Conversation/Action.hs +++ b/libs/wire-api/src/Wire/API/Conversation/Action.hs @@ -43,11 +43,11 @@ import Data.Singletons.TH import qualified Data.Swagger as S import Data.Time.Clock import Imports -import Wire.API.Arbitrary (Arbitrary (..)) import Wire.API.Conversation import Wire.API.Conversation.Action.Tag import Wire.API.Conversation.Role import Wire.API.Event.Conversation +import Wire.Arbitrary (Arbitrary (..)) -- | We use this type family instead of a sum type to be able to define -- individual effects per conversation action. See 'HasConversationActionEffects'. diff --git a/libs/wire-api/src/Wire/API/Conversation/Action/Tag.hs b/libs/wire-api/src/Wire/API/Conversation/Action/Tag.hs index 1f53eb6fb0d..3445e3794ff 100644 --- a/libs/wire-api/src/Wire/API/Conversation/Action/Tag.hs +++ b/libs/wire-api/src/Wire/API/Conversation/Action/Tag.hs @@ -26,7 +26,7 @@ import Data.Schema hiding (tag) import Data.Singletons.TH import Imports import Test.QuickCheck (elements) -import Wire.API.Arbitrary (Arbitrary (..)) +import Wire.Arbitrary (Arbitrary (..)) data ConversationActionTag = ConversationJoinTag diff --git a/libs/wire-api/src/Wire/API/Conversation/Bot.hs b/libs/wire-api/src/Wire/API/Conversation/Bot.hs index 05112e9cd90..4b4da2f0466 100644 --- a/libs/wire-api/src/Wire/API/Conversation/Bot.hs +++ b/libs/wire-api/src/Wire/API/Conversation/Bot.hs @@ -30,10 +30,10 @@ import Data.Aeson import Data.Id import Data.Json.Util ((#)) import Imports -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) import Wire.API.Event.Conversation (Event) import Wire.API.User.Client.Prekey (Prekey) import Wire.API.User.Profile (Asset, ColourId, Locale, Name) +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) -------------------------------------------------------------------------------- -- AddBot diff --git a/libs/wire-api/src/Wire/API/Conversation/Code.hs b/libs/wire-api/src/Wire/API/Conversation/Code.hs index e953df6d475..2120f5a6808 100644 --- a/libs/wire-api/src/Wire/API/Conversation/Code.hs +++ b/libs/wire-api/src/Wire/API/Conversation/Code.hs @@ -43,7 +43,7 @@ import qualified Data.Swagger as S import qualified Data.Swagger.Build.Api as Doc import Imports import qualified URI.ByteString as URI -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) data ConversationCode = ConversationCode { conversationKey :: Code.Key, diff --git a/libs/wire-api/src/Wire/API/Conversation/Member.hs b/libs/wire-api/src/Wire/API/Conversation/Member.hs index b01ac7a559f..f79d8e530e1 100644 --- a/libs/wire-api/src/Wire/API/Conversation/Member.hs +++ b/libs/wire-api/src/Wire/API/Conversation/Member.hs @@ -51,9 +51,9 @@ import qualified Data.Swagger as S import qualified Data.Swagger.Build.Api as Doc import Imports import qualified Test.QuickCheck as QC -import Wire.API.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) import Wire.API.Conversation.Role import Wire.API.Provider.Service (ServiceRef, modelServiceRef) +import Wire.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) data ConvMembers = ConvMembers { cmSelf :: Member, diff --git a/libs/wire-api/src/Wire/API/Conversation/Protocol.hs b/libs/wire-api/src/Wire/API/Conversation/Protocol.hs index 8f5a16a28c6..718caa3071d 100644 --- a/libs/wire-api/src/Wire/API/Conversation/Protocol.hs +++ b/libs/wire-api/src/Wire/API/Conversation/Protocol.hs @@ -37,11 +37,11 @@ import Control.Lens (makePrisms, (?~)) import Data.Aeson (FromJSON (..), ToJSON (..)) import Data.Schema import Imports -import Wire.API.Arbitrary import Wire.API.Conversation.Action.Tag import Wire.API.MLS.CipherSuite import Wire.API.MLS.Epoch import Wire.API.MLS.Group +import Wire.Arbitrary data ProtocolTag = ProtocolProteusTag | ProtocolMLSTag deriving stock (Eq, Show, Enum, Bounded, Generic) diff --git a/libs/wire-api/src/Wire/API/Conversation/Role.hs b/libs/wire-api/src/Wire/API/Conversation/Role.hs index c015e430982..e215b72db88 100644 --- a/libs/wire-api/src/Wire/API/Conversation/Role.hs +++ b/libs/wire-api/src/Wire/API/Conversation/Role.hs @@ -83,7 +83,7 @@ import qualified Deriving.Swagger as S import GHC.TypeLits import Imports import qualified Test.QuickCheck as QC -import Wire.API.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) +import Wire.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) -------------------------------------------------------------------------------- -- Role diff --git a/libs/wire-api/src/Wire/API/Conversation/Typing.hs b/libs/wire-api/src/Wire/API/Conversation/Typing.hs index dc1480b6e70..d18131b3c6c 100644 --- a/libs/wire-api/src/Wire/API/Conversation/Typing.hs +++ b/libs/wire-api/src/Wire/API/Conversation/Typing.hs @@ -33,7 +33,7 @@ import Data.Schema import qualified Data.Swagger as S import qualified Data.Swagger.Build.Api as Doc import Imports -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) newtype TypingData = TypingData { tdStatus :: TypingStatus diff --git a/libs/wire-api/src/Wire/API/CustomBackend.hs b/libs/wire-api/src/Wire/API/CustomBackend.hs index 010fb466a69..75b0e976a77 100644 --- a/libs/wire-api/src/Wire/API/CustomBackend.hs +++ b/libs/wire-api/src/Wire/API/CustomBackend.hs @@ -28,7 +28,7 @@ import Data.Schema import qualified Data.Swagger as S import Deriving.Aeson import Imports -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) data CustomBackend = CustomBackend { backendConfigJsonUrl :: HttpsUrl, diff --git a/libs/wire-api/src/Wire/API/Error/Brig.hs b/libs/wire-api/src/Wire/API/Error/Brig.hs index e8ca3a3bb46..544a3755cde 100644 --- a/libs/wire-api/src/Wire/API/Error/Brig.hs +++ b/libs/wire-api/src/Wire/API/Error/Brig.hs @@ -146,7 +146,7 @@ type instance 'StaticError 403 "mls-identity-mismatch" - "Prekey credential does not match qualified client ID" + "Key package credential does not match qualified client ID" -- | docs/reference/user/registration.md {#RefRestrictRegistration}. type instance MapError 'UserCreationRestricted = 'StaticError 403 "user-creation-restricted" "This instance does not allow creation of personal users or teams." diff --git a/libs/wire-api/src/Wire/API/Error/Galley.hs b/libs/wire-api/src/Wire/API/Error/Galley.hs index 1d9eaf4eb40..7ba3239c8b0 100644 --- a/libs/wire-api/src/Wire/API/Error/Galley.hs +++ b/libs/wire-api/src/Wire/API/Error/Galley.hs @@ -79,6 +79,8 @@ data GalleyError | MLSStaleMessage | MLSCommitMissingReferences | MLSSelfRemovalNotAllowed + | MLSGroupConversationMismatch + | MLSClientSenderUserMismatch | -- NoBindingTeamMembers | NoBindingTeam @@ -194,6 +196,10 @@ type instance MapError 'MLSCommitMissingReferences = 'StaticError 409 "mls-commi type instance MapError 'MLSSelfRemovalNotAllowed = 'StaticError 409 "mls-self-removal-not-allowed" "Self removal from group is not allowed" +type instance MapError 'MLSGroupConversationMismatch = 'StaticError 409 "mls-group-conversation-mismatch" "Conversation ID resolved from Group ID does not match submitted Conversation ID" + +type instance MapError 'MLSClientSenderUserMismatch = 'StaticError 409 "mls-client-sender-user-mismatch" "User ID resolved from Client ID does not match message's sender user ID" + type instance MapError 'NoBindingTeamMembers = 'StaticError 403 "non-binding-team-members" "Both users must be members of the same binding team" type instance MapError 'NoBindingTeam = 'StaticError 403 "no-binding-team" "Operation allowed only on binding teams" diff --git a/libs/wire-api/src/Wire/API/Event/Conversation.hs b/libs/wire-api/src/Wire/API/Event/Conversation.hs index 6534e7ff8ab..9571ed8d86c 100644 --- a/libs/wire-api/src/Wire/API/Event/Conversation.hs +++ b/libs/wire-api/src/Wire/API/Event/Conversation.hs @@ -79,12 +79,12 @@ import Data.Time import Imports import qualified Test.QuickCheck as QC import URI.ByteString () -import Wire.API.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) import Wire.API.Conversation import Wire.API.Conversation.Code (ConversationCode (..)) import Wire.API.Conversation.Role import Wire.API.Conversation.Typing (TypingData (..)) import Wire.API.User (QualifiedUserIdList (..)) +import Wire.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) -------------------------------------------------------------------------------- -- Event diff --git a/libs/wire-api/src/Wire/API/Event/Team.hs b/libs/wire-api/src/Wire/API/Event/Team.hs index 3dae7964b81..4dc8100e13a 100644 --- a/libs/wire-api/src/Wire/API/Event/Team.hs +++ b/libs/wire-api/src/Wire/API/Event/Team.hs @@ -54,9 +54,9 @@ import qualified Data.Swagger.Build.Api as Doc import Data.Time (UTCTime) import Imports import qualified Test.QuickCheck as QC -import Wire.API.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) import Wire.API.Team (Team, TeamUpdateData, modelUpdateData) import Wire.API.Team.Permission (Permissions) +import Wire.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) -------------------------------------------------------------------------------- -- Event diff --git a/libs/wire-api/src/Wire/API/MLS/CipherSuite.hs b/libs/wire-api/src/Wire/API/MLS/CipherSuite.hs index 3b31e199d60..9308a1b1e29 100644 --- a/libs/wire-api/src/Wire/API/MLS/CipherSuite.hs +++ b/libs/wire-api/src/Wire/API/MLS/CipherSuite.hs @@ -31,9 +31,9 @@ import qualified Data.Swagger as S import qualified Data.Swagger.Internal.Schema as S import Data.Word import Imports -import Wire.API.Arbitrary import Wire.API.MLS.Credential import Wire.API.MLS.Serialisation +import Wire.Arbitrary newtype CipherSuite = CipherSuite {cipherSuiteNumber :: Word16} deriving stock (Eq, Show) @@ -84,10 +84,11 @@ csHash MLS_128_DHKEMX25519_AES128GCM_SHA256_Ed25519 ctx value = HKDF.expand (HKDF.extract @SHA256 (mempty :: ByteString) value) ctx 16 csVerifySignature :: CipherSuiteTag -> ByteString -> ByteString -> ByteString -> Bool -csVerifySignature MLS_128_DHKEMX25519_AES128GCM_SHA256_Ed25519 pub x sig = fromMaybe False . maybeCryptoError $ do - pub' <- Ed25519.publicKey pub - sig' <- Ed25519.signature sig - pure $ Ed25519.verify pub' x sig' +csVerifySignature MLS_128_DHKEMX25519_AES128GCM_SHA256_Ed25519 pub x sig = + fromMaybe False . maybeCryptoError $ do + pub' <- Ed25519.publicKey pub + sig' <- Ed25519.signature sig + pure $ Ed25519.verify pub' x sig' csSignatureScheme :: CipherSuiteTag -> SignatureSchemeTag csSignatureScheme MLS_128_DHKEMX25519_AES128GCM_SHA256_Ed25519 = Ed25519 diff --git a/libs/wire-api/src/Wire/API/MLS/Commit.hs b/libs/wire-api/src/Wire/API/MLS/Commit.hs index a3c595d7078..7b4729cf6d1 100644 --- a/libs/wire-api/src/Wire/API/MLS/Commit.hs +++ b/libs/wire-api/src/Wire/API/MLS/Commit.hs @@ -26,6 +26,7 @@ data Commit = Commit { cProposals :: [ProposalOrRef], cPath :: Maybe UpdatePath } + deriving (Eq, Show) instance ParseMLS Commit where parseMLS = Commit <$> parseMLSVector @Word32 parseMLS <*> parseMLSOptional parseMLS @@ -34,6 +35,7 @@ data UpdatePath = UpdatePath { upLeaf :: RawMLS KeyPackage, upNodes :: [UpdatePathNode] } + deriving (Eq, Show) instance ParseMLS UpdatePath where parseMLS = UpdatePath <$> parseMLS <*> parseMLSVector @Word32 parseMLS @@ -42,6 +44,7 @@ data UpdatePathNode = UpdatePathNode { upnPublicKey :: ByteString, upnSecret :: [HPKECiphertext] } + deriving (Eq, Show) instance ParseMLS UpdatePathNode where parseMLS = UpdatePathNode <$> parseMLSBytes @Word16 <*> parseMLSVector @Word32 parseMLS @@ -50,6 +53,7 @@ data HPKECiphertext = HPKECiphertext { hcOutput :: ByteString, hcCiphertext :: ByteString } + deriving (Eq, Show) instance ParseMLS HPKECiphertext where parseMLS = HPKECiphertext <$> parseMLSBytes @Word16 <*> parseMLSBytes @Word16 diff --git a/libs/wire-api/src/Wire/API/MLS/Credential.hs b/libs/wire-api/src/Wire/API/MLS/Credential.hs index b2af79881aa..6cd03be33fc 100644 --- a/libs/wire-api/src/Wire/API/MLS/Credential.hs +++ b/libs/wire-api/src/Wire/API/MLS/Credential.hs @@ -24,6 +24,7 @@ import Control.Lens ((?~)) import Data.Aeson (FromJSON (..), FromJSONKey (..), ToJSON (..), ToJSONKey (..)) import qualified Data.Aeson as Aeson import qualified Data.Aeson.Types as Aeson +import Data.Bifunctor import Data.Binary import Data.Binary.Get import Data.Binary.Parser @@ -37,8 +38,8 @@ import qualified Data.Text as T import Data.UUID import Imports import Web.HttpApiData -import Wire.API.Arbitrary import Wire.API.MLS.Serialisation +import Wire.Arbitrary -- | An MLS credential. -- @@ -156,3 +157,41 @@ instance ParseMLS ClientIdentity where mkClientIdentity :: Qualified UserId -> ClientId -> ClientIdentity mkClientIdentity (Qualified uid domain) = ClientIdentity domain uid + +-- | Possible uses of a private key in the context of MLS. +data SignaturePurpose + = -- | Creating external remove proposals. + RemovalPurpose + deriving (Eq, Ord, Show, Bounded, Enum) + +signaturePurposeName :: SignaturePurpose -> Text +signaturePurposeName RemovalPurpose = "removal" + +signaturePurposeFromName :: Text -> Either String SignaturePurpose +signaturePurposeFromName name = + note ("Unsupported signature purpose " <> T.unpack name) + . getAlt + $ flip foldMap [minBound .. maxBound] $ \s -> + guard (signaturePurposeName s == name) $> s + +instance FromJSON SignaturePurpose where + parseJSON = + Aeson.withText "SignaturePurpose" $ + either fail pure . signaturePurposeFromName + +instance FromJSONKey SignaturePurpose where + fromJSONKey = + Aeson.FromJSONKeyTextParser $ + either fail pure . signaturePurposeFromName + +instance S.ToParamSchema SignaturePurpose where + toParamSchema _ = mempty & S.type_ ?~ S.SwaggerString + +instance FromHttpApiData SignaturePurpose where + parseQueryParam = first T.pack . signaturePurposeFromName + +instance ToJSON SignaturePurpose where + toJSON = Aeson.String . signaturePurposeName + +instance ToJSONKey SignaturePurpose where + toJSONKey = Aeson.toJSONKeyText signaturePurposeName diff --git a/libs/wire-api/src/Wire/API/MLS/Epoch.hs b/libs/wire-api/src/Wire/API/MLS/Epoch.hs index 79c27bae432..c5fae61fa99 100644 --- a/libs/wire-api/src/Wire/API/MLS/Epoch.hs +++ b/libs/wire-api/src/Wire/API/MLS/Epoch.hs @@ -19,10 +19,11 @@ module Wire.API.MLS.Epoch where +import Data.Binary import Data.Schema import Imports -import Wire.API.Arbitrary import Wire.API.MLS.Serialisation +import Wire.Arbitrary newtype Epoch = Epoch {epochNumber :: Word64} deriving stock (Eq, Show) @@ -30,3 +31,6 @@ newtype Epoch = Epoch {epochNumber :: Word64} instance ParseMLS Epoch where parseMLS = Epoch <$> parseMLS + +instance SerialiseMLS Epoch where + serialiseMLS (Epoch n) = put n diff --git a/libs/wire-api/src/Wire/API/MLS/Extension.hs b/libs/wire-api/src/Wire/API/MLS/Extension.hs index 5899a916b52..18b1d551d2d 100644 --- a/libs/wire-api/src/Wire/API/MLS/Extension.hs +++ b/libs/wire-api/src/Wire/API/MLS/Extension.hs @@ -47,9 +47,9 @@ import Data.Binary import Data.Singletons.TH import Data.Time.Clock.POSIX import Imports -import Wire.API.Arbitrary import Wire.API.MLS.CipherSuite import Wire.API.MLS.Serialisation +import Wire.Arbitrary newtype ProtocolVersion = ProtocolVersion {pvNumber :: Word8} deriving newtype (Eq, Ord, Show, Binary, Arbitrary) @@ -75,6 +75,11 @@ data Extension = Extension instance ParseMLS Extension where parseMLS = Extension <$> parseMLS <*> parseMLSBytes @Word32 +instance SerialiseMLS Extension where + serialiseMLS (Extension ty d) = do + serialiseMLS ty + serialiseMLSBytes @Word32 d + data ExtensionTag = CapabilitiesExtensionTag | LifetimeExtensionTag diff --git a/libs/wire-api/src/Wire/API/MLS/Group.hs b/libs/wire-api/src/Wire/API/MLS/Group.hs index 7935726e2fe..54a5fff5138 100644 --- a/libs/wire-api/src/Wire/API/MLS/Group.hs +++ b/libs/wire-api/src/Wire/API/MLS/Group.hs @@ -27,8 +27,8 @@ import Data.Qualified import Data.Schema import qualified Data.Swagger as S import Imports -import Wire.API.Arbitrary import Wire.API.MLS.Serialisation +import Wire.Arbitrary newtype GroupId = GroupId {unGroupId :: ByteString} deriving (Eq, Show, Generic) @@ -41,6 +41,9 @@ instance IsString GroupId where instance ParseMLS GroupId where parseMLS = GroupId <$> parseMLSBytes @Word8 +instance SerialiseMLS GroupId where + serialiseMLS (GroupId gid) = serialiseMLSBytes @Word8 gid + instance ToSchema GroupId where schema = GroupId diff --git a/libs/wire-api/src/Wire/API/MLS/KeyPackage.hs b/libs/wire-api/src/Wire/API/MLS/KeyPackage.hs index ceb4ead408f..3d39d778c5f 100644 --- a/libs/wire-api/src/Wire/API/MLS/KeyPackage.hs +++ b/libs/wire-api/src/Wire/API/MLS/KeyPackage.hs @@ -42,19 +42,22 @@ import Control.Lens hiding (set, (.=)) import Data.Aeson (FromJSON, ToJSON) import Data.Binary import Data.Binary.Get +import Data.Binary.Put +import qualified Data.ByteString as B import Data.Id import Data.Json.Util import Data.Qualified import Data.Schema import qualified Data.Swagger as S import Imports +import Test.QuickCheck import Web.HttpApiData -import Wire.API.Arbitrary import Wire.API.MLS.CipherSuite import Wire.API.MLS.Context import Wire.API.MLS.Credential import Wire.API.MLS.Extension import Wire.API.MLS.Serialisation +import Wire.Arbitrary data KeyPackageUpload = KeyPackageUpload {kpuKeyPackages :: [RawMLS KeyPackage]} @@ -117,12 +120,18 @@ newtype KeyPackageRef = KeyPackageRef {unKeyPackageRef :: ByteString} deriving (FromHttpApiData, ToHttpApiData, S.ToParamSchema) via Base64ByteString deriving (ToJSON, FromJSON, S.ToSchema) via (Schema KeyPackageRef) +instance Arbitrary KeyPackageRef where + arbitrary = KeyPackageRef . B.pack <$> vectorOf 16 arbitrary + instance ToSchema KeyPackageRef where schema = named "KeyPackageRef" $ unKeyPackageRef .= fmap KeyPackageRef base64Schema instance ParseMLS KeyPackageRef where parseMLS = KeyPackageRef <$> getByteString 16 +instance SerialiseMLS KeyPackageRef where + serialiseMLS = putByteString . unKeyPackageRef + -- | Compute key package ref given a ciphersuite and the raw key package data. kpRef :: CipherSuiteTag -> KeyPackageData -> KeyPackageRef kpRef cs = diff --git a/libs/wire-api/src/Wire/API/MLS/Keys.hs b/libs/wire-api/src/Wire/API/MLS/Keys.hs new file mode 100644 index 00000000000..df7989ffd95 --- /dev/null +++ b/libs/wire-api/src/Wire/API/MLS/Keys.hs @@ -0,0 +1,66 @@ +{-# LANGUAGE GeneralizedNewtypeDeriving #-} + +-- This file is part of the Wire Server implementation. +-- +-- Copyright (C) 2022 Wire Swiss GmbH +-- +-- This program is free software: you can redistribute it and/or modify it under +-- the terms of the GNU Affero General Public License as published by the Free +-- Software Foundation, either version 3 of the License, or (at your option) any +-- later version. +-- +-- This program is distributed in the hope that it will be useful, but WITHOUT +-- ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +-- FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more +-- details. +-- +-- You should have received a copy of the GNU Affero General Public License along +-- with this program. If not, see . + +module Wire.API.MLS.Keys + ( MLSKeys (..), + MLSPublicKeys (..), + mlsKeysToPublic, + ) +where + +import Crypto.PubKey.Ed25519 +import Data.Aeson (FromJSON (..), ToJSON (..)) +import Data.ByteArray +import Data.Json.Util +import qualified Data.Map as Map +import Data.Schema +import qualified Data.Swagger as S +import Imports +import Wire.API.MLS.Credential + +data MLSKeys = MLSKeys + { mlsKeyPair_ed25519 :: Maybe (SecretKey, PublicKey) + } + +instance Semigroup MLSKeys where + MLSKeys Nothing <> MLSKeys ed2 = MLSKeys ed2 + MLSKeys ed1 <> MLSKeys _ = MLSKeys ed1 + +instance Monoid MLSKeys where + mempty = MLSKeys Nothing + +newtype MLSPublicKeys = MLSPublicKeys + { unMLSPublicKeys :: Map SignaturePurpose (Map SignatureSchemeTag ByteString) + } + deriving (FromJSON, ToJSON, S.ToSchema) via Schema MLSPublicKeys + deriving newtype (Semigroup, Monoid) + +instance ToSchema MLSPublicKeys where + schema = + named "MLSKeys" $ + MLSPublicKeys <$> unMLSPublicKeys + .= map_ (map_ base64Schema) + +mlsKeysToPublic1 :: MLSKeys -> Map SignatureSchemeTag ByteString +mlsKeysToPublic1 (MLSKeys mEd25519key) = + fold $ Map.singleton Ed25519 . convert . snd <$> mEd25519key + +mlsKeysToPublic :: (SignaturePurpose -> MLSKeys) -> MLSPublicKeys +mlsKeysToPublic f = flip foldMap [minBound .. maxBound] $ \purpose -> + MLSPublicKeys (Map.singleton purpose (mlsKeysToPublic1 (f purpose))) diff --git a/libs/wire-api/src/Wire/API/MLS/Message.hs b/libs/wire-api/src/Wire/API/MLS/Message.hs index 4cda326cf01..721f63c9c35 100644 --- a/libs/wire-api/src/Wire/API/MLS/Message.hs +++ b/libs/wire-api/src/Wire/API/MLS/Message.hs @@ -36,19 +36,27 @@ module Wire.API.MLS.Message MLSPlainTextSym0, MLSCipherTextSym0, MLSMessageSendingStatus (..), + KnownFormatTag (..), verifyMessageSignature, + mkRemoveProposalMessage, ) where import Control.Lens ((?~)) +import Crypto.Error +import Crypto.PubKey.Ed25519 import qualified Data.Aeson as A import Data.Binary import Data.Binary.Get +import Data.Binary.Put +import qualified Data.ByteArray as BA +-- import qualified Data.ByteString as BS import Data.Json.Util import Data.Schema import Data.Singletons.TH import qualified Data.Swagger as S import Imports +import Test.QuickCheck hiding (label) import Wire.API.Event.Conversation import Wire.API.MLS.CipherSuite import Wire.API.MLS.Commit @@ -57,6 +65,7 @@ import Wire.API.MLS.Group import Wire.API.MLS.KeyPackage import Wire.API.MLS.Proposal import Wire.API.MLS.Serialisation +import Wire.Arbitrary (GenericUniform (..)) data WireFormatTag = MLSPlainText | MLSCipherText deriving (Bounded, Enum, Eq, Show) @@ -73,26 +82,55 @@ data instance MessageExtraFields 'MLSPlainText = MessageExtraFields msgConfirmation :: Maybe ByteString, msgMembership :: Maybe ByteString } + deriving (Generic) + deriving (Arbitrary) via (GenericUniform (MessageExtraFields 'MLSPlainText)) instance ParseMLS (MessageExtraFields 'MLSPlainText) where parseMLS = MessageExtraFields - <$> parseMLSBytes @Word16 - <*> parseMLSOptional (parseMLSBytes @Word8) - <*> parseMLSOptional (parseMLSBytes @Word8) + <$> label "msgSignature" (parseMLSBytes @Word16) + <*> label "msgConfirmation" (parseMLSOptional (parseMLSBytes @Word8)) + <*> label "msgMembership" (parseMLSOptional (parseMLSBytes @Word8)) + +instance SerialiseMLS (MessageExtraFields 'MLSPlainText) where + serialiseMLS (MessageExtraFields sig mconf mmemb) = do + serialiseMLSBytes @Word16 sig + serialiseMLSOptional (serialiseMLSBytes @Word8) mconf + serialiseMLSOptional (serialiseMLSBytes @Word8) mmemb data instance MessageExtraFields 'MLSCipherText = NoExtraFields instance ParseMLS (MessageExtraFields 'MLSCipherText) where parseMLS = pure NoExtraFields +deriving instance Eq (MessageExtraFields 'MLSPlainText) + +deriving instance Eq (MessageExtraFields 'MLSCipherText) + +deriving instance Show (MessageExtraFields 'MLSPlainText) + +deriving instance Show (MessageExtraFields 'MLSCipherText) + data Message (tag :: WireFormatTag) = Message { msgTBS :: RawMLS (MessageTBS tag), msgExtraFields :: MessageExtraFields tag } +deriving instance Eq (Message 'MLSPlainText) + +deriving instance Eq (Message 'MLSCipherText) + +deriving instance Show (Message 'MLSPlainText) + +deriving instance Show (Message 'MLSCipherText) + instance ParseMLS (Message 'MLSPlainText) where - parseMLS = Message <$> parseMLS <*> parseMLS + parseMLS = Message <$> label "tbs" parseMLS <*> label "MessageExtraFields" parseMLS + +instance SerialiseMLS (Message 'MLSPlainText) where + serialiseMLS (Message msgTBS msgExtraFields) = do + putByteString (rmRaw msgTBS) + serialiseMLS msgExtraFields instance ParseMLS (Message 'MLSCipherText) where parseMLS = Message <$> parseMLS <*> parseMLS @@ -105,6 +143,20 @@ data KnownFormatTag (tag :: WireFormatTag) = KnownFormatTag instance ParseMLS (KnownFormatTag tag) where parseMLS = parseMLS @WireFormatTag $> KnownFormatTag +instance SerialiseMLS (KnownFormatTag 'MLSPlainText) where + serialiseMLS _ = put (fromMLSEnum @Word8 MLSPlainText) + +instance SerialiseMLS (KnownFormatTag 'MLSCipherText) where + serialiseMLS _ = put (fromMLSEnum @Word8 MLSCipherText) + +deriving instance Eq (KnownFormatTag 'MLSPlainText) + +deriving instance Eq (KnownFormatTag 'MLSCipherText) + +deriving instance Show (KnownFormatTag 'MLSPlainText) + +deriving instance Show (KnownFormatTag 'MLSCipherText) + data MessageTBS (tag :: WireFormatTag) = MessageTBS { tbsMsgFormat :: KnownFormatTag tag, tbsMsgGroupId :: GroupId, @@ -146,6 +198,23 @@ instance ParseMLS (MessageTBS 'MLSCipherText) where p <- parseMLSBytes @Word32 pure $ MessageTBS f g e d s (CipherText ct p) +instance SerialiseMLS (MessageTBS 'MLSPlainText) where + serialiseMLS (MessageTBS f g e d s p) = do + serialiseMLS f + serialiseMLS g + serialiseMLS e + serialiseMLS s + serialiseMLSBytes @Word32 d + serialiseMLS p + +deriving instance Eq (MessageTBS 'MLSPlainText) + +deriving instance Eq (MessageTBS 'MLSCipherText) + +deriving instance Show (MessageTBS 'MLSPlainText) + +deriving instance Show (MessageTBS 'MLSCipherText) + data SomeMessage where SomeMessage :: Sing tag -> Message tag -> SomeMessage @@ -161,6 +230,7 @@ instance ParseMLS SomeMessage where data family Sender (tag :: WireFormatTag) :: * data instance Sender 'MLSCipherText = EncryptedSender {esData :: ByteString} + deriving (Eq, Show) instance ParseMLS (Sender 'MLSCipherText) where parseMLS = EncryptedSender <$> parseMLSBytes @Word8 @@ -171,20 +241,44 @@ data SenderTag = MemberSenderTag | PreconfiguredSenderTag | NewMemberSenderTag instance ParseMLS SenderTag where parseMLS = parseMLSEnum @Word8 "sender type" +instance SerialiseMLS SenderTag where + serialiseMLS = serialiseMLSEnum @Word8 + +-- NOTE: according to the spec, the preconfigured sender case contains a +-- bytestring, not a u32. However, as of 2022-08-02, the openmls fork used by +-- the clients is using a u32 here. data instance Sender 'MLSPlainText = MemberSender KeyPackageRef - | PreconfiguredSender ByteString + | PreconfiguredSender Word32 | NewMemberSender + deriving (Eq, Show, Generic) instance ParseMLS (Sender 'MLSPlainText) where parseMLS = parseMLS >>= \case MemberSenderTag -> MemberSender <$> parseMLS - PreconfiguredSenderTag -> PreconfiguredSender <$> parseMLSBytes @Word8 + PreconfiguredSenderTag -> PreconfiguredSender <$> get NewMemberSenderTag -> pure NewMemberSender +instance SerialiseMLS (Sender 'MLSPlainText) where + serialiseMLS (MemberSender r) = do + serialiseMLS MemberSenderTag + serialiseMLS r + serialiseMLS (PreconfiguredSender x) = do + serialiseMLS PreconfiguredSenderTag + put x + serialiseMLS NewMemberSender = serialiseMLS NewMemberSender + data family MessagePayload (tag :: WireFormatTag) :: * +deriving instance Eq (MessagePayload 'MLSPlainText) + +deriving instance Eq (MessagePayload 'MLSCipherText) + +deriving instance Show (MessagePayload 'MLSPlainText) + +deriving instance Show (MessagePayload 'MLSCipherText) + data instance MessagePayload 'MLSCipherText = CipherText { msgContentType :: Word8, msgCipherText :: ByteString @@ -211,6 +305,17 @@ instance ParseMLS (MessagePayload 'MLSPlainText) where ProposalMessageTag -> ProposalMessage <$> parseMLS CommitMessageTag -> CommitMessage <$> parseMLS +instance SerialiseMLS ContentType where + serialiseMLS = serialiseMLSEnum @Word8 + +instance SerialiseMLS (MessagePayload 'MLSPlainText) where + serialiseMLS (ProposalMessage raw) = do + serialiseMLS ProposalMessageTag + putByteString (rmRaw raw) + -- We do not need to serialise Commit and Application messages, + -- so the next case is left as a stub + serialiseMLS _ = pure () + data MLSMessageSendingStatus = MLSMessageSendingStatus { mmssEvents :: [Event], mmssTime :: UTCTimeMillis @@ -235,3 +340,24 @@ instance ToSchema MLSMessageSendingStatus where verifyMessageSignature :: CipherSuiteTag -> Message 'MLSPlainText -> ByteString -> Bool verifyMessageSignature cs msg pubkey = csVerifySignature cs pubkey (rmRaw (msgTBS msg)) (msgSignature (msgExtraFields msg)) + +mkRemoveProposalMessage :: + SecretKey -> + PublicKey -> + GroupId -> + Epoch -> + KeyPackageRef -> + Maybe (Message 'MLSPlainText) +mkRemoveProposalMessage priv pub gid epoch ref = maybeCryptoError $ do + let tbs = + mkRawMLS $ + MessageTBS + { tbsMsgFormat = KnownFormatTag, + tbsMsgGroupId = gid, + tbsMsgEpoch = epoch, + tbsMsgAuthData = mempty, + tbsMsgSender = PreconfiguredSender 0, + tbsMsgPayload = ProposalMessage (mkRemoveProposal ref) + } + let sig = BA.convert $ sign priv pub (rmRaw tbs) + pure (Message tbs (MessageExtraFields sig Nothing Nothing)) diff --git a/libs/wire-api/src/Wire/API/MLS/Proposal.hs b/libs/wire-api/src/Wire/API/MLS/Proposal.hs index 6f6e43a705c..4264a319183 100644 --- a/libs/wire-api/src/Wire/API/MLS/Proposal.hs +++ b/libs/wire-api/src/Wire/API/MLS/Proposal.hs @@ -1,3 +1,4 @@ +{-# LANGUAGE RecordWildCards #-} -- This file is part of the Wire Server implementation. -- -- Copyright (C) 2022 Wire Swiss GmbH @@ -18,17 +19,20 @@ module Wire.API.MLS.Proposal where +import Control.Arrow import Control.Lens (makePrisms) import Data.Binary import Data.Binary.Get +import Data.Binary.Put +import qualified Data.ByteString.Lazy as LBS import Imports -import Wire.API.Arbitrary import Wire.API.MLS.CipherSuite import Wire.API.MLS.Context import Wire.API.MLS.Extension import Wire.API.MLS.Group import Wire.API.MLS.KeyPackage import Wire.API.MLS.Serialisation +import Wire.Arbitrary data ProposalTag = AddProposalTag @@ -45,6 +49,9 @@ data ProposalTag instance ParseMLS ProposalTag where parseMLS = parseMLSEnum @Word16 "proposal type" +instance SerialiseMLS ProposalTag where + serialiseMLS = serialiseMLSEnum @Word16 + data Proposal = AddProposal (RawMLS KeyPackage) | UpdateProposal KeyPackage @@ -69,6 +76,23 @@ instance ParseMLS Proposal where GroupContextExtensionsProposalTag -> GroupContextExtensionsProposal <$> parseMLSVector @Word32 parseMLS +mkRemoveProposal :: KeyPackageRef -> RawMLS Proposal +mkRemoveProposal ref = RawMLS bytes (RemoveProposal ref) + where + bytes = LBS.toStrict . runPut $ do + serialiseMLS RemoveProposalTag + serialiseMLS ref + +serialiseAppAckProposal :: [MessageRange] -> Put +serialiseAppAckProposal mrs = do + serialiseMLS AppAckProposalTag + serialiseMLSVector @Word32 serialiseMLS mrs + +mkAppAckProposal :: [MessageRange] -> RawMLS Proposal +mkAppAckProposal = uncurry RawMLS . (bytes &&& AppAckProposal) + where + bytes = LBS.toStrict . runPut . serialiseAppAckProposal + -- | Compute the proposal ref given a ciphersuite and the raw proposal data. proposalRef :: CipherSuiteTag -> RawMLS Proposal -> ProposalRef proposalRef cs = @@ -125,10 +149,13 @@ instance ParseMLS ReInit where data MessageRange = MessageRange { mrSender :: KeyPackageRef, mrFirstGeneration :: Word32, - mrLastGenereation :: Word32 + mrLastGeneration :: Word32 } deriving stock (Eq, Show) +instance Arbitrary MessageRange where + arbitrary = MessageRange <$> arbitrary <*> arbitrary <*> arbitrary + instance ParseMLS MessageRange where parseMLS = MessageRange @@ -136,6 +163,12 @@ instance ParseMLS MessageRange where <*> parseMLS <*> parseMLS +instance SerialiseMLS MessageRange where + serialiseMLS MessageRange {..} = do + serialiseMLS mrSender + serialiseMLS mrFirstGeneration + serialiseMLS mrLastGeneration + data ProposalOrRefTag = InlineTag | RefTag deriving stock (Bounded, Enum, Eq, Show) diff --git a/libs/wire-api/src/Wire/API/MLS/Serialisation.hs b/libs/wire-api/src/Wire/API/MLS/Serialisation.hs index cc823d304bc..6510ac31008 100644 --- a/libs/wire-api/src/Wire/API/MLS/Serialisation.hs +++ b/libs/wire-api/src/Wire/API/MLS/Serialisation.hs @@ -17,10 +17,16 @@ module Wire.API.MLS.Serialisation ( ParseMLS (..), + SerialiseMLS (..), parseMLSVector, + serialiseMLSVector, parseMLSBytes, + serialiseMLSBytes, + serialiseMLSBytesLazy, parseMLSOptional, + serialiseMLSOptional, parseMLSEnum, + serialiseMLSEnum, BinaryMLS (..), MLSEnumError (..), fromMLSEnum, @@ -34,6 +40,7 @@ module Wire.API.MLS.Serialisation rawMLSSchema, mlsSwagger, parseRawMLS, + mkRawMLS, ) where @@ -44,7 +51,10 @@ import Data.Aeson (FromJSON (..)) import qualified Data.Aeson as Aeson import Data.Bifunctor import Data.Binary +import Data.Binary.Builder import Data.Binary.Get +import Data.Binary.Put +import qualified Data.ByteString as BS import qualified Data.ByteString.Lazy as LBS import Data.Json.Util import Data.Proxy @@ -57,6 +67,10 @@ import Imports class ParseMLS a where parseMLS :: Get a +-- | Convert a value to "TLS presentation" format. +class SerialiseMLS a where + serialiseMLS :: a -> Put + parseMLSVector :: forall w a. (Binary w, Integral w) => Get a -> Get [a] parseMLSVector getItem = do len <- get @w @@ -70,16 +84,41 @@ parseMLSVector getItem = do pos <- bytesRead (:) x <$> (if pos < endPos then go endPos else pure []) +serialiseMLSVector :: + forall w a. + (Binary w, Integral w) => + (a -> Put) -> + [a] -> + Put +serialiseMLSVector p = + serialiseMLSBytesLazy @w . toLazyByteString . execPut . traverse_ p + parseMLSBytes :: forall w. (Binary w, Integral w) => Get ByteString parseMLSBytes = do len <- fromIntegral <$> get @w getByteString len +serialiseMLSBytes :: forall w. (Binary w, Integral w) => ByteString -> Put +serialiseMLSBytes x = do + put @w (fromIntegral (BS.length x)) + putByteString x + +serialiseMLSBytesLazy :: forall w. (Binary w, Integral w) => LBS.ByteString -> Put +serialiseMLSBytesLazy x = do + put @w (fromIntegral (LBS.length x)) + putLazyByteString x + parseMLSOptional :: Get a -> Get (Maybe a) parseMLSOptional g = do b <- getWord8 sequenceA $ guard (b /= 0) $> g +serialiseMLSOptional :: (a -> Put) -> Maybe a -> Put +serialiseMLSOptional _p Nothing = putWord8 0 +serialiseMLSOptional p (Just x) = do + putWord8 1 + p x + -- | Parse a positive tag for an enumeration. The value 0 is considered -- "reserved", and all other values are shifted down by 1 to get the -- corresponding enumeration index. This makes it possible to parse enumeration @@ -91,6 +130,13 @@ parseMLSEnum :: Get a parseMLSEnum name = toMLSEnum name =<< get @w +serialiseMLSEnum :: + forall w a. + (Enum a, Integral w, Binary w) => + a -> + Put +serialiseMLSEnum = put . fromMLSEnum @w + data MLSEnumError = MLSEnumUnknown | MLSEnumInvalid toMLSEnum' :: forall a w. (Bounded a, Enum a, Integral w) => w -> Either MLSEnumError a @@ -117,6 +163,10 @@ instance ParseMLS Word32 where parseMLS = get instance ParseMLS Word64 where parseMLS = get +instance SerialiseMLS Word16 where serialiseMLS = put + +instance SerialiseMLS Word32 where serialiseMLS = put + -- | A wrapper to generate a 'ParseMLS' instance given a 'Binary' instance. newtype BinaryMLS a = BinaryMLS a @@ -201,3 +251,9 @@ parseRawMLS p = do instance ParseMLS a => ParseMLS (RawMLS a) where parseMLS = parseRawMLS parseMLS + +instance SerialiseMLS (RawMLS a) where + serialiseMLS = putByteString . rmRaw + +mkRawMLS :: SerialiseMLS a => a -> RawMLS a +mkRawMLS x = RawMLS (LBS.toStrict (runPut (serialiseMLS x))) x diff --git a/libs/wire-api/src/Wire/API/Message.hs b/libs/wire-api/src/Wire/API/Message.hs index 173dd5aad7d..f5676a013f2 100644 --- a/libs/wire-api/src/Wire/API/Message.hs +++ b/libs/wire-api/src/Wire/API/Message.hs @@ -89,10 +89,10 @@ import Imports import qualified Proto.Otr import qualified Proto.Otr_Fields as Proto.Otr import Servant (FromHttpApiData (..)) -import Wire.API.Arbitrary (Arbitrary (..), GenericUniform (..)) import qualified Wire.API.Message.Proto as Proto import Wire.API.ServantProto (FromProto (..), ToProto (..)) import Wire.API.User.Client (QualifiedUserClientMap (QualifiedUserClientMap), QualifiedUserClients, UserClientMap (..), UserClients (..), modelOtrClientMap, modelUserClients) +import Wire.Arbitrary (Arbitrary (..), GenericUniform (..)) -------------------------------------------------------------------------------- -- Message diff --git a/libs/wire-api/src/Wire/API/Notification.hs b/libs/wire-api/src/Wire/API/Notification.hs index d46ed7bfe43..04a2c8c55ff 100644 --- a/libs/wire-api/src/Wire/API/Notification.hs +++ b/libs/wire-api/src/Wire/API/Notification.hs @@ -51,7 +51,7 @@ import qualified Data.Swagger as S import qualified Data.Swagger.Build.Api as Doc import Data.Time.Clock (UTCTime) import Imports -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) type NotificationId = Id QueuedNotification diff --git a/libs/wire-api/src/Wire/API/Properties.hs b/libs/wire-api/src/Wire/API/Properties.hs index 6297e62ef2a..83a04aa07c0 100644 --- a/libs/wire-api/src/Wire/API/Properties.hs +++ b/libs/wire-api/src/Wire/API/Properties.hs @@ -39,7 +39,7 @@ import qualified Data.Swagger.Build.Api as Doc import Data.Text.Ascii import Imports import Servant -import Wire.API.Arbitrary (Arbitrary) +import Wire.Arbitrary (Arbitrary) newtype PropertyKeysAndValues = PropertyKeysAndValues (Map PropertyKey PropertyValue) deriving newtype (ToJSON) diff --git a/libs/wire-api/src/Wire/API/Provider.hs b/libs/wire-api/src/Wire/API/Provider.hs index f6f0314810f..146517bf9ce 100644 --- a/libs/wire-api/src/Wire/API/Provider.hs +++ b/libs/wire-api/src/Wire/API/Provider.hs @@ -59,12 +59,12 @@ import Data.Json.Util import Data.Misc (HttpsUrl (..), PlainTextPassword (..)) import Data.Range import Imports -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) import Wire.API.Conversation.Code as Code import Wire.API.Provider.Service (ServiceToken (..)) import Wire.API.Provider.Service.Tag (ServiceTag (..)) import Wire.API.User.Identity (Email) import Wire.API.User.Profile (Name) +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) -------------------------------------------------------------------------------- -- Provider diff --git a/libs/wire-api/src/Wire/API/Provider/Bot.hs b/libs/wire-api/src/Wire/API/Provider/Bot.hs index cd51e2c405f..b8aabf3af01 100644 --- a/libs/wire-api/src/Wire/API/Provider/Bot.hs +++ b/libs/wire-api/src/Wire/API/Provider/Bot.hs @@ -36,9 +36,9 @@ import Data.Handle (Handle) import Data.Id import Data.Json.Util ((#)) import Imports -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) import Wire.API.Conversation.Member (OtherMember (..)) import Wire.API.User.Profile (ColourId, Name) +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) -------------------------------------------------------------------------------- -- BotConvView diff --git a/libs/wire-api/src/Wire/API/Provider/External.hs b/libs/wire-api/src/Wire/API/Provider/External.hs index 8fb9e8f698f..246b6aa5317 100644 --- a/libs/wire-api/src/Wire/API/Provider/External.hs +++ b/libs/wire-api/src/Wire/API/Provider/External.hs @@ -27,10 +27,10 @@ import Data.Aeson import Data.Id import Data.Json.Util ((#)) import Imports -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) import Wire.API.Provider.Bot (BotConvView, BotUserView) import Wire.API.User.Client.Prekey (LastPrekey, Prekey) import Wire.API.User.Profile (Asset, ColourId, Locale, Name) +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) -------------------------------------------------------------------------------- -- NewBotRequest diff --git a/libs/wire-api/src/Wire/API/Provider/Service.hs b/libs/wire-api/src/Wire/API/Provider/Service.hs index c533c69639b..329c1e0ad6b 100644 --- a/libs/wire-api/src/Wire/API/Provider/Service.hs +++ b/libs/wire-api/src/Wire/API/Provider/Service.hs @@ -74,9 +74,9 @@ import qualified Data.Text as Text import Data.Text.Ascii import qualified Data.Text.Encoding as Text import Imports -import Wire.API.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) import Wire.API.Provider.Service.Tag (ServiceTag (..)) import Wire.API.User.Profile (Asset, Name) +import Wire.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) -------------------------------------------------------------------------------- -- ServiceRef diff --git a/libs/wire-api/src/Wire/API/Provider/Service/Tag.hs b/libs/wire-api/src/Wire/API/Provider/Service/Tag.hs index 9029be7e13d..60be42a4273 100644 --- a/libs/wire-api/src/Wire/API/Provider/Service/Tag.hs +++ b/libs/wire-api/src/Wire/API/Provider/Service/Tag.hs @@ -50,7 +50,7 @@ import qualified Data.Set as Set import qualified Data.Text.Encoding as Text import GHC.TypeLits (KnownNat, Nat) import Imports -import Wire.API.Arbitrary (Arbitrary (..), GenericUniform (..)) +import Wire.Arbitrary (Arbitrary (..), GenericUniform (..)) -------------------------------------------------------------------------------- -- ServiceTag diff --git a/libs/wire-api/src/Wire/API/Push/V2/Token.hs b/libs/wire-api/src/Wire/API/Push/V2/Token.hs index fec07c9f765..ef3cc837545 100644 --- a/libs/wire-api/src/Wire/API/Push/V2/Token.hs +++ b/libs/wire-api/src/Wire/API/Push/V2/Token.hs @@ -49,7 +49,7 @@ import Data.Id import Data.Json.Util import qualified Data.Swagger.Build.Api as Doc import Imports -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) -------------------------------------------------------------------------------- -- PushToken diff --git a/libs/wire-api/src/Wire/API/Routes/Internal/Brig.hs b/libs/wire-api/src/Wire/API/Routes/Internal/Brig.hs index 08e00501e38..eec1d208efd 100644 --- a/libs/wire-api/src/Wire/API/Routes/Internal/Brig.hs +++ b/libs/wire-api/src/Wire/API/Routes/Internal/Brig.hs @@ -59,6 +59,7 @@ import Wire.API.Routes.MultiVerb import Wire.API.Routes.Named import Wire.API.Team.Feature import Wire.API.User +import Wire.API.User.Client type EJPDRequest = Summary @@ -244,7 +245,7 @@ type GetConversationByKeyPackageRef = ) type GetMLSClients = - Summary "Return all MLS-enabled clients of a user" + Summary "Return all clients and all MLS-capable clients of a user" :> "clients" :> CanThrow 'UserNotFound :> Capture "user" UserId @@ -252,7 +253,7 @@ type GetMLSClients = :> MultiVerb1 'GET '[Servant.JSON] - (Respond 200 "MLS clients" (Set ClientId)) + (Respond 200 "MLS clients" (Set ClientInfo)) type MapKeyPackageRefs = Summary "Insert bundle into the KeyPackage ref mapping. Only for tests." diff --git a/libs/wire-api/src/Wire/API/Routes/Internal/Brig/EJPD.hs b/libs/wire-api/src/Wire/API/Routes/Internal/Brig/EJPD.hs index cfaa67ed9ce..efd26df2ee0 100644 --- a/libs/wire-api/src/Wire/API/Routes/Internal/Brig/EJPD.hs +++ b/libs/wire-api/src/Wire/API/Routes/Internal/Brig/EJPD.hs @@ -32,11 +32,11 @@ import Data.Swagger (ToSchema) import Deriving.Swagger (CamelToSnake, CustomSwagger (..), FieldLabelModifier, StripSuffix) import Imports hiding (head) import Test.QuickCheck (Arbitrary) -import Wire.API.Arbitrary (GenericUniform (..)) import Wire.API.Connection (Relation) import Wire.API.Team.Member (NewListType) import Wire.API.User.Identity (Email, Phone) import Wire.API.User.Profile (Name) +import Wire.Arbitrary (GenericUniform (..)) newtype EJPDRequestBody = EJPDRequestBody {ejpdRequestBody :: [Handle]} deriving stock (Eq, Show, Generic) diff --git a/libs/wire-api/src/Wire/API/Routes/Public/Brig.hs b/libs/wire-api/src/Wire/API/Routes/Public/Brig.hs index 8cd0455dd5a..0bc1a9ca28b 100644 --- a/libs/wire-api/src/Wire/API/Routes/Public/Brig.hs +++ b/libs/wire-api/src/Wire/API/Routes/Public/Brig.hs @@ -19,15 +19,17 @@ module Wire.API.Routes.Public.Brig where +import Data.ByteString.Conversion import Data.Code (Timeout) import Data.CommaSeparatedList (CommaSeparatedList) import Data.Domain import Data.Handle import Data.Id as Id import Data.Misc (IpAddr) +import Data.Nonce (Nonce) import Data.Qualified (Qualified (..)) import Data.Range -import Data.SOP (I (..), NS (..)) +import Data.SOP import Data.Swagger hiding (Contact, Header) import Imports hiding (head) import Servant (JSON) @@ -533,6 +535,34 @@ type UserClientAPI = :> "prekeys" :> Get '[JSON] [PrekeyId] ) + :<|> NonceAPI + +type NonceAPI = + -- be aware that the order matters, if get was first, then head requests would be routed to the get handler + NewNonce "head-nonce" 'HEAD 200 + :<|> NewNonce "get-nonce" 'GET 204 + +type NewNonce name method statusCode = + Named + name + ( Summary "Get a new nonce for a client CSR, specified in the response header `Replay-Nonce` as a uuidv4 in base64url encoding" + :> ZUser + :> "clients" + :> CaptureClientId "client" + :> "nonce" + :> MultiVerb1 + method + '[JSON] + (WithHeaders '[Header "Replay-Nonce" NonceHeader, Header "Cache-Control" Text] Nonce (RespondEmpty statusCode "No Content")) + ) + +newtype NonceHeader = NonceHeader Nonce + deriving (Eq, Show) + deriving newtype (FromByteString, ToByteString, ToParamSchema, ToHttpApiData, FromHttpApiData) + +instance AsHeaders '[NonceHeader, Text] () Nonce where + fromHeaders (I (NonceHeader nonce) :* (_ :* Nil), _) = nonce + toHeaders nonce = (I (NonceHeader nonce) :* (I "no-store" :* Nil), ()) type ClientAPI = Named diff --git a/libs/wire-api/src/Wire/API/Routes/Public/Galley.hs b/libs/wire-api/src/Wire/API/Routes/Public/Galley.hs index 45d6886809c..719fc9b33fc 100644 --- a/libs/wire-api/src/Wire/API/Routes/Public/Galley.hs +++ b/libs/wire-api/src/Wire/API/Routes/Public/Galley.hs @@ -41,6 +41,7 @@ import Wire.API.Error import qualified Wire.API.Error.Brig as BrigError import Wire.API.Error.Galley import Wire.API.Event.Conversation +import Wire.API.MLS.Keys import Wire.API.MLS.Message import Wire.API.MLS.Serialisation import Wire.API.MLS.Servant @@ -1332,6 +1333,8 @@ type MLSMessagingAPI = :> CanThrow 'MLSStaleMessage :> CanThrow 'MLSUnsupportedMessage :> CanThrow 'MLSUnsupportedProposal + :> CanThrow 'MLSClientSenderUserMismatch + :> CanThrow 'MLSGroupConversationMismatch :> CanThrow 'MissingLegalholdConsent :> CanThrow MLSProposalFailure :> "messages" @@ -1356,6 +1359,8 @@ type MLSMessagingAPI = :> CanThrow 'MLSStaleMessage :> CanThrow 'MLSUnsupportedMessage :> CanThrow 'MLSUnsupportedProposal + :> CanThrow 'MLSClientSenderUserMismatch + :> CanThrow 'MLSGroupConversationMismatch :> CanThrow 'MissingLegalholdConsent :> CanThrow MLSProposalFailure :> "messages" @@ -1363,6 +1368,12 @@ type MLSMessagingAPI = :> ReqBody '[MLS] (RawMLS SomeMessage) :> MultiVerb1 'POST '[JSON] (Respond 201 "Message sent" MLSMessageSendingStatus) ) + :<|> Named + "mls-public-keys" + ( Summary "Get public keys used by the backend to sign external proposals" + :> "public-keys" + :> MultiVerb1 'GET '[JSON] (Respond 200 "Public keys" MLSPublicKeys) + ) type MLSAPI = LiftNamed (ZLocalUser :> "mls" :> MLSMessagingAPI) diff --git a/libs/wire-api/src/Wire/API/Team.hs b/libs/wire-api/src/Wire/API/Team.hs index 8bb151eca0d..bcebae30ac7 100644 --- a/libs/wire-api/src/Wire/API/Team.hs +++ b/libs/wire-api/src/Wire/API/Team.hs @@ -89,9 +89,9 @@ import qualified Data.Swagger.Build.Api as Doc import qualified Data.Text.Encoding as T import Imports import Test.QuickCheck.Gen (suchThat) -import Wire.API.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) import Wire.API.Asset (AssetKey) import Wire.API.Team.Member (TeamMember) +import Wire.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) -------------------------------------------------------------------------------- -- Team diff --git a/libs/wire-api/src/Wire/API/Team/Conversation.hs b/libs/wire-api/src/Wire/API/Team/Conversation.hs index f6fb56f92a7..3fd614cd6b8 100644 --- a/libs/wire-api/src/Wire/API/Team/Conversation.hs +++ b/libs/wire-api/src/Wire/API/Team/Conversation.hs @@ -43,7 +43,7 @@ import Data.Proxy import Data.Swagger import qualified Data.Swagger.Build.Api as Doc import Imports -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) -------------------------------------------------------------------------------- -- TeamConversation diff --git a/libs/wire-api/src/Wire/API/Team/Export.hs b/libs/wire-api/src/Wire/API/Team/Export.hs index ac9cd8a824f..7e0c464fc9b 100644 --- a/libs/wire-api/src/Wire/API/Team/Export.hs +++ b/libs/wire-api/src/Wire/API/Team/Export.hs @@ -30,12 +30,12 @@ import Data.String.Conversions (cs) import Data.Vector (fromList) import Imports import Test.QuickCheck (Arbitrary) -import Wire.API.Arbitrary (GenericUniform (GenericUniform)) import Wire.API.Team.Role (Role) import Wire.API.User (Name) import Wire.API.User.Identity (Email) import Wire.API.User.Profile (ManagedBy) import Wire.API.User.RichInfo (RichInfo) +import Wire.Arbitrary (GenericUniform (GenericUniform)) data TeamExportUser = TeamExportUser { tExportDisplayName :: Name, diff --git a/libs/wire-api/src/Wire/API/Team/Feature.hs b/libs/wire-api/src/Wire/API/Team/Feature.hs index fecff851f29..1e2a33f803d 100644 --- a/libs/wire-api/src/Wire/API/Team/Feature.hs +++ b/libs/wire-api/src/Wire/API/Team/Feature.hs @@ -109,9 +109,9 @@ import Imports import Servant (FromHttpApiData (..), ToHttpApiData (..)) import Test.QuickCheck.Arbitrary (arbitrary) import Test.QuickCheck.Gen (suchThat) -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) import Wire.API.Conversation.Protocol (ProtocolTag (ProtocolProteusTag)) import Wire.API.MLS.CipherSuite (CipherSuiteTag (MLS_128_DHKEMX25519_AES128GCM_SHA256_Ed25519)) +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) ---------------------------------------------------------------------- -- FeatureTag @@ -997,6 +997,7 @@ data AllFeatureConfigs = AllFeatureConfigs { afcLegalholdStatus :: WithStatus LegalholdConfig, afcSSOStatus :: WithStatus SSOConfig, afcTeamSearchVisibilityAvailable :: WithStatus SearchVisibilityAvailableConfig, + afcSearchVisibilityInboundConfig :: WithStatus SearchVisibilityInboundConfig, afcValidateSAMLEmails :: WithStatus ValidateSAMLEmailsConfig, afcDigitalSignatures :: WithStatus DigitalSignaturesConfig, afcAppLock :: WithStatus AppLockConfig, @@ -1006,8 +1007,7 @@ data AllFeatureConfigs = AllFeatureConfigs afcSelfDeletingMessages :: WithStatus SelfDeletingMessagesConfig, afcGuestLink :: WithStatus GuestLinksConfig, afcSndFactorPasswordChallenge :: WithStatus SndFactorPasswordChallengeConfig, - afcMLS :: WithStatus MLSConfig, - afcSearchVisibilityInboundConfig :: WithStatus SearchVisibilityInboundConfig + afcMLS :: WithStatus MLSConfig } deriving stock (Eq, Show) deriving (FromJSON, ToJSON, S.ToSchema) via (Schema AllFeatureConfigs) @@ -1019,6 +1019,7 @@ instance ToSchema AllFeatureConfigs where <$> afcLegalholdStatus .= featureField <*> afcSSOStatus .= featureField <*> afcTeamSearchVisibilityAvailable .= featureField + <*> afcSearchVisibilityInboundConfig .= featureField <*> afcValidateSAMLEmails .= featureField <*> afcDigitalSignatures .= featureField <*> afcAppLock .= featureField @@ -1029,7 +1030,6 @@ instance ToSchema AllFeatureConfigs where <*> afcGuestLink .= featureField <*> afcSndFactorPasswordChallenge .= featureField <*> afcMLS .= featureField - <*> afcSearchVisibilityInboundConfig .= featureField where featureField :: forall cfg. diff --git a/libs/wire-api/src/Wire/API/Team/Invitation.hs b/libs/wire-api/src/Wire/API/Team/Invitation.hs index 3e68bec925d..4476698097f 100644 --- a/libs/wire-api/src/Wire/API/Team/Invitation.hs +++ b/libs/wire-api/src/Wire/API/Team/Invitation.hs @@ -34,10 +34,10 @@ import Data.Id import Data.Json.Util import qualified Data.Swagger.Build.Api as Doc import Imports -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) import Wire.API.Team.Role (Role, defaultRole, typeRole) import Wire.API.User.Identity (Email, Phone) import Wire.API.User.Profile (Locale, Name) +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) -------------------------------------------------------------------------------- -- InvitationRequest diff --git a/libs/wire-api/src/Wire/API/Team/LegalHold.hs b/libs/wire-api/src/Wire/API/Team/LegalHold.hs index 91fb866f585..59f5b40cdf8 100644 --- a/libs/wire-api/src/Wire/API/Team/LegalHold.hs +++ b/libs/wire-api/src/Wire/API/Team/LegalHold.hs @@ -37,10 +37,10 @@ import Data.Schema import qualified Data.Swagger as S hiding (info) import Deriving.Aeson import Imports -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) import Wire.API.Provider import Wire.API.Provider.Service (ServiceKeyPEM) import Wire.API.User.Client.Prekey +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) -------------------------------------------------------------------------------- -- NewLegalHoldService diff --git a/libs/wire-api/src/Wire/API/Team/LegalHold/External.hs b/libs/wire-api/src/Wire/API/Team/LegalHold/External.hs index 197f0ac513c..ea892087bfe 100644 --- a/libs/wire-api/src/Wire/API/Team/LegalHold/External.hs +++ b/libs/wire-api/src/Wire/API/Team/LegalHold/External.hs @@ -36,8 +36,8 @@ import Data.Id import Data.Json.Util ((#)) import Data.Swagger import Imports -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) import Wire.API.User.Client.Prekey +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) -------------------------------------------------------------------------------- -- initiate diff --git a/libs/wire-api/src/Wire/API/Team/Member.hs b/libs/wire-api/src/Wire/API/Team/Member.hs index 96388778716..3b87022987f 100644 --- a/libs/wire-api/src/Wire/API/Team/Member.hs +++ b/libs/wire-api/src/Wire/API/Team/Member.hs @@ -71,8 +71,8 @@ import Data.Schema import qualified Data.Swagger.Schema as S import GHC.TypeLits import Imports -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) import Wire.API.Team.Permission (Permissions) +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) data PermissionTag = Required | Optional diff --git a/libs/wire-api/src/Wire/API/Team/Permission.hs b/libs/wire-api/src/Wire/API/Team/Permission.hs index 43dd13189b3..e6fe0aafb4e 100644 --- a/libs/wire-api/src/Wire/API/Team/Permission.hs +++ b/libs/wire-api/src/Wire/API/Team/Permission.hs @@ -57,8 +57,8 @@ import Data.Singletons.TH import qualified Data.Swagger as S import qualified Data.Swagger.Build.Api as Doc import Imports -import Wire.API.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) import Wire.API.Util.Aeson (CustomEncoded (..)) +import Wire.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) -------------------------------------------------------------------------------- -- Permissions diff --git a/libs/wire-api/src/Wire/API/Team/Role.hs b/libs/wire-api/src/Wire/API/Team/Role.hs index b10f37a2e4f..2879766d4b3 100644 --- a/libs/wire-api/src/Wire/API/Team/Role.hs +++ b/libs/wire-api/src/Wire/API/Team/Role.hs @@ -38,7 +38,7 @@ import qualified Data.Swagger.Model.Api as Doc import qualified Data.Text as T import Imports import Servant.API (FromHttpApiData, parseQueryParam) -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) -- Note [team roles] -- ~~~~~~~~~~~~ diff --git a/libs/wire-api/src/Wire/API/Team/SearchVisibility.hs b/libs/wire-api/src/Wire/API/Team/SearchVisibility.hs index 90e67cece18..175ec24b473 100644 --- a/libs/wire-api/src/Wire/API/Team/SearchVisibility.hs +++ b/libs/wire-api/src/Wire/API/Team/SearchVisibility.hs @@ -35,7 +35,7 @@ import qualified Data.Swagger as S import qualified Data.Swagger.Build.Api as Doc import Deriving.Aeson import Imports -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) -------------------------------------------------------------------------------- -- TeamSearchVisibility @@ -60,7 +60,7 @@ import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) -- Name: can be found by same team only -- @ -- --- See also: 'FeatureTeamSearchVisibility', 'TeamSearchVisibilityEnabled'. +-- See also: 'FeatureTeamSearchVisibilityAvailability'. data TeamSearchVisibility = SearchVisibilityStandard | SearchVisibilityNoNameOutsideTeam diff --git a/libs/wire-api/src/Wire/API/User.hs b/libs/wire-api/src/Wire/API/User.hs index 8af4f071f6b..87f9d1fa3e8 100644 --- a/libs/wire-api/src/Wire/API/User.hs +++ b/libs/wire-api/src/Wire/API/User.hs @@ -157,7 +157,6 @@ import Servant (FromHttpApiData (..), ToHttpApiData (..), type (.++)) import qualified Test.QuickCheck as QC import URI.ByteString (serializeURIRef) import qualified Web.Cookie as Web -import Wire.API.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) import Wire.API.Error import Wire.API.Error.Brig import qualified Wire.API.Error.Brig as E @@ -169,6 +168,7 @@ import Wire.API.User.Auth (CookieLabel) import Wire.API.User.Identity import Wire.API.User.Profile import Wire.API.User.RichInfo +import Wire.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) -------------------------------------------------------------------------------- -- UserIdList diff --git a/libs/wire-api/src/Wire/API/User/Activation.hs b/libs/wire-api/src/Wire/API/User/Activation.hs index de14a9f0f37..385ceaa59cf 100644 --- a/libs/wire-api/src/Wire/API/User/Activation.hs +++ b/libs/wire-api/src/Wire/API/User/Activation.hs @@ -48,9 +48,9 @@ import qualified Data.Swagger as S import qualified Data.Swagger.Build.Api as Doc import Data.Text.Ascii import Imports -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) import Wire.API.User.Identity import Wire.API.User.Profile +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) -------------------------------------------------------------------------------- -- ActivationTarget diff --git a/libs/wire-api/src/Wire/API/User/Auth.hs b/libs/wire-api/src/Wire/API/User/Auth.hs index d81da1511f3..c38537045a1 100644 --- a/libs/wire-api/src/Wire/API/User/Auth.hs +++ b/libs/wire-api/src/Wire/API/User/Auth.hs @@ -56,7 +56,6 @@ import Data.Aeson import qualified Data.Aeson.Types as Aeson import Data.ByteString.Conversion import Data.Code as Code -import Data.Handle (Handle) import Data.Id (UserId) import Data.Misc (PlainTextPassword (..)) import Data.Schema (ToSchema) @@ -65,8 +64,9 @@ import qualified Data.Swagger.Build.Api as Doc import Data.Text.Lazy.Encoding (decodeUtf8, encodeUtf8) import Data.Time.Clock (UTCTime) import Imports -import Wire.API.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) -import Wire.API.User.Identity (Email, Phone) +import Wire.API.User.Auth2 +import Wire.API.User.Identity (Phone) +import Wire.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) -------------------------------------------------------------------------------- -- Login @@ -131,42 +131,12 @@ loginLabel :: Login -> Maybe CookieLabel loginLabel (PasswordLogin _ _ l _) = l loginLabel (SmsLogin _ _ l) = l --------------------------------------------------------------------------------- --- LoginId - -data LoginId - = LoginByEmail Email - | LoginByPhone Phone - | LoginByHandle Handle - deriving stock (Eq, Show, Generic) - deriving (Arbitrary) via (GenericUniform LoginId) - -instance FromJSON LoginId where - parseJSON = withObject "LoginId" $ \o -> do - email <- fmap LoginByEmail <$> (o .:? "email") - phone <- fmap LoginByPhone <$> (o .:? "phone") - handle <- fmap LoginByHandle <$> (o .:? "handle") - maybe - (fail "'email', 'phone' or 'handle' required") - pure - (email <|> phone <|> handle) - --- NB. You might be tempted to rewrite this by applying (<|>) to --- parsers themselves. However, the code as it is right now has a --- property that if (e.g.) the email is present but unparseable, --- parsing will fail. If you change it to use (<|>), unparseable --- email (or phone, etc) will just cause the next parser to be --- chosen, instead of failing early. - loginIdPair :: LoginId -> Aeson.Pair loginIdPair = \case LoginByEmail s -> "email" .= s LoginByPhone s -> "phone" .= s LoginByHandle s -> "handle" .= s -instance ToJSON LoginId where - toJSON loginId = object [loginIdPair loginId] - -------------------------------------------------------------------------------- -- LoginCode diff --git a/libs/wire-api/src/Wire/API/User/Auth2.hs b/libs/wire-api/src/Wire/API/User/Auth2.hs new file mode 100644 index 00000000000..5183c382109 --- /dev/null +++ b/libs/wire-api/src/Wire/API/User/Auth2.hs @@ -0,0 +1,65 @@ +{-# LANGUAGE StrictData #-} + +-- This file is part of the Wire Server implementation. +-- +-- Copyright (C) 2022 Wire Swiss GmbH +-- +-- This program is free software: you can redistribute it and/or modify it under +-- the terms of the GNU Affero General Public License as published by the Free +-- Software Foundation, either version 3 of the License, or (at your option) any +-- later version. +-- +-- This program is distributed in the hope that it will be useful, but WITHOUT +-- ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +-- FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more +-- details. +-- +-- You should have received a copy of the GNU Affero General Public License along +-- with this program. If not, see . + +-- FUTUREWORK: replace `Wire.API.User.Auth` with this module once everything in `Auth` is migrated to schema-profunctor +module Wire.API.User.Auth2 where + +import Data.Aeson (FromJSON (..), ToJSON (..)) +import qualified Data.Aeson.Types as A +import Data.Handle (Handle) +import Data.Schema +import qualified Data.Swagger as S +import Data.Tuple.Extra (fst3, snd3, thd3) +import Imports +import Wire.API.User.Identity (Email, Phone) +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) + +-------------------------------------------------------------------------------- +-- LoginId + +data LoginId + = LoginByEmail Email + | LoginByPhone Phone + | LoginByHandle Handle + deriving stock (Eq, Show, Generic) + deriving (Arbitrary) via (GenericUniform LoginId) + deriving (ToJSON, FromJSON, S.ToSchema) via (Schema LoginId) + +-- NB. this should fail if (e.g.) the email is present but unparseable even if the JSON contains a valid phone number or handle. +-- See tests in `Test.Wire.API.User.Auth`. +instance ToSchema LoginId where + schema = + object "LoginId" $ + fromLoginId .= tupleSchema `withParser` validate + where + fromLoginId :: LoginId -> (Maybe Email, Maybe Phone, Maybe Handle) + fromLoginId = \case + LoginByEmail e -> (Just e, Nothing, Nothing) + LoginByPhone p -> (Nothing, Just p, Nothing) + LoginByHandle h -> (Nothing, Nothing, Just h) + tupleSchema :: ObjectSchema SwaggerDoc (Maybe Email, Maybe Phone, Maybe Handle) + tupleSchema = + (,,) + <$> fst3 .= maybe_ (optField "email" schema) + <*> snd3 .= maybe_ (optField "phone" schema) + <*> thd3 .= maybe_ (optField "handle" schema) + validate :: (Maybe Email, Maybe Phone, Maybe Handle) -> A.Parser LoginId + validate (mEmail, mPhone, mHandle) = + maybe (fail "'email', 'phone' or 'handle' required") pure $ + (LoginByEmail <$> mEmail) <|> (LoginByPhone <$> mPhone) <|> (LoginByHandle <$> mHandle) diff --git a/libs/wire-api/src/Wire/API/User/Client.hs b/libs/wire-api/src/Wire/API/User/Client.hs index e399b90f4c4..6f38cc75f67 100644 --- a/libs/wire-api/src/Wire/API/User/Client.hs +++ b/libs/wire-api/src/Wire/API/User/Client.hs @@ -23,6 +23,9 @@ module Wire.API.User.Client ClientCapability (..), ClientCapabilityList (..), + -- * ClientInfo + ClientInfo (..), + -- * UserClients UserClientMap (..), UserClientPrekeyMap (..), @@ -105,10 +108,10 @@ import Deriving.Swagger StripPrefix, ) import Imports -import Wire.API.Arbitrary (Arbitrary (arbitrary), GenericUniform (..), generateExample, mapOf', setOf') import Wire.API.MLS.Credential import Wire.API.User.Auth (CookieLabel) import Wire.API.User.Client.Prekey as Prekey +import Wire.Arbitrary (Arbitrary (arbitrary), GenericUniform (..), generateExample, mapOf', setOf') ---------------------------------------------------------------------- -- ClientCapability, ClientCapabilityList @@ -332,6 +335,26 @@ qualifiedUserClientPrekeyMapFromList :: qualifiedUserClientPrekeyMapFromList = mkQualifiedUserClientPrekeyMap . Map.fromList . map qToPair +-------------------------------------------------------------------------------- +-- ClientInfo + +-- | A client, together with extra information about it. +data ClientInfo = ClientInfo + { -- | The ID of this client. + ciId :: ClientId, + -- | Whether this client is MLS-capable. + ciMLS :: Bool + } + deriving stock (Eq, Ord, Show) + deriving (Swagger.ToSchema, FromJSON, ToJSON) via Schema ClientInfo + +instance ToSchema ClientInfo where + schema = + object "ClientInfo" $ + ClientInfo + <$> ciId .= field "id" schema + <*> ciMLS .= field "mls" schema + -------------------------------------------------------------------------------- -- UserClients diff --git a/libs/wire-api/src/Wire/API/User/Client/Prekey.hs b/libs/wire-api/src/Wire/API/User/Client/Prekey.hs index c94dab86c25..8b29e54afac 100644 --- a/libs/wire-api/src/Wire/API/User/Client/Prekey.hs +++ b/libs/wire-api/src/Wire/API/User/Client/Prekey.hs @@ -41,7 +41,7 @@ import Data.Schema import qualified Data.Swagger as S import qualified Data.Swagger.Build.Api as Doc import Imports -import Wire.API.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) +import Wire.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) newtype PrekeyId = PrekeyId {keyId :: Word16} deriving stock (Eq, Ord, Show, Generic) diff --git a/libs/wire-api/src/Wire/API/User/Handle.hs b/libs/wire-api/src/Wire/API/User/Handle.hs index c62bbd81fb7..2fa2955d738 100644 --- a/libs/wire-api/src/Wire/API/User/Handle.hs +++ b/libs/wire-api/src/Wire/API/User/Handle.hs @@ -38,7 +38,7 @@ import Data.Schema import qualified Data.Swagger as S import qualified Data.Swagger.Build.Api as Doc import Imports -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) -------------------------------------------------------------------------------- -- UserHandleInfo diff --git a/libs/wire-api/src/Wire/API/User/Identity.hs b/libs/wire-api/src/Wire/API/User/Identity.hs index 802412efd52..9c1f0ae55cc 100644 --- a/libs/wire-api/src/Wire/API/User/Identity.hs +++ b/libs/wire-api/src/Wire/API/User/Identity.hs @@ -79,8 +79,8 @@ import qualified Test.QuickCheck as QC import qualified Text.Email.Validate as Email.V import qualified URI.ByteString as URI import URI.ByteString.QQ (uri) -import Wire.API.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) import Wire.API.User.Profile (fromName, mkName) +import Wire.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) -------------------------------------------------------------------------------- -- UserIdentity diff --git a/libs/wire-api/src/Wire/API/User/IdentityProvider.hs b/libs/wire-api/src/Wire/API/User/IdentityProvider.hs index b90af4f5ab2..e573b19a93d 100644 --- a/libs/wire-api/src/Wire/API/User/IdentityProvider.hs +++ b/libs/wire-api/src/Wire/API/User/IdentityProvider.hs @@ -39,8 +39,8 @@ import SAML2.WebSSO (IdPConfig) import qualified SAML2.WebSSO as SAML import SAML2.WebSSO.Types.TH (deriveJSONOptions) import Servant.API as Servant hiding (MkLink, URI (..)) -import Wire.API.Arbitrary (Arbitrary, GenericUniform (GenericUniform)) import Wire.API.User.Orphans (samlSchemaOptions) +import Wire.Arbitrary (Arbitrary, GenericUniform (GenericUniform)) -- | The identity provider type used in Spar. type IdP = IdPConfig WireIdP diff --git a/libs/wire-api/src/Wire/API/User/Password.hs b/libs/wire-api/src/Wire/API/User/Password.hs index 044002f6253..986ef301a6f 100644 --- a/libs/wire-api/src/Wire/API/User/Password.hs +++ b/libs/wire-api/src/Wire/API/User/Password.hs @@ -42,8 +42,8 @@ import Data.Range (Ranged (..)) import qualified Data.Swagger.Build.Api as Doc import Data.Text.Ascii import Imports -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) import Wire.API.User.Identity +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) -------------------------------------------------------------------------------- -- NewPasswordReset diff --git a/libs/wire-api/src/Wire/API/User/Profile.hs b/libs/wire-api/src/Wire/API/User/Profile.hs index bed58b36948..37b3557e31c 100644 --- a/libs/wire-api/src/Wire/API/User/Profile.hs +++ b/libs/wire-api/src/Wire/API/User/Profile.hs @@ -68,9 +68,9 @@ import qualified Data.Swagger as S import qualified Data.Swagger.Build.Api as Doc import qualified Data.Text as Text import Imports -import Wire.API.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) import Wire.API.Asset (AssetKey (..)) import Wire.API.User.Orphans () +import Wire.Arbitrary (Arbitrary (arbitrary), GenericUniform (..)) -------------------------------------------------------------------------------- -- Name diff --git a/libs/wire-api/src/Wire/API/User/RichInfo.hs b/libs/wire-api/src/Wire/API/User/RichInfo.hs index c9debf31d91..d0a7be408dc 100644 --- a/libs/wire-api/src/Wire/API/User/RichInfo.hs +++ b/libs/wire-api/src/Wire/API/User/RichInfo.hs @@ -63,7 +63,7 @@ import qualified Data.Swagger.Build.Api as Doc import qualified Data.Text as Text import Imports import qualified Test.QuickCheck as QC -import Wire.API.Arbitrary (Arbitrary (arbitrary)) +import Wire.Arbitrary (Arbitrary (arbitrary)) -------------------------------------------------------------------------------- -- RichInfo diff --git a/libs/wire-api/src/Wire/API/User/Scim.hs b/libs/wire-api/src/Wire/API/User/Scim.hs index d77d06deb3d..129ec635312 100644 --- a/libs/wire-api/src/Wire/API/User/Scim.hs +++ b/libs/wire-api/src/Wire/API/User/Scim.hs @@ -84,11 +84,11 @@ import Web.Scim.Schema.Schema (Schema (CustomSchema)) import qualified Web.Scim.Schema.Schema as Scim import qualified Web.Scim.Schema.User as Scim import qualified Web.Scim.Schema.User as Scim.User -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) import Wire.API.User.Identity (Email) import Wire.API.User.Profile as BT import qualified Wire.API.User.RichInfo as RI import Wire.API.User.Saml () +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) ---------------------------------------------------------------------------- -- Schemas diff --git a/libs/wire-api/src/Wire/API/User/Search.hs b/libs/wire-api/src/Wire/API/User/Search.hs index bf0790a18a2..86732ebc52b 100644 --- a/libs/wire-api/src/Wire/API/User/Search.hs +++ b/libs/wire-api/src/Wire/API/User/Search.hs @@ -54,10 +54,10 @@ import qualified Data.Text as T import Imports import Servant.API (FromHttpApiData) import Web.Internal.HttpApiData (parseQueryParam) -import Wire.API.Arbitrary (Arbitrary, GenericUniform (..)) import Wire.API.Team.Role (Role) import Wire.API.User (ManagedBy) import Wire.API.User.Identity (Email (..)) +import Wire.Arbitrary (Arbitrary, GenericUniform (..)) -------------------------------------------------------------------------------- -- SearchResult diff --git a/libs/wire-api/src/Wire/API/UserMap.hs b/libs/wire-api/src/Wire/API/UserMap.hs index 25547ea6feb..81916f85710 100644 --- a/libs/wire-api/src/Wire/API/UserMap.hs +++ b/libs/wire-api/src/Wire/API/UserMap.hs @@ -31,8 +31,8 @@ import qualified Data.Text as Text import Data.Typeable (typeRep) import Imports import Test.QuickCheck (Arbitrary (..)) -import Wire.API.Arbitrary (generateExample, mapOf') import Wire.API.Wrapped (Wrapped) +import Wire.Arbitrary (generateExample, mapOf') newtype UserMap a = UserMap {userMap :: Map UserId a} deriving stock (Eq, Show) diff --git a/libs/wire-api/test/unit/Main.hs b/libs/wire-api/test/unit/Main.hs index 8bd8aeefb91..74dcadd90ec 100644 --- a/libs/wire-api/test/unit/Main.hs +++ b/libs/wire-api/test/unit/Main.hs @@ -28,11 +28,13 @@ import qualified Test.Wire.API.MLS as MLS import qualified Test.Wire.API.Roundtrip.Aeson as Roundtrip.Aeson import qualified Test.Wire.API.Roundtrip.ByteString as Roundtrip.ByteString import qualified Test.Wire.API.Roundtrip.CSV as Roundtrip.CSV +import qualified Test.Wire.API.Roundtrip.MLS as Roundtrip.MLS import qualified Test.Wire.API.Routes as Routes import qualified Test.Wire.API.Swagger as Swagger import qualified Test.Wire.API.Team.Export as Team.Export import qualified Test.Wire.API.Team.Member as Team.Member import qualified Test.Wire.API.User as User +import qualified Test.Wire.API.User.Auth as User.Auth import qualified Test.Wire.API.User.RichInfo as User.RichInfo import qualified Test.Wire.API.User.Search as User.Search @@ -47,8 +49,10 @@ main = User.tests, User.Search.tests, User.RichInfo.tests, + User.Auth.tests, Roundtrip.Aeson.tests, Roundtrip.ByteString.tests, + Roundtrip.MLS.tests, Swagger.tests, Roundtrip.CSV.tests, Routes.tests, diff --git a/libs/wire-api/test/unit/Test/Wire/API/Call/Config.hs b/libs/wire-api/test/unit/Test/Wire/API/Call/Config.hs index a7106562c52..57cb41f4c05 100644 --- a/libs/wire-api/test/unit/Test/Wire/API/Call/Config.hs +++ b/libs/wire-api/test/unit/Test/Wire/API/Call/Config.hs @@ -22,8 +22,8 @@ import qualified Data.Aeson.KeyMap as KeyMap import Imports import Test.Tasty import Test.Tasty.QuickCheck hiding (total) -import Wire.API.Arbitrary () import Wire.API.Call.Config (RTCConfiguration, TurnURI, isTcp, isTls, isUdp, limitServers) +import Wire.Arbitrary () tests :: TestTree tests = diff --git a/libs/wire-api/test/unit/Test/Wire/API/MLS.hs b/libs/wire-api/test/unit/Test/Wire/API/MLS.hs index 06ce1bd0625..d13573dd596 100644 --- a/libs/wire-api/test/unit/Test/Wire/API/MLS.hs +++ b/libs/wire-api/test/unit/Test/Wire/API/MLS.hs @@ -17,22 +17,34 @@ module Test.Wire.API.MLS where +import Control.Concurrent.Async +import qualified Crypto.PubKey.Ed25519 as Ed25519 +import Data.ByteArray import qualified Data.ByteString as BS import qualified Data.ByteString.Lazy as LBS import Data.Domain import Data.Either.Combinators import Data.Hex import Data.Id +import Data.Json.Util (toBase64Text) +import Data.Qualified import qualified Data.Text as T +import qualified Data.Text as Text import qualified Data.UUID as UUID +import qualified Data.UUID.V4 as UUID import Imports +import System.Exit +import System.FilePath (()) +import System.Process import Test.Tasty import Test.Tasty.HUnit +import UnliftIO (withSystemTempDirectory) import Wire.API.MLS.CipherSuite import Wire.API.MLS.Commit import Wire.API.MLS.Credential import Wire.API.MLS.Epoch import Wire.API.MLS.Extension +import Wire.API.MLS.Group import Wire.API.MLS.KeyPackage import Wire.API.MLS.Message import Wire.API.MLS.Proposal @@ -47,7 +59,8 @@ tests = testCase "parse application message" testParseApplication, testCase "parse welcome message" testParseWelcome, testCase "key package ref" testKeyPackageRef, - testCase "validate message signature" testVerifyMLSPlainTextWithKey + testCase "validate message signature" testVerifyMLSPlainTextWithKey, + testCase "create signed remove proposal" testRemoveProposalMessageSignature ] testParseKeyPackage :: IO () @@ -154,3 +167,82 @@ testVerifyMLSPlainTextWithKey = do assertBool "message signature verification failed" $ verifyMessageSignature MLS_128_DHKEMX25519_AES128GCM_SHA256_Ed25519 msg pubkey + +testRemoveProposalMessageSignature :: IO () +testRemoveProposalMessageSignature = withSystemTempDirectory "mls" $ \tmp -> do + qcid <- do + let c = newClientId 0x3ae58155 + usr <- flip Qualified (Domain "example.com") <$> (Id <$> UUID.nextRandom) + pure (userClientQid usr c) + void . liftIO $ spawn (cli qcid tmp ["init", qcid]) Nothing + + qcid2 <- do + let c = newClientId 0x4ae58157 + usr <- flip Qualified (Domain "example.com") <$> (Id <$> UUID.nextRandom) + pure (userClientQid usr c) + void . liftIO $ spawn (cli qcid2 tmp ["init", qcid2]) Nothing + kp <- liftIO $ decodeMLSError <$> spawn (cli qcid2 tmp ["key-package", "create"]) Nothing + liftIO $ BS.writeFile (tmp qcid2) (rmRaw kp) + + let groupFilename = "group" + let gid = GroupId "abcd" + createGroup tmp qcid groupFilename gid + + void $ liftIO $ spawn (cli qcid tmp ["member", "add", "--group", tmp groupFilename, "--in-place", tmp qcid2]) Nothing + + secretKey <- Ed25519.generateSecretKey + let publicKey = Ed25519.toPublic secretKey + let message = fromJust (mkRemoveProposalMessage secretKey publicKey gid (Epoch 1) (fromJust (kpRef' kp))) + let messageFilename = "signed-message.mls" + BS.writeFile (tmp messageFilename) (rmRaw (mkRawMLS message)) + let signerKeyFilename = "signer-key.bin" + BS.writeFile (tmp signerKeyFilename) (convert publicKey) + + void . liftIO $ spawn (cli qcid tmp ["check-signature", "--group", tmp groupFilename, "--message", tmp messageFilename, "--signer-key", tmp signerKeyFilename]) Nothing + +createGroup :: FilePath -> String -> String -> GroupId -> IO () +createGroup tmp store groupName gid = do + groupJSON <- + liftIO $ + spawn + ( cli + store + tmp + ["group", "create", T.unpack (toBase64Text (unGroupId gid))] + ) + Nothing + liftIO $ BS.writeFile (tmp groupName) groupJSON + +decodeMLSError :: ParseMLS a => ByteString -> a +decodeMLSError s = case decodeMLS' s of + Left e -> error ("Could not parse MLS object: " <> Text.unpack e) + Right x -> x + +userClientQid :: Qualified UserId -> ClientId -> String +userClientQid usr c = + show (qUnqualified usr) + <> ":" + <> T.unpack (client c) + <> "@" + <> T.unpack (domainText (qDomain usr)) + +spawn :: HasCallStack => CreateProcess -> Maybe ByteString -> IO ByteString +spawn cp minput = do + (mout, ex) <- withCreateProcess + cp + { std_out = CreatePipe, + std_in = if isJust minput then CreatePipe else Inherit + } + $ \minh mouth _ ph -> + let writeInput = for_ ((,) <$> minput <*> minh) $ \(input, inh) -> + BS.hPutStr inh input >> hClose inh + readOutput = (,) <$> traverse BS.hGetContents mouth <*> waitForProcess ph + in snd <$> concurrently writeInput readOutput + case (mout, ex) of + (Just out, ExitSuccess) -> pure out + _ -> assertFailure "Failed spawning process" + +cli :: String -> FilePath -> [String] -> CreateProcess +cli store tmp args = + proc "mls-test-cli" $ + ["--store", tmp (store <> ".db")] <> args diff --git a/libs/wire-api/test/unit/Test/Wire/API/Roundtrip/ByteString.hs b/libs/wire-api/test/unit/Test/Wire/API/Roundtrip/ByteString.hs index 38e4ac790a8..6240b9c361b 100644 --- a/libs/wire-api/test/unit/Test/Wire/API/Roundtrip/ByteString.hs +++ b/libs/wire-api/test/unit/Test/Wire/API/Roundtrip/ByteString.hs @@ -22,7 +22,6 @@ import Imports import qualified Test.Tasty as T import Test.Tasty.QuickCheck (Arbitrary, counterexample, testProperty, (===)) import Type.Reflection (typeRep) -import qualified Wire.API.Arbitrary as Arbitrary () import qualified Wire.API.Asset as Asset import qualified Wire.API.Call.Config as Call.Config import qualified Wire.API.Conversation.Code as Conversation.Code @@ -42,6 +41,7 @@ import qualified Wire.API.User.IdentityProvider as User.IdentityProvider import qualified Wire.API.User.Password as User.Password import qualified Wire.API.User.Profile as User.Profile import qualified Wire.API.User.Search as User.Search +import qualified Wire.Arbitrary as Arbitrary () tests :: T.TestTree tests = diff --git a/libs/wire-api/test/unit/Test/Wire/API/Roundtrip/MLS.hs b/libs/wire-api/test/unit/Test/Wire/API/Roundtrip/MLS.hs new file mode 100644 index 00000000000..34aaeeb9ffe --- /dev/null +++ b/libs/wire-api/test/unit/Test/Wire/API/Roundtrip/MLS.hs @@ -0,0 +1,111 @@ +{-# LANGUAGE GeneralizedNewtypeDeriving #-} +-- This file is part of the Wire Server implementation. +-- +-- Copyright (C) 2022 Wire Swiss GmbH +-- +-- This program is free software: you can redistribute it and/or modify it under +-- the terms of the GNU Affero General Public License as published by the Free +-- Software Foundation, either version 3 of the License, or (at your option) any +-- later version. +-- +-- This program is distributed in the hope that it will be useful, but WITHOUT +-- ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +-- FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more +-- details. +-- +-- You should have received a copy of the GNU Affero General Public License along +-- with this program. If not, see . +{-# OPTIONS_GHC -Wwarn #-} + +module Test.Wire.API.Roundtrip.MLS (tests) where + +import Data.Binary.Put +import Imports +import qualified Test.Tasty as T +import Test.Tasty.QuickCheck +import Type.Reflection (typeRep) +import Wire.API.MLS.Extension +import Wire.API.MLS.KeyPackage +import Wire.API.MLS.Message +import Wire.API.MLS.Proposal +import Wire.API.MLS.Serialisation + +tests :: T.TestTree +tests = + T.localOption (T.Timeout (60 * 1000000) "60s") . T.testGroup "MLS roundtrip tests" $ + [ testRoundTrip @KeyPackageRef, + testRoundTrip @RemoveProposalSender, + testRoundTrip @RemoveProposalMessage, + testRoundTrip @RemoveProposalPayload, + testRoundTrip @AppAckProposalTest, + testRoundTrip @ExtensionVector + ] + +testRoundTrip :: + forall a. + (Arbitrary a, Typeable a, ParseMLS a, SerialiseMLS a, Eq a, Show a) => + T.TestTree +testRoundTrip = testProperty msg trip + where + msg = show (typeRep @a) + trip (v :: a) = + counterexample (show (runPut (serialiseMLS v))) $ + Right v === (decodeMLS . runPut . serialiseMLS) v + +-------------------------------------------------------------------------------- +-- auxiliary types + +newtype RemoveProposalMessage = RemoveProposalMessage (Message 'MLSPlainText) + deriving newtype (ParseMLS, SerialiseMLS, Eq, Show) + +newtype RemoveProposalTBS = RemoveProposalTBS (MessageTBS 'MLSPlainText) + deriving newtype (ParseMLS, SerialiseMLS, Eq, Show) + +instance Arbitrary RemoveProposalTBS where + arbitrary = + fmap RemoveProposalTBS $ + MessageTBS KnownFormatTag + <$> arbitrary + <*> arbitrary + <*> arbitrary + <*> (unRemoveProposalSender <$> arbitrary) + <*> (unRemoveProposalPayload <$> arbitrary) + +instance Arbitrary RemoveProposalMessage where + arbitrary = do + RemoveProposalTBS tbs <- arbitrary + RemoveProposalMessage + <$> (Message (mkRawMLS tbs) <$> arbitrary) + +newtype RemoveProposalPayload = RemoveProposalPayload {unRemoveProposalPayload :: MessagePayload 'MLSPlainText} + deriving newtype (ParseMLS, SerialiseMLS, Eq, Show) + +instance Arbitrary RemoveProposalPayload where + arbitrary = RemoveProposalPayload . ProposalMessage . mkRemoveProposal <$> arbitrary + +newtype RemoveProposalSender = RemoveProposalSender + {unRemoveProposalSender :: Sender 'MLSPlainText} + deriving newtype (ParseMLS, SerialiseMLS, Eq, Show) + +instance Arbitrary RemoveProposalSender where + arbitrary = RemoveProposalSender . PreconfiguredSender <$> arbitrary + +newtype AppAckProposalTest = AppAckProposalTest Proposal + deriving newtype (ParseMLS, Eq, Show) + +instance Arbitrary AppAckProposalTest where + arbitrary = AppAckProposalTest . AppAckProposal <$> arbitrary + +instance SerialiseMLS AppAckProposalTest where + serialiseMLS (AppAckProposalTest (AppAckProposal mrs)) = serialiseAppAckProposal mrs + serialiseMLS _ = serialiseAppAckProposal [] + +newtype ExtensionVector = ExtensionVector [Extension] + deriving newtype (Arbitrary, Eq, Show) + +instance ParseMLS ExtensionVector where + parseMLS = ExtensionVector <$> parseMLSVector @Word32 (parseMLS @Extension) + +instance SerialiseMLS ExtensionVector where + serialiseMLS (ExtensionVector exts) = do + serialiseMLSVector @Word32 serialiseMLS exts diff --git a/libs/wire-api/test/unit/Test/Wire/API/User/Auth.hs b/libs/wire-api/test/unit/Test/Wire/API/User/Auth.hs new file mode 100644 index 00000000000..3c56333dcd1 --- /dev/null +++ b/libs/wire-api/test/unit/Test/Wire/API/User/Auth.hs @@ -0,0 +1,45 @@ +-- This file is part of the Wire Server implementation. +-- +-- Copyright (C) 2022 Wire Swiss GmbH +-- +-- This program is free software: you can redistribute it and/or modify it under +-- the terms of the GNU Affero General Public License as published by the Free +-- Software Foundation, either version 3 of the License, or (at your option) any +-- later version. +-- +-- This program is distributed in the hope that it will be useful, but WITHOUT +-- ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +-- FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more +-- details. +-- +-- You should have received a copy of the GNU Affero General Public License along +-- with this program. If not, see . +module Test.Wire.API.User.Auth where + +import qualified Data.Aeson as Aeson +import Imports +import qualified Test.Tasty as T +import Test.Tasty.HUnit +import Wire.API.User +import Wire.API.User.Auth + +tests :: T.TestTree +tests = T.testGroup "Auth" [loginIdHappyCase, loginIdFailFastOnEmail, loginIdFailFastOnPhone] + +loginIdHappyCase :: T.TestTree +loginIdHappyCase = testCase "LoginId parser: valid email" $ do + let actual :: Maybe LoginId = Aeson.decode "{\"email\":\"foo@bar.com\"}" + let expected = Just $ LoginByEmail (Email {emailLocal = "foo", emailDomain = "bar.com"}) + assertEqual "should succeed" expected actual + +loginIdFailFastOnEmail :: T.TestTree +loginIdFailFastOnEmail = testCase "LoginId parser: invalid email, valid phone" $ do + let actual :: Maybe LoginId = Aeson.decode "{\"email\":\"invalid-email\",\"phone\":\"+123456789\"}" + let expected = Nothing + assertEqual "should fail if any provided login id is invalid" expected actual + +loginIdFailFastOnPhone :: T.TestTree +loginIdFailFastOnPhone = testCase "LoginId parser: invalid phone, valid email" $ do + let actual :: Maybe LoginId = Aeson.decode "{\"email\":\"foo@bar.com\",\"phone\":\"invalid-phone\"}" + let expected = Nothing + assertEqual "should fail if any provided login id is invalid" expected actual diff --git a/libs/wire-api/wire-api.cabal b/libs/wire-api/wire-api.cabal index d99d652103c..622041a2522 100644 --- a/libs/wire-api/wire-api.cabal +++ b/libs/wire-api/wire-api.cabal @@ -13,7 +13,6 @@ build-type: Simple library -- cabal-fmt: expand src exposed-modules: - Wire.API.Arbitrary Wire.API.Asset Wire.API.Call.Config Wire.API.Connection @@ -48,6 +47,7 @@ library Wire.API.MLS.Extension Wire.API.MLS.Group Wire.API.MLS.KeyPackage + Wire.API.MLS.Keys Wire.API.MLS.Message Wire.API.MLS.Proposal Wire.API.MLS.Serialisation @@ -107,6 +107,7 @@ library Wire.API.User Wire.API.User.Activation Wire.API.User.Auth + Wire.API.User.Auth2 Wire.API.User.Client Wire.API.User.Client.Prekey Wire.API.User.Handle @@ -612,11 +613,13 @@ test-suite wire-api-tests Test.Wire.API.Roundtrip.Aeson Test.Wire.API.Roundtrip.ByteString Test.Wire.API.Roundtrip.CSV + Test.Wire.API.Roundtrip.MLS Test.Wire.API.Routes Test.Wire.API.Swagger Test.Wire.API.Team.Export Test.Wire.API.Team.Member Test.Wire.API.User + Test.Wire.API.User.Auth Test.Wire.API.User.RichInfo Test.Wire.API.User.Search @@ -670,13 +673,16 @@ test-suite wire-api-tests aeson >=2.0.1.0 , aeson-pretty , aeson-qq + , async , base + , binary , bytestring , bytestring-arbitrary >=0.1.3 , bytestring-conversion , case-insensitive , cassava , containers >=0.5 + , cryptonite , currency-codes , directory , either @@ -687,10 +693,12 @@ test-suite wire-api-tests , iso3166-country-codes , iso639 , lens + , memory , metrics-wai , mime , pem , pretty + , process , proto-lens , QuickCheck , saml2-web-sso @@ -706,6 +714,7 @@ test-suite wire-api-tests , text , time , types-common >=0.16 + , unliftio , unordered-containers , uri-bytestring , uuid diff --git a/libs/wire-message-proto-lens/wire-message-proto-lens.cabal b/libs/wire-message-proto-lens/wire-message-proto-lens.cabal index 96992606143..4afca0f0635 100644 --- a/libs/wire-message-proto-lens/wire-message-proto-lens.cabal +++ b/libs/wire-message-proto-lens/wire-message-proto-lens.cabal @@ -75,7 +75,7 @@ library base , proto-lens-runtime - build-tool-depends: proto-lens-protoc:proto-lens-protoc -any + build-tool-depends: proto-lens-protoc:proto-lens-protoc default-language: Haskell2010 autogen-modules: Proto.Otr diff --git a/nix/pkgs/mls_test_cli/default.nix b/nix/pkgs/mls_test_cli/default.nix index f8b84ccf7eb..4a9c64213de 100644 --- a/nix/pkgs/mls_test_cli/default.nix +++ b/nix/pkgs/mls_test_cli/default.nix @@ -9,18 +9,18 @@ rustPlatform.buildRustPackage rec { name = "mls-test-cli-${version}"; - version = "0.3.0"; + version = "0.4.0"; nativeBuildInputs = [ pkg-config perl ]; buildInputs = [ libsodium ]; src = fetchFromGitHub { owner = "wireapp"; repo = "mls-test-cli"; - sha256 = "sha256-5CjGd7Di58XvseC0zmkU2Xc5t2qH/g1a6cjDDQvrCsU="; - rev = "aeead948a40d968119c847741f4610a25ab94595"; + sha256 = "sha256-6G01eONZb/61MrO/Py+ix7Psz+jl+3Cn7xUMez3osxw="; + rev = "d01258a290546a01a62dca21ba3d0e3863a288b4"; }; doCheck = false; - cargoSha256 = "sha256-UOB+fiHjz2xUP50CN766aT9TDVpd5Ebd+EDxrddmJbo="; + cargoSha256 = "sha256-frzVXP0lxXhPhfNL4zleHj2WSMwmQfCdTqkTbHXBFEI="; cargoDepsHook = '' - mkdir -p mls-test-cli-0.3.0-vendor.tar.gz/ring/.git + mkdir -p mls-test-cli-${version}-vendor.tar.gz/ring/.git ''; } diff --git a/nix/sources.json b/nix/sources.json index 32398b4aebc..55d7283a6ce 100644 --- a/nix/sources.json +++ b/nix/sources.json @@ -1,14 +1,14 @@ { "nixpkgs": { - "branch": "nixpkgs-unstable", - "description": "A read-only mirror of NixOS/nixpkgs tracking the released channels. Send issues and PRs to", - "homepage": "https://github.com/NixOS/nixpkgs", - "owner": "NixOS", + "branch": "hls-hlint-plugin-workaround", + "description": "Wire's fork of NixOS/nixpkgs. Use until HLS > 1.7.0.0 is available in NixOS/nixpkgs", + "homepage": "https://github.com/wireapp/nixpkgs", + "owner": "wireapp", "repo": "nixpkgs", - "rev": "964d60ff2e6bc76c0618962da52859603784fa78", - "sha256": "1qvs9a59araglrrbzp5zdx81nmjgaxpfp36yl708il0lyqdjawvd", + "rev": "c1f2214fb86c79f577460a3acd1b41ba284178c0", + "sha256": "0hdc53q8xmx6wgcyilxwy450dwhzwakdbbil1hg3vraprr1kpxcp", "type": "tarball", - "url": "https://github.com/NixOS/nixpkgs/archive/964d60ff2e6bc76c0618962da52859603784fa78.tar.gz", + "url": "https://github.com/wireapp/nixpkgs/archive/c1f2214fb86c79f577460a3acd1b41ba284178c0.tar.gz", "url_template": "https://github.com///archive/.tar.gz" } } diff --git a/services/brig/brig.cabal b/services/brig/brig.cabal index b91e6632499..661b91679ea 100644 --- a/services/brig/brig.cabal +++ b/services/brig/brig.cabal @@ -49,11 +49,11 @@ library Brig.Data.Instances Brig.Data.LoginCode Brig.Data.MLS.KeyPackage + Brig.Data.Nonce Brig.Data.Properties Brig.Data.Types Brig.Data.User Brig.Data.UserKey - Brig.Data.UserPendingActivation Brig.Effects.BlacklistPhonePrefixStore Brig.Effects.BlacklistPhonePrefixStore.Cassandra Brig.Effects.BlacklistStore @@ -89,6 +89,8 @@ library Brig.Sem.CodeStore.Cassandra Brig.Sem.PasswordResetStore Brig.Sem.PasswordResetStore.CodeStore + Brig.Sem.UserPendingActivationStore + Brig.Sem.UserPendingActivationStore.Cassandra Brig.SMTP Brig.Team.API Brig.Team.DB @@ -166,7 +168,7 @@ library ghc-options: -O2 -Wall -Wincomplete-uni-patterns -Wincomplete-record-updates -Wpartial-fields -fwarn-tabs -optP-Wno-nonportable-include-path - -funbox-strict-fields + -funbox-strict-fields -fplugin=Polysemy.Plugin build-depends: aeson >=2.0.1.0 @@ -239,6 +241,7 @@ library , optparse-applicative >=0.11 , pem >=0.2 , polysemy + , polysemy-plugin , polysemy-wire-zoo , proto-lens >=0.1 , random-shuffle >=0.0.3 @@ -540,6 +543,7 @@ executable brig-integration , optparse-applicative , pem , polysemy + , polysemy-wire-zoo , process , proto-lens , QuickCheck @@ -650,6 +654,8 @@ executable brig-schema V69_MLSKeyPackageRefMapping V70_UserEmailUnvalidated V71_AddTableVCodesThrottle + V72_AddNonceTable + V73_ReplaceNonceTable V9 hs-source-dirs: schema/src diff --git a/services/brig/brig.integration.yaml b/services/brig/brig.integration.yaml index 16f6ad2c355..05e2eaaec98 100644 --- a/services/brig/brig.integration.yaml +++ b/services/brig/brig.integration.yaml @@ -188,6 +188,7 @@ optSettings: - domain: example.com search_policy: full_search set2FACodeGenerationDelaySecs: 5 + setNonceTtlSecs: 5 logLevel: Warn # ^ NOTE: We log too much in brig, if we set this to Info like other services, running tests diff --git a/services/brig/schema/src/Main.hs b/services/brig/schema/src/Main.hs index eb565236d6d..20667d3a2bf 100644 --- a/services/brig/schema/src/Main.hs +++ b/services/brig/schema/src/Main.hs @@ -81,6 +81,8 @@ import qualified V68_AddMLSPublicKeys import qualified V69_MLSKeyPackageRefMapping import qualified V70_UserEmailUnvalidated import qualified V71_AddTableVCodesThrottle +import qualified V72_AddNonceTable +import qualified V73_ReplaceNonceTable import qualified V9 main :: IO () @@ -151,7 +153,9 @@ main = do V68_AddMLSPublicKeys.migration, V69_MLSKeyPackageRefMapping.migration, V70_UserEmailUnvalidated.migration, - V71_AddTableVCodesThrottle.migration + V71_AddTableVCodesThrottle.migration, + V72_AddNonceTable.migration, + V73_ReplaceNonceTable.migration -- When adding migrations here, don't forget to update -- 'schemaVersion' in Brig.App diff --git a/services/brig/schema/src/V72_AddNonceTable.hs b/services/brig/schema/src/V72_AddNonceTable.hs new file mode 100644 index 00000000000..32c7b71012a --- /dev/null +++ b/services/brig/schema/src/V72_AddNonceTable.hs @@ -0,0 +1,38 @@ +{-# LANGUAGE QuasiQuotes #-} + +-- This file is part of the Wire Server implementation. +-- +-- Copyright (C) 2022 Wire Swiss GmbH +-- +-- This program is free software: you can redistribute it and/or modify it under +-- the terms of the GNU Affero General Public License as published by the Free +-- Software Foundation, either version 3 of the License, or (at your option) any +-- later version. +-- +-- This program is distributed in the hope that it will be useful, but WITHOUT +-- ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +-- FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more +-- details. +-- +-- You should have received a copy of the GNU Affero General Public License along +-- with this program. If not, see . + +module V72_AddNonceTable + ( migration, + ) +where + +import Cassandra.Schema +import Imports +import Text.RawString.QQ + +migration :: Migration +migration = + Migration 72 "Add table nonce" $ do + schema' + [r| + CREATE TABLE IF NOT EXISTS client_nonce + ( nonce uuid + , PRIMARY KEY (nonce) + ) WITH default_time_to_live = 300; + |] diff --git a/services/brig/schema/src/V73_ReplaceNonceTable.hs b/services/brig/schema/src/V73_ReplaceNonceTable.hs new file mode 100644 index 00000000000..94b47f817a2 --- /dev/null +++ b/services/brig/schema/src/V73_ReplaceNonceTable.hs @@ -0,0 +1,44 @@ +{-# LANGUAGE QuasiQuotes #-} + +-- This file is part of the Wire Server implementation. +-- +-- Copyright (C) 2022 Wire Swiss GmbH +-- +-- This program is free software: you can redistribute it and/or modify it under +-- the terms of the GNU Affero General Public License as published by the Free +-- Software Foundation, either version 3 of the License, or (at your option) any +-- later version. +-- +-- This program is distributed in the hope that it will be useful, but WITHOUT +-- ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +-- FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more +-- details. +-- +-- You should have received a copy of the GNU Affero General Public License along +-- with this program. If not, see . + +module V73_ReplaceNonceTable + ( migration, + ) +where + +import Cassandra.Schema +import Imports +import Text.RawString.QQ + +migration :: Migration +migration = + Migration 73 "Replace nonce with a better one" $ do + schema' + [r| + DROP TABLE IF EXISTS client_nonce + |] + schema' + [r| + CREATE TABLE IF NOT EXISTS nonce + ( user uuid, + , key text, + , nonce uuid + , primary key (user, key) + ) WITH default_time_to_live = 300; + |] diff --git a/services/brig/src/Brig/API.hs b/services/brig/src/Brig/API.hs index 61483fc0bb1..d724135c50f 100644 --- a/services/brig/src/Brig/API.hs +++ b/services/brig/src/Brig/API.hs @@ -27,16 +27,19 @@ import Brig.Effects.BlacklistPhonePrefixStore (BlacklistPhonePrefixStore) import Brig.Effects.BlacklistStore (BlacklistStore) import Brig.Sem.CodeStore import Brig.Sem.PasswordResetStore (PasswordResetStore) +import Brig.Sem.UserPendingActivationStore (UserPendingActivationStore) import qualified Data.Swagger.Build.Api as Doc import Network.Wai.Routing (Routes) import Polysemy sitemap :: + forall r p. Members '[ CodeStore, PasswordResetStore, BlacklistStore, - BlacklistPhonePrefixStore + BlacklistPhonePrefixStore, + UserPendingActivationStore p ] r => Routes Doc.ApiBuilder (Handler r) () diff --git a/services/brig/src/Brig/API/Client.hs b/services/brig/src/Brig/API/Client.hs index 8dcb9ce149b..691bc360600 100644 --- a/services/brig/src/Brig/API/Client.hs +++ b/services/brig/src/Brig/API/Client.hs @@ -181,7 +181,6 @@ addClientWithReAuthPolicy policy u con ip new = do ( MonadReader Env m, MonadMask m, MonadHttp m, - MonadIO m, HasRequestId m, Log.MonadLogger m, MonadClient m @@ -228,7 +227,6 @@ rmClient u con clt pw = claimPrekey :: ( MonadReader Env m, - MonadIO m, MonadMask m, MonadHttp m, HasRequestId m, @@ -267,10 +265,6 @@ claimLocalPrekey protectee user client = do claimRemotePrekey :: ( MonadReader Env m, - MonadIO m, - MonadMask m, - MonadHttp m, - HasRequestId m, Log.MonadLogger m, MonadClient m ) => @@ -394,7 +388,6 @@ execDelete u con c = do -- (and possibly duplicated) client data. noPrekeys :: ( MonadReader Env m, - MonadIO m, MonadMask m, MonadHttp m, HasRequestId m, diff --git a/services/brig/src/Brig/API/Error.hs b/services/brig/src/Brig/API/Error.hs index 388a889f52b..c3e8341498f 100644 --- a/services/brig/src/Brig/API/Error.hs +++ b/services/brig/src/Brig/API/Error.hs @@ -18,7 +18,6 @@ module Brig.API.Error where import Brig.API.Types -import Brig.Options (DomainsBlockedForRegistration) import Brig.Phone (PhoneException (..)) import Brig.Types.Common (PhoneBudgetTimeout (..)) import Control.Monad.Error.Class hiding (Error) @@ -414,7 +413,7 @@ legalHoldNotEnabled = Wai.mkError status403 "legalhold-not-enabled" "LegalHold m -- (the tautological constraint in the type signature is added so that once we remove the -- feature, ghc will guide us here.) -customerExtensionBlockedDomain :: (DomainsBlockedForRegistration ~ DomainsBlockedForRegistration) => Domain -> Wai.Error +customerExtensionBlockedDomain :: Domain -> Wai.Error customerExtensionBlockedDomain domain = Wai.mkError (mkStatus 451 "Unavailable For Legal Reasons") "domain-blocked-for-registration" msg where msg = diff --git a/services/brig/src/Brig/API/Federation.hs b/services/brig/src/Brig/API/Federation.hs index 716400e411b..f6cbc7ea957 100644 --- a/services/brig/src/Brig/API/Federation.hs +++ b/services/brig/src/Brig/API/Federation.hs @@ -60,12 +60,11 @@ import Wire.API.Federation.API.Brig import Wire.API.Federation.API.Common import Wire.API.Federation.Version import Wire.API.MLS.KeyPackage -import Wire.API.Message (UserClients) import Wire.API.Routes.Internal.Brig.Connection import Wire.API.Routes.Named import Wire.API.Team.LegalHold (LegalholdProtectee (LegalholdPlusFederationNotImplemented)) import Wire.API.User (UserProfile) -import Wire.API.User.Client (PubClient, UserClientPrekeyMap) +import Wire.API.User.Client import Wire.API.User.Client.Prekey import Wire.API.User.Search import Wire.API.UserMap (UserMap) @@ -213,7 +212,7 @@ searchUsers domain (SearchRequest searchTerm) = do getUserClients :: Domain -> GetUserClients -> (Handler r) (UserMap (Set PubClient)) getUserClients _ (GetUserClients uids) = API.lookupLocalPubClientsBulk uids !>> clientError -getMLSClients :: Domain -> MLSClientsRequest -> Handler r (Set ClientId) +getMLSClients :: Domain -> MLSClientsRequest -> Handler r (Set ClientInfo) getMLSClients _domain mcr = do Internal.getMLSClients (mcrUserId mcr) (mcrSignatureScheme mcr) diff --git a/services/brig/src/Brig/API/Internal.hs b/services/brig/src/Brig/API/Internal.hs index c2e871490f4..83bf3e28cec 100644 --- a/services/brig/src/Brig/API/Internal.hs +++ b/services/brig/src/Brig/API/Internal.hs @@ -46,6 +46,7 @@ import Brig.Options hiding (internalEvents, sesQueue) import qualified Brig.Provider.API as Provider import Brig.Sem.CodeStore (CodeStore) import Brig.Sem.PasswordResetStore (PasswordResetStore) +import Brig.Sem.UserPendingActivationStore (UserPendingActivationStore) import qualified Brig.Team.API as Team import Brig.Team.DB (lookupInvitationByEmail) import Brig.Types.Connection @@ -57,7 +58,6 @@ import qualified Brig.User.API.Auth as Auth import qualified Brig.User.API.Search as Search import qualified Brig.User.EJPD import qualified Brig.User.Search.Index as Index -import Cassandra (MonadClient) import Control.Error hiding (bool) import Control.Lens (view) import Data.Aeson hiding (json) @@ -100,7 +100,13 @@ import Wire.API.User.RichInfo --------------------------------------------------------------------------- -- Sitemap (servant) -servantSitemap :: Members '[BlacklistStore] r => ServerT BrigIRoutes.API (Handler r) +servantSitemap :: + Members + '[ BlacklistStore, + UserPendingActivationStore p + ] + r => + ServerT BrigIRoutes.API (Handler r) servantSitemap = ejpdAPI :<|> accountAPI :<|> mlsAPI :<|> getVerificationCode :<|> teamsAPI :<|> userAPI ejpdAPI :: ServerT BrigIRoutes.EJPD_API (Handler r) @@ -125,7 +131,13 @@ mlsAPI = :<|> getMLSClients :<|> mapKeyPackageRefsInternal -accountAPI :: Member BlacklistStore r => ServerT BrigIRoutes.AccountAPI (Handler r) +accountAPI :: + Members + '[ BlacklistStore, + UserPendingActivationStore p + ] + r => + ServerT BrigIRoutes.AccountAPI (Handler r) accountAPI = Named @"createUserNoVerify" createUserNoVerify :<|> Named @"createUserNoVerifySpar" createUserNoVerifySpar @@ -172,19 +184,22 @@ getConvIdByKeyPackageRef = runMaybeT . mapMaybeT wrapClientE . Data.keyPackageRe postKeyPackageRef :: KeyPackageRef -> KeyPackageRef -> Handler r () postKeyPackageRef ref = lift . wrapClient . Data.updateKeyPackageRef ref -getMLSClients :: UserId -> SignatureSchemeTag -> Handler r (Set ClientId) -getMLSClients usr ss = do - results <- lift (wrapClient (API.lookupUsersClientIds (pure usr))) >>= getResult - keys <- lift . wrapClient $ pooledMapConcurrentlyN 16 getKey (toList results) - pure . Set.fromList . map fst . filter (isJust . snd) $ keys +getMLSClients :: UserId -> SignatureSchemeTag -> Handler r (Set ClientInfo) +getMLSClients usr _ss = do + -- FUTUREWORK: check existence of key packages with a given ciphersuite + lusr <- qualifyLocal usr + allClients <- lift (wrapClient (API.lookupUsersClientIds (pure usr))) >>= getResult + clientInfo <- lift . wrapClient $ pooledMapConcurrentlyN 16 (getValidity lusr) (toList allClients) + pure . Set.fromList . map (uncurry ClientInfo) $ clientInfo where getResult [] = pure mempty getResult ((u, cs) : rs) | u == usr = pure cs | otherwise = getResult rs - getKey :: MonadClient m => ClientId -> m (ClientId, Maybe LByteString) - getKey cid = (cid,) <$> Data.lookupMLSPublicKey usr cid ss + getValidity lusr cid = + fmap ((cid,) . (> 0)) $ + Data.countKeyPackages lusr cid mapKeyPackageRefsInternal :: KeyPackageBundle -> Handler r () mapKeyPackageRefsInternal bundle = do @@ -192,7 +207,7 @@ mapKeyPackageRefsInternal bundle = do for_ (kpbEntries bundle) $ \e -> Data.mapKeyPackageRef (kpbeRef e) (kpbeUser e) (kpbeClient e) -getVerificationCode :: UserId -> VerificationAction -> (Handler r) (Maybe Code.Value) +getVerificationCode :: UserId -> VerificationAction -> Handler r (Maybe Code.Value) getVerificationCode uid action = do user <- wrapClientE $ Api.lookupUser NoPendingInvitations uid maybe (pure Nothing) (lookupCode action) (userEmail =<< user) @@ -214,7 +229,8 @@ sitemap :: '[ CodeStore, PasswordResetStore, BlacklistStore, - BlacklistPhonePrefixStore + BlacklistPhonePrefixStore, + UserPendingActivationStore p ] r => Routes a (Handler r) () @@ -420,7 +436,14 @@ internalListFullClients :: UserSet -> (AppT r) UserClientsFull internalListFullClients (UserSet usrs) = UserClientsFull <$> wrapClient (Data.lookupClientsBulk (Set.toList usrs)) -createUserNoVerify :: Member BlacklistStore r => NewUser -> (Handler r) (Either RegisterError SelfProfile) +createUserNoVerify :: + Members + '[ BlacklistStore, + UserPendingActivationStore p + ] + r => + NewUser -> + (Handler r) (Either RegisterError SelfProfile) createUserNoVerify uData = lift . runExceptT $ do result <- API.createUser uData let acc = createdAccount result diff --git a/services/brig/src/Brig/API/Public.hs b/services/brig/src/Brig/API/Public.hs index 09ed538c151..fa51d1b9255 100644 --- a/services/brig/src/Brig/API/Public.hs +++ b/services/brig/src/Brig/API/Public.hs @@ -41,6 +41,7 @@ import Brig.App import qualified Brig.Calling.API as Calling import qualified Brig.Code as Code import qualified Brig.Data.Connection as Data +import Brig.Data.Nonce as Nonce import qualified Brig.Data.User as Data import qualified Brig.Data.UserKey as UserKey import Brig.Effects.BlacklistPhonePrefixStore (BlacklistPhonePrefixStore) @@ -50,6 +51,7 @@ import Brig.Options hiding (internalEvents, sesQueue) import qualified Brig.Provider.API as Provider import Brig.Sem.CodeStore (CodeStore) import Brig.Sem.PasswordResetStore (PasswordResetStore) +import Brig.Sem.UserPendingActivationStore (UserPendingActivationStore) import qualified Brig.Team.API as Team import qualified Brig.Team.Email as Team import Brig.Types.Activation (ActivationPair) @@ -80,6 +82,7 @@ import Data.Handle (Handle, parseHandle) import Data.Id as Id import qualified Data.Map.Strict as Map import Data.Misc (IpAddr (..)) +import Data.Nonce (Nonce, randomNonce) import Data.Qualified import Data.Range import qualified Data.Swagger as S @@ -183,10 +186,11 @@ swaggerDocsAPI (Just V1) = swaggerDocsAPI Nothing = swaggerDocsAPI (Just maxBound) servantSitemap :: - forall r. + forall r p. Members '[ BlacklistStore, - BlacklistPhonePrefixStore + BlacklistPhonePrefixStore, + UserPendingActivationStore p ] r => ServerT BrigAPI (Handler r) @@ -248,6 +252,8 @@ servantSitemap = userAPI :<|> selfAPI :<|> accountAPI :<|> clientAPI :<|> prekey :<|> Named @"get-client" getClient :<|> Named @"get-client-capabilities" getClientCapabilities :<|> Named @"get-client-prekeys" getClientPrekeys + :<|> Named @"head-nonce" newNonce + :<|> Named @"get-nonce" newNonce connectionAPI :: ServerT ConnectionAPI (Handler r) connectionAPI = @@ -613,8 +619,22 @@ getRichInfo self user = do getClientPrekeys :: UserId -> ClientId -> (Handler r) [Public.PrekeyId] getClientPrekeys usr clt = lift (wrapClient $ API.lookupPrekeyIds usr clt) +newNonce :: UserId -> ClientId -> (Handler r) Nonce +newNonce uid cid = do + ttl <- setNonceTtlSecs <$> view settings + nonce <- randomNonce + lift $ wrapClient $ Nonce.insertNonce ttl uid (client cid) nonce + pure nonce + -- | docs/reference/user/registration.md {#RefRegistration} -createUser :: Member BlacklistStore r => Public.NewUserPublic -> (Handler r) (Either Public.RegisterError Public.RegisterSuccess) +createUser :: + Members + '[ BlacklistStore, + UserPendingActivationStore p + ] + r => + Public.NewUserPublic -> + (Handler r) (Either Public.RegisterError Public.RegisterSuccess) createUser (Public.NewUserPublic new) = lift . runExceptT $ do API.checkRestrictedUserCreation new for_ (Public.newUserEmail new) $ mapExceptT wrapHttp . checkWhitelistWithError RegisterErrorWhitelistError . Left @@ -870,7 +890,7 @@ sendActivationCode Public.SendActivationCode {..} = do -- -- The tautological constraint in the type signature is added so that once we remove the -- feature, ghc will guide us here. -customerExtensionCheckBlockedDomains :: (DomainsBlockedForRegistration ~ DomainsBlockedForRegistration) => Public.Email -> (Handler r) () +customerExtensionCheckBlockedDomains :: Public.Email -> (Handler r) () customerExtensionCheckBlockedDomains email = do mBlockedDomains <- asks (fmap domainsBlockedForRegistration . setCustomerExtensions . view settings) for_ mBlockedDomains $ \(DomainsBlockedForRegistration blockedDomains) -> do diff --git a/services/brig/src/Brig/API/User.hs b/services/brig/src/Brig/API/User.hs index 3ba1eb47c31..afd82173ec9 100644 --- a/services/brig/src/Brig/API/User.hs +++ b/services/brig/src/Brig/API/User.hs @@ -106,8 +106,6 @@ import Brig.Data.User import qualified Brig.Data.User as Data import Brig.Data.UserKey import qualified Brig.Data.UserKey as Data -import Brig.Data.UserPendingActivation -import qualified Brig.Data.UserPendingActivation as Data import Brig.Effects.BlacklistPhonePrefixStore (BlacklistPhonePrefixStore) import qualified Brig.Effects.BlacklistPhonePrefixStore as BlacklistPhonePrefixStore import Brig.Effects.BlacklistStore (BlacklistStore) @@ -122,6 +120,8 @@ import Brig.Sem.CodeStore (CodeStore) import qualified Brig.Sem.CodeStore as E import Brig.Sem.PasswordResetStore (PasswordResetStore) import qualified Brig.Sem.PasswordResetStore as E +import Brig.Sem.UserPendingActivationStore (UserPendingActivation (..), UserPendingActivationStore) +import qualified Brig.Sem.UserPendingActivationStore as UserPendingActivationStore import qualified Brig.Team.DB as Team import Brig.Types.Activation (ActivationPair) import Brig.Types.Connection @@ -277,7 +277,15 @@ createUserSpar new = do pure $ CreateUserTeam tid nm -- docs/reference/user/registration.md {#RefRegistration} -createUser :: forall r. Member BlacklistStore r => NewUser -> ExceptT RegisterError (AppT r) CreateUserResult +createUser :: + forall r p. + Members + '[ BlacklistStore, + UserPendingActivationStore p + ] + r => + NewUser -> + ExceptT RegisterError (AppT r) CreateUserResult createUser new = do (email, phone) <- validateEmailAndPhone new @@ -448,8 +456,8 @@ createUser new = do field "user" (toByteString uid) . field "team" (toByteString $ Team.iiTeam ii) . msg (val "Accepting invitation") + liftSem $ UserPendingActivationStore.remove uid wrapClient $ do - Data.usersPendingActivationRemove uid Team.deleteInvitation (Team.inTeam inv) (Team.inInvitation inv) addUserToTeamSSO :: UserAccount -> TeamId -> UserIdentity -> ExceptT RegisterError (AppT r) CreateUserTeam @@ -512,7 +520,15 @@ initAccountFeatureConfig uid = do -- | 'createUser' is becoming hard to maintian, and instead of adding more case distinctions -- all over the place there, we add a new function that handles just the one new flow where -- users are invited to the team via scim. -createUserInviteViaScim :: Member BlacklistStore r => UserId -> NewUserScimInvitation -> ExceptT Error.Error (AppT r) UserAccount +createUserInviteViaScim :: + Members + '[ BlacklistStore, + UserPendingActivationStore p + ] + r => + UserId -> + NewUserScimInvitation -> + ExceptT Error.Error (AppT r) UserAccount createUserInviteViaScim uid (NewUserScimInvitation tid loc name rawEmail) = do email <- either (const . throwE . Error.StdError $ errorToWai @'E.InvalidEmail) pure (validateEmail rawEmail) let emKey = userEmailKey email @@ -526,7 +542,7 @@ createUserInviteViaScim uid (NewUserScimInvitation tid loc name rawEmail) = do ttl <- setTeamInvitationTimeout <$> view settings now <- liftIO =<< view currentTime pure $ addUTCTime (realToFrac ttl) now - lift . wrapClient $ Data.usersPendingActivationAdd (UserPendingActivation uid expiresAt) + lift . liftSem $ UserPendingActivationStore.add (UserPendingActivation uid expiresAt) let activated = -- treating 'PendingActivation' as 'Active', but then 'Brig.Data.User.toIdentity' @@ -1220,11 +1236,8 @@ verifyDeleteUser d = do -- other owner left. deleteAccount :: ( MonadLogger m, - MonadCatch m, - MonadThrow m, MonadIndexIO m, MonadReader Env m, - MonadIO m, MonadMask m, MonadHttp m, HasRequestId m, diff --git a/services/brig/src/Brig/AWS.hs b/services/brig/src/Brig/AWS.hs index 85c081a5149..640bb3c8309 100644 --- a/services/brig/src/Brig/AWS.hs +++ b/services/brig/src/Brig/AWS.hs @@ -245,7 +245,7 @@ throwA :: Either AWS.Error a -> Amazon a throwA = either (throwM . GeneralError) pure execCatch :: - (AWSRequest a, MonadUnliftIO m, MonadCatch m, MonadThrow m, MonadIO m) => + (AWSRequest a, MonadUnliftIO m, MonadCatch m) => AWS.Env -> a -> m (Either AWS.Error (AWSResponse a)) @@ -255,7 +255,7 @@ execCatch e cmd = AWS.send e cmd exec :: - (AWSRequest a, MonadCatch m, MonadThrow m, MonadIO m) => + (AWSRequest a, MonadCatch m, MonadIO m) => AWS.Env -> a -> m (AWSResponse a) diff --git a/services/brig/src/Brig/App.hs b/services/brig/src/Brig/App.hs index 7cb9b5a26e3..aa6765dcef1 100644 --- a/services/brig/src/Brig/App.hs +++ b/services/brig/src/Brig/App.hs @@ -148,7 +148,7 @@ import Util.Options import Wire.API.User schemaVersion :: Int32 -schemaVersion = 71 +schemaVersion = 73 ------------------------------------------------------------------------------- -- Environment @@ -494,7 +494,7 @@ instance MonadLogger (AppT r) where instance MonadLogger (ExceptT err (AppT r)) where log l m = lift (LC.log l m) -instance MonadIO m => MonadHttp (AppT r) where +instance MonadHttp (AppT r) where handleRequestWithCont req handler = do manager <- view httpManager liftIO $ withResponse req manager handler @@ -577,7 +577,7 @@ instance MonadIndexIO (AppT r) where instance MonadIndexIO (AppT r) => MonadIndexIO (ExceptT err (AppT r)) where liftIndexIO m = view indexEnv >>= \e -> runIndexIO e m -instance Monad m => HasRequestId (AppT r) where +instance HasRequestId (AppT r) where getRequestId = view requestId locationOf :: (MonadIO m, MonadReader Env m) => IP -> m (Maybe Location) diff --git a/services/brig/src/Brig/CanonicalInterpreter.hs b/services/brig/src/Brig/CanonicalInterpreter.hs index 72ae9ee2cb9..6460a79c42f 100644 --- a/services/brig/src/Brig/CanonicalInterpreter.hs +++ b/services/brig/src/Brig/CanonicalInterpreter.hs @@ -9,17 +9,21 @@ import Brig.Sem.CodeStore (CodeStore) import Brig.Sem.CodeStore.Cassandra (codeStoreToCassandra, interpretClientToIO) import Brig.Sem.PasswordResetStore (PasswordResetStore) import Brig.Sem.PasswordResetStore.CodeStore (passwordResetStoreToCodeStore) +import Brig.Sem.UserPendingActivationStore (UserPendingActivationStore) +import Brig.Sem.UserPendingActivationStore.Cassandra (userPendingActivationStoreToCassandra) import qualified Cassandra as Cas import Control.Lens ((^.)) import Imports import Polysemy (Embed, Final, embedToFinal, runFinal) import Wire.Sem.Now (Now) import Wire.Sem.Now.IO (nowToIOAction) +import Wire.Sem.Paging.Cassandra (InternalPaging) type BrigCanonicalEffects = '[ BlacklistPhonePrefixStore, BlacklistStore, PasswordResetStore, + UserPendingActivationStore InternalPaging, Now, CodeStore, Embed Cas.Client, @@ -34,6 +38,7 @@ runBrigToIO e (AppT ma) = . interpretClientToIO (e ^. casClient) . codeStoreToCassandra @Cas.Client . nowToIOAction (e ^. currentTime) + . userPendingActivationStoreToCassandra . passwordResetStoreToCodeStore . interpretBlacklistStoreToCassandra @Cas.Client . interpretBlacklistPhonePrefixStoreToCassandra @Cas.Client diff --git a/services/brig/src/Brig/Data/Activation.hs b/services/brig/src/Brig/Data/Activation.hs index d7954609fe0..71efc8a17f9 100644 --- a/services/brig/src/Brig/Data/Activation.hs +++ b/services/brig/src/Brig/Data/Activation.hs @@ -132,7 +132,7 @@ activateKey k c u = verifyCode k c >>= pickUser >>= activate for_ oldKey $ lift . deleteKey pure . Just $ foldKey (EmailActivated uid) (PhoneActivated uid) key where - updateEmailAndDeleteEmailUnvalidated :: MonadClient m => UserId -> Email -> m () + updateEmailAndDeleteEmailUnvalidated :: UserId -> Email -> m () updateEmailAndDeleteEmailUnvalidated u' email = updateEmail u' email <* deleteEmailUnvalidated u' claim key uid = do diff --git a/services/brig/src/Brig/Data/MLS/KeyPackage.hs b/services/brig/src/Brig/Data/MLS/KeyPackage.hs index e7bcb1f9609..130cd96f851 100644 --- a/services/brig/src/Brig/Data/MLS/KeyPackage.hs +++ b/services/brig/src/Brig/Data/MLS/KeyPackage.hs @@ -212,19 +212,19 @@ updateKeyPackageRef prevRef newRef = -------------------------------------------------------------------------------- -- Utilities -backupKeyPackageMeta :: MonadClient m => KeyPackageRef -> MaybeT m (ClientId, Qualified ConvId, Qualified UserId) +backupKeyPackageMeta :: MonadClient m => KeyPackageRef -> MaybeT m (ClientId, Maybe (Qualified ConvId), Qualified UserId) backupKeyPackageMeta ref = do (clientId, convId, convDomain, userDomain, userId) <- MaybeT . retry x1 $ query1 q (params LocalQuorum (Identity ref)) - pure (clientId, Qualified convId convDomain, Qualified userId userDomain) + pure (clientId, Qualified <$> convId <*> convDomain, Qualified userId userDomain) where - q :: PrepQuery R (Identity KeyPackageRef) (ClientId, ConvId, Domain, Domain, UserId) + q :: PrepQuery R (Identity KeyPackageRef) (ClientId, Maybe ConvId, Maybe Domain, Domain, UserId) q = "SELECT client, conv, conv_domain, domain, user FROM mls_key_package_refs WHERE ref = ?" -restoreKeyPackageMeta :: MonadClient m => KeyPackageRef -> (ClientId, Qualified ConvId, Qualified UserId) -> m () +restoreKeyPackageMeta :: MonadClient m => KeyPackageRef -> (ClientId, Maybe (Qualified ConvId), Qualified UserId) -> m () restoreKeyPackageMeta ref (clientId, convId, userId) = do - write q (params LocalQuorum (ref, clientId, qUnqualified convId, qDomain convId, qDomain userId, qUnqualified userId)) + write q (params LocalQuorum (ref, clientId, qUnqualified <$> convId, qDomain <$> convId, qDomain userId, qUnqualified userId)) where - q :: PrepQuery W (KeyPackageRef, ClientId, ConvId, Domain, Domain, UserId) () + q :: PrepQuery W (KeyPackageRef, ClientId, Maybe ConvId, Maybe Domain, Domain, UserId) () q = "INSERT INTO mls_key_package_refs (ref, client, conv, conv_domain, domain, user) VALUES (?, ?, ?, ?, ?, ?)" deleteKeyPackage :: MonadClient m => KeyPackageRef -> m () diff --git a/services/brig/src/Brig/Data/Nonce.hs b/services/brig/src/Brig/Data/Nonce.hs new file mode 100644 index 00000000000..9653ea43138 --- /dev/null +++ b/services/brig/src/Brig/Data/Nonce.hs @@ -0,0 +1,69 @@ +-- This file is part of the Wire Server implementation. +-- +-- Copyright (C) 2022 Wire Swiss GmbH +-- +-- This program is free software: you can redistribute it and/or modify it under +-- the terms of the GNU Affero General Public License as published by the Free +-- Software Foundation, either version 3 of the License, or (at your option) any +-- later version. +-- +-- This program is distributed in the hope that it will be useful, but WITHOUT +-- ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +-- FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more +-- details. +-- +-- You should have received a copy of the GNU Affero General Public License along +-- with this program. If not, see . + +module Brig.Data.Nonce + ( insertNonce, + lookupAndDeleteNonce, + ) +where + +import Brig.App (Env) +import Brig.Data.Instances () +import Cassandra +import Control.Lens hiding (from) +import Data.Id (UserId) +import Data.Nonce (Nonce, NonceTtlSecs) +import Imports + +insertNonce :: + (MonadClient m, MonadReader Brig.App.Env m) => + NonceTtlSecs -> + UserId -> + Text -> + Nonce -> + m () +insertNonce ttl uid key nonce = retry x5 . write insert $ params LocalQuorum (uid, key, nonce, ttl) + where + insert :: PrepQuery W (UserId, Text, Nonce, NonceTtlSecs) () + insert = "INSERT INTO nonce (user, key, nonce) VALUES (?, ?, ?) USING TTL ?" + +lookupAndDeleteNonce :: + (MonadClient m, MonadReader Env m) => + UserId -> + Text -> + m (Maybe Nonce) +lookupAndDeleteNonce uid key = lookupNonce uid key <* deleteNonce uid key + +lookupNonce :: + (MonadClient m, MonadReader Env m) => + UserId -> + Text -> + m (Maybe Nonce) +lookupNonce uid key = (runIdentity <$$>) . retry x5 . query1 get $ params LocalQuorum (uid, key) + where + get :: PrepQuery R (UserId, Text) (Identity Nonce) + get = "SELECT nonce FROM nonce WHERE user = ? AND key = ?" + +deleteNonce :: + (MonadClient m, MonadReader Env m) => + UserId -> + Text -> + m () +deleteNonce uid key = retry x5 . write delete $ params LocalQuorum (uid, key) + where + delete :: PrepQuery W (UserId, Text) () + delete = "DELETE FROM nonce WHERE user = ? AND key = ?" diff --git a/services/brig/src/Brig/Data/User.hs b/services/brig/src/Brig/Data/User.hs index badd37afe12..7aa3adbb824 100644 --- a/services/brig/src/Brig/Data/User.hs +++ b/services/brig/src/Brig/Data/User.hs @@ -434,7 +434,7 @@ lookupUserTeam u = (runIdentity =<<) <$> retry x1 (query1 teamSelect (params LocalQuorum (Identity u))) -lookupAuth :: MonadClient m => (MonadClient m) => UserId -> m (Maybe (Maybe Password, AccountStatus)) +lookupAuth :: MonadClient m => UserId -> m (Maybe (Maybe Password, AccountStatus)) lookupAuth u = fmap f <$> retry x1 (query1 authSelect (params LocalQuorum (Identity u))) where f (pw, st) = (pw, fromMaybe Active st) diff --git a/services/brig/src/Brig/Data/UserPendingActivation.hs b/services/brig/src/Brig/Data/UserPendingActivation.hs deleted file mode 100644 index 23a56684281..00000000000 --- a/services/brig/src/Brig/Data/UserPendingActivation.hs +++ /dev/null @@ -1,62 +0,0 @@ --- This file is part of the Wire Server implementation. --- --- Copyright (C) 2022 Wire Swiss GmbH --- --- This program is free software: you can redistribute it and/or modify it under --- the terms of the GNU Affero General Public License as published by the Free --- Software Foundation, either version 3 of the License, or (at your option) any --- later version. --- --- This program is distributed in the hope that it will be useful, but WITHOUT --- ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS --- FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more --- details. --- --- You should have received a copy of the GNU Affero General Public License along --- with this program. If not, see . - -module Brig.Data.UserPendingActivation - ( usersPendingActivationAdd, - usersPendingActivationList, - usersPendingActivationRemove, - usersPendingActivationRemoveMultiple, - UserPendingActivation (..), - ) -where - -import Cassandra -import Data.Id (UserId) -import Data.Time (UTCTime) -import Imports - -data UserPendingActivation = UserPendingActivation - { upaUserId :: !UserId, - upaDay :: !UTCTime - } - deriving stock (Eq, Show, Ord) - -usersPendingActivationAdd :: MonadClient m => UserPendingActivation -> m () -usersPendingActivationAdd (UserPendingActivation uid expiresAt) = do - retry x5 . write insertExpiration . params LocalQuorum $ (uid, expiresAt) - where - insertExpiration :: PrepQuery W (UserId, UTCTime) () - insertExpiration = "INSERT INTO users_pending_activation (user, expires_at) VALUES (?, ?)" - -usersPendingActivationList :: MonadClient m => m (Page UserPendingActivation) -usersPendingActivationList = do - uncurry UserPendingActivation <$$> retry x1 (paginate selectExpired (params LocalQuorum ())) - where - selectExpired :: PrepQuery R () (UserId, UTCTime) - selectExpired = - "SELECT user, expires_at FROM users_pending_activation" - -usersPendingActivationRemove :: MonadClient m => UserId -> m () -usersPendingActivationRemove uid = usersPendingActivationRemoveMultiple [uid] - -usersPendingActivationRemoveMultiple :: MonadClient m => [UserId] -> m () -usersPendingActivationRemoveMultiple uids = - retry x5 . write deleteExpired . params LocalQuorum $ Identity uids - where - deleteExpired :: PrepQuery W (Identity [UserId]) () - deleteExpired = - "DELETE FROM users_pending_activation WHERE user IN ?" diff --git a/services/brig/src/Brig/Federation/Client.hs b/services/brig/src/Brig/Federation/Client.hs index 23ab8a2cb22..7a93b4371c0 100644 --- a/services/brig/src/Brig/Federation/Client.hs +++ b/services/brig/src/Brig/Federation/Client.hs @@ -138,7 +138,6 @@ notifyUserDeleted :: ( MonadReader Env m, MonadIO m, HasFedEndpoint 'Brig api "on-user-deleted-connections", - HasClient ClientM api, HasClient (FederatorClient 'Brig) api ) => Local UserId -> diff --git a/services/brig/src/Brig/IO/Intra.hs b/services/brig/src/Brig/IO/Intra.hs index 51d1b6c57a4..c22c8a178ae 100644 --- a/services/brig/src/Brig/IO/Intra.hs +++ b/services/brig/src/Brig/IO/Intra.hs @@ -143,11 +143,8 @@ import Wire.API.User.Client onUserEvent :: ( MonadLogger m, - MonadCatch m, - MonadThrow m, MonadIndexIO m, MonadReader Env m, - MonadIO m, MonadMask m, MonadHttp m, HasRequestId m, @@ -215,11 +212,8 @@ onClientEvent orig conn e = do updateSearchIndex :: ( MonadClient m, - MonadLogger m, MonadCatch m, MonadLogger m, - MonadCatch m, - MonadThrow m, MonadIndexIO m ) => UserId -> @@ -277,7 +271,6 @@ dispatchNotifications :: Log.MonadLogger m, MonadReader Env m, MonadMask m, - MonadCatch m, MonadHttp m, HasRequestId m, MonadUnliftIO m, @@ -314,7 +307,6 @@ notifyUserDeletionLocals :: Log.MonadLogger m, MonadReader Env m, MonadMask m, - MonadCatch m, MonadHttp m, HasRequestId m, MonadUnliftIO m, @@ -331,7 +323,6 @@ notifyUserDeletionLocals deleted conn event = do notifyUserDeletionRemotes :: forall m. ( MonadReader Env m, - MonadIO m, MonadClient m, MonadLogger m ) => @@ -358,7 +349,7 @@ notifyUserDeletionRemotes deleted = do whenLeft eitherFErr $ logFederationError (tDomain uids) - logFederationError :: Log.MonadLogger m => Domain -> FederationError -> m () + logFederationError :: Domain -> FederationError -> m () logFederationError domain fErr = Log.err $ Log.msg ("Federation error while notifying remote backends of a user deletion." :: ByteString) @@ -372,7 +363,6 @@ push :: Log.MonadLogger m, MonadReader Env m, MonadMask m, - MonadCatch m, MonadHttp m, HasRequestId m ) => @@ -404,7 +394,6 @@ rawPush :: Log.MonadLogger m, MonadReader Env m, MonadMask m, - MonadCatch m, MonadHttp m, HasRequestId m ) => @@ -458,7 +447,6 @@ notify :: Log.MonadLogger m, MonadReader Env m, MonadMask m, - MonadCatch m, MonadHttp m, HasRequestId m, MonadUnliftIO m @@ -499,7 +487,6 @@ notifySelf :: Log.MonadLogger m, MonadReader Env m, MonadMask m, - MonadCatch m, MonadHttp m, HasRequestId m, MonadUnliftIO m @@ -518,7 +505,6 @@ notifySelf events orig route conn = notifyContacts :: forall m. ( MonadReader Env m, - MonadIO m, MonadMask m, MonadHttp m, HasRequestId m, @@ -538,7 +524,7 @@ notifyContacts events orig route conn = do notify events orig route conn $ list1 orig <$> liftA2 (++) contacts teamContacts where - contacts :: MonadClient m => m [UserId] + contacts :: m [UserId] contacts = lookupContactList orig teamContacts :: m [UserId] @@ -929,8 +915,7 @@ upsertOne2OneConversation :: MonadIO m, MonadMask m, MonadHttp m, - HasRequestId m, - MonadLogger m + HasRequestId m ) => UpsertOne2OneConversationRequest -> m UpsertOne2OneConversationResponse @@ -1079,8 +1064,7 @@ lookupPushToken :: MonadIO m, MonadMask m, MonadHttp m, - HasRequestId m, - MonadLogger m + HasRequestId m ) => UserId -> m [Push.PushToken] diff --git a/services/brig/src/Brig/Index/Eval.hs b/services/brig/src/Brig/Index/Eval.hs index 7e01426715f..e7629392ea2 100644 --- a/services/brig/src/Brig/Index/Eval.hs +++ b/services/brig/src/Brig/Index/Eval.hs @@ -111,7 +111,7 @@ runCommand l = \case . C.setProtocolVersion C.V4 $ C.defSettings -waitForTaskToComplete :: forall a m. (ES.MonadBH m, MonadIO m, MonadThrow m, FromJSON a) => Int -> ES.TaskNodeId -> m () +waitForTaskToComplete :: forall a m. (ES.MonadBH m, MonadThrow m, FromJSON a) => Int -> ES.TaskNodeId -> m () waitForTaskToComplete timeoutSeconds taskNodeId = do -- Delay is 0.1 seconds, so retries are limited to timeoutSeconds * 10 let policy = constantDelay 100000 <> limitRetries (timeoutSeconds * 10) @@ -130,7 +130,7 @@ waitForTaskToComplete timeoutSeconds taskNodeId = do isTaskComplete (Left e) = throwM $ ReindexFromAnotherIndexError $ "Error response while getting task: " <> show e isTaskComplete (Right taskRes) = pure $ ES.taskResponseCompleted taskRes - errTaskGet :: MonadThrow m => ES.EsError -> m x + errTaskGet :: ES.EsError -> m x errTaskGet e = throwM $ ReindexFromAnotherIndexError $ "Error response while getting task: " <> show e newtype ReindexFromAnotherIndexError = ReindexFromAnotherIndexError String diff --git a/services/brig/src/Brig/Index/Migrations.hs b/services/brig/src/Brig/Index/Migrations.hs index efe394ecd04..d0879d3a0d3 100644 --- a/services/brig/src/Brig/Index/Migrations.hs +++ b/services/brig/src/Brig/Index/Migrations.hs @@ -95,7 +95,7 @@ mkEnv l es cas galleyEndpoint = do $ C.defSettings initLogger = pure l -createMigrationsIndexIfNotPresent :: (MonadThrow m, MonadIO m, ES.MonadBH m, Log.MonadLogger m) => m () +createMigrationsIndexIfNotPresent :: (MonadThrow m, ES.MonadBH m, Log.MonadLogger m) => m () createMigrationsIndexIfNotPresent = do unlessM (ES.indexExists indexName) $ do @@ -111,7 +111,7 @@ createMigrationsIndexIfNotPresent = throwM $ err (show response) -failIfIndexAbsent :: (MonadThrow m, MonadIO m, ES.MonadBH m) => ES.IndexName -> m () +failIfIndexAbsent :: (MonadThrow m, ES.MonadBH m) => ES.IndexName -> m () failIfIndexAbsent targetIndex = unlessM (ES.indexExists targetIndex) @@ -135,7 +135,7 @@ runMigration ver = do . Log.field "expectedVersion" vmax . Log.field "foundVersion" ver -persistVersion :: (Monad m, MonadThrow m, MonadIO m) => MigrationVersion -> MigrationActionT m () +persistVersion :: (MonadThrow m, MonadIO m) => MigrationVersion -> MigrationActionT m () persistVersion v = let docId = ES.DocId . Text.pack . show $ migrationVersion v in do diff --git a/services/brig/src/Brig/InternalEvent/Process.hs b/services/brig/src/Brig/InternalEvent/Process.hs index 3e1baa90e57..abfd920e89e 100644 --- a/services/brig/src/Brig/InternalEvent/Process.hs +++ b/services/brig/src/Brig/InternalEvent/Process.hs @@ -43,10 +43,8 @@ import UnliftIO (timeout) onEvent :: ( Log.MonadLogger m, MonadCatch m, - MonadThrow m, MonadIndexIO m, MonadReader Env m, - MonadIO m, MonadMask m, MonadHttp m, HasRequestId m, diff --git a/services/brig/src/Brig/Options.hs b/services/brig/src/Brig/Options.hs index a53eeaab217..dab2dbb1ebf 100644 --- a/services/brig/src/Brig/Options.hs +++ b/services/brig/src/Brig/Options.hs @@ -38,6 +38,7 @@ import Data.Domain (Domain (..)) import Data.Id import Data.LanguageCodes (ISO639_1 (EN)) import Data.Misc (HttpsUrl) +import Data.Nonce import Data.Range import Data.Schema import Data.Scientific (toBoundedInteger) @@ -51,10 +52,10 @@ import Imports import qualified Network.DNS as DNS import System.Logger.Extended (Level, LogFormat) import Util.Options -import Wire.API.Arbitrary (Arbitrary, arbitrary) import qualified Wire.API.Team.Feature as Public import Wire.API.User import Wire.API.User.Search (FederatedUserSearchPolicy) +import Wire.Arbitrary (Arbitrary, arbitrary) newtype Timeout = Timeout { timeoutDiff :: NominalDiffTime @@ -589,7 +590,10 @@ data Settings = Settings setEnableDevelopmentVersions :: Maybe Bool, -- | Minimum delay in seconds between consecutive attempts to generate a new verification code. -- use `set2FACodeGenerationDelaySecs` as the getter function which always provides a default value - set2FACodeGenerationDelaySecsInternal :: !(Maybe Int) + set2FACodeGenerationDelaySecsInternal :: !(Maybe Int), + -- | The time-to-live of a nonce in seconds. + -- use `setNonceTtlSecs` as the getter function which always provides a default value + setNonceTtlSecsInternal :: !(Maybe NonceTtlSecs) } deriving (Show, Generic) @@ -617,6 +621,12 @@ def2FACodeGenerationDelaySecs = 5 * 60 -- 5 minutes set2FACodeGenerationDelaySecs :: Settings -> Int set2FACodeGenerationDelaySecs = fromMaybe def2FACodeGenerationDelaySecs . set2FACodeGenerationDelaySecsInternal +defaultNonceTtlSecs :: NonceTtlSecs +defaultNonceTtlSecs = NonceTtlSecs $ 5 * 60 -- 5 minutes + +setNonceTtlSecs :: Settings -> NonceTtlSecs +setNonceTtlSecs = fromMaybe defaultNonceTtlSecs . setNonceTtlSecsInternal + -- | The analog to `GT.FeatureFlags`. This type tracks only the things that we need to -- express our current cloud business logic. -- @@ -797,6 +807,7 @@ instance FromJSON Settings where "setDefaultTemplateLocaleInternal" -> "setDefaultTemplateLocale" "setVerificationCodeTimeoutInternal" -> "setVerificationTimeout" "set2FACodeGenerationDelaySecsInternal" -> "set2FACodeGenerationDelaySecs" + "setNonceTtlSecsInternal" -> "setNonceTtlSecs" other -> other } diff --git a/services/brig/src/Brig/Provider/API.hs b/services/brig/src/Brig/Provider/API.hs index 937cc677955..061cdf4011d 100644 --- a/services/brig/src/Brig/Provider/API.hs +++ b/services/brig/src/Brig/Provider/API.hs @@ -689,7 +689,6 @@ deleteService pid sid del = do finishDeleteService :: ( MonadReader Env m, - MonadIO m, MonadMask m, MonadHttp m, HasRequestId m, @@ -715,7 +714,6 @@ finishDeleteService pid sid = do deleteAccountH :: ( MonadReader Env m, - MonadIO m, MonadMask m, MonadHttp m, MonadClient m, @@ -730,7 +728,6 @@ deleteAccountH (pid ::: req) = do deleteAccount :: ( MonadReader Env m, - MonadIO m, MonadMask m, MonadHttp m, MonadClient m, @@ -1109,7 +1106,6 @@ activate pid old new = do deleteBot :: ( MonadHttp m, MonadReader Env m, - MonadIO m, MonadMask m, HasRequestId m, MonadLogger m, diff --git a/services/brig/src/Brig/Run.hs b/services/brig/src/Brig/Run.hs index 3e20d9fddf8..8a5eba0a1c3 100644 --- a/services/brig/src/Brig/Run.hs +++ b/services/brig/src/Brig/Run.hs @@ -36,13 +36,13 @@ import qualified Brig.AWS.SesNotification as SesNotification import Brig.App import qualified Brig.Calling as Calling import Brig.CanonicalInterpreter -import Brig.Data.UserPendingActivation (UserPendingActivation (..), usersPendingActivationList, usersPendingActivationRemoveMultiple) import qualified Brig.InternalEvent.Process as Internal import Brig.Options hiding (internalEvents, sesQueue) import qualified Brig.Queue as Queue +import Brig.Sem.UserPendingActivationStore (UserPendingActivation (UserPendingActivation), UserPendingActivationStore) +import qualified Brig.Sem.UserPendingActivationStore as UsersPendingActivationStore import Brig.Types.Intra (AccountStatus (PendingInvitation)) import Brig.Version -import Cassandra (Page (Page)) import qualified Control.Concurrent.Async as Async import Control.Exception.Safe (catchAny) import Control.Lens (view, (.~), (^.)) @@ -67,6 +67,7 @@ import Network.Wai.Routing.Route (App) import Network.Wai.Utilities (lookupRequestId) import Network.Wai.Utilities.Server import qualified Network.Wai.Utilities.Server as Server +import Polysemy (Members) import Servant (Context ((:.)), (:<|>) (..)) import qualified Servant import System.Logger (msg, val, (.=), (~~)) @@ -76,6 +77,7 @@ import Wire.API.Routes.API import Wire.API.Routes.Public.Brig import Wire.API.Routes.Version import Wire.API.Routes.Version.Wai +import qualified Wire.Sem.Paging as P -- FUTUREWORK: If any of these async threads die, we will have no clue about it -- and brig could start misbehaving. We should ensure that brig dies whenever a @@ -180,7 +182,7 @@ bodyParserErrorFormatter _ _ errMsg = Servant.errHeaders = [(HTTP.hContentType, HTTPMedia.renderHeader (Servant.contentType (Proxy @Servant.JSON)))] } -pendingActivationCleanup :: forall r. AppT r () +pendingActivationCleanup :: forall r p. (P.Paging p, Members '[UserPendingActivationStore p] r) => AppT r () pendingActivationCleanup = do safeForever "pendingActivationCleanup" $ do now <- liftIO =<< view currentTime @@ -200,7 +202,7 @@ pendingActivationCleanup = do if isExpired && isPendingInvitation then Just uid else Nothing ) - wrapClient . usersPendingActivationRemoveMultiple $ + liftSem . UsersPendingActivationStore.removeMultiple $ catMaybes ( uids <&> \(isExpired, _isPendingInvitation, uid) -> if isExpired then Just uid else Nothing @@ -218,13 +220,13 @@ pendingActivationCleanup = do forExpirationsPaged :: ([UserPendingActivation] -> (AppT r) ()) -> (AppT r) () forExpirationsPaged f = do - go =<< wrapClient usersPendingActivationList + go =<< liftSem (UsersPendingActivationStore.list Nothing) where - go :: Page UserPendingActivation -> (AppT r) () - go (Page hasMore result nextPage) = do - f result - when hasMore $ - go =<< wrapClient (lift nextPage) + go :: P.Page p UserPendingActivation -> (AppT r) () + go p = do + f (P.pageItems p) + when (P.pageHasMore p) $ do + go =<< liftSem (UsersPendingActivationStore.list $ Just $ P.pageState p) threadDelayRandom :: (AppT r) () threadDelayRandom = do diff --git a/services/brig/src/Brig/Sem/UserPendingActivationStore.hs b/services/brig/src/Brig/Sem/UserPendingActivationStore.hs new file mode 100644 index 00000000000..a23f1d5a878 --- /dev/null +++ b/services/brig/src/Brig/Sem/UserPendingActivationStore.hs @@ -0,0 +1,27 @@ +{-# LANGUAGE TemplateHaskell #-} + +module Brig.Sem.UserPendingActivationStore where + +import Data.Id +import Data.Time.Clock +import Imports +import Polysemy +import Wire.Sem.Paging + +data UserPendingActivation = UserPendingActivation + { upaUserId :: !UserId, + upaDay :: !UTCTime + } + deriving stock (Eq, Show, Ord) + +data UserPendingActivationStore p m a where + Add :: UserPendingActivation -> UserPendingActivationStore p m () + List :: + Maybe (PagingState p UserPendingActivation) -> + UserPendingActivationStore p m (Page p UserPendingActivation) + RemoveMultiple :: [UserId] -> UserPendingActivationStore p m () + +makeSem ''UserPendingActivationStore + +remove :: forall p r. Member (UserPendingActivationStore p) r => UserId -> Sem r () +remove uid = removeMultiple [uid] diff --git a/services/brig/src/Brig/Sem/UserPendingActivationStore/Cassandra.hs b/services/brig/src/Brig/Sem/UserPendingActivationStore/Cassandra.hs new file mode 100644 index 00000000000..9521af20d44 --- /dev/null +++ b/services/brig/src/Brig/Sem/UserPendingActivationStore/Cassandra.hs @@ -0,0 +1,49 @@ +module Brig.Sem.UserPendingActivationStore.Cassandra + ( userPendingActivationStoreToCassandra, + ) +where + +import Brig.Sem.UserPendingActivationStore +import Cassandra +import Data.Id (UserId) +import Data.Time (UTCTime) +import Imports +import Polysemy +import Polysemy.Internal.Tactics +import qualified Wire.Sem.Paging.Cassandra as PC + +userPendingActivationStoreToCassandra :: + forall r a. + (Member (Embed Client) r) => + Sem (UserPendingActivationStore PC.InternalPaging ': r) a -> + Sem r a +userPendingActivationStoreToCassandra = + interpretH $ + liftT . embed @Client . \case + Add upa -> usersPendingActivationAdd upa + List Nothing -> (flip PC.mkInternalPage pure) =<< usersPendingActivationList + List (Just ps) -> PC.ipNext ps + RemoveMultiple uids -> usersPendingActivationRemoveMultiple uids + +usersPendingActivationAdd :: MonadClient m => UserPendingActivation -> m () +usersPendingActivationAdd (UserPendingActivation uid expiresAt) = do + retry x5 . write insertExpiration . params LocalQuorum $ (uid, expiresAt) + where + insertExpiration :: PrepQuery W (UserId, UTCTime) () + insertExpiration = "INSERT INTO users_pending_activation (user, expires_at) VALUES (?, ?)" + +usersPendingActivationList :: MonadClient m => m (Page UserPendingActivation) +usersPendingActivationList = do + uncurry UserPendingActivation <$$> retry x1 (paginate selectExpired (params LocalQuorum ())) + where + selectExpired :: PrepQuery R () (UserId, UTCTime) + selectExpired = + "SELECT user, expires_at FROM users_pending_activation" + +usersPendingActivationRemoveMultiple :: MonadClient m => [UserId] -> m () +usersPendingActivationRemoveMultiple uids = + retry x5 . write deleteExpired . params LocalQuorum $ Identity uids + where + deleteExpired :: PrepQuery W (Identity [UserId]) () + deleteExpired = + "DELETE FROM users_pending_activation WHERE user IN ?" diff --git a/services/brig/src/Brig/Team/API.hs b/services/brig/src/Brig/Team/API.hs index a8fefecf984..9afa8b0a361 100644 --- a/services/brig/src/Brig/Team/API.hs +++ b/services/brig/src/Brig/Team/API.hs @@ -35,6 +35,7 @@ import qualified Brig.Email as Email import qualified Brig.IO.Intra as Intra import Brig.Options (setMaxTeamSize, setTeamInvitationTimeout) import qualified Brig.Phone as Phone +import Brig.Sem.UserPendingActivationStore (UserPendingActivationStore) import qualified Brig.Team.DB as DB import Brig.Team.Email import Brig.Team.Util (ensurePermissionToAddUser, ensurePermissions) @@ -60,7 +61,7 @@ import Network.Wai.Routing import Network.Wai.Utilities hiding (code, message) import Network.Wai.Utilities.Swagger (document) import qualified Network.Wai.Utilities.Swagger as Doc -import Polysemy (Member) +import Polysemy (Member, Members) import System.Logger (Msg) import qualified System.Logger.Class as Log import Util.Logging (logFunction, logTeam) @@ -188,7 +189,13 @@ routesPublic = do Doc.response 200 "Invitation successful." Doc.end Doc.response 403 "No permission (not admin or owner of this team)." Doc.end -routesInternal :: Member BlacklistStore r => Routes a (Handler r) () +routesInternal :: + Members + '[ BlacklistStore, + UserPendingActivationStore p + ] + r => + Routes a (Handler r) () routesInternal = do get "/i/teams/invitations/by-email" (continue getInvitationByEmailH) $ accept "application" "json" @@ -280,12 +287,26 @@ createInvitationPublic uid tid body = do context (createInvitation' tid inviteeRole (Just (inviterUid inviter)) (inviterEmail inviter) body) -createInvitationViaScimH :: Member BlacklistStore r => JSON ::: JsonRequest NewUserScimInvitation -> (Handler r) Response +createInvitationViaScimH :: + Members + '[ BlacklistStore, + UserPendingActivationStore p + ] + r => + JSON ::: JsonRequest NewUserScimInvitation -> + (Handler r) Response createInvitationViaScimH (_ ::: req) = do body <- parseJsonBody req setStatus status201 . json <$> createInvitationViaScim body -createInvitationViaScim :: Member BlacklistStore r => NewUserScimInvitation -> (Handler r) UserAccount +createInvitationViaScim :: + Members + '[ BlacklistStore, + UserPendingActivationStore p + ] + r => + NewUserScimInvitation -> + (Handler r) UserAccount createInvitationViaScim newUser@(NewUserScimInvitation tid loc name email) = do env <- ask let inviteeRole = defaultRole diff --git a/services/brig/src/Brig/Team/DB.hs b/services/brig/src/Brig/Team/DB.hs index 3e094ccff8b..0f6063c4d6d 100644 --- a/services/brig/src/Brig/Team/DB.hs +++ b/services/brig/src/Brig/Team/DB.hs @@ -174,7 +174,7 @@ deleteInvitation t i = do cqlInvitationEmail :: PrepQuery W (Email, TeamId) () cqlInvitationEmail = "DELETE FROM team_invitation_email WHERE email = ? AND team = ?" -deleteInvitations :: (MonadClient m, MonadUnliftIO m) => TeamId -> m () +deleteInvitations :: (MonadClient m) => TeamId -> m () deleteInvitations t = liftClient $ runConduit $ diff --git a/services/brig/src/Brig/User/API/Auth.hs b/services/brig/src/Brig/User/API/Auth.hs index aaa24efe998..a528297b1b2 100644 --- a/services/brig/src/Brig/User/API/Auth.hs +++ b/services/brig/src/Brig/User/API/Auth.hs @@ -396,7 +396,7 @@ tokenRequest = opt (userToken ||| legalHoldUserToken) .&. opt (accessToken ||| l tokenQuery :: r -> Result P.Error ByteString tokenQuery = query "access_token" - cookieErr :: ZAuth.UserTokenLike u => Result P.Error (List1 (ZAuth.Token u)) -> Result P.Error (List1 (ZAuth.Token u)) + cookieErr :: Result P.Error (List1 (ZAuth.Token u)) -> Result P.Error (List1 (ZAuth.Token u)) cookieErr x@Okay {} = x cookieErr (Fail x) = Fail (setMessage "Invalid user token" (P.setStatus status403 x)) diff --git a/services/brig/src/Brig/User/Auth.hs b/services/brig/src/Brig/User/Auth.hs index 89223c047ca..d85d7786706 100644 --- a/services/brig/src/Brig/User/Auth.hs +++ b/services/brig/src/Brig/User/Auth.hs @@ -137,7 +137,6 @@ login :: ( MonadReader Env m, MonadMask m, MonadHttp m, - MonadIO m, HasRequestId m, Log.MonadLogger m, MonadClient m, @@ -182,7 +181,6 @@ verifyCode :: ( MonadReader Env m, MonadMask m, MonadHttp m, - MonadIO m, HasRequestId m, Log.MonadLogger m, MonadClient m @@ -295,7 +293,6 @@ revokeAccess u pw cc ll = do catchSuspendInactiveUser :: ( MonadClient m, - Log.MonadLogger m, MonadIndexIO m, MonadReader Env m, MonadMask m, @@ -449,7 +446,6 @@ ssoLogin :: ( MonadClient m, MonadReader Env m, ZAuth.MonadZAuth m, - ZAuth.MonadZAuth m, Log.MonadLogger m, MonadIndexIO m, MonadMask m, diff --git a/services/brig/src/Brig/User/Auth/Cookie.hs b/services/brig/src/Brig/User/Auth/Cookie.hs index 9e379dc64a1..05f24ff8c73 100644 --- a/services/brig/src/Brig/User/Auth/Cookie.hs +++ b/services/brig/src/Brig/User/Auth/Cookie.hs @@ -48,7 +48,6 @@ import qualified Brig.User.Auth.DB.Cookie as DB import qualified Brig.ZAuth as ZAuth import Cassandra import Control.Lens (to, view) -import Control.Monad.Catch import Data.ByteString.Conversion import Data.Id import qualified Data.List as List @@ -70,9 +69,7 @@ import Wire.API.User.Auth newCookie :: ( ZAuth.UserTokenLike u, MonadReader Env m, - MonadIO m, ZAuth.MonadZAuth m, - MonadThrow m, MonadClient m ) => UserId -> @@ -103,8 +100,6 @@ newCookie uid typ label = do nextCookie :: ( ZAuth.UserTokenLike u, MonadReader Env m, - MonadIO m, - MonadIO m, Log.MonadLogger m, ZAuth.MonadZAuth m, MonadClient m @@ -161,7 +156,7 @@ renewCookie old = do -- 'suspendCookiesOlderThanSecs'. Call this always before 'newCookie', 'nextCookie', -- 'newCookieLimited' if there is a chance that the user should be suspended (we don't do it -- implicitly because of cyclical dependencies). -mustSuspendInactiveUser :: (MonadReader Env m, MonadIO m, MonadClient m) => UserId -> m Bool +mustSuspendInactiveUser :: (MonadReader Env m, MonadClient m) => UserId -> m Bool mustSuspendInactiveUser uid = view (settings . to setSuspendInactiveUsers) >>= \case Nothing -> pure False @@ -232,7 +227,6 @@ revokeCookies u ids labels = do newCookieLimited :: ( ZAuth.UserTokenLike t, MonadReader Env m, - MonadIO m, MonadClient m, ZAuth.MonadZAuth m ) => diff --git a/services/brig/src/Brig/User/Search/Index.hs b/services/brig/src/Brig/User/Search/Index.hs index 1374594aa87..8dfc420437e 100644 --- a/services/brig/src/Brig/User/Search/Index.hs +++ b/services/brig/src/Brig/User/Search/Index.hs @@ -171,7 +171,7 @@ withAdditionalESUrl action = do -------------------------------------------------------------------------------- -- Updates -reindex :: (MonadLogger m, MonadCatch m, MonadThrow m, MonadIndexIO m, C.MonadClient m) => UserId -> m () +reindex :: (MonadLogger m, MonadCatch m, MonadIndexIO m, C.MonadClient m) => UserId -> m () reindex u = do ixu <- lookupIndexUser u updateIndex (maybe (IndexDeleteUser u) (IndexUpdateUser IndexUpdateIfNewerVersion) ixu) @@ -682,7 +682,7 @@ mappingName :: ES.MappingName mappingName = ES.MappingName "user" lookupIndexUser :: - (MonadCatch m, MonadThrow m, MonadLogger m, MonadIndexIO m, C.MonadClient m) => + (MonadCatch m, MonadIndexIO m, C.MonadClient m) => UserId -> m (Maybe IndexUser) lookupIndexUser = lookupForIndex @@ -911,7 +911,7 @@ getTeamSearchVisibilityInboundMulti tids = do serviceRequest' :: forall m. - (MonadIO m, MonadMask m, MonadCatch m, MonadHttp m) => + (MonadIO m, MonadMask m, MonadHttp m) => LT.Text -> Endpoint -> StdMethod -> diff --git a/services/brig/test/integration/API/Internal.hs b/services/brig/test/integration/API/Internal.hs index b27269772bd..93d0eebb759 100644 --- a/services/brig/test/integration/API/Internal.hs +++ b/services/brig/test/integration/API/Internal.hs @@ -57,6 +57,7 @@ import Wire.API.Team.Feature import qualified Wire.API.Team.Feature as ApiFt import qualified Wire.API.Team.Member as Team import Wire.API.User +import Wire.API.User.Client tests :: Opt.Opts -> Manager -> Cass.ClientState -> Brig -> Endpoint -> Gundeck -> Galley -> IO TestTree tests opts mgr db brig brigep gundeck galley = do @@ -68,13 +69,15 @@ tests opts mgr db brig brigep gundeck galley = do test mgr "suspend and unsuspend user" $ testSuspendUser db brig, test mgr "suspend non existing user and verify no db entry" $ testSuspendNonExistingUser db brig, - testGroup "mls/key-packages" $ - [ test mgr "fresh get" $ testKpcFreshGet brig, - test mgr "put,get" $ testKpcPutGet brig, - test mgr "get,get" $ testKpcGetGet brig, - test mgr "put,put" $ testKpcPutPut brig, - test mgr "add key package ref" $ testAddKeyPackageRef brig - ] + test mgr "mls/clients" $ testGetMlsClients brig, + testGroup + "mls/key-packages" + $ [ test mgr "fresh get" $ testKpcFreshGet brig, + test mgr "put,get" $ testKpcPutGet brig, + test mgr "get,get" $ testKpcGetGet brig, + test mgr "put,put" $ testKpcPutPut brig, + test mgr "add key package ref" $ testAddKeyPackageRef brig + ] ] testSuspendUser :: forall m. TestConstraints m => Cass.ClientState -> Brig -> m () @@ -222,6 +225,31 @@ testFeatureConferenceCallingByAccount (Opt.optSettings -> settings) mgr db brig check $ ApiFt.WithStatusNoLock ApiFt.FeatureStatusDisabled ApiFt.ConferenceCallingConfig ApiFt.FeatureTTLUnlimited check' +testGetMlsClients :: Brig -> Http () +testGetMlsClients brig = do + qusr <- userQualifiedId <$> randomUser brig + c <- createClient brig qusr 0 + (cs0 :: Set ClientInfo) <- + responseJsonError + =<< get + ( brig + . paths ["i", "mls", "clients", toByteString' (qUnqualified qusr)] + . queryItem "sig_scheme" "ed25519" + ) + liftIO $ toList cs0 @?= [ClientInfo c False] + + withSystemTempDirectory "mls" $ \tmp -> + uploadKeyPackages brig tmp def qusr c 2 + + (cs1 :: Set ClientInfo) <- + responseJsonError + =<< get + ( brig + . paths ["i", "mls", "clients", toByteString' (qUnqualified qusr)] + . queryItem "sig_scheme" "ed25519" + ) + liftIO $ toList cs1 @?= [ClientInfo c True] + keyPackageCreate :: HasCallStack => Brig -> Http KeyPackageRef keyPackageCreate brig = do uid <- userQualifiedId <$> randomUser brig diff --git a/services/brig/test/integration/API/User/Client.hs b/services/brig/test/integration/API/User/Client.hs index 0c8662246c2..239de959d72 100644 --- a/services/brig/test/integration/API/User/Client.hs +++ b/services/brig/test/integration/API/User/Client.hs @@ -38,6 +38,7 @@ import Data.Default import Data.Id hiding (client) import qualified Data.List1 as List1 import qualified Data.Map as Map +import Data.Nonce (isValidBase64UrlEncodedUUID) import Data.Qualified (Qualified (..)) import Data.Range (unsafeRange) import qualified Data.Set as Set @@ -105,7 +106,8 @@ tests _cl _at opts p db b c g = test p "get /clients/:client - 404" $ testMissingClient b, test p "get /clients/:client - 200" $ testMLSClient b, test p "post /clients - 200 multiple temporary" $ testAddMultipleTemporary b g, - test p "client/prekeys/race" $ testPreKeyRace b + test p "client/prekeys/race" $ testPreKeyRace b, + test p "get/head nonce/clients" $ testNewNonce b ] testAddGetClientVerificationCode :: DB.ClientState -> Brig -> Galley -> Http () @@ -946,6 +948,22 @@ testPreKeyRace brig = do liftIO $ assertEqual "duplicate prekeys" (length regular) (length (nub regular)) deleteClient brig uid (clientId c) (Just defPasswordText) !!! const 200 === statusCode +testNewNonce :: Brig -> Http () +testNewNonce brig = do + n1 <- check Util.getNonce 204 + n2 <- check Util.headNonce 200 + lift $ assertBool "nonces are should not be equal" (n1 /= n2) + where + check f status = do + uid <- userId <$> randomUser brig + cid <- randomClient + response <- f brig uid cid Http () testCan'tDeleteLegalHoldClient brig = do let hasPassword = False diff --git a/services/brig/test/integration/API/User/Util.hs b/services/brig/test/integration/API/User/Util.hs index 9b3741dbbc8..8c23cb39a0e 100644 --- a/services/brig/test/integration/API/User/Util.hs +++ b/services/brig/test/integration/API/User/Util.hs @@ -551,3 +551,27 @@ setTeamFeatureLockStatus galley tid status = lookupCode :: MonadIO m => DB.ClientState -> Code.Key -> Code.Scope -> m (Maybe Code.Code) lookupCode db k = liftIO . DB.runClient db . Code.lookup k + +getNonce :: + (MonadIO m, MonadHttp m) => + Brig -> + UserId -> + ClientId -> + m ResponseLBS +getNonce = nonce get + +headNonce :: + (MonadIO m, MonadHttp m) => + Brig -> + UserId -> + ClientId -> + m ResponseLBS +headNonce = nonce Bilge.head + +nonce :: ((Request -> c) -> t) -> (Request -> c) -> UserId -> ClientId -> t +nonce m brig uid cid = + m + ( brig + . paths ["clients", toByteString' cid, "nonce"] + . zUser uid + ) diff --git a/services/brig/test/integration/Main.hs b/services/brig/test/integration/Main.hs index 7321ee93d3c..3e51716247c 100644 --- a/services/brig/test/integration/Main.hs +++ b/services/brig/test/integration/Main.hs @@ -61,6 +61,7 @@ import Util import Util.Options import Util.Test import Wire.API.Federation.API +import Wire.Sem.Paging.Cassandra (InternalPaging) data BackendConf = BackendConf { remoteBrig :: Endpoint, @@ -157,7 +158,7 @@ runTests iConf brigOpts otherArgs = do assertEqual "inconcistent sitemap" mempty - (pathsConsistencyCheck . treeToPaths . compile $ Brig.API.sitemap @BrigCanonicalEffects), + (pathsConsistencyCheck . treeToPaths . compile $ Brig.API.sitemap @BrigCanonicalEffects @InternalPaging), userApi, providerApi, searchApis, diff --git a/services/federator/src/Federator/Monitor/Internal.hs b/services/federator/src/Federator/Monitor/Internal.hs index 219acaa6c8e..a7fa5c05afa 100644 --- a/services/federator/src/Federator/Monitor/Internal.hs +++ b/services/federator/src/Federator/Monitor/Internal.hs @@ -45,7 +45,7 @@ import qualified System.Logger.Message as Log import System.Posix.ByteString (RawFilePath) import System.Posix.Files import System.X509 -import Wire.API.Arbitrary +import Wire.Arbitrary import qualified Wire.Sem.Logger.TinyLog as Log data Monitor = Monitor diff --git a/services/galley/galley.cabal b/services/galley/galley.cabal index 948dd33b483..f10581474ee 100644 --- a/services/galley/galley.cabal +++ b/services/galley/galley.cabal @@ -32,6 +32,7 @@ library Galley.API.Message Galley.API.MLS Galley.API.MLS.KeyPackage + Galley.API.MLS.Keys Galley.API.MLS.Message Galley.API.MLS.Welcome Galley.API.One2One @@ -57,10 +58,8 @@ library Galley.Cassandra.CustomBackend Galley.Cassandra.Instances Galley.Cassandra.LegalHold - Galley.Cassandra.Paging Galley.Cassandra.Proposal Galley.Cassandra.Queries - Galley.Cassandra.ResultSet Galley.Cassandra.SearchVisibility Galley.Cassandra.Services Galley.Cassandra.Store @@ -87,7 +86,6 @@ library Galley.Effects.LegalHoldStore Galley.Effects.ListItems Galley.Effects.MemberStore - Galley.Effects.Paging Galley.Effects.ProposalStore Galley.Effects.Queue Galley.Effects.RemoteConversationListStore @@ -115,6 +113,7 @@ library Galley.Intra.Team Galley.Intra.User Galley.Intra.Util + Galley.Keys Galley.Monad Galley.Options Galley.Queue @@ -175,6 +174,8 @@ library aeson >=2.0.1.0 , amazonka >=1.4.5 , amazonka-sqs >=1.4.5 + , asn1-encoding + , asn1-types , async >=2.0 , base >=4.6 && <5 , base64-bytestring >=1.0 @@ -270,6 +271,7 @@ library , warp >=3.0 , wire-api , wire-api-federation + , x509 default-language: Haskell2010 @@ -425,6 +427,7 @@ executable galley-integration , base , base64-bytestring , bilge + , binary , brig-types , bytestring , bytestring-conversion @@ -435,6 +438,7 @@ executable galley-integration , comonad , containers , cookie + , cryptonite , currency-codes , data-default , data-timeout @@ -459,6 +463,7 @@ executable galley-integration , kan-extensions , lens , lens-aeson + , memory , metrics-wai , mtl , optparse-applicative diff --git a/services/galley/galley.integration.yaml b/services/galley/galley.integration.yaml index c5068b63784..be7d1fe7d62 100644 --- a/services/galley/galley.integration.yaml +++ b/services/galley/galley.integration.yaml @@ -42,6 +42,9 @@ settings: # Once set, DO NOT change it: if you do, existing users may have a broken experience and/or stop working # Remember to keep it the same in Brig federationDomain: example.com + mlsPrivateKeyPaths: + removal: + ed25519: test/resources/ed25519.pem featureFlags: # see #RefConfigOptions in `/docs/reference` sso: disabled-by-default diff --git a/services/galley/src/Galley/API/Federation.hs b/services/galley/src/Galley/API/Federation.hs index 2446ad1a917..4a93480e8cf 100644 --- a/services/galley/src/Galley/API/Federation.hs +++ b/services/galley/src/Galley/API/Federation.hs @@ -81,6 +81,7 @@ import Wire.API.Federation.API.Galley (ConversationUpdateResponse) import qualified Wire.API.Federation.API.Galley as F import Wire.API.Federation.Error import Wire.API.MLS.Credential +import Wire.API.MLS.Message import Wire.API.MLS.Serialisation import Wire.API.MLS.Welcome import Wire.API.Message @@ -581,9 +582,13 @@ sendMLSMessage remoteDomain msr = loc <- qualifyLocal () let sender = toRemoteUnsafe remoteDomain (F.msrSender msr) raw <- either (throw . mlsProtocolError) pure $ decodeMLS' (fromBase64ByteString (F.msrRawMessage msr)) - mapToGalleyError @MLSMessageStaticErrors $ - F.MLSMessageResponseUpdates . map lcuUpdate - <$> postMLSMessage loc (qUntagged sender) Nothing raw + mapToGalleyError @MLSMessageStaticErrors $ do + case rmValue raw of + SomeMessage _ msg -> do + qcnv <- E.getConversationIdByGroupId (msgGroupId msg) >>= noteS @'ConvNotFound + when (qUnqualified qcnv /= F.msrConvId msr) $ throwS @'MLSGroupConversationMismatch + F.MLSMessageResponseUpdates . map lcuUpdate + <$> postMLSMessage loc (qUntagged sender) qcnv Nothing raw class ToGalleyRuntimeError (effs :: EffectRow) r where mapToGalleyError :: diff --git a/services/galley/src/Galley/API/Internal.hs b/services/galley/src/Galley/API/Internal.hs index 3257f1f9d1b..04497a56828 100644 --- a/services/galley/src/Galley/API/Internal.hs +++ b/services/galley/src/Galley/API/Internal.hs @@ -49,7 +49,6 @@ import Galley.API.Teams.Features import qualified Galley.API.Update as Update import Galley.API.Util import Galley.App -import Galley.Cassandra.Paging import Galley.Cassandra.TeamFeatures import qualified Galley.Data.Conversation as Data import Galley.Effects @@ -59,7 +58,6 @@ import Galley.Effects.FederatorAccess import Galley.Effects.GundeckAccess import Galley.Effects.LegalHoldStore as LegalHoldStore import Galley.Effects.MemberStore -import Galley.Effects.Paging import Galley.Effects.TeamStore import qualified Galley.Intra.Push as Intra import Galley.Monad @@ -107,6 +105,8 @@ import Wire.API.Team import Wire.API.Team.Feature import Wire.API.Team.Member import Wire.API.Team.SearchVisibility +import Wire.Sem.Paging +import Wire.Sem.Paging.Cassandra type LegalHoldFeatureStatusChangeErrors = '( 'ActionDenied 'RemoveConversationMember, diff --git a/services/galley/src/Galley/API/LegalHold.hs b/services/galley/src/Galley/API/LegalHold.hs index 88993ec0d45..302ef62ad57 100644 --- a/services/galley/src/Galley/API/LegalHold.hs +++ b/services/galley/src/Galley/API/LegalHold.hs @@ -48,13 +48,11 @@ import Galley.API.Error import Galley.API.Query (iterateConversations) import Galley.API.Update (removeMemberFromLocalConv) import Galley.API.Util -import Galley.Cassandra.Paging import qualified Galley.Data.Conversation as Data import Galley.Effects import Galley.Effects.BrigAccess import Galley.Effects.FireAndForget import qualified Galley.Effects.LegalHoldStore as LegalHoldData -import Galley.Effects.Paging import qualified Galley.Effects.TeamFeatureStore as TeamFeatures import Galley.Effects.TeamMemberStore import Galley.Effects.TeamStore @@ -81,6 +79,8 @@ import qualified Wire.API.Team.LegalHold as Public import Wire.API.Team.LegalHold.External hiding (userId) import Wire.API.Team.Member import Wire.API.User.Client.Prekey +import Wire.Sem.Paging +import Wire.Sem.Paging.Cassandra assertLegalHoldEnabledForTeam :: forall db r. diff --git a/services/galley/src/Galley/API/MLS.hs b/services/galley/src/Galley/API/MLS.hs index 1e89f9bd590..fc85496141b 100644 --- a/services/galley/src/Galley/API/MLS.hs +++ b/services/galley/src/Galley/API/MLS.hs @@ -20,8 +20,10 @@ module Galley.API.MLS postMLSMessage, postMLSMessageFromLocalUser, postMLSMessageFromLocalUserV1, + getMLSPublicKeys, ) where +import Galley.API.MLS.Keys import Galley.API.MLS.Message import Galley.API.MLS.Welcome diff --git a/services/galley/src/Galley/API/MLS/Keys.hs b/services/galley/src/Galley/API/MLS/Keys.hs new file mode 100644 index 00000000000..43890eb9ed3 --- /dev/null +++ b/services/galley/src/Galley/API/MLS/Keys.hs @@ -0,0 +1,35 @@ +-- This file is part of the Wire Server implementation. +-- +-- Copyright (C) 2022 Wire Swiss GmbH +-- +-- This program is free software: you can redistribute it and/or modify it under +-- the terms of the GNU Affero General Public License as published by the Free +-- Software Foundation, either version 3 of the License, or (at your option) any +-- later version. +-- +-- This program is distributed in the hope that it will be useful, but WITHOUT +-- ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +-- FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more +-- details. +-- +-- You should have received a copy of the GNU Affero General Public License along +-- with this program. If not, see . + +module Galley.API.MLS.Keys where + +import Control.Lens (view) +import Data.Id +import Data.Qualified +import Galley.Env +import Imports +import Polysemy +import Polysemy.Input +import Wire.API.MLS.Keys + +getMLSPublicKeys :: + Member (Input Env) r => + Local UserId -> + Sem r MLSPublicKeys +getMLSPublicKeys _ = do + keys <- inputs (view mlsKeys) + pure $ mlsKeysToPublic keys diff --git a/services/galley/src/Galley/API/MLS/Message.hs b/services/galley/src/Galley/API/MLS/Message.hs index bb1182fdaac..8e0ea8ef87f 100644 --- a/services/galley/src/Galley/API/MLS/Message.hs +++ b/services/galley/src/Galley/API/MLS/Message.hs @@ -81,9 +81,11 @@ import Wire.API.MLS.Proposal import qualified Wire.API.MLS.Proposal as Proposal import Wire.API.MLS.Serialisation import Wire.API.Message +import Wire.API.User.Client type MLSMessageStaticErrors = '[ ErrorS 'ConvAccessDenied, + ErrorS 'ConvMemberNotFound, ErrorS 'ConvNotFound, ErrorS 'MLSUnsupportedMessage, ErrorS 'MLSStaleMessage, @@ -93,7 +95,9 @@ type MLSMessageStaticErrors = ErrorS 'MLSClientMismatch, ErrorS 'MLSUnsupportedProposal, ErrorS 'MLSCommitMissingReferences, - ErrorS 'MLSSelfRemovalNotAllowed + ErrorS 'MLSSelfRemovalNotAllowed, + ErrorS 'MLSClientSenderUserMismatch, + ErrorS 'MLSGroupConversationMismatch ] postMLSMessageFromLocalUserV1 :: @@ -109,6 +113,8 @@ postMLSMessageFromLocalUserV1 :: ErrorS 'MLSSelfRemovalNotAllowed, ErrorS 'MLSStaleMessage, ErrorS 'MLSUnsupportedMessage, + ErrorS 'MLSClientSenderUserMismatch, + ErrorS 'MLSGroupConversationMismatch, ErrorS 'MissingLegalholdConsent, Input (Local ()), ProposalStore, @@ -121,9 +127,11 @@ postMLSMessageFromLocalUserV1 :: ConnId -> RawMLS SomeMessage -> Sem r [Event] -postMLSMessageFromLocalUserV1 lusr conn msg = - map lcuEvent - <$> postMLSMessage lusr (qUntagged lusr) (Just conn) msg +postMLSMessageFromLocalUserV1 lusr conn smsg = case rmValue smsg of + SomeMessage _ msg -> do + qcnv <- getConversationIdByGroupId (msgGroupId msg) >>= noteS @'ConvNotFound + map lcuEvent + <$> postMLSMessage lusr (qUntagged lusr) qcnv (Just conn) smsg postMLSMessageFromLocalUser :: ( HasProposalEffects r, @@ -138,6 +146,8 @@ postMLSMessageFromLocalUser :: ErrorS 'MLSSelfRemovalNotAllowed, ErrorS 'MLSStaleMessage, ErrorS 'MLSUnsupportedMessage, + ErrorS 'MLSClientSenderUserMismatch, + ErrorS 'MLSGroupConversationMismatch, ErrorS 'MissingLegalholdConsent, Input (Local ()), ProposalStore, @@ -164,12 +174,15 @@ postMLSMessage :: Error InternalError, ErrorS 'ConvAccessDenied, ErrorS 'ConvNotFound, + ErrorS 'ConvMemberNotFound, ErrorS 'MLSUnsupportedMessage, ErrorS 'MLSStaleMessage, ErrorS 'MLSProposalNotFound, ErrorS 'MissingLegalholdConsent, ErrorS 'MLSCommitMissingReferences, ErrorS 'MLSSelfRemovalNotAllowed, + ErrorS 'MLSClientSenderUserMismatch, + ErrorS 'MLSGroupConversationMismatch, Resource, TinyLog, ProposalStore, @@ -179,19 +192,55 @@ postMLSMessage :: ) => Local x -> Qualified UserId -> + Qualified ConvId -> Maybe ConnId -> RawMLS SomeMessage -> Sem r [LocalConversationUpdate] -postMLSMessage loc qusr con smsg = case rmValue smsg of +postMLSMessage loc qusr qcnv con smsg = case rmValue smsg of SomeMessage _ msg -> do - -- fetch conversation ID - qcnv <- getConversationIdByGroupId (msgGroupId msg) >>= noteS @'ConvNotFound + unless (msgEpoch msg == Epoch 0) $ + flip unless (throwS @'MLSClientSenderUserMismatch) =<< isUserSender qusr smsg foldQualified loc (postMLSMessageToLocalConv qusr con smsg) (postMLSMessageToRemoteConv loc qusr con smsg) qcnv +-- | Check that the MLS client who created the message belongs to the user who +-- is the sender of the REST request, identified by HTTP header. +-- +-- This is only relevant in an ongoing conversation. The check should be skipped +-- in case of +-- * encrypted messages in which we don't have access to the sending client's +-- key package, +-- * messages sent by the backend, and +-- * external add proposals which propose fresh key packages for new clients and +-- thus the validity of the key package cannot be known at the time of this +-- check. +-- For these cases the function will return True. +isUserSender :: + ( Members + '[ ErrorS 'MLSKeyPackageRefNotFound, + BrigAccess + ] + r + ) => + Qualified UserId -> + RawMLS SomeMessage -> + Sem r Bool +isUserSender qusr smsg = case rmValue smsg of + SomeMessage tag msg -> case tag of + -- skip encrypted message + SMLSCipherText -> pure True + SMLSPlainText -> case msgSender msg of + -- skip message sent by backend + PreconfiguredSender _ -> pure True + -- skip external add proposal + NewMemberSender -> pure True + MemberSender ref -> do + ci <- derefKeyPackage ref + pure $ fmap fst (cidQualifiedClient ci) == qusr + postMLSMessageToLocalConv :: ( HasProposalEffects r, Members @@ -258,6 +307,9 @@ postMLSMessageToRemoteConv :: postMLSMessageToRemoteConv loc qusr con smsg rcnv = do -- only local users can send messages to remote conversations lusr <- foldQualified loc pure (\_ -> throwS @'ConvAccessDenied) qusr + -- only members may send messages to the remote conversation + flip unless (throwS @'ConvMemberNotFound) =<< checkLocalMemberRemoteConv (tUnqualified lusr) rcnv + resp <- runFederated rcnv $ fedClient @'Galley @"send-mls-message" $ @@ -365,7 +417,7 @@ processCommit qusr con lconv epoch sender commit = do senderRef <- maybe (pure currentRef) - ( (& note (mlsProtocolError "Could not compute key package ref")) + ( note (mlsProtocolError "Could not compute key package ref") . kpRef' . upLeaf ) @@ -437,7 +489,7 @@ applyProposalRef conv _groupId _epoch (Inline p) = do suite <- preview (to convProtocol . _ProtocolMLS . to cnvmlsCipherSuite) conv & noteS @'ConvNotFound - checkProposal suite p + checkProposalCipherSuite suite p applyProposal p applyProposal :: HasProposalEffects r => Proposal -> Sem r ProposalAction @@ -450,9 +502,9 @@ applyProposal (AddProposal kp) = do applyProposal (RemoveProposal ref) = do qclient <- cidQualifiedClient <$> derefKeyPackage ref pure (paRemoveClient qclient) -applyProposal _ = throwS @'MLSUnsupportedProposal +applyProposal _ = pure mempty -checkProposal :: +checkProposalCipherSuite :: Members '[ Error MLSProtocolError, ProposalStore @@ -461,7 +513,7 @@ checkProposal :: CipherSuiteTag -> Proposal -> Sem r () -checkProposal suite (AddProposal kpRaw) = do +checkProposalCipherSuite suite (AddProposal kpRaw) = do let kp = rmValue kpRaw unless (kpCipherSuite kp == tagCipherSuite suite) . throw @@ -472,7 +524,7 @@ checkProposal suite (AddProposal kpRaw) = do <> " and the cipher suite of the proposal's key package " <> show (cipherSuiteNumber (kpCipherSuite kp)) <> " do not match." -checkProposal _suite _prop = pure () +checkProposalCipherSuite _suite _prop = pure () processProposal :: HasProposalEffects r => @@ -500,14 +552,84 @@ processProposal qusr conv msg prop = do -- -- is the user a member of the conversation? loc <- qualifyLocal () - isMember' <- foldQualified loc (fmap isJust . getLocalMember (convId conv) . tUnqualified) (fmap isJust . getRemoteMember (convId conv)) qusr + isMember' <- + foldQualified + loc + ( fmap isJust + . getLocalMember (convId conv) + . tUnqualified + ) + ( fmap isJust + . getRemoteMember (convId conv) + ) + qusr unless isMember' $ throwS @'ConvNotFound -- FUTUREWORK: validate the member's conversation role + let propValue = rmValue prop + checkProposalCipherSuite suiteTag propValue + when (isExternalProposal msg) $ do + checkExternalProposalSignature suiteTag msg prop + checkExternalProposalUser qusr propValue let propRef = proposalRef suiteTag prop - checkProposal suiteTag (rmValue prop) storeProposal (msgGroupId msg) (msgEpoch msg) propRef prop +checkExternalProposalSignature :: + Members + '[ ErrorS 'MLSUnsupportedProposal + ] + r => + CipherSuiteTag -> + Message 'MLSPlainText -> + RawMLS Proposal -> + Sem r () +checkExternalProposalSignature csTag msg prop = case rmValue prop of + AddProposal kp -> do + let pubKey = bcSignatureKey . kpCredential $ rmValue kp + unless (verifyMessageSignature csTag msg pubKey) $ throwS @'MLSUnsupportedProposal + _ -> pure () -- FUTUREWORK: check signature of other proposals as well + +isExternalProposal :: Message 'MLSPlainText -> Bool +isExternalProposal msg = case msgSender msg of + NewMemberSender -> True + PreconfiguredSender _ -> True + _ -> False + +-- check owner/subject of the key package exists and belongs to the user +checkExternalProposalUser :: + Members + '[ BrigAccess, + ErrorS 'MLSUnsupportedProposal, + Input (Local ()) + ] + r => + Qualified UserId -> + Proposal -> + Sem r () +checkExternalProposalUser qusr prop = do + loc <- qualifyLocal () + foldQualified + loc + ( \lusr -> case prop of + AddProposal keyPackage -> do + ClientIdentity {ciUser, ciClient} <- + either + (const $ throwS @'MLSUnsupportedProposal) + pure + $ decodeMLS' @ClientIdentity (bcIdentity . kpCredential . rmValue $ keyPackage) + -- requesting user must match key package owner + when (tUnqualified lusr /= ciUser) $ throwS @'MLSUnsupportedProposal + -- client referenced in key package must be one of the user's clients + UserClients {userClients} <- lookupClients [ciUser] + maybe + (throwS @'MLSUnsupportedProposal) + (flip when (throwS @'MLSUnsupportedProposal) . Set.null . Set.filter (== ciClient)) + $ userClients Map.!? ciUser + _ -> throwS @'MLSUnsupportedProposal + ) + (const $ pure ()) -- FUTUREWORK: check external proposals from remote backends + qusr + executeProposalAction :: forall r. ( Member BrigAccess r, @@ -544,16 +666,35 @@ executeProposalAction qusr con lconv action = do -- FUTUREWORK: remove this check after remote admins are implemented in federation https://wearezeta.atlassian.net/browse/FS-216 foldQualified lconv (\_ -> pure ()) (\_ -> throwS @'MLSUnsupportedProposal) qusr - -- check that all clients of each user are added to the conversation - for_ newUserClients $ \(qtarget, newclients) -> do - -- final set of clients in the conversation - let clients = newclients <> Map.findWithDefault mempty qtarget cm - -- get list of mls clients from brig - allClients <- getMLSClients lconv qtarget ss - -- if not all clients have been added to the conversation, return an error - when (clients /= allClients) $ do - -- FUTUREWORK: turn this error into a proper response - throwS @'MLSClientMismatch + -- for each user, we compare their clients with the ones being added to the conversation + for_ newUserClients $ \(qtarget, newclients) -> case Map.lookup qtarget cm of + -- user is already present, skip check in this case + Just _ -> pure () + -- new user + Nothing -> do + -- final set of clients in the conversation + let clients = newclients <> Map.findWithDefault mempty qtarget cm + -- get list of mls clients from brig + clientInfo <- getMLSClients lconv qtarget ss + let allClients = Set.map ciId clientInfo + let allMLSClients = Set.map ciId (Set.filter ciMLS clientInfo) + -- We check the following condition: + -- allMLSClients ⊆ clients ⊆ allClients + -- i.e. + -- - if a client has at least 1 key package, it has to be added + -- - if a client is being added, it has to still exist + -- + -- The reason why we can't simply check that clients == allMLSClients is + -- that a client with no remaining key packages might be added by a user + -- who just fetched its last key package. + unless + ( Set.isSubsetOf allMLSClients clients + && Set.isSubsetOf clients allClients + ) + $ do + -- unless (Set.isSubsetOf allClients clients) $ do + -- FUTUREWORK: turn this error into a proper response + throwS @'MLSClientMismatch membersToRemove <- catMaybes <$> for removeUserClients (uncurry (checkRemoval lconv ss)) @@ -572,7 +713,7 @@ executeProposalAction qusr con lconv action = do -- For these clients there is nothing left to do checkRemoval :: Local x -> SignatureSchemeTag -> Qualified UserId -> Set ClientId -> Sem r (Maybe (Qualified UserId)) checkRemoval loc ss qtarget clients = do - allClients <- getMLSClients loc qtarget ss + allClients <- Set.map ciId <$> getMLSClients loc qtarget ss let allClientsDontExist = Set.null (clients `Set.intersection` allClients) if allClientsDontExist then pure Nothing @@ -600,7 +741,7 @@ executeProposalAction qusr con lconv action = do $ ConversationJoin users roleNameWireMember removeMembers :: NonEmpty (Qualified UserId) -> Sem r [LocalConversationUpdate] - removeMembers users = + removeMembers = handleNoChanges . handleMLSProposalFailures @ProposalErrors . fmap pure @@ -609,7 +750,6 @@ executeProposalAction qusr con lconv action = do lconv qusr con - $ users handleNoChanges :: Monoid a => Sem (Error NoChanges ': r) a -> Sem r a handleNoChanges = fmap fold . runError @@ -658,9 +798,8 @@ propagateMessage loc qusr conv con raw = do foldMap (uncurry mkPush) (cToList =<< lclients) -- send to remotes - (traverse_ handleError =<<) - . runFederatedConcurrentlyEither (map remoteMemberQualify (Data.convRemoteMembers conv)) - $ \(tUnqualified -> rs) -> + traverse_ handleError <=< runFederatedConcurrentlyEither (map remoteMemberQualify (Data.convRemoteMembers conv)) $ + \(tUnqualified -> rs) -> fedClient @'Galley @"on-mls-message-sent" $ RemoteMLSMessage { rmmTime = now, @@ -696,10 +835,14 @@ getMLSClients :: Local x -> Qualified UserId -> SignatureSchemeTag -> - Sem r (Set ClientId) + Sem r (Set ClientInfo) getMLSClients loc = foldQualified loc getLocalMLSClients getRemoteMLSClients -getRemoteMLSClients :: Member FederatorAccess r => Remote UserId -> SignatureSchemeTag -> Sem r (Set ClientId) +getRemoteMLSClients :: + Member FederatorAccess r => + Remote UserId -> + SignatureSchemeTag -> + Sem r (Set ClientInfo) getRemoteMLSClients rusr ss = do runFederated rusr $ fedClient @'Brig @"get-mls-clients" $ diff --git a/services/galley/src/Galley/API/Public/Servant.hs b/services/galley/src/Galley/API/Public/Servant.hs index 9ee3de4c20c..21378d52784 100644 --- a/services/galley/src/Galley/API/Public/Servant.hs +++ b/services/galley/src/Galley/API/Public/Servant.hs @@ -161,6 +161,7 @@ servantSitemap = mkNamedAPI @"mls-welcome-message" postMLSWelcome <@> mkNamedAPI @"mls-message-v1" postMLSMessageFromLocalUserV1 <@> mkNamedAPI @"mls-message" postMLSMessageFromLocalUser + <@> mkNamedAPI @"mls-public-keys" getMLSPublicKeys customBackend :: API CustomBackendAPI GalleyEffects customBackend = mkNamedAPI @"get-custom-backend-by-domain" getCustomBackendByDomain diff --git a/services/galley/src/Galley/API/Query.hs b/services/galley/src/Galley/API/Query.hs index 1e55dbe5d73..a84e13cab01 100644 --- a/services/galley/src/Galley/API/Query.hs +++ b/services/galley/src/Galley/API/Query.hs @@ -68,7 +68,6 @@ import qualified Data.Set as Set import Galley.API.Error import qualified Galley.API.Mapping as Mapping import Galley.API.Util -import Galley.Cassandra.Paging import qualified Galley.Data.Conversation as Data import Galley.Data.Types (Code (codeConversation)) import Galley.Effects @@ -105,6 +104,7 @@ import Wire.API.Federation.Error import qualified Wire.API.Provider.Bot as Public import qualified Wire.API.Routes.MultiTablePaging as Public import Wire.API.Team.Feature as Public hiding (setStatus) +import Wire.Sem.Paging.Cassandra getBotConversationH :: Members '[ConversationStore, ErrorS 'ConvNotFound, Input (Local ())] r => diff --git a/services/galley/src/Galley/API/Teams.hs b/services/galley/src/Galley/API/Teams.hs index d8ad841f396..ca2643e5618 100644 --- a/services/galley/src/Galley/API/Teams.hs +++ b/services/galley/src/Galley/API/Teams.hs @@ -87,7 +87,6 @@ import qualified Galley.API.Teams.Notifications as APITeamQueue import qualified Galley.API.Update as API import Galley.API.Util import Galley.App -import Galley.Cassandra.Paging import qualified Galley.Data.Conversation as Data import Galley.Data.Services (BotMember) import Galley.Effects @@ -98,7 +97,6 @@ import qualified Galley.Effects.GundeckAccess as E import qualified Galley.Effects.LegalHoldStore as Data import qualified Galley.Effects.ListItems as E import qualified Galley.Effects.MemberStore as E -import qualified Galley.Effects.Paging as E import qualified Galley.Effects.Queue as E import qualified Galley.Effects.SearchVisibilityStore as SearchVisibilityData import qualified Galley.Effects.SparAccess as Spar @@ -150,6 +148,8 @@ import Wire.API.User (ScimUserInfo (..), User, UserIdList, UserSSOId (UserScimEx import qualified Wire.API.User as U import Wire.API.User.Identity (UserSSOId (UserSSOId)) import Wire.API.User.RichInfo (RichInfo) +import qualified Wire.Sem.Paging as E +import Wire.Sem.Paging.Cassandra getTeamH :: forall r. diff --git a/services/galley/src/Galley/API/Teams/Features.hs b/services/galley/src/Galley/API/Teams/Features.hs index b3dc7ed0b33..260781cec44 100644 --- a/services/galley/src/Galley/API/Teams/Features.hs +++ b/services/galley/src/Galley/API/Teams/Features.hs @@ -52,12 +52,10 @@ import Galley.API.LegalHold (isLegalHoldEnabledForTeam) import qualified Galley.API.LegalHold as LegalHold import Galley.API.Teams (ensureNotTooLargeToActivateLegalHold) import Galley.API.Util (assertTeamExists, getTeamMembersForFanout, membersToRecipients, permissionCheck) -import Galley.Cassandra.Paging import Galley.Effects import Galley.Effects.BrigAccess (getAccountConferenceCallingConfigClient, updateSearchVisibilityInbound) import Galley.Effects.ConversationStore as ConversationStore import Galley.Effects.GundeckAccess -import Galley.Effects.Paging import qualified Galley.Effects.SearchVisibilityStore as SearchVisibilityData import Galley.Effects.TeamFeatureStore import qualified Galley.Effects.TeamFeatureStore as TeamFeatures @@ -79,6 +77,8 @@ import qualified Wire.API.Event.FeatureConfig as Event import qualified Wire.API.Routes.Internal.Galley.TeamFeatureNoConfigMulti as Multi import Wire.API.Team.Feature import Wire.API.Team.Member +import Wire.Sem.Paging +import Wire.Sem.Paging.Cassandra data DoAuth = DoAuth UserId | DontDoAuth @@ -427,6 +427,7 @@ getAllFeatureConfigsForServer = <$> getConfigForServer @db @LegalholdConfig <*> getConfigForServer @db @SSOConfig <*> getConfigForServer @db @SearchVisibilityAvailableConfig + <*> getConfigForServer @db @SearchVisibilityInboundConfig <*> getConfigForServer @db @ValidateSAMLEmailsConfig <*> getConfigForServer @db @DigitalSignaturesConfig <*> getConfigForServer @db @AppLockConfig @@ -437,7 +438,6 @@ getAllFeatureConfigsForServer = <*> getConfigForServer @db @GuestLinksConfig <*> getConfigForServer @db @SndFactorPasswordChallengeConfig <*> getConfigForServer @db @MLSConfig - <*> getConfigForServer @db @SearchVisibilityInboundConfig getAllFeatureConfigsUser :: forall db r. @@ -460,6 +460,7 @@ getAllFeatureConfigsUser uid = <$> getConfigForUser @db @LegalholdConfig uid <*> getConfigForUser @db @SSOConfig uid <*> getConfigForUser @db @SearchVisibilityAvailableConfig uid + <*> getConfigForUser @db @SearchVisibilityInboundConfig uid <*> getConfigForUser @db @ValidateSAMLEmailsConfig uid <*> getConfigForUser @db @DigitalSignaturesConfig uid <*> getConfigForUser @db @AppLockConfig uid @@ -470,7 +471,6 @@ getAllFeatureConfigsUser uid = <*> getConfigForUser @db @GuestLinksConfig uid <*> getConfigForUser @db @SndFactorPasswordChallengeConfig uid <*> getConfigForUser @db @MLSConfig uid - <*> getConfigForUser @db @SearchVisibilityInboundConfig uid getAllFeatureConfigsTeam :: forall db r. @@ -492,6 +492,7 @@ getAllFeatureConfigsTeam tid = <$> getConfigForTeam @db @LegalholdConfig tid <*> getConfigForTeam @db @SSOConfig tid <*> getConfigForTeam @db @SearchVisibilityAvailableConfig tid + <*> getConfigForTeam @db @SearchVisibilityInboundConfig tid <*> getConfigForTeam @db @ValidateSAMLEmailsConfig tid <*> getConfigForTeam @db @DigitalSignaturesConfig tid <*> getConfigForTeam @db @AppLockConfig tid @@ -502,7 +503,6 @@ getAllFeatureConfigsTeam tid = <*> getConfigForTeam @db @GuestLinksConfig tid <*> getConfigForTeam @db @SndFactorPasswordChallengeConfig tid <*> getConfigForTeam @db @MLSConfig tid - <*> getConfigForTeam @db @SearchVisibilityInboundConfig tid -- | Note: this is an internal function which doesn't cover all features, e.g. LegalholdConfig genericGetConfigForTeam :: diff --git a/services/galley/src/Galley/App.hs b/services/galley/src/Galley/App.hs index 7ea627cc3b4..a145b9b446d 100644 --- a/services/galley/src/Galley/App.hs +++ b/services/galley/src/Galley/App.hs @@ -80,6 +80,7 @@ import Galley.Env import Galley.External import Galley.Intra.Effects import Galley.Intra.Federator +import Galley.Keys import Galley.Options import Galley.Queue import qualified Galley.Queue as Q @@ -157,6 +158,7 @@ createEnv m o = do <$> Q.new 16000 <*> initExtEnv <*> maybe (pure Nothing) (fmap Just . Aws.mkEnv l mgr) (o ^. optJournal) + <*> loadAllMLSKeys (fold (o ^. optSettings . setMlsPrivateKeyPaths)) initCassandra :: Opts -> Logger -> IO ClientState initCassandra o l = do diff --git a/services/galley/src/Galley/Cassandra/Conversation/Members.hs b/services/galley/src/Galley/Cassandra/Conversation/Members.hs index 0edf47b65c1..12fa07c4d17 100644 --- a/services/galley/src/Galley/Cassandra/Conversation/Members.hs +++ b/services/galley/src/Galley/Cassandra/Conversation/Members.hs @@ -321,6 +321,16 @@ filterRemoteConvMembers users (qUntagged -> Qualified conv dom) = . retry x1 $ query Cql.selectRemoteConvMembers (params LocalQuorum (user, dom, conv)) +lookupLocalMemberRemoteConv :: + UserId -> + Remote ConvId -> + Client (Maybe UserId) +lookupLocalMemberRemoteConv userId (qUntagged -> Qualified conv dom) = + runIdentity + <$$> retry + x5 + (query1 Cql.selectRemoteConvMembers (params LocalQuorum (userId, dom, conv))) + removeLocalMembersFromRemoteConv :: -- | The conversation to remove members from Remote ConvId -> @@ -374,6 +384,7 @@ interpretMemberStoreToCassandra = interpret $ \case GetLocalMembers cid -> embedClient $ members cid GetRemoteMember cid uid -> embedClient $ lookupRemoteMember cid (tDomain uid) (tUnqualified uid) GetRemoteMembers rcid -> embedClient $ lookupRemoteMembers rcid + CheckLocalMemberRemoteConv uid rcnv -> fmap (not . null) $ embedClient $ lookupLocalMemberRemoteConv uid rcnv SelectRemoteMembers uids rcnv -> embedClient $ filterRemoteConvMembers uids rcnv SetSelfMember qcid luid upd -> embedClient $ updateSelfMember qcid luid upd SetOtherMember lcid quid upd -> diff --git a/services/galley/src/Galley/Cassandra/ConversationList.hs b/services/galley/src/Galley/Cassandra/ConversationList.hs index ae8c03eb06f..93ae9352d72 100644 --- a/services/galley/src/Galley/Cassandra/ConversationList.hs +++ b/services/galley/src/Galley/Cassandra/ConversationList.hs @@ -27,14 +27,13 @@ import Data.Id import Data.Qualified import Data.Range import Galley.Cassandra.Instances () -import Galley.Cassandra.Paging import qualified Galley.Cassandra.Queries as Cql -import Galley.Cassandra.ResultSet import Galley.Cassandra.Store import Galley.Effects.ListItems import Imports hiding (max) import Polysemy import Polysemy.Input +import Wire.Sem.Paging.Cassandra -- | Deprecated, use 'localConversationIdsPageFrom' conversationIdsFrom :: diff --git a/services/galley/src/Galley/Cassandra/ResultSet.hs b/services/galley/src/Galley/Cassandra/ResultSet.hs deleted file mode 100644 index 5d8f801f88a..00000000000 --- a/services/galley/src/Galley/Cassandra/ResultSet.hs +++ /dev/null @@ -1,51 +0,0 @@ --- This file is part of the Wire Server implementation. --- --- Copyright (C) 2022 Wire Swiss GmbH --- --- This program is free software: you can redistribute it and/or modify it under --- the terms of the GNU Affero General Public License as published by the Free --- Software Foundation, either version 3 of the License, or (at your option) any --- later version. --- --- This program is distributed in the hope that it will be useful, but WITHOUT --- ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS --- FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more --- details. --- --- You should have received a copy of the GNU Affero General Public License along --- with this program. If not, see . - -module Galley.Cassandra.ResultSet where - -import Cassandra -import Imports - --- We use this newtype to highlight the fact that the 'Page' wrapped in here --- can not reliably used for paging. --- --- The reason for this is that Cassandra returns 'hasMore' as true if the --- page size requested is equal to result size. To work around this we --- actually request for one additional element and drop the last value if --- necessary. This means however that 'nextPage' does not work properly as --- we would miss a value on every page size. --- Thus, and since we don't want to expose the ResultSet constructor --- because it gives access to `nextPage`, we give accessors to the results --- and a more typed `hasMore` (ResultSetComplete | ResultSetTruncated) -data ResultSet a = ResultSet - { resultSetResult :: [a], - resultSetType :: ResultSetType - } - deriving stock (Show, Functor, Foldable, Traversable) - --- | A more descriptive type than using a simple bool to represent `hasMore` -data ResultSetType - = ResultSetComplete - | ResultSetTruncated - deriving stock (Eq, Show) - -mkResultSet :: Page a -> ResultSet a -mkResultSet page = ResultSet (result page) typ - where - typ - | hasMore page = ResultSetTruncated - | otherwise = ResultSetComplete diff --git a/services/galley/src/Galley/Cassandra/Team.hs b/services/galley/src/Galley/Cassandra/Team.hs index 54c36ea42dc..3156f2fc613 100644 --- a/services/galley/src/Galley/Cassandra/Team.hs +++ b/services/galley/src/Galley/Cassandra/Team.hs @@ -41,9 +41,7 @@ import Data.UUID.V4 (nextRandom) import qualified Galley.Aws as Aws import qualified Galley.Cassandra.Conversation as C import Galley.Cassandra.LegalHold (isTeamLegalholdWhitelisted) -import Galley.Cassandra.Paging import qualified Galley.Cassandra.Queries as Cql -import Galley.Cassandra.ResultSet import Galley.Cassandra.Store import Galley.Effects.ListItems import Galley.Effects.TeamMemberStore @@ -61,6 +59,7 @@ import Wire.API.Team import Wire.API.Team.Conversation import Wire.API.Team.Member import Wire.API.Team.Permission (Perm (SetBilling), Permissions, self) +import Wire.Sem.Paging.Cassandra interpretTeamStoreToCassandra :: Members '[Embed IO, Input Env, Input ClientState] r => diff --git a/services/galley/src/Galley/Effects.hs b/services/galley/src/Galley/Effects.hs index ca74c9d98d5..3cdcd04dd4f 100644 --- a/services/galley/src/Galley/Effects.hs +++ b/services/galley/src/Galley/Effects.hs @@ -62,7 +62,6 @@ where import Data.Id import Data.Qualified import Data.Time.Clock -import Galley.Cassandra.Paging import Galley.Cassandra.TeamFeatures (Cassandra) import Galley.Effects.BotAccess import Galley.Effects.BrigAccess @@ -94,6 +93,7 @@ import Polysemy.Error import Polysemy.Input import Polysemy.TinyLog import Wire.API.Error +import Wire.Sem.Paging.Cassandra -- All the possible high-level effects. type GalleyEffects1 = diff --git a/services/galley/src/Galley/Effects/BrigAccess.hs b/services/galley/src/Galley/Effects/BrigAccess.hs index 32933ff8594..31390adcd95 100644 --- a/services/galley/src/Galley/Effects/BrigAccess.hs +++ b/services/galley/src/Galley/Effects/BrigAccess.hs @@ -127,7 +127,7 @@ data BrigAccess m a where RemoveLegalHoldClientFromUser :: UserId -> BrigAccess m () GetAccountConferenceCallingConfigClient :: UserId -> BrigAccess m (WithStatusNoLock ConferenceCallingConfig) GetClientByKeyPackageRef :: KeyPackageRef -> BrigAccess m (Maybe ClientIdentity) - GetLocalMLSClients :: Local UserId -> SignatureSchemeTag -> BrigAccess m (Set ClientId) + GetLocalMLSClients :: Local UserId -> SignatureSchemeTag -> BrigAccess m (Set ClientInfo) AddKeyPackageRef :: KeyPackageRef -> Qualified UserId -> ClientId -> Qualified ConvId -> BrigAccess m () UpdateKeyPackageRef :: KeyPackageUpdate -> BrigAccess m () UpdateSearchVisibilityInbound :: diff --git a/services/galley/src/Galley/Effects/ListItems.hs b/services/galley/src/Galley/Effects/ListItems.hs index 8853909a1f6..0e1e4c8e690 100644 --- a/services/galley/src/Galley/Effects/ListItems.hs +++ b/services/galley/src/Galley/Effects/ListItems.hs @@ -24,9 +24,9 @@ module Galley.Effects.ListItems where import Data.Id -import Galley.Effects.Paging import Imports import Polysemy +import Wire.Sem.Paging -- | General pagination-aware list-by-user effect data ListItems p i m a where diff --git a/services/galley/src/Galley/Effects/MemberStore.hs b/services/galley/src/Galley/Effects/MemberStore.hs index 51606436f2d..11dbe2f836e 100644 --- a/services/galley/src/Galley/Effects/MemberStore.hs +++ b/services/galley/src/Galley/Effects/MemberStore.hs @@ -32,6 +32,7 @@ module Galley.Effects.MemberStore getLocalMembers, getRemoteMember, getRemoteMembers, + checkLocalMemberRemoteConv, selectRemoteMembers, -- * Update members @@ -64,6 +65,7 @@ data MemberStore m a where GetLocalMembers :: ConvId -> MemberStore m [LocalMember] GetRemoteMember :: ConvId -> Remote UserId -> MemberStore m (Maybe RemoteMember) GetRemoteMembers :: ConvId -> MemberStore m [RemoteMember] + CheckLocalMemberRemoteConv :: UserId -> Remote ConvId -> MemberStore m Bool SelectRemoteMembers :: [UserId] -> Remote ConvId -> MemberStore m ([UserId], Bool) SetSelfMember :: Qualified ConvId -> Local UserId -> MemberUpdate -> MemberStore m () SetOtherMember :: Local ConvId -> Qualified UserId -> OtherMemberUpdate -> MemberStore m () diff --git a/services/galley/src/Galley/Effects/RemoteConversationListStore.hs b/services/galley/src/Galley/Effects/RemoteConversationListStore.hs index a942f7c6e9f..54a076818ab 100644 --- a/services/galley/src/Galley/Effects/RemoteConversationListStore.hs +++ b/services/galley/src/Galley/Effects/RemoteConversationListStore.hs @@ -26,10 +26,10 @@ where import Data.Id import Data.Qualified -import Galley.Effects.Paging import Galley.Types.Conversations.Members import Imports import Polysemy +import Wire.Sem.Paging data RemoteConversationListStore p m a where ListRemoteConversations :: diff --git a/services/galley/src/Galley/Effects/TeamMemberStore.hs b/services/galley/src/Galley/Effects/TeamMemberStore.hs index eac886b9463..84f2dbca287 100644 --- a/services/galley/src/Galley/Effects/TeamMemberStore.hs +++ b/services/galley/src/Galley/Effects/TeamMemberStore.hs @@ -27,10 +27,10 @@ module Galley.Effects.TeamMemberStore where import Data.Id -import Galley.Effects.Paging import Imports import Polysemy import Wire.API.Team.Member +import Wire.Sem.Paging data TeamMemberStore p m a where ListTeamMembers :: diff --git a/services/galley/src/Galley/Effects/TeamStore.hs b/services/galley/src/Galley/Effects/TeamStore.hs index 5441919cca1..bf2fdbb5664 100644 --- a/services/galley/src/Galley/Effects/TeamStore.hs +++ b/services/galley/src/Galley/Effects/TeamStore.hs @@ -79,7 +79,6 @@ where import Data.Id import Data.Range import Galley.Effects.ListItems -import Galley.Effects.Paging import Galley.Types.Teams import Galley.Types.Teams.Intra import Imports @@ -91,6 +90,7 @@ import Wire.API.Team import Wire.API.Team.Conversation import Wire.API.Team.Member (HardTruncationLimit, TeamMember, TeamMemberList) import Wire.API.Team.Permission +import Wire.Sem.Paging data TeamStore m a where CreateTeamMember :: TeamId -> TeamMember -> TeamStore m () diff --git a/services/galley/src/Galley/Env.hs b/services/galley/src/Galley/Env.hs index 8958e3cdc73..76eecaa2cf3 100644 --- a/services/galley/src/Galley/Env.hs +++ b/services/galley/src/Galley/Env.hs @@ -38,6 +38,8 @@ import qualified OpenSSL.X509.SystemStore as Ssl import Ssl.Util import System.Logger import Util.Options +import Wire.API.MLS.Credential +import Wire.API.MLS.Keys import Wire.API.Team.Member data DeleteItem = TeamItem TeamId UserId (Maybe ConnId) @@ -55,7 +57,8 @@ data Env = Env _cstate :: ClientState, _deleteQueue :: Q.Queue DeleteItem, _extEnv :: ExtEnv, - _aEnv :: Maybe Aws.Env + _aEnv :: Maybe Aws.Env, + _mlsKeys :: SignaturePurpose -> MLSKeys } -- | Environment specific to the communication with external diff --git a/services/galley/src/Galley/Intra/Client.hs b/services/galley/src/Galley/Intra/Client.hs index 6eab2c9754c..43ff1ab5b1d 100644 --- a/services/galley/src/Galley/Intra/Client.hs +++ b/services/galley/src/Galley/Intra/Client.hs @@ -185,7 +185,7 @@ getClientByKeyPackageRef ref = do else pure Nothing -- | Calls 'Brig.API.Internal.getMLSClients'. -getLocalMLSClients :: Local UserId -> SignatureSchemeTag -> App (Set ClientId) +getLocalMLSClients :: Local UserId -> SignatureSchemeTag -> App (Set ClientInfo) getLocalMLSClients lusr ss = call Brig diff --git a/services/galley/src/Galley/Keys.hs b/services/galley/src/Galley/Keys.hs new file mode 100644 index 00000000000..129b42396a3 --- /dev/null +++ b/services/galley/src/Galley/Keys.hs @@ -0,0 +1,90 @@ +-- This file is part of the Wire Server implementation. +-- +-- Copyright (C) 2022 Wire Swiss GmbH +-- +-- This program is free software: you can redistribute it and/or modify it under +-- the terms of the GNU Affero General Public License as published by the Free +-- Software Foundation, either version 3 of the License, or (at your option) any +-- later version. +-- +-- This program is distributed in the hope that it will be useful, but WITHOUT +-- ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +-- FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more +-- details. +-- +-- You should have received a copy of the GNU Affero General Public License along +-- with this program. If not, see . + +-- | Handling of MLS private keys used for signing external proposals. +module Galley.Keys + ( MLSPrivateKeyPaths, + loadAllMLSKeys, + ) +where + +import Control.Exception +import Crypto.PubKey.Ed25519 +import Data.ASN1.BinaryEncoding +import Data.ASN1.Encoding +import Data.ASN1.Types +import Data.Bifunctor +import qualified Data.ByteString.Lazy as LBS +import qualified Data.Map as Map +import Data.PEM +import Data.X509 +import Imports +import Wire.API.MLS.Credential +import Wire.API.MLS.Keys + +type MLSPrivateKeyPaths = Map SignaturePurpose (Map SignatureSchemeTag FilePath) + +data MLSPrivateKeyException = MLSPrivateKeyException + { mpkePath :: FilePath, + mpkeMsg :: String + } + deriving (Eq, Show, Typeable) + +instance Exception MLSPrivateKeyException where + displayException e = mpkePath e <> ": " <> mpkeMsg e + +mapToFunction :: (Ord k, Monoid m) => Map k m -> k -> m +mapToFunction m x = Map.findWithDefault mempty x m + +loadAllMLSKeys :: MLSPrivateKeyPaths -> IO (SignaturePurpose -> MLSKeys) +loadAllMLSKeys = fmap mapToFunction . traverse loadMLSKeys + +loadMLSKeys :: Map SignatureSchemeTag FilePath -> IO MLSKeys +loadMLSKeys m = + MLSKeys + <$> traverse loadEd25519KeyPair (Map.lookup Ed25519 m) + +loadEd25519KeyPair :: FilePath -> IO (SecretKey, PublicKey) +loadEd25519KeyPair path = do + bytes <- LBS.readFile path + priv <- + either (throwIO . MLSPrivateKeyException path) pure $ + decodeEd25519PrivateKey bytes + pure (priv, toPublic priv) + +decodeEd25519PrivateKey :: + LByteString -> + Either String SecretKey +decodeEd25519PrivateKey bytes = do + pems <- pemParseLBS bytes + pem <- expectOne "private key" pems + let content = pemContent pem + asn1 <- first displayException (decodeASN1' BER content) + (priv, remainder) <- fromASN1 asn1 + expectEmpty remainder + case priv of + PrivKeyEd25519 sec -> pure sec + _ -> Left $ "invalid signature scheme (expected ed25519)" + where + expectOne :: String -> [a] -> Either String a + expectOne label [] = Left $ "no " <> label <> " found" + expectOne _ [x] = pure x + expectOne label _ = Left $ "found multiple " <> label <> "s" + + expectEmpty :: [a] -> Either String () + expectEmpty [] = pure () + expectEmpty _ = Left "extraneous ASN.1 data" diff --git a/services/galley/src/Galley/Options.hs b/services/galley/src/Galley/Options.hs index d8782870b6b..2d822419134 100644 --- a/services/galley/src/Galley/Options.hs +++ b/services/galley/src/Galley/Options.hs @@ -29,6 +29,7 @@ module Galley.Options setDeleteConvThrottleMillis, setFederationDomain, setEnableIndexedBillingTeamMembers, + setMlsPrivateKeyPaths, setFeatureFlags, defConcurrentDeletionEvents, defDeleteConvThrottleMillis, @@ -57,6 +58,7 @@ import Data.Aeson.TH (deriveFromJSON) import Data.Domain (Domain) import Data.Misc import Data.Range +import Galley.Keys import Galley.Types.Teams import Imports import System.Logger.Extended (Level, LogFormat) @@ -103,6 +105,7 @@ data Settings = Settings -- the owners. -- Defaults to false. _setEnableIndexedBillingTeamMembers :: !(Maybe Bool), + _setMlsPrivateKeyPaths :: !(Maybe MLSPrivateKeyPaths), -- | FUTUREWORK: 'setFeatureFlags' should be renamed to 'setFeatureConfigs' in all types. _setFeatureFlags :: !FeatureFlags } diff --git a/services/galley/test/integration/API/MLS.hs b/services/galley/test/integration/API/MLS.hs index b5fb706cffc..086a206f7d2 100644 --- a/services/galley/test/integration/API/MLS.hs +++ b/services/galley/test/integration/API/MLS.hs @@ -26,10 +26,16 @@ import Bilge hiding (head) import Bilge.Assert import Cassandra import Control.Arrow -import Control.Lens (view) +import Control.Lens (view, (^..)) +import Crypto.Error +import qualified Crypto.PubKey.Ed25519 as C import qualified Data.Aeson as Aeson +import Data.Aeson.Lens +import Data.Binary.Put import qualified Data.ByteString as BS +import qualified Data.ByteString.Base64.URL as B64U import Data.ByteString.Conversion +import qualified Data.ByteString.Lazy as LBS import Data.Default import Data.Domain import Data.Id @@ -37,6 +43,7 @@ import Data.Json.Util hiding ((#)) import qualified Data.List.NonEmpty as NE import qualified Data.List.NonEmpty as NonEmpty import Data.List1 hiding (head) +import qualified Data.Map as Map import Data.Qualified import Data.Range import qualified Data.Set as Set @@ -64,10 +71,15 @@ import Wire.API.Event.Conversation import Wire.API.Federation.API.Common import Wire.API.Federation.API.Galley import Wire.API.MLS.CipherSuite +import Wire.API.MLS.Credential import Wire.API.MLS.Group (convToGroupId) +import Wire.API.MLS.KeyPackage +import Wire.API.MLS.Keys import Wire.API.MLS.Message +import Wire.API.MLS.Serialisation import Wire.API.Message import Wire.API.Routes.Version +import Wire.API.User.Client tests :: IO TestSetup -> TestTree tests s = @@ -75,7 +87,9 @@ tests s = "MLS" [ testGroup "Message" - [test s "sender must be part of conversation" testSenderNotInConversation], + [ test s "sender must be part of conversation" testSenderNotInConversation, + test s "send other user's commit" testSendAnotherUsersCommit + ], testGroup "Welcome" [ test s "local welcome" testLocalWelcome, @@ -92,18 +106,17 @@ tests s = [ test s "add user to a conversation" testAddUser, test s "add user (not connected)" testAddUserNotConnected, test s "add user (partial client list)" testAddUserPartial, + test s "add client of existing user" testAddClientPartial, test s "add user with some non-MLS clients" testAddUserWithProteusClients, test s "add new client of an already-present user to a conversation" testAddNewClient, test s "send a stale commit" testStaleCommit, test s "add remote user to a conversation" testAddRemoteUser, test s "return error when commit is locked" testCommitLock, - test s "add remote user to a conversation" testAddRemoteUser, test s "add user to a conversation with proposal + commit" testAddUserBareProposalCommit, test s "post commit that references a unknown proposal" testUnknownProposalRefCommit, test s "post commit that is not referencing all proposals" testCommitNotReferencingAllProposals, test s "admin removes user from a conversation" testAdminRemovesUserFromConv, test s "admin removes user from a conversation but doesn't list all clients" testRemoveClientsIncomplete, - test s "user tries to remove themselves from conversation" testUserRemovesThemselvesFromConv, test s "anyone removes a non-existing client from a group" (testRemoveDeletedClient True), test s "anyone removes an existing client from group, but the user has other clients" (testRemoveDeletedClient False), test s "admin removes only strict subset of clients from a user" testRemoveSubset @@ -118,12 +131,14 @@ tests s = ], testGroup "Local Sender/Remote Conversation" - [ test s "send application message" testLocalToRemote + [ test s "send application message" testLocalToRemote, + test s "non-member sends application message" testLocalToRemoteNonMember ], testGroup "Remote Sender/Local Conversation" [ test s "POST /federation/send-mls-message" testRemoteToLocal, - test s "POST /federation/send-mls-message, remote user is not a conversation member" testRemoteNonMemberToLocal + test s "POST /federation/send-mls-message, remote user is not a conversation member" testRemoteNonMemberToLocal, + test s "POST /federation/send-mls-message, remote user sends to wrong conversation" testRemoteToLocalWrongConversation ], testGroup "Remote Sender/Remote Conversation" @@ -134,7 +149,14 @@ tests s = "Proposal" [ test s "add a new client to a non-existing conversation" propNonExistingConv, test s "add a new client to an existing conversation" propExistingConv, - test s "add a new client in an invalid epoch" propInvalidEpoch + test s "add a new client in an invalid epoch" propInvalidEpoch, + test s "forward an unsupported proposal" propUnsupported + ], + testGroup + "External Proposal" + [ test s "member adds new client" testExternalAddProposal, + test s "non-member adds new client" testExternalAddProposalWrongUser, + test s "member adds unknown new client" testExternalAddProposalWrongClient ], testGroup "Protocol mismatch" @@ -142,7 +164,8 @@ tests s = test s "add users bypassing MLS" testAddUsersDirectly, test s "remove users bypassing MLS" testRemoveUsersDirectly, test s "send proteus message to an MLS conversation" testProteusMessage - ] + ], + test s "public keys" testPublicKeys ] postMLSConvFail :: TestM () @@ -186,21 +209,7 @@ testSenderNotInConversation = do liftIO $ setupCommit tmp alice "group" "group" $ toList (pClients bob) - - void . liftIO $ - spawn - ( cli - (pClientQid bob) - tmp - [ "group", - "from-welcome", - "--group-out", - tmp "group", - tmp "welcome" - ] - ) - Nothing - + liftIO $ mergeWelcome tmp (pClientQid bob) "group" "group" "welcome" message <- liftIO $ createMessage tmp bob "group" "some text" -- send the message as bob, who is not in the conversation @@ -412,6 +421,17 @@ testAddUserPartial = do (creator, commit) <- withSystemTempDirectory "mls" $ \tmp -> do -- Bob has 3 clients, Charlie has 2 (alice, [bob, charlie]) <- withLastPrekeys $ setupParticipants tmp def ((,LocalUser) <$> [3, 2]) + + -- upload one more key package for each of bob's clients + -- this makes sure the unused client has at least one key package, and + -- therefore will be considered MLS-capable + for_ (pClients bob) $ \(cid, c) -> do + kp <- + liftIO $ + decodeMLSError + =<< spawn (cli cid tmp ["key-package", "create"]) Nothing + addKeyPackage def {mapKeyPackage = False, setPublicKey = False} (pUserId bob) c kp + void $ setupGroup tmp CreateConv alice "group" (commit, _) <- liftIO . setupCommit tmp alice "group" "group" $ @@ -433,6 +453,36 @@ testAddUserPartial = do do + withLastPrekeys $ do + (alice, [bob]) <- setupParticipants tmp def ((,LocalUser) <$> [1]) + (groupId, conversation) <- lift $ setupGroup tmp CreateConv alice "group" + (commit, welcome) <- liftIO . setupCommit tmp alice "group" "group" $ pClients bob + let setup = + MessagingSetup + { creator = alice, + users = [bob], + .. + } + lift $ testSuccessfulCommit setup + + -- create more clients for Bob, only take the first one + nc <- fmap head . replicateM 2 $ do + setupUserClient tmp CreateWithKey True (pUserId bob) + + -- add new client + (commit', welcome') <- + liftIO $ + setupCommit + tmp + alice + "group" + "group" + [(userClientQid (pUserId bob) nc, nc)] + + lift $ testSuccessfulCommitWithNewUsers setup {commit = commit', welcome = welcome'} [] + testAddNewClient :: TestM () testAddNewClient = do withSystemTempDirectory "mls" $ \tmp -> withLastPrekeys $ do @@ -454,6 +504,28 @@ testAddNewClient = do -- and the corresponding commit is sent lift $ testSuccessfulCommitWithNewUsers MessagingSetup {..} [] +testSendAnotherUsersCommit :: TestM () +testSendAnotherUsersCommit = do + withSystemTempDirectory "mls" $ \tmp -> withLastPrekeys $ do + -- bob starts with a single client + (creator, users@[bob]) <- setupParticipants tmp def [(1, LocalUser)] + (groupId, conversation) <- lift $ setupGroup tmp CreateConv creator "group" + + -- creator sends first commit message + do + (commit, welcome) <- liftIO $ setupCommit tmp creator "group" "group" (pClients bob) + lift $ testSuccessfulCommit MessagingSetup {..} + + do + -- then bob adds a new client + c <- setupUserClient tmp CreateWithKey True (pUserId bob) + let bobC = (userClientQid (pUserId bob) c, c) + -- which gets added to the group + (commit, _welcome) <- liftIO $ setupCommit tmp creator "group" "group" [bobC] + -- and the corresponding commit is sent from bob instead of the creator + err <- lift (responseJsonError =<< postMessage (qUnqualified (pUserId bob)) commit do liftIO $ Wai.label err @?= "mls-client-mismatch" -testUserRemovesThemselvesFromConv :: TestM () -testUserRemovesThemselvesFromConv = withSystemTempDirectory "mls" $ \tmp -> do - MessagingSetup {..} <- aliceInvitesBobWithTmp tmp (2, LocalUser) def {createConv = CreateConv} - let [bob] = users - - testSuccessfulCommit MessagingSetup {users = [bob], ..} - - -- FUTUREWORK: create commit as bob, when the openmls library supports removing own clients - (removalCommit, _mbWelcome) <- liftIO $ setupRemoveCommit tmp creator "group" "group" (pClients bob) - - -- bob tries to leave the conversation by removing all its clients - err <- - responseJsonError - =<< postMessage (qUnqualified (pUserId bob)) removalCommit - TestM () testRemoveDeletedClient deleteClientBefore = withSystemTempDirectory "mls" $ \tmp -> do (creator, [bob, dee]) <- withLastPrekeys $ setupParticipants tmp def [(2, LocalUser), (1, LocalUser)] @@ -795,7 +849,35 @@ testRemoveDeletedClient deleteClientBefore = withSystemTempDirectory "mls" $ \tm deleteClient (qUnqualified (pUserId bob)) (snd bobClient2) (Just defPassword) !!! statusCode === const 200 - (removalCommit, _mbWelcome) <- liftIO $ setupRemoveCommit tmp creator "group" "group" [bobClient2] + void . liftIO $ + spawn + ( cli + (pClientQid bob) + tmp + [ "group", + "from-welcome", + "--group-out", + tmp "group", + tmp "welcome" + ] + ) + Nothing + + void . liftIO $ + spawn + ( cli + (pClientQid dee) + tmp + [ "group", + "from-welcome", + "--group-out", + tmp "group", + tmp "welcome" + ] + ) + Nothing + + (removalCommit, _mbWelcome) <- liftIO $ setupRemoveCommit tmp dee "group" "group" [bobClient2] -- dee (which is not an admin) commits removal of bob's deleted client let doCommitRemoval = postMessage (qUnqualified (pUserId dee)) removalCommit @@ -855,7 +937,7 @@ testRemoteAppMessage = withSystemTempDirectory "mls" $ \tmp -> do pure . Aeson.encode . Set.fromList - . map snd + . map (flip ClientInfo True . snd) . toList . pClients $ bob @@ -914,8 +996,8 @@ testRemoteAppMessage = withSystemTempDirectory "mls" $ \tmp -> do -- -- In the test: -- --- setup: 2 10 11 --- skipped: 1 3 5 6 7 8 9 13 +-- setup: 2 5 10 11 +-- skipped: 1 3 6 7 8 9 13 -- faked: 4 -- actual test step: 12 14 testLocalToRemote :: TestM () @@ -931,19 +1013,7 @@ testLocalToRemote = withSystemTempDirectory "mls" $ \tmp -> do } -- step 10 - void . liftIO $ - spawn - ( cli - (pClientQid bob) - tmp - [ "group", - "from-welcome", - "--group-out", - tmp "groupB.json", - tmp "welcome" - ] - ) - Nothing + liftIO $ mergeWelcome tmp (pClientQid bob) "group" "groupB.json" "welcome" -- step 11 message <- liftIO $ @@ -969,6 +1039,29 @@ testLocalToRemote = withSystemTempDirectory "mls" $ \tmp -> do (qDomain (pUserId alice)) nrc + -- A notifies B about bob being in the conversation (Join event): step 5 + now <- liftIO getCurrentTime + let cu = + ConversationUpdate + { cuTime = now, + cuOrigUserId = pUserId alice, + cuConvId = qUnqualified qcnv, + cuAlreadyPresentUsers = [qUnqualified $ pUserId bob], + cuAction = + SomeConversationAction + SConversationJoinTag + ConversationJoin + { cjUsers = pure (pUserId bob), + cjRole = roleNameWireMember + } + } + void $ + runFedClient + @"on-conversation-updated" + fedGalleyClient + (qDomain (pUserId alice)) + cu + let mock req = case frRPC req of "send-mls-message" -> pure (Aeson.encode (MLSMessageResponseUpdates [])) rpc -> assertFailure $ "unmocked RPC called: " <> T.unpack rpc @@ -999,6 +1092,65 @@ testLocalToRemote = withSystemTempDirectory "mls" $ \tmp -> do msrSender bdy @?= qUnqualified (pUserId bob) msrRawMessage bdy @?= Base64ByteString message +testLocalToRemoteNonMember :: TestM () +testLocalToRemoteNonMember = withSystemTempDirectory "mls" $ \tmp -> do + let domain = Domain "faraway.example.com" + -- step 2 + MessagingSetup {creator = alice, users = [bob], ..} <- + aliceInvitesBobWithTmp + tmp + (1, LocalUser) + def + { creatorOrigin = RemoteUser domain + } + + -- step 10 + liftIO $ mergeWelcome tmp (pClientQid bob) "group" "groupB.json" "welcome" + -- step 11 + message <- + liftIO $ + spawn + ( cli + (pClientQid bob) + tmp + ["message", "--group", tmp "groupB.json", "hi"] + ) + Nothing + + fedGalleyClient <- view tsFedGalleyClient + + -- register remote conversation: step 4 + qcnv <- randomQualifiedId (qDomain (pUserId alice)) + let nrc = + NewRemoteConversation (qUnqualified qcnv) $ + ProtocolMLS (ConversationMLSData groupId (Epoch 1) MLS_128_DHKEMX25519_AES128GCM_SHA256_Ed25519) + void $ + runFedClient + @"on-new-remote-conversation" + fedGalleyClient + (qDomain (pUserId alice)) + nrc + + let mock req = case frRPC req of + "send-mls-message" -> pure (Aeson.encode (MLSMessageResponseUpdates [])) + rpc -> assertFailure $ "unmocked RPC called: " <> T.unpack rpc + + void $ + withTempMockFederator' mock $ do + galley <- viewGalley + + -- bob sends a message: step 12 + post + ( galley . paths ["mls", "messages"] + . zUser (qUnqualified (pUserId bob)) + . zConn "conn" + . content "message/mls" + . bytes message + ) + !!! do + const 404 === statusCode + const (Just "no-conversation-member") === fmap Wai.label . responseJsonError + testAppMessage :: TestM () testAppMessage = withSystemTempDirectory "mls" $ \tmp -> do (creator, users) <- withLastPrekeys $ setupParticipants tmp def ((,LocalUser) <$> [1, 2, 3]) @@ -1050,19 +1202,7 @@ testAppMessage2 = do void $ postCommit setup let bob = head users - void . liftIO $ - spawn - ( cli - (pClientQid bob) - tmp - [ "group", - "from-welcome", - "--group-out", - tmp "group", - tmp "welcome" - ] - ) - Nothing + liftIO $ mergeWelcome tmp (pClientQid bob) "group" "group" "welcome" message <- liftIO $ createMessage tmp bob "group" "some text" @@ -1183,7 +1323,7 @@ testRemoteToLocal = do pure . Aeson.encode . Set.fromList - . map snd + . map (flip ClientInfo True . snd) . toList . pClients $ bob @@ -1191,19 +1331,7 @@ testRemoteToLocal = do void . withTempMockFederator' mockedResponse $ postCommit setup - void . liftIO $ - spawn - ( cli - (pClientQid bob) - tmp - [ "group", - "from-welcome", - "--group-out", - tmp "groupB.json", - tmp "welcome" - ] - ) - Nothing + liftIO $ mergeWelcome tmp (pClientQid bob) "group" "groupB.json" "welcome" message <- liftIO $ spawn @@ -1237,6 +1365,72 @@ testRemoteToLocal = do WS.assertMatch_ (5 # Second) ws $ wsAssertMLSMessage conversation (pUserId bob) message +testRemoteToLocalWrongConversation :: TestM () +testRemoteToLocalWrongConversation = do + -- alice is local, bob is remote + -- alice creates a local conversation and invites bob + -- bob then sends a message to the conversation + + let bobDomain = Domain "faraway.example.com" + + -- Simulate the whole MLS setup for both clients first. In reality, + -- backend calls would need to happen in order for bob to get ahold of a + -- welcome message, but that should not affect the correctness of the test. + + (MessagingSetup {..}, message) <- withSystemTempDirectory "mls" $ \tmp -> do + setup <- + aliceInvitesBobWithTmp + tmp + (1, RemoteUser bobDomain) + def + { createConv = CreateConv + } + bob <- assertOne (users setup) + let mockedResponse fedReq = + case frRPC fedReq of + "mls-welcome" -> pure (Aeson.encode EmptyResponse) + "on-new-remote-conversation" -> pure (Aeson.encode EmptyResponse) + "on-conversation-updated" -> pure (Aeson.encode ()) + "get-mls-clients" -> + pure + . Aeson.encode + . Set.fromList + . map (flip ClientInfo True . snd) + . toList + . pClients + $ bob + ms -> assertFailure ("unmocked endpoint called: " <> cs ms) + + void . withTempMockFederator' mockedResponse $ + postCommit setup + liftIO $ mergeWelcome tmp (pClientQid bob) "group" "groupB.json" "welcome" + message <- + liftIO $ + spawn + ( cli + (pClientQid bob) + tmp + ["message", "--group", tmp "groupB.json", "hello from another backend"] + ) + Nothing + pure (setup, message) + + let bob = head users + + fedGalleyClient <- view tsFedGalleyClient + + -- actual test + randomConfId <- randomId + let msr = + MessageSendRequest + { msrConvId = randomConfId, + msrSender = qUnqualified (pUserId bob), + msrRawMessage = Base64ByteString message + } + + resp <- runFedClient @"send-mls-message" fedGalleyClient bobDomain msr + liftIO $ resp @?= MLSMessageResponseError MLSGroupConversationMismatch + testRemoteNonMemberToLocal :: TestM () testRemoteNonMemberToLocal = do -- alice is local, bob is remote @@ -1258,19 +1452,7 @@ testRemoteNonMemberToLocal = do { createConv = CreateConv } bob <- assertOne (users setup) - void . liftIO $ - spawn - ( cli - (pClientQid bob) - tmp - [ "group", - "from-welcome", - "--group-out", - tmp "groupB.json", - tmp "welcome" - ] - ) - Nothing + liftIO $ mergeWelcome tmp (pClientQid bob) "group" "groupB.json" "welcome" message <- liftIO $ spawn @@ -1376,11 +1558,146 @@ propInvalidEpoch = withSystemTempDirectory "mls" $ \tmp -> do err <- responseJsonError =<< postMessage (qUnqualified (pUserId creator)) prop - do + (creator, [bob]) <- withLastPrekeys $ setupParticipants tmp def [(1, LocalUser)] + (groupId, conversation) <- setupGroup tmp CreateConv creator "group" + + bobClient1 <- assertOne . toList $ pClients bob + (commit, welcome) <- + liftIO $ + setupCommit tmp creator "group" "group" $ + NonEmpty.tail (pClients creator) <> [bobClient1] + testSuccessfulCommit MessagingSetup {users = [bob], ..} + + liftIO $ mergeWelcome tmp (fst bobClient1) "group" "group" "welcome" + + bobClient2Qid <- + userClientQid (pUserId bob) + <$> withLastPrekeys (setupUserClient tmp CreateWithKey True (pUserId bob)) + externalProposal <- liftIO $ createExternalProposal tmp bobClient2Qid "group" "group" + postMessage (qUnqualified (pUserId bob)) externalProposal !!! const 201 === statusCode + +testExternalAddProposalWrongUser :: TestM () +testExternalAddProposalWrongUser = withSystemTempDirectory "mls" $ \tmp -> do + (creator, [bob, charly]) <- withLastPrekeys $ setupParticipants tmp def [(1, LocalUser), (1, LocalUser)] + (groupId, conversation) <- setupGroup tmp CreateConv creator "group" + + bobClient1 <- assertOne . toList $ pClients bob + charlyClient1 <- assertOne . toList $ pClients charly + (commit, welcome) <- + liftIO $ + setupCommit tmp creator "group" "group" $ + NonEmpty.tail (pClients creator) <> [bobClient1, charlyClient1] + testSuccessfulCommit MessagingSetup {users = [bob, charly], ..} + + liftIO $ mergeWelcome tmp (fst bobClient1) "group" "group" "welcome" + + bobClient2Qid <- + userClientQid (pUserId bob) + <$> withLastPrekeys (setupUserClient tmp CreateWithKey True (pUserId bob)) + externalProposal <- liftIO $ createExternalProposal tmp bobClient2Qid "group" "group" + postMessage (qUnqualified (pUserId charly)) externalProposal !!! do + const 422 === statusCode + const (Just "mls-unsupported-proposal") === fmap Wai.label . responseJsonError + +testExternalAddProposalWrongClient :: TestM () +testExternalAddProposalWrongClient = withSystemTempDirectory "mls" $ \tmp -> do + (creator, [bob, charly]) <- withLastPrekeys $ setupParticipants tmp def [(1, LocalUser), (1, LocalUser)] + (groupId, conversation) <- setupGroup tmp CreateConv creator "group" + + bobClient1 <- assertOne . toList $ pClients bob + charlyClient1 <- assertOne . toList $ pClients charly + (commit, welcome) <- + liftIO $ + setupCommit tmp creator "group" "group" $ + NonEmpty.tail (pClients creator) <> [bobClient1, charlyClient1] + testSuccessfulCommit MessagingSetup {users = [bob, charly], ..} + + liftIO $ mergeWelcome tmp (fst bobClient1) "group" "group" "welcome" + + bobClient2Qid <- + userClientQid (pUserId bob) + <$> withLastPrekeys (setupUserClient tmp CreateWithoutKey True (pUserId bob)) + externalProposal <- liftIO $ createExternalProposal tmp bobClient2Qid "group" "group" + postMessage (qUnqualified (pUserId charly)) externalProposal !!! do + const 422 === statusCode + const (Just "mls-unsupported-proposal") === fmap Wai.label . responseJsonError + +-- FUTUREWORK: test processing a commit containing the external proposal +testPublicKeys :: TestM () +testPublicKeys = do + u <- randomId + g <- viewGalley + keys <- + responseJsonError + =<< get + ( g + . paths ["mls", "public-keys"] + . zUser u + ) + do + MessagingSetup {..} <- aliceInvitesBobWithTmp tmp (1, LocalUser) def {createConv = CreateConv} + aliceKP <- liftIO $ do + d <- BS.readFile (tmp pClientQid creator) + either (\e -> assertFailure ("could not parse key package: " <> T.unpack e)) pure $ + decodeMLS' d + let alicePublicKey = bcSignatureKey $ kpCredential aliceKP + + -- "\0 " corresponds to 0020 in TLS encoding, which is the length of the + -- following public key + file <- + liftIO . BS.readFile $ + tmp pClientQid creator <> ".db" cs (B64U.encode $ "\0 " <> alicePublicKey) + let s = + file ^.. key "signature_private_key" . key "value" . _Array . traverse . _Integer + & fmap fromIntegral + & BS.pack + let (privKey, pubKey) = BS.splitAt 32 s + liftIO $ alicePublicKey @?= pubKey + let aliceRef = + kpRef + MLS_128_DHKEMX25519_AES128GCM_SHA256_Ed25519 + . KeyPackageData + . rmRaw + . kpTBS + $ aliceKP + let Just appAckMsg = + maybeCryptoError $ + mkAppAckProposalMessage + groupId + (Epoch 0) + aliceRef + [] + <$> C.secretKey privKey + <*> C.publicKey pubKey + msgSerialised = + LBS.toStrict . runPut . serialiseMLS $ appAckMsg + + postMessage (qUnqualified . pUserId $ creator) msgSerialised + !!! const 201 === statusCode diff --git a/services/galley/test/integration/API/MLS/Util.hs b/services/galley/test/integration/API/MLS/Util.hs index fdb1e876188..c4cca7bf2cd 100644 --- a/services/galley/test/integration/API/MLS/Util.hs +++ b/services/galley/test/integration/API/MLS/Util.hs @@ -25,6 +25,8 @@ import Bilge.Assert import Control.Lens (preview, to, view) import Control.Monad.Catch import qualified Control.Monad.State as State +import Crypto.PubKey.Ed25519 +import qualified Data.ByteArray as BA import qualified Data.ByteString as BS import Data.ByteString.Conversion import Data.Default @@ -51,6 +53,7 @@ import Wire.API.Event.Conversation import Wire.API.MLS.Credential import Wire.API.MLS.KeyPackage import Wire.API.MLS.Message +import Wire.API.MLS.Proposal import Wire.API.MLS.Serialisation import Wire.API.User.Client import Wire.API.User.Client.Prekey @@ -96,6 +99,19 @@ data MessagingSetup = MessagingSetup } deriving (Show) +data AddKeyPackage = AddKeyPackage + { mapKeyPackage :: Bool, + setPublicKey :: Bool + } + deriving (Show) + +instance Default AddKeyPackage where + def = + AddKeyPackage + { mapKeyPackage = True, + setPublicKey = True + } + cli :: String -> FilePath -> [String] -> CreateProcess cli store tmp args = proc "mls-test-cli" $ @@ -157,7 +173,7 @@ setupUserClient tmp doCreateClients mapKeyPackage usr = do -- does not have to be created and it is remote, pretend to have claimed its -- key package. case doCreateClients of - CreateWithKey -> addKeyPackage mapKeyPackage usr c kp + CreateWithKey -> addKeyPackage def {mapKeyPackage = mapKeyPackage} usr c kp DontCreateClients | localDomain /= qDomain usr -> do brig <- view tsBrig let bundle = @@ -170,7 +186,8 @@ setupUserClient tmp doCreateClients mapKeyPackage usr = do kpbeKeyPackage = KeyPackageData $ rmRaw kp } when mapKeyPackage $ mapRemoteKeyPackageRef brig bundle - _ -> pure () + DontCreateClients -> pure () + CreateWithoutKey -> pure () pure c @@ -329,6 +346,29 @@ setupRemoveCommit tmp admin groupName newGroupName clients = do True -> Just <$> BS.readFile welcomeFile pure (commit, welcome) +mergeWelcome :: + (HasCallStack) => + String -> + String -> + String -> + String -> + String -> + IO () +mergeWelcome tmp clientQid groupIn groupOut welcomeIn = + void $ + spawn + ( cli + clientQid + tmp + [ groupIn, + "from-welcome", + "--group-out", + tmp groupOut, + tmp welcomeIn + ] + ) + Nothing + bareAddProposal :: HasCallStack => String -> @@ -380,6 +420,28 @@ pendingProposalsCommit tmp creator groupName = do True -> Just <$> BS.readFile welcomeFile pure (commit, welcome) +createExternalProposal :: + HasCallStack => + String -> + String -> + String -> + String -> + IO ByteString +createExternalProposal tmp creatorClientQid groupIn groupOut = do + spawn + ( cli + creatorClientQid + tmp + $ [ "external-proposal", + "--group-in", + tmp groupIn, + "--group-out", + tmp groupOut, + "add" + ] + ) + Nothing + createMessage :: HasCallStack => String -> @@ -427,18 +489,26 @@ aliceInvitesBobWithTmp tmp bobConf opts@SetupOptions {..} = do .. } -addKeyPackage :: HasCallStack => Bool -> Qualified UserId -> ClientId -> RawMLS KeyPackage -> TestM () -addKeyPackage mapKeyPackage u c kp = do - let update = defUpdateClient {updateClientMLSPublicKeys = Map.singleton Ed25519 (bcSignatureKey (kpCredential (rmValue kp)))} - -- set public key +addKeyPackage :: + HasCallStack => + AddKeyPackage -> + Qualified UserId -> + ClientId -> + RawMLS KeyPackage -> + TestM () +addKeyPackage AddKeyPackage {..} u c kp = do brig <- view tsBrig - put - ( brig - . paths ["clients", toByteString' c] - . zUser (qUnqualified u) - . json update - ) - !!! const 200 === statusCode + + when setPublicKey $ do + -- set public key + let update = defUpdateClient {updateClientMLSPublicKeys = Map.singleton Ed25519 (bcSignatureKey (kpCredential (rmValue kp)))} + put + ( brig + . paths ["clients", toByteString' c] + . zUser (qUnqualified u) + . json update + ) + !!! const 200 === statusCode -- upload key package post @@ -514,3 +584,25 @@ postWelcome uid welcome = do . content "message/mls" . bytes welcome ) + +mkAppAckProposalMessage :: + GroupId -> + Epoch -> + KeyPackageRef -> + [MessageRange] -> + SecretKey -> + PublicKey -> + Message 'MLSPlainText +mkAppAckProposalMessage gid epoch ref mrs priv pub = do + let tbs = + mkRawMLS $ + MessageTBS + { tbsMsgFormat = KnownFormatTag, + tbsMsgGroupId = gid, + tbsMsgEpoch = epoch, + tbsMsgAuthData = mempty, + tbsMsgSender = MemberSender ref, + tbsMsgPayload = ProposalMessage (mkAppAckProposal mrs) + } + sig = BA.convert $ sign priv pub (rmRaw tbs) + in (Message tbs (MessageExtraFields sig Nothing Nothing)) diff --git a/services/galley/test/resources/ed25519.pem b/services/galley/test/resources/ed25519.pem new file mode 120000 index 00000000000..2b827f502d6 --- /dev/null +++ b/services/galley/test/resources/ed25519.pem @@ -0,0 +1 @@ +../../../../deploy/services-demo/conf/ed25519.pem \ No newline at end of file diff --git a/services/spar/spar.cabal b/services/spar/spar.cabal index b19d657dda0..9cb59ca37ee 100644 --- a/services/spar/spar.cabal +++ b/services/spar/spar.cabal @@ -383,7 +383,7 @@ executable spar-integration -Wno-redundant-constraints -Werror -threaded -rtsopts -with-rtsopts=-N - build-tool-depends: hspec-discover:hspec-discover -any + build-tool-depends: hspec-discover:hspec-discover build-depends: aeson , aeson-qq @@ -802,7 +802,7 @@ test-suite spec -Wno-redundant-constraints -Werror -threaded -rtsopts -with-rtsopts=-N - build-tool-depends: hspec-discover:hspec-discover -any + build-tool-depends: hspec-discover:hspec-discover build-depends: aeson , aeson-qq diff --git a/stack.yaml b/stack.yaml index 8bdbcfd2551..330a3c96a7c 100644 --- a/stack.yaml +++ b/stack.yaml @@ -146,6 +146,10 @@ extra-deps: - git: https://gitlab.com/axeman/swagger commit: e2d3f5b5274b8d8d301b5377b0af4319cea73f9e +# Use forked cql-io for https://gitlab.com/twittner/cql-io/-/merge_requests/20 +- git: https://gitlab.com/axeman/cql-io + commit: c2b6aa995b5817ed7c78c53f72d5aa586ef87c36 + ############################################################ # Wire packages ############################################################ @@ -215,7 +219,6 @@ extra-deps: - HsOpenSSL-x509-system-0.1.0.4 - cql-4.0.3 -- cql-io-1.1.1 - primitive-extras-0.10.1.1 - text-format-0.3.2 - hex-0.2.0 diff --git a/stack.yaml.lock b/stack.yaml.lock index b9126164479..ae8292bdbd6 100644 --- a/stack.yaml.lock +++ b/stack.yaml.lock @@ -371,6 +371,17 @@ packages: original: git: https://gitlab.com/axeman/swagger commit: e2d3f5b5274b8d8d301b5377b0af4319cea73f9e +- completed: + name: cql-io + version: 1.1.1 + git: https://gitlab.com/axeman/cql-io + pantry-tree: + size: 2172 + sha256: 4eead69907e2fc081d66b9d0ab4f73234f7636220c995147f499777dd14b9250 + commit: c2b6aa995b5817ed7c78c53f72d5aa586ef87c36 + original: + git: https://gitlab.com/axeman/cql-io + commit: c2b6aa995b5817ed7c78c53f72d5aa586ef87c36 - completed: name: cryptobox-haskell version: 0.1.1 @@ -728,13 +739,6 @@ packages: sha256: 94433b7c7c46bea532fdc64c6988643a48e39b643948003b27e5bde1bdad3b24 original: hackage: cql-4.0.3 -- completed: - hackage: cql-io-1.1.1@sha256:897ef0811b227c8b1a269b29b9c1ebfb09c46f00d66834e2e8c6f19ea7f90f7d,4611 - pantry-tree: - size: 2067 - sha256: 7ced76ae95b51fa1669b4fcaeec3825b5cb8cf1f4e37c53d0bddf6234742eba8 - original: - hackage: cql-io-1.1.1 - completed: hackage: primitive-extras-0.10.1.1@sha256:47c4d211166bc31ebdb053f610e4b5387c01d00bde81840e59438469cef6c94e,2955 pantry-tree: diff --git a/tools/api-simulations/README.md b/tools/api-simulations/README.md index 7c872a10629..b9db899eb17 100644 --- a/tools/api-simulations/README.md +++ b/tools/api-simulations/README.md @@ -193,3 +193,128 @@ The output of a typical (successful) run looks like Ignored: 0 Missed: 0 + +## Load tests and configuration option examples + +(legacy usage; untested if this still works, provided as an example in case api-simulations will see more usage in the future) + +Using a `mailboxes.json` and `users.txt` (for wire employees: check `s3://z-config/simulator/` folder for an example), in the past the following ansible script was run to run a smoke or conversation load test. File snippet provided as an example of configuration options used. + +Adapt the actual run to a modern dockerized environment (i.e. without the use of ansible). + +``` +- name: run smoke test + hosts: simulator + vars: + log_level: Info + api_host: '{{ external_env_domain }}' # must match BRIG_COOKIE_DOMAIN + api_port: '{{ routing_table.nginz.web.local }}' + email_sender: # must match BRIG_EMAIL_SENDER + prod: 'accounts@wire.com' + internal: false + tasks: + # api-loadtest and api-smoketest now take a mandatory argument 'users-federation-domain' + # introduced as a side-effect in https://github.com/wireapp/wire-server/pull/1283 + # which we parse here from brig's config + - name: parse federationDomain from brig + shell: "yq .config.setFederationDomain.{{ khan_env }} < roles/brig/vars/main.yml " + register: federationDomain + delegate_to: localhost + + - name: update /etc/hosts + lineinfile: > + dest=/etc/hosts + regexp='.*{{ api_host }}$' + line="127.0.0.1 {{ api_host }}" + state=present + when: internal and simulation is defined and simulation == 'smoketest' + + - shell: LOG_LEVEL={{ log_level }} LOG_BUFFER=0 + /opt/api-simulations/bin/api-smoketest + {% if internal %} + --api-host='{{ api_host }}' + --api-port='{{ api_port }}' + {% else %} + --api-ssl + --api-host='{{ khan_env }}-nginz-https.{{ api_host }}' + --api-port=443 + --api-websocket-host='{{ khan_env }}-nginz-ssl.{{ api_host }}' + --api-websocket-port=443 + {% endif %} + --mailbox-config=/etc/api-simulations/mailboxes.json + --users-federation-domain='{{ federationDomain.stdout }}' + {% if email_sender[khan_env] is defined %} + --sender-email='{{ email_sender[khan_env] }}' + {% endif %} + --report-dir=/tmp/simulator + --enable-asserts + 2>&1 | ./run + args: + chdir: /etc/sv/simulator/log + when: simulation is defined and simulation == 'smoketest' + tags: + - simulation + - remote + +- name: run conversation load test + hosts: simulator + vars: + log_level: Info + api_host: '{{ external_env_domain }}' # must match BRIG_COOKIE_DOMAIN + api_port: '{{ routing_table.nginz.web.local }}' + conversations_total: 1 + conversation_bots_min: 2 + conversation_bots_max: 5 + bot_messages_min: 5 + bot_messages_max: 10 + bot_assets_min: 1 + bot_assets_max: 1 + message_length_min: 1 + message_length_max: 100 + asset_size_min: 10 + asset_size_max: 1000 + assets_noinline: false + tasks: + # api-loadtest and api-smoketest now take a mandatory argument 'users-federation-domain' + # introduced as a side-effect in https://github.com/wireapp/wire-server/pull/1283 + # which we parse here from brig's config + - name: parse federationDomain from brig + shell: "yq .config.setFederationDomain.{{ khan_env }} < roles/brig/vars/main.yml " + register: federationDomain + delegate_to: localhost + + - name: update /etc/hosts + lineinfile: > + dest=/etc/hosts + regexp='.*{{ api_host }}$' + line="127.0.0.1 {{ api_host }}" + state=present + when: simulation is defined and simulation == 'conversations' + + - shell: LOG_LEVEL={{ log_level }} LOG_BUFFER=128 + /opt/api-simulations/bin/api-loadtest + --api-host='{{ api_host }}' + --api-port='{{ api_port }}' + --users-file=/etc/api-simulations/users.txt + --users-federation-domain='{{ federationDomain.stdout }}' + --report-dir=/tmp/simulator + --enable-asserts + --conversations-total {{ conversations_total }} + --conversation-bots-min {{ conversation_bots_min }} + --conversation-bots-max {{ conversation_bots_max }} + --bot-messages-max {{ bot_messages_max }} + --bot-messages-min {{ bot_messages_min }} + --bot-assets-min {{ bot_assets_min }} + --bot-assets-max {{ bot_assets_max }} + --message-length-min {{ message_length_min }} + --message-length-max {{ message_length_max }} + --asset-size-min {{ asset_size_min }} + --asset-size-max {{ asset_size_max }} + {% if assets_noinline %} + --assets-noinline + {% endif %} + 2>&1 | ./run + args: + chdir: /etc/sv/simulator/log + when: simulation is defined and simulation == 'conversations' +``` diff --git a/tools/convert-to-cabal/README.md b/tools/convert-to-cabal/README.md index 52e78d1695c..1514458cb07 100644 --- a/tools/convert-to-cabal/README.md +++ b/tools/convert-to-cabal/README.md @@ -1,86 +1,13 @@ # How to convert the project to cabal -1. Run +Run - ```bash - ./tools/convert-to-cabal/generate.sh - ``` - - This will generate these files - - `cabal.project.freeze` - - `cabal.project` - -2. Create a `cabal.project.local` file with - - ``` - optimization: False - ``` - - This configures that local builds fast without optimization. - - To make sure Haskell Language Server also builds all projects without optimization run this: - - ```bash - echo "optimization: False" > ./cabal.project.local - ./hack/bin/cabal-project-local-template.sh "ghc-options: -O0" >> ./cabal.project.local - ``` - - Note: cabal v2-repl (which is run by hie-bios (HLS)) seem to be ignoring "optimization" flag for local dependencies, this is why we need to specify `ghc-options` explicitely. - - -# How to use the project with cabal - -1. Update your environment. - ```bash - cabal update - ``` - - Add this to your .envrc.local - ```bash - export WIRE_BUILD_WITH_CABAL=1 - ``` - - You should be able to build wire-server with cabal now: - - ```bash - make install # using cabal - make c package=brig # to build and install all of brig's executables - make c package=brig test=1 # also run unit tests - make ci package=brig pattern="delete" # build and run brig's integration tests - ``` - -2. For Haskell Language Server change `hie.yaml` to use cabal - ```bash - WIRE_BUILD_WITH_CABAL=1 make hie.yaml - ``` - - - -## Notes - -- `cabal v2-repl` (used by hie-bios) seem to be ignoring "optimization" flag for local dependencies. However it respects ghc-options - -``` -package foo - ghc-options: -O0 +```bash +./tools/convert-to-cabal/generate.sh ``` -- With new cabal build there doesn't seem to be any way of running tests as part of a build. You have to run the tests manually. - https://github.com/haskell/cabal/issues/7267 - -- Nix integration (`nix: True` in `~/.cabal/config`) is not supported in new-build. - https://github.com/haskell/cabal/issues/4646 - That's why you have to enter the environment defined by .envrc to use cabal. +This will generate these files +- `cabal.project.freeze` +- `cabal.project` -- cabal oddity? Specifying `--ghc-options` twice yields different result - - if run - ``` - cabal build --ghc-options "-O0" exe:brig - ``` - - and then - ``` - cabal build --ghc-options "-O0" --ghc-options "-O0" exe:brig - ``` - Cabal will retry to build brig and _all_ of its dependencies + Note: cabal v2-repl (which is run by hie-bios (HLS)) seem to be ignoring "optimization" flag for local dependencies, this is why we need to specify `ghc-options` explicitely.