Merge pull request #456 from eMPee584/synapse-workers

Synapse workers

CHANGELOG.md

@@ -1,3 +1,18 @@
+# 2021-02-19
+
+## Synapse workers support
+
+After [lots and lots of work](https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/456) (done over many months by [Marcel Partap](https://github.com/eMPee584), [Max Klenk](https://github.com/maxklenk), a few others from the [Technical University of Dresden, Germany](https://tu-dresden.de/) and various other contributors), support for Synapse workers has finally landed.
+
+Having support for workers makes the playbook suitable for larger homeserver deployments.
+
+Our setup is not yet perfect (we don't support all types of workers; scaling some of them (like `pusher`, `federation_sender`) beyond a single instance is not yet supported). Still, it's a great start and can already power homeservers with thousands of users, like the [Matrix deployment at TU Dresden](https://doc.matrix.tu-dresden.de/en/) discussed in [Matrix Live S06E09 - TU Dresden on their Matrix deployment](https://www.youtube.com/watch?v=UHJX2pmT2gk).
+
+By default, workers are disabled and Synapse runs as a single process (homeservers don't necessarily need the complexity and increased memory requirements of running a worker-based setup).
+
+To enable Synapse workers, follow our [Load balancing with workers](docs/configuring-playbook-synapse.md#load-balancing-with-workers) documentation.
+
+
 # 2021-02-12
 
 ## (Potential Breaking Change) Monitoring/metrics support using Prometheus and Grafana

docs/configuring-playbook-synapse.md

@@ -18,6 +18,35 @@ Alternatively, **if there is no pre-defined variable** for a Synapse setting you
 - or, if extending the configuration is still not powerful enough for your needs, you can **override the configuration completely** using `matrix_synapse_configuration` (or `matrix_synapse_configuration_yaml`). You can find information about this in [`roles/matrix-synapse/defaults/main.yml`](../roles/matrix-synapse/defaults/main.yml).
 
 
+## Load balancing with workers
+
+To have Synapse gracefully handle thousands of users, worker support should be enabled. It factors out some homeserver tasks and spreads the load of incoming client and server-to-server traffic between multiple processes. More information can be found in the [official Synapse workers documentation](https://github.com/matrix-org/synapse/blob/master/docs/workers.md).
+
+To enable Synapse worker support, update your `inventory/host_vars/matrix.DOMAIN/vars.yml` file:
+
+```yaml
+matrix_synapse_workers_enabled: true
+```
+
+We support a few configuration presets (`matrix_synapse_workers_preset: one-of-each` being the default configuration):
+- `little-federation-helper` - a very minimal worker configuration to improve federation performance
+- `one-of-each` - one worker of each supported type
+
+If you'd like more customization power, you can start with one of the presets and tweak various `matrix_synapse_workers_*_count` variables manually.
+
+If you increase worker counts too much, you may need to increase the maximum number of Postgres connections too (example):
+
+```yaml
+matrix_postgres_process_extra_arguments: [
+  "-c 'max_connections=200'"
+]
+```
+
+If you're using the default setup (the `matrix-nginx-proxy` webserver being enabled) or you're using your own `nginx` server (which imports the configuration files generated by the playbook), you're good to go. If you use some other webserver, you may need to tweak your reverse-proxy setup manually to forward traffic to the various workers.
+
+In case any problems occur, make sure to have a look at the [list of synapse issues about workers](https://github.com/matrix-org/synapse/issues?q=workers+in%3Atitle) and your `journalctl --unit 'matrix-*'`.
+
+
 ## Synapse Admin
 
 Certain Synapse administration tasks (managing users and rooms, etc.) can be performed via a web user-interace, if you install [Synapse Admin](configuring-playbook-synapse-admin.md).

group_vars/matrix_servers

@@ -18,6 +18,10 @@
 
 matrix_identity_server_url: "{{ ('https://' + matrix_server_fqn_matrix) if matrix_ma1sd_enabled else None }}"
 
+# If Synapse workers are enabled and matrix-nginx-proxy is disabled, certain APIs may not work over 'http://matrix-synapse:8008'.
+# This is because we explicitly disable them for the main Synapse process.
+matrix_homeserver_container_url: "{{ 'http://matrix-nginx-proxy:12080' if matrix_nginx_proxy_enabled else 'http://matrix-synapse:8008' }}"
+
 ######################################################################
 #
 # /matrix-base
@@ -283,7 +287,7 @@ matrix_mautrix_signal_systemd_required_services_list: |
 
 matrix_mautrix_signal_homeserver_domain: '{{ matrix_domain }}'
 
-matrix_mautrix_signal_homeserver_address: "{{ 'http://matrix-synapse:8008' if matrix_synapse_enabled else '' }}"
+matrix_mautrix_signal_homeserver_address: "{{ matrix_homeserver_container_url }}"
 
 matrix_mautrix_signal_homeserver_token: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'si.hs.token') | to_uuid }}"
 
@@ -673,7 +677,8 @@ matrix_corporal_systemd_required_services_list: |
     (['matrix-synapse.service'])
   }}
 
-matrix_corporal_matrix_homeserver_api_endpoint: "http://matrix-synapse:8008"
+# This goes to Synapse's vhost
+matrix_corporal_matrix_homeserver_api_endpoint: "{{ matrix_homeserver_container_url }}"
 
 matrix_corporal_matrix_auth_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret }}"
 
@@ -915,7 +920,7 @@ matrix_ma1sd_synapsesql_connection: //{{ matrix_synapse_database_host }}/{{ matr
 
 matrix_ma1sd_dns_overwrite_enabled: true
 matrix_ma1sd_dns_overwrite_homeserver_client_name: "{{ matrix_server_fqn_matrix }}"
-matrix_ma1sd_dns_overwrite_homeserver_client_value: "http://{{ 'matrix-corporal:41080' if matrix_corporal_enabled else 'matrix-synapse:8008' }}"
+matrix_ma1sd_dns_overwrite_homeserver_client_value: "http://{{ matrix_nginx_proxy_proxy_matrix_client_api_addr_with_container }}"
 
 # By default, we send mail through the `matrix-mailer` service.
 matrix_ma1sd_threepid_medium_email_identity_from: "{{ matrix_mailer_sender_address }}"
@@ -962,8 +967,8 @@ matrix_ma1sd_database_password: "{{ matrix_synapse_macaroon_secret_key | passwor
 # If that's not the case, you may wish to disable this and take care of proxying yourself.
 matrix_nginx_proxy_enabled: true
 
-matrix_nginx_proxy_proxy_matrix_client_api_addr_with_container: "{{ 'matrix-corporal:41080' if matrix_corporal_enabled else 'matrix-synapse:8008' }}"
-matrix_nginx_proxy_proxy_matrix_client_api_addr_sans_container: "{{ '127.0.0.1:41080' if matrix_corporal_enabled else '127.0.0.1:8008' }}"
+matrix_nginx_proxy_proxy_matrix_client_api_addr_with_container: "{{ 'matrix-corporal:41080' if matrix_corporal_enabled else 'matrix-nginx-proxy:12080' }}"
+matrix_nginx_proxy_proxy_matrix_client_api_addr_sans_container: "{{ '127.0.0.1:41080' if matrix_corporal_enabled else '127.0.0.1:12080' }}"
 matrix_nginx_proxy_proxy_matrix_client_api_client_max_body_size_mb: "{{ matrix_synapse_max_upload_size_mb }}"
 
 matrix_nginx_proxy_proxy_matrix_client_api_forwarded_location_synapse_admin_api_enabled: "{{ matrix_synapse_admin_enabled }}"
@@ -987,8 +992,12 @@ matrix_nginx_proxy_proxy_matrix_identity_api_addr_sans_container: "127.0.0.1:809
 # By default, we do TLS termination for the Matrix Federation API (port 8448) at matrix-nginx-proxy.
 # Unless this is handled there OR Synapse's federation listener port is disabled, we'll reverse-proxy.
 matrix_nginx_proxy_proxy_matrix_federation_api_enabled: "{{ matrix_synapse_federation_port_enabled and not matrix_synapse_tls_federation_listener_enabled }}"
-matrix_nginx_proxy_proxy_matrix_federation_api_addr_with_container: "matrix-synapse:8048"
-matrix_nginx_proxy_proxy_matrix_federation_api_addr_sans_container: "127.0.0.1:8048"
+matrix_nginx_proxy_proxy_matrix_federation_api_addr_with_container: "matrix-nginx-proxy:12088"
+matrix_nginx_proxy_proxy_matrix_federation_api_addr_sans_container: "127.0.0.1:12088"
+
+# Settings controlling matrix-synapse-proxy.conf
+matrix_nginx_proxy_proxy_synapse_enabled: "{{ matrix_synapse_enabled }}"
+matrix_nginx_proxy_proxy_synapse_federation_api_enabled: "{{ matrix_nginx_proxy_proxy_matrix_federation_api_enabled }}"
 
 matrix_nginx_proxy_container_federation_host_bind_port: "{{ matrix_federation_public_port }}"
 
@@ -1005,6 +1014,16 @@ matrix_nginx_proxy_proxy_matrix_user_directory_search_addr_sans_container: "{{ m
 
 matrix_nginx_proxy_self_check_validate_certificates: "{{ false if matrix_ssl_retrieval_method == 'self-signed' else true }}"
 
+matrix_nginx_proxy_synapse_presence_disabled: "{{ not matrix_synapse_use_presence }}"
+
+matrix_nginx_proxy_synapse_workers_enabled: "{{ matrix_synapse_workers_enabled }}"
+matrix_nginx_proxy_synapse_workers_list: "{{ matrix_synapse_workers_enabled_list }}"
+matrix_nginx_proxy_synapse_generic_worker_client_server_locations: "{{ matrix_synapse_workers_generic_worker_client_server_endpoints }}"
+matrix_nginx_proxy_synapse_generic_worker_federation_locations: "{{ matrix_synapse_workers_generic_worker_federation_endpoints }}"
+matrix_nginx_proxy_synapse_media_repository_locations: "{{matrix_synapse_workers_media_repository_endpoints|default([]) }}"
+matrix_nginx_proxy_synapse_user_dir_locations: "{{ matrix_synapse_workers_user_dir_endpoints|default([]) }}"
+matrix_nginx_proxy_synapse_frontend_proxy_locations: "{{ matrix_synapse_workers_frontend_proxy_endpoints|default([]) }}"
+
 matrix_nginx_proxy_systemd_wanted_services_list: |
   {{
     (['matrix-synapse.service'])
@@ -1214,6 +1233,22 @@ matrix_postgres_import_databases_to_ignore: |
 
 
 
+######################################################################
+#
+# matrix-redis
+#
+######################################################################
+
+matrix_redis_enabled: "{{ matrix_synapse_workers_enabled }}"
+
+######################################################################
+#
+# /matrix-redis
+#
+######################################################################
+
+
+
 ######################################################################
 #
 # matrix-client-element
@@ -1294,6 +1329,9 @@ matrix_synapse_container_metrics_api_host_bind_port: "{{ '127.0.0.1:9100' if (ma
 #
 # For exposing the Synapse Manhole port (plain HTTP) to the local host.
 matrix_synapse_container_manhole_api_host_bind_port: "{{ '127.0.0.1:9000' if matrix_synapse_manhole_enabled else '' }}"
+#
+# For exposing the Synapse worker (and metrics) ports to the local host.
+matrix_synapse_workers_container_host_bind_address: "{{ '127.0.0.1' if (matrix_synapse_workers_enabled and not matrix_nginx_proxy_enabled) else '' }}"
 
 matrix_synapse_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'synapse.db') | to_uuid }}"
 
@@ -1348,6 +1386,11 @@ matrix_synapse_systemd_wanted_services_list: |
     (['matrix-mailer.service'] if matrix_mailer_enabled else [])
   }}
 
+# Synapse workers (used for parallel load-scaling) need Redis for IPC.
+matrix_synapse_redis_enabled: "{{ matrix_redis_enabled }}"
+matrix_synapse_redis_host: "{{ 'matrix-redis' if matrix_redis_enabled else '' }}"
+matrix_synapse_redis_password: "{{ matrix_redis_connection_password if matrix_redis_enabled else '' }}"
+
 ######################################################################
 #
 # /matrix-synapse
@@ -1465,7 +1508,7 @@ matrix_registration_riot_instance: "{{ ('https://' + matrix_server_fqn_element)
 
 matrix_registration_shared_secret: "{{ matrix_synapse_registration_shared_secret if matrix_synapse_enabled else '' }}"
 
-matrix_registration_server_location: "{{ 'http://matrix-synapse:8008' if matrix_synapse_enabled else '' }}"
+matrix_registration_server_location: "{{ matrix_homeserver_container_url }}"
 
 matrix_registration_api_validate_certs: "{{ false if matrix_ssl_retrieval_method == 'self-signed' else true }}"
 

roles/matrix-base/defaults/main.yml

@@ -76,6 +76,11 @@ matrix_ntpd_service: "{{ 'ntpd' if ansible_os_family == 'RedHat' or ansible_dist
 
 matrix_homeserver_url: "https://{{ matrix_server_fqn_matrix }}"
 
+# Specifies where the homeserver is on the container network.
+# Where this is depends on whether there's a reverse-proxy in front of it, etc.
+# This likely gets overriden elsewhere.
+matrix_homeserver_container_url: "http://matrix-synapse:8008"
+
 matrix_identity_server_url: ~
 
 matrix_integration_manager_rest_url: ~

roles/matrix-base/templates/usr-local-bin/matrix-remove-all.j2

@@ -15,11 +15,14 @@ if [ "$sure" != "Yes, I really want to remove everything!" ]; then
 	exit 0
 else
 	echo "Stop and remove matrix services"
-	for s in $(find {{ matrix_systemd_path }}/ -name "matrix-*" -printf "%f\n"); do
-		systemctl stop $s
+
+	for s in $(find {{ matrix_systemd_path }}/ -type f -name "matrix-*" -printf "%f\n"); do
+		systemctl disable --now $s
 		rm -f {{ matrix_systemd_path }}/$s
 	done
+
 	systemctl daemon-reload
+
 	echo "Remove matrix scripts"
 	find {{ matrix_local_bin_path }}/ -name "matrix-*" -delete
 	echo "Remove unused Docker images and resources"

roles/matrix-bot-matrix-reminder-bot/defaults/main.yml

@@ -58,7 +58,7 @@ matrix_bot_matrix_reminder_bot_matrix_user_id: '@{{ matrix_bot_matrix_reminder_b
 # The password that the bot uses to authenticate.
 matrix_bot_matrix_reminder_bot_matrix_user_password: ''
 
-matrix_bot_matrix_reminder_bot_matrix_homeserver_url: 'http://matrix-synapse:8008'
+matrix_bot_matrix_reminder_bot_matrix_homeserver_url: "{{ matrix_homeserver_container_url }}"
 
 # The timezone to use when creating reminders.
 # Examples: 'Europe/London', 'Etc/UTC'

roles/matrix-bridge-appservice-irc/defaults/main.yml

@@ -14,7 +14,7 @@ matrix_appservice_irc_base_path: "{{ matrix_base_data_path }}/appservice-irc"
 matrix_appservice_irc_config_path: "{{ matrix_appservice_irc_base_path }}/config"
 matrix_appservice_irc_data_path: "{{ matrix_appservice_irc_base_path }}/data"
 
-matrix_appservice_irc_homeserver_url: 'http://matrix-synapse:8008'
+matrix_appservice_irc_homeserver_url: "{{ matrix_homeserver_container_url }}"
 matrix_appservice_irc_homeserver_media_url: 'https://{{ matrix_server_fqn_matrix }}'
 matrix_appservice_irc_homeserver_domain: '{{ matrix_domain }}'
 matrix_appservice_irc_homeserver_enablePresence: true

roles/matrix-bridge-mautrix-facebook/defaults/main.yml

@@ -16,7 +16,7 @@ matrix_mautrix_facebook_config_path: "{{ matrix_mautrix_facebook_base_path }}/co
 matrix_mautrix_facebook_data_path: "{{ matrix_mautrix_facebook_base_path }}/data"
 matrix_mautrix_facebook_docker_src_files_path: "{{ matrix_mautrix_facebook_base_path }}/docker-src"
 
-matrix_mautrix_facebook_homeserver_address: 'http://matrix-synapse:8008'
+matrix_mautrix_facebook_homeserver_address: "{{ matrix_homeserver_container_url }}"
 matrix_mautrix_facebook_homeserver_domain: '{{ matrix_domain }}'
 matrix_mautrix_facebook_appservice_address: 'http://matrix-mautrix-facebook:29319'
 

roles/matrix-bridge-mautrix-hangouts/defaults/main.yml

@@ -18,7 +18,7 @@ matrix_mautrix_hangouts_docker_src_files_path: "{{ matrix_mautrix_hangouts_base_
 
 matrix_mautrix_hangouts_public_endpoint: '/mautrix-hangouts'
 
-matrix_mautrix_hangouts_homeserver_address: 'http://matrix-synapse:8008'
+matrix_mautrix_hangouts_homeserver_address: "{{ matrix_homeserver_container_url }}"
 matrix_mautrix_hangouts_homeserver_domain: '{{ matrix_domain }}'
 matrix_mautrix_hangouts_appservice_address: 'http://matrix-mautrix-hangouts:8080'
 

roles/matrix-bridge-mautrix-telegram/defaults/main.yml

@@ -25,7 +25,7 @@ matrix_mautrix_telegram_bot_token: disabled
 # Example: /741a0483-ba17-4682-9900-30bd7269f1cc
 matrix_mautrix_telegram_public_endpoint: ''
 
-matrix_mautrix_telegram_homeserver_address: 'http://matrix-synapse:8008'
+matrix_mautrix_telegram_homeserver_address: "{{ matrix_homeserver_container_url }}"
 matrix_mautrix_telegram_homeserver_domain: '{{ matrix_domain }}'
 matrix_mautrix_telegram_appservice_address: 'http://matrix-mautrix-telegram:8080'
 matrix_mautrix_telegram_appservice_public_external: 'https://{{ matrix_server_fqn_matrix }}{{ matrix_mautrix_telegram_public_endpoint }}'

roles/matrix-bridge-mx-puppet-discord/defaults/main.yml

@@ -22,7 +22,7 @@ matrix_mx_puppet_discord_docker_src_files_path: "{{ matrix_mx_puppet_discord_bas
 
 matrix_mx_puppet_discord_appservice_port: "8432"
 
-matrix_mx_puppet_discord_homeserver_address: 'http://matrix-synapse:8008'
+matrix_mx_puppet_discord_homeserver_address: "{{ matrix_homeserver_container_url }}"
 matrix_mx_puppet_discord_homeserver_domain: '{{ matrix_domain }}'
 matrix_mx_puppet_discord_appservice_address: 'http://matrix-mx-puppet-discord:{{ matrix_mx_puppet_discord_appservice_port }}'
 

roles/matrix-bridge-mx-puppet-instagram/defaults/main.yml

@@ -16,7 +16,7 @@ matrix_mx_puppet_instagram_data_path: "{{ matrix_mx_puppet_instagram_base_path }
 matrix_mx_puppet_instagram_docker_src_files_path: "{{ matrix_mx_puppet_instagram_base_path }}/docker-src"
 
 matrix_mx_puppet_instagram_appservice_port: "8440"
-matrix_mx_puppet_instagram_homeserver_address: 'http://matrix-synapse:8008'
+matrix_mx_puppet_instagram_homeserver_address: "{{ matrix_homeserver_container_url }}"
 matrix_mx_puppet_instagram_homeserver_domain: '{{ matrix_domain }}'
 matrix_mx_puppet_instagram_appservice_address: 'http://matrix-mx-puppet-instagram:{{ matrix_mx_puppet_instagram_appservice_port }}'
 

roles/matrix-bridge-mx-puppet-skype/defaults/main.yml

@@ -17,7 +17,7 @@ matrix_mx_puppet_skype_docker_src_files_path: "{{ matrix_mx_puppet_skype_base_pa
 
 matrix_mx_puppet_skype_appservice_port: "8438"
 
-matrix_mx_puppet_skype_homeserver_address: 'http://matrix-synapse:8008'
+matrix_mx_puppet_skype_homeserver_address: "{{ matrix_homeserver_container_url }}"
 matrix_mx_puppet_skype_appservice_address: 'http://matrix-mx-puppet-skype:{{ matrix_mx_puppet_skype_appservice_port }}'
 
 # "@user:server.com" to allow specific user

roles/matrix-bridge-mx-puppet-slack/defaults/main.yml

@@ -22,7 +22,7 @@ matrix_mx_puppet_slack_docker_src_files_path: "{{ matrix_mx_puppet_slack_base_pa
 
 matrix_mx_puppet_slack_appservice_port: "8432"
 
-matrix_mx_puppet_slack_homeserver_address: 'http://matrix-synapse:8008'
+matrix_mx_puppet_slack_homeserver_address: "{{ matrix_homeserver_container_url }}"
 matrix_mx_puppet_slack_homeserver_domain: '{{ matrix_domain }}'
 matrix_mx_puppet_slack_appservice_address: 'http://matrix-mx-puppet-slack:{{ matrix_mx_puppet_slack_appservice_port }}'
 

roles/matrix-bridge-mx-puppet-steam/defaults/main.yml

@@ -22,7 +22,7 @@ matrix_mx_puppet_steam_docker_src_files_path: "{{ matrix_mx_puppet_steam_base_pa
 
 matrix_mx_puppet_steam_appservice_port: "8432"
 
-matrix_mx_puppet_steam_homeserver_address: 'http://matrix-synapse:8008'
+matrix_mx_puppet_steam_homeserver_address: "{{ matrix_homeserver_container_url }}"
 matrix_mx_puppet_steam_homeserver_domain: '{{ matrix_domain }}'
 matrix_mx_puppet_steam_appservice_address: 'http://matrix-mx-puppet-steam:{{ matrix_mx_puppet_steam_appservice_port }}'
 

roles/matrix-bridge-mx-puppet-twitter/defaults/main.yml

@@ -22,7 +22,7 @@ matrix_mx_puppet_twitter_docker_src_files_path: "{{ matrix_mx_puppet_twitter_bas
 
 matrix_mx_puppet_twitter_appservice_port: "8432"
 
-matrix_mx_puppet_twitter_homeserver_address: 'http://matrix-synapse:8008'
+matrix_mx_puppet_twitter_homeserver_address: "{{ matrix_homeserver_container_url }}"
 matrix_mx_puppet_twitter_homeserver_domain: '{{ matrix_domain }}'
 matrix_mx_puppet_twitter_appservice_address: 'http://matrix-mx-puppet-twitter:{{ matrix_mx_puppet_twitter_appservice_port }}'
 

roles/matrix-dimension/templates/config.yaml.j2

@@ -13,7 +13,7 @@ homeserver:
 
   # The URL that Dimension, go-neb, and other services provisioned by Dimension should
   # use to access the homeserver with.
-  clientServerUrl: "http://matrix-synapse:8008"
+  clientServerUrl: "{{ matrix_homeserver_container_url }}"
 
   # The URL that Dimension should use when trying to communicate with federated APIs on
   # the homeserver. If not supplied or left empty Dimension will try to resolve the address