update to 1.12 broke workers setup

[airblag]

Description

I changed my setup for using workers a few days ago. Today, after updating to 1.12 (debian repository from matrix.org), my workers are not starting anymore.

Example of the log for a synchrotron worker :

2020-03-24 00:51:40,018 - root - 257 - WARNING - None- ***** STARTING SERVER *****
2020-03-24 00:51:40,018 - root - 258 - WARNING - None- Server /opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/app/synchrotron.py version 1.12.0
2020-03-24 00:51:40,230 - synapse.app.synchrotron - 245 - WARNING - - Starting daemon.
2020-03-24 00:51:40,332 - twisted - 192 - ERROR - - Traceback (most recent call last):
2020-03-24 00:51:40,332 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/app/_base.py", line 277, in start
2020-03-24 00:51:40,332 - twisted - 192 - ERROR - -     hs.start_listening(listeners)
2020-03-24 00:51:40,333 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/app/generic_worker.py", line 575, in start_listening
2020-03-24 00:51:40,333 - twisted - 192 - ERROR - -     self.get_tcp_replication().start_replication(self)
2020-03-24 00:51:40,333 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/server.py", line 568, in _get
2020-03-24 00:51:40,333 - twisted - 192 - ERROR - -     dep = builder()
2020-03-24 00:51:40,334 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/app/generic_worker.py", line 581, in build_tcp_replication
2020-03-24 00:51:40,334 - twisted - 192 - ERROR - -     return GenericWorkerReplicationHandler(self)
2020-03-24 00:51:40,334 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/app/generic_worker.py", line 604, in __init__
2020-03-24 00:51:40,334 - twisted - 192 - ERROR - -     self.send_handler = FederationSenderHandler(hs, self)
2020-03-24 00:51:40,335 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/app/generic_worker.py", line 734, in __init__
2020-03-24 00:51:40,335 - twisted - 192 - ERROR - -     self.federation_sender = hs.get_federation_sender()
2020-03-24 00:51:40,335 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/server.py", line 568, in _get
2020-03-24 00:51:40,335 - twisted - 192 - ERROR - -     dep = builder()
2020-03-24 00:51:40,335 - twisted - 192 - ERROR - -   File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/server.py", line 445, in build_federation_sender
2020-03-24 00:51:40,336 - twisted - 192 - ERROR - -     raise Exception("Workers cannot send federation traffic")
2020-03-24 00:51:40,336 - twisted - 192 - ERROR - - Exception: Workers cannot send federation traffic
2020-03-24 00:51:40,337 - twisted - 192 - CRITICAL - - Unhandled Error
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/internet/base.py", line 1282, in run
    self.startRunning(installSignalHandlers=installSignalHandlers)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/internet/base.py", line 1262, in startRunning
    ReactorBase.startRunning(self)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/internet/base.py", line 770, in startRunning
    self.fireSystemEvent('startup')
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/internet/base.py", line 721, in fireSystemEvent
    event.fireEvent()
--- <exception caught here> ---
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/twisted/internet/base.py", line 428, in fireEvent
    result = callable(*args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.7/site-packages/synapse/app/_base.py", line 297, in start
    sys.exit(1)
builtins.SystemExit: 1

/etc/matrix-synapse/worker/synchrotron-1.yaml:

worker_app: synapse.app.synchrotron

# The replication listener on the synapse to talk to.
worker_replication_host: 127.0.0.1
worker_replication_port: 9092
worker_replication_http_port: 9093

worker_listeners:
 - type: http
   port: 8083
   resources:
     - names:
       - client

worker_daemonize: True
worker_pid_file: /var/run/synapse/synchrotron1.pid
worker_log_config: /etc/matrix-synapse/synchrotron1-log.yaml

Steps to reproduce

worked with 1.11 and stopped with the update

Version information

• Homeserver:

selfhosted

• Version:

{
"python_version": "3.7.3",
"server_version": "1.12.0"
}

• Install method:

apt, matrix.org repository

[erikjohnston]

Ah, looks like we're not correctly ignoring send_federation config option (which defaults to True) on non-federation sending workers. As a hacky fix you should be able to add send_federation: false to your worker config file

[airblag]

Thanks, the send_federation: false parameter made my day !

[anoadragon453]

As #7133 has merged, I believe this can now be closed.