retry device resync doesn't follow exponentially back off algorithm

[benvei]

Description

As far as I'm aware matrix-synapse should stop syncing to dead servers using an exponential back off algorithm.
Reference:
#4979
#5113

This actually doesn't happen if the error is "No Content-Type header received from remote server". Most likely the server owner disabled the server and left the reverse proxy online or the domain got resold and now there is another HTTP/HTTPS service on that domain.

In my case my server tries to contact it almost twice a minute and is spamming my logs. The exponential back off algorithm does not work in that case.

Logs in my case:

`2021-03-12 12:12:11,582 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1834 - {GET-O-15116} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:12:11,583 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1834 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:12:40,812 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1835 - {GET-O-15120} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:12:40,813 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1835 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:13:10,854 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1836 - {GET-O-15126} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:13:10,854 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1836 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:13:40,815 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1837 - {GET-O-15130} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:13:40,815 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1837 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:14:10,813 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1838 - {GET-O-15134} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:14:10,814 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1838 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:14:40,814 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1839 - {GET-O-15138} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:14:40,815 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1839 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:15:10,814 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1840 - {GET-O-15142} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:15:10,814 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1840 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server

Steps to reproduce

• install synapse
• find a user whose server is returning no content-type header
• check logs

Version information

• Homeserver:

• Version: 1.29.0+buster1

• Install method: Debian Package Manager

• Platform: Debian, LXC Container

[erikjohnston]

Looks like this is indeed the case, as we're only backing off it we get an error code:

synapse/synapse/util/retryutils.py

Lines 163 to 166 in aff1eb7

	elif not issubclass(exc_type, Exception):
	# avoid treating exceptions which don't derive from Exception as
	# failures; this is mostly so as not to catch defer._DefGen.
	valid_err_code = True

[richvdh]

#8917 is probably relevant here