Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Synchrotron stopped relaying new messages #7345

Closed
Half-Shot opened this issue Apr 25, 2020 · 15 comments
Closed

Synchrotron stopped relaying new messages #7345

Half-Shot opened this issue Apr 25, 2020 · 15 comments
Assignees
@richvdh
Labels
A-Workers Problems related to running Synapse in Worker Mode (or replication)

Comments

@Half-Shot
Copy link
Collaborator

Half-Shot commented Apr 25, 2020 •
edited
Loading

I noticed today that my messages had stopped arriving after some time, but /sync was happily running as normal (no clientside errors). I checked the synchrotron and noticed that I was hitting the long poll timeout (30s) for my user pretty much reliably, and the sync token wasn't advancing.

And to be sure, I checked that my federation workers were all working, and they seemed to be processing incoming traffic just fine. I've also checked my certificates, and they are okay too.

2020-04-24 23:36:10,512 - synapse.state - 497 - INFO - GET-109649- Resolving state for !tdtPuWhSgVBBsTHMhG:matrix.org with 2 groups
2020-04-24 23:36:10,512 - synapse.state - 520 - INFO - GET-109649- Resolving conflicted state for '!tdtPuWhSgVBBsTHMhG:matrix.org'
2020-04-24 23:36:10,513 - synapse.state.v1 - 80 - INFO - GET-109649- Asking for 190/190 conflicted events
2020-04-24 23:36:10,522 - synapse.state - 497 - INFO - GET-109649- Resolving state for !FTFmvhuoHbAAzjYtZe:matrix.org with 8 groups
2020-04-24 23:36:10,522 - synapse.state - 520 - INFO - GET-109649- Resolving conflicted state for '!FTFmvhuoHbAAzjYtZe:matrix.org'
2020-04-24 23:36:10,527 - synapse.state.v1 - 80 - INFO - GET-109649- Asking for 447/447 conflicted events
2020-04-24 23:36:10,542 - synapse.metrics - 464 - INFO - - Collecting gc 1
2020-04-24 23:36:10,586 - synapse.state.v1 - 112 - INFO - GET-109649- Asking for 3/3 auth events
2020-04-24 23:36:10,676 - synapse.state.v1 - 112 - INFO - GET-109649- Asking for 4/4 auth events
2020-04-24 23:36:11,349 - synapse.access.http.8018 - 301 - INFO - GET-109649- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 1.083sec/0.000sec (0.165sec, 0.008sec) (0.078sec/2.126sec/57) 15178B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879948_272948375_38408_18234089_321641_1391_199653_1182298_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDr
oid; MatrixAndroidSDK_X 0.0.1)" [660 dbevts]
2020-04-24 23:36:15,103 - synapse.access.http.8018 - 301 - INFO - GET-109650- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.152sec/0.000sec (0.031sec, 0.001sec) (0.036sec/0.348sec/17) 14608B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879950_272948375_38420_18234097_321641_1391_199653_1182298_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15
 Chrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:15,374 - synapse.access.http.8018 - 301 - INFO - GET-109651- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.037sec/0.000sec (0.017sec, 0.000sec) (0.001sec/0.018sec/1) 1384B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38427_18234109_321641_1391_199653_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C
hrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:16,972 - synapse.access.http.8018 - 301 - INFO - GET-109653- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.059sec/0.000sec (0.019sec, 0.000sec) (0.001sec/0.036sec/2) 2707B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879969_272948375_38425_18234106_321641_1391_199653_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroi
d; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:36:18,871 - synapse.access.http.8018 - 301 - INFO - GET-109652- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 3.378sec/0.000sec (0.011sec, 0.000sec) (0.001sec/0.038sec/2) 559B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38428_18234112_321641_1391_199653_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Ch
rome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:21,919 - synapse.access.http.8018 - 301 - INFO - GET-109654- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 2.983sec/0.000sec (0.007sec, 0.000sec) (0.000sec/0.000sec/0) 759B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38428_18234112_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Ch
rome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:22,853 - synapse.access.http.8018 - 301 - INFO - GET-109655- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.860sec/0.000sec (0.012sec, 0.000sec) (0.001sec/0.018sec/1) 848B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38429_18234112_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Ch
rome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:25,040 - synapse.access.http.8018 - 301 - INFO - GET-109656- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 2.045sec/0.000sec (0.010sec, 0.000sec) (0.000sec/0.000sec/0) 803B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38429_18234113_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Ch
rome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:27,238 - synapse.access.http.8018 - 301 - INFO - GET-109657- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 2.120sec/0.000sec (0.041sec, 0.000sec) (0.001sec/0.045sec/2) 848B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234113_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Ch
rome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:27,795 - synapse.access.http.8018 - 301 - INFO - GET-109658- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.476sec/0.000sec (0.014sec, 0.000sec) (0.000sec/0.018sec/1) 851B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234114_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Ch
rome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:28,130 - synapse.access.http.8018 - 301 - INFO - GET-109659- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.252sec/0.000sec (0.017sec, 0.000sec) (0.007sec/0.018sec/1) 853B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234115_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Ch
rome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:36:28,130 - synapse.metrics - 464 - INFO - - Collecting gc 1
2020-04-24 23:36:28,139 - synapse.metrics - 464 - INFO - - Collecting gc 2
2020-04-24 23:36:39,442 - synapse.access.http.8018 - 301 - INFO - GET-109661- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.145sec/0.000sec (0.034sec, 0.000sec) (0.001sec/0.105sec/4) 2394B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38428_18234112_321641_1391_199653_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; M
atrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:36:58,590 - synapse.access.http.8018 - 301 - INFO - GET-109660- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.013sec/0.000sec (0.014sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C
hrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:37:10,582 - synapse.access.http.8018 - 301 - INFO - GET-109662- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.010sec/0.000sec (0.008sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroi
d; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:37:10,840 - synapse.access.http.8018 - 301 - INFO - GET-109664- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.012sec/0.000sec (0.009sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; Ma
trixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:37:28,678 - synapse.access.http.8018 - 301 - INFO - GET-109663- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.008sec/0.000sec (0.006sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C
hrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:37:41,678 - synapse.access.http.8018 - 301 - INFO - GET-109665- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.008sec/0.000sec (0.005sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroi
d; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:37:41,856 - synapse.access.http.8018 - 301 - INFO - GET-109667- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.007sec/0.000sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; Ma
trixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:37:58,751 - synapse.access.http.8018 - 301 - INFO - GET-109666- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.009sec/0.000sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C
hrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:38:12,332 - synapse.access.http.8018 - 301 - INFO - GET-109668- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.009sec/0.000sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroi
d; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:38:12,458 - synapse.access.http.8018 - 301 - INFO - GET-109670- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.008sec/0.000sec (0.008sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; Ma
trixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:38:20,666 - synapse.storage.data_stores.main.event_push_actions - 499 - INFO - event_push_action_stream_orderings-221- Searching for stream ordering 1 month ago
2020-04-24 23:38:20,866 - synapse.storage.data_stores.main.event_push_actions - 504 - INFO - event_push_action_stream_orderings-221- Found stream ordering 1 month ago: it's 36262982
2020-04-24 23:38:20,867 - synapse.storage.data_stores.main.event_push_actions - 506 - INFO - event_push_action_stream_orderings-221- Searching for stream ordering 1 day ago
2020-04-24 23:38:21,025 - synapse.storage.data_stores.main.event_push_actions - 511 - INFO - event_push_action_stream_orderings-221- Found stream ordering 1 day ago: it's 36857827
2020-04-24 23:38:28,839 - synapse.access.http.8018 - 301 - INFO - GET-109669- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.012sec/0.000sec (0.008sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C
hrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:38:42,594 - synapse.access.http.8018 - 301 - INFO - GET-109671- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.008sec/0.000sec (0.005sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroi
d; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:38:42,781 - synapse.access.http.8018 - 301 - INFO - GET-109673- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.007sec/0.000sec (0.007sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; Ma
trixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:38:58,920 - synapse.access.http.8018 - 301 - INFO - GET-109672- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.013sec/0.000sec (0.008sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C
hrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:39:13,273 - synapse.access.http.8018 - 301 - INFO - GET-109674- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.009sec/0.000sec (0.008sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroi
d; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:39:13,487 - synapse.access.http.8018 - 301 - INFO - GET-109676- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.013sec/0.000sec (0.011sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; Ma
trixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:39:29,040 - synapse.access.http.8018 - 301 - INFO - GET-109675- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.016sec/0.000sec (0.013sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C
hrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:39:43,976 - synapse.access.http.8018 - 301 - INFO - GET-109677- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.008sec/0.000sec (0.003sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:39:44,237 - synapse.access.http.8018 - 301 - INFO - GET-109679- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.007sec/0.000sec (0.007sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:39:59,121 - synapse.access.http.8018 - 301 - INFO - GET-109678- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.013sec/0.000sec (0.013sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Chrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:40:14,737 - synapse.access.http.8018 - 301 - INFO - GET-109680- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.014sec/0.000sec (0.011sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:40:29,227 - synapse.access.http.8018 - 301 - INFO - GET-109681- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.014sec/0.000sec (0.010sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Chrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:40:33,459 - synapse.access.http.8018 - 301 - INFO - GET-109683- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 0.013sec/0.000sec (0.011sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=0&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:40:59,315 - synapse.access.http.8018 - 301 - INFO - GET-109682- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.010sec/0.000sec (0.005sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 Chrome/78.0.3904.130 Electron/7.1.14 Safari/537.36" [0 dbevts]
2020-04-24 23:41:03,993 - synapse.access.http.8018 - 301 - INFO - GET-109684- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.015sec/0.000sec (0.010sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=181&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "RiotX/0.19.0-dev (Linux; U; Android 10; Pixel 3a Build/QQ2A.200405.00; Flavour FDroid; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-04-24 23:41:29,497 - synapse.access.http.8018 - 301 - INFO - GET-109685- - - 8018 - {@Half-Shot:half-shot.uk} Processed request: 30.009sec/0.000sec (0.005sec, 0.000sec) (0.000sec/0.000sec/0) 374B 200 "GET /_matrix/client/r0/sync?filter=174&timeout=30000&since=s36879970_272948375_38430_18234116_321641_1391_199654_1182299_48 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.5.15 C:

It has been going like this from 23:38 last night up until now.
@Half-Shot
Copy link
Collaborator Author

Restarting the worker seemed to get it running again.

@Half-Shot
Copy link
Collaborator Author

Ah, and it quickly fell apart. The master process appears to be screaming a lot about:

2020-04-25 15:14:28,809 - synapse.http.client - 318 - INFO - as-sender-105d7bcabb777ad734c4db8675bb5896c18beb4e1ea45d5c4a3a411e0b3a9f8e-3609- Received response to PUT http://localhost:12450/transactions/367266?access_token=<redacted>: 200
2020-04-25 15:14:28,816 - synapse.replication.tcp.resource - 169 - INFO - replication_notifier-137927- Failed to handle stream events
2020-04-25 15:14:28,816 - synapse.metrics.background_process_metrics - 215 - ERROR - replication_notifier-137927- Background process 'replication_notifier' threw an exception
Traceback (most recent call last):
  File "/opt/synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: [(36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@mfaure:matrix.org', '$15449765203163589VTibv:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_htoc^:matrix.org', '$15498973977237451lRpuU:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@mike_new:mikenew.io', '$1549469192151SUawy:mikenew.io'), (36879971, '!QtykxKocfZaZOUrTwp:ma
trix.org', 'm.room.member', '@duncanturk:synapse.duncanturk.com', '$15471013481428EBVoH:synapse.duncanturk.com'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@aelequin:matrix.org', '$15628785501751584YcIrd:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@mj:matrix.ucw.cz', '$15609802944QuyZy:matrix.ucw.cz'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@ant
onov:halogen.city', '$1554751835268PiitW:halogen.city'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_yanosz:matrix.org', '$1539093345170475beZRk:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_dc_irc_bridge:matrix.org', '$15525394782757049ftBmX:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@snk:nobeta.org', '$155068652752ywF
CP:nobeta.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_ManyRaptors:matrix.org', '$15350844691475147zHalG:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Mutter:matrix.org', '$1550691114253095FNVVn:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_grin[m]:matrix.org', '$15605059211912460COSIn:matrix.org'), (368799
71, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_adru_:matrix.org', '$15352505244429584zhOZK:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_psychnosis:matrix.org', '$1536830715595391cnThL:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@midar:matrix.org', '$1568548157474580xeSJe:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 
'm.room.member', '@freenode_JonOsterman:matrix.org', '$15686366511803808kJqIO:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_sandra24xi:matrix.org', '$15734300293241205Jfoyj:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Solprefixer:matrix.org', '$1534422199123209SMpzq:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member',
 '@freenode_hernan604:matrix.org', '$1532138248298124ODYiD:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_KibaFox:matrix.org', '$1566108001467241LdsgV:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_eido1on:matrix.org', '$15344958761140681uGGit:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_graphene:matri
x.org', '$1558541094151819VxbRC:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_kalamaja_:matrix.org', '$15477765452033083isFOa:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@philip:trailingwhite.space', '$15494866193968UpZzl:trailingwhite.space'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Tobias928:matrix.org', '$15576200
355949126QEpNm:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@okapii:matrix.org', '$1551201211128322aSZRy:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@G4sD0c:matrix.org', '$1567959819445692FmmRg:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_jrl:matrix.org', '$1569107689949399LPFjv:matrix.org'), (36879971, '!QtykxKocfZa
ZOUrTwp:matrix.org', 'm.room.member', '@freenode_hilario:matrix.org', '$15713532531374805DaqUy:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_detran:matrix.org', '$1554145233336312vEaIl:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_dc_irc_bridge1:matrix.org', '$15492545341289686IfEEE:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 
'm.room.member', '@freenode_stee3:matrix.org', '$1532467670323396LFFAS:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_ks1en:matrix.org', '$15625500441686201DVVnf:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_TheBloodyMaster1:matrix.org', '$1569991165219560iORRp:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@phil
rw:matrix.rosenberg-watt.com', '$15605467771262PXbFR:matrix.rosenberg-watt.com'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_xvnvx-:matrix.org', '$15431988362961092DYzDF:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_clarjon_1:matrix.org', '$15346019861242697ILSKq:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@dylanrod:mat
rix.org', '$15456240461126230msBaS:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_tlaxkit:matrix.org', '$15400810221822899mSckR:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@simonpatapon:matrix.patapon.lol', '$15539913543qDcwy:matrix.patapon.lol'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_coumbole_:matrix.org', '$1541058276906845hFFny:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_georgie:matrix.org', '$1568427530489262EZhGh:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Whidou_:matrix.org', '$15691897691468494qKpew:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@kechpaja:matrix.org', '$1566287710370480bmjrE:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_hehehehe:matrix.org', '$1543016200866329zOqec:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@jm.strauss:matrix.org', '$1586684896564238FMSmG:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Morpheus_B:matrix.org', '$15297493881036032cVXQT:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@h90-:matrix.org', '$1536358558702889iAZYq:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_banc:matrix.org', '$15293441861636921FptLB:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@ml_mx_test:matrix.org', '$1546187749689590vpUvg:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@test.user.12381209:matrix.org', '$15426473271297140vUWDM:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Wouldnlite:matrix.org', '$1555452282355523RDYUv:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@fcoimbrar:matrix.org', '$1553765486366766bsxgz:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_djanos-:matrix.org', '$15539405272916163ydEmk:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@sixecho:matrix.org', '$157383207334247lbBPA:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@sameer.isa27:matrix.org', '$1549289269569101nfBme:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_i1nfusion:matrix.org', '$15575319025190317hqShz:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_malSet_:matrix.org', '$1569086482793534uBGYc:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@darksiders:darksiders.me', '$155697162144VhOXl:darksiders.me'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@felix:chat.matrix4me.de', '$15877713704mLoRT:chat.matrix4me.de'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_e0f:matrix.org', '$15431911342731002srBrv:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@eibhear:matrix.org', '$1552397095432828RHZJu:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Not_White_Rabbit:matrix.org', '$1536735890654282TIlad:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_jistr|mtgs:matrix.org', '$1571691219128941kaCjo:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@matroid:matrix.org', '$15573687911918058Pzeim:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Guest35:matrix.org', '$15516513584184196YITis:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_CcxWrk:matrix.org', '$15397982216810427UXule:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Vikulin:matrix.org', '$15360438862585714rleKj:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_fxrs:matrix.org', '$1534794797262796hDurT:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@avere:banter.city', '$15485076691016UauhC:banter.city'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_SoniEx2:matrix.org', '$15613568361503853LHrjP:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_rachelfi1h:matrix.org', '$155930606172089AWaha:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_evilem:matrix.org', '$15418771602244393VqNXX:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_tracyiot376:matrix.org', '$1536702779399191WlsRo:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Roybent:matrix.org', '$15420864544215478CMFWY:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_milkt:matrix.org', '$15635795131004581tOOsI:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@bjoern.schanz:matrix.org', '$155258946877500RNUvB:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_SocksPls:matrix.org', '$1539137650661188QRQPV:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_fredcy_:matrix.org', '$15615631251479604RhswL:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Sh|m0d4:matrix.org', '$15674761332336217xbGWQ:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@tristan:kretzer.it', '$15500931970TOMKd:kretzer.it'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_dabeowulf:matrix.org', '$1539142122711430kxUIi:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_zetorian:matrix.org', '$1547074379571416hkZeS:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@_x_half-shot=20=28xmpp=29=2f=23matrix=23matrix.org=40half-shot.uk:half-shot.uk', '$15724593328uHJKw:half-shot.uk'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@wtp2018:matrix.org', '$1530870816277887DMtQv:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@wuhlheide:matrix.org', '$15498960082623309fZzsf:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@benoit0816:matrix.org', '$1568888854213905dMMhw:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_lazydude[m]:matrix.org', '$15528197872685794mPayS:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@mujx:feneas.org', '$15550217676651QsrzQ:feneas.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@risc-j508:matrix.org', '$15687419811091673hBcYn:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_MorpheusB:matrix.org', '$15719022421162306hMajg:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@riegel:matrix.org', '$155529774083701xdEQj:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Notkea:matrix.org', '$15363462961872490iKoMd:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@condar:matrix.org', '$1562265305416808IglRw:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@anoboy666:matrix.org', '$155631056952733uDYYu:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@hawkowl:atleastfornow.net', '$15495418452tTgfA:atleastfornow.net'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_thunders1ore:matrix.org', '$1534422164122485rHvRm:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_crivrc:matrix.org', '$15706186421306928WYsfu:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_sparklefarkle:matrix.org', '$15359746391833656XPoBi:matrix.org')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: [(36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@mfaure:matrix.org', '$15449765203163589VTibv:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_htoc^:matrix.org', '$15498973977237451lRpuU:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@mike_new:mikenew.io', '$1549469192151SUawy:mikenew.io'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@duncanturk:synapse.duncanturk.com', '$15471013481428EBVoH:synapse.duncanturk.com'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@aelequin:matrix.org', '$15628785501751584YcIrd:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@mj:matrix.ucw.cz', '$15609802944QuyZy:matrix.ucw.cz'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@antonov:halogen.city', '$1554751835268PiitW:halogen.city'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_yanosz:matrix.org', '$1539093345170475beZRk:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_dc_irc_bridge:matrix.org', '$15525394782757049ftBmX:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@snk:nobeta.org', '$155068652752ywFCP:nobeta.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_ManyRaptors:matrix.org', '$15350844691475147zHalG:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Mutter:matrix.org', '$1550691114253095FNVVn:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_grin[m]:matrix.org', '$15605059211912460COSIn:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_adru_:matrix.org', '$15352505244429584zhOZK:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_psychnosis:matrix.org', '$1536830715595391cnThL:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@midar:matrix.org', '$1568548157474580xeSJe:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_JonOsterman:matrix.org', '$15686366511803808kJqIO:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_sandra24xi:matrix.org', '$15734300293241205Jfoyj:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Solprefixer:matrix.org', '$1534422199123209SMpzq:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_hernan604:matrix.org', '$1532138248298124ODYiD:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_KibaFox:matrix.org', '$1566108001467241LdsgV:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_eido1on:matrix.org', '$15344958761140681uGGit:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_graphene:matrix.org', '$1558541094151819VxbRC:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_kalamaja_:matrix.org', '$15477765452033083isFOa:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@philip:trailingwhite.space', '$15494866193968UpZzl:trailingwhite.space'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Tobias928:matrix.org', '$15576200355949126QEpNm:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@okapii:matrix.org', '$1551201211128322aSZRy:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@G4sD0c:matrix.org', '$1567959819445692FmmRg:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_jrl:matrix.org', '$1569107689949399LPFjv:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_hilario:matrix.org', '$15713532531374805DaqUy:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_detran:matrix.org', '$1554145233336312vEaIl:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_dc_irc_bridge1:matrix.org', '$15492545341289686IfEEE:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_stee3:matrix.org', '$1532467670323396LFFAS:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_ks1en:matrix.org', '$15625500441686201DVVnf:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_TheBloodyMaster1:matrix.org', '$1569991165219560iORRp:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@philrw:matrix.rosenberg-watt.com', '$15605467771262PXbFR:matrix.rosenberg-watt.com'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_xvnvx-:matrix.org', '$15431988362961092DYzDF:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_clarjon_1:matrix.org', '$15346019861242697ILSKq:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@dylanrod:matrix.org', '$15456240461126230msBaS:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_tlaxkit:matrix.org', '$15400810221822899mSckR:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@simonpatapon:matrix.patapon.lol', '$15539913543qDcwy:matrix.patapon.lol'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_coumbole_:matrix.org', '$1541058276906845hFFny:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_georgie:matrix.org', '$1568427530489262EZhGh:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Whidou_:matrix.org', '$15691897691468494qKpew:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@kechpaja:matrix.org', '$1566287710370480bmjrE:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_hehehehe:matrix.org', '$1543016200866329zOqec:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@jm.strauss:matrix.org', '$1586684896564238FMSmG:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Morpheus_B:matrix.org', '$15297493881036032cVXQT:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@h90-:matrix.org', '$1536358558702889iAZYq:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_banc:matrix.org', '$15293441861636921FptLB:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@ml_mx_test:matrix.org', '$1546187749689590vpUvg:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@test.user.12381209:matrix.org', '$15426473271297140vUWDM:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Wouldnlite:matrix.org', '$1555452282355523RDYUv:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@fcoimbrar:matrix.org', '$1553765486366766bsxgz:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_djanos-:matrix.org', '$15539405272916163ydEmk:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@sixecho:matrix.org', '$157383207334247lbBPA:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@sameer.isa27:matrix.org', '$1549289269569101nfBme:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_i1nfusion:matrix.org', '$15575319025190317hqShz:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_malSet_:matrix.org', '$1569086482793534uBGYc:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@darksiders:darksiders.me', '$155697162144VhOXl:darksiders.me'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@felix:chat.matrix4me.de', '$15877713704mLoRT:chat.matrix4me.de'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_e0f:matrix.org', '$15431911342731002srBrv:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@eibhear:matrix.org', '$1552397095432828RHZJu:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Not_White_Rabbit:matrix.org', '$1536735890654282TIlad:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_jistr|mtgs:matrix.org', '$1571691219128941kaCjo:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@matroid:matrix.org', '$15573687911918058Pzeim:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Guest35:matrix.org', '$15516513584184196YITis:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_CcxWrk:matrix.org', '$15397982216810427UXule:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Vikulin:matrix.org', '$15360438862585714rleKj:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_fxrs:matrix.org', '$1534794797262796hDurT:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@avere:banter.city', '$15485076691016UauhC:banter.city'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_SoniEx2:matrix.org', '$15613568361503853LHrjP:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_rachelfi1h:matrix.org', '$155930606172089AWaha:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_evilem:matrix.org', '$15418771602244393VqNXX:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_tracyiot376:matrix.org', '$1536702779399191WlsRo:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Roybent:matrix.org', '$15420864544215478CMFWY:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_milkt:matrix.org', '$15635795131004581tOOsI:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@bjoern.schanz:matrix.org', '$155258946877500RNUvB:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_SocksPls:matrix.org', '$1539137650661188QRQPV:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_fredcy_:matrix.org', '$15615631251479604RhswL:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Sh|m0d4:matrix.org', '$15674761332336217xbGWQ:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@tristan:kretzer.it', '$15500931970TOMKd:kretzer.it'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_dabeowulf:matrix.org', '$1539142122711430kxUIi:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_zetorian:matrix.org', '$1547074379571416hkZeS:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@_x_half-shot=20=28xmpp=29=2f=23matrix=23matrix.org=40half-shot.uk:half-shot.uk', '$15724593328uHJKw:half-shot.uk'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@wtp2018:matrix.org', '$1530870816277887DMtQv:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@wuhlheide:matrix.org', '$15498960082623309fZzsf:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@benoit0816:matrix.org', '$1568888854213905dMMhw:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_lazydude[m]:matrix.org', '$15528197872685794mPayS:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@mujx:feneas.org', '$15550217676651QsrzQ:feneas.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@risc-j508:matrix.org', '$15687419811091673hBcYn:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_MorpheusB:matrix.org', '$15719022421162306hMajg:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@riegel:matrix.org', '$155529774083701xdEQj:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_Notkea:matrix.org', '$15363462961872490iKoMd:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@condar:matrix.org', '$1562265305416808IglRw:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@anoboy666:matrix.org', '$155631056952733uDYYu:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@hawkowl:atleastfornow.net', '$15495418452tTgfA:atleastfornow.net'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_thunders1ore:matrix.org', '$1534422164122485rHvRm:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_crivrc:matrix.org', '$15706186421306928WYsfu:matrix.org'), (36879971, '!QtykxKocfZaZOUrTwp:matrix.org', 'm.room.member', '@freenode_sparklefarkle:matrix.org', '$15359746391833656XPoBi:matrix.org')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/synapse/lib/python3.7/site-packages/synapse/metrics/background_process_metrics.py", line 213, in run
    return (yield result)
  File "/opt/synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/resource.py", line 166, in _run_notifier_loop
    updates, current_token, limited = await stream.get_updates()
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/streams/_base.py", line 134, in get_updates
    self.last_token, current_token
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/streams/_base.py", line 159, in get_updates_since
    from_token, upto_token, limit,
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/streams/_base.py", line 178, in update_function
    assert len(updates) <= limit
AssertionError
2020-04-25 15:14:28,858 - synapse.push.httppusher - 195 - INFO - httppush.process-2284- Processing 1 unprocessed push actions for @Half-Shot:half-shot.uk/im.vector.app.android/cw24so2GjJs:APA91bGnJYSh8nHSoHV28c-IHKZolYqc76SCbFkaDzgVYt0n2Li2IwQNqrMDC0i4ssX6_AdGnmxyd-ZQE_J-tErjwi8K1zrZ__C71U9b9lVQgrpZ-Win2gGrZCPoprLat1mPATyPfVX2 starting at stream_ordering 36888411

@babolivier
Copy link
Contributor

This last error seems to be #7308, which should have been fixed by #7337 although apparently other AssertionErrors can still happen despite that :/

@Half-Shot
Copy link
Collaborator Author

Oh, it's worth pointing out that I might not have been quite up to date...I'll try forcing myself to latest develop and see if that helps.

@Half-Shot
Copy link
Collaborator Author

it did not

@eMPee584 eMPee584 mentioned this issue Apr 25, 2020
Merged
@richvdh
Copy link
Member

richvdh commented Apr 25, 2020

Yup there are still bugs here as per #7308. Half-Shot can you share a stacktrace with latest develop?

@Half-Shot
Copy link
Collaborator Author 

Traceback (most recent call last):
  File "/opt/synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: [(36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_Pempri=2fdisroot=40chat.disroot.org:bau-ha.us', '$1572513494483ARvOY:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_newbie=2fdisroot=40chat.disroot.org:bau-ha.us', '$15797231134407DbccO:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_michelc:matrix.org', '$15774803232684446OXYIq:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_33kk=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581691057869qwfyT:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_erbzn=20=2fdisroot=40chat.disroot.org:bau-ha.us', '$15694168238668ejgDi:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_golanv=2fdisroot=40chat.disroot.org:bau-ha.us', '$1572527774846TekpE:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_lukas=2fdisroot=40chat.disroot.org:bau-ha.us', '$15798981442818TWtxY:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_Neurognostic_:matrix.org', '$15816546711199735iVoNs:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_domayor7=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581685121744fFtRX:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_AngryArt=2fdisroot=40chat.disroot.org:bau-ha.us', '$1572540843397QyBuz:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_wikwg9=2fdisroot=40chat.disroot.org:bau-ha.us', '$15715141501877vSqWP:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_baldeagle=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581578706889GznWF:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_Manson=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581683033702fSjRq:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_sintrenton=2fdisroot=40chat.disroot.org:bau-ha.us', '$1570022688514Mzcny:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_willpate=2fdisroot=40chat.disroot.org:bau-ha.us', '$1572532753269afhHl:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_bonber=2fdisroot=40chat.disroot.org:bau-ha.us', '$1570024587559htYII:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@[email protected]:disroot.org', '$148320022659tPVvI:disroot.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@ako:disroot.org', '$14828968831161RIEmH:disroot.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_coldbuddy=2fdisroot=40chat.disroot.org:bau-ha.us', '$15797292574589ldplO:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_cc__:matrix.org', '$1581477965334072nLqHq:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_outloudvi[m]:matrix.org', '$1568414692354648ZjaxW:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_icMikaela:matrix.org', '$15527884382355807cglWS:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_hekanju=2fdisroot=40chat.disroot.org:bau-ha.us', '$15817564342148nRfWM:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@Ciblia:matrix.org', '$14875843601338755aoxIF:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_tulsi=2fdisroot=40chat.disroot.org:bau-ha.us', '$15597705089011kdknj:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_expresionvocaloid=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581552021240JdAlH:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@[email protected]:disroot.org', '$14817577361HHxIb:disroot.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_cc_:matrix.org', '$15795036653073557bazgf:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ozwald1=2fdisroot=40chat.disroot.org:bau-ha.us', '$157848178218806BPZSC:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_basedude:matrix.org', '$1581981956206514NWSuC:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_jh=2fdisroot=40chat.disroot.org:bau-ha.us', '$15701727904496PzlpJ:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_userdebug=2fdisroot=40chat.disroot.org:bau-ha.us', '$156011843485naJvu:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_maryjane=2fdisroot=40chat.disroot.org:bau-ha.us', '$156215398215wRLpH:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_muntashir=2fdisroot=40chat.disroot.org:bau-ha.us', '$15736709891108iMkGh:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_oxo=2fdisroot=40chat.disroot.org:bau-ha.us', '$157776490011238ndfKx:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_mars=2fdisroot=40chat.disroot.org:bau-ha.us', '$15610171791521lDEws:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_anam_:matrix.org', '$15794610782848027gKObS:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_awal=2fdisroot=40chat.disroot.org:bau-ha.us', '$15619195565420gtHuw:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_clackety:matrix.org', '$1581981967206617EIcHL:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_coffeebird=2fdisroot=40chat.disroot.org:bau-ha.us', '$15799600881294bqUpd:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_learningc:matrix.org', '$1581982036207290rMdAj:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_amam5=2fdisroot=40chat.disroot.org:bau-ha.us', '$15819078625064wUKbr:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_techbolt:matrix.org', '$1581982171208302UeyVR:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_nly:matrix.org', '$1581982073207557qvsLO:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_linuxturtle=2fdisroot=40chat.disroot.org:bau-ha.us', '$15788870471004NioHg:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_anon003=2fdisroot=40chat.disroot.org:bau-ha.us', '$15649291498697huXxU:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_Nuts_:matrix.org', '$1581981924206165BasRn:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ironmonkey=2fdisroot=40chat.disroot.org:bau-ha.us', '$15596783686136AMFRx:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_diego.m51=2fdisroot=40chat.disroot.org:bau-ha.us', '$15764786081764JPrLN:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_toktr=2fdisroot=40chat.disroot.org:bau-ha.us', '$15787462713806rdUEJ:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_anam1:matrix.org', '$15856066292028898SqoHp:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_intuxikated[m]:matrix.org', '$1581982011207014mcTRp:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_crayon=2fdisroot=40chat.disroot.org:bau-ha.us', '$15748640221597PeIWX:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_freelancehero=2fdisroot=40chat.disroot.org:bau-ha.us', '$15766942317764sfedM:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_steve-_-:matrix.org', '$1581982168208278hkrXe:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_gmar=2fdisroot=40chat.disroot.org:bau-ha.us', '$15648527237255EdYZr:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_shahab:matrix.org', '$1581982151208154eNyoR:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_dingen:matrix.org', '$1581981981206735ZuDaN:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_coffeecat=2fdisroot=40chat.disroot.org:bau-ha.us', '$1559930340460LzsPu:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_massimiliano:matrix.org', '$1581982050207399jPLNK:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_draxaris=2fdisroot=40chat.disroot.org:bau-ha.us', '$15792705211025ggfbc:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_insanity=2fdisroot=40chat.disroot.org:bau-ha.us', '$15764764951668zsoor:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_vigilian=2fdisroot=40chat.disroot.org:bau-ha.us', '$15818530323802whQfB:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_ajeremias:matrix.org', '$1581981945206392RveoE:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_azhar=2fdisroot=40chat.disroot.org:bau-ha.us', '$1570033096739sEVkN:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_skrzyp1:matrix.org', '$15856066262028868MwMuk:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_perflyst:matrix.org', '$1581982088207679SVFJp:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_bou_:matrix.org', '$1581981962206567sHRuC:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_naimishavasi=2fdisroot=40chat.disroot.org:bau-ha.us', '$15597542318481auPjG:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_=5bAlien=5d=2fdisroot=40chat.disroot.org:bau-ha.us', '$15827418482574PilQq:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_steve=2fdisroot=40chat.disroot.org:bau-ha.us', '$155989704923411BIxaS:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_gongzisun=2fdisroot=40chat.disroot.org:bau-ha.us', '$15754102044760MCAZn:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_lurch:matrix.org', '$1581982042207347taXPq:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ricardog08=2fdisroot=40chat.disroot.org:bau-ha.us', '$1576731958466PFaJB:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_jvuitton=2fdisroot=40chat.disroot.org:bau-ha.us', '$15799569861208HKMYK:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_oier=2fdisroot=40chat.disroot.org:bau-ha.us', '$15797935136359PBfdS:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_atilano=2fdisroot=40chat.disroot.org:bau-ha.us', '$15799653941474xgemx:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_perflyst[x]:matrix.org', '$1581982090207694hnQPR:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_golanv[m]:matrix.org', '$1581981996206881vYiiJ:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_sum_one=2fdisroot=40chat.disroot.org:bau-ha.us', '$1572541412433TJxHz:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_mitfree[m]:matrix.org', '$1581982062207478jWarf:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_antilopa:matrix.org', '$15792581791680716dYvHn:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_andy=2fdisroot=40chat.disroot.org:bau-ha.us', '$15649236058575nbson:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_Sasha1=2fdisroot=40chat.disroot.org:bau-ha.us', '$15764646841487DrxLQ:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_TheAceOfHearts:matrix.org', '$1581981933206280UVbIt:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_tlaxkit:matrix.org', '$1568414704354840RNUAF:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_salonitanicus=2fdisroot=40chat.disroot.org:bau-ha.us', '$15793799921860JssEl:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_strypey=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581669173452EWNlB:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_andresflamenco=2fdisroot=40chat.disroot.org:bau-ha.us', '$15818221063313CfDpM:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_adn=2fdisroot=40chat.disroot.org:bau-ha.us', '$15747491651878rTzes:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_meaz=2fdisroot=40chat.disroot.org:bau-ha.us', '$15638995173197JHscv:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_GyroW_:matrix.org', '$15856066302028921zeAAc:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ignacio=2fdisroot=40chat.disroot.org:bau-ha.us', '$1576159219813QmNwQ:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_mikelpz_:matrix.org', '$15856066302028928hBcOk:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ryzendeath=2fdisroot=40chat.disroot.org:bau-ha.us', '$15596287384626aTuJH:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_pinguingitkeyz=2fdisroot=40chat.disroot.org:bau-ha.us', '$15763980482945xJCAP:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_masnavan=2fdisroot=40chat.disroot.org:bau-ha.us', '$15647598165568WtmKA:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_aerro=2fdisroot=40chat.disroot.org:bau-ha.us', '$15817812352571RnloL:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_pempri=2fdisroot=40chat.disroot.org:bau-ha.us', '$15764762471660JALHF:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_oscar=2fdisroot=40chat.disroot.org:bau-ha.us', '$15648337076842BqDjP:bau-ha.us')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: [(36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_Pempri=2fdisroot=40chat.disroot.org:bau-ha.us', '$1572513494483ARvOY:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_newbie=2fdisroot=40chat.disroot.org:bau-ha.us', '$15797231134407DbccO:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_michelc:matrix.org', '$15774803232684446OXYIq:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_33kk=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581691057869qwfyT:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_erbzn=20=2fdisroot=40chat.disroot.org:bau-ha.us', '$15694168238668ejgDi:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_golanv=2fdisroot=40chat.disroot.org:bau-ha.us', '$1572527774846TekpE:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_lukas=2fdisroot=40chat.disroot.org:bau-ha.us', '$15798981442818TWtxY:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_Neurognostic_:matrix.org', '$15816546711199735iVoNs:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_domayor7=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581685121744fFtRX:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_AngryArt=2fdisroot=40chat.disroot.org:bau-ha.us', '$1572540843397QyBuz:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_wikwg9=2fdisroot=40chat.disroot.org:bau-ha.us', '$15715141501877vSqWP:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_baldeagle=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581578706889GznWF:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_Manson=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581683033702fSjRq:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_sintrenton=2fdisroot=40chat.disroot.org:bau-ha.us', '$1570022688514Mzcny:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_willpate=2fdisroot=40chat.disroot.org:bau-ha.us', '$1572532753269afhHl:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_bonber=2fdisroot=40chat.disroot.org:bau-ha.us', '$1570024587559htYII:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@[email protected]:disroot.org', '$148320022659tPVvI:disroot.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@ako:disroot.org', '$14828968831161RIEmH:disroot.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_coldbuddy=2fdisroot=40chat.disroot.org:bau-ha.us', '$15797292574589ldplO:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_cc__:matrix.org', '$1581477965334072nLqHq:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_outloudvi[m]:matrix.org', '$1568414692354648ZjaxW:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_icMikaela:matrix.org', '$15527884382355807cglWS:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_hekanju=2fdisroot=40chat.disroot.org:bau-ha.us', '$15817564342148nRfWM:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@Ciblia:matrix.org', '$14875843601338755aoxIF:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_tulsi=2fdisroot=40chat.disroot.org:bau-ha.us', '$15597705089011kdknj:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_expresionvocaloid=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581552021240JdAlH:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@[email protected]:disroot.org', '$14817577361HHxIb:disroot.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_cc_:matrix.org', '$15795036653073557bazgf:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ozwald1=2fdisroot=40chat.disroot.org:bau-ha.us', '$157848178218806BPZSC:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_basedude:matrix.org', '$1581981956206514NWSuC:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_jh=2fdisroot=40chat.disroot.org:bau-ha.us', '$15701727904496PzlpJ:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_userdebug=2fdisroot=40chat.disroot.org:bau-ha.us', '$156011843485naJvu:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_maryjane=2fdisroot=40chat.disroot.org:bau-ha.us', '$156215398215wRLpH:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_muntashir=2fdisroot=40chat.disroot.org:bau-ha.us', '$15736709891108iMkGh:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_oxo=2fdisroot=40chat.disroot.org:bau-ha.us', '$157776490011238ndfKx:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_mars=2fdisroot=40chat.disroot.org:bau-ha.us', '$15610171791521lDEws:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_anam_:matrix.org', '$15794610782848027gKObS:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_awal=2fdisroot=40chat.disroot.org:bau-ha.us', '$15619195565420gtHuw:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_clackety:matrix.org', '$1581981967206617EIcHL:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_coffeebird=2fdisroot=40chat.disroot.org:bau-ha.us', '$15799600881294bqUpd:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_learningc:matrix.org', '$1581982036207290rMdAj:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_amam5=2fdisroot=40chat.disroot.org:bau-ha.us', '$15819078625064wUKbr:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_techbolt:matrix.org', '$1581982171208302UeyVR:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_nly:matrix.org', '$1581982073207557qvsLO:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_linuxturtle=2fdisroot=40chat.disroot.org:bau-ha.us', '$15788870471004NioHg:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_anon003=2fdisroot=40chat.disroot.org:bau-ha.us', '$15649291498697huXxU:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_Nuts_:matrix.org', '$1581981924206165BasRn:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ironmonkey=2fdisroot=40chat.disroot.org:bau-ha.us', '$15596783686136AMFRx:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_diego.m51=2fdisroot=40chat.disroot.org:bau-ha.us', '$15764786081764JPrLN:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_toktr=2fdisroot=40chat.disroot.org:bau-ha.us', '$15787462713806rdUEJ:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_anam1:matrix.org', '$15856066292028898SqoHp:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_intuxikated[m]:matrix.org', '$1581982011207014mcTRp:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_crayon=2fdisroot=40chat.disroot.org:bau-ha.us', '$15748640221597PeIWX:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_freelancehero=2fdisroot=40chat.disroot.org:bau-ha.us', '$15766942317764sfedM:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_steve-_-:matrix.org', '$1581982168208278hkrXe:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_gmar=2fdisroot=40chat.disroot.org:bau-ha.us', '$15648527237255EdYZr:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_shahab:matrix.org', '$1581982151208154eNyoR:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_dingen:matrix.org', '$1581981981206735ZuDaN:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_coffeecat=2fdisroot=40chat.disroot.org:bau-ha.us', '$1559930340460LzsPu:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_massimiliano:matrix.org', '$1581982050207399jPLNK:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_draxaris=2fdisroot=40chat.disroot.org:bau-ha.us', '$15792705211025ggfbc:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_insanity=2fdisroot=40chat.disroot.org:bau-ha.us', '$15764764951668zsoor:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_vigilian=2fdisroot=40chat.disroot.org:bau-ha.us', '$15818530323802whQfB:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_ajeremias:matrix.org', '$1581981945206392RveoE:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_azhar=2fdisroot=40chat.disroot.org:bau-ha.us', '$1570033096739sEVkN:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_skrzyp1:matrix.org', '$15856066262028868MwMuk:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_perflyst:matrix.org', '$1581982088207679SVFJp:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_bou_:matrix.org', '$1581981962206567sHRuC:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_naimishavasi=2fdisroot=40chat.disroot.org:bau-ha.us', '$15597542318481auPjG:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_=5bAlien=5d=2fdisroot=40chat.disroot.org:bau-ha.us', '$15827418482574PilQq:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_steve=2fdisroot=40chat.disroot.org:bau-ha.us', '$155989704923411BIxaS:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_gongzisun=2fdisroot=40chat.disroot.org:bau-ha.us', '$15754102044760MCAZn:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_lurch:matrix.org', '$1581982042207347taXPq:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ricardog08=2fdisroot=40chat.disroot.org:bau-ha.us', '$1576731958466PFaJB:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_jvuitton=2fdisroot=40chat.disroot.org:bau-ha.us', '$15799569861208HKMYK:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_oier=2fdisroot=40chat.disroot.org:bau-ha.us', '$15797935136359PBfdS:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_atilano=2fdisroot=40chat.disroot.org:bau-ha.us', '$15799653941474xgemx:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_perflyst[x]:matrix.org', '$1581982090207694hnQPR:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_golanv[m]:matrix.org', '$1581981996206881vYiiJ:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_sum_one=2fdisroot=40chat.disroot.org:bau-ha.us', '$1572541412433TJxHz:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_mitfree[m]:matrix.org', '$1581982062207478jWarf:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_antilopa:matrix.org', '$15792581791680716dYvHn:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_andy=2fdisroot=40chat.disroot.org:bau-ha.us', '$15649236058575nbson:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_Sasha1=2fdisroot=40chat.disroot.org:bau-ha.us', '$15764646841487DrxLQ:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_TheAceOfHearts:matrix.org', '$1581981933206280UVbIt:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_tlaxkit:matrix.org', '$1568414704354840RNUAF:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_salonitanicus=2fdisroot=40chat.disroot.org:bau-ha.us', '$15793799921860JssEl:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_strypey=2fdisroot=40chat.disroot.org:bau-ha.us', '$1581669173452EWNlB:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_andresflamenco=2fdisroot=40chat.disroot.org:bau-ha.us', '$15818221063313CfDpM:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_adn=2fdisroot=40chat.disroot.org:bau-ha.us', '$15747491651878rTzes:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_meaz=2fdisroot=40chat.disroot.org:bau-ha.us', '$15638995173197JHscv:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_GyroW_:matrix.org', '$15856066302028921zeAAc:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ignacio=2fdisroot=40chat.disroot.org:bau-ha.us', '$1576159219813QmNwQ:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_mikelpz_:matrix.org', '$15856066302028928hBcOk:matrix.org'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ryzendeath=2fdisroot=40chat.disroot.org:bau-ha.us', '$15596287384626aTuJH:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_pinguingitkeyz=2fdisroot=40chat.disroot.org:bau-ha.us', '$15763980482945xJCAP:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_masnavan=2fdisroot=40chat.disroot.org:bau-ha.us', '$15647598165568WtmKA:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_aerro=2fdisroot=40chat.disroot.org:bau-ha.us', '$15817812352571RnloL:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_pempri=2fdisroot=40chat.disroot.org:bau-ha.us', '$15764762471660JALHF:bau-ha.us'), (36889546, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_oscar=2fdisroot=40chat.disroot.org:bau-ha.us', '$15648337076842BqDjP:bau-ha.us')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/synapse/lib/python3.7/site-packages/synapse/metrics/background_process_metrics.py", line 213, in run
    return (yield result)
  File "/opt/synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/resource.py", line 166, in _run_notifier_loop
    updates, current_token, limited = await stream.get_updates()
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/streams/_base.py", line 134, in get_updates
    self.last_token, current_token
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/streams/_base.py", line 159, in get_updates_since
    from_token, upto_token, limit,
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/streams/_base.py", line 178, in update_function
    assert len(updates) <= limit
AssertionError
2020-04-25 18:10:14,980 - synapse.access.http.9092 - 301 - INFO - GET-9613- ::ffff:127.0.0.1 - 9092 - {None} Processed request: 0.053sec/-0.000sec (0.051sec, 0.000sec) (0.000sec/0.000sec/0) 783423B 200 "GET /_synapse/metrics/master HTTP/1.0" "Prometheus/2.17.1" [0 dbevts]
2020-04-25 18:10:15,242 - synapse.logging.context - 355 - WARNING - renew_attestations-4- Re-starting finished log context renew_attestations-4
2020-04-25 18:10:15,244 - synapse.http.matrixfederationclient - 408 - INFO - renew_attestations-4- {POST-O-432} [krtdex.com] Sending request: POST matrix://krtdex.com/_matrix/federation/v1/groups/%2Bcops%3Ahalf-shot.uk/renew_attestation/%40seequ%3Akrtdex.com; timeout 60.000000s
2020-04-25 18:10:15,246 - synapse.http.federation.matrix_federation_agent - 242 - INFO - renew_attestations-4- Connecting to m.krtdex.com:8448
2020-04-25 18:10:15,949 - synapse.replication.http.federation - 160 - INFO - POST-9614- Got 'm.receipt' edu from matrix.org
2020-04-25 18:10:16,112 - synapse.replication.http.federation - 116 - INFO - POST-9615- Got 1 events from federation
2020-04-25 18:10:16,194 - synapse.access.http.9593 - 301 - INFO - POST-9614- 127.0.0.1 - 9593 - {None} Processed request: 0.245sec/0.000sec (0.004sec, 0.000sec) (0.005sec/0.235sec/5) 4B 200 "POST /_synapse/replication/fed_send_edu/m.receipt/kteZMhPYbI HTTP/1.1" "Synapse/1.12.4rc1" [0 dbevts]
2020-04-25 18:10:16,214 - synapse.logging.context - 355 - WARNING - renew_attestations-4- Re-starting finished log context renew_attestations-4
2020-04-25 18:10:16,215 - synapse.http.matrixfederationclient - 408 - INFO - renew_attestations-4- {POST-O-431} [krtdex.com] Sending request: POST matrix://krtdex.com/_matrix/federation/v1/groups/%2Bteam-scrollup%3Ahalf-shot.uk/renew_attestation/%40seequ%3Akrtdex.com; timeout 60.000000s
2020-04-25 18:10:16,217 - synapse.http.federation.matrix_federation_agent - 242 - INFO - renew_attestations-4- Connecting to m.krtdex.com:8448
2020-04-25 18:10:16,231 - synapse.replication.tcp.resource - 169 - INFO - replication_notifier-5473- Failed to handle stream events
2020-04-25 18:10:16,232 - synapse.metrics.background_process_metrics - 215 - ERROR - replication_notifier-5473- Background process 'replication_notifier' threw an exception
...etc etc...

@richvdh
Copy link
Member

richvdh commented Apr 27, 2020

sorry @Half-Shot can you include the ERROR line above the traceback?

@richvdh
Copy link
Member

richvdh commented Apr 27, 2020

(this looks closely related to #7308 but a slightly different code path)

@Half-Shot
Copy link
Collaborator Author

@richvdh my logs have since rotated but the traceback was cased by the error below it (it was looping the same error and traceback in the logs over and over).

I can try to run develop again and capture fresh logs, if useful.

@richvdh
Copy link
Member

richvdh commented Apr 27, 2020

ah don't worry. Will have a stare at the code and try and track it down.

@richvdh
Copy link
Member

richvdh commented Apr 27, 2020

I'm struggling to figure out which stream could be throwing that assertion. I'll add some better logging and then get you to try again.

@richvdh
Copy link
Member

richvdh commented Apr 27, 2020

Oh wait, it says:

2020-04-25 18:10:16,231 - synapse.replication.tcp.resource - 169 - INFO - replication_notifier-5473- Failed to handle stream events

but then... that makes no sense, because as of #7337, the events stream doesn't go via tcp/streams/_base.py:178. @Half-Shot : are you sure that stack trace came from develop?

@richvdh richvdh self-assigned this Apr 27, 2020
@Half-Shot
Copy link
Collaborator Author 

2020-04-28 17:15:02,264 - synapse.access.http.9593 - 301 - INFO - POST-6164- 127.0.0.1 - 9593 - {None} Processed request: 0.258sec/0.000sec (0.004sec, 0.001sec) (0.003sec/0.250sec/5) 4B 200 "POST /_synapse/replication/fed_send_edu/m.receipt/pApQEXzcZG HTTP/1.1" "Synapse/1.12.4" [0 dbevts]
2020-04-28 17:15:02,268 - synapse.replication.http.send_event - 119 - INFO - POST-6166- Got event to send with ID: $7EhWbiwsfRAJnGlaZ-BUyGkX-ueNNAghtQbMDmqQQ1c into room: !TdAwENXmXuMrCrFEFX:maunium.net
2020-04-28 17:15:02,289 - synapse.replication.tcp.resource - 166 - INFO - replication_notifier-3139- Failed to handle stream events
2020-04-28 17:15:02,289 - synapse.metrics.background_process_metrics - 215 - ERROR - replication_notifier-3139- Background process 'replication_notifier' threw an exception
Traceback (most recent call last):
  File "/opt/synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: [(36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@lucio776:matrix.org', '$1557941942501905Sfzou:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@theloudspeaker:matrix.org', '$1577126616423568yPobd:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@babel:matrix.org', '$155887605030618Idvhk:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_mdjanic:matrix.org', '$15787689151746958uFEpF:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_traumsch1le:matrix.org', '$15817340261773906pCrKc:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_joffamac=2fdisroot=40chat.disroot.org:bau-ha.us', '$156832202421036GaODX:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_soila=2fdisroot=40chat.disroot.org:bau-ha.us', '$15610770572287kNVpI:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_userdebug=2fdisroot=40chat.disroot.org:bau-ha.us', '$156011843485naJvu:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_outloudvi[m]:matrix.org', '$15527884852356229JIVyi:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_maryjane=2fdisroot=40chat.disroot.org:bau-ha.us', '$15708084934410ATnIV:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_adho=2fdisroot=40chat.disroot.org:bau-ha.us', '$15757335581994YrcXV:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_uggo=2fdisroot=40chat.disroot.org:bau-ha.us', '$15623700031264ubARR:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_pwr=2fdisroot=40chat.disroot.org:bau-ha.us', '$15620757306854vDIBY:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_awal=2fdisroot=40chat.disroot.org:bau-ha.us', '$15618775895009KnrMA:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_wikwg9=2fdisroot=40chat.disroot.org:bau-ha.us', '$15714053802598hnJCR:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_errorabend=2fdisroot=40chat.disroot.org:bau-ha.us', '$15688424278967FZakH:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_amam5=2fdisroot=40chat.disroot.org:bau-ha.us', '$15734361532161BbYFO:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_geuse=2fdisroot=40chat.disroot.org:bau-ha.us', '$1575234594504DTPaY:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_marmulak=2fdisroot=40chat.disroot.org:bau-ha.us', '$15747969853227oYCqq:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_mmzz=2fdisroot=40chat.disroot.org:bau-ha.us', '$1563753312441XqjtN:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_dhanesh95|xmpp:matrix.org', '$1568414689354606MdVjS:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_debacle=2fdisroot=40chat.disroot.org:bau-ha.us', '$15595831783257CAoKs:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_diego.m51=2fdisroot=40chat.disroot.org:bau-ha.us', '$15649050868112JhRHH:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@ako:disroot.org', '$149684567160411BCmrr:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_tobi=2fdisroot=40chat.disroot.org:bau-ha.us', '$15616482092884rSwis:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_sabisil=2fdisroot=40chat.disroot.org:bau-ha.us', '$15606345081050TaKHE:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_post2me=2fdisroot=40chat.disroot.org:bau-ha.us', '$15608360532818lRamH:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_abdhessuk=2fdisroot=40chat.disroot.org:bau-ha.us', '$15610712722234qEfyD:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_perflyst_=2fdisroot=40chat.disroot.org:bau-ha.us', '$15608333262786vnMDI:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_bcome=2fdisroot=40chat.disroot.org:bau-ha.us', '$15610749252269kFLfr:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_alexesc=2fdisroot=40chat.disroot.org:bau-ha.us', '$15688338008706RADlo:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_neonbukowskii=2fdisroot=40chat.disroot.org:bau-ha.us', '$15734145301732aNRaI:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_dasj19_:matrix.org', '$1565646951342278vjPvO:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_tuxita=2fdisroot=40chat.disroot.org:bau-ha.us', '$15617342993781lRlBx:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_bartender=2fdisroot=40chat.disroot.org:bau-ha.us', '$15608420272895wVKnX:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_valeford=2fdisroot=40chat.disroot.org:bau-ha.us', '$15649203998498dOqvO:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@[email protected]:disroot.org', '$149684567760514XBBLD:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_cr5t=2fdisroot=40chat.disroot.org:bau-ha.us', '$1573889623721yIEfs:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_szymon.wierzbicki=2fdisroot=40chat.disroot.org:bau-ha.us', '$15734011631363QWLez:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_zuc=2fdisroot=40chat.disroot.org:bau-ha.us', '$15610700932229tffQN:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_tblu=2fdisroot=40chat.disroot.org:bau-ha.us', '$15734287442071LYdxF:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_steve-_-:matrix.org', '$15549365014950517rEeDx:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@Ciblia:matrix.org', '$1497678116800470KNbQf:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_nrice=2fdisroot=40chat.disroot.org:bau-ha.us', '$1560581808481BPuIZ:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_coffeecat=2fdisroot=40chat.disroot.org:bau-ha.us', '$1559930340460LzsPu:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ale=2fdisroot=40chat.disroot.org:bau-ha.us', '$15618649674903jBEFq:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@mikaela:privacytools.io', '$15814444571375IMnnM:privacytools.io'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_alhassanaraouf=2fdisroot=40chat.disroot.org:bau-ha.us', '$15767749621007QqQxk:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_joselp=2fdisroot=40chat.disroot.org:bau-ha.us', '$157480289413xGagW:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ebi=2fdisroot=40chat.disroot.org:bau-ha.us', '$15596080624202apyJP:bau-ha.us')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: [(36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@lucio776:matrix.org', '$1557941942501905Sfzou:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@theloudspeaker:matrix.org', '$1577126616423568yPobd:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@babel:matrix.org', '$155887605030618Idvhk:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_mdjanic:matrix.org', '$15787689151746958uFEpF:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_traumsch1le:matrix.org', '$15817340261773906pCrKc:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_joffamac=2fdisroot=40chat.disroot.org:bau-ha.us', '$156832202421036GaODX:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_soila=2fdisroot=40chat.disroot.org:bau-ha.us', '$15610770572287kNVpI:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_userdebug=2fdisroot=40chat.disroot.org:bau-ha.us', '$156011843485naJvu:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_outloudvi[m]:matrix.org', '$15527884852356229JIVyi:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_maryjane=2fdisroot=40chat.disroot.org:bau-ha.us', '$15708084934410ATnIV:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_adho=2fdisroot=40chat.disroot.org:bau-ha.us', '$15757335581994YrcXV:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_uggo=2fdisroot=40chat.disroot.org:bau-ha.us', '$15623700031264ubARR:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_pwr=2fdisroot=40chat.disroot.org:bau-ha.us', '$15620757306854vDIBY:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_awal=2fdisroot=40chat.disroot.org:bau-ha.us', '$15618775895009KnrMA:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_wikwg9=2fdisroot=40chat.disroot.org:bau-ha.us', '$15714053802598hnJCR:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_errorabend=2fdisroot=40chat.disroot.org:bau-ha.us', '$15688424278967FZakH:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_amam5=2fdisroot=40chat.disroot.org:bau-ha.us', '$15734361532161BbYFO:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_geuse=2fdisroot=40chat.disroot.org:bau-ha.us', '$1575234594504DTPaY:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_marmulak=2fdisroot=40chat.disroot.org:bau-ha.us', '$15747969853227oYCqq:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_mmzz=2fdisroot=40chat.disroot.org:bau-ha.us', '$1563753312441XqjtN:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_dhanesh95|xmpp:matrix.org', '$1568414689354606MdVjS:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_debacle=2fdisroot=40chat.disroot.org:bau-ha.us', '$15595831783257CAoKs:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_diego.m51=2fdisroot=40chat.disroot.org:bau-ha.us', '$15649050868112JhRHH:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@ako:disroot.org', '$149684567160411BCmrr:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_tobi=2fdisroot=40chat.disroot.org:bau-ha.us', '$15616482092884rSwis:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_sabisil=2fdisroot=40chat.disroot.org:bau-ha.us', '$15606345081050TaKHE:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_post2me=2fdisroot=40chat.disroot.org:bau-ha.us', '$15608360532818lRamH:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_abdhessuk=2fdisroot=40chat.disroot.org:bau-ha.us', '$15610712722234qEfyD:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_perflyst_=2fdisroot=40chat.disroot.org:bau-ha.us', '$15608333262786vnMDI:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_bcome=2fdisroot=40chat.disroot.org:bau-ha.us', '$15610749252269kFLfr:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_alexesc=2fdisroot=40chat.disroot.org:bau-ha.us', '$15688338008706RADlo:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_neonbukowskii=2fdisroot=40chat.disroot.org:bau-ha.us', '$15734145301732aNRaI:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_dasj19_:matrix.org', '$1565646951342278vjPvO:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_tuxita=2fdisroot=40chat.disroot.org:bau-ha.us', '$15617342993781lRlBx:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_bartender=2fdisroot=40chat.disroot.org:bau-ha.us', '$15608420272895wVKnX:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_valeford=2fdisroot=40chat.disroot.org:bau-ha.us', '$15649203998498dOqvO:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@[email protected]:disroot.org', '$149684567760514XBBLD:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_cr5t=2fdisroot=40chat.disroot.org:bau-ha.us', '$1573889623721yIEfs:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_szymon.wierzbicki=2fdisroot=40chat.disroot.org:bau-ha.us', '$15734011631363QWLez:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_zuc=2fdisroot=40chat.disroot.org:bau-ha.us', '$15610700932229tffQN:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_tblu=2fdisroot=40chat.disroot.org:bau-ha.us', '$15734287442071LYdxF:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@freenode_steve-_-:matrix.org', '$15549365014950517rEeDx:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@Ciblia:matrix.org', '$1497678116800470KNbQf:matrix.org'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_nrice=2fdisroot=40chat.disroot.org:bau-ha.us', '$1560581808481BPuIZ:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_coffeecat=2fdisroot=40chat.disroot.org:bau-ha.us', '$1559930340460LzsPu:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ale=2fdisroot=40chat.disroot.org:bau-ha.us', '$15618649674903jBEFq:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@mikaela:privacytools.io', '$15814444571375IMnnM:privacytools.io'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_alhassanaraouf=2fdisroot=40chat.disroot.org:bau-ha.us', '$15767749621007QqQxk:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_joselp=2fdisroot=40chat.disroot.org:bau-ha.us', '$157480289413xGagW:bau-ha.us'), (36946554, '!uDQoIebqsjEEtmWLrO:disroot.org', 'm.room.member', '@_xmpp_ebi=2fdisroot=40chat.disroot.org:bau-ha.us', '$15596080624202apyJP:bau-ha.us')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/synapse/lib/python3.7/site-packages/synapse/metrics/background_process_metrics.py", line 213, in run
    return (yield result)
  File "/opt/synapse/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/resource.py", line 163, in _run_notifier_loop
    updates, current_token, limited = await stream.get_updates()
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/streams/_base.py", line 138, in get_updates
    self.last_token, current_token
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/streams/_base.py", line 163, in get_updates_since
    from_token, upto_token, _STREAM_UPDATE_TARGET_ROW_COUNT,
  File "/opt/synapse/lib/python3.7/site-packages/synapse/replication/tcp/streams/events.py", line 180, in _update_function
    assert len(state_rows) < target_row_count
AssertionError
2020-04-28 17:15:02,465 - synapse.access.http.9593 - 301 - INFO - POST-6166- 127.0.0.1 - 9593 - {@Half-Shot:half-shot.uk} Processed request: 0.197sec/0.000sec (0.003sec, 0.000sec) (0.001sec/0.018sec/1) 2B 200 "POST /_synapse/replication/send_event/%247EhWbiwsfRAJnGlaZ-BUyGkX-ueNNAghtQbMDmqQQ1c/nWxGBrDBCH HTTP/1.1" "Synapse/1.12.4" [0 dbevts]

fresh logs from develop

@richvdh
Copy link
Member

richvdh commented Apr 28, 2020

thanks, looks like a dup of #7308

@richvdh richvdh closed this as completed Apr 28, 2020
@richvdh richvdh added the A-Workers Problems related to running Synapse in Worker Mode (or replication) label Feb 16, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@richvdh richvdh

Labels
A-Workers Problems related to running Synapse in Worker Mode (or replication)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@richvdh @Half-Shot @babolivier