Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

"Failed to calculate hosts in room" error from outbound federation #12509

Closed
richvdh opened this issue Apr 20, 2022 · 4 comments · Fixed by #12570
Closed

"Failed to calculate hosts in room" error from outbound federation #12509

richvdh opened this issue Apr 20, 2022 · 4 comments · Fixed by #12570
Assignees
@richvdh
Labels
T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. X-Regression Something broke which worked on a previous release

Comments

@richvdh
Copy link
Member

richvdh commented Apr 20, 2022

Presumably introduced by #12191. Suspect this means that outbound federation is broken in some rooms? Or it might be transient?

2022-04-20 00:27:42,101 - synapse.federation.sender - 392 - ERROR - process_event_queue_for_federation-6803872 - Failed to calculate hosts in room for event: <event_id>
Capture point (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/synapse/src/synapse/app/federation_sender.py", line 22, in <module>
    start(sys.argv[1:])
  File "/home/synapse/src/synapse/app/generic_worker.py", line 505, in start
    _base.start_worker_reactor("synapse-generic-worker", config)
  File "/home/synapse/src/synapse/app/_base.py", line 125, in start_worker_reactor
    run_command=run_command,
  File "/home/synapse/src/synapse/app/_base.py", line 180, in start_reactor
    run()
  File "/home/synapse/src/synapse/app/_base.py", line 162, in run
    run_command()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/base.py", line 1315, in run
    self.mainLoop()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/base.py", line 1325, in mainLoop
    reactorBaseSelf.runUntilCurrent()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/base.py", line 964, in runUntilCurrent
    f(*a, **kw)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 661, in callback
    self._startRunCallbacks(result)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
    self._runCallbacks()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
    current.result, *args, **kwargs
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1750, in gotResult
    current_context.run(_inlineCallbacks, r, gen, status)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1663, in _inlineCallbacks
    status.deferred.callback(getattr(e, "value", None))
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 661, in callback
    self._startRunCallbacks(result)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
    self._runCallbacks()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
    current.result, *args, **kwargs
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 661, in callback
    self._startRunCallbacks(result)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
    self._runCallbacks()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
    current.result, *args, **kwargs
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1750, in gotResult
    current_context.run(_inlineCallbacks, r, gen, status)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1740, in _inlineCallbacks
    status.deferred.errback()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 700, in errback
    self._startRunCallbacks(fail)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
    self._runCallbacks()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
    current.result, *args, **kwargs
  File "/home/synapse/src/synapse/util/caches/descriptors.py", line 505, in errback_all
    d1.errback(f)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 700, in errback
    self._startRunCallbacks(fail)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
    self._runCallbacks()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
    current.result, *args, **kwargs
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1750, in gotResult
    current_context.run(_inlineCallbacks, r, gen, status)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1657, in _inlineCallbacks
    cast(Failure, result).throwExceptionIntoGenerator, gen
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/python/failure.py", line 489, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
Traceback (most recent call last):
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
    result = current_context.run(gen.send, result)
StopIteration: []

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/synapse/src/synapse/federation/sender/__init__.py", line 387, in handle_event
    event.room_id, event_ids=event.prev_event_ids()
  File "/home/synapse/src/synapse/state/__init__.py", line 257, in get_hosts_in_room_at_events
    entry = await self.resolve_state_groups_for_events(room_id, event_ids)
  File "/home/synapse/src/synapse/util/metrics.py", line 106, in measured_func
    r = await func(self, *args, **kwargs)
  File "/home/synapse/src/synapse/state/__init__.py", line 425, in resolve_state_groups_for_events
    room_id, event_ids
  File "/home/synapse/src/synapse/storage/state.py", line 582, in get_state_groups_ids
    event_to_groups = await self.stores.main._get_state_group_for_events(event_ids)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
    result = current_context.run(gen.send, result)
  File "/home/synapse/src/synapse/storage/databases/main/state.py", line 331, in _get_state_group_for_events
    raise RuntimeError("No state group for unknown or outlier event %s" % e)
RuntimeError: No state group for unknown or outlier event <different event id>
@richvdh
Copy link
Member Author

richvdh commented Apr 20, 2022

See https://sentry.matrix.org/sentry/synapse-matrixorg/issues/246739

@reivilibre reivilibre added T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. X-Regression Something broke which worked on a previous release labels Apr 21, 2022
@reivilibre
Copy link
Contributor

If we suspect this is breaking outbound federation for some rooms, it seems like we might want to investigate this one urgently.

@Ezwen
Copy link

Ezwen commented Apr 21, 2022

I shared a slightly similar error log in this other issue (more precisely, the message is similar, even though the stack is different): #12507 (comment)

@richvdh
Copy link
Member Author

richvdh commented Apr 27, 2022 •
edited
Loading

Ok, so this is being caused by:

  • rejections for invites received over federation.
  • knock events we sent for federated rooms

Neither of these are meant to be sent over federation, so the exception does not reflect a functional problem. However, we clearly need to do something about the log spam.

richvdh added a commit that referenced this issue Apr 27, 2022
@richvdh
Exclude OOB memberships from the federation sender
78dad8a 
As the comment says, there is no need to process such events, and indeed we
need to avoid doing so.

Fixes #12509.
@richvdh richvdh mentioned this issue Apr 27, 2022
Merged
richvdh added a commit that referenced this issue May 3, 2022
@richvdh
Exclude OOB memberships from the federation sender (#12570)
db2edf5 
As the comment says, there is no need to process such events, and indeed we
need to avoid doing so.

Fixes #12509.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@richvdh richvdh

Labels
T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. X-Regression Something broke which worked on a previous release
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Exclude OOB memberships from the federation sender matrix-org/synapse
3 participants
@Ezwen @richvdh @reivilibre