Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix deduplicating of membership events to not create unused state groups. #17164

Merged
merged 6 commits into from
May 30, 2024
Merged

Fix deduplicating of membership events to not create unused state groups. #17164

merged 6 commits into from
May 30, 2024

Conversation

erikjohnston
Copy link
Member

We try and deduplicate in two places: 1) really early on, and 2) just before we persist the event. The first case was broken due to it occuring before the profile information was added, and so it thought the event contents were different.

The second case did catch it and handle it correctly, however doing so creates a redundant state group leading to bloat.

Fixes #3791

@erikjohnston
Fix deduplicating of membership events
9b35d46 
We try and deduplicate in two places: 1) really early on, and 2) just
before we persist the event. The first case was broken due to it
occuring before the profile information was added, and so it thought the
event contents were different.

The second case did catch it and handle it correctly, however doing so
creates a redundant state group leading to bloat.
@erikjohnston
Newsfile
277a0c6
@erikjohnston
Check membership is valid before using it
6844afd
@erikjohnston erikjohnston marked this pull request as ready for review May 7, 2024 14:56
@erikjohnston erikjohnston requested a review from a team as a code owner May 7, 2024 14:56
devonh
devonh reviewed May 14, 2024
View reviewed changes
Copy link
Member

@devonh devonh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fix looks good to me & makes sense as to why it fixes the issue.
I would feel more comfortable if someone more familiar with this section of code also takes a look before this PR is approved.

tests/handlers/test_room_member.py Outdated Show resolved Hide resolved
synapse/handlers/message.py Show resolved Hide resolved
@devonh
Copy link
Member

devonh commented May 14, 2024

This line in the create_event function doc is now out of date:

synapse/synapse/handlers/message.py

Line 597 in 68dca80

Adds display names to Join membership events.

@erikjohnston erikjohnston requested a review from a team May 15, 2024 11:44
anoadragon453
anoadragon453 approved these changes May 29, 2024
View reviewed changes
Copy link
Member

@anoadragon453 anoadragon453 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple nits, but otherwise LGTM!

synapse/handlers/room_member.py Outdated
@@ -106,6 +106,13 @@ def __init__(self, hs: "HomeServer"):
self.event_auth_handler = hs.get_event_auth_handler()
self._worker_lock_handler = hs.get_worker_locks_handler()

self.membership_types_to_include_profile_data_in = {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: self.membership_types_to_include_profile_data_in should have a leading underscore.

tests/handlers/test_room_member.py Outdated Show resolved Hide resolved
@erikjohnston erikjohnston enabled auto-merge (squash) May 30, 2024 10:30
@erikjohnston erikjohnston merged commit 4e3868d into develop May 30, 2024
38 checks passed
@erikjohnston erikjohnston deleted the erikj/properly_deduplicate branch May 30, 2024 11:33
H-Shay pushed a commit to H-Shay/hq_synapse that referenced this pull request May 31, 2024
@erikjohnston @H-Shay
Fix deduplicating of membership events to not create unused state gro…
44bee74 
…ups. (element-hq#17164)

We try and deduplicate in two places: 1) really early on, and 2) just
before we persist the event. The first case was broken due to it
occuring before the profile information was added, and so it thought the
event contents were different.

The second case did catch it and handle it correctly, however doing so
creates a redundant state group leading to bloat.

Fixes element-hq#3791
Mic92 pushed a commit to Mic92/synapse that referenced this pull request Jun 14, 2024
@erikjohnston @Mic92
Fix deduplicating of membership events to not create unused state gro…
17f6266 
…ups. (element-hq#17164)

We try and deduplicate in two places: 1) really early on, and 2) just
before we persist the event. The first case was broken due to it
occuring before the profile information was added, and so it thought the
event contents were different.

The second case did catch it and handle it correctly, however doing so
creates a redundant state group leading to bloat.

Fixes element-hq#3791
yingziwu added a commit to yingziwu/synapse that referenced this pull request Jun 26, 2024
@yingziwu
Merge tag 'v1.109.0' into develop
a31c075 
- Fix the building of binary wheels for macOS by switching to macOS 12 CI runners. ([\#17319](element-hq/synapse#17319))

- When rolling back to a previous Synapse version and then forwards again to this release, don't require server operators to manually run SQL. ([\#17305](element-hq/synapse#17305), [\#17309](element-hq/synapse#17309))

- Use the release branch for sytest in release-branch PRs. ([\#17306](element-hq/synapse#17306))

- Fix bug where one-time-keys were not always included in `/sync` response when using workers. Introduced in v1.109.0rc1. ([\#17275](element-hq/synapse#17275))
- Fix bug where `/sync` could get stuck due to edge case in device lists handling. Introduced in v1.109.0rc1. ([\#17292](element-hq/synapse#17292))

- Add the ability to auto-accept invites on the behalf of users. See the [`auto_accept_invites`](https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html#auto-accept-invites) config option for details. ([\#17147](element-hq/synapse#17147))
- Add experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync/e2ee` endpoint for to-device messages and device encryption info. ([\#17167](element-hq/synapse#17167))
- Support [MSC3916](matrix-org/matrix-spec-proposals#3916) by adding unstable media endpoints to `/_matrix/client`. ([\#17213](element-hq/synapse#17213))
- Add logging to tasks managed by the task scheduler, showing CPU and database usage. ([\#17219](element-hq/synapse#17219))

- Fix deduplicating of membership events to not create unused state groups. ([\#17164](element-hq/synapse#17164))
- Fix bug where duplicate events could be sent down sync when using workers that are overloaded. ([\#17215](element-hq/synapse#17215))
- Ignore attempts to send to-device messages to bad users, to avoid log spam when we try to connect to the bad server. ([\#17240](element-hq/synapse#17240))
- Fix handling of duplicate concurrent uploading of device one-time-keys. ([\#17241](element-hq/synapse#17241))
- Fix reporting of default tags to Sentry, such as worker name. Broke in v1.108.0. ([\#17251](element-hq/synapse#17251))
- Fix bug where typing updates would not be sent when using workers after a restart. ([\#17252](element-hq/synapse#17252))

- Update the LemonLDAP documentation to say that claims should be explicitly included in the returned `id_token`, as Synapse won't request them. ([\#17204](element-hq/synapse#17204))

- Improve DB usage when fetching related events. ([\#17083](element-hq/synapse#17083))
- Log exceptions when failing to auto-join new user according to the `auto_join_rooms` option. ([\#17176](element-hq/synapse#17176))
- Reduce work of calculating outbound device lists updates. ([\#17211](element-hq/synapse#17211))
- Improve performance of calculating device lists changes in `/sync`. ([\#17216](element-hq/synapse#17216))
- Move towards using `MultiWriterIdGenerator` everywhere. ([\#17226](element-hq/synapse#17226))
- Replaces all usages of `StreamIdGenerator` with `MultiWriterIdGenerator`. ([\#17229](element-hq/synapse#17229))
- Change the `allow_unsafe_locale` config option to also apply when setting up new databases. ([\#17238](element-hq/synapse#17238))
- Fix errors in logs about closing incorrect logging contexts when media gets rejected by a module. ([\#17239](element-hq/synapse#17239), [\#17246](element-hq/synapse#17246))
- Clean out invalid destinations from `device_federation_outbox` table. ([\#17242](element-hq/synapse#17242))
- Stop logging errors when receiving invalid User IDs in key querys requests. ([\#17250](element-hq/synapse#17250))

* Bump anyhow from 1.0.83 to 1.0.86. ([\#17220](element-hq/synapse#17220))
* Bump bcrypt from 4.1.2 to 4.1.3. ([\#17224](element-hq/synapse#17224))
* Bump lxml from 5.2.1 to 5.2.2. ([\#17261](element-hq/synapse#17261))
* Bump mypy-zope from 1.0.3 to 1.0.4. ([\#17262](element-hq/synapse#17262))
* Bump phonenumbers from 8.13.35 to 8.13.37. ([\#17235](element-hq/synapse#17235))
* Bump prometheus-client from 0.19.0 to 0.20.0. ([\#17233](element-hq/synapse#17233))
* Bump pyasn1 from 0.5.1 to 0.6.0. ([\#17223](element-hq/synapse#17223))
* Bump pyicu from 2.13 to 2.13.1. ([\#17236](element-hq/synapse#17236))
* Bump pyopenssl from 24.0.0 to 24.1.0. ([\#17234](element-hq/synapse#17234))
* Bump serde from 1.0.201 to 1.0.202. ([\#17221](element-hq/synapse#17221))
* Bump serde from 1.0.202 to 1.0.203. ([\#17232](element-hq/synapse#17232))
* Bump twine from 5.0.0 to 5.1.0. ([\#17225](element-hq/synapse#17225))
* Bump types-psycopg2 from 2.9.21.20240311 to 2.9.21.20240417. ([\#17222](element-hq/synapse#17222))
* Bump types-pyopenssl from 24.0.0.20240311 to 24.1.0.20240425. ([\#17260](element-hq/synapse#17260))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@devonh devonh devonh left review comments

@anoadragon453 anoadragon453 anoadragon453 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

we create redundant state groups when we get a no-op membership update
3 participants
@erikjohnston @devonh @anoadragon453