Complement Existing members see new members' presence is flaky

[reivilibre]

It took me 363 runs (~3 hours of CI) to reproduce this, so maybe it's not the highest priority flake around.

The issue seems to be that Bob's presence information is not coming down sync.

• Complement test link: https://github.com/matrix-org/complement/blob/d784f7d96677e8b4267da9729d5c601c1e99e741/tests/csapi/rooms_members_local_test.go#L38-L61
• Equivalent SyTest link: https://github.com/matrix-org/sytest/blob/a94cd1dc2d6102e5a9e94659e2dca243e9b72208/tests/30rooms/02members-local.pl#L104-L117
• Logs for failing test: TestMembersLocal.log

I believe the reason we expect Bob's presence to come down sync is because /join involves sending an event and that should update his presence.

I can't see the code that should trigger this in Synapse, but since this succeeds most of the time it must be there.
In the test logs for this test, I didn't spot any requests to /_synapse/replication/presence_set_state/ for Bob, which I would have expected to see. (I do see some for Alice.)

For this reason, I hesitate to brush it off as 'slow replication'. I also note that the requests in the log don't seem to be that slow around the time of the failure, so I don't suspect CPU contention.

[DMRobertson]

Seen again at https://github.com/matrix-org/synapse/runs/7541314250?check_suite_focus=true#step:4:3959 (on a monolith sqlite deployment?)

[DMRobertson]

https://github.com/matrix-org/synapse/runs/7742123843?check_suite_focus=true

[richvdh]

maybe this is happening more frequently now? Also seen at https://github.com/matrix-org/synapse/runs/7783161292?check_suite_focus=true (on a monolith)

[richvdh]

keyword: TestMembersLocal/Parallel/Existing_members_see_new_members'_presence

[DMRobertson]

https://github.com/matrix-org/synapse/runs/8206808042?check_suite_focus=true

[DMRobertson]

https://github.com/matrix-org/synapse/runs/8213472770?check_suite_focus=true

[clokep]

https://github.com/matrix-org/synapse/actions/runs/3080827578/jobs/4978642028#logs

[DMRobertson]

https://github.com/matrix-org/synapse/actions/runs/3127323400/jobs/5073871243

[DMRobertson]

https://github.com/matrix-org/synapse/actions/runs/3137293711/jobs/5095334318

[DMRobertson]

https://github.com/matrix-org/synapse/actions/runs/3144138134/jobs/5109828689

[DMRobertson]

https://github.com/matrix-org/synapse/actions/runs/3244698433/jobs/5321264235

[DMRobertson]

https://github.com/matrix-org/synapse/actions/runs/3250081130/jobs/5333349627

[DMRobertson]

https://github.com/matrix-org/synapse/actions/runs/3280866173/jobs/5402221795

[DMRobertson]

https://github.com/matrix-org/synapse/actions/runs/3321575592/jobs/5489496717

[DMRobertson]

https://github.com/matrix-org/synapse/actions/runs/3369210209/jobs/5588651779

[DMRobertson]

https://github.com/matrix-org/synapse/actions/runs/3375799267/jobs/5602795377

[DMRobertson]

https://github.com/matrix-org/synapse/actions/runs/3439211524/jobs/5736298006

[DMRobertson]

https://github.com/matrix-org/synapse/actions/runs/3462005537/jobs/5780406423

[squahtx]

On SQLite in monolith mode, the test fails reliably if modified so that Bob's join appears in an initial sync. Bob's presence is offline(!) and it's only sent during incremental syncs, so this appears to be an issue with the test setup, or Synapse not setting Bob's presence to online upon a join.

[squahtx]

We appear to only bump presence when Bob sends a regular message.

synapse/synapse/handlers/message.py

Lines 1918 to 1921 in 86c5a71

	if event.type == EventTypes.Message:
	# We don't want to block sending messages on any presence code. This
	# matters as sometimes presence code can take a while.
	run_in_background(self._bump_active_time, requester.user)

That condition has been around since f70e622 from 2014.

However, b31ec21 modified the bump to only transition from unavailable to online and changes nothing when a user is offline. So it seems that users intentionally do not go from offline to online when sending messages.

synapse/synapse/handlers/presence.py

Lines 964 to 965 in 86c5a71

	if prev_state.state == PresenceState.UNAVAILABLE:
	new_fields["state"] = PresenceState.ONLINE