Support unread message counts server-side

[delijati]

This is way too much code to be implemented in every client [1]

[1] matrix-org/matrix-python-sdk#161

[ara4n]

I agree that the balance is wrong, and irritatingly I'm failing to remember how we've ended up like this. Just to confirm the current situation:

• The server calculates the number of unread highlighted notifications for a room (the unread_notifications.highlight_count field of the /sync response)
• It also calculates the number of unread notifications in general (the unread_notifications.notification-count field.

You're right that it doesn't track the number of unread messages in general though, and this has to be inferred by the client by looping through all the messages which the client has received in a room since we last sent a Read Receipt, and summing them up.

Possible reasons for this are:

1. It's quite a heavy operation for the server to do, so we offloaded it to the client (whereas doing it for the subset of messages which are notifications is less heavy as there's less data)
2. We're forced into calculating the unread notification counts serverside anyway to update push notification badges
3. Clients can apply whatever weird filtering rules they like to show given events, so in the end it's really up to the client to decide whether a given message should actually nudge the unread count for a room or not. (e.g. should an m.call.candidates event in a room nudge the unread count or not?).

In practice, calculating the unread counts clientside is still a trainwreck, as evinced by element-hq/element-web#3363, so we really do need to find a way to fix this.

Perhaps one solution would be to define stricter rules on the events which should constitute an "unread messages" in the spec, and thus have that metric be consistent across all clients. This could in turn tie into the long-awaited "human readable event representation" work (matrix-org/matrix-spec-proposals#92 and its many descendants), as in the end the concept of "unreadness" is very much a human representation question. The same metric that we use to decide that a message is human readable could then also be used to calculate unreadness serverside, and everyone wins.

[komzbike]

+1

[ara4n]

To summarise the current situation:

• Servers still calculate badge counts for the number of rooms with missed notifications and missed highlights.
• Clients still have to calculate whether a given room should be marked as unread or not, because only they know which events they happen to render to users to be visible.
• The bugs which contributed the most to unreliable unread counts on Riot/Web over in Room unread counts are unreliable (metabug) element-hq/element-web#3363 (which I described as a trainwreck back in Nov 2017) have since been solved.

It's true that the server could guess which events the client can render, but this feels like it could be fragile and lead to further unpredictable behaviour.

Alternatively, we could mandate which events the client should try to render (showing fallback if they can't) - for instance, when matrix-org/matrix-spec-proposals#1767 comes to pass, we could declare that any event with an m.text field should be considered to contribute to the unread count - and for E2E rooms, use the tricks in matrix-org/matrix-spec-proposals#1796 to tag the events to indicate the existence of a human-visible message.

This would then (modulo performance considerations) let us calculate full unread counts serverside - which could be useful for "catch up on the last 100 unread messages in this room" style UIs. However, we've got by without these semantics so far without missing it that much. It would be necessary for full parity with Slack however.

This is unlikely to happen anytime soon unless someone desperately wants it.

[richvdh]

this is related to matrix-org/matrix-spec-proposals#2654, though I don't think it will do much to change the situation here.

[clokep]

Note that MSC2654 was implemented in Synapse in #8059, albeit behind an experimental flag.