Inject state blocks from incremental polls into the room timeline #71

DMRobertson · 2023-04-13T18:45:09Z

Closes #18. This is a best-effort attempt to ensure that room state self-heals after the poller "misses" a state event. We don't attempt to retrieve any missed message events.

For a more robust solution, the homeserver should natively implement sliding sync.

#73 also helps, by making it less likely that the poller sees a gap in the first place.

tests-e2e/client_test.go

tests-integration/poller_test.go

tests-e2e/gappy_state_test.go

We fetch those NIDs above in `eventIDToNID` by adding a `RETURNING` clause to the `INSERT` statement.

…state

DMRobertson · 2023-04-17T19:23:06Z

(Sorry for force pushing, I'd forgotten that this branch was already PRed.)

state/accumulator.go

DMRobertson · 2023-04-17T19:31:57Z

state/event_table.go

+	WITH maybe_unknown_events(event_id) AS (SELECT unnest($1::text[]))
+	SELECT event_id
+	FROM maybe_unknown_events LEFT JOIN syncv3_events USING(event_id)
+	WHERE event_nid IS NULL;`


I did test this in a postgres playground, but I think this might be worth a unittest.

Please do. There is event_table_test.go to do that.

We don't use this style of query anywhere in the proxy atm. I'm guessing you do this so you can "select then negate" all on the DB side? I'd expected to see something more like:

SELECT event_id FROM syncv3_events WHERE room_id=$1 AND event_id=ANY($2)

but then you'd need to negate on the Go side to find the missing ones. I wonder which is better.

Yeah, the thinking was basically to get the DB to tell me which events are unknown, rather than getting it to tell me which events are known and then computing the complement. (I'm assuming that we know about most state events, so we may as well get the DB to transmit the smaller set back to us.)

I don't think it makes a huge difference if the state block and room are small. I'm slightly worried about hitting this query for Matrix HQ with state block from a new poller's initial sync.

kegsay

Mostly LGTM, just a few things.

state/accumulator.go

kegsay · 2023-04-19T09:04:16Z

state/accumulator.go

+			for i := range state {
+				eventIDs[i] = gjson.ParseBytes(state[i]).Get("event_id").Str
+			}
+			unknownEventIDs, err := a.eventsTable.SelectUnknownEventIDs(txn, eventIDs)


This could be improved if we made use of the room ID as it can cut lookup times down a lot.

Oh, I'd just assumed this was indexed on event id---maybe we should add that index?

Err, the event_id column is UNIQUE, so it will be index will exist. Not sure why the room ID would help here?

The room ID can help as it cuts down the dataset immediately, rather than relying on a the mahoosive event ID index of all events ever. In the past it has been beneficial to include this information when running EXPLAIN.

I remain sceptical.

-- Matrix HQ \set room_id '!OGEhHVWSdvArJzumhm:matrix.org' -- 20 event Ids taken from Matrix HQ, plus two fake event IDs \set event_ids {$fakeevent1,$4jteAg_T0BMXWiSRzZbptOF9zETeomXbJoDtg6zs9SU,$Iu-74VzJ0B4Xc6XgBDAKoWYZkFcUGCuJ_vLd47Cf4Ok,$OIqjRnK_rXBApl_8x4s0Iq--OH969cikkvjGYW5Seq8,$7EYkESAKR-t6wKLG7rjPyyVcZdQ-v0YQtaLbzrLJ2vI,$6LYIKlvAlC1ILMhIsWx-M9IDOarFBLM_Ni609bz1NDI,$w1sLe53ZqsRc9vebfLS8JCUyq3ql9DRhN-lERWsZRHQ,$A6k3YCykEOHCzsbzxQjdupwNdt4hr4AeS6IWOnEG1H4,$K1EGNY8C-McIJJ5PBbd1GlUFMuw84S_QlbomqnYMiYY,$QgyACr1TujWs846B5tiAkbM9JkBEzUoMK6QgOfXDc64,$f2aefO_6yYMhSFRtQWYkTDdie87ldgTJD1h3JQ2F_3Q,$pYU_XgUXF0kQXvtSjG0boNdMHtXGBIFfwBDqxZuU1IE,$cZ5XF2o7ObQqROkJZWSubgm4cLgBX8j1nkrcrHv0rZ0,$wXTQjHKMTsMBqj3B4BLN40L0-7tz1NbG8WesWgGcjwY,$csN_i4UoBTk5jFzdDAxnKSmKg0Hp6U7Af_ifht07D-o,$NBsWMM0MJ_XcAt9gk9ZJxV3_v3tPfvmZ1azHI3Wb31A,$bf1IpaCxsC2Te4mWrBXcJZqesoh1HsssdaI_U1OzXuQ,$nsLyOZZRywB1bGwEVNR3qQFLuY3hGOuM-26J6IAvsgk,$-b0T_rqr5-vE1FiqwYOriXkTE0c8oMXj0a9uSeMnWVw,$qx3qMhu6ba_ceFcv9ADA5J45CTUhhAMDz44eOVmJwR8,$ugwvSmk0LfVLpxk0cloQ6tQxjacdp0ojurri3pN4bl4,$notanevent2}

Query without room ID:

EXPLAIN ANALYZE VERBOSE WITH maybe_unknown_events(event_id) AS (SELECT unnest(:'event_ids'::text[])) SELECT * FROM maybe_unknown_events LEFT JOIN syncv3_events USING(event_id) WHERE event_nid IS NULL;

QUERY PLAN -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Nested Loop Left Join (cost=0.56..189.27 rows=1 width=648) (actual time=0.030..0.239 rows=2 loops=1) Output: (unnest('{$fakeevent1,$4jteAg_T0BMXWiSRzZbptOF9zETeomXbJoDtg6zs9SU,$Iu-74VzJ0B4Xc6XgBDAKoWYZkFcUGCuJ_vLd47Cf4Ok,$OIqjRnK_rXBApl_8x4s0Iq--OH969cikkvjGYW5Seq8,$7EYkESAKR-t6wKLG7rjPyyVcZdQ-v0YQtaLbzrLJ2vI,$6LYIKlvAlC1ILMhIsWx-M9IDOarFBLM_Ni609bz1NDI,$w1sLe53ZqsRc9vebfLS8JCUyq 3ql9DRhN-lERWsZRHQ,$A6k3YCykEOHCzsbzxQjdupwNdt4hr4AeS6IWOnEG1H4,$K1EGNY8C-McIJJ5PBbd1GlUFMuw84S_QlbomqnYMiYY,$QgyACr1TujWs846B5tiAkbM9JkBEzUoMK6QgOfXDc64,$f2aefO_6yYMhSFRtQWYkTDdie87ldgTJD1h3JQ2F_3Q,$pYU_XgUXF0kQXvtSjG0boNdMHtXGBIFfwBDqxZuU1IE,$cZ5XF2o7ObQqROkJZWSubgm4cLgBX8j1nkrcrHv 0rZ0,$wXTQjHKMTsMBqj3B4BLN40L0-7tz1NbG8WesWgGcjwY,$csN_i4UoBTk5jFzdDAxnKSmKg0Hp6U7Af_ifht07D-o,$NBsWMM0MJ_XcAt9gk9ZJxV3_v3tPfvmZ1azHI3Wb31A,$bf1IpaCxsC2Te4mWrBXcJZqesoh1HsssdaI_U1OzXuQ,$nsLyOZZRywB1bGwEVNR3qQFLuY3hGOuM-26J6IAvsgk,$-b0T_rqr5-vE1FiqwYOriXkTE0c8oMXj0a9uSeMnWVw,$qx3qMhu6 ba_ceFcv9ADA5J45CTUhhAMDz44eOVmJwR8,$ugwvSmk0LfVLpxk0cloQ6tQxjacdp0ojurri3pN4bl4,$notanevent2}'::text[])), syncv3_events.event_nid, syncv3_events.before_state_snapshot_id, syncv3_events.event_replaces_nid, syncv3_events.room_id, syncv3_events.event_type, syncv3_events.state_key, sync v3_events.prev_batch, syncv3_events.membership, syncv3_events.is_state, syncv3_events.event Inner Unique: true Filter: (syncv3_events.event_nid IS NULL) Rows Removed by Filter: 20 -> ProjectSet (cost=0.00..0.13 rows=22 width=32) (actual time=0.003..0.007 rows=22 loops=1) Output: unnest('{$fakeevent1,$4jteAg_T0BMXWiSRzZbptOF9zETeomXbJoDtg6zs9SU,$Iu-74VzJ0B4Xc6XgBDAKoWYZkFcUGCuJ_vLd47Cf4Ok,$OIqjRnK_rXBApl_8x4s0Iq--OH969cikkvjGYW5Seq8,$7EYkESAKR-t6wKLG7rjPyyVcZdQ-v0YQtaLbzrLJ2vI,$6LYIKlvAlC1ILMhIsWx-M9IDOarFBLM_Ni609bz1NDI,$w1sLe53ZqsRc9vebfLS8 JCUyq3ql9DRhN-lERWsZRHQ,$A6k3YCykEOHCzsbzxQjdupwNdt4hr4AeS6IWOnEG1H4,$K1EGNY8C-McIJJ5PBbd1GlUFMuw84S_QlbomqnYMiYY,$QgyACr1TujWs846B5tiAkbM9JkBEzUoMK6QgOfXDc64,$f2aefO_6yYMhSFRtQWYkTDdie87ldgTJD1h3JQ2F_3Q,$pYU_XgUXF0kQXvtSjG0boNdMHtXGBIFfwBDqxZuU1IE,$cZ5XF2o7ObQqROkJZWSubgm4cLgBX8j1nk rcrHv0rZ0,$wXTQjHKMTsMBqj3B4BLN40L0-7tz1NbG8WesWgGcjwY,$csN_i4UoBTk5jFzdDAxnKSmKg0Hp6U7Af_ifht07D-o,$NBsWMM0MJ_XcAt9gk9ZJxV3_v3tPfvmZ1azHI3Wb31A,$bf1IpaCxsC2Te4mWrBXcJZqesoh1HsssdaI_U1OzXuQ,$nsLyOZZRywB1bGwEVNR3qQFLuY3hGOuM-26J6IAvsgk,$-b0T_rqr5-vE1FiqwYOriXkTE0c8oMXj0a9uSeMnWVw,$qx3 qMhu6ba_ceFcv9ADA5J45CTUhhAMDz44eOVmJwR8,$ugwvSmk0LfVLpxk0cloQ6tQxjacdp0ojurri3pN4bl4,$notanevent2}'::text[]) -> Result (cost=0.00..0.01 rows=1 width=0) (actual time=0.001..0.001 rows=1 loops=1) -> Index Scan using syncv3_events_event_id_key on public.syncv3_events (cost=0.56..8.58 rows=1 width=658) (actual time=0.010..0.010 rows=1 loops=22) Output: syncv3_events.event_nid, syncv3_events.event_id, syncv3_events.before_state_snapshot_id, syncv3_events.event_replaces_nid, syncv3_events.room_id, syncv3_events.event_type, syncv3_events.state_key, syncv3_events.prev_batch, syncv3_events.membership, syncv3_events.is_s tate, syncv3_events.event Index Cond: (syncv3_events.event_id = (unnest('{$fakeevent1,$4jteAg_T0BMXWiSRzZbptOF9zETeomXbJoDtg6zs9SU,$Iu-74VzJ0B4Xc6XgBDAKoWYZkFcUGCuJ_vLd47Cf4Ok,$OIqjRnK_rXBApl_8x4s0Iq--OH969cikkvjGYW5Seq8,$7EYkESAKR-t6wKLG7rjPyyVcZdQ-v0YQtaLbzrLJ2vI,$6LYIKlvAlC1ILMhIsWx-M9IDOarFBLM_Ni 609bz1NDI,$w1sLe53ZqsRc9vebfLS8JCUyq3ql9DRhN-lERWsZRHQ,$A6k3YCykEOHCzsbzxQjdupwNdt4hr4AeS6IWOnEG1H4,$K1EGNY8C-McIJJ5PBbd1GlUFMuw84S_QlbomqnYMiYY,$QgyACr1TujWs846B5tiAkbM9JkBEzUoMK6QgOfXDc64,$f2aefO_6yYMhSFRtQWYkTDdie87ldgTJD1h3JQ2F_3Q,$pYU_XgUXF0kQXvtSjG0boNdMHtXGBIFfwBDqxZuU1IE,$cZ5 XF2o7ObQqROkJZWSubgm4cLgBX8j1nkrcrHv0rZ0,$wXTQjHKMTsMBqj3B4BLN40L0-7tz1NbG8WesWgGcjwY,$csN_i4UoBTk5jFzdDAxnKSmKg0Hp6U7Af_ifht07D-o,$NBsWMM0MJ_XcAt9gk9ZJxV3_v3tPfvmZ1azHI3Wb31A,$bf1IpaCxsC2Te4mWrBXcJZqesoh1HsssdaI_U1OzXuQ,$nsLyOZZRywB1bGwEVNR3qQFLuY3hGOuM-26J6IAvsgk,$-b0T_rqr5-vE1Fiqw YOriXkTE0c8oMXj0a9uSeMnWVw,$qx3qMhu6ba_ceFcv9ADA5J45CTUhhAMDz44eOVmJwR8,$ugwvSmk0LfVLpxk0cloQ6tQxjacdp0ojurri3pN4bl4,$notanevent2}'::text[]))) Query Identifier: -467502754092800539 Planning Time: 0.142 ms Execution Time: 0.264 ms

Query also checking room ID (note it is not just a case of appending WHERE room_id = ...):

EXPLAIN ANALYZE VERBOSE WITH maybe_unknown_events(event_id) AS (SELECT unnest(:'event_ids'::text[])) SELECT * FROM maybe_unknown_events LEFT JOIN syncv3_events ON maybe_unknown_events.event_id = syncv3_events.event_id AND room_id = :'room_id' WHERE event_nid IS NULL;

QUERY PLAN -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------- Nested Loop Left Join (cost=0.56..189.33 rows=1 width=690) (actual time=0.053..0.262 rows=2 loops=1) Output: (unnest('{$fakeevent1,$4jteAg_T0BMXWiSRzZbptOF9zETeomXbJoDtg6zs9SU,$Iu-74VzJ0B4Xc6XgBDAKoWYZkFcUGCuJ_vLd47Cf4Ok,$OIqjRnK_rXBApl_8x4s0Iq--OH969cikkvjGYW5Seq8,$7EYkESAKR-t6wKLG7rjPyyVcZdQ-v0YQtaLbzrLJ2vI,$6LYIKlvAlC1ILMhIsWx-M9IDOarFBLM_Ni609bz1NDI,$w1sLe53ZqsRc9vebfLS8JCUyq 3ql9DRhN-lERWsZRHQ,$A6k3YCykEOHCzsbzxQjdupwNdt4hr4AeS6IWOnEG1H4,$K1EGNY8C-McIJJ5PBbd1GlUFMuw84S_QlbomqnYMiYY,$QgyACr1TujWs846B5tiAkbM9JkBEzUoMK6QgOfXDc64,$f2aefO_6yYMhSFRtQWYkTDdie87ldgTJD1h3JQ2F_3Q,$pYU_XgUXF0kQXvtSjG0boNdMHtXGBIFfwBDqxZuU1IE,$cZ5XF2o7ObQqROkJZWSubgm4cLgBX8j1nkrcrHv 0rZ0,$wXTQjHKMTsMBqj3B4BLN40L0-7tz1NbG8WesWgGcjwY,$csN_i4UoBTk5jFzdDAxnKSmKg0Hp6U7Af_ifht07D-o,$NBsWMM0MJ_XcAt9gk9ZJxV3_v3tPfvmZ1azHI3Wb31A,$bf1IpaCxsC2Te4mWrBXcJZqesoh1HsssdaI_U1OzXuQ,$nsLyOZZRywB1bGwEVNR3qQFLuY3hGOuM-26J6IAvsgk,$-b0T_rqr5-vE1FiqwYOriXkTE0c8oMXj0a9uSeMnWVw,$qx3qMhu6 ba_ceFcv9ADA5J45CTUhhAMDz44eOVmJwR8,$ugwvSmk0LfVLpxk0cloQ6tQxjacdp0ojurri3pN4bl4,$notanevent2}'::text[])), syncv3_events.event_nid, syncv3_events.event_id, syncv3_events.before_state_snapshot_id, syncv3_events.event_replaces_nid, syncv3_events.room_id, syncv3_events.event_type, syncv 3_events.state_key, syncv3_events.prev_batch, syncv3_events.membership, syncv3_events.is_state, syncv3_events.event Inner Unique: true Filter: (syncv3_events.event_nid IS NULL) Rows Removed by Filter: 20 -> ProjectSet (cost=0.00..0.13 rows=22 width=32) (actual time=0.003..0.006 rows=22 loops=1) Output: unnest('{$fakeevent1,$4jteAg_T0BMXWiSRzZbptOF9zETeomXbJoDtg6zs9SU,$Iu-74VzJ0B4Xc6XgBDAKoWYZkFcUGCuJ_vLd47Cf4Ok,$OIqjRnK_rXBApl_8x4s0Iq--OH969cikkvjGYW5Seq8,$7EYkESAKR-t6wKLG7rjPyyVcZdQ-v0YQtaLbzrLJ2vI,$6LYIKlvAlC1ILMhIsWx-M9IDOarFBLM_Ni609bz1NDI,$w1sLe53ZqsRc9vebfLS8 JCUyq3ql9DRhN-lERWsZRHQ,$A6k3YCykEOHCzsbzxQjdupwNdt4hr4AeS6IWOnEG1H4,$K1EGNY8C-McIJJ5PBbd1GlUFMuw84S_QlbomqnYMiYY,$QgyACr1TujWs846B5tiAkbM9JkBEzUoMK6QgOfXDc64,$f2aefO_6yYMhSFRtQWYkTDdie87ldgTJD1h3JQ2F_3Q,$pYU_XgUXF0kQXvtSjG0boNdMHtXGBIFfwBDqxZuU1IE,$cZ5XF2o7ObQqROkJZWSubgm4cLgBX8j1nk rcrHv0rZ0,$wXTQjHKMTsMBqj3B4BLN40L0-7tz1NbG8WesWgGcjwY,$csN_i4UoBTk5jFzdDAxnKSmKg0Hp6U7Af_ifht07D-o,$NBsWMM0MJ_XcAt9gk9ZJxV3_v3tPfvmZ1azHI3Wb31A,$bf1IpaCxsC2Te4mWrBXcJZqesoh1HsssdaI_U1OzXuQ,$nsLyOZZRywB1bGwEVNR3qQFLuY3hGOuM-26J6IAvsgk,$-b0T_rqr5-vE1FiqwYOriXkTE0c8oMXj0a9uSeMnWVw,$qx3 qMhu6ba_ceFcv9ADA5J45CTUhhAMDz44eOVmJwR8,$ugwvSmk0LfVLpxk0cloQ6tQxjacdp0ojurri3pN4bl4,$notanevent2}'::text[]) -> Result (cost=0.00..0.01 rows=1 width=0) (actual time=0.001..0.001 rows=1 loops=1) -> Index Scan using syncv3_events_event_id_key on public.syncv3_events (cost=0.56..8.58 rows=1 width=658) (actual time=0.011..0.011 rows=1 loops=22) Output: syncv3_events.event_nid, syncv3_events.event_id, syncv3_events.before_state_snapshot_id, syncv3_events.event_replaces_nid, syncv3_events.room_id, syncv3_events.event_type, syncv3_events.state_key, syncv3_events.prev_batch, syncv3_events.membership, syncv3_events.is_s tate, syncv3_events.event Index Cond: (syncv3_events.event_id = (unnest('{$fakeevent1,$4jteAg_T0BMXWiSRzZbptOF9zETeomXbJoDtg6zs9SU,$Iu-74VzJ0B4Xc6XgBDAKoWYZkFcUGCuJ_vLd47Cf4Ok,$OIqjRnK_rXBApl_8x4s0Iq--OH969cikkvjGYW5Seq8,$7EYkESAKR-t6wKLG7rjPyyVcZdQ-v0YQtaLbzrLJ2vI,$6LYIKlvAlC1ILMhIsWx-M9IDOarFBLM_Ni 609bz1NDI,$w1sLe53ZqsRc9vebfLS8JCUyq3ql9DRhN-lERWsZRHQ,$A6k3YCykEOHCzsbzxQjdupwNdt4hr4AeS6IWOnEG1H4,$K1EGNY8C-McIJJ5PBbd1GlUFMuw84S_QlbomqnYMiYY,$QgyACr1TujWs846B5tiAkbM9JkBEzUoMK6QgOfXDc64,$f2aefO_6yYMhSFRtQWYkTDdie87ldgTJD1h3JQ2F_3Q,$pYU_XgUXF0kQXvtSjG0boNdMHtXGBIFfwBDqxZuU1IE,$cZ5 XF2o7ObQqROkJZWSubgm4cLgBX8j1nkrcrHv0rZ0,$wXTQjHKMTsMBqj3B4BLN40L0-7tz1NbG8WesWgGcjwY,$csN_i4UoBTk5jFzdDAxnKSmKg0Hp6U7Af_ifht07D-o,$NBsWMM0MJ_XcAt9gk9ZJxV3_v3tPfvmZ1azHI3Wb31A,$bf1IpaCxsC2Te4mWrBXcJZqesoh1HsssdaI_U1OzXuQ,$nsLyOZZRywB1bGwEVNR3qQFLuY3hGOuM-26J6IAvsgk,$-b0T_rqr5-vE1Fiqw YOriXkTE0c8oMXj0a9uSeMnWVw,$qx3qMhu6ba_ceFcv9ADA5J45CTUhhAMDz44eOVmJwR8,$ugwvSmk0LfVLpxk0cloQ6tQxjacdp0ojurri3pN4bl4,$notanevent2}'::text[]))) Filter: (syncv3_events.room_id = '!OGEhHVWSdvArJzumhm:matrix.org'::text) Query Identifier: 2484314538527637739 Planning Time: 0.163 ms Execution Time: 0.285 ms

AFAICS the only difference is an additional

Filter: (syncv3_events.room_id = '!OGEhHVWSdvArJzumhm:matrix.org'::text)

at the bottom of the second query plan, which removes no rows anyway. The time diff (~+40 microseconds) is positive. (Though this isn't a statistical analysis; that diff may be just noise.)

state/accumulator.go

kegsay · 2023-04-19T09:10:19Z

state/event_table.go

+	WITH maybe_unknown_events(event_id) AS (SELECT unnest($1::text[]))
+	SELECT event_id
+	FROM maybe_unknown_events LEFT JOIN syncv3_events USING(event_id)
+	WHERE event_nid IS NULL;`


Please do. There is event_table_test.go to do that.

state/event_table.go

sync2/poller.go

tests-e2e/gappy_state_test.go

tests-integration/poller_test.go

Need to work out some kind of local linter here

Suppose I build `MatchRoomTimelineMostRecent(1, []Event{C})`. this matcher was given a list `[A, B, C]` of three events. The matcher should allow that, but before this patch it wouldn't. I don't think this is the intention?

Some txns are commited, so the tests aren't isolated.

state/event_table_test.go

tests-integration/poller_test.go

This reverts commit 6c9aa09.

tests-integration/poller_test.go

Co-authored-by: kegsay <[email protected]>

David Robertson added 2 commits April 13, 2023 19:36

Integration test

38c1bda

E2E test case draft

1d59167

DMRobertson commented Apr 13, 2023

View reviewed changes

tests-e2e/client_test.go Show resolved Hide resolved

kegsay reviewed Apr 14, 2023

View reviewed changes

tests-integration/poller_test.go Show resolved Hide resolved

tests-integration/poller_test.go Outdated Show resolved Hide resolved

tests-e2e/gappy_state_test.go Outdated Show resolved Hide resolved

David Robertson added 6 commits April 14, 2023 20:22

WIP: update test cases

46059df

Update test case again

419e1ab

Don't reselect NIDs after initialising a room

4b90454

We fetch those NIDs above in `eventIDToNID` by adding a `RETURNING` clause to the `INSERT` statement.

We don't need eventIDs either

a2cba6e

Merge branch 'dmr/dont-select-after-insert-returning' into dmr/gappy-…

c19b561

…state

Introduce return struct for Initialise

666823d

DMRobertson force-pushed the dmr/gappy-state branch from e1a49bb to accec63 Compare April 17, 2023 19:10

David Robertson added 2 commits April 17, 2023 20:21

Initialise: handle state blocks from a gappy sync

4ba80d2

TODO comment

9fdb001

DMRobertson force-pushed the dmr/gappy-state branch from accec63 to 9fdb001 Compare April 17, 2023 19:21

Fixup test code

33b174d

DMRobertson commented Apr 17, 2023

View reviewed changes

state/accumulator.go Show resolved Hide resolved

DMRobertson commented Apr 17, 2023

View reviewed changes

David Robertson added 12 commits April 17, 2023 20:43

Propagate err message

ac9651c

Fix db query

5d8560c

Improve logging

b7a8e7d

Fix the logging

e3008e6

TODO note

2406da5

Return set of unknown events from the db

f28f7d0

Simply Initialise to bail out early

9d53a87

Propagate prepend events to poller, and inject

2bae784

Fix tests

5621423

Always look for unknown state events

9af0471

Better error checking in test

b5893b1

Fixup test

aa07188

DMRobertson mentioned this pull request Apr 18, 2023

Don't reselect NIDs after initialising a room #75

Closed

DMRobertson marked this pull request as ready for review April 18, 2023 18:06

DMRobertson changed the title ~~WIP: tests for gappy state behaviour~~ Inject state blocks from incremental polls into the room timeline Apr 18, 2023

kegsay requested changes Apr 19, 2023

View reviewed changes

David Robertson added 11 commits April 19, 2023 13:05

Use test helper instead

7f03a3d

SelectUnknownEventIDs expects a txn

e6aac43

Poller test description

b54ba7c

Test new events table query

5dd4315

Tidy up accumulator event ID handling

76066f9

Fix capitalisation

5a9fae1

Fix test function args too

b32e5da

Need to work out some kind of local linter here

Fix MatchRoomTimelineMostRecent

00dd396

Suppose I build `MatchRoomTimelineMostRecent(1, []Event{C})`. this matcher was given a list `[A, B, C]` of three events. The matcher should allow that, but before this patch it wouldn't. I don't think this is the intention?

Fix running unit test alongside other tests

bea931d

Some txns are commited, so the tests aren't isolated.

Tidyup unit test

1680952

Test memberships are updated after gappy sync

6e7d0cb

DMRobertson requested a review from kegsay April 19, 2023 17:39

kegsay approved these changes Apr 21, 2023

View reviewed changes

state/event_table_test.go Show resolved Hide resolved

state/event_table_test.go Show resolved Hide resolved

tests-integration/poller_test.go Show resolved Hide resolved

David Robertson added 5 commits April 22, 2023 00:24

Check the new query reports multiple unknown IDs

aa09d12

SelectByIDs has an ordering guarantee

8324f6e

Pass room ID to new query

6c9aa09

Revert "Pass room ID to new query"

2755384

This reverts commit 6c9aa09.

Update integration test

82d1d58

kegsay approved these changes Apr 24, 2023

View reviewed changes

tests-integration/poller_test.go Outdated Show resolved Hide resolved

typo

3274cf2

Co-authored-by: kegsay <[email protected]>

DMRobertson merged commit 206ee81 into main Apr 24, 2023

DMRobertson mentioned this pull request Apr 27, 2023

Deflake gappy state integration test #88

Merged

DMRobertson deleted the dmr/gappy-state branch May 3, 2023 10:48

nimau mentioned this pull request Jun 28, 2023

account_data returned by /_matrix/client/unstable/org.matrix.msc3575/sync are sometimes inconsistent #189

Closed

DMRobertson mentioned this pull request Jul 18, 2023

Persist large gappy state blocks as a single snapshot #211

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Inject state blocks from incremental polls into the room timeline #71

Inject state blocks from incremental polls into the room timeline #71

DMRobertson commented Apr 13, 2023 •

edited

Loading

DMRobertson commented Apr 17, 2023

DMRobertson Apr 17, 2023

kegsay Apr 19, 2023

kegsay Apr 19, 2023

DMRobertson Apr 19, 2023 •

edited

Loading

kegsay left a comment

kegsay Apr 19, 2023

DMRobertson Apr 19, 2023

DMRobertson Apr 19, 2023

kegsay Apr 21, 2023

DMRobertson Apr 24, 2023 •

edited

Loading

kegsay Apr 19, 2023

Inject state blocks from incremental polls into the room timeline #71

Inject state blocks from incremental polls into the room timeline #71

Conversation

DMRobertson commented Apr 13, 2023 • edited Loading

DMRobertson commented Apr 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DMRobertson Apr 19, 2023 • edited Loading

Choose a reason for hiding this comment

kegsay left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DMRobertson Apr 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DMRobertson commented Apr 13, 2023 •

edited

Loading

DMRobertson Apr 19, 2023 •

edited

Loading

DMRobertson Apr 24, 2023 •

edited

Loading