-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Background process 'stats.notify_new_event' fails upsert due to missing index after DB restore #14470
Comments
What does postgres say if you connect to the db with |
@DMRobertson sure:
|
Are there any indexes mentioned just below that output? There should be a |
Also: have you always been using postgres, or have you migrated from sqlite in the past? |
@DMRobertson this is everything:
And yes always been a postgres setup. |
That's very odd. You haven't manually deleted any indices by any chance, have you? What does Also, what do the following queries return?
|
No I have not removed them.
|
@DMRobertson am I'm missing a migration? |
I'm not sure what's happened; I'll check everything next week. Related (indices on m.org differing from mainline synapse): #11893 |
Did you ever dump and restore your database? I think there's been cases of that losing indices in the past. |
@DMRobertson @clokep yes I did. Due to the utf8 database issue conversion I had to dump and recreate it. |
When I looked into the table I found a duplicate user_id. Thats why I could not apply the contraint. Why this happened I don't know.
I have removed the duplicate item and added the index. Now it looks like this:
and the error is also gone. Makes me wounder which other indexed might not have been applied like in #11893 |
That's not quite the same: we sometimes use matrix.org as a testing ground to see if new indices might be effective. It sounds like your backup and restore process wasn't watertight? For what it's worth, here's the list of indices that Synapse should have as of 1.72.0rc1. CREATE INDEX access_tokens_device_id ON access_tokens USING btree (user_id, device_id);
CREATE INDEX account_data_stream_id ON account_data USING btree (user_id, stream_id);
CREATE INDEX application_services_txns_id ON application_services_txns USING btree (as_id);
CREATE INDEX batch_events_batch_id ON batch_events USING btree (batch_id);
CREATE INDEX batch_events_room_id ON batch_events USING btree (room_id);
CREATE INDEX cache_invalidation_stream_by_instance_instance_index ON cache_invalidation_stream_by_instance USING btree (instance_name, stream_id);
CREATE INDEX current_state_delta_stream_idx ON current_state_delta_stream USING btree (stream_id);
CREATE INDEX current_state_events_member_index ON current_state_events USING btree (state_key) WHERE (type = 'm.room.member'::text);
CREATE INDEX deleted_pushers_stream_id ON deleted_pushers USING btree (stream_id);
CREATE INDEX destination_rooms_room_id ON destination_rooms USING btree (room_id);
CREATE INDEX device_auth_providers_devices ON device_auth_providers USING btree (user_id, device_id);
CREATE INDEX device_auth_providers_sessions ON device_auth_providers USING btree (auth_provider_id, auth_provider_session_id);
CREATE INDEX device_federation_inbox_sender_id ON device_federation_inbox USING btree (origin, message_id);
CREATE INDEX device_federation_outbox_destination_id ON device_federation_outbox USING btree (destination, stream_id);
CREATE INDEX device_federation_outbox_id ON device_federation_outbox USING btree (stream_id);
CREATE INDEX device_inbox_stream_id_user_id ON device_inbox USING btree (stream_id, user_id);
CREATE INDEX device_inbox_user_stream_id ON device_inbox USING btree (user_id, device_id, stream_id);
CREATE INDEX device_lists_changes_in_stream_id_unconverted ON device_lists_changes_in_room USING btree (stream_id) WHERE (NOT converted_to_destinations);
CREATE INDEX device_lists_outbound_pokes_id ON device_lists_outbound_pokes USING btree (destination, stream_id);
CREATE INDEX device_lists_outbound_pokes_stream ON device_lists_outbound_pokes USING btree (stream_id);
CREATE INDEX device_lists_outbound_pokes_user ON device_lists_outbound_pokes USING btree (destination, user_id);
CREATE INDEX device_lists_remote_resync_ts_idx ON device_lists_remote_resync USING btree (added_ts);
CREATE INDEX device_lists_stream_id ON device_lists_stream USING btree (stream_id, user_id);
CREATE INDEX device_lists_stream_user_id ON device_lists_stream USING btree (user_id, device_id);
CREATE INDEX e2e_cross_signing_signatures2_idx ON e2e_cross_signing_signatures USING btree (user_id, target_user_id, target_device_id);
CREATE INDEX evauth_edges_id ON event_auth USING btree (event_id);
CREATE INDEX ev_b_extrem_id ON event_backward_extremities USING btree (event_id);
CREATE INDEX ev_b_extrem_room ON event_backward_extremities USING btree (room_id);
CREATE INDEX ev_edges_prev_id ON event_edges USING btree (prev_event_id);
CREATE INDEX event_auth_chain_links_idx ON event_auth_chain_links USING btree (origin_chain_id, target_chain_id);
CREATE INDEX event_auth_chain_to_calculate_rm_id ON event_auth_chain_to_calculate USING btree (room_id);
CREATE INDEX event_contains_url_index ON events USING btree (room_id, topological_ordering, stream_ordering) WHERE ((contains_url = true) AND (outlier = false));
CREATE INDEX event_expiry_expiry_ts_idx ON event_expiry USING btree (expiry_ts);
CREATE INDEX event_failed_pull_attempts_room_id ON event_failed_pull_attempts USING btree (room_id);
CREATE INDEX event_labels_room_id_label_idx ON event_labels USING btree (room_id, label, topological_ordering);
CREATE INDEX event_push_actions_highlights_index ON event_push_actions USING btree (user_id, room_id, topological_ordering, stream_ordering) WHERE (highlight = 1);
CREATE INDEX event_push_actions_rm_tokens ON event_push_actions USING btree (user_id, room_id, topological_ordering, stream_ordering);
CREATE INDEX event_push_actions_room_id_user_id ON event_push_actions USING btree (room_id, user_id);
CREATE INDEX event_push_actions_staging_id ON event_push_actions_staging USING btree (event_id);
CREATE INDEX event_push_actions_stream_highlight_index ON event_push_actions USING btree (highlight, stream_ordering) WHERE (highlight = 0);
CREATE INDEX event_push_actions_stream_ordering ON event_push_actions USING btree (stream_ordering, user_id);
CREATE INDEX event_push_actions_thread_id_null ON event_push_actions USING btree (thread_id) WHERE (thread_id IS NULL);
CREATE INDEX event_push_actions_u_highlight ON event_push_actions USING btree (user_id, stream_ordering);
CREATE INDEX event_push_summary_thread_id_null ON event_push_summary USING btree (thread_id) WHERE (thread_id IS NULL);
CREATE INDEX event_relations_relates ON event_relations USING btree (relates_to_id, relation_type, aggregation_key);
CREATE INDEX event_search_ev_ridx ON event_search USING btree (room_id);
CREATE INDEX event_search_fts_idx ON event_search USING gin (vector);
CREATE INDEX events_order_room ON events USING btree (room_id, topological_ordering, stream_ordering);
CREATE INDEX events_room_stream ON events USING btree (room_id, stream_ordering);
CREATE INDEX events_ts ON events USING btree (origin_server_ts, stream_ordering);
CREATE INDEX event_to_state_groups_sg_index ON event_to_state_groups USING btree (state_group);
CREATE INDEX event_txn_id_ts ON event_txn_id USING btree (inserted_ts);
CREATE INDEX ev_extrem_id ON event_forward_extremities USING btree (event_id);
CREATE INDEX ev_extrem_room ON event_forward_extremities USING btree (room_id);
CREATE INDEX federation_inbound_events_staging_room ON federation_inbound_events_staging USING btree (room_id, received_ts);
CREATE INDEX ignored_users_ignored_user_id ON ignored_users USING btree (ignored_user_id);
CREATE INDEX insertion_event_edges_event_id ON insertion_event_edges USING btree (event_id);
CREATE INDEX insertion_event_edges_insertion_prev_event_id ON insertion_event_edges USING btree (insertion_prev_event_id);
CREATE INDEX insertion_event_edges_insertion_room_id ON insertion_event_edges USING btree (room_id);
CREATE INDEX insertion_event_extremities_room_id ON insertion_event_extremities USING btree (room_id);
CREATE INDEX insertion_events_next_batch_id ON insertion_events USING btree (next_batch_id);
CREATE INDEX insertion_events_room_id ON insertion_events USING btree (room_id);
CREATE INDEX local_current_membership_room_idx ON local_current_membership USING btree (room_id);
CREATE INDEX local_media_repository_thumbnails_media_id ON local_media_repository_thumbnails USING btree (media_id);
CREATE INDEX local_media_repository_url_cache_by_url_download_ts ON local_media_repository_url_cache USING btree (url, download_ts);
CREATE INDEX local_media_repository_url_cache_expires_idx ON local_media_repository_url_cache USING btree (expires_ts);
CREATE INDEX local_media_repository_url_cache_media_idx ON local_media_repository_url_cache USING btree (media_id);
CREATE INDEX local_media_repository_url_idx ON local_media_repository USING btree (created_ts) WHERE (url_cache IS NOT NULL);
CREATE INDEX login_tokens_auth_provider_idx ON login_tokens USING btree (auth_provider_id, auth_provider_session_id);
CREATE INDEX login_tokens_expiry_time_idx ON login_tokens USING btree (expiry_ts);
CREATE INDEX monthly_active_users_time_stamp ON monthly_active_users USING btree ("timestamp");
CREATE INDEX open_id_tokens_ts_valid_until_ms ON open_id_tokens USING btree (ts_valid_until_ms);
CREATE INDEX partial_state_events_room_id_idx ON partial_state_events USING btree (room_id);
CREATE INDEX presence_stream_id ON presence_stream USING btree (stream_id, user_id);
CREATE INDEX presence_stream_state_not_offline_idx ON presence_stream USING btree (state) WHERE (state <> 'offline'::text);
CREATE INDEX presence_stream_user_id ON presence_stream USING btree (user_id);
CREATE INDEX public_room_index ON rooms USING btree (is_public);
CREATE INDEX push_rules_enable_user_name ON push_rules_enable USING btree (user_name);
CREATE INDEX push_rules_stream_id ON push_rules_stream USING btree (stream_id);
CREATE INDEX push_rules_stream_user_stream_id ON push_rules_stream USING btree (user_id, stream_id);
CREATE INDEX push_rules_user_name ON push_rules USING btree (user_name);
CREATE INDEX receipts_linearized_id ON receipts_linearized USING btree (stream_id);
CREATE INDEX receipts_linearized_room_stream ON receipts_linearized USING btree (room_id, stream_id);
CREATE INDEX receipts_linearized_user ON receipts_linearized USING btree (user_id);
CREATE INDEX received_transactions_ts ON received_transactions USING btree (ts);
CREATE INDEX redactions_have_censored_ts ON redactions USING btree (received_ts) WHERE (NOT have_censored);
CREATE INDEX redactions_redacts ON redactions USING btree (redacts);
CREATE INDEX refresh_tokens_next_token_id ON refresh_tokens USING btree (next_token_id) WHERE (next_token_id IS NOT NULL);
CREATE INDEX room_account_data_stream_id ON room_account_data USING btree (user_id, stream_id);
CREATE INDEX room_aliases_id ON room_aliases USING btree (room_id);
CREATE INDEX room_alias_servers_alias ON room_alias_servers USING btree (room_alias);
CREATE INDEX room_memberships_room_id ON room_memberships USING btree (room_id);
CREATE INDEX room_memberships_user_id ON room_memberships USING btree (user_id);
CREATE INDEX room_memberships_user_room_forgotten ON room_memberships USING btree (user_id, room_id) WHERE (forgotten = 1);
CREATE INDEX room_retention_max_lifetime_idx ON room_retention USING btree (max_lifetime);
CREATE INDEX state_group_edges_prev_idx ON state_group_edges USING btree (prev_state_group);
CREATE INDEX state_groups_room_id_idx ON state_groups USING btree (room_id);
CREATE INDEX state_groups_state_type_idx ON state_groups_state USING btree (state_group, type, state_key);
CREATE INDEX stream_ordering_to_exterm_idx ON stream_ordering_to_exterm USING btree (stream_ordering);
CREATE INDEX stream_ordering_to_exterm_rm_idx ON stream_ordering_to_exterm USING btree (room_id, stream_ordering);
CREATE INDEX threads_ordering_idx ON threads USING btree (room_id, topological_ordering, stream_ordering);
CREATE INDEX threepid_validation_token_session_id ON threepid_validation_token USING btree (session_id);
CREATE INDEX user_daily_visits_ts_idx ON user_daily_visits USING btree ("timestamp");
CREATE INDEX user_daily_visits_uts_idx ON user_daily_visits USING btree (user_id, "timestamp");
CREATE INDEX user_directory_room_idx ON user_directory USING btree (room_id);
CREATE INDEX user_directory_search_fts_idx ON user_directory_search USING gin (vector);
CREATE INDEX user_external_ids_user_id_idx ON user_external_ids USING btree (user_id);
CREATE INDEX user_ips_device_id ON user_ips USING btree (user_id, device_id, last_seen);
CREATE INDEX user_ips_last_seen_only ON user_ips USING btree (last_seen);
CREATE INDEX user_ips_last_seen ON user_ips USING btree (user_id, last_seen);
CREATE INDEX users_creation_ts ON users USING btree (creation_ts);
CREATE INDEX users_have_local_media ON local_media_repository USING btree (user_id, created_ts);
CREATE INDEX users_in_public_rooms_r_idx ON users_in_public_rooms USING btree (room_id);
CREATE INDEX users_who_share_private_rooms_o_idx ON users_who_share_private_rooms USING btree (other_user_id);
CREATE INDEX users_who_share_private_rooms_r_idx ON users_who_share_private_rooms USING btree (room_id);
CREATE INDEX user_threepids_medium_address ON user_threepids USING btree (medium, address);
CREATE INDEX user_threepids_user_id ON user_threepids USING btree (user_id);
CREATE UNIQUE INDEX appservice_room_list_idx ON appservice_room_list USING btree (appservice_id, network_id, room_id);
CREATE UNIQUE INDEX blocked_rooms_idx ON blocked_rooms USING btree (room_id);
CREATE UNIQUE INDEX cache_invalidation_stream_by_instance_id ON cache_invalidation_stream_by_instance USING btree (stream_id);
CREATE UNIQUE INDEX chunk_events_event_id ON batch_events USING btree (event_id);
CREATE UNIQUE INDEX device_lists_changes_in_stream_id ON device_lists_changes_in_room USING btree (stream_id, room_id);
CREATE UNIQUE INDEX device_lists_outbound_last_success_unique_idx ON device_lists_outbound_last_success USING btree (destination, user_id);
CREATE UNIQUE INDEX device_lists_remote_cache_unique_id ON device_lists_remote_cache USING btree (user_id, device_id);
CREATE UNIQUE INDEX device_lists_remote_extremeties_unique_idx ON device_lists_remote_extremeties USING btree (user_id);
CREATE UNIQUE INDEX device_lists_remote_pending_user_device_id ON device_lists_remote_pending USING btree (user_id, device_id);
CREATE UNIQUE INDEX device_lists_remote_resync_idx ON device_lists_remote_resync USING btree (user_id);
CREATE UNIQUE INDEX e2e_cross_signing_keys_idx ON e2e_cross_signing_keys USING btree (user_id, keytype, stream_id);
CREATE UNIQUE INDEX e2e_cross_signing_keys_stream_idx ON e2e_cross_signing_keys USING btree (stream_id);
CREATE UNIQUE INDEX e2e_room_keys_versions_idx ON e2e_room_keys_versions USING btree (user_id, version);
CREATE UNIQUE INDEX e2e_room_keys_with_version_idx ON e2e_room_keys USING btree (user_id, version, room_id, session_id);
CREATE UNIQUE INDEX erased_users_user ON erased_users USING btree (user_id);
CREATE UNIQUE INDEX event_auth_chains_c_seq_index ON event_auth_chains USING btree (chain_id, sequence_number);
CREATE UNIQUE INDEX event_edges_event_id_prev_event_id_idx ON event_edges USING btree (event_id, prev_event_id);
CREATE UNIQUE INDEX event_push_summary_unique_index2 ON event_push_summary USING btree (user_id, room_id, thread_id);
CREATE UNIQUE INDEX event_relations_id ON event_relations USING btree (event_id);
CREATE UNIQUE INDEX event_search_event_id_idx ON event_search USING btree (event_id);
CREATE UNIQUE INDEX events_stream_ordering ON events USING btree (stream_ordering);
CREATE UNIQUE INDEX event_txn_id_event_id ON event_txn_id USING btree (event_id);
CREATE UNIQUE INDEX event_txn_id_txn_id ON event_txn_id USING btree (room_id, user_id, token_id, txn_id);
CREATE UNIQUE INDEX federation_inbound_events_staging_instance_event ON federation_inbound_events_staging USING btree (origin, event_id);
CREATE UNIQUE INDEX federation_stream_position_instance ON federation_stream_position USING btree (type, instance_name);
CREATE UNIQUE INDEX ignored_users_uniqueness ON ignored_users USING btree (ignorer_user_id, ignored_user_id);
CREATE UNIQUE INDEX insertion_event_extremities_event_id ON insertion_event_extremities USING btree (event_id);
CREATE UNIQUE INDEX insertion_events_event_id ON insertion_events USING btree (event_id);
CREATE UNIQUE INDEX instance_map_idx ON instance_map USING btree (instance_name);
CREATE UNIQUE INDEX local_current_membership_idx ON local_current_membership USING btree (user_id, room_id);
CREATE UNIQUE INDEX local_media_repository_thumbn_media_id_width_height_method_key ON local_media_repository_thumbnails USING btree (media_id, thumbnail_width, thumbnail_height, thumbnail_type, thumbnail_method);
CREATE UNIQUE INDEX monthly_active_users_users ON monthly_active_users USING btree (user_id);
CREATE UNIQUE INDEX ratelimit_override_idx ON ratelimit_override USING btree (user_id);
CREATE UNIQUE INDEX receipts_graph_unique_index ON receipts_graph USING btree (room_id, receipt_type, user_id) WHERE (thread_id IS NULL);
CREATE UNIQUE INDEX receipts_linearized_unique_index ON receipts_linearized USING btree (room_id, receipt_type, user_id) WHERE (thread_id IS NULL);
CREATE UNIQUE INDEX remote_media_repository_thumbn_media_origin_id_width_height_met ON remote_media_cache_thumbnails USING btree (media_origin, media_id, thumbnail_width, thumbnail_height, thumbnail_type, thumbnail_method);
CREATE UNIQUE INDEX room_stats_earliest_token_idx ON room_stats_earliest_token USING btree (room_id);
CREATE UNIQUE INDEX room_stats_state_room ON room_stats_state USING btree (room_id);
CREATE UNIQUE INDEX state_group_edges_unique_idx ON state_group_edges USING btree (state_group, prev_state_group);
CREATE UNIQUE INDEX stream_positions_idx ON stream_positions USING btree (stream_name, instance_name);
CREATE UNIQUE INDEX threepid_guest_access_tokens_index ON threepid_guest_access_tokens USING btree (medium, address);
CREATE UNIQUE INDEX user_directory_search_user_idx ON user_directory_search USING btree (user_id);
CREATE UNIQUE INDEX user_directory_user_idx ON user_directory USING btree (user_id);
CREATE UNIQUE INDEX user_filters_unique ON user_filters USING btree (user_id, filter_id);
CREATE UNIQUE INDEX user_ips_user_token_ip_unique_index ON user_ips USING btree (user_id, access_token, ip);
CREATE UNIQUE INDEX user_signature_stream_idx ON user_signature_stream USING btree (stream_id);
CREATE UNIQUE INDEX users_in_public_rooms_u_idx ON users_in_public_rooms USING btree (user_id, room_id);
CREATE UNIQUE INDEX users_who_share_private_rooms_u_idx ON users_who_share_private_rooms USING btree (user_id, other_user_id, room_id);
CREATE UNIQUE INDEX user_threepid_id_server_idx ON user_threepid_id_server USING btree (user_id, medium, address, id_server);
CREATE UNIQUE INDEX worker_locks_key ON worker_locks USING btree (lock_name, lock_key); I'd suggest you compare that against the list of indices from #14470 (comment) and see if any more are missing. It's worth doing it sooner rather than later, to prevent similar errors in the future. For future reference, I created the dump of indices by running the following from a 1.72.0rc1 checkout:
|
@johansmitsnl, I'm going to close this since it sounds like your errors have been sorted. If there are any other errors, or if you have trouble getting the right set of indices, please reopen and we can take another look. |
Description
Same as in #14219 getting the error in the log.
Steps to reproduce
It occurs very often in the log, not sure if it triggered by something.
Homeserver
homeserver
Synapse Version
Noticed since 1.71.0, now running 1.72.0rc1
Installation Method
Docker (matrixdotorg/synapse)
Platform
Debian docker
Relevant log output
Anything else that would be useful to know?
No response
The text was updated successfully, but these errors were encountered: