Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Background process 'stats.notify_new_event' fails upsert due to missing index after DB restore #14470

Closed
johansmitsnl opened this issue Nov 16, 2022 · 14 comments
Labels
X-Needs-Info This issue is blocked awaiting information from the reporter

Comments

@johansmitsnl
Copy link

Description

Same as in #14219 getting the error in the log.

Steps to reproduce

It occurs very often in the log, not sure if it triggered by something.

Homeserver

homeserver

Synapse Version

Noticed since 1.71.0, now running 1.72.0rc1

Installation Method

Docker (matrixdotorg/synapse)

Platform

Debian docker

Relevant log output

2022-11-16 20:06:34,042 - synapse.storage.SQL - 442 - DEBUG - stats.notify_new_event-87 - [SQL time] {bulk_update_stats_delta-2a42} 0.000437 sec
2022-11-16 20:06:34,042 - synapse.storage.SQL - 416 - DEBUG - stats.notify_new_event-87 - [SQL] {bulk_update_stats_delta-2a42} INSERT INTO user_stats_current (user_id, completed_delta_stream_id, joined_rooms) VALUES (?, ?, ?) ON CONFLICT (user_id) DO UPDATE SET completed_delta_stream_id = EXCLUDED.completed_delta_stream_id, joined_rooms = EXCLUDED.joined_rooms + COALESCE(user_stats_current.joined_rooms, 0)
2022-11-16 20:06:34,042 - synapse.storage.SQL - 416 - DEBUG - stats.notify_new_event-87 - [SQL] {bulk_update_stats_delta-2a42} INSERT INTO user_stats_current (user_id, completed_delta_stream_id, joined_rooms) VALUES (?, ?, ?) ON CONFLICT (user_id) DO UPDATE SET completed_delta_stream_id = EXCLUDED.completed_delta_stream_id, joined_rooms = EXCLUDED.joined_rooms + COALESCE(user_stats_current.joined_rooms, 0)
2022-11-16 20:06:34,042 - synapse.storage.SQL - 421 - DEBUG - stats.notify_new_event-87 - [SQL values] {bulk_update_stats_delta-2a42} ['@ha-xx:xx.xx', 673097, 1]
2022-11-16 20:06:34,042 - synapse.storage.SQL - 421 - DEBUG - stats.notify_new_event-87 - [SQL values] {bulk_update_stats_delta-2a42} ['@ha-xx:xx.xx', 673097, 1]
2022-11-16 20:06:34,042 - synapse.storage.SQL - 438 - DEBUG - stats.notify_new_event-87 - [SQL FAIL] {bulk_update_stats_delta-2a42} there is no unique or exclusion constraint matching the ON CONFLICT specification

2022-11-16 20:06:34,042 - synapse.storage.SQL - 438 - DEBUG - stats.notify_new_event-87 - [SQL FAIL] {bulk_update_stats_delta-2a42} there is no unique or exclusion constraint matching the ON CONFLICT specification

2022-11-16 20:06:34,043 - synapse.storage.SQL - 442 - DEBUG - stats.notify_new_event-87 - [SQL time] {bulk_update_stats_delta-2a42} 0.000483 sec
2022-11-16 20:06:34,043 - synapse.storage.SQL - 442 - DEBUG - stats.notify_new_event-87 - [SQL time] {bulk_update_stats_delta-2a42} 0.000483 sec
2022-11-16 20:06:34,044 - synapse.metrics.background_process_metrics - 244 - ERROR - stats.notify_new_event-87 - Background process 'stats.notify_new_event' threw an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/synapse/metrics/background_process_metrics.py", line 242, in run
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/synapse/handlers/stats.py", line 74, in process
    await self._unsafe_process()
  File "/usr/local/lib/python3.9/site-packages/synapse/handlers/stats.py", line 127, in _unsafe_process
    await self.store.bulk_update_stats_delta(
  File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/stats.py", line 338, in bulk_update_stats_delta
    await self.db_pool.runInteraction(
  File "/usr/local/lib/python3.9/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.9/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.9/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.9/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.9/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.9/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.9/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.9/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.9/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/stats.py", line 322, in _bulk_update_stats_delta_txn
    self._update_stats_delta_txn(
  File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/stats.py", line 422, in _update_stats_delta_txn
    self._upsert_with_additive_relatives_txn(
  File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/stats.py", line 483, in _upsert_with_additive_relatives_txn
    txn.execute(sql, qargs)
  File "/usr/local/lib/python3.9/site-packages/synapse/storage/database.py", line 388, in execute
    self._do_execute(self.txn.execute, sql, *args)
  File "/usr/local/lib/python3.9/site-packages/synapse/storage/database.py", line 436, in _do_execute
    return func(sql, *args, **kwargs)
psycopg2.errors.InvalidColumnReference: there is no unique or exclusion constraint matching the ON CONFLICT specification

Anything else that would be useful to know?

No response

@DMRobertson
Copy link
Contributor

What does postgres say if you connect to the db with psql and run \d user_stats_current?

@DMRobertson DMRobertson added the X-Needs-Info This issue is blocked awaiting information from the reporter label Nov 17, 2022
@johansmitsnl
Copy link
Author

@DMRobertson sure:

                  Table "public.user_stats_current"
          Column           |  Type  | Collation | Nullable | Default 
---------------------------+--------+-----------+----------+---------
 user_id                   | text   |           | not null | 
 joined_rooms              | bigint |           | not null | 
 completed_delta_stream_id | bigint |           | not null | 

@DMRobertson
Copy link
Contributor

Are there any indexes mentioned just below that output? There should be a user_stats_current_pkey or similar.

@DMRobertson
Copy link
Contributor

Also: have you always been using postgres, or have you migrated from sqlite in the past?

@johansmitsnl
Copy link
Author

@DMRobertson this is everything:

psql -h localhost synapse synapse
psql (14.5)
Type "help" for help.

synapse=> \d user_stats_current
                  Table "public.user_stats_current"
          Column           |  Type  | Collation | Nullable | Default 
---------------------------+--------+-----------+----------+---------
 user_id                   | text   |           | not null | 
 joined_rooms              | bigint |           | not null | 
 completed_delta_stream_id | bigint |           | not null | 

synapse=>

And yes always been a postgres setup.

@DMRobertson
Copy link
Contributor

DMRobertson commented Nov 18, 2022

That's very odd. You haven't manually deleted any indices by any chance, have you?

What does \di show you from psql?

Also, what do the following queries return?

  • SELECT * FROM background_updates;
  • SELECT * FROM applied_schema_deltas;
  • SELECT * FROM schema_version;

@johansmitsnl
Copy link
Author

No I have not removed them.

synapse=> \di
                                                           List of relations
 Schema |                              Name                               | Type  |  Owner  |                   Table                   
--------+-----------------------------------------------------------------+-------+---------+-------------------------------------------
 public | access_tokens_device_id                                         | index | synapse | access_tokens
 public | access_tokens_pkey                                              | index | synapse | access_tokens
 public | access_tokens_token_key                                         | index | synapse | access_tokens
 public | account_data_stream_id                                          | index | synapse | account_data
 public | account_data_uniqueness                                         | index | synapse | account_data
 public | account_validity_pkey                                           | index | synapse | account_validity
 public | application_services_state_pkey                                 | index | synapse | application_services_state
 public | application_services_txns_as_id_txn_id_key                      | index | synapse | application_services_txns
 public | application_services_txns_id                                    | index | synapse | application_services_txns
 public | applied_module_schemas_module_name_file_key                     | index | synapse | applied_module_schemas
 public | applied_schema_deltas_version_file_key                          | index | synapse | applied_schema_deltas
 public | appservice_room_list_idx                                        | index | synapse | appservice_room_list
 public | appservice_stream_position_lock_key                             | index | synapse | appservice_stream_position
 public | background_updates_uniqueness                                   | index | synapse | background_updates
 public | batch_events_batch_id                                           | index | synapse | batch_events
 public | batch_events_room_id                                            | index | synapse | batch_events
 public | blocked_rooms_idx                                               | index | synapse | blocked_rooms
 public | cache_invalidation_stream_by_instance_id                        | index | synapse | cache_invalidation_stream_by_instance
 public | cache_invalidation_stream_by_instance_instance_index            | index | synapse | cache_invalidation_stream_by_instance
 public | chunk_events_event_id                                           | index | synapse | batch_events
 public | current_state_delta_stream_idx                                  | index | synapse | current_state_delta_stream
 public | current_state_events_event_id_key                               | index | synapse | current_state_events
 public | current_state_events_member_index                               | index | synapse | current_state_events
 public | current_state_events_room_id_type_state_key_key                 | index | synapse | current_state_events
 public | dehydrated_devices_pkey                                         | index | synapse | dehydrated_devices
 public | deleted_pushers_stream_id                                       | index | synapse | deleted_pushers
 public | destination_rooms_pkey                                          | index | synapse | destination_rooms
 public | destination_rooms_room_id                                       | index | synapse | destination_rooms
 public | destinations_pkey                                               | index | synapse | destinations
 public | device_auth_providers_devices                                   | index | synapse | device_auth_providers
 public | device_auth_providers_sessions                                  | index | synapse | device_auth_providers
 public | device_federation_inbox_sender_id                               | index | synapse | device_federation_inbox
 public | device_federation_outbox_destination_id                         | index | synapse | device_federation_outbox
 public | device_federation_outbox_id                                     | index | synapse | device_federation_outbox
 public | device_inbox_stream_id_user_id                                  | index | synapse | device_inbox
 public | device_inbox_user_stream_id                                     | index | synapse | device_inbox
 public | device_lists_changes_in_stream_id                               | index | synapse | device_lists_changes_in_room
 public | device_lists_changes_in_stream_id_unconverted                   | index | synapse | device_lists_changes_in_room
 public | device_lists_outbound_last_success_unique_idx                   | index | synapse | device_lists_outbound_last_success
 public | device_lists_outbound_pokes_id                                  | index | synapse | device_lists_outbound_pokes
 public | device_lists_outbound_pokes_stream                              | index | synapse | device_lists_outbound_pokes
 public | device_lists_outbound_pokes_user                                | index | synapse | device_lists_outbound_pokes
 public | device_lists_remote_cache_unique_id                             | index | synapse | device_lists_remote_cache
 public | device_lists_remote_extremeties_unique_idx                      | index | synapse | device_lists_remote_extremeties
 public | device_lists_remote_pending_pkey                                | index | synapse | device_lists_remote_pending
 public | device_lists_remote_pending_user_device_id                      | index | synapse | device_lists_remote_pending
 public | device_lists_remote_resync_idx                                  | index | synapse | device_lists_remote_resync
 public | device_lists_remote_resync_ts_idx                               | index | synapse | device_lists_remote_resync
 public | device_lists_stream_id                                          | index | synapse | device_lists_stream
 public | device_lists_stream_user_id                                     | index | synapse | device_lists_stream
 public | device_uniqueness                                               | index | synapse | devices
 public | e2e_cross_signing_keys_idx                                      | index | synapse | e2e_cross_signing_keys
 public | e2e_cross_signing_keys_stream_idx                               | index | synapse | e2e_cross_signing_keys
 public | e2e_cross_signing_signatures2_idx                               | index | synapse | e2e_cross_signing_signatures
 public | e2e_device_keys_json_uniqueness                                 | index | synapse | e2e_device_keys_json
 public | e2e_fallback_keys_json_uniqueness                               | index | synapse | e2e_fallback_keys_json
 public | e2e_one_time_keys_json_uniqueness                               | index | synapse | e2e_one_time_keys_json
 public | e2e_room_keys_versions_idx                                      | index | synapse | e2e_room_keys_versions
 public | e2e_room_keys_with_version_idx                                  | index | synapse | e2e_room_keys
 public | erased_users_user                                               | index | synapse | erased_users
 public | ev_b_extrem_id                                                  | index | synapse | event_backward_extremities
 public | ev_b_extrem_room                                                | index | synapse | event_backward_extremities
 public | ev_edges_prev_id                                                | index | synapse | event_edges
 public | ev_extrem_id                                                    | index | synapse | event_forward_extremities
 public | ev_extrem_room                                                  | index | synapse | event_forward_extremities
 public | evauth_edges_id                                                 | index | synapse | event_auth
 public | event_auth_chain_links_idx                                      | index | synapse | event_auth_chain_links
 public | event_auth_chain_to_calculate_pkey                              | index | synapse | event_auth_chain_to_calculate
 public | event_auth_chain_to_calculate_rm_id                             | index | synapse | event_auth_chain_to_calculate
 public | event_auth_chains_c_seq_index                                   | index | synapse | event_auth_chains
 public | event_auth_chains_pkey                                          | index | synapse | event_auth_chains
 public | event_backward_extremities_event_id_room_id_key                 | index | synapse | event_backward_extremities
 public | event_contains_url_index                                        | index | synapse | events
 public | event_edges_event_id_prev_event_id_idx                          | index | synapse | event_edges
 public | event_expiry_expiry_ts_idx                                      | index | synapse | event_expiry
 public | event_expiry_pkey                                               | index | synapse | event_expiry
 public | event_failed_pull_attempts_pkey                                 | index | synapse | event_failed_pull_attempts
 public | event_failed_pull_attempts_room_id                              | index | synapse | event_failed_pull_attempts
 public | event_forward_extremities_event_id_room_id_key                  | index | synapse | event_forward_extremities
 public | event_id_user_id_profile_tag_uniqueness                         | index | synapse | event_push_actions
 public | event_json_event_id_key                                         | index | synapse | event_json
 public | event_labels_pkey                                               | index | synapse | event_labels
 public | event_labels_room_id_label_idx                                  | index | synapse | event_labels
 public | event_push_actions_highlights_index                             | index | synapse | event_push_actions
 public | event_push_actions_rm_tokens                                    | index | synapse | event_push_actions
 public | event_push_actions_room_id_user_id                              | index | synapse | event_push_actions
 public | event_push_actions_staging_id                                   | index | synapse | event_push_actions_staging
 public | event_push_actions_stream_highlight_index                       | index | synapse | event_push_actions
 public | event_push_actions_stream_ordering                              | index | synapse | event_push_actions
 public | event_push_actions_thread_id_null                               | index | synapse | event_push_actions
 public | event_push_actions_u_highlight                                  | index | synapse | event_push_actions
 public | event_push_summary_last_receipt_stream_id_lock_key              | index | synapse | event_push_summary_last_receipt_stream_id
 public | event_push_summary_stream_ordering_lock_key                     | index | synapse | event_push_summary_stream_ordering
 public | event_push_summary_thread_id_null                               | index | synapse | event_push_summary
 public | event_push_summary_unique_index2                                | index | synapse | event_push_summary
 public | event_relations_id                                              | index | synapse | event_relations
 public | event_relations_relates                                         | index | synapse | event_relations
 public | event_reports_pkey                                              | index | synapse | event_reports
 public | event_search_ev_ridx                                            | index | synapse | event_search
 public | event_search_event_id_idx                                       | index | synapse | event_search
 public | event_search_fts_idx                                            | index | synapse | event_search
 public | event_to_state_groups_event_id_key                              | index | synapse | event_to_state_groups
 public | event_to_state_groups_sg_index                                  | index | synapse | event_to_state_groups
 public | event_txn_id_event_id                                           | index | synapse | event_txn_id
 public | event_txn_id_ts                                                 | index | synapse | event_txn_id
 public | event_txn_id_txn_id                                             | index | synapse | event_txn_id
 public | events_event_id_key                                             | index | synapse | events
 public | events_order_room                                               | index | synapse | events
 public | events_room_stream                                              | index | synapse | events
 public | events_stream_ordering                                          | index | synapse | events
 public | events_ts                                                       | index | synapse | events
 public | ex_outlier_stream_pkey                                          | index | synapse | ex_outlier_stream
 public | federation_inbound_events_staging_instance_event                | index | synapse | federation_inbound_events_staging
 public | federation_inbound_events_staging_room                          | index | synapse | federation_inbound_events_staging
 public | federation_stream_position_instance                             | index | synapse | federation_stream_position
 public | ignored_users_ignored_user_id                                   | index | synapse | ignored_users
 public | ignored_users_uniqueness                                        | index | synapse | ignored_users
 public | insertion_event_edges_event_id                                  | index | synapse | insertion_event_edges
 public | insertion_event_edges_insertion_prev_event_id                   | index | synapse | insertion_event_edges
 public | insertion_event_edges_insertion_room_id                         | index | synapse | insertion_event_edges
 public | insertion_event_extremities_event_id                            | index | synapse | insertion_event_extremities
 public | insertion_event_extremities_room_id                             | index | synapse | insertion_event_extremities
 public | insertion_events_event_id                                       | index | synapse | insertion_events
 public | insertion_events_next_batch_id                                  | index | synapse | insertion_events
 public | insertion_events_room_id                                        | index | synapse | insertion_events
 public | instance_map_idx                                                | index | synapse | instance_map
 public | instance_map_pkey                                               | index | synapse | instance_map
 public | local_current_membership_idx                                    | index | synapse | local_current_membership
 public | local_current_membership_room_idx                               | index | synapse | local_current_membership
 public | local_media_repository_media_id_key                             | index | synapse | local_media_repository
 public | local_media_repository_thumbn_media_id_width_height_method_key  | index | synapse | local_media_repository_thumbnails
 public | local_media_repository_thumbnails_media_id                      | index | synapse | local_media_repository_thumbnails
 public | local_media_repository_url_cache_by_url_download_ts             | index | synapse | local_media_repository_url_cache
 public | local_media_repository_url_cache_expires_idx                    | index | synapse | local_media_repository_url_cache
 public | local_media_repository_url_cache_media_idx                      | index | synapse | local_media_repository_url_cache
 public | local_media_repository_url_idx                                  | index | synapse | local_media_repository
 public | login_tokens_auth_provider_idx                                  | index | synapse | login_tokens
 public | login_tokens_expiry_time_idx                                    | index | synapse | login_tokens
 public | login_tokens_pkey                                               | index | synapse | login_tokens
 public | medium_address                                                  | index | synapse | user_threepids
 public | monthly_active_users_time_stamp                                 | index | synapse | monthly_active_users
 public | monthly_active_users_users                                      | index | synapse | monthly_active_users
 public | open_id_tokens_pkey                                             | index | synapse | open_id_tokens
 public | open_id_tokens_ts_valid_until_ms                                | index | synapse | open_id_tokens
 public | partial_state_events_event_id_key                               | index | synapse | partial_state_events
 public | partial_state_events_room_id_idx                                | index | synapse | partial_state_events
 public | partial_state_rooms_pkey                                        | index | synapse | partial_state_rooms
 public | partial_state_rooms_servers_room_id_server_name_key             | index | synapse | partial_state_rooms_servers
 public | presence_stream_id                                              | index | synapse | presence_stream
 public | presence_stream_state_not_offline_idx                           | index | synapse | presence_stream
 public | presence_stream_user_id                                         | index | synapse | presence_stream
 public | presence_user_id_key                                            | index | synapse | presence
 public | profiles_user_id_key                                            | index | synapse | profiles
 public | public_room_index                                               | index | synapse | rooms
 public | push_rules_enable_pkey                                          | index | synapse | push_rules_enable
 public | push_rules_enable_user_name                                     | index | synapse | push_rules_enable
 public | push_rules_enable_user_name_rule_id_key                         | index | synapse | push_rules_enable
 public | push_rules_pkey                                                 | index | synapse | push_rules
 public | push_rules_stream_id                                            | index | synapse | push_rules_stream
 public | push_rules_stream_user_stream_id                                | index | synapse | push_rules_stream
 public | push_rules_user_name                                            | index | synapse | push_rules
 public | push_rules_user_name_rule_id_key                                | index | synapse | push_rules
 public | pusher_throttle_pkey                                            | index | synapse | pusher_throttle
 public | pushers2_app_id_pushkey_user_name_key                           | index | synapse | pushers
 public | pushers2_pkey                                                   | index | synapse | pushers
 public | ratelimit_override_idx                                          | index | synapse | ratelimit_override
 public | receipts_graph_unique_index                                     | index | synapse | receipts_graph
 public | receipts_graph_uniqueness_thread                                | index | synapse | receipts_graph
 public | receipts_linearized_id                                          | index | synapse | receipts_linearized
 public | receipts_linearized_room_stream                                 | index | synapse | receipts_linearized
 public | receipts_linearized_unique_index                                | index | synapse | receipts_linearized
 public | receipts_linearized_uniqueness_thread                           | index | synapse | receipts_linearized
 public | receipts_linearized_user                                        | index | synapse | receipts_linearized
 public | received_transactions_transaction_id_origin_key                 | index | synapse | received_transactions
 public | received_transactions_ts                                        | index | synapse | received_transactions
 public | redactions_event_id_key                                         | index | synapse | redactions
 public | redactions_have_censored_ts                                     | index | synapse | redactions
 public | redactions_redacts                                              | index | synapse | redactions
 public | refresh_tokens_next_token_id                                    | index | synapse | refresh_tokens
 public | refresh_tokens_pkey                                             | index | synapse | refresh_tokens
 public | refresh_tokens_token_key                                        | index | synapse | refresh_tokens
 public | registration_tokens_token_key                                   | index | synapse | registration_tokens
 public | rejections_event_id_key                                         | index | synapse | rejections
 public | remote_media_cache_media_origin_media_id_key                    | index | synapse | remote_media_cache
 public | remote_media_repository_thumbn_media_origin_id_width_height_met | index | synapse | remote_media_cache_thumbnails
 public | room_account_data_stream_id                                     | index | synapse | room_account_data
 public | room_account_data_uniqueness                                    | index | synapse | room_account_data
 public | room_alias_servers_alias                                        | index | synapse | room_alias_servers
 public | room_aliases_id                                                 | index | synapse | room_aliases
 public | room_aliases_room_alias_key                                     | index | synapse | room_aliases
 public | room_depth_room_id_key                                          | index | synapse | room_depth
 public | room_memberships_event_id_key                                   | index | synapse | room_memberships
 public | room_memberships_room_id                                        | index | synapse | room_memberships
 public | room_memberships_user_id                                        | index | synapse | room_memberships
 public | room_memberships_user_room_forgotten                            | index | synapse | room_memberships
 public | room_retention_max_lifetime_idx                                 | index | synapse | room_retention
 public | room_retention_pkey                                             | index | synapse | room_retention
 public | room_stats_current_pkey                                         | index | synapse | room_stats_current
 public | room_stats_earliest_token_idx                                   | index | synapse | room_stats_earliest_token
 public | room_stats_state_room                                           | index | synapse | room_stats_state
 public | room_tag_revisions_uniqueness                                   | index | synapse | room_tags_revisions
 public | room_tag_uniqueness                                             | index | synapse | room_tags
 public | rooms_pkey                                                      | index | synapse | rooms
 public | schema_compat_version_lock_key                                  | index | synapse | schema_compat_version
 public | schema_version_lock_key                                         | index | synapse | schema_version
 public | server_keys_json_uniqueness                                     | index | synapse | server_keys_json
 public | server_signature_keys_server_name_key_id_key                    | index | synapse | server_signature_keys
 public | sessions_session_type_session_id_key                            | index | synapse | sessions
 public | state_events_event_id_key                                       | index | synapse | state_events
 public | state_group_edges_prev_idx                                      | index | synapse | state_group_edges
 public | state_group_edges_unique_idx                                    | index | synapse | state_group_edges
 public | state_groups_pkey                                               | index | synapse | state_groups
 public | state_groups_room_id_idx                                        | index | synapse | state_groups
 public | state_groups_state_type_idx                                     | index | synapse | state_groups_state
 public | stats_incremental_position_lock_key                             | index | synapse | stats_incremental_position
 public | stream_ordering_to_exterm_idx                                   | index | synapse | stream_ordering_to_exterm
 public | stream_ordering_to_exterm_rm_idx                                | index | synapse | stream_ordering_to_exterm
 public | stream_positions_idx                                            | index | synapse | stream_positions
 public | threads_ordering_idx                                            | index | synapse | threads
 public | threads_uniqueness                                              | index | synapse | threads
 public | threepid_guest_access_tokens_index                              | index | synapse | threepid_guest_access_tokens
 public | threepid_validation_session_pkey                                | index | synapse | threepid_validation_session
 public | threepid_validation_token_pkey                                  | index | synapse | threepid_validation_token
 public | threepid_validation_token_session_id                            | index | synapse | threepid_validation_token
 public | ui_auth_sessions_credentials_session_id_stage_type_key          | index | synapse | ui_auth_sessions_credentials
 public | ui_auth_sessions_ips_session_id_ip_user_agent_key               | index | synapse | ui_auth_sessions_ips
 public | ui_auth_sessions_session_id_key                                 | index | synapse | ui_auth_sessions
 public | user_daily_visits_ts_idx                                        | index | synapse | user_daily_visits
 public | user_daily_visits_uts_idx                                       | index | synapse | user_daily_visits
 public | user_directory_room_idx                                         | index | synapse | user_directory
 public | user_directory_search_fts_idx                                   | index | synapse | user_directory_search
 public | user_directory_search_user_idx                                  | index | synapse | user_directory_search
 public | user_directory_stream_pos_lock_key                              | index | synapse | user_directory_stream_pos
 public | user_directory_user_idx                                         | index | synapse | user_directory
 public | user_external_ids_auth_provider_external_id_key                 | index | synapse | user_external_ids
 public | user_external_ids_user_id_idx                                   | index | synapse | user_external_ids
 public | user_filters_unique                                             | index | synapse | user_filters
 public | user_ips_device_id                                              | index | synapse | user_ips
 public | user_ips_last_seen                                              | index | synapse | user_ips
 public | user_ips_last_seen_only                                         | index | synapse | user_ips
 public | user_ips_user_token_ip_unique_index                             | index | synapse | user_ips
 public | user_signature_stream_idx                                       | index | synapse | user_signature_stream
 public | user_threepid_id_server_idx                                     | index | synapse | user_threepid_id_server
 public | user_threepids_medium_address                                   | index | synapse | user_threepids
 public | user_threepids_user_id                                          | index | synapse | user_threepids
 public | users_creation_ts                                               | index | synapse | users
 public | users_have_local_media                                          | index | synapse | local_media_repository
 public | users_in_public_rooms_r_idx                                     | index | synapse | users_in_public_rooms
 public | users_in_public_rooms_u_idx                                     | index | synapse | users_in_public_rooms
 public | users_name_key                                                  | index | synapse | users
 public | users_to_send_full_presence_to_pkey                             | index | synapse | users_to_send_full_presence_to
 public | users_who_share_private_rooms_o_idx                             | index | synapse | users_who_share_private_rooms
 public | users_who_share_private_rooms_r_idx                             | index | synapse | users_who_share_private_rooms
 public | users_who_share_private_rooms_u_idx                             | index | synapse | users_who_share_private_rooms
 public | worker_locks_key                                                | index | synapse | worker_locks
(255 rows)
SELECT * FROM background_updates;
 update_name | progress_json | depends_on | ordering 
-------------+---------------+------------+----------
(0 rows)
SELECT * FROM applied_schema_deltas;
 version |                               file                                
---------+-------------------------------------------------------------------
      55 | 55/access_token_expiry.sql
      55 | 55/track_threepid_validations.sql
      55 | 55/users_alter_deactivated.sql
      56 | 56/current_state_events_membership.sql
      56 | 56/current_state_events_membership_mk2.sql
      56 | 56/room_membership_idx.sql
      56 | 56/add_spans_to_device_lists.sql
      56 | 56/destinations_failure_ts.sql
      56 | 56/destinations_retry_interval_type.sql.postgres
      56 | 56/devices_last_seen.sql
      56 | 56/fix_room_keys_index.sql
      56 | 56/redaction_censor.sql
      56 | 56/redaction_censor2.sql
      56 | 56/stats_separated.sql
      56 | 56/user_external_ids.sql
      56 | 56/users_in_public_rooms_idx.sql
      56 | 56/redaction_censor3_fix_update.sql.postgres
      56 | 56/drop_unused_event_tables.sql
      56 | 56/hidden_devices.sql
      56 | 56/public_room_list_idx.sql
      56 | 56/signing_keys.sql
      56 | 56/unique_user_filter_index.py
      56 | 56/delete_keys_from_deleted_backups.sql
      56 | 56/event_labels.sql
      56 | 56/event_labels_background_update.sql
      56 | 56/event_expiry.sql
      56 | 56/redaction_censor4.sql
      56 | 56/room_key_etag.sql
      56 | 56/room_retention.sql
      56 | 56/signing_keys_nonunique_signatures.sql
      56 | 56/device_stream_id_insert.sql
      56 | 56/nuke_empty_communities_from_db.sql
      56 | 56/remove_tombstoned_rooms_from_directory.sql
      56 | 56/state_group_room_idx.sql
      57 | 57/local_current_membership.py
      57 | 57/delete_old_current_state_events.sql
      57 | 57/device_list_remote_cache_stale.sql
      57 | 57/rooms_version_column.sql
      57 | 57/rooms_version_column_2.sql.postgres
      57 | 57/rooms_version_column_3.sql.postgres
      57 | 57/remove_sent_outbound_pokes.sql
      58 | 58/00background_update_ordering.sql
      58 | 58/02remove_dup_outbound_pokes.sql
      58 | 58/03persist_ui_auth.sql
      58 | 58/05cache_instance.sql.postgres
      58 | 58/06dlols_unique_idx.py
      58 | 58/08_media_safe_from_quarantine.sql.postgres
      58 | 58/10drop_local_rejections_stream.sql
      58 | 58/10federation_pos_instance_name.sql
      58 | 58/11user_id_seq.py
      58 | 58/12room_stats.sql
      58 | 58/07persist_ui_auth_ips.sql
      58 | 58/09shadow_ban.sql
      58 | 58/13remove_presence_allow_inbound.sql
      58 | 58/15unread_count.sql
      58 | 58/07add_method_to_thumbnail_constraint.sql.postgres
      58 | 58/10_pushrules_enabled_delete_obsolete.sql
      58 | 58/14events_instance_name.sql
      58 | 58/14events_instance_name.sql.postgres
      58 | 58/15_catchup_destination_rooms.sql
      58 | 58/16populate_stats_process_rooms_fix.sql
      58 | 58/17_catchup_last_successful.sql
      58 | 58/18stream_positions.sql
      58 | 58/11dehydration.sql
      58 | 58/11fallback.sql
      58 | 58/19instance_map.sql.postgres
      58 | 58/19txn_id.sql
      58 | 58/20instance_name_event_tables.sql
      58 | 58/20user_daily_visits.sql
      58 | 58/21as_device_stream.sql
      58 | 58/21drop_device_max_stream_id.sql
      58 | 58/22puppet_token.sql
      58 | 58/22users_have_local_media.sql
      58 | 58/23e2e_cross_signing_keys_idx.sql
      58 | 58/24drop_event_json_index.sql
      58 | 58/25user_external_ids_user_id_idx.sql
      58 | 58/26access_token_last_validated.sql
      58 | 58/27local_invites.sql
      58 | 58/28drop_last_used_column.sql.postgres
      58 | 58/28rejected_events_metadata.sql
      59 | 59/01ignored_user.py
      59 | 59/02shard_send_to_device.sql
      59 | 59/03shard_send_to_device_sequence.sql.postgres
      59 | 59/04_event_auth_chains.sql
      59 | 59/04_event_auth_chains.sql.postgres
      59 | 59/04drop_account_data.sql
      59 | 59/05cache_invalidation.sql
      59 | 59/06chain_cover_index.sql
      59 | 59/06shard_account_data.sql
      59 | 59/06shard_account_data.sql.postgres
      59 | 59/07shard_account_data_fix.sql
      59 | 59/08delete_pushers_for_deactivated_accounts.sql
      59 | 59/08delete_stale_pushers.sql
      59 | 59/09rejected_events_metadata.sql
      59 | 59/10delete_purged_chain_cover.sql
      59 | 59/11drop_thumbnail_constraint.sql.postgres
      59 | 59/12account_validity_token_used_ts_ms.sql
      59 | 59/12presence_stream_instance.sql
      59 | 59/12presence_stream_instance_seq.sql.postgres
      59 | 59/13users_to_send_full_presence_to.sql
      59 | 59/11add_knock_members_to_stats.sql
      59 | 59/15locks.sql
      59 | 59/16federation_inbound_staging.sql
      59 | 59/14refresh_tokens.sql
      60 | 60/01recreate_stream_ordering.sql.postgres
      60 | 60/02change_stream_ordering_columns.sql.postgres
      61 | 61/01change_appservices_txns.sql.postgres
      61 | 61/02drop_redundant_room_depth_index.sql
      61 | 61/02state_groups_state_n_distinct.sql.postgres
      61 | 61/03recreate_min_depth.py
      61 | 61/01insertion_event_lookups.sql
      62 | 62/01insertion_event_extremities.sql
      63 | 63/01create_registration_tokens.sql
      63 | 63/02delete_unlinked_email_pushers.sql
      63 | 63/03session_store.sql
      63 | 63/02populate-rooms-creator.sql
      63 | 63/04add_presence_stream_not_offline_index.sql
      64 | 64/01msc2716_chunk_to_batch_rename.sql.postgres
      65 | 65/01msc2716_insertion_event_edges.sql
      65 | 65/02_thread_relations.sql
      65 | 65/03remove_hidden_devices_from_device_inbox.sql
      65 | 65/04_local_group_updates.sql
      65 | 65/06remove_deleted_devices_from_device_inbox.sql
      65 | 65/05_remove_room_stats_historical_and_user_stats_historical.sql
      65 | 65/07_arbitrary_relations.sql
      65 | 65/08_device_inbox_background_updates.sql
      65 | 65/10_expirable_refresh_tokens.sql
      65 | 65/11_devices_auth_provider_session.sql
      67 | 67/01drop_public_room_list_stream.sql
      68 | 68/01event_columns.sql
      68 | 68/02_msc2409_add_device_id_appservice_stream_type.sql
      68 | 68/03_delete_account_data_for_deactivated_accounts.sql
      68 | 68/04_refresh_tokens_index_next_token_id.sql
      68 | 68/04partial_state_rooms.sql
      68 | 68/05partial_state_rooms_triggers.py
      68 | 68/06_msc3202_add_device_list_appservice_stream_type.sql
      69 | 69/01as_txn_seq.py
      69 | 69/01device_list_oubound_by_room.sql
      69 | 69/02cache_invalidation_index.sql
      70 | 70/08_state_group_edges_unique.sql
      70 | 70/01clean_table_purged_rooms.sql
      71 | 71/01rebuild_event_edges.sql.postgres
      71 | 71/01remove_noop_background_updates.sql
      71 | 71/02event_push_summary_unique.sql
      72 | 72/01event_push_summary_receipt.sql
      72 | 72/02event_push_actions_index.sql
      72 | 72/01add_room_type_to_state_stats.sql
      72 | 72/03bg_populate_events_columns.py
      72 | 72/03drop_event_reference_hashes.sql
      72 | 72/03remove_groups.sql
      72 | 72/04drop_column_application_services_state_last_txn.sql.postgres
      72 | 72/05remove_unstable_private_read_receipts.sql
      72 | 72/05receipts_event_stream_ordering.sql
      72 | 72/06add_consent_ts_to_users.sql
      72 | 72/06thread_notifications.sql
      72 | 72/07force_update_current_state_events_membership.py
      72 | 72/07thread_receipts.sql.postgres
      72 | 72/08begin_cache_invalidation_seq_at_2.sql.postgres
      72 | 72/08thread_receipts.sql
      73 | 73/01event_failed_pull_attempts.sql
      73 | 73/02room_id_indexes_for_purging.sql
      73 | 73/02add_pusher_enabled.sql
      73 | 73/03pusher_device_id.sql
      73 | 73/03users_approved_column.sql
      73 | 73/04partial_join_details.sql
      73 | 73/04pending_device_list_updates.sql
      73 | 73/05old_push_actions.sql.postgres
      73 | 73/06thread_notifications_thread_id_idx.sql
      73 | 73/08thread_receipts_non_null.sql.postgres
      73 | 73/09partial_joined_via_destination.sql
      73 | 73/09threads_table.sql
      73 | 73/10_update_sqlite_fts4_tokenizer.py
      73 | 73/10login_tokens.sql
      73 | 73/11event_search_room_id_n_distinct.sql.postgres
(174 rows)
SELECT * FROM schema_version;
 lock | version | upgraded 
------+---------+----------
 X    |      73 | t
(1 row)

@johansmitsnl
Copy link
Author

@DMRobertson am I'm missing a migration?

@DMRobertson
Copy link
Contributor

I'm not sure what's happened; I'll check everything next week.

Related (indices on m.org differing from mainline synapse): #11893

@clokep
Copy link
Member

clokep commented Nov 18, 2022

Did you ever dump and restore your database? I think there's been cases of that losing indices in the past.

@johansmitsnl
Copy link
Author

@DMRobertson @clokep yes I did. Due to the utf8 database issue conversion I had to dump and recreate it.

@johansmitsnl
Copy link
Author

johansmitsnl commented Nov 19, 2022

When I looked into the table I found a duplicate user_id. Thats why I could not apply the contraint. Why this happened I don't know.

synapse=> ALTER TABLE ONLY user_stats_current
synapse->     ADD CONSTRAINT user_stats_current_pkey PRIMARY KEY (user_id);
ERROR:  could not create unique index "user_stats_current_pkey"
DETAIL:  Key (user_id)=(@xxx.xxx:xxx.net) is duplicated.

I have removed the duplicate item and added the index. Now it looks like this:

\d user_stats_current
                  Table "public.user_stats_current"
          Column           |  Type  | Collation | Nullable | Default 
---------------------------+--------+-----------+----------+---------
 user_id                   | text   |           | not null | 
 joined_rooms              | bigint |           | not null | 
 completed_delta_stream_id | bigint |           | not null | 
Indexes:
    "user_stats_current_pkey" PRIMARY KEY, btree (user_id)

and the error is also gone.

Makes me wounder which other indexed might not have been applied like in #11893

@DMRobertson
Copy link
Contributor

Makes me wounder which other indexed might not have been applied like in #11893

That's not quite the same: we sometimes use matrix.org as a testing ground to see if new indices might be effective. It sounds like your backup and restore process wasn't watertight?

For what it's worth, here's the list of indices that Synapse should have as of 1.72.0rc1.

CREATE INDEX access_tokens_device_id ON access_tokens USING btree (user_id, device_id);
CREATE INDEX account_data_stream_id ON account_data USING btree (user_id, stream_id);
CREATE INDEX application_services_txns_id ON application_services_txns USING btree (as_id);
CREATE INDEX batch_events_batch_id ON batch_events USING btree (batch_id);
CREATE INDEX batch_events_room_id ON batch_events USING btree (room_id);
CREATE INDEX cache_invalidation_stream_by_instance_instance_index ON cache_invalidation_stream_by_instance USING btree (instance_name, stream_id);
CREATE INDEX current_state_delta_stream_idx ON current_state_delta_stream USING btree (stream_id);
CREATE INDEX current_state_events_member_index ON current_state_events USING btree (state_key) WHERE (type = 'm.room.member'::text);
CREATE INDEX deleted_pushers_stream_id ON deleted_pushers USING btree (stream_id);
CREATE INDEX destination_rooms_room_id ON destination_rooms USING btree (room_id);
CREATE INDEX device_auth_providers_devices ON device_auth_providers USING btree (user_id, device_id);
CREATE INDEX device_auth_providers_sessions ON device_auth_providers USING btree (auth_provider_id, auth_provider_session_id);
CREATE INDEX device_federation_inbox_sender_id ON device_federation_inbox USING btree (origin, message_id);
CREATE INDEX device_federation_outbox_destination_id ON device_federation_outbox USING btree (destination, stream_id);
CREATE INDEX device_federation_outbox_id ON device_federation_outbox USING btree (stream_id);
CREATE INDEX device_inbox_stream_id_user_id ON device_inbox USING btree (stream_id, user_id);
CREATE INDEX device_inbox_user_stream_id ON device_inbox USING btree (user_id, device_id, stream_id);
CREATE INDEX device_lists_changes_in_stream_id_unconverted ON device_lists_changes_in_room USING btree (stream_id) WHERE (NOT converted_to_destinations);
CREATE INDEX device_lists_outbound_pokes_id ON device_lists_outbound_pokes USING btree (destination, stream_id);
CREATE INDEX device_lists_outbound_pokes_stream ON device_lists_outbound_pokes USING btree (stream_id);
CREATE INDEX device_lists_outbound_pokes_user ON device_lists_outbound_pokes USING btree (destination, user_id);
CREATE INDEX device_lists_remote_resync_ts_idx ON device_lists_remote_resync USING btree (added_ts);
CREATE INDEX device_lists_stream_id ON device_lists_stream USING btree (stream_id, user_id);
CREATE INDEX device_lists_stream_user_id ON device_lists_stream USING btree (user_id, device_id);
CREATE INDEX e2e_cross_signing_signatures2_idx ON e2e_cross_signing_signatures USING btree (user_id, target_user_id, target_device_id);
CREATE INDEX evauth_edges_id ON event_auth USING btree (event_id);
CREATE INDEX ev_b_extrem_id ON event_backward_extremities USING btree (event_id);
CREATE INDEX ev_b_extrem_room ON event_backward_extremities USING btree (room_id);
CREATE INDEX ev_edges_prev_id ON event_edges USING btree (prev_event_id);
CREATE INDEX event_auth_chain_links_idx ON event_auth_chain_links USING btree (origin_chain_id, target_chain_id);
CREATE INDEX event_auth_chain_to_calculate_rm_id ON event_auth_chain_to_calculate USING btree (room_id);
CREATE INDEX event_contains_url_index ON events USING btree (room_id, topological_ordering, stream_ordering) WHERE ((contains_url = true) AND (outlier = false));
CREATE INDEX event_expiry_expiry_ts_idx ON event_expiry USING btree (expiry_ts);
CREATE INDEX event_failed_pull_attempts_room_id ON event_failed_pull_attempts USING btree (room_id);
CREATE INDEX event_labels_room_id_label_idx ON event_labels USING btree (room_id, label, topological_ordering);
CREATE INDEX event_push_actions_highlights_index ON event_push_actions USING btree (user_id, room_id, topological_ordering, stream_ordering) WHERE (highlight = 1);
CREATE INDEX event_push_actions_rm_tokens ON event_push_actions USING btree (user_id, room_id, topological_ordering, stream_ordering);
CREATE INDEX event_push_actions_room_id_user_id ON event_push_actions USING btree (room_id, user_id);
CREATE INDEX event_push_actions_staging_id ON event_push_actions_staging USING btree (event_id);
CREATE INDEX event_push_actions_stream_highlight_index ON event_push_actions USING btree (highlight, stream_ordering) WHERE (highlight = 0);
CREATE INDEX event_push_actions_stream_ordering ON event_push_actions USING btree (stream_ordering, user_id);
CREATE INDEX event_push_actions_thread_id_null ON event_push_actions USING btree (thread_id) WHERE (thread_id IS NULL);
CREATE INDEX event_push_actions_u_highlight ON event_push_actions USING btree (user_id, stream_ordering);
CREATE INDEX event_push_summary_thread_id_null ON event_push_summary USING btree (thread_id) WHERE (thread_id IS NULL);
CREATE INDEX event_relations_relates ON event_relations USING btree (relates_to_id, relation_type, aggregation_key);
CREATE INDEX event_search_ev_ridx ON event_search USING btree (room_id);
CREATE INDEX event_search_fts_idx ON event_search USING gin (vector);
CREATE INDEX events_order_room ON events USING btree (room_id, topological_ordering, stream_ordering);
CREATE INDEX events_room_stream ON events USING btree (room_id, stream_ordering);
CREATE INDEX events_ts ON events USING btree (origin_server_ts, stream_ordering);
CREATE INDEX event_to_state_groups_sg_index ON event_to_state_groups USING btree (state_group);
CREATE INDEX event_txn_id_ts ON event_txn_id USING btree (inserted_ts);
CREATE INDEX ev_extrem_id ON event_forward_extremities USING btree (event_id);
CREATE INDEX ev_extrem_room ON event_forward_extremities USING btree (room_id);
CREATE INDEX federation_inbound_events_staging_room ON federation_inbound_events_staging USING btree (room_id, received_ts);
CREATE INDEX ignored_users_ignored_user_id ON ignored_users USING btree (ignored_user_id);
CREATE INDEX insertion_event_edges_event_id ON insertion_event_edges USING btree (event_id);
CREATE INDEX insertion_event_edges_insertion_prev_event_id ON insertion_event_edges USING btree (insertion_prev_event_id);
CREATE INDEX insertion_event_edges_insertion_room_id ON insertion_event_edges USING btree (room_id);
CREATE INDEX insertion_event_extremities_room_id ON insertion_event_extremities USING btree (room_id);
CREATE INDEX insertion_events_next_batch_id ON insertion_events USING btree (next_batch_id);
CREATE INDEX insertion_events_room_id ON insertion_events USING btree (room_id);
CREATE INDEX local_current_membership_room_idx ON local_current_membership USING btree (room_id);
CREATE INDEX local_media_repository_thumbnails_media_id ON local_media_repository_thumbnails USING btree (media_id);
CREATE INDEX local_media_repository_url_cache_by_url_download_ts ON local_media_repository_url_cache USING btree (url, download_ts);
CREATE INDEX local_media_repository_url_cache_expires_idx ON local_media_repository_url_cache USING btree (expires_ts);
CREATE INDEX local_media_repository_url_cache_media_idx ON local_media_repository_url_cache USING btree (media_id);
CREATE INDEX local_media_repository_url_idx ON local_media_repository USING btree (created_ts) WHERE (url_cache IS NOT NULL);
CREATE INDEX login_tokens_auth_provider_idx ON login_tokens USING btree (auth_provider_id, auth_provider_session_id);
CREATE INDEX login_tokens_expiry_time_idx ON login_tokens USING btree (expiry_ts);
CREATE INDEX monthly_active_users_time_stamp ON monthly_active_users USING btree ("timestamp");
CREATE INDEX open_id_tokens_ts_valid_until_ms ON open_id_tokens USING btree (ts_valid_until_ms);
CREATE INDEX partial_state_events_room_id_idx ON partial_state_events USING btree (room_id);
CREATE INDEX presence_stream_id ON presence_stream USING btree (stream_id, user_id);
CREATE INDEX presence_stream_state_not_offline_idx ON presence_stream USING btree (state) WHERE (state <> 'offline'::text);
CREATE INDEX presence_stream_user_id ON presence_stream USING btree (user_id);
CREATE INDEX public_room_index ON rooms USING btree (is_public);
CREATE INDEX push_rules_enable_user_name ON push_rules_enable USING btree (user_name);
CREATE INDEX push_rules_stream_id ON push_rules_stream USING btree (stream_id);
CREATE INDEX push_rules_stream_user_stream_id ON push_rules_stream USING btree (user_id, stream_id);
CREATE INDEX push_rules_user_name ON push_rules USING btree (user_name);
CREATE INDEX receipts_linearized_id ON receipts_linearized USING btree (stream_id);
CREATE INDEX receipts_linearized_room_stream ON receipts_linearized USING btree (room_id, stream_id);
CREATE INDEX receipts_linearized_user ON receipts_linearized USING btree (user_id);
CREATE INDEX received_transactions_ts ON received_transactions USING btree (ts);
CREATE INDEX redactions_have_censored_ts ON redactions USING btree (received_ts) WHERE (NOT have_censored);
CREATE INDEX redactions_redacts ON redactions USING btree (redacts);
CREATE INDEX refresh_tokens_next_token_id ON refresh_tokens USING btree (next_token_id) WHERE (next_token_id IS NOT NULL);
CREATE INDEX room_account_data_stream_id ON room_account_data USING btree (user_id, stream_id);
CREATE INDEX room_aliases_id ON room_aliases USING btree (room_id);
CREATE INDEX room_alias_servers_alias ON room_alias_servers USING btree (room_alias);
CREATE INDEX room_memberships_room_id ON room_memberships USING btree (room_id);
CREATE INDEX room_memberships_user_id ON room_memberships USING btree (user_id);
CREATE INDEX room_memberships_user_room_forgotten ON room_memberships USING btree (user_id, room_id) WHERE (forgotten = 1);
CREATE INDEX room_retention_max_lifetime_idx ON room_retention USING btree (max_lifetime);
CREATE INDEX state_group_edges_prev_idx ON state_group_edges USING btree (prev_state_group);
CREATE INDEX state_groups_room_id_idx ON state_groups USING btree (room_id);
CREATE INDEX state_groups_state_type_idx ON state_groups_state USING btree (state_group, type, state_key);
CREATE INDEX stream_ordering_to_exterm_idx ON stream_ordering_to_exterm USING btree (stream_ordering);
CREATE INDEX stream_ordering_to_exterm_rm_idx ON stream_ordering_to_exterm USING btree (room_id, stream_ordering);
CREATE INDEX threads_ordering_idx ON threads USING btree (room_id, topological_ordering, stream_ordering);
CREATE INDEX threepid_validation_token_session_id ON threepid_validation_token USING btree (session_id);
CREATE INDEX user_daily_visits_ts_idx ON user_daily_visits USING btree ("timestamp");
CREATE INDEX user_daily_visits_uts_idx ON user_daily_visits USING btree (user_id, "timestamp");
CREATE INDEX user_directory_room_idx ON user_directory USING btree (room_id);
CREATE INDEX user_directory_search_fts_idx ON user_directory_search USING gin (vector);
CREATE INDEX user_external_ids_user_id_idx ON user_external_ids USING btree (user_id);
CREATE INDEX user_ips_device_id ON user_ips USING btree (user_id, device_id, last_seen);
CREATE INDEX user_ips_last_seen_only ON user_ips USING btree (last_seen);
CREATE INDEX user_ips_last_seen ON user_ips USING btree (user_id, last_seen);
CREATE INDEX users_creation_ts ON users USING btree (creation_ts);
CREATE INDEX users_have_local_media ON local_media_repository USING btree (user_id, created_ts);
CREATE INDEX users_in_public_rooms_r_idx ON users_in_public_rooms USING btree (room_id);
CREATE INDEX users_who_share_private_rooms_o_idx ON users_who_share_private_rooms USING btree (other_user_id);
CREATE INDEX users_who_share_private_rooms_r_idx ON users_who_share_private_rooms USING btree (room_id);
CREATE INDEX user_threepids_medium_address ON user_threepids USING btree (medium, address);
CREATE INDEX user_threepids_user_id ON user_threepids USING btree (user_id);
CREATE UNIQUE INDEX appservice_room_list_idx ON appservice_room_list USING btree (appservice_id, network_id, room_id);
CREATE UNIQUE INDEX blocked_rooms_idx ON blocked_rooms USING btree (room_id);
CREATE UNIQUE INDEX cache_invalidation_stream_by_instance_id ON cache_invalidation_stream_by_instance USING btree (stream_id);
CREATE UNIQUE INDEX chunk_events_event_id ON batch_events USING btree (event_id);
CREATE UNIQUE INDEX device_lists_changes_in_stream_id ON device_lists_changes_in_room USING btree (stream_id, room_id);
CREATE UNIQUE INDEX device_lists_outbound_last_success_unique_idx ON device_lists_outbound_last_success USING btree (destination, user_id);
CREATE UNIQUE INDEX device_lists_remote_cache_unique_id ON device_lists_remote_cache USING btree (user_id, device_id);
CREATE UNIQUE INDEX device_lists_remote_extremeties_unique_idx ON device_lists_remote_extremeties USING btree (user_id);
CREATE UNIQUE INDEX device_lists_remote_pending_user_device_id ON device_lists_remote_pending USING btree (user_id, device_id);
CREATE UNIQUE INDEX device_lists_remote_resync_idx ON device_lists_remote_resync USING btree (user_id);
CREATE UNIQUE INDEX e2e_cross_signing_keys_idx ON e2e_cross_signing_keys USING btree (user_id, keytype, stream_id);
CREATE UNIQUE INDEX e2e_cross_signing_keys_stream_idx ON e2e_cross_signing_keys USING btree (stream_id);
CREATE UNIQUE INDEX e2e_room_keys_versions_idx ON e2e_room_keys_versions USING btree (user_id, version);
CREATE UNIQUE INDEX e2e_room_keys_with_version_idx ON e2e_room_keys USING btree (user_id, version, room_id, session_id);
CREATE UNIQUE INDEX erased_users_user ON erased_users USING btree (user_id);
CREATE UNIQUE INDEX event_auth_chains_c_seq_index ON event_auth_chains USING btree (chain_id, sequence_number);
CREATE UNIQUE INDEX event_edges_event_id_prev_event_id_idx ON event_edges USING btree (event_id, prev_event_id);
CREATE UNIQUE INDEX event_push_summary_unique_index2 ON event_push_summary USING btree (user_id, room_id, thread_id);
CREATE UNIQUE INDEX event_relations_id ON event_relations USING btree (event_id);
CREATE UNIQUE INDEX event_search_event_id_idx ON event_search USING btree (event_id);
CREATE UNIQUE INDEX events_stream_ordering ON events USING btree (stream_ordering);
CREATE UNIQUE INDEX event_txn_id_event_id ON event_txn_id USING btree (event_id);
CREATE UNIQUE INDEX event_txn_id_txn_id ON event_txn_id USING btree (room_id, user_id, token_id, txn_id);
CREATE UNIQUE INDEX federation_inbound_events_staging_instance_event ON federation_inbound_events_staging USING btree (origin, event_id);
CREATE UNIQUE INDEX federation_stream_position_instance ON federation_stream_position USING btree (type, instance_name);
CREATE UNIQUE INDEX ignored_users_uniqueness ON ignored_users USING btree (ignorer_user_id, ignored_user_id);
CREATE UNIQUE INDEX insertion_event_extremities_event_id ON insertion_event_extremities USING btree (event_id);
CREATE UNIQUE INDEX insertion_events_event_id ON insertion_events USING btree (event_id);
CREATE UNIQUE INDEX instance_map_idx ON instance_map USING btree (instance_name);
CREATE UNIQUE INDEX local_current_membership_idx ON local_current_membership USING btree (user_id, room_id);
CREATE UNIQUE INDEX local_media_repository_thumbn_media_id_width_height_method_key ON local_media_repository_thumbnails USING btree (media_id, thumbnail_width, thumbnail_height, thumbnail_type, thumbnail_method);
CREATE UNIQUE INDEX monthly_active_users_users ON monthly_active_users USING btree (user_id);
CREATE UNIQUE INDEX ratelimit_override_idx ON ratelimit_override USING btree (user_id);
CREATE UNIQUE INDEX receipts_graph_unique_index ON receipts_graph USING btree (room_id, receipt_type, user_id) WHERE (thread_id IS NULL);
CREATE UNIQUE INDEX receipts_linearized_unique_index ON receipts_linearized USING btree (room_id, receipt_type, user_id) WHERE (thread_id IS NULL);
CREATE UNIQUE INDEX remote_media_repository_thumbn_media_origin_id_width_height_met ON remote_media_cache_thumbnails USING btree (media_origin, media_id, thumbnail_width, thumbnail_height, thumbnail_type, thumbnail_method);
CREATE UNIQUE INDEX room_stats_earliest_token_idx ON room_stats_earliest_token USING btree (room_id);
CREATE UNIQUE INDEX room_stats_state_room ON room_stats_state USING btree (room_id);
CREATE UNIQUE INDEX state_group_edges_unique_idx ON state_group_edges USING btree (state_group, prev_state_group);
CREATE UNIQUE INDEX stream_positions_idx ON stream_positions USING btree (stream_name, instance_name);
CREATE UNIQUE INDEX threepid_guest_access_tokens_index ON threepid_guest_access_tokens USING btree (medium, address);
CREATE UNIQUE INDEX user_directory_search_user_idx ON user_directory_search USING btree (user_id);
CREATE UNIQUE INDEX user_directory_user_idx ON user_directory USING btree (user_id);
CREATE UNIQUE INDEX user_filters_unique ON user_filters USING btree (user_id, filter_id);
CREATE UNIQUE INDEX user_ips_user_token_ip_unique_index ON user_ips USING btree (user_id, access_token, ip);
CREATE UNIQUE INDEX user_signature_stream_idx ON user_signature_stream USING btree (stream_id);
CREATE UNIQUE INDEX users_in_public_rooms_u_idx ON users_in_public_rooms USING btree (user_id, room_id);
CREATE UNIQUE INDEX users_who_share_private_rooms_u_idx ON users_who_share_private_rooms USING btree (user_id, other_user_id, room_id);
CREATE UNIQUE INDEX user_threepid_id_server_idx ON user_threepid_id_server USING btree (user_id, medium, address, id_server);
CREATE UNIQUE INDEX worker_locks_key ON worker_locks USING btree (lock_name, lock_key);

I'd suggest you compare that against the list of indices from #14470 (comment) and see if any more are missing. It's worth doing it sooner rather than later, to prevent similar errors in the future.


For future reference, I created the dump of indices by running the following from a 1.72.0rc1 checkout:

docker run --rm -e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres -p 5432:5432 postgres:11-alpine 

# separately:
scripts-dev/make_full_schema.sh -p postgres -n 73 -o synapse/storage/schema
$ cat synapse/storage/schema/{common,main,state}/full_schemas/73/*.postgres | \grep "CREATE.* INDEX " | sort | tee index

@DMRobertson DMRobertson changed the title there is no unique or exclusion constraint matching the ON CONFLICT specification Background process 'stats.notify_new_event' fails upsert due to missing index after DB restore Nov 21, 2022
@DMRobertson
Copy link
Contributor

@johansmitsnl, I'm going to close this since it sounds like your errors have been sorted.

If there are any other errors, or if you have trouble getting the right set of indices, please reopen and we can take another look.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
X-Needs-Info This issue is blocked awaiting information from the reporter
Projects
None yet
Development

No branches or pull requests

3 participants