clickhouse: prevent replicated tables from starting in read-only mode.

On start, ClickHouse compares the local state of each distributed table to its
distributed state. If it finds a discrepancy, it starts the table in read-only
mode. When this happens, oximeter can't write new records to the relevant
table(s). In the past, we've worked around this by manually instructing
ClickHouse using the `force_restore_data` sentinel file, but this requires
manual detection and intervention each time a table starts up in read-only
mode. This patch sets the `replicated_max_ratio_of_wrong_parts` flag to 1.0 so
that ClickHouse always accepts shared state, and never starts tables in
read-only mode.

As described in ClickHouse/ClickHouse#66527, this
appears to be a bug, or at least an ergonomic flaw, in ClickHouse. One replica
of a table can routinely fall behind the others, e.g. due to restart or network
partition, and shouldn't require manual intervention to start back up.

Part of #8595.

Author: jmcarp

clickhouse-admin/types/src/config.rs

@@ -201,9 +201,13 @@ impl ReplicaConfig {
         <max_tasks_in_queue>1000</max_tasks_in_queue>
      </distributed_ddl>
 
-     <!-- Disable sparse column serialization, which we expect to not need -->
      <merge_tree>
+        <!-- Disable sparse column serialization, which we expect to not need -->
         <ratio_of_defaults_for_sparse_serialization>1.0</ratio_of_defaults_for_sparse_serialization>
+
+        <!-- Prevent ClickHouse from setting distributed tables to read-only. -->
+        <!-- See https://github.com/oxidecomputer/omicron/issues/8595 for details. -->
+        <replicated_max_ratio_of_wrong_parts>1.0</replicated_max_ratio_of_wrong_parts>
     </merge_tree>
 {macros}
 {remote_servers}

clickhouse-admin/types/testutils/replica-server-config.xml

@@ -104,9 +104,13 @@
         <max_tasks_in_queue>1000</max_tasks_in_queue>
      </distributed_ddl>
 
-     <!-- Disable sparse column serialization, which we expect to not need -->
      <merge_tree>
+        <!-- Disable sparse column serialization, which we expect to not need -->
         <ratio_of_defaults_for_sparse_serialization>1.0</ratio_of_defaults_for_sparse_serialization>
+
+        <!-- Prevent ClickHouse from setting distributed tables to read-only. -->
+        <!-- See https://github.com/oxidecomputer/omicron/issues/8595 for details. -->
+        <replicated_max_ratio_of_wrong_parts>1.0</replicated_max_ratio_of_wrong_parts>
     </merge_tree>
 
     <macros>