Removed ingesters blocks transfer support (#2996)

* Removed ingesters blocks transfer support

Signed-off-by: Marco Pracucci <[email protected]>

* Updated CHANGELOG

Signed-off-by: Marco Pracucci <[email protected]>

* Small improvements based on code review feedback

Signed-off-by: Marco Pracucci <[email protected]>

* Improved documentation based on feedback receive

Signed-off-by: Marco Pracucci <[email protected]>

* Simplified explaination about WAL loading

Signed-off-by: Marco Pracucci <[email protected]>

Author: pracucci

CHANGELOG.md

@@ -2,6 +2,11 @@
 
 ## master / unreleased
 
+* [CHANGE] Experimental blocks storage: removed the support to transfer blocks between ingesters on shutdown. When running the Cortex blocks storage, ingesters are expected to run with a persistent disk. The following metrics have been removed: #2996
+  * `cortex_ingester_sent_files`
+  * `cortex_ingester_received_files`
+  * `cortex_ingester_received_bytes_total`
+  * `cortex_ingester_sent_bytes_total`
 * [ENHANCEMENT] Query-tee: added a small tolerance to floating point sample values comparison. #2994
 * [BUGFIX] Query-frontend: Fixed rounding for incoming query timestamps, to be 100% Prometheus compatible. #2990
 

development/tsdb-blocks-storage-s3-single-binary/config/cortex.yaml

@@ -13,8 +13,6 @@ ingester_client:
     use_gzip_compression: true
 
 ingester:
-  max_transfer_retries: 1
-
   lifecycler:
     # We want to start immediately.
     join_after: 0

development/tsdb-blocks-storage-s3/config/cortex.yaml

@@ -13,8 +13,6 @@ ingester_client:
     use_gzip_compression: true
 
 ingester:
-  max_transfer_retries: 1
-
   lifecycler:
     # We want to start immediately.
     join_after: 0

docs/architecture.md

@@ -153,17 +153,18 @@ Incoming series are not immediately written to the storage but kept in memory an
 
 Ingesters contain a **lifecycler** which manages the lifecycle of an ingester and stores the **ingester state** in the [hash ring](#the-hash-ring). Each ingester could be in one of the following states:
 
-1. `PENDING` is an ingester's state when it just started and is waiting for a hand-over from another ingester that is `LEAVING`. If no hand-over occurs within the configured timeout period ("auto-join timeout", configurable via `-ingester.join-after` option), the ingester will join the ring with a new set of random tokens (ie. during a scale up). When hand-over process starts, state changes to `JOINING`.
-
-2. `JOINING` is an ingester's state in two situations. First, ingester will switch to a `JOINING` state from `PENDING` state after auto-join timeout. In this case, ingester will generate tokens, store them into the ring, optionally observe the ring for token conflicts and then move to `ACTIVE` state. Second, ingester will also switch into a `JOINING` state as a result of another `LEAVING` ingester initiating a hand-over process with `PENDING` (which then switches to `JOINING` state). `JOINING` ingester then receives series and tokens from `LEAVING` ingester, and if everything goes well, `JOINING` ingester switches to `ACTIVE` state. If hand-over process fails, `JOINING` ingester will move back to `PENDING` state and either wait for another hand-over or auto-join timeout.
-
-3. `ACTIVE` is an ingester's state when it is fully initialized. It may receive both write and read requests for tokens it owns.
-
-4. `LEAVING` is an ingester's state when it is shutting down. It cannot receive write requests anymore, while it could still receive read requests for series it has in memory. While in this state, the ingester may look for a `PENDING` ingester to start a hand-over process with, used to transfer the state from `LEAVING` ingester to the `PENDING` one, during a rolling update (`PENDING` ingester moves to `JOINING` state during hand-over process). If there is no new ingester to accept hand-over, ingester in `LEAVING` state will flush data to storage instead.
-
-5. `UNHEALTHY` is an ingester's state when it has failed to heartbeat to the ring's KV Store. While in this state, distributors skip the ingester while building the replication set for incoming series and the ingester does not receive write or read requests.
-
-For more information about the hand-over process, please check out the [Ingester hand-over](guides/ingester-handover.md) documentation.
+- **`PENDING`**<br />
+  The ingester has just started. While in this state, the ingester doesn't receive neither write and read requests, and could be waiting for time series data transfer from another ingester if running the chunks storage and the [hand-over](guides/ingesters-rolling-updates.md#chunks-storage-with-wal-disabled-hand-over) is enabled.
+- **`JOINING`**<br />
+  The ingester is starting up and joining the ring. While in this state the ingester doesn't receive neither write and read requests. The ingester will join the ring using tokens received by a leaving ingester as part of the [hand-over](guides/ingesters-rolling-updates.md#chunks-storage-with-wal-disabled-hand-over) process (if enabled), otherwise it could load tokens from disk (if `-ingester.tokens-file-path` is configured) or generate a set of new random ones. Finally, the ingester optionally observes the ring for tokens conflicts and then, once any conflict is resolved, will move to `ACTIVE` state.
+- **`ACTIVE`**<br />
+  The ingester is up and running. While in this state the ingester can receive both write and read requests.
+- **`LEAVING`**<br />
+  The ingester is shutting down and leaving the ring. While in this state the ingester doesn't receive write requests, while it could receive read requests.
+- **`UNHEALTHY`**<br />
+  The ingester has failed to heartbeat to the ring's KV Store. While in this state, distributors skip the ingester while building the replication set for incoming series and the ingester does not receive write or read requests.
+
+_The ingester states are interally used for different purposes, including the series hand-over process supported by the chunks storage. For more information about it, please check out the [Ingester hand-over](guides/ingesters-rolling-updates.md#chunks-storage-with-wal-disabled-hand-over) documentation._
 
 Ingesters are **semi-stateful**.
 

docs/blocks-storage/querier.md

@@ -475,9 +475,8 @@ blocks_storage:
     # CLI flag: -experimental.blocks-storage.tsdb.wal-compression-enabled
     [wal_compression_enabled: <boolean> | default = false]
 
-    # If true, and transfer of blocks on shutdown fails or is disabled,
-    # incomplete blocks are flushed to storage instead. If false, incomplete
-    # blocks will be reused after restart, and uploaded when finished.
+    # True to flush blocks to storage on shutdown. If false, incomplete blocks
+    # will be reused after restart.
     # CLI flag: -experimental.blocks-storage.tsdb.flush-blocks-on-shutdown
     [flush_blocks_on_shutdown: <boolean> | default = false]
 

docs/blocks-storage/store-gateway.md

@@ -502,9 +502,8 @@ blocks_storage:
     # CLI flag: -experimental.blocks-storage.tsdb.wal-compression-enabled
     [wal_compression_enabled: <boolean> | default = false]
 
-    # If true, and transfer of blocks on shutdown fails or is disabled,
-    # incomplete blocks are flushed to storage instead. If false, incomplete
-    # blocks will be reused after restart, and uploaded when finished.
+    # True to flush blocks to storage on shutdown. If false, incomplete blocks
+    # will be reused after restart.
     # CLI flag: -experimental.blocks-storage.tsdb.flush-blocks-on-shutdown
     [flush_blocks_on_shutdown: <boolean> | default = false]
 

docs/configuration/arguments.md

@@ -306,11 +306,11 @@ It also talks to a KVStore and has it's own copies of the same flags used by the
 
 - `-ingester.join-after`
 
-   How long to wait in PENDING state during the [hand-over process](../guides/ingester-handover.md). (default 0s)
+   How long to wait in PENDING state during the [hand-over process](../guides/ingesters-rolling-updates.md#chunks-storage-with-wal-disabled-hand-over) (supported only by the chunks storage). (default 0s)
 
 - `-ingester.max-transfer-retries`
 
-   How many times a LEAVING ingester tries to find a PENDING ingester during the [hand-over process](../guides/ingester-handover.md). Each attempt takes a second or so. Negative value or zero disables hand-over process completely. (default 10)
+   How many times a LEAVING ingester tries to find a PENDING ingester during the [hand-over process](../guides/ingesters-rolling-updates.md#chunks-storage-with-wal-disabled-hand-over) (supported only by the chunks storage). Negative value or zero disables hand-over process completely. (default 10)
 
 - `-ingester.normalise-tokens`
 

docs/configuration/config-file-reference.md

@@ -543,7 +543,8 @@ lifecycler:
   [availability_zone: <string> | default = ""]
 
 # Number of times to try and transfer chunks before falling back to flushing.
-# Negative value or zero disables hand-over.
+# Negative value or zero disables hand-over. This feature is supported only by
+# the chunks storage.
 # CLI flag: -ingester.max-transfer-retries
 [max_transfer_retries: <int> | default = 10]
 
@@ -3263,9 +3264,8 @@ tsdb:
   # CLI flag: -experimental.blocks-storage.tsdb.wal-compression-enabled
   [wal_compression_enabled: <boolean> | default = false]
 
-  # If true, and transfer of blocks on shutdown fails or is disabled, incomplete
-  # blocks are flushed to storage instead. If false, incomplete blocks will be
-  # reused after restart, and uploaded when finished.
+  # True to flush blocks to storage on shutdown. If false, incomplete blocks
+  # will be reused after restart.
   # CLI flag: -experimental.blocks-storage.tsdb.flush-blocks-on-shutdown
   [flush_blocks_on_shutdown: <boolean> | default = false]
 

docs/configuration/single-process-config-blocks-gossip-1.yaml

@@ -29,9 +29,6 @@ ingester_client:
     use_gzip_compression: true
 
 ingester:
-  # Disable blocks transfers on ingesters shutdown or rollout.
-  max_transfer_retries: 0
-
   lifecycler:
     # The address to advertise for this ingester.  Will be autodiscovered by
     # looking up address on eth0 or en0; can be specified if this fails.

docs/configuration/single-process-config-blocks-gossip-2.yaml

@@ -29,9 +29,6 @@ ingester_client:
     use_gzip_compression: true
 
 ingester:
-  # Disable blocks transfers on ingesters shutdown or rollout.
-  max_transfer_retries: 0
-
   lifecycler:
     # The address to advertise for this ingester.  Will be autodiscovered by
     # looking up address on eth0 or en0; can be specified if this fails.