Add warning for multi workers

The default configuration for `s3_object_key_format` is racy with multiple workers: it can generate the same S3 path name across several workers. So change the logic to show a dataloss warning for multiple workers (not only for multiple threads). Signed-off-by: daipom <[email protected]> Reviewed-by: Fujimoto Seiji <[email protected]>
fluent · Mar 11, 2022 · f55d70f · f55d70f
1 parent 5bfcef0
commit f55d70f
Showing 1 changed file with 3 additions and 2 deletions.
diff --git a/lib/fluent/plugin/out_s3.rb b/lib/fluent/plugin/out_s3.rb
@@ -461,8 +461,9 @@ def check_s3_path_safety(conf)
         log.warn "The default value of s3_object_key_format will use ${chunk_id} instead of %{index} to avoid object conflict in v2"
       end
 
-      if (@buffer_config.flush_thread_count > 1) && ['${chunk_id}', '%{uuid_flush}'].none? { |key| @s3_object_key_format.include?(key) }
-        log.warn "No ${chunk_id} or %{uuid_flush} in s3_object_key_format with multiple flush threads. Recommend to set ${chunk_id} or %{uuid_flush} to avoid data lost by object conflict"
+      is_working_on_parallel = @buffer_config.flush_thread_count > 1 || system_config.workers > 1
+      if is_working_on_parallel && ['${chunk_id}', '%{uuid_flush}'].none? { |key| @s3_object_key_format.include?(key) }
+        log.warn "No ${chunk_id} or %{uuid_flush} in s3_object_key_format with multiple flush threads or multiple workers. Recommend to set ${chunk_id} or %{uuid_flush} to avoid data lost by object conflict"
       end
     end