[S3 OUTPUT] odd errors when workers are set

[cdancy]

We're trying to get more throughput with our S3 output plugin as we noticed that, when using the rewrite_tag filter plugin (basically to split a single containers logs into X number of streams), that plugin is generating loads of emitter files on disk and no amount of increasing the upload_timeout on the S3 plugin seems to help. So we thought maybe using workers would work now as I know in the past the support wasn't there. We tried it and are getting odd results. Below are 2 different errors we see when setting workers 2 that we don't see otherwise.

The emitter buffer directory seems to grow endlessly with thousands of very small files until our persistent volume runs out of space.

Error 1:

2025/07/09 22:08:05] [ info] [output:s3:s3.0] Successfully uploaded object /logs/application/cluster-integ-test/rate-limit-testing/web-container/hello-world/2025-07-09/stdout/220300-mETRinTB.gz
[2025/07/09 22:08:05] [ warn] [s3_key] Invalid Tag delimiter: does not exist in tag. tag=15613978119417515792-14826480349264227722, format=/logs/application/cluster-integ-test/$TAG[1]/$TAG[2]/$TAG[3]/%Y-%m-%d/$TAG[4]/%H%M%S-$UUID.gz
[2025/07/09 22:08:05] [ warn] [s3_key] Invalid / Out of bounds tag part: At most 10 tag parts ($TAG[0] - $TAG[9]) can be processed. tag=15613978119417515792-14826480349264227722, format=/logs/application/cluster-integ-test/$TAG[1]/$TAG[2]/$TAG[3]/%Y-%m-%d/$TAG[4]/%H%M%S-$UUID.gz, delimiters=.
[2025/07/09 22:08:05] [ info] [output:s3:s3.0] Successfully uploaded object /logs/application/cluster-integ-test/15613978119417515792-14826480349264227722[1]/15613978119417515792-14826480349264227722[2]/15613978119417515792-14826480349264227722[3]/2025-07-09/15613978119417515792-14826480349264227722[4]/220300-qOpanvDL.gz
[2025/07/09 22:08:05] [engine] caught signal (SIGSEGV)
#0  0x55af3d1b8d66      in  cb_s3_flush() at plugins/out_s3/s3.c:2223
#1  0x55af3d6cd026      in  co_init() at lib/monkey/deps/flb_libco/amd64.c:117
#2  0xffffffffffffffff  in  ???() at ???:0

Error 2:

[2025/07/09 22:09:07] [ info] [input:storage_backlog:storage_backlog.16] queueing emitter.17:1-1752097667.555759947.flb
[2025/07/09 22:09:07] [ info] [input:storage_backlog:storage_backlog.16] queueing emitter.17:1-1752097667.619441841.flb
#0  0x56005ab26fde      in  s3_store_file_delete() at plugins/out_s3/s3_store.c:424
#1  0x56005ab22a09      in  upload_data() at plugins/out_s3/s3.c:1085
#2  0x56005ab24c86      in  send_upload_request() at plugins/out_s3/s3.c:1662
#3  0x56005ab24dbf      in  s3_upload_queue() at plugins/out_s3/s3.c:1719
#4  0x56005ab25d66      in  cb_s3_flush() at plugins/out_s3/s3.c:2223
#5  0x56005b03a026      in  co_init() at lib/monkey/deps/flb_libco/amd64.c:117
#6  0xffffffffffffffff  in  ???() at ???:0

@edsiper @PettitWesley any ideas?

[shadowshot-x]

I am facing the exact same issue with 4.0.7 for this

[sanchoooo]

Version 3.29, when workers set to 15, causes similar error. Removed workers and no more restarts.

[OUTPUT]
    Name                         s3
    Match                        *
    bucket                       supersecret-log
     workers                      15
    json_date_key                false
    use_put_object               Off
    # compression                  gzip
    s3_key_format                /%Y/%m/%d/$TAG/%H-$INDEX-$UUID.json

#0  0x561d23680ebf      in  cio_file_write() at lib/chunkio/src/cio_file.c:1008
#1  0x561d2340e83a      in  flb_fstore_file_append() at src/flb_fstore.c:289
#2  0x561d233cb5c6      in  s3_store_buffer_put() at plugins/out_s3/s3_store.c:190
#3  0x561d233ca5dd      in  buffer_chunk() at plugins/out_s3/s3.c:1676
#4  0x561d233ca5dd      in  cb_s3_flush() at plugins/out_s3/s3.c:2243
#5  0x561d2386be46      in  co_init() at lib/monkey/deps/flb_libco/amd64.c:117
#6  0xffffffffffffffff  in  ???() at ???:0