Skip to content

Commit

Permalink
Use a more resilient s3 object name by default
Browse files Browse the repository at this point in the history
The s3 plugin uses a default object key that is problematic in a few ways.

1. It makes HEAD requests for each chunk it uploads, starting from 1 each time.
If you have uploaded 2000 log files within the same time slice, it will make
2001 HEAD requests to figure out if it exists.

fluent/fluent-plugin-s3#160

2. The above check is not thread-safe, and two threads can race and decide to
use the same `%{index}` value, with the loser of the race overwriting the chunk
from the winner.

fluent/fluent-plugin-s3#326

This is planned to change for v2, but there's no clear path to v2 right now.
The plugin does warn already if you use multiple threads and don't use either
`%{chunk_id}` or `%{uuid_hash}` in the object key.
  • Loading branch information
worr committed Oct 22, 2020
1 parent 9c81632 commit 3878f7c
Show file tree
Hide file tree
Showing 3 changed files with 6 additions and 2 deletions.
4 changes: 2 additions & 2 deletions pkg/sdk/model/output/s3.go
Original file line number Diff line number Diff line change
Expand Up @@ -127,8 +127,8 @@ type S3OutputConfig struct {
GrantFullControl string `json:"grant_full_control,omitempty"`
// The length of `%{hex_random}` placeholder(4-16)
HexRandomLength string `json:"hex_random_length,omitempty"`
// The format of S3 object keys (default: %{path}%{time_slice}_%{index}.%{file_extension})
S3ObjectKeyFormat string `json:"s3_object_key_format,omitempty"`
// The format of S3 object keys (default: %{path}%{time_slice}_%{uuid_hash}_%{index}.%{file_extension})
S3ObjectKeyFormat string `json:"s3_object_key_format,omitempty" plugin:"default:%{path}%{time_slice}_%{uuid_hash}_%{index}.%{file_extension}"`
// S3 bucket name
S3Bucket string `json:"s3_bucket"`
// Archive format on S3
Expand Down
1 change: 1 addition & 0 deletions pkg/sdk/model/output/s3_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ buffer:
@id test
path logs/${tag}/%Y/%m/%d/
s3_bucket logging-amazon-s3
s3_object_key_format %{path}%{time_slice}_%{uuid_hash}_%{index}.%{file_extension}
s3_region eu-central-1
<buffer tag,time>
@type file
Expand Down
3 changes: 3 additions & 0 deletions pkg/sdk/model/render/fluent_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -479,6 +479,7 @@ func TestRenderS3(t *testing.T) {
@id test
path /var/buffer
s3_bucket test_bucket
s3_object_key_format %{path}%{time_slice}_%{uuid_hash}_%{index}.%{file_extension}
<buffer tag,time>
@type file
path asd
Expand All @@ -502,6 +503,7 @@ func TestRenderS3(t *testing.T) {
@id test
path /var/buffer
s3_bucket test_bucket
s3_object_key_format %{path}%{time_slice}_%{uuid_hash}_%{index}.%{file_extension}
<instance_profile_credentials>
</instance_profile_credentials>`,
},
Expand All @@ -519,6 +521,7 @@ func TestRenderS3(t *testing.T) {
@id test
path /var/buffer
s3_bucket test_bucket
s3_object_key_format %{path}%{time_slice}_%{uuid_hash}_%{index}.%{file_extension}
<shared_credentials>
path e
profile_name f
Expand Down

0 comments on commit 3878f7c

Please sign in to comment.