Unable to download firelens s3 config file

[drbly]

I am using the send-to-multiple-destinations example. I added the log-router container definition and edited our app container definition to use the awsfirelens log driver.

When I update my CloudFormation template with these changes it gets stuck starting and stopping tasks over and over again. Each task gives the error
Unable to download firelens s3 config file: unable to download s3 config extra.conf from bucket mybucket: MissingRegion: could not find region configuration

The task role has all of the S3 permissions it should need, and I tried adding those permissions to the execution role as well, but I still get the same error. So I don't think this is an S3 error, but I am not sure what the MissingRegion: could not find region configuration error is referring to.

[CarmenAPuccio]

Can you upload your IAM permissions here for review? Also, did you ensure that the permissions match the documentation as described here?

[drbly]

I added these statements to an S3 policy on both the task role and execution role

{
    "Effect": "Allow",
    "Action": [
        "s3:GetObject"
    ],
    "Resource": [
        "arn:aws:s3:::mybucket/extra.conf"
    ]
},
{
    "Effect": "Allow",
    "Action": [
        "s3:GetBucketLocation"
    ],
    "Resource": [
        "arn:aws:s3:::mybucket"
    ]
}

But I still get the same error.

[CarmenAPuccio]

Can you upload you CF template as well?

[drbly]

"apiBackendTask": {
	"Condition": "notFirstRun",
	"Type": "AWS::ECS::TaskDefinition",
	"Properties": {
		"ContainerDefinitions": [
			{
				"Environment": [
					{
						"Name": "SPRING_PROFILES_ACTIVE",
						"Value": {"Ref": "target"}
					},
					{
						"Name": "DB_ENDPOINT",
						"Value": {"Fn::FindInMap": [{"Ref": "target"}, {"Ref": "AWS::Region"}, "jdbc"]}
					},
					{
						"Name": "endpoint",
						"Value": {"Fn::Sub": "cognito-idp.${AWS::Region}.amazonaws.com"}
					},
					{
						"Name": "region",
						"Value": {"Ref": "AWS::Region"}
					},
					{
						"Name": "poolId",
						"Value": {"Fn::FindInMap": [{"Ref": "target"}, {"Ref": "AWS::Region"}, "poolId"]}
					},
					{
						"Name": "identityPoolId",
						"Value": {"Fn::FindInMap": [{"Ref": "target"}, {"Ref": "AWS::Region"}, "identityPoolId"]}
					},
					{
						"Name": "BUCKET_NAME",
						"Value": {"Fn::FindInMap": [{"Ref": "target"}, {"Ref": "AWS::Region"}, "bucket"]}
					}
				],
				"Essential": true,
				"Image": {"Ref": "beimage"},
				"LogConfiguration": {
					"LogDriver": "awsfirelens"
				},
				"MemoryReservation": 320,
				"Name": {
					"Fn::Sub": [
						"${name}-${target}",
						{
							"name": {
								"Fn::FindInMap": ["all", "all", "beName"]
							}
						}
					]
				},
				"PortMappings": [
					{
						"ContainerPort": {"Fn::FindInMap": ["all", "all", "bePort"]}
					}
				],
				"Secrets": [
					{
						"Name": "clientId",
						"ValueFrom": {"Fn::FindInMap": [{"Ref": "target"}, {"Ref": "AWS::Region"}, "clientId"]}
					}
				]
			},
			{
				"Essential": true,
				"Image": "amazon/aws-for-fluent-bit:latest",
				"Name": "log_router",
				"FirelensConfiguration": {
					"Type": "fluentbit",
					"Options": {
						"config-file-type": "s3",
						"config-file-value": "arn:aws:s3:::mybucket/extra.conf"
					}
				},
				"LogConfiguration": {
					"LogDriver": "awslogs",
					"Options": {
						"awslogs-group": {"Ref": "apiBackendLogGroup"},
						"awslogs-region": {"Ref": "AWS::Region"},
						"awslogs-stream-prefix": "accessLogs"
					}
				},
				"MemoryReservation": 50
			}
		],
		"ExecutionRoleArn": {"Ref": "executionRole"},
		"NetworkMode": "bridge",
		"RequiresCompatibilities": ["EC2"],
		"Tags": [
			{
				"Key": "Owner",
				"Value": {"Ref": "contact"}
			}
		],
		"TaskRoleArn": {"Ref": "containerRole"}
	}
}

[PettitWesley]

This is an ECS Agent issue: aws/amazon-ecs-agent#2354

A fix is being worked on.

[cwiggs]

ecs agent 1.37.0 fixed this issue. I tested it today and can confirm it now works.