chore(deps): Update aws-config crate

[jszwedko]

Also adds the aws-runtime crate as some types were moved there.

Closes: #20662

[datadog-vectordotdev]

Datadog Report

Branch report: jszwedko/update-aws-config
Commit report: cc7a433
Test service: vector

✅ 0 Failed, 7 Passed, 0 Skipped, 25.44s Total Time

[balonik]

@jszwedko could you please check the failing checks so this can get merged?

[jszwedko]

@jszwedko could you please check the failing checks so this can get merged?

Apologies for the delay. I had looked a bit before, the integration tests for AWS are failing, but I wasn't able to sort it out 😓 I'll try to take another pass soon.

[balonik]

@jszwedko maybe the issue is with the integration test itself? I am just guessing here, because I don't know what mockwatchlogs image does, but it seems like it is used to emulate AWS Cloudwatch, because back in the days (that container is 5 years old) localstack didn't support Cloudwatch Logs?
Using localstack for all Cloudwatch tests (metrics are using localstack) could do the trick.

[datadog-vectordotdev]

Datadog Report

Branch report: jszwedko/update-aws-config
Commit report: fbd978c
Test service: vector

✅ 0 Failed, 7 Passed, 0 Skipped, 25.47s Total Time

[jszwedko]

@jszwedko maybe the issue is with the integration test itself? I am just guessing here, because I don't know what mockwatchlogs image does, but it seems like it is used to emulate AWS Cloudwatch, because back in the days (that container is 5 years old) localstack didn't support Cloudwatch Logs? Using localstack for all Cloudwatch tests (metrics are using localstack) could do the trick.

Good observation! I opened a PR to switch from mockwatchlogs to localstack here: #21114. The integration tests are currently failing, but if we can fix that that might unblock this. I'll try to look at it again soon, but also happy to see someone else take a look if they have time.

[balonik]

@jszwedko now after we fixed the integration tests in another PR, could you please rebase this branch and see if it works now?

[jszwedko]

@jszwedko now after we fixed the integration tests in another PR, could you please rebase this branch and see if it works now?

👍 thanks for the bump on this. I merged in master 🤞

[jszwedko]

Looks like we are still seeing:

2024-09-06T20:47:02.214518Z  WARN request{request_id=1}: vector::sinks::util::retries: Retrying after error. error=CloudwatchError::DescribeLogStreams: dispatch failure internal_log_rate_limit=true

I'll try to dig into this later unless someone else gets to it first. It may just be a matter of a missing feature flag.

[chlunde]

Looks like we are still seeing:

2024-09-06T20:47:02.214518Z  WARN request{request_id=1}: vector::sinks::util::retries: Retrying after error. error=CloudwatchError::DescribeLogStreams: dispatch failure internal_log_rate_limit=true

I'll try to dig into this later unless someone else gets to it first. It may just be a matter of a missing feature flag.

@jszwedko I wonder if the logging should be improved here, there's also typically a message in the AWS error response, which tells what's actually wrong. vector never logs this. This makes it harder to diagnose any issue with the AWS config, like missing permissions. I haven't had the bandwidth to report a proper issue on this.

[jszwedko]

Looks like we are still seeing:

2024-09-06T20:47:02.214518Z  WARN request{request_id=1}: vector::sinks::util::retries: Retrying after error. error=CloudwatchError::DescribeLogStreams: dispatch failure internal_log_rate_limit=true

I'll try to dig into this later unless someone else gets to it first. It may just be a matter of a missing feature flag.

@jszwedko I wonder if the logging should be improved here, there's also typically a message in the AWS error response, which tells what's actually wrong. vector never logs this. This makes it harder to diagnose any issue with the AWS config, like missing permissions. I haven't had the bandwidth to report a proper issue on this.

I believe "dispatch failure" is an error returned by the SDK before it even makes the request, and so there wouldn't be a response from AWS, but I could be wrong.

[milas]

FYI dispatch failure internal_log_rate_limit=true doesn't appear to be new / introduced by this PR:

• Sink healthcheck failed for aws_s3 #19636
• S3/SQS source events fails sporadically #20314 (comment)
• vector_component_errors_total metric is empty even when there's an error on cloudwatch sink #21339

[jszwedko]

FYI dispatch failure internal_log_rate_limit=true doesn't appear to be new / introduced by this PR:

• Sink healthcheck failed for aws_s3 #19636
• S3/SQS source events fails sporadically #20314 (comment)
• vector_component_errors_total metric is empty even when there's an error on cloudwatch sink #21339

It is new in that the integration tests previously didn't hit it, but it does seem like it can potentially be hit in other circumstances too (thanks fro the links!).

[jszwedko]

/ci-run-integration-aws

[Denton-L]

rip, it looks like the integration tests are failing. I don't know Rust at all so sorry if this is an obvious statement, but would it be possible to wrap the error being returned with https://docs.rs/aws-sdk-cloudwatch/latest/aws_sdk_cloudwatch/error/struct.DisplayErrorContext.html so we can get more debug output?

[pront]

rip, it looks like the integration tests are failing. I don't know Rust at all so sorry if this is an obvious statement, but would it be possible to wrap the error being returned with https://docs.rs/aws-sdk-cloudwatch/latest/aws_sdk_cloudwatch/error/struct.DisplayErrorContext.html so we can get more debug output?

There are some details in the CI logs:

thread 'sinks::aws_cloudwatch_logs::integration_tests::cloudwatch_healthcheck' panicked at src/sinks/aws_cloudwatch_logs/integration_tests.rs:461:39:
called `Result::unwrap()` on an `Err` value: DescribeLogGroupsFailed { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: InterceptorError { kind: ReadAfterSerialization, interceptor_name: Some("UserAgentInterceptor"), source: Some(MissingApiMetadata) }, connection: Unknown } }) }

I think this needs further debugging.