Prometheus Remote Write to Amazon Prometheus?

[generate-me12]

Hi,
I have Vector running on an ec2 instance. I am collecting prometheus metrics on this instance and trying to use prometheus_remote_write sink to forward these metrics to Amazon Managed Prometheus workspace. The AMP workspace is created and uses an VPC interface endpoint. Usually we don't generate access keys, we just use instance roles or profiles. How does authentication for this sink work? I tried just supplying the assume_role in auth section but I'm running into:

WARN sink{component_kind="sink" component_id=prometheus sink component_type=prometheus_remote_write}: aws_config::imds::region: failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: other: No HTTP client was available to send this request. Enable the rustlscrate feature or configure a HTTP client to fix this. (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: "No HTTP client was available to send this request. Enable therustls crate feature or configure a HTTP client to fix this.", connection: Unknown } }) })) If I remove it then, healthcheck just says:

sh-4.2$ vector validate √ Loaded ["vector.yaml"] √ Component configuration √ Health check "print" 2025-01-22T19:57:35.514440Z ERROR vector::topology::builder: msg="Healthcheck failed." error=Unexpected status: 403 Forbidden component_kind="sink" component_type="prometheus_remote_write" component_id=prometheus_sink x Health check for "prometheus_sink" failed: Unexpected status: 403 Forbidden
Is providing the access keys the only way to authenticate ? Thank you.

[jszwedko]

Hmm, that sounds like it may be a bug. Can you share what Vector version you are trying?

[generate-me12]

Yes, the version number is vector 0.36.0

[jszwedko]

Ah, you might be running into one of the known issues from that release. See: https://vector.dev/releases/0.36.0/#known-issues

[generate-me12]

Ah, you might be running into one of the known issues from that release. See: https://vector.dev/releases/0.36.0/#known-issues

Regarding the 403 Forbidden error when healthcheck is enabled, does this mean that some aws credentials are missing? If I don't fill in the assume_role and just:
auth: strategy: aws region: us-west-2 aws: region: us-west-2, does vector infer from the environment variables or we have to fill out either assume_role or provide access key and secret access keys?

[jszwedko]

It should pick up credentials from the normal places (environment, IMDSv2, local config / credentials files). The Rust AWS SDK documentation has some details: https://docs.aws.amazon.com/sdk-for-rust/latest/dg/credentials.html

If you are having trouble with AWS credentials I recommend setting VECTOR_LOG=aws_config=trace to get verbose debug logs about the AWS authentication.

Also note that I think that the health check for Prometheus Remote Write doesn't work with Amazon Managed Prometheus. This is tracked here here: #8279

[generate-me12]

Ok, that's great. I'll look into those things. Thank you!