Working version for Lambda - AWS Managed Prometheus integration #248

pcolazurdo · 2022-06-14T17:16:50Z

This PR shows a fully working Lambda function based on Python sending metrics to AWS Managed Prometheus using an OTel Layer.

A few comments:

As force_flush is not defined yet to wait for the actual collector to flush the metrics we add a fake wait of 1 second at the end of the function call for it to succeed. This should be removed once the SDK/API team solve the discussions around force_flush functionality for FaaS
The current version of the Python opentelemetry-api module (1.11.1) has issues with the protobuf version which makes the library unusable (more info). To solve this we are asking the Function to use the python implementation of the protobuf compiler on the fly by adding an Environment variable PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION: python in the template.yaml for the function.
Also, the current version of the opentelemetry-api module (1.11.1) marks the metrics api as internal, that's why _metrics is used. Once the next stable version is released, I'll update the code with a new PR.

…es - pabcol@

…cification#1287

linux-foundation-easycla · 2022-06-14T17:16:52Z

✅ login: EddieEldridge / name: Medik (52268bf)
✅ login: vasireddy99 / name: Vasi Vasireddy (a9a4d80, 34e37f2, 7208181, c44e382)
❌ - login: @pcolazurdo / name: Pablo Colazurdo . The commit (5593c75, 25d8121, 85b97a7, 094b2be, add5813, d76c827, 33f9a28, aad8809, 28045f7, 5492ec2, e5d6194) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please submit a support request ticket.

NathanielRN · 2022-06-17T03:14:48Z

collector/config.yaml

+  awsprometheusremotewrite:
+    endpoint: "https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/ws-12345678-1234-1234-1234-123456789012/api/v1/remote_write"
+    aws_auth:
+      service: "aps"
+      region: "eu-west-1"


Is this a real endpoint that is used? If it's not, would this cause a problem if downstream repositories are using this file?

I suspect this should be replaced with actual workspace from customer.
I think one addition to the PR would be a readme/documentation about what is added and how to use it

This was a sample number to show how to fill the endpoint URL - I will add a comment

Rather than changing the default configuration here, perhaps include a configuration with the sample app that includes its use. Also, note that the awsprometheusremotewrite exporter is deprecated and should instead use the prometheusremotewrite exporter and sigv4auth extension for authentication.

Thank you @Aneurysm9 - I'll update those

python/sample-apps/template.yml

python/sample-apps/function/lambda_function.py

Aneurysm9 · 2022-06-22T22:50:27Z

collector/config.yaml

+  awsprometheusremotewrite:
+    endpoint: "https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/ws-12345678-1234-1234-1234-123456789012/api/v1/remote_write"
+    aws_auth:
+      service: "aps"
+      region: "eu-west-1"


Rather than changing the default configuration here, perhaps include a configuration with the sample app that includes its use. Also, note that the awsprometheusremotewrite exporter is deprecated and should instead use the prometheusremotewrite exporter and sigv4auth extension for authentication.

python/sample-apps/function/lambda_function.py

python/sample-apps/template.yml

python/sample-apps/function/lambda_function.py

NathanielRN · 2022-07-07T20:08:00Z

@pcolazurdo Are you able to fix the unit tests?

I'll defer to @Aneurysm9 for the approval here 🙂

* Update Collector version to 0.54.0 * Update to go version 1.18

…ry#260)

pcolazurdo · 2022-08-01T18:57:26Z

Updated to match latest commit on main. Some comments:

go.mod and go.sum are different because I had to include awsprometheuswriter. This is because the current collector doesn't seem to include the sigv4auth extension
config.yaml is modified on the collector/ path because there isn't a mechanism defined to override configuration. I would have used AOT_CONFIG_CONTENT environment variable as discussed here but this option doesn't seem to be available in the stripped-down version version of the collector in the project.

erichsueh3 · 2022-08-01T22:18:24Z

Updated to match latest commit on main. Some comments:

go.mod and go.sum are different because I had to include awsprometheuswriter. This is because the current collector doesn't seem to include the sigv4auth extension

config.yaml is modified on the collector/ path because there isn't a mechanism defined to override configuration. I would have used AOT_CONFIG_CONTENT environment variable as discussed here but this option doesn't seem to be available in the stripped-down version version of the collector in the project.

Hi @pcolazurdo, can you clarify on your first point? The sigv4auth extension has been included since 0.48, and the current collector version is latest, at 0.56.

pcolazurdo · 2022-08-02T11:45:35Z

@erichsueh3 I've tried initialising the extension and I just get an extension crash on Lambda. As soon as I add:

extensions:
  sigv4auth:
    region: "us-west-2"
...
service:
  extensions: [sigv4auth]

I get:

2022/08/02 11:41:36 Failed to start the extension: unable to start, otelcol state is 3

Also, looking in the source code in the repo for sigv4authextension, I couldn't find any reference to it, not even in the go.mod.

Aneurysm9 · 2022-08-02T16:15:53Z

collector/lambdacomponents/default.go

@@ -23,6 +23,7 @@ import (
 	"go.opentelemetry.io/collector/receiver/otlpreceiver"
 	"go.uber.org/multierr"

+	"github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsprometheusremotewriteexporter"


The sigv4auth extension should be added here rather than the awsprometheusremotewrite exporter.

I didn't find the right way to add it. Looking at the code in the collector directory and the code in https://github.com/aws-observability/aws-otel-collector regarding the extensions - specifically this , they look different (no ExtensionFactoryMap is present) so I didn't know what the strategy to add extensions in this stripped-down version was. Adding awsprometheuswriter was straightforward and didn't need major modifications in the existing collector. It looks like we I could to sync the code in https://github.com/aws-observability/aws-otel-collector/tree/2d4dd74947dc95826001a82e23a51c47a2d7f43b/pkg/lambdacomponents with the code here? I think this should be a totally different PR, am I right?

You can add the ExtensionFactoryMap in this PR, and add the sigv4 extension instead of the awsprometheusremotewrite exporter, it doesn't have to be in a separate PR

vasireddy99 · 2022-09-27T16:00:51Z

python/sample-apps/template.yml

      Environment:
        Variables:
          AWS_LAMBDA_EXEC_WRAPPER: /opt/python/otel-instrument
+          PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION: python  # needed because https://github.com/open-telemetry/opentelemetry-python/issues/2717 / when new release of library is released this should be solved and can be removed 


This issue is now resolved as the opentelemetry-python has pinned the protobuf version.

vasireddy99 · 2022-09-27T16:01:21Z

This example in the PR is really helpful.

prometheusremotewriteexporter with sigv4 auth extension can now be used as PR is merged.
I second this comment as it would be the be better case example as in the downstream repo used for integration tests.
force_flush() is now supported for metrics in v1.12.0, although this issue still exists in opentelemetry-instrumentation-aws-lambda.

pcolazurdo · 2023-01-06T11:37:29Z

Lot has changed in the project since I built this demo and now. It served its purpose but it is now mostly obsolete. I'll close this for now to avoid people getting confused, and try to create a new one in the coming weeks (no promise)

pcolazurdo added 5 commits June 8, 2022 08:43

Working version for Lambda - Prometheus integration - pabcol@

5593c75

Working version for Lambda - Prometheus integration - collector chang…

25d8121

…es - pabcol@

Removed sleep and using force_flush mechanism

85b97a7

Added delay after force_flush due to open-telemetry/opentelemetry-spe…

094b2be

…cification#1287

Working version for Lambda - AWS Managed Prometheus integration

add5813

pcolazurdo requested review from codeboten, wangzlei and NathanielRN as code owners June 14, 2022 17:16

Updated README and add CI badges (open-telemetry#249)

52268bf

NathanielRN reviewed Jun 17, 2022

View reviewed changes

python/sample-apps/template.yml Outdated Show resolved Hide resolved

NathanielRN reviewed Jun 17, 2022

View reviewed changes

python/sample-apps/function/lambda_function.py Show resolved Hide resolved

Aneurysm9 reviewed Jun 22, 2022

View reviewed changes

pcolazurdo and others added 2 commits June 25, 2022 19:25

reverted back to original timeout and python version

d76c827

Merge branch 'main' of github.com:pcolazurdo/opentelemetry-lambda

33f9a28

vasireddy99 and others added 8 commits July 8, 2022 09:41

Lambda Layer Update : Update Collector 0.54 (open-telemetry#256)

a9a4d80

* Update Collector version to 0.54.0 * Update to go version 1.18

Update Open-telemetry Java to 1.15.0 (open-telemetry#257)

34e37f2

Lambda Layer Update : OTel JS 1.4 and Contrib (open-telemetry#258)

7208181

Update JS Endpoint + use go1.17 in go.mod for collector (open-telemet…

c44e382

…ry#260)

Applying metrics demo to python sample app with collector to 0.54

aad8809

Applying metrics demo to python sample app with collector to 0.54

28045f7

Updated config.yaml to show remote endpoint configuration sample

5492ec2

Merge branch 'main' of codecommit://pabcol-opentelemetry-lambda

e5d6194

Aneurysm9 reviewed Aug 2, 2022

View reviewed changes

vasireddy99 reviewed Sep 27, 2022

View reviewed changes

pcolazurdo closed this Jan 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Working version for Lambda - AWS Managed Prometheus integration #248

Working version for Lambda - AWS Managed Prometheus integration #248

pcolazurdo commented Jun 14, 2022 •

edited

Loading

linux-foundation-easycla bot commented Jun 14, 2022 •

edited

Loading

NathanielRN Jun 17, 2022

jeromeinsf Jun 17, 2022

pcolazurdo Jun 17, 2022

Aneurysm9 Jun 22, 2022

pcolazurdo Jun 23, 2022

Aneurysm9 Jun 22, 2022

NathanielRN commented Jul 7, 2022

pcolazurdo commented Aug 1, 2022 •

edited

Loading

erichsueh3 commented Aug 1, 2022

pcolazurdo commented Aug 2, 2022 •

edited

Loading

Aneurysm9 Aug 2, 2022

pcolazurdo Aug 2, 2022

erichsueh3 Aug 2, 2022

vasireddy99 Sep 27, 2022

vasireddy99 commented Sep 27, 2022

pcolazurdo commented Jan 6, 2023

Working version for Lambda - AWS Managed Prometheus integration #248

Working version for Lambda - AWS Managed Prometheus integration #248

Conversation

pcolazurdo commented Jun 14, 2022 • edited Loading

linux-foundation-easycla bot commented Jun 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NathanielRN commented Jul 7, 2022

pcolazurdo commented Aug 1, 2022 • edited Loading

erichsueh3 commented Aug 1, 2022

pcolazurdo commented Aug 2, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vasireddy99 commented Sep 27, 2022

pcolazurdo commented Jan 6, 2023

pcolazurdo commented Jun 14, 2022 •

edited

Loading

linux-foundation-easycla bot commented Jun 14, 2022 •

edited

Loading

pcolazurdo commented Aug 1, 2022 •

edited

Loading

pcolazurdo commented Aug 2, 2022 •

edited

Loading