Add Env variables in OTLP exporter #1101

jan25 · 2020-09-13T11:00:53Z

Description

Adds support for Env variables in OTLP trace and metrics exporter. Implemented subset of variables from exporter spec, other variables will be implemented as the usecases are added to otlp exporter.

Fixes #1004 (issue)

Type of change

New feature (non-breaking change which adds functionality)
This change requires a documentation update

How Has This Been Tested?

Write unit tests

Checklist:

Followed the style guidelines of this project
Changelogs have been updated
Unit tests have been added
Documentation has been updated

exporter/opentelemetry-exporter-otlp/src/opentelemetry/exporter/otlp/exporter.py

...ter/opentelemetry-exporter-otlp/src/opentelemetry/exporter/otlp/metrics_exporter/__init__.py

jan25 · 2020-09-21T17:46:05Z

We need to wait for this PR for spec updates, so we follow the correct default for OTEL_.._INSECURE env variable.

Also, for PR checklist:

I'll update readme file for exporter to mention about env variables
Unit tests could be tricky to implement, my idea is to start a grpc server in unit test with tls enabled and listening to specific endpoint/port and client sends a sample span to the server through using env vars. Any suggestions here?

exporter/opentelemetry-exporter-otlp/src/opentelemetry/exporter/otlp/exporter.py

lzchen · 2020-09-21T18:54:30Z

@jan25
Can you add an entry in the CHANGELOG?

Unit tests could be tricky to implement, my idea is to start a grpc server in unit test with tls enabled and listening to specific endpoint/port and client sends a sample span to the server through using env vars. Any suggestions here?

This seems like an integration test. For a unit test, you simply need to test the constructor populates the desired fields for each use case, as well as your file opening logic works as expected for credentials.

codeboten · 2020-10-01T15:44:59Z

@jan25 is this still a WIP?

jan25 · 2020-10-01T15:58:07Z

@codeboten yes, I'll add unit tests and unmark WIP. Also, need to revert small changes i made assuming exporter changes will be merged.

exporter/opentelemetry-exporter-otlp/src/opentelemetry/exporter/otlp/exporter.py

lzchen · 2020-10-12T15:05:38Z

exporter/opentelemetry-exporter-otlp/src/opentelemetry/exporter/otlp/exporter.py

        credentials: ChannelCredentials object for server authentication
        metadata: Metadata to send when exporting
    """

    def __init__(
        self,
-        endpoint: str = "localhost:55680",
+        endpoint: Optional[str] = None,
+        insecure: Optional[bool] = None,


Missing protocol, certificate_file in constructor, compression and timeout?

We could add env vars for protocol compression when those usecases are implemented. Currently exporter uses grpc only and no implementation for compression.

certificate_file is wrapped in ChannelCredentials object which is passed down from OTLPSpanExporter|MetricExporter. I'll change timeout logic to use env var too.

Compression is implemented in a separate PR https://github.com/open-telemetry/opentelemetry-python/pull/1141/files#diff-77db88bbe988c1d7f7d9f92f9e05400259a026e7203f7e9ee12aef76b3b0fdb3R142

exporter/opentelemetry-exporter-otlp/src/opentelemetry/exporter/otlp/exporter.py

lzchen

Some comments.

Signed-off-by: Abhilash Gnan <[email protected]>

codeboten · 2020-10-22T16:55:56Z

@jan25 looks like some tests are still failing, is the PR still a work in progress or is it ready for review?

jan25 · 2020-10-23T07:39:14Z

@codeboten Tests are failing on older Python versions due to unittest api changes, I've tried a few things but didn't succeed, I'll have to fix them. Otherwise this is ready to review

codeboten

Overall this looks pretty good! Just a request for change on the changelog. It would be good have information about the exporter environment variables in docs somewhere

codeboten · 2020-10-23T15:42:41Z

exporter/opentelemetry-exporter-otlp/CHANGELOG.md

 - Update OpenTelemetry protos to v0.5.0
  ([#1143](https://github.com/open-telemetry/opentelemetry-python/pull/1143))
+- Add Env variables in OTLP exporter
+  ([#1101](https://github.com/open-telemetry/opentelemetry-python/pull/1101))


this should be moved to unreleased

exporter/opentelemetry-exporter-otlp/CHANGELOG.md

Signed-off-by: Abhilash Gnan <[email protected]>

jan25 · 2020-10-27T18:37:44Z

@codeboten Thanks! I've fixed the tests and updated changelog too.

Do you suggest we add a link to exporter-spec in README.rst file? Perhaps we can also add about whats currently implemented?

codeboten · 2020-10-28T17:14:39Z

Do you suggest we add a link to exporter-spec in README.rst file? Perhaps we can also add about whats currently implemented?

I added a note to capture this work in the issue to review documentation, I don't want to block this PR on it and there are a bunch of places we need to review env variables anyways #1133 (comment)

codeboten

PR looks great!

NathanielRN

This works, just leaving my comments about some UX issues I had today.

NathanielRN · 2020-10-30T21:50:09Z

exporter/opentelemetry-exporter-otlp/src/opentelemetry/exporter/otlp/exporter.py

@@ -113,6 +116,16 @@ def _get_resource_data(
    return resource_data


+def _load_credential_from_file(filepath) -> ChannelCredentials:
+    try:
+        with open(filepath, "rb") as f:


Tried to use the OTLP Exporter today and I think I ran into an issue. If you don't have the OTEL_PYTHON_EXPORTER_OTLP_CERTIFICATE Env variable set, then the call to Configuration().EXPORTER_OTLP_CERTIFICATE

https://github.com/open-telemetry/opentelemetry-python/blob/master/exporter/opentelemetry-exporter-otlp/src/opentelemetry/exporter/otlp/exporter.py#L175

will return None instead of a String. Then this line will throw error

TypeError: expected str, bytes or os.PathLike object, not NoneType

which is not caught by FileNotFoundError below.

I suggest opening a new issue to track these problems instead of continuing an already closed PR thread.

NathanielRN · 2020-10-30T21:50:23Z

exporter/opentelemetry-exporter-otlp/src/opentelemetry/exporter/otlp/trace_exporter/__init__.py

+                "endpoint": endpoint
+                or Configuration().EXPORTER_OTLP_SPAN_ENDPOINT,
+                "insecure": insecure,
+                "credentials": credentials,


If I set OTLPExporter(insecure=True) this works fine, but if I leave that out, I get a message

File "/Users/enowell/git/opentelemetry-python/exporter/opentelemetry-exporter-otlp/src/opentelemetry/exporter/otlp/exporter.py", line 178, in __init__ self._client = self._stub(secure_channel(endpoint, credentials)) File "/Users/enowell/Library/Python/3.8/lib/python/site-packages/grpc/__init__.py", line 1943, in secure_channel if credentials._credentials is _insecure_channel_credentials: AttributeError: 'NoneType' object has no attribute '_credentials'

which makes sense because insecure=None and credentials=None. I think a follow up would be to throw an Error Message if insecure=None (or False) && credentials=None saying "Error: Either provide credentials or label as insecure."

@NathanielRN Thanks for reporting this. I also ran into this issue. Causes a failure in the example code.

jan25 · 2020-10-31T08:57:27Z

@NathanielRN thanks for the review! I'll work on fixing them and open a PR. I couldn't think of way to add such case in unittests, will try again.

…1101 (open-telemetry#1099) This addresses open-telemetry#1101 open-telemetry#752, and open-telemetry#1013 - Fix context loss when used in some async cases

jan25 requested a review from a team September 13, 2020 11:00

owais reviewed Sep 15, 2020

View reviewed changes

exporter/opentelemetry-exporter-otlp/src/opentelemetry/exporter/otlp/exporter.py Outdated Show resolved Hide resolved

owais reviewed Sep 15, 2020

View reviewed changes

exporter/opentelemetry-exporter-otlp/src/opentelemetry/exporter/otlp/exporter.py Outdated Show resolved Hide resolved

...ter/opentelemetry-exporter-otlp/src/opentelemetry/exporter/otlp/metrics_exporter/__init__.py Show resolved Hide resolved

codeboten added the release:required-for-ga To be resolved before GA release label Sep 17, 2020

jan25 force-pushed the issue-1004 branch from 1a7f03d to 0ac9bbc Compare September 20, 2020 14:32

jan25 changed the title ~~WIP: Add Env variables in OTLP exporter~~ Add Env variables in OTLP exporter Sep 21, 2020

lzchen reviewed Sep 21, 2020

View reviewed changes

exporter/opentelemetry-exporter-otlp/src/opentelemetry/exporter/otlp/exporter.py Outdated Show resolved Hide resolved

jan25 changed the title ~~Add Env variables in OTLP exporter~~ [WIP] Add Env variables in OTLP exporter Sep 21, 2020

codeboten mentioned this pull request Sep 24, 2020

[Issue #1107] Add gzip compression support for OTLP exporter #1141

Merged

3 tasks

jan25 changed the title ~~[WIP] Add Env variables in OTLP exporter~~ Add Env variables in OTLP exporter Oct 11, 2020

lzchen reviewed Oct 12, 2020

View reviewed changes

exporter/opentelemetry-exporter-otlp/src/opentelemetry/exporter/otlp/exporter.py Outdated Show resolved Hide resolved

lzchen reviewed Oct 12, 2020

View reviewed changes

exporter/opentelemetry-exporter-otlp/src/opentelemetry/exporter/otlp/exporter.py Show resolved Hide resolved

lzchen requested changes Oct 12, 2020

View reviewed changes

lzchen approved these changes Oct 14, 2020

View reviewed changes

jan25 added 10 commits October 16, 2020 11:28

Add env vars for endpoint, credentials, headers

416d55d

Read cert from file

a24a88d

Use configuration library

ced438a

Drop .env local setup

da5a7a2

Drop .env local setup;

045ad91

Ensure setting default insecure variable

710a60b

Lint

d6995de

Fix tests

47a5dcb

Lint

c704a55

Print error message

168771f

jan25 and others added 8 commits October 16, 2020 11:52

rename metadata to headers

af89d7d

Signed-off-by: Abhilash Gnan <[email protected]>

lint

f0d415d

Signed-off-by: Abhilash Gnan <[email protected]>

optional args

f9ae58d

Signed-off-by: Abhilash Gnan <[email protected]>

lint

4e70e10

Signed-off-by: Abhilash Gnan <[email protected]>

lint

eff69d4

Signed-off-by: Abhilash Gnan <[email protected]>

Add timeout env variable

82e78de

Signed-off-by: Abhilash Gnan <[email protected]>

lint

49b2ed6

Signed-off-by: Abhilash Gnan <[email protected]>

Merge branch 'master' into issue-1004

bdaebb1

Merge branch 'master' into issue-1004

a9f5bae

codeboten suggested changes Oct 23, 2020

View reviewed changes

jan25 added 3 commits October 27, 2020 19:20

Fix unittest api issues

500f8c7

Signed-off-by: Abhilash Gnan <[email protected]>

Fix changelog

5a5708c

Signed-off-by: Abhilash Gnan <[email protected]>

Remove unused imports

593f459

Signed-off-by: Abhilash Gnan <[email protected]>

alrex added 2 commits October 28, 2020 10:09

Merge branch 'master' into issue-1004

6f37169

Update CHANGELOG.md

9483813

codeboten approved these changes Oct 28, 2020

View reviewed changes

Merge branch 'master' into issue-1004

2ff2492

codeboten merged commit 42abfb7 into open-telemetry:master Oct 28, 2020

jan25 deleted the issue-1004 branch October 28, 2020 20:21

NathanielRN reviewed Oct 30, 2020

View reviewed changes

jan25 mentioned this pull request Nov 9, 2020

OTLP exporter: Handle error case when no credentials supplied #1366

Merged

8 tasks

chenqi0805 mentioned this pull request Nov 13, 2020

Upgrade proto to 0.10.0 opendistro-for-elasticsearch/data-prepper#154

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Env variables in OTLP exporter #1101

Add Env variables in OTLP exporter #1101

jan25 commented Sep 13, 2020 •

edited

Loading

jan25 commented Sep 21, 2020

lzchen commented Sep 21, 2020

codeboten commented Oct 1, 2020

jan25 commented Oct 1, 2020 •

edited

Loading

lzchen Oct 12, 2020

jan25 Oct 13, 2020

codeboten Oct 14, 2020

lzchen left a comment

codeboten commented Oct 22, 2020

jan25 commented Oct 23, 2020

codeboten left a comment

codeboten Oct 23, 2020

jan25 commented Oct 27, 2020

codeboten commented Oct 28, 2020

codeboten left a comment

NathanielRN left a comment

NathanielRN Oct 30, 2020 •

edited

Loading

lzchen Nov 2, 2020

NathanielRN Oct 30, 2020

rgstephens Nov 7, 2020

jan25 commented Oct 31, 2020

Add Env variables in OTLP exporter #1101

Add Env variables in OTLP exporter #1101

Conversation

jan25 commented Sep 13, 2020 • edited Loading

Description

Type of change

How Has This Been Tested?

Checklist:

jan25 commented Sep 21, 2020

lzchen commented Sep 21, 2020

codeboten commented Oct 1, 2020

jan25 commented Oct 1, 2020 • edited Loading

lzchen Oct 12, 2020

Choose a reason for hiding this comment

jan25 Oct 13, 2020

Choose a reason for hiding this comment

codeboten Oct 14, 2020

Choose a reason for hiding this comment

lzchen left a comment

Choose a reason for hiding this comment

codeboten commented Oct 22, 2020

jan25 commented Oct 23, 2020

codeboten left a comment

Choose a reason for hiding this comment

codeboten Oct 23, 2020

Choose a reason for hiding this comment

jan25 commented Oct 27, 2020

codeboten commented Oct 28, 2020

codeboten left a comment

Choose a reason for hiding this comment

NathanielRN left a comment

Choose a reason for hiding this comment

NathanielRN Oct 30, 2020 • edited Loading

Choose a reason for hiding this comment

lzchen Nov 2, 2020

Choose a reason for hiding this comment

NathanielRN Oct 30, 2020

Choose a reason for hiding this comment

rgstephens Nov 7, 2020

Choose a reason for hiding this comment

jan25 commented Oct 31, 2020

jan25 commented Sep 13, 2020 •

edited

Loading

jan25 commented Oct 1, 2020 •

edited

Loading

NathanielRN Oct 30, 2020 •

edited

Loading