OTLP/GRPC Exporter Implementation #272

SeanHood · 2021-03-04T21:30:31Z

What's working?

Sends spans to the Otel Collector
Spans contain: Attributes, Status Code, Kind, Start/End timestamp, Lib Version
Successfully sent OTLP to Honeycomb.io

What's not working?

Need to rebase main onto this branch
Trace ID and Span ID are too long, and have been fudged to trim them.
Some tests. Mostly for SpanConverter, lacking in tests for Exporter
Events on spans haven't been tested
Child spans haven't been tested
Example code needs to be updated (been testing with laravel-otel)
Need to check the status returned

Generally just unfinished

Anything more to add?

Thank you to the Ruby OTLP Exporter, lots of inspiration was taken from there
Thanks to @riticksingh for starting off the OTLP/GRPC work

Signed-off-by:Ritick Gautam <[email protected]>

updated .gitignore

updated script file updated Makefile

What's working? * Sends spans to the Otel Collector * Spans contain: Attributes, Status Code, Kind, Start/End timestamp, Lib Version * Sucessfully sent OTLP to Honeycomb.io What's not working? * Need to rebase main onto this branch * Trace ID and Span ID are too long, and have been fudged to trim them. * No tests. Unit/Integ * Events on spans haven't been tested * Child spans haven't been tested * Example code needs to be updated (been testing with laravel-otel) * Need to check the status returned * Links not implemented * Generally just unfinished Anything more to add? * Thank you to the Ruby OTLP Exporter, lots of inspiration was taken from there

linux-foundation-easycla · 2021-03-04T21:30:35Z

The committers are authorized under a signed CLA.

✅ Ritick Gautam (dc64b04, dfe7b26, 7205339, dc6fe75, 09a405a, 3444fb9, ffae565, 1cc76d4, 7a3fdcd, d536ec8, 40bf96f, e02b931, b691299, 912584d, 9bb160f)
✅ Sean Hood (cf54582)

bobstrecansky · 2021-03-05T15:21:11Z

@zsistla - I know you were planning on working on something similar, wanted to tag you for awareness.

contrib/OtlpGrpc/Exporter.php

zsistla · 2021-03-05T17:22:54Z

Hi Bob, Thanks for the heads up, this was pretty much what I was working on. I guess I'll try to give him comments instead. regards, Amber

…

On Fri, Mar 5, 2021 at 7:21 AM Bob Strecansky ***@***.***> wrote: @zsistla <https://github.com/zsistla> - I know you were planning on working on something similar, wanted to tag you for awareness. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#272 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKNQUTZAMCDNQ5VV227NP2TTCDZHRANCNFSM4YUATN5A> .

-- Amber Sistla Software Engineer New Relic

…tests

… `make style` to format the code

contrib/OtlpGrpc/Exporter.php

zsistla · 2021-03-21T19:34:08Z

contrib/OtlpGrpc/Exporter.php

+        $this->compression = getenv('OTEL_EXPORTER_OTLP_COMPRESSION') ?: $compression; // TODO
+        $this->timeout =(int) getenv('OTEL_EXPORTER_OTLP_TIMEOUT') ?: $timeout; // TODO
+
+        $this->spanConverter = new SpanConverter('foo');


Other exporters have passed the service name (foo in this case) in via the constructer but it can also be set from OTEL_RESOURCE_ATTRIBUTES

So I've looked into this a little more, I see that most of the PHP Exporters take in the serviceName. So I went down the rabbit hole of OTEL_RESOURCE_ATTRIBUTES in other languages, and they seem to pass in service.name etc into a Resource on the TracerProvider rather than passing the service name via the Exporter. See these two examples:

https://github.com/lightstep/opentelemetry-examples/blob/main/go/opentelemetry/server/server.go#L116
https://github.com/honeycombio/examples/blob/main/python-otlp/app.py#L20

Which is swaying me more to removing the $serviceName from the Exporter entirely. What are your thoughts?

If OTEL_RESOURCE_ATTRIBUTES exists it is prioritized; if it doesn't exist, then the service.name is set in the tracer and should be pulled from there to set the service name for export.

You can get the ResourceInfo from the tracer then iterate through the keys and set the attributes on the span like python.
https://github.com/open-telemetry/opentelemetry-python/blob/main/exporter/opentelemetry-exporter-otlp-proto-grpc/src/opentelemetry/exporter/otlp/proto/grpc/exporter.py#L127

zsistla · 2021-03-21T20:04:45Z

The progress you made on this is fantastic! Once you make the metadata not hardcoded, it'll be completely flexible to send to whatever backend(s) the collector is set up for. If you comment out those lines, the make collector-trace will automatically send it.

Actually, I don't think the metadata doesn't even needs to be passed here. You can set it in the collector configuration file. For example, the config file for honeycomb would be something like this.

Great job :)

SeanHood · 2021-04-07T17:44:37Z

tests/Contrib/Unit/OTLPGrpcSpanConverterTest.php

+
+        $sdk->setSpanStatus(SpanStatus::OK);
+
+        $sdk->end($end_time); // Setting the end timestamp does not work how I expect it to


This is the line that I'm struggling to make the test pass. /cc @bobstrecansky

Tried passing in a timestamp

Tried not passing a timestamp, but fetching it later with $sdk->getEnd() and passing this to the Proto Span didn't work either.

I've ran into complications with setting an end time/duration when trying to write database instrumentation in Laravel too.

I spoke with Sean about this in DM, I think we can use:

Clock::get())->timestamp()

To accomplish his timing demands here.

SeanHood · 2021-04-09T09:30:51Z

I asked on one of the weekly calls, but I forgot to add a comment here.

I think we need to generate the proto files and add them into the repo. In the call it seemed like a good idea.

One less thing end users will need to do
Will help get tests to pass for this PR (without generating them just for the tests)

bobstrecansky · 2021-04-09T21:34:13Z

Definitely happy to add the proto files to the repo!

I pushed the proto files and build is complaining, not sure if we want to exclude this dir from linting or not? Testing it formatted so I can try to get this build green.

1. Moved out ResourcesSpans into the SpanConverter 2. Time is almost fixed, well tests pass but... I learned the clock didn't work as I expected it to. Wall clock and monotonic clock are entirely unrelated. I still think this needs more work but I _think_ it tests that the duration is correctly set on the otlp span. 3. ResourceInfo is now populated on otlp ResourceSpans.

hrodic · 2021-04-14T13:52:23Z

https://github.com/open-telemetry/opentelemetry-proto

…p-grpc

No, we do not want generated code to be formatted with php-cs-fixer This reverts commit 996b473.

… it's implemented

SeanHood · 2021-05-05T16:09:31Z

Hey all,

Flagging this as ready for review. There's a few things which still need to be fixed but I believe it's 90% there.

Links are not implemented (Needs work in the SDK I believe)
InstrumentationLibrary is not implemented (Needs work in the SDK I believe, comparing how it's implemented in opentelemetry-go)
Exporter could use some more tests either acceptance or mocking the Grpc Client
A number of the SpanConverter functions should really return an empty object for the respective type, rather than a partially constructed object.
Missing any form of retries on Exporter errors: https://github.com/open-telemetry/opentelemetry-specification/blob/318745aaf6fb173eb5cbc887ab76796bd817b3d6/specification/protocol/exporter.md#retry

Since it's almost there, merging it will open the floor for more contributors to resolve some of these without being blocked.

codecov · 2021-05-05T16:58:22Z

Codecov Report

Merging #272 (f68a7f1) into main (8bc993e) will decrease coverage by 0.00%.
The diff coverage is 92.10%.

@@             Coverage Diff              @@
##               main     #272      +/-   ##
============================================
- Coverage     94.09%   94.09%   -0.01%     
- Complexity      495      551      +56     
============================================
  Files            47       49       +2     
  Lines          1185     1337     +152     
============================================
+ Hits           1115     1258     +143     
- Misses           70       79       +9

Impacted Files	Coverage Δ	Complexity Δ
contrib/OtlpGrpc/Exporter.php	`92.06% <92.06%> (ø)`	`25.00 <25.00> (?)`
contrib/OtlpGrpc/SpanConverter.php	`92.13% <92.13%> (ø)`	`31.00 <31.00> (?)`
sdk/Trace/Span.php	`93.26% <0.00%> (+2.88%)`	`42.00% <0.00%> (ø%)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8bc993e...f68a7f1. Read the comment docs.

bobstrecansky

The codecov part of this is unimportant to me right now. I think we can merge this.

zsistla

Great job on all your work on this!!!

riticksingh and others added 16 commits November 12, 2020 00:28

Adding Proto to opentelemetry-php

dc64b04

Signed-off-by:Ritick Gautam <[email protected]>

Updated the script for generating proto folder in root

dfe7b26

Moved shell file in script folder

7205339

updated .gitignore

Removed opentelemetry-proto

dc6fe75

Bug fixed

09a405a

added docker-compose.proto.yaml

3444fb9

updated script file updated Makefile

Update docker-compose.proto.yaml

ffae565

Updated README.md

1cc76d4

Merge remote-tracking branch 'upstream/master'

7a3fdcd

OtlpGrpc

d536ec8

Added Request-response

40bf96f

Example Updated

e02b931

Updated Span Converter

b691299

small fixes

912584d

modified: SpanConvertor.php

9bb160f

bobstrecansky reviewed Mar 5, 2021

View reviewed changes

contrib/OtlpGrpc/Exporter.php Show resolved Hide resolved

contrib/OtlpGrpc/Exporter.php Show resolved Hide resolved

contrib/OtlpGrpc/Exporter.php Outdated Show resolved Hide resolved

bobstrecansky mentioned this pull request Mar 5, 2021

OpenTelemetry Exporter fail with : 400 Bad Request json: cannot unmarshal array into Go value of type map[string]json.RawMessage #265

Closed

SeanHood added 5 commits March 8, 2021 22:05

Fixed traceid and spanid hacks; Now supports Events

354853f

Use localhost:4317 for the default host; + cleanup

7f9530c

Merge remote-tracking branch 'origin/main' into otlp-grpc

8c63f52

Split code between Exporter.php and SpanConverter.php; Beginnings of …

580acf8

…tests

More progress on tests, array's now correctly get converted. Also ran…

b8469e6

… `make style` to format the code

zsistla reviewed Mar 21, 2021

View reviewed changes

contrib/OtlpGrpc/Exporter.php Outdated Show resolved Hide resolved

zsistla reviewed Mar 21, 2021

View reviewed changes

bobstrecansky mentioned this pull request Mar 23, 2021

OTLP/grpc #230

Closed

Further progress on Tests for the Exporter

621175c

SeanHood commented Apr 7, 2021

View reviewed changes

SeanHood added 3 commits April 9, 2021 23:29

SpanConverter now translates attrs on Events

5c54d21

Add opentelemtry-proto files

49fa87d

Ran `make style' on proto. Do we want this?

996b473

I pushed the proto files and build is complaining, not sure if we want to exclude this dir from linting or not? Testing it formatted so I can try to get this build green.

SeanHood force-pushed the otlp-grpc branch from 034be3b to 996b473 Compare April 13, 2021 19:43

SeanHood added 9 commits April 21, 2021 17:59

Fix some of the complaints from static analysis

03a767a

Merge branch 'main' of github.com:SeanHood/opentelemetry-php into otl…

40ce35f

…p-grpc

Revert "Ran `make style' on proto. Do we want this?"

0eb6cfd

No, we do not want generated code to be formatted with php-cs-fixer This reverts commit 996b473.

make style

469afde

Static analysis fixes

06b68f7

Hopefully this adds the grpc extension

1b2c91a

Hopefully this adds the grpc extension

b46f981

Some Psalm fixes

a0fd792

Added a few comments; disable testing of InstrumentationLibrary until…

f68a7f1

… it's implemented

SeanHood changed the title ~~(WIP): Partially working OTLP/GRPC Exporter~~ OTLP/GRPC Exporter Implementation May 5, 2021

SeanHood marked this pull request as ready for review May 5, 2021 16:02

SeanHood requested review from beniamin, Fahmy-Mohammed and morrisonlevi as code owners May 5, 2021 16:02

bobstrecansky approved these changes May 5, 2021

View reviewed changes

zsistla approved these changes May 5, 2021

View reviewed changes

bobstrecansky merged commit 8a13501 into open-telemetry:main May 5, 2021

bobstrecansky mentioned this pull request May 11, 2021

Feature - OpenTelemetry Exporter #130

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OTLP/GRPC Exporter Implementation #272

OTLP/GRPC Exporter Implementation #272

SeanHood commented Mar 4, 2021 •

edited

Loading

linux-foundation-easycla bot commented Mar 4, 2021 •

edited

Loading

bobstrecansky commented Mar 5, 2021

zsistla commented Mar 5, 2021 via email

zsistla Mar 21, 2021

SeanHood Mar 24, 2021

zsistla Apr 9, 2021

zsistla commented Mar 21, 2021

SeanHood Apr 7, 2021

bobstrecansky Apr 7, 2021

SeanHood commented Apr 9, 2021

bobstrecansky commented Apr 9, 2021

hrodic commented Apr 14, 2021

SeanHood commented May 5, 2021 •

edited

Loading

codecov bot commented May 5, 2021 •

edited

Loading

bobstrecansky left a comment

zsistla left a comment


		$sdk->setSpanStatus(SpanStatus::OK);

		$sdk->end($end_time); // Setting the end timestamp does not work how I expect it to

OTLP/GRPC Exporter Implementation #272

OTLP/GRPC Exporter Implementation #272

Conversation

SeanHood commented Mar 4, 2021 • edited Loading

linux-foundation-easycla bot commented Mar 4, 2021 • edited Loading

bobstrecansky commented Mar 5, 2021

zsistla commented Mar 5, 2021 via email

zsistla Mar 21, 2021

Choose a reason for hiding this comment

SeanHood Mar 24, 2021

Choose a reason for hiding this comment

zsistla Apr 9, 2021

Choose a reason for hiding this comment

zsistla commented Mar 21, 2021

SeanHood Apr 7, 2021

Choose a reason for hiding this comment

bobstrecansky Apr 7, 2021

Choose a reason for hiding this comment

SeanHood commented Apr 9, 2021

bobstrecansky commented Apr 9, 2021

hrodic commented Apr 14, 2021

SeanHood commented May 5, 2021 • edited Loading

codecov bot commented May 5, 2021 • edited Loading

Codecov Report

bobstrecansky left a comment

Choose a reason for hiding this comment

zsistla left a comment

Choose a reason for hiding this comment

SeanHood commented Mar 4, 2021 •

edited

Loading

linux-foundation-easycla bot commented Mar 4, 2021 •

edited

Loading

SeanHood commented May 5, 2021 •

edited

Loading

codecov bot commented May 5, 2021 •

edited

Loading