Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Issue #1107] Add gzip compression support for OTLP exporter #1141

Merged
merged 33 commits into from
Nov 3, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
25220c3
[Issue #1107] Add gzip compression support for OTLP exporter
wilguo Sep 20, 2020
67b577e
[Issue #1107] Fix lint error
wilguo Sep 21, 2020
a4e78b5
[Issue #1107] Update changelog for OTLP exporter
wilguo Sep 21, 2020
6c75f3c
Merge branch 'master' into gzip-compression-feature
wilguo Sep 21, 2020
e2f5b07
[Issue #1107] Refactor OTLP exporter to pass compression value direct…
wilguo Sep 23, 2020
3cb72fd
[Issue #1107] Fix unused compression import error for OTLP trace expo…
wilguo Sep 23, 2020
75a594e
[Issue #1107] Change compression argument to use enum
wilguo Sep 23, 2020
5847ac2
[Issue #1107] Fix lint error
wilguo Sep 23, 2020
a88df17
[Issue #1107] Change doc-string for compression argument and modify E…
wilguo Sep 24, 2020
2c358f3
[Issue #1107] Add environment variable support for gzip compression
wilguo Sep 24, 2020
5cc9bf7
Update exporter/opentelemetry-exporter-otlp/src/opentelemetry/exporte…
wilguo Sep 28, 2020
36141e4
[Issue #1107] Change Enum from type GRPCCompression to OTLPCompression
wilguo Sep 28, 2020
0042962
Change priority for compression argument over environment variable
wilguo Oct 1, 2020
c747b7f
Merge branch 'master' into gzip-compression-feature
wilguo Oct 1, 2020
7285fd5
Update exporter/opentelemetry-exporter-otlp/src/opentelemetry/exporte…
wilguo Oct 1, 2020
842bfb7
Update exporter/opentelemetry-exporter-otlp/CHANGELOG.md
wilguo Oct 1, 2020
fc84520
[WIP] Add tests for OTLP compression
wilguo Oct 6, 2020
d300df2
[WIP] Add tests for OTLP Exporter compression
wilguo Oct 7, 2020
14f6635
WIP Add test cases for otlp compression
wilguo Oct 15, 2020
464ded3
Add environment variable to docs, fix enum parsing for OTLPCompressio…
wilguo Oct 25, 2020
7b37856
Merge branch 'master' into gzip-compression-feature
wilguo Oct 25, 2020
4871937
Fix lint errors
wilguo Oct 25, 2020
0020023
Merge branch 'gzip-compression-feature' of https://github.com/open-o1…
wilguo Oct 25, 2020
c411c42
Fix lint errors for OTLP test file
wilguo Oct 25, 2020
42982f3
Delete unnecessary imports from OTLP trace exporter test
wilguo Oct 26, 2020
d624203
Merge branch 'master' into gzip-compression-feature
wilguo Nov 2, 2020
e1fab0b
Fix lint errors
wilguo Nov 2, 2020
5083591
Fix arguments for insecure_channel and lint error
wilguo Nov 2, 2020
cf5a5e6
Change compression environment variable to use Configuration()
wilguo Nov 2, 2020
6e6a528
Fix lint error
wilguo Nov 2, 2020
965f0e9
Remove unused os import for OTLP Exporter
wilguo Nov 2, 2020
3799944
Merge branch 'master' into gzip-compression-feature
wilguo Nov 2, 2020
dba6143
Update exporter/opentelemetry-exporter-otlp/CHANGELOG.md
wilguo Nov 2, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions exporter/opentelemetry-exporter-otlp/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

## Unreleased

- Add Gzip compression for exporter
([#1141](https://github.com/open-telemetry/opentelemetry-python/pull/1141))
## Version 0.15b0

Released 2020-11-02
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,16 @@
.. _OTLP: https://github.com/open-telemetry/opentelemetry-collector/
.. _OpenTelemetry: https://github.com/open-telemetry/opentelemetry-python/

.. envvar:: OTEL_EXPORTER_OTLP_COMPRESSION

The :envvar:`OTEL_EXPORTER_OTLP_COMPRESSION` environment variable allows a
compression algorithm to be passed to the OTLP exporter. The compression
algorithms that are supported include gzip and no compression. The value should
be in the format of a string "gzip" for gzip compression, and no value specified
if no compression is the desired choice.
Additional details are available `in the specification
<https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/protocol/exporter.md#opentelemetry-protocol-exporter>`_.

.. code:: python

from opentelemetry import trace
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,19 +14,20 @@

"""OTLP Exporter"""

import enum
import logging
import os
from abc import ABC, abstractmethod
from collections.abc import Mapping, Sequence
from time import sleep
from typing import Any, Callable, Dict, Generic, List, Optional
from typing import Sequence as TypingSequence
from typing import Text, Tuple, TypeVar
from typing import Text, TypeVar

from backoff import expo
from google.rpc.error_details_pb2 import RetryInfo
from grpc import (
ChannelCredentials,
Compression,
RpcError,
StatusCode,
insecure_channel,
Expand All @@ -47,6 +48,10 @@
ExportResultT = TypeVar("ExportResultT")


class OTLPCompression(enum.Enum):
gzip = "gzip"


def _translate_key_values(key: Text, value: Any) -> KeyValue:

if isinstance(value, bool):
Expand Down Expand Up @@ -137,6 +142,7 @@ class OTLPExporterMixin(
insecure: Connection type
credentials: ChannelCredentials object for server authentication
metadata: Metadata to send when exporting
compression: Compression algorithm to be used in channel
timeout: Backend request timeout in seconds
"""

Expand All @@ -147,6 +153,7 @@ def __init__(
credentials: Optional[ChannelCredentials] = None,
headers: Optional[str] = None,
timeout: Optional[int] = None,
compression: str = None,
aabmass marked this conversation as resolved.
Show resolved Hide resolved
):
super().__init__()

Expand All @@ -169,13 +176,40 @@ def __init__(
)
self._collector_span_kwargs = None

if compression is None:
compression_algorithm = Compression.NoCompression
elif (
compression in OTLPCompression._value2member_map_
and OTLPCompression(compression) is OTLPCompression.gzip
):
compression_algorithm = Compression.Gzip
else:
compression_str = Configuration().EXPORTER_OTLP_INSECURE or None
if compression_str is None:
compression_algorithm = Compression.NoCompression
elif (
compression_str in OTLPCompression._value2member_map_
and OTLPCompression(compression_str) is OTLPCompression.gzip
):
compression_algorithm = Compression.Gzip
else:
raise ValueError(
"OTEL_EXPORTER_OTLP_COMPRESSION environment variable does not match gzip."
)

if insecure:
self._client = self._stub(insecure_channel(endpoint))
self._client = self._stub(
insecure_channel(endpoint, compression=compression_algorithm)
)
else:
credentials = credentials or _load_credential_from_file(
Configuration().EXPORTER_OTLP_CERTIFICATE
)
self._client = self._stub(secure_channel(endpoint, credentials))
self._client = self._stub(
secure_channel(
endpoint, credentials, compression=compression_algorithm
)
)

@abstractmethod
def _translate_data(
Expand Down