-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NH-2313 Add basic TraceState handling and W3C trace context propagation #11
Merged
Merged
Changes from all commits
Commits
Show all changes
78 commits
Select commit
Hold shift + click to select a range
23e4f8e
Add stub for SolarWindsFormat custom propagator
tammy-baylis-swi b591d4f
(WIP) tracestate sometimes gets updated with span_id
tammy-baylis-swi 9ceb4df
Add helpers to format span_id, trace_flags
tammy-baylis-swi a5701a1
Add temp manual trace_state update logic
tammy-baylis-swi a6533bc
Rm SolarWindsFormat.extract duplicate code from TraceContextTextMapPr…
tammy-baylis-swi 05b8887
Fix propagator use of tracestate.update
tammy-baylis-swi a7c6244
Rename to SwSampler, WIP decision-tracestate-attributes management
tammy-baylis-swi d91e276
WIP attributes
tammy-baylis-swi ae29003
Rm unnecessary attributes touching by Propagator
tammy-baylis-swi 394ae52
SwSampler adds/updates sw.w3c.tracestate, sw.tracestate_parent_id
tammy-baylis-swi f6ff7f4
SwSampler adds/updates sw.parent_span_id
tammy-baylis-swi c380527
Refactor/Break up _SwSampler.should_sample into several helpers
tammy-baylis-swi 5659f5b
Fix format of args sent to Context::getDecisions
tammy-baylis-swi f17b2b8
Fix duplication of sw.parent_span_id by Sampler; stop invalid attribu…
tammy-baylis-swi 52e655a
install_requires less strict for Django ASGI support
tammy-baylis-swi 69a487a
Mv SolarWindsFormat to SolarWindsPropagator
tammy-baylis-swi cb5800e
Mv ot_ao_transformer to w3c_transformer, used by Exporter Sampler Pro…
tammy-baylis-swi 537cf0a
Simplify variables used in Propagator.inject
tammy-baylis-swi 88529a1
Refactor of _SwSampler
tammy-baylis-swi 761e34f
Rm ununsed variables
tammy-baylis-swi 07ec508
Fix trace_flags and some log formatting
tammy-baylis-swi 77dbfb6
Rm f-strings for older Python versions
tammy-baylis-swi c4ca055
Mv tracestate sw value format checks to _SwSampler.calculate_liboboe_…
tammy-baylis-swi 30899bf
Misc fixups for style and comments
tammy-baylis-swi 7e4e8c9
Sampler lets liboboe do all the decision-making
tammy-baylis-swi d9c0e63
Fix flag format at trace_state and attribute update
tammy-baylis-swi 5204536
Update Propagator debug logs for clarity
tammy-baylis-swi 12d4cbd
Fix sw.tracestate_parent_id assignment
tammy-baylis-swi bd315a6
Preserve non-vendor values of injected tracestate header
tammy-baylis-swi 8f5a215
Add TraceOptions class and an init with headers from propagator.extract
tammy-baylis-swi 60522b7
Mv TraceOptions to XTraceOptions; add from_context and iter,str overl…
tammy-baylis-swi 3e175ba
Sampler passes xtraceoptions from otel context to liboboe
tammy-baylis-swi 38639c9
Sampler passes liboboe tracestate from header tracestate, not tracepa…
tammy-baylis-swi f95bf46
Fix XTraceOptions init when other headers provided
tammy-baylis-swi af11112
Add TT signature handling; Sampler gives liboboe None if parent not v…
tammy-baylis-swi 322949b
Add SolarWindsResponsePropagator (wip); up otel-instrumentation version
tammy-baylis-swi ff201c7
Sampler creates xtraceoptions_response for tracestate piggyback; Resp…
tammy-baylis-swi 05f03c7
Fix sampler tracestate creation; mv XTraceOptions str overload to to_…
tammy-baylis-swi 3b62bc5
Sampler create x-trace-options-response if parent tracestate exists
tammy-baylis-swi 1dfd489
Fix bug in to_options_header
tammy-baylis-swi 99d2fc4
Add W3CTransformer classmethods
tammy-baylis-swi 32df3d0
Fix Sampler OBOE_SETTINGS_UNSET of tracing_mode, sample_rate, trigger…
tammy-baylis-swi 7444d43
String formatting a bit less brittle
tammy-baylis-swi 84a863e
Fix trace_state KV add at inject
tammy-baylis-swi de88565
Style and comment for ResponsePropagator
tammy-baylis-swi ec2c317
Python 3.6+, otel SDK 1.10.0, instrumentation 0.29.b0
tammy-baylis-swi 59b1bac
xtraceoptions log lines to debug level
tammy-baylis-swi 45f1910
Assign several keys/attrs as constants
tammy-baylis-swi 552398c
Fix sw.tracestate_parent_id, sw.w3c.tracestate KV/attribute creation
tammy-baylis-swi 58acd70
Rm _LiboboeDecision class to use dict instead
tammy-baylis-swi 783cac0
distro set_global_textmap hardcode instead of env var for OTEL_PROPAG…
tammy-baylis-swi b58e29b
traceparent_from_context returns lowercase
tammy-baylis-swi b08517e
Fixups: variable assignments, rm ununsed param
tammy-baylis-swi f7f487a
Fix comment
tammy-baylis-swi 3aad13d
NH-11236 sampler creates Service Entry Internal KVs
tammy-baylis-swi ffccf28
SWKeys KV added to root span, sw_keys always str, trigger_trace as in…
tammy-baylis-swi 3a90bdb
Simplify x-trace-options handling
tammy-baylis-swi 118c1bc
Sampler always sets SWKeys KV if available; calculate_attributes refa…
tammy-baylis-swi 23a12a2
Sampler sets bucket/sample KVs if liboboe decision not cont'd
tammy-baylis-swi 526df98
Merge branch 'add-custom-propagator' into NH-11236-service-entry-inte…
tammy-baylis-swi 7077e6a
Latest otel 1.11.0/0.30b0, rename entry point to solarwinds_propagato…
tammy-baylis-swi f9171c8
Exporter and Propagator(s) are env var configurable
tammy-baylis-swi 0aef14c
NH-9150 Use oboe_api fns for no default timestamp
tammy-baylis-swi c77afc9
Merge pull request #13 from appoptics/nh-9150-use-oboe-api-no-default…
tammy-baylis-swi 68a0e63
Merge pull request #12 from appoptics/NH-11236-service-entry-internal…
tammy-baylis-swi 7b75e65
Separate concerns: env vars in distro, component init in configurator
tammy-baylis-swi 600e5b2
Custom Propagator gets tracestate from carrier in case of upstream pr…
tammy-baylis-swi ede5686
Call TraceState.from_header correctly
tammy-baylis-swi 2842209
NH-2313 adjust CompositePropagator config with OTEL_PROPAGATORS
tammy-baylis-swi 7ffb1a9
solarwinds_sampler cannot be env default because not in OTel _KNOWN_S…
tammy-baylis-swi 02bde11
Merge pull request #15 from appoptics/NH-2313-adjust-propagator-confi…
tammy-baylis-swi 54a0e7d
Merge pull request #16 from appoptics/NH-2313-propagator-gets-tracest…
tammy-baylis-swi 1241e7a
load_entry_point instead of iter for singulars
tammy-baylis-swi 1b830a1
Merge pull request #14 from appoptics/NH-12018-distro-and-configurato…
tammy-baylis-swi 73be566
Move _initialize_solarwinds_reporter from Exporter to Configurator
tammy-baylis-swi 126e030
Fix non-SW sampler,exporter configure
tammy-baylis-swi 01f1e98
Always use SW Sampler
tammy-baylis-swi 7e758df
Merge pull request #17 from appoptics/NH-12018-oboe-reporter-in-confi…
tammy-baylis-swi File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1,10 @@ | ||
__version__ = "0.0.1" | ||
|
||
COMMA = "," | ||
COMMA_W3C_SANITIZED = "...." | ||
EQUALS = "=" | ||
EQUALS_W3C_SANITIZED = "####" | ||
SW_TRACESTATE_KEY = "sw" | ||
OTEL_CONTEXT_SW_OPTIONS_KEY = "sw_xtraceoptions" | ||
OTEL_CONTEXT_SW_SIGNATURE_KEY = "sw_signature" | ||
DEFAULT_SW_TRACES_EXPORTER = "solarwinds_exporter" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,156 @@ | ||
"""Module to initialize OpenTelemetry SDK components to work with SolarWinds backend""" | ||
|
||
import logging | ||
from os import environ | ||
from pkg_resources import ( | ||
iter_entry_points, | ||
load_entry_point | ||
) | ||
|
||
from opentelemetry import trace | ||
from opentelemetry.environment_variables import ( | ||
OTEL_PROPAGATORS, | ||
OTEL_TRACES_EXPORTER | ||
) | ||
from opentelemetry.instrumentation.propagators import set_global_response_propagator | ||
from opentelemetry.propagate import set_global_textmap | ||
from opentelemetry.propagators.composite import CompositePropagator | ||
from opentelemetry.sdk._configuration import _OTelSDKConfigurator | ||
from opentelemetry.sdk.environment_variables import OTEL_TRACES_SAMPLER | ||
from opentelemetry.sdk.trace import ( | ||
sampling, | ||
TracerProvider | ||
) | ||
from opentelemetry.sdk.trace.export import BatchSpanProcessor | ||
|
||
from opentelemetry_distro_solarwinds import DEFAULT_SW_TRACES_EXPORTER | ||
from opentelemetry_distro_solarwinds.extension.oboe import Reporter | ||
from opentelemetry_distro_solarwinds.response_propagator import SolarWindsTraceResponsePropagator | ||
|
||
logger = logging.getLogger(__name__) | ||
|
||
class SolarWindsConfigurator(_OTelSDKConfigurator): | ||
"""OpenTelemetry Configurator for initializing SolarWinds-reporting SDK components""" | ||
|
||
# Cannot set as env default because not part of OTel Python _KNOWN_SAMPLERS | ||
# https://github.com/open-telemetry/opentelemetry-python/blob/main/opentelemetry-sdk/src/opentelemetry/sdk/trace/sampling.py#L364-L380 | ||
_DEFAULT_SW_TRACES_SAMPLER = "solarwinds_sampler" | ||
|
||
def _configure(self, **kwargs): | ||
"""Configure OTel sampler, exporter, propagator, response propagator""" | ||
reporter = self._initialize_solarwinds_reporter() | ||
self._configure_sampler() | ||
self._configure_exporter(reporter) | ||
self._configure_propagator() | ||
# Set global HTTP response propagator | ||
set_global_response_propagator(SolarWindsTraceResponsePropagator()) | ||
|
||
def _configure_sampler(self): | ||
"""Always configure SolarWinds OTel sampler""" | ||
try: | ||
sampler = load_entry_point( | ||
"opentelemetry_distro_solarwinds", | ||
"opentelemetry_traces_sampler", | ||
self._DEFAULT_SW_TRACES_SAMPLER | ||
)() | ||
except: | ||
logger.exception( | ||
"Failed to load configured sampler {}".format( | ||
self._DEFAULT_SW_TRACES_SAMPLER | ||
) | ||
) | ||
raise | ||
trace.set_tracer_provider( | ||
TracerProvider(sampler=sampler) | ||
) | ||
|
||
def _configure_exporter(self, reporter): | ||
"""Configure SolarWinds or env-specified OTel span exporter. | ||
Initialization of SolarWinds exporter requires a liboboe reporter.""" | ||
exporter = None | ||
environ_exporter_name = environ.get(OTEL_TRACES_EXPORTER) | ||
|
||
if environ_exporter_name == DEFAULT_SW_TRACES_EXPORTER: | ||
try: | ||
exporter = load_entry_point( | ||
"opentelemetry_distro_solarwinds", | ||
"opentelemetry_traces_exporter", | ||
environ_exporter_name | ||
)(reporter) | ||
except: | ||
logger.exception( | ||
"Failed to load configured exporter {} with reporter".format( | ||
environ_exporter_name | ||
) | ||
) | ||
raise | ||
else: | ||
try: | ||
exporter = next( | ||
iter_entry_points( | ||
"opentelemetry_traces_exporter", | ||
environ_exporter_name | ||
) | ||
).load()() | ||
except: | ||
logger.exception( | ||
"Failed to load configured exporter {}".format( | ||
environ_exporter_name | ||
) | ||
) | ||
raise | ||
span_processor = BatchSpanProcessor(exporter) | ||
trace.get_tracer_provider().add_span_processor(span_processor) | ||
|
||
def _configure_propagator(self): | ||
"""Configure CompositePropagator with SolarWinds and other propagators""" | ||
propagators = [] | ||
environ_propagators_names = environ.get(OTEL_PROPAGATORS).split(",") | ||
for propagator_name in environ_propagators_names: | ||
try: | ||
propagators.append( | ||
next( | ||
iter_entry_points("opentelemetry_propagator", propagator_name) | ||
).load()() | ||
) | ||
except Exception: | ||
logger.exception( | ||
"Failed to load configured propagator {}".format( | ||
propagator_name | ||
) | ||
) | ||
raise | ||
set_global_textmap(CompositePropagator(propagators)) | ||
|
||
def _initialize_solarwinds_reporter(self) -> Reporter: | ||
"""Initialize SolarWinds reporter used by sampler and exporter. This establishes collector and sampling settings in a background thread.""" | ||
log_level = environ.get('SOLARWINDS_DEBUG_LEVEL', 3) | ||
try: | ||
log_level = int(log_level) | ||
except ValueError: | ||
log_level = 3 | ||
# TODO make some of these customizable | ||
return Reporter( | ||
hostname_alias='', | ||
log_level=log_level, | ||
log_file_path='', | ||
max_transactions=-1, | ||
max_flush_wait_time=-1, | ||
events_flush_interval=-1, | ||
max_request_size_bytes=-1, | ||
reporter='ssl', | ||
host=environ.get('SOLARWINDS_COLLECTOR', ''), | ||
service_key=environ.get('SOLARWINDS_SERVICE_KEY', ''), | ||
trusted_path='', | ||
buffer_size=-1, | ||
trace_metrics=-1, | ||
histogram_precision=-1, | ||
token_bucket_capacity=-1, | ||
token_bucket_rate=-1, | ||
file_single=0, | ||
ec2_metadata_timeout=1000, | ||
grpc_proxy='', | ||
stdout_clear_nonblocking=0, | ||
is_grpc_clean_hack_enabled=False, | ||
w3c_trace_format=1, | ||
) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,24 +1,52 @@ | ||
"""Module to configure OpenTelemetry agent to work with SolarWinds backend""" | ||
"""Module to configure OpenTelemetry to work with SolarWinds backend""" | ||
|
||
from opentelemetry import trace | ||
import logging | ||
from os import environ | ||
|
||
from opentelemetry.environment_variables import ( | ||
OTEL_PROPAGATORS, | ||
OTEL_TRACES_EXPORTER | ||
) | ||
from opentelemetry.instrumentation.distro import BaseDistro | ||
from opentelemetry.sdk.trace import TracerProvider | ||
from opentelemetry.sdk.trace.export import BatchSpanProcessor | ||
|
||
from opentelemetry_distro_solarwinds.exporter import SolarWindsSpanExporter | ||
from opentelemetry_distro_solarwinds.sampler import ParentBasedAoSampler | ||
from opentelemetry_distro_solarwinds import DEFAULT_SW_TRACES_EXPORTER | ||
|
||
logger = logging.getLogger(__name__) | ||
|
||
class SolarWindsDistro(BaseDistro): | ||
"""SolarWinds custom distro for OpenTelemetry agents. | ||
"""OpenTelemetry Distro for SolarWinds reporting environment""" | ||
|
||
_TRACECONTEXT_PROPAGATOR = "tracecontext" | ||
_SW_PROPAGATOR = "solarwinds_propagator" | ||
_DEFAULT_SW_PROPAGATORS = [ | ||
_TRACECONTEXT_PROPAGATOR, | ||
"baggage", | ||
_SW_PROPAGATOR, | ||
] | ||
|
||
With this custom distro, the following functionality is introduced: | ||
- no functionality added at this time | ||
""" | ||
def _configure(self, **kwargs): | ||
# automatically make use of custom SolarWinds sampler | ||
trace.set_tracer_provider( | ||
TracerProvider(sampler=ParentBasedAoSampler())) | ||
# Automatically configure the SolarWinds Span exporter | ||
span_exporter = BatchSpanProcessor(SolarWindsSpanExporter()) | ||
trace.get_tracer_provider().add_span_processor(span_exporter) | ||
"""Configure OTel exporter and propagators""" | ||
environ.setdefault(OTEL_TRACES_EXPORTER, DEFAULT_SW_TRACES_EXPORTER) | ||
|
||
environ_propagators = environ.get( | ||
OTEL_PROPAGATORS, | ||
",".join(self._DEFAULT_SW_PROPAGATORS) | ||
).split(",") | ||
# If not using the default propagators, | ||
# can any arbitrary list BUT | ||
# (1) must include tracecontext and solarwinds_propagator | ||
# (2) tracecontext must be before solarwinds_propagator | ||
if environ_propagators != self._DEFAULT_SW_PROPAGATORS: | ||
if not self._TRACECONTEXT_PROPAGATOR in environ_propagators or \ | ||
not self._SW_PROPAGATOR in environ_propagators: | ||
raise ValueError("Must include tracecontext and solarwinds_propagator in OTEL_PROPAGATORS to use SolarWinds Observability.") | ||
|
||
if environ_propagators.index(self._SW_PROPAGATOR) \ | ||
< environ_propagators.index(self._TRACECONTEXT_PROPAGATOR): | ||
raise ValueError("tracecontext must be before solarwinds_propagator in OTEL_PROPAGATORS to use SolarWinds Observability.") | ||
environ[OTEL_PROPAGATORS] = ",".join(environ_propagators) | ||
|
||
logger.debug("Configured SolarWindsDistro: {}, {}".format( | ||
environ.get(OTEL_TRACES_EXPORTER), | ||
environ.get(OTEL_PROPAGATORS) | ||
)) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit and inherited code, but let's rename this to something like
startSpan
/spanEntry
and the matching onestopSpan
/spanExit
or similar, just to clarify this about a span rather than entire trace.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that's going to be a separate liboboe PR (oboe itself or the SWIG interface only if possible?), related to NH-9150, NH-7246, and this comment: https://swicloud.atlassian.net/wiki/spaces/NIT/pages/2823618801/liboboe+and+OTel+Python+custom-distro?focusedCommentId=2826010703#comment-2826010703
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah makes sense.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We've now addressed this in 0aef14c by switching to use the new
Context.createEntry
functions Jerry added to oboe_api in NH-9150.