Using opentelemetry-instrument breaks Sentry SDK #894

Maraket · 2022-02-02T22:13:01Z

Describe your environment

Python 3.6.8
Centos 7
Supervisord
Sentry_sdk 1.4.3
Environment Variables Set:
** OTEL_SERVICE_NAME=testthing
** SENTRY_URL=

Steps to reproduce
Using the below proof of concept python script:

import os
import sentry_sdk


print('about to run')
if os.environ.get('SENTRY_URL', None):      # pragma: unreachable
    sentry_sdk.init(os.environ['SENTRY_URL'])
    print('sentry run')

When the script is executed as python3 proof.py the output is:

about to run
sentry run

Though when executing opentelemetry-instrument python3 proof.py the result is:

You are using Python 3.6. This version does not support timestamps with nanosecond precision and the OpenTelemetry SDK will use millisecond precision instead. Please refer to PEP 564 for more information. Please upgrade to Python 3.7 or newer to use nanosecond precision.
about to run
Traceback (most recent call last):
  File "proof.py", line 7, in <module>
    sentry_sdk.init(os.environ['SENTRY_URL'])
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/hub.py", line 105, in _init
    client = Client(*args, **kwargs)  # type: ignore
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/client.py", line 86, in __init__
    self._init_impl()
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/client.py", line 124, in _init_impl
    "auto_enabling_integrations"
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/__init__.py", line 120, in setup_integrations
    type(integration).setup_once()
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/falcon.py", line 113, in setup_once
    _patch_handle_exception()
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/falcon.py", line 139, in _patch_handle_exception
    original_handle_exception = falcon.API._handle_exception
AttributeError: 'functools.partial' object has no attribute '_handle_exception'

What is the expected behavior?
Ideally sentry should execute without errors and provide the following response from the sample script:

about to run
sentry run

What is the actual behavior?
Recieved the error:

You are using Python 3.6. This version does not support timestamps with nanosecond precision and the OpenTelemetry SDK will use millisecond precision instead. Please refer to PEP 564 for more information. Please upgrade to Python 3.7 or newer to use nanosecond precision.
about to run
Traceback (most recent call last):
  File "proof.py", line 7, in <module>
    sentry_sdk.init(os.environ['SENTRY_URL'])
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/hub.py", line 105, in _init
    client = Client(*args, **kwargs)  # type: ignore
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/client.py", line 86, in __init__
    self._init_impl()
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/client.py", line 124, in _init_impl
    "auto_enabling_integrations"
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/__init__.py", line 120, in setup_integrations
    type(integration).setup_once()
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/falcon.py", line 113, in setup_once
    _patch_handle_exception()
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/falcon.py", line 139, in _patch_handle_exception
    original_handle_exception = falcon.API._handle_exception
AttributeError: 'functools.partial' object has no attribute '_handle_exception'

The text was updated successfully, but these errors were encountered:

owais · 2022-02-02T22:24:29Z

Looks like an issue with Falcon. Which version of Falcon are you using? Please share output of pip freeze

Maraket · 2022-02-02T22:52:44Z

aiocontextvars==0.2.2
apispec==5.1.1
ari==0.1.3
backoff==1.10.0
boto3==1.20.46
botocore==1.23.46
certifi==2021.10.8
charset-normalizer==2.0.11
contextvars==2.4
dataclasses==0.8
datadog==0.42.0
Deprecated==1.2.13
dnspython==2.1.0
falcon==2.0.0
googleapis-common-protos==1.54.0
grpcio==1.43.0
idna==3.3
immutables==0.16
inflection==0.5.1
Jinja2==3.0.3
jmespath==0.10.0
MarkupSafe==2.0.1
marshmallow==3.14.1
opentelemetry-api==1.9.1
opentelemetry-exporter-jaeger-thrift==1.9.1
opentelemetry-exporter-otlp-proto-grpc==1.9.1
opentelemetry-instrumentation==0.28b1
opentelemetry-instrumentation-aws-lambda==0.28b1
opentelemetry-instrumentation-dbapi==0.28b1
opentelemetry-instrumentation-falcon==0.28b1
opentelemetry-instrumentation-grpc==0.28b1
opentelemetry-instrumentation-logging==0.28b1
opentelemetry-instrumentation-redis==0.28b1
opentelemetry-instrumentation-requests==0.28b1
opentelemetry-instrumentation-sqlite3==0.28b1
opentelemetry-instrumentation-urllib==0.28b1
opentelemetry-instrumentation-urllib3==0.28b1
opentelemetry-instrumentation-wsgi==0.28b1
opentelemetry-propagator-aws-xray==1.0.1
opentelemetry-propagator-b3==1.9.1
opentelemetry-proto==1.9.1
opentelemetry-sdk==1.9.1
opentelemetry-semantic-conventions==0.28b1
opentelemetry-util-http==0.28b1
panoramisk==1.4
protobuf==3.19.4
PyJWT==2.3.0
python-dateutil==2.8.2
PyYAML==6.0
redis==3.5.3
requests==2.27.1
s3transfer==0.5.0
sentry-sdk==1.4.3
six==1.16.0
splunk-opentelemetry==1.4.1
supervisor==4.2.2
swaggerpy==0.2.1
thrift==0.15.0
typing_extensions==4.0.1
urllib3==1.26.8
uWSGI==2.0.20
websocket-client==1.2.3
wrapt==1.13.3

Maraket · 2022-02-02T23:03:18Z

I should mention that the application in question doesn't use Falcon, there is a second application that runs on the same instance that does though, and that is executed using uwsgi which runs without any issues and pushes telemetry data.

owais · 2022-02-03T00:33:11Z

The issue is here: https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/instrumentation/opentelemetry-instrumentation-falcon/src/opentelemetry/instrumentation/falcon/__init__.py#L151

We wrap the Falcon API class into a partial function so when it is initialized, it receives some default params. This does not break the public Falcon API as it remains a callable but it breaks Sentry because Sentry also seems to patch Falcon and assumes certain knowledge about Falcon internals.

I think we can change our implementation so instead of using a partial we use an actual class and pass in the kwargs some other way.

Maraket · 2022-02-03T01:11:44Z

As a side note, using a virtual environment (which is the fix we are going for in the interim) should solve this problem as if you are using falcon you would likely be running your application via uwsgi which seems to not suffer this issue

owais · 2022-02-03T01:24:37Z

Would it be possible for you to apply these changes to your local copy of opentelemetry-instrumentation-falcon and verify if this fixes the problem?

https://github.com/open-telemetry/opentelemetry-python-contrib/pull/895/files

Maraket · 2022-02-03T06:12:29Z

It's changed to a new error:

You are using Python 3.6. This version does not support timestamps with nanosecond precision and the OpenTelemetry SDK will use millisecond precision instead. Please refer to PEP 564 for more information. Please upgrade to Python 3.7 or newer to use nanosecond precision.
You are using Python 3.6. This version does not support timestamps with nanosecond precision and the OpenTelemetry SDK will use millisecond precision instead. Please refer to PEP 564 for more information. Please upgrade to Python 3.7 or newer to use nanosecond precision.
2022-02-03 06:01:58,614 ERROR [sitecustomize] [sitecustomize.py:87] [trace_id=0 span_id=0 resource.service.name=test_thing] - Instrumenting of falcon failed
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 84, in _load_instrumentors
    distro.load_instrumentor(entry_point, skip_dep_check=True)
  File "/usr/local/lib/python3.6/site-packages/splunk_otel/distro.py", line 44, in load_instrumentor
    super().load_instrumentor(entry_point, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/distro.py", line 63, in load_instrumentor
    instrumentor().instrument(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/instrumentor.py", line 110, in instrument
    **kwargs
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/falcon/__init__.py", line 156, in _instrument
    setattr(falcon, _instrument_app, API)
NameError: name 'API' is not defined
2022-02-03 06:01:58,623 ERROR [sitecustomize] [sitecustomize.py:119] [trace_id=0 span_id=0 resource.service.name=test_thing] - Failed to auto initialize opentelemetry
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 117, in initialize
    _load_instrumentors(distro)
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 88, in _load_instrumentors
    raise exc
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 84, in _load_instrumentors
    distro.load_instrumentor(entry_point, skip_dep_check=True)
  File "/usr/local/lib/python3.6/site-packages/splunk_otel/distro.py", line 44, in load_instrumentor
    super().load_instrumentor(entry_point, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/distro.py", line 63, in load_instrumentor
    instrumentor().instrument(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/instrumentor.py", line 110, in instrument
    **kwargs
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/falcon/__init__.py", line 156, in _instrument
    setattr(falcon, _instrument_app, API)
NameError: name 'API' is not defined
about to run
sentry run

A similar error message is occuring now in the uwsgi app now though:

2022-02-03 06:11:42.954 ERROR    Instrumenting of falcon failed
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/splunk_otel/tracing.py", line 82, in _load_instrumentors
    entry_point.load()().instrument()
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/instrumentor.py", line 110, in instrument
    **kwargs
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/falcon/__init__.py", line 156, in _instrument
    setattr(falcon, _instrument_app, API)
NameError: name 'API' is not defined
2022-02-03 06:11:43.345 WARNING  Already shutdown, dropping span.
Sentry is attempting to send 2 pending error messages
Waiting up to 2 seconds

owais · 2022-02-03T08:36:42Z

Yes, of course. I made a silly mistake just before my last push. Fixed now. Please test again if possible. Thanks!

Maraket · 2022-02-03T21:32:42Z

@owais seems to be working

Maraket added the bug Something isn't working label Feb 2, 2022

owais transferred this issue from open-telemetry/opentelemetry-python Feb 3, 2022

owais self-assigned this Feb 3, 2022

owais mentioned this issue Feb 3, 2022

Safer patching for Falcon API #895

Merged

11 tasks

owais closed this as completed in #895 Feb 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Using opentelemetry-instrument breaks Sentry SDK #894

Using opentelemetry-instrument breaks Sentry SDK #894

Maraket commented Feb 2, 2022

owais commented Feb 2, 2022

Maraket commented Feb 2, 2022

Maraket commented Feb 2, 2022

owais commented Feb 3, 2022

Maraket commented Feb 3, 2022

owais commented Feb 3, 2022

Maraket commented Feb 3, 2022

owais commented Feb 3, 2022

Maraket commented Feb 3, 2022

Using opentelemetry-instrument breaks Sentry SDK #894

Using opentelemetry-instrument breaks Sentry SDK #894

Comments

Maraket commented Feb 2, 2022

owais commented Feb 2, 2022

Maraket commented Feb 2, 2022

Maraket commented Feb 2, 2022

owais commented Feb 3, 2022

Maraket commented Feb 3, 2022

owais commented Feb 3, 2022

Maraket commented Feb 3, 2022

owais commented Feb 3, 2022

Maraket commented Feb 3, 2022