Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using opentelemetry-instrument breaks Sentry SDK #894

Closed
Maraket opened this issue Feb 2, 2022 · 9 comments · Fixed by #895
Closed

Using opentelemetry-instrument breaks Sentry SDK #894

Maraket opened this issue Feb 2, 2022 · 9 comments · Fixed by #895
Assignees
Labels
bug Something isn't working

Comments

@Maraket
Copy link

Maraket commented Feb 2, 2022

Describe your environment

  • Python 3.6.8
  • Centos 7
  • Supervisord
  • Sentry_sdk 1.4.3
  • Environment Variables Set:
    ** OTEL_SERVICE_NAME=testthing
    ** SENTRY_URL=

Steps to reproduce
Using the below proof of concept python script:

import os
import sentry_sdk


print('about to run')
if os.environ.get('SENTRY_URL', None):      # pragma: unreachable
    sentry_sdk.init(os.environ['SENTRY_URL'])
    print('sentry run')

When the script is executed as python3 proof.py the output is:

about to run
sentry run

Though when executing opentelemetry-instrument python3 proof.py the result is:

You are using Python 3.6. This version does not support timestamps with nanosecond precision and the OpenTelemetry SDK will use millisecond precision instead. Please refer to PEP 564 for more information. Please upgrade to Python 3.7 or newer to use nanosecond precision.
about to run
Traceback (most recent call last):
  File "proof.py", line 7, in <module>
    sentry_sdk.init(os.environ['SENTRY_URL'])
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/hub.py", line 105, in _init
    client = Client(*args, **kwargs)  # type: ignore
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/client.py", line 86, in __init__
    self._init_impl()
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/client.py", line 124, in _init_impl
    "auto_enabling_integrations"
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/__init__.py", line 120, in setup_integrations
    type(integration).setup_once()
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/falcon.py", line 113, in setup_once
    _patch_handle_exception()
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/falcon.py", line 139, in _patch_handle_exception
    original_handle_exception = falcon.API._handle_exception
AttributeError: 'functools.partial' object has no attribute '_handle_exception'

What is the expected behavior?
Ideally sentry should execute without errors and provide the following response from the sample script:

about to run
sentry run

What is the actual behavior?
Recieved the error:

You are using Python 3.6. This version does not support timestamps with nanosecond precision and the OpenTelemetry SDK will use millisecond precision instead. Please refer to PEP 564 for more information. Please upgrade to Python 3.7 or newer to use nanosecond precision.
about to run
Traceback (most recent call last):
  File "proof.py", line 7, in <module>
    sentry_sdk.init(os.environ['SENTRY_URL'])
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/hub.py", line 105, in _init
    client = Client(*args, **kwargs)  # type: ignore
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/client.py", line 86, in __init__
    self._init_impl()
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/client.py", line 124, in _init_impl
    "auto_enabling_integrations"
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/__init__.py", line 120, in setup_integrations
    type(integration).setup_once()
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/falcon.py", line 113, in setup_once
    _patch_handle_exception()
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/falcon.py", line 139, in _patch_handle_exception
    original_handle_exception = falcon.API._handle_exception
AttributeError: 'functools.partial' object has no attribute '_handle_exception'
@Maraket Maraket added the bug Something isn't working label Feb 2, 2022
@owais
Copy link
Contributor

owais commented Feb 2, 2022

Looks like an issue with Falcon. Which version of Falcon are you using? Please share output of pip freeze

@Maraket
Copy link
Author

Maraket commented Feb 2, 2022

aiocontextvars==0.2.2
apispec==5.1.1
ari==0.1.3
backoff==1.10.0
boto3==1.20.46
botocore==1.23.46
certifi==2021.10.8
charset-normalizer==2.0.11
contextvars==2.4
dataclasses==0.8
datadog==0.42.0
Deprecated==1.2.13
dnspython==2.1.0
falcon==2.0.0
googleapis-common-protos==1.54.0
grpcio==1.43.0
idna==3.3
immutables==0.16
inflection==0.5.1
Jinja2==3.0.3
jmespath==0.10.0
MarkupSafe==2.0.1
marshmallow==3.14.1
opentelemetry-api==1.9.1
opentelemetry-exporter-jaeger-thrift==1.9.1
opentelemetry-exporter-otlp-proto-grpc==1.9.1
opentelemetry-instrumentation==0.28b1
opentelemetry-instrumentation-aws-lambda==0.28b1
opentelemetry-instrumentation-dbapi==0.28b1
opentelemetry-instrumentation-falcon==0.28b1
opentelemetry-instrumentation-grpc==0.28b1
opentelemetry-instrumentation-logging==0.28b1
opentelemetry-instrumentation-redis==0.28b1
opentelemetry-instrumentation-requests==0.28b1
opentelemetry-instrumentation-sqlite3==0.28b1
opentelemetry-instrumentation-urllib==0.28b1
opentelemetry-instrumentation-urllib3==0.28b1
opentelemetry-instrumentation-wsgi==0.28b1
opentelemetry-propagator-aws-xray==1.0.1
opentelemetry-propagator-b3==1.9.1
opentelemetry-proto==1.9.1
opentelemetry-sdk==1.9.1
opentelemetry-semantic-conventions==0.28b1
opentelemetry-util-http==0.28b1
panoramisk==1.4
protobuf==3.19.4
PyJWT==2.3.0
python-dateutil==2.8.2
PyYAML==6.0
redis==3.5.3
requests==2.27.1
s3transfer==0.5.0
sentry-sdk==1.4.3
six==1.16.0
splunk-opentelemetry==1.4.1
supervisor==4.2.2
swaggerpy==0.2.1
thrift==0.15.0
typing_extensions==4.0.1
urllib3==1.26.8
uWSGI==2.0.20
websocket-client==1.2.3
wrapt==1.13.3

@Maraket
Copy link
Author

Maraket commented Feb 2, 2022

I should mention that the application in question doesn't use Falcon, there is a second application that runs on the same instance that does though, and that is executed using uwsgi which runs without any issues and pushes telemetry data.

@owais owais transferred this issue from open-telemetry/opentelemetry-python Feb 3, 2022
@owais
Copy link
Contributor

owais commented Feb 3, 2022

The issue is here: https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/instrumentation/opentelemetry-instrumentation-falcon/src/opentelemetry/instrumentation/falcon/__init__.py#L151

We wrap the Falcon API class into a partial function so when it is initialized, it receives some default params. This does not break the public Falcon API as it remains a callable but it breaks Sentry because Sentry also seems to patch Falcon and assumes certain knowledge about Falcon internals.

I think we can change our implementation so instead of using a partial we use an actual class and pass in the kwargs some other way.

@owais owais self-assigned this Feb 3, 2022
@Maraket
Copy link
Author

Maraket commented Feb 3, 2022

As a side note, using a virtual environment (which is the fix we are going for in the interim) should solve this problem as if you are using falcon you would likely be running your application via uwsgi which seems to not suffer this issue

@owais owais mentioned this issue Feb 3, 2022
11 tasks
@owais
Copy link
Contributor

owais commented Feb 3, 2022

Would it be possible for you to apply these changes to your local copy of opentelemetry-instrumentation-falcon and verify if this fixes the problem?

https://github.com/open-telemetry/opentelemetry-python-contrib/pull/895/files

@Maraket
Copy link
Author

Maraket commented Feb 3, 2022

It's changed to a new error:

You are using Python 3.6. This version does not support timestamps with nanosecond precision and the OpenTelemetry SDK will use millisecond precision instead. Please refer to PEP 564 for more information. Please upgrade to Python 3.7 or newer to use nanosecond precision.
You are using Python 3.6. This version does not support timestamps with nanosecond precision and the OpenTelemetry SDK will use millisecond precision instead. Please refer to PEP 564 for more information. Please upgrade to Python 3.7 or newer to use nanosecond precision.
2022-02-03 06:01:58,614 ERROR [sitecustomize] [sitecustomize.py:87] [trace_id=0 span_id=0 resource.service.name=test_thing] - Instrumenting of falcon failed
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 84, in _load_instrumentors
    distro.load_instrumentor(entry_point, skip_dep_check=True)
  File "/usr/local/lib/python3.6/site-packages/splunk_otel/distro.py", line 44, in load_instrumentor
    super().load_instrumentor(entry_point, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/distro.py", line 63, in load_instrumentor
    instrumentor().instrument(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/instrumentor.py", line 110, in instrument
    **kwargs
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/falcon/__init__.py", line 156, in _instrument
    setattr(falcon, _instrument_app, API)
NameError: name 'API' is not defined
2022-02-03 06:01:58,623 ERROR [sitecustomize] [sitecustomize.py:119] [trace_id=0 span_id=0 resource.service.name=test_thing] - Failed to auto initialize opentelemetry
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 117, in initialize
    _load_instrumentors(distro)
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 88, in _load_instrumentors
    raise exc
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 84, in _load_instrumentors
    distro.load_instrumentor(entry_point, skip_dep_check=True)
  File "/usr/local/lib/python3.6/site-packages/splunk_otel/distro.py", line 44, in load_instrumentor
    super().load_instrumentor(entry_point, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/distro.py", line 63, in load_instrumentor
    instrumentor().instrument(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/instrumentor.py", line 110, in instrument
    **kwargs
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/falcon/__init__.py", line 156, in _instrument
    setattr(falcon, _instrument_app, API)
NameError: name 'API' is not defined
about to run
sentry run

A similar error message is occuring now in the uwsgi app now though:

2022-02-03 06:11:42.954 ERROR    Instrumenting of falcon failed
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/splunk_otel/tracing.py", line 82, in _load_instrumentors
    entry_point.load()().instrument()
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/instrumentor.py", line 110, in instrument
    **kwargs
  File "/usr/local/lib/python3.6/site-packages/opentelemetry/instrumentation/falcon/__init__.py", line 156, in _instrument
    setattr(falcon, _instrument_app, API)
NameError: name 'API' is not defined
2022-02-03 06:11:43.345 WARNING  Already shutdown, dropping span.
Sentry is attempting to send 2 pending error messages
Waiting up to 2 seconds

@owais
Copy link
Contributor

owais commented Feb 3, 2022

Yes, of course. I made a silly mistake just before my last push. Fixed now. Please test again if possible. Thanks!

@Maraket
Copy link
Author

Maraket commented Feb 3, 2022

@owais seems to be working

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants