Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exception in detector <opentelemetry.resource.detector.azure.vm.AzureVMResourceDetector object at 0x735573070750>, ignoring #33301

Closed
Davidnoguerabuxeda79 opened this issue Nov 27, 2023 · 9 comments
Closed

Exception in detector <opentelemetry.resource.detector.azure.vm.AzureVMResourceDetector object at 0x735573070750>, ignoring #33301

Davidnoguerabuxeda79 opened this issue Nov 27, 2023 · 9 comments
Labels
customer-reported Issues that are reported by GitHub users external to the Azure organization. issue-addressed Workflow: The Azure SDK team believes it to be addressed and ready to close. Monitor - Distro Monitor OpenTelemetry Distro Monitor Monitor, Monitor Ingestion, Monitor Query question The issue doesn't require a change to the product in order to be resolved. Most issues start as that

Comments

@Davidnoguerabuxeda79
Copy link

  • Package Name: opentelemetry.resource.detector.azure.vm.AzureVMResourceDetector
  • Package Version: opentelemetry sdk-1.21.0-py3-none-any. whl
  • Operating System: Linux
  • Python Version: 3.11

Describe the bug
A Basic webapp it takes 10m to start the container with opentelemetry with default configuration on python, the same code in one public webapp or in a public app service environment it works fine, but on a isolated service environment the traffic goes over a firewall and to a proxy and it takes long time to start the container.

2023-11-27T15:45:30.236Z INFO - docker run -d --expose=8000 --name az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69 -e WEBSITE_SITE_NAME=az-euw-cxb-inn-idmia-caso07-webapp04-pre -e WEBSITE_AUTH_ENABLED=False -e PORT=8000 -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=az-euw-cxb-inn-idmia-caso07-webapp04-pre.az-euw-cxb-inn-idmia-tr-appse02-pre.appserviceenvironment.net -e WEBSITE_INSTANCE_ID=e7adbffe7a9306cf3b69e5759917ea1be4e8f496a9807afa71e7308ee38d5df3 -e HTTP_LOGGING_ENABLED=1 -e WEBSITE_USE_DIAGNOSTIC_SERVER=False appsvc/python:3.11_20230810.1.tuxprod python -m uvicorn main:app --host 0.0.0.0

2023-11-27T15:45:32.147Z INFO - Initiating warmup request to container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69_msiProxy for site az-euw-cxb-inn-idmia-caso07-webapp04-pre
2023-11-27T15:45:32.182Z INFO - Container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69_msiProxy for site az-euw-cxb-inn-idmia-caso07-webapp04-pre initialized successfully and is ready to serve requests.
2023-11-27T15:45:32.183Z INFO - Initiating warmup request to container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69 for site az-euw-cxb-inn-idmia-caso07-webapp04-pre
2023-11-27T15:45:47.577Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 15.429939 sec
2023-11-27T15:46:02.669Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 30.5222708 sec
2023-11-27T15:46:53.829Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 81.6822779 sec
2023-11-27T15:47:08.945Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 96.7977411 sec
2023-11-27T15:47:24.135Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 111.9885401 sec
2023-11-27T15:47:39.317Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 127.1690888 sec
2023-11-27T15:47:54.554Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 142.4067304 sec
2023-11-27T15:48:09.645Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 157.497903 sec
2023-11-27T15:48:53.263Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 201.1167256 sec
2023-11-27T15:49:08.390Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 216.2429898 sec
2023-11-27T15:49:23.509Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 231.3624886 sec
2023-11-27T15:49:38.596Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 246.4487662 sec
2023-11-27T15:49:53.702Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 261.5554278 sec
2023-11-27T15:50:08.811Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 276.6646104 sec
2023-11-27T15:50:23.930Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 291.7827126 sec
2023-11-27T15:51:05.728Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 333.581189 sec
2023-11-27T15:51:20.844Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 348.6968061 sec
2023-11-27T15:51:35.928Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 363.7812372 sec
2023-11-27T15:51:51.078Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 378.9314144 sec
2023-11-27T15:52:06.242Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 394.0948935 sec
2023-11-27T15:52:21.862Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 409.7148284 sec
2023-11-27T15:52:37.271Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 425.1240091 sec
2023-11-27T15:52:52.680Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 440.5328074 sec
2023-11-27T15:53:26.125Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 473.9760907 sec
2023-11-27T15:53:41.238Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 489.0909973 sec
2023-11-27T15:53:56.364Z INFO - Waiting for response to warmup request for container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69. Elapsed time = 504.2169282 sec
2023-11-27T15:54:32.198Z INFO - Container az-euw-cxb-inn-idmia-caso07-webapp04-pre_3_e5a2ac69 for site az-euw-cxb-inn-idmia-caso07-webapp04-pre initialized successfully and is ready to serve requests.

This is the log of message warning of the app during the deployment:

2023-11-27T15:54:52.333304060Z _____
2023-11-27T15:54:52.333332861Z / _ \ __________ _________ ____
2023-11-27T15:54:52.333336561Z / /\ __ / | _ __ / __
2023-11-27T15:54:52.333339361Z / | / /| | /| | /\ /
2023-11-27T15:54:52.333342061Z | /_____ _/ || __ >
2023-11-27T15:54:52.333345461Z / / /
2023-11-27T15:54:52.333349361Z A P P S E R V I C E O N L I N U X
2023-11-27T15:54:52.333353461Z
2023-11-27T15:54:52.333357461Z Documentation: http://aka.ms/webapp-linux
2023-11-27T15:54:52.333361861Z Python 3.11.4
2023-11-27T15:54:52.333365061Z Note: Any data outside '/home' is not persisted
2023-11-27T15:54:52.732183448Z Starting OpenBSD Secure Shell server: sshd.
2023-11-27T15:54:52.780885296Z Site's appCommandLine: python -m uvicorn main:app --host 0.0.0.0
2023-11-27T15:54:52.937716660Z Starting periodic command scheduler: cron.
2023-11-27T15:54:52.941068098Z Launching oryx with: create-script -appPath /home/site/wwwroot -output /opt/startup/startup.sh -virtualEnvName antenv -defaultApp /opt/defaultsite -userStartupCommand 'python -m uvicorn main:app --host 0.0.0.0'
2023-11-27T15:54:52.966342782Z Found build manifest file at '/home/site/wwwroot/oryx-manifest.toml'. Deserializing it...
2023-11-27T15:54:52.978332217Z Build Operation ID: d0b9b338b4e4d0aa
2023-11-27T15:54:52.978353217Z Oryx Version: 0.2.20230707.1, Commit: 0bd28e69919b5e8beba451e8677e3345f0be8361, ReleaseTagName: 20230707.1
2023-11-27T15:54:52.978368818Z Output is compressed. Extracting it...
2023-11-27T15:54:52.978373918Z Extracting '/home/site/wwwroot/output.tar.gz' to directory '/tmp/8dbec192cced8d7'...
2023-11-27T15:54:54.317775886Z App path is set to '/tmp/8dbec192cced8d7'
2023-11-27T15:55:00.322497123Z Writing output script to '/opt/startup/startup.sh'
2023-11-27T15:55:06.394262936Z Using packages from virtual environment antenv located at /tmp/8dbec192cced8d7/antenv.
2023-11-27T15:55:06.394299637Z Updated PYTHONPATH to '/opt/startup/app_logs:/tmp/8dbec192cced8d7/antenv/lib/python3.11/site-packages'
2023-11-27T15:55:16.326474197Z AppInsightsFastApi.zip
2023-11-27T15:57:25.758467401Z Exception in detector <opentelemetry.resource.detector.azure.vm.AzureVMResourceDetector object at 0x73557218ce10>, ignoring
2023-11-27T15:59:36.905010353Z Exception in detector <opentelemetry.resource.detector.azure.vm.AzureVMResourceDetector object at 0x735572882850>, ignoring
2023-11-27T16:03:54.285590933Z INFO: Started server process [76]
2023-11-27T16:03:54.294562533Z INFO: Waiting for application startup.
2023-11-27T16:03:54.294586734Z INFO: Application startup complete.
2023-11-27T16:03:54.294592834Z INFO: Uvicorn running on http://0.0.0.0:8000/ (Press CTRL+C to quit)

Checking the log it seems a possible problem on AzureVMResourceDetector, but I'm not sure. May be he needs to reach some url or ip and client firewall or proxy is blocking, if that is the case we can configure noproxy variable or a route to bypass the firewall, but we need to know which.

Can you provide some guidance to what communications requieres opentelemetry to work properly?
Azure app insights is working and resolving it with private link scope and we are getting the rights.

Regards.

To Reproduce
Steps to reproduce the behavior:

  1. Try to deploy one linux python isolated webapp using opentelemetry with nsg, onprem firewall & proxy.
  2. I attached the zip file of the application.

Expected behavior
The opentelemetry application should start in less than 2m

Screenshots
If applicable, add screenshots to help explain your problem.
We can do a session if you want or I can provide the python code that are using.

Additional context
Same application without being rear a proxy + nsg + firewall it works.
@github-actions github-actions bot added customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Nov 27, 2023
@lzchen
Copy link
Member

lzchen commented Nov 27, 2023

Duplicate of #33295

@lzchen lzchen marked this as a duplicate of #33295 Nov 27, 2023
@Davidnoguerabuxeda79
Copy link
Author

I don't know if it's the same problem, in my case it takes 10m to start the app, and it's running on an isolated webapp, I just saw this message on the app log meanwhile it was trying to start.

@jeremydvoss jeremydvoss mentioned this issue Nov 27, 2023
Merged
7 tasks
@xiangyan99 xiangyan99 added Monitor Monitor, Monitor Ingestion, Monitor Query and removed needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. labels Nov 27, 2023
@github-actions github-actions bot added the needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team label Nov 27, 2023
@jeremydvoss jeremydvoss added the Monitor - Distro Monitor OpenTelemetry Distro label Dec 5, 2023
@jeremydvoss
Copy link
Member

We have release azure-monitor-opentelemetry=1.1.1 . This includes the ability to disable resource detectors. For instance, in order to disable the VM detector, but leave the App Service detector on, the customer can set their environment variable OTEL_EXPERIMENTAL_RESOURCE_DETECTORS="azure_app_service"

Let me know if this solves the problem.

@xiangyan99 xiangyan99 added the issue-addressed Workflow: The Azure SDK team believes it to be addressed and ready to close. label Dec 7, 2023
@github-actions github-actions bot removed the needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team label Dec 7, 2023
@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 7, 2023

Hi @Davidnoguerabuxeda79. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

@github-actions GitHub Actions
Copy link

Hi @Davidnoguerabuxeda79, since you haven’t asked that we /unresolve the issue, we’ll close this out. If you believe further discussion is needed, please add a comment /unresolve to reopen the issue.

@jeremydvoss
Copy link
Member

I may have reproduced this issue in Functions. My app fails to start within the time limit if the VM resource detector is enabled. Since this only happened for me in a single function app and does not occur locally. It is unclear why the VM detector can sometimes take so long.

Will investigate more.

@Davidnoguerabuxeda79
Copy link
Author

Davidnoguerabuxeda79 commented Dec 16, 2023 via email

@jeremydvoss
Copy link
Member

If you are experiencing issues related to the AzureVMResourceDetector, please update the package to https://pypi.org/project/opentelemetry-resource-detector-azure/0.1.1/

@jeremydvoss
Copy link
Member

The issue stems from an unclear timeout in the OTel SDK. My fix will be in the next release. In order to not trigger the 5 second timeout, the VM Resource Detector now sets its own timeout to 4s. Please update to opentelemetry-resource-detector-azure=0.1.3

@github-actions github-actions bot locked and limited conversation to collaborators Apr 28, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
customer-reported Issues that are reported by GitHub users external to the Azure organization. issue-addressed Workflow: The Azure SDK team believes it to be addressed and ready to close. Monitor - Distro Monitor OpenTelemetry Distro Monitor Monitor, Monitor Ingestion, Monitor Query question The issue doesn't require a change to the product in order to be resolved. Most issues start as that
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jeremydvoss @lzchen @xiangyan99 @Davidnoguerabuxeda79