Automatic build and push to repo #11

LotharSee · 2016-07-04T16:04:32Z

Use CircleCI to manage our versions and deploy wheels automatically.
The built wheels are pushed to our S3 repo. These are usable as wheelhouses thanks to mkwheelhouse. As a result, a single URL has to be added to the pip config/CLI to find ddtrace.

Here is the organisation.

"Dev" versions

Any push to master or develop trigger a build, then a deploy if it succeed.
The version of such a wheel is __version__+$BRANCH$BUILD_NUM. For example: 0.2.0+master123 or 0.2.1+develop142.
These are the packages we will use in dogweb. The idea would be to use develop as a way to tests branches on staging, not yet ready to be merged to master.
dogweb prod would use master versions of ddtrace.

For both staging and dogweb, we will need to manually update the dogweb's setup.py to upgrade ddtrace. Since ddtrace versions used by staging and prod might evolve independently, I'd suggest to ship the version changes independently (different commit / branch), and to push the develop versions only to the staging branch.

To keep the repo "not-too-public" (pypi.datadoghq.com is a way too common name), put all these wheels in the dir apm_dev.

To install the wheel from the version I just pushed.

pip install --find-links=https://s3.amazonaws.com/pypi.datadoghq.com/apm_dev/index.html ddtrace==0.2.0+develop93

"Public" versions

A "public" version is a version that we will officially tag and "release" to our first used. It allows us to run versions on dogweb prod without pushing them to our users.

To release a new version:

on master
Increase the package version__version__ in ddtrace/__init__.py (ex: 0.2.1)
Commit
Tag with the version vX.Y.Z (ex: v0.2.1)
Push

CircleCI will automatically build and push it to a different directory apm_unstable. This way there is no risk for our "dev" packages to break our users.

To install it (example):

pip install --find-links=https://s3.amazonaws.com/pypi.datadoghq.com/apm_unstable/index.html ddtrace==0.2.1

Finally, here is what it looks like when switching dogweb to this new structure:
https://github.com/DataDog/dogweb/compare/benjamin/ddtrace-wheelhouse

Make CircleCI build the wheel (branch develop, master, tag vA.B.C) and publish it to our S3 repo

talwai · 2016-07-05T12:10:27Z

ddtrace/__init__.py

typo datadaog

talwai · 2016-07-05T12:12:53Z

looks good to me! 👍

LeoCavaille · 2016-07-05T12:16:48Z

LGTM 👍

awang · 2016-07-05T14:18:56Z

can we rename the dirs from apm_{dev,unstable} to trace_{dev,unstable}? to make the public directory a little less conspicuous

…est (#11…" This reverts commit 8c4d62d.

## Description https://datadoghq.atlassian.net/browse/PROF-13114 This PR makes sure the Python Profiler's signal handler (for `SIGSEGV` and `SIGBUS`) is properly installed when the Sampler thread starts. Note that this (reinstalling our signal handler) does NOT break any other signal handler (Python's or another extension's) as our signal handler only swallows faults / jumps to the recovery path if it's been "armed" (otherwise it re-raises). What matters is that we should be the "first responder" when a fault happens. This is an attempt to fix a crash we saw in the testing environment where some workloads receive segmentation faults clearly coming from `safe_memcpy` in FastAPI / Django apps. The "real" root cause isn't yet known of me – Django and FastAPI don't seem to use `PYTHONFAULTHANDLER` or `faulthandler` based on the Github code – but after deploying those changes, we stopped seeing those crashes (0 in the past 4 days). <img width="2089" height="480" alt="image" src="https://github.com/user-attachments/assets/c554ec0c-cfae-4311-bded-8082d8f79ed9" /> ``` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x0000755d4295c18f safe_memcpy #1 0x0000755d42954879 copy_memory #2 0x0000755d42956826 GenInfo::create #3 0x0000755d42958c25 TaskInfo::create #4 0x0000755d42958cc7 TaskInfo::create #5 0x0000755d4295a618 ThreadInfo::unwind_tasks #6 0x0000755d4295b98b std::_Function_handler<void (_ts*, ThreadInfo&), Datadog::Sampler::sampling_thread(unsigned long)::{lambda(InterpreterInfo&)#1}::operator()(InterpreterInfo&) const::{lambda(_ts*, ThreadInfo&)#1}>::_M_invoke #7 0x0000755d42959906 for_each_thread #8 0x0000755d429599e5 std::_Function_handler<void (InterpreterInfo&), Datadog::Sampler::sampling_thread(unsigned long)::{lambda(InterpreterInfo&)#1}>::_M_invoke #9 0x0000755d4295c5b2 Datadog::Sampler::sampling_thread #10 0x0000755d4295c685 call_sampling_thread #11 0x0000755d45aeeea7 start_thread #12 0x0000755d45c05def clone ```

## Description https://datadoghq.atlassian.net/browse/PROF-13112 This is an attempt to address the following crash. There seems to be a case (that I wasn't able to reproduce in a Docker image, but maybe my "code environment" didn't match the customer's exactly) where using `uvloop` results in a crash caused by `PeriodicThread_start` after `uvloop` tries to restart Threads after a fork. ``` #0 0x00007f9a7acdbefa cfree #1 0x00007f9a7accc6b5 pthread_create #2 0x00007f9a7a63aaa5 std::thread::_M_start_thread #3 0x00007f9a7a639d18 PeriodicThread_start #4 0x00007f9a2e71d565 __pyx_f_6uvloop_4loop_9UVProcess__after_fork (uvloop/loop.c:120214:3) #5 0x00007f9a2e6369a8 __pyx_f_6uvloop_4loop___get_fork_handler (uvloop/loop.c:163075:24) #6 0x00007f9a7ad17073 __fork #7 0x00007f9a2e732d62 uv__spawn_and_init_child_fork (src/unix/process.c:831:10) #8 0x00007f9a2e732d62 uv__spawn_and_init_child (src/unix/process.c:919:9) #9 0x00007f9a2e732d62 uv_spawn (src/unix/process.c:1013:18) #10 0x00007f9a2e71fb87 __pyx_f_6uvloop_4loop_9UVProcess__init (uvloop/loop.c:119056:19) #11 0x00007f9a2e711bf7 __pyx_f_6uvloop_4loop_18UVProcessTransport_new (uvloop/loop.c:126866:16) #12 0x00007f9a2e712aa7 __pyx_gb_6uvloop_4loop_4Loop_116generator16 (uvloop/loop.c:54030:28) #13 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #14 0x00007f9a2e699f8a __Pyx_Coroutine_AmSend (uvloop/loop.c:196492:18) #15 0x00007f9a2e69a052 __Pyx_Coroutine_Yield_From_Coroutine (uvloop/loop.c:197380:14) #16 0x00007f9a2e69b0e5 __Pyx_Coroutine_Yield_From (uvloop/loop.c:197408:16) #17 0x00007f9a2e69b0e5 __pyx_gb_6uvloop_4loop_4Loop_122generator18 (uvloop/loop.c:55002:15) #18 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #19 0x00007f9a2e69bb86 __Pyx_Generator_Next (uvloop/loop.c:196581:18) #20 0x00007f9a2e6398eb __Pyx_PyObject_Call (uvloop/loop.c:191431:15) #21 0x00007f9a2e6398eb __Pyx_PyObject_FastCallDict (uvloop/loop.c:191552:16) #22 0x00007f9a2e715a69 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66873:27) #23 0x00007f9a2e71996b __pyx_f_6uvloop_4loop_4Loop__on_idle (uvloop/loop.c:17975:25) #24 0x00007f9a2e713e52 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66927:24) #25 0x00007f9a2e715c88 __pyx_f_6uvloop_4loop_cb_idle_callback (uvloop/loop.c:87335:19) #26 0x00007f9a2e731311 uv__run_idle (unix/loop-watcher.c:68:1) #27 0x00007f9a2e72e647 uv_run (src/unix/core.c:439:5) #28 0x00007f9a2e64fdb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (uvloop/loop.c:18458:23) #29 0x00007f9a2e6b7e50 __pyx_f_6uvloop_4loop_4Loop__run (uvloop/loop.c:18876:18) #30 0x00007f9a2e6c8cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (uvloop/loop.c:31528:18) #31 0x00007f9a2e6c8cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (uvloop/loop.c:31331:13) #32 0x00007f9a7b065c25 PyObject_VectorcallMethod #33 0x00007f9a2e6ccd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (uvloop/loop.c:33768:23) #34 0x00007f9a2e6ce591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (uvloop/loop.c:33318:13) #35 0x00007f9a7b039358 PyObject_Vectorcall ``` ## Fix The `_after_fork` boolean field marks that this thread object is in a "post-fork zombie state." When the flag is set to true, Thread methods (e.g. `join`) become no-ops because the threads do not exist anymore so we should not try to do something with them. By checking that same flag, we can tell that we are trying to start a Thread that doesn't really exist and so we shouldn't try to do it.

## Description https://datadoghq.atlassian.net/browse/PROF-13112 This is an attempt to address the following crash. There seems to be a case (that I wasn't able to reproduce in a Docker image, but maybe my "code environment" didn't match the customer's exactly) where using `uvloop` results in a crash caused by `PeriodicThread_start` after `uvloop` tries to restart Threads after a fork. ``` #0 0x00007f9a7acdbefa cfree #1 0x00007f9a7accc6b5 pthread_create #2 0x00007f9a7a63aaa5 std::thread::_M_start_thread #3 0x00007f9a7a639d18 PeriodicThread_start #4 0x00007f9a2e71d565 __pyx_f_6uvloop_4loop_9UVProcess__after_fork (uvloop/loop.c:120214:3) #5 0x00007f9a2e6369a8 __pyx_f_6uvloop_4loop___get_fork_handler (uvloop/loop.c:163075:24) #6 0x00007f9a7ad17073 __fork #7 0x00007f9a2e732d62 uv__spawn_and_init_child_fork (src/unix/process.c:831:10) #8 0x00007f9a2e732d62 uv__spawn_and_init_child (src/unix/process.c:919:9) #9 0x00007f9a2e732d62 uv_spawn (src/unix/process.c:1013:18) #10 0x00007f9a2e71fb87 __pyx_f_6uvloop_4loop_9UVProcess__init (uvloop/loop.c:119056:19) #11 0x00007f9a2e711bf7 __pyx_f_6uvloop_4loop_18UVProcessTransport_new (uvloop/loop.c:126866:16) #12 0x00007f9a2e712aa7 __pyx_gb_6uvloop_4loop_4Loop_116generator16 (uvloop/loop.c:54030:28) #13 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #14 0x00007f9a2e699f8a __Pyx_Coroutine_AmSend (uvloop/loop.c:196492:18) #15 0x00007f9a2e69a052 __Pyx_Coroutine_Yield_From_Coroutine (uvloop/loop.c:197380:14) #16 0x00007f9a2e69b0e5 __Pyx_Coroutine_Yield_From (uvloop/loop.c:197408:16) #17 0x00007f9a2e69b0e5 __pyx_gb_6uvloop_4loop_4Loop_122generator18 (uvloop/loop.c:55002:15) #18 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #19 0x00007f9a2e69bb86 __Pyx_Generator_Next (uvloop/loop.c:196581:18) #20 0x00007f9a2e6398eb __Pyx_PyObject_Call (uvloop/loop.c:191431:15) #21 0x00007f9a2e6398eb __Pyx_PyObject_FastCallDict (uvloop/loop.c:191552:16) #22 0x00007f9a2e715a69 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66873:27) #23 0x00007f9a2e71996b __pyx_f_6uvloop_4loop_4Loop__on_idle (uvloop/loop.c:17975:25) #24 0x00007f9a2e713e52 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66927:24) #25 0x00007f9a2e715c88 __pyx_f_6uvloop_4loop_cb_idle_callback (uvloop/loop.c:87335:19) #26 0x00007f9a2e731311 uv__run_idle (unix/loop-watcher.c:68:1) #27 0x00007f9a2e72e647 uv_run (src/unix/core.c:439:5) #28 0x00007f9a2e64fdb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (uvloop/loop.c:18458:23) #29 0x00007f9a2e6b7e50 __pyx_f_6uvloop_4loop_4Loop__run (uvloop/loop.c:18876:18) #30 0x00007f9a2e6c8cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (uvloop/loop.c:31528:18) #31 0x00007f9a2e6c8cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (uvloop/loop.c:31331:13) #32 0x00007f9a7b065c25 PyObject_VectorcallMethod #33 0x00007f9a2e6ccd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (uvloop/loop.c:33768:23) #34 0x00007f9a2e6ce591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (uvloop/loop.c:33318:13) #35 0x00007f9a7b039358 PyObject_Vectorcall ``` ## Fix The `_after_fork` boolean field marks that this thread object is in a "post-fork zombie state." When the flag is set to true, Thread methods (e.g. `join`) become no-ops because the threads do not exist anymore so we should not try to do something with them. By checking that same flag, we can tell that we are trying to start a Thread that doesn't really exist and so we shouldn't try to do it. (cherry picked from commit 4c69fdd)

…15858) Backport 4c69fdd from #15798 to 4.1. ## Description https://datadoghq.atlassian.net/browse/PROF-13112 This is an attempt to address the following crash. There seems to be a case (that I wasn't able to reproduce in a Docker image, but maybe my "code environment" didn't match the customer's exactly) where using `uvloop` results in a crash caused by `PeriodicThread_start` after `uvloop` tries to restart Threads after a fork. ``` #0 0x00007f9a7acdbefa cfree #1 0x00007f9a7accc6b5 pthread_create #2 0x00007f9a7a63aaa5 std::thread::_M_start_thread #3 0x00007f9a7a639d18 PeriodicThread_start #4 0x00007f9a2e71d565 __pyx_f_6uvloop_4loop_9UVProcess__after_fork (uvloop/loop.c:120214:3) #5 0x00007f9a2e6369a8 __pyx_f_6uvloop_4loop___get_fork_handler (uvloop/loop.c:163075:24) #6 0x00007f9a7ad17073 __fork #7 0x00007f9a2e732d62 uv__spawn_and_init_child_fork (src/unix/process.c:831:10) #8 0x00007f9a2e732d62 uv__spawn_and_init_child (src/unix/process.c:919:9) #9 0x00007f9a2e732d62 uv_spawn (src/unix/process.c:1013:18) #10 0x00007f9a2e71fb87 __pyx_f_6uvloop_4loop_9UVProcess__init (uvloop/loop.c:119056:19) #11 0x00007f9a2e711bf7 __pyx_f_6uvloop_4loop_18UVProcessTransport_new (uvloop/loop.c:126866:16) #12 0x00007f9a2e712aa7 __pyx_gb_6uvloop_4loop_4Loop_116generator16 (uvloop/loop.c:54030:28) #13 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #14 0x00007f9a2e699f8a __Pyx_Coroutine_AmSend (uvloop/loop.c:196492:18) #15 0x00007f9a2e69a052 __Pyx_Coroutine_Yield_From_Coroutine (uvloop/loop.c:197380:14) #16 0x00007f9a2e69b0e5 __Pyx_Coroutine_Yield_From (uvloop/loop.c:197408:16) #17 0x00007f9a2e69b0e5 __pyx_gb_6uvloop_4loop_4Loop_122generator18 (uvloop/loop.c:55002:15) #18 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #19 0x00007f9a2e69bb86 __Pyx_Generator_Next (uvloop/loop.c:196581:18) #20 0x00007f9a2e6398eb __Pyx_PyObject_Call (uvloop/loop.c:191431:15) #21 0x00007f9a2e6398eb __Pyx_PyObject_FastCallDict (uvloop/loop.c:191552:16) #22 0x00007f9a2e715a69 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66873:27) #23 0x00007f9a2e71996b __pyx_f_6uvloop_4loop_4Loop__on_idle (uvloop/loop.c:17975:25) #24 0x00007f9a2e713e52 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66927:24) #25 0x00007f9a2e715c88 __pyx_f_6uvloop_4loop_cb_idle_callback (uvloop/loop.c:87335:19) #26 0x00007f9a2e731311 uv__run_idle (unix/loop-watcher.c:68:1) #27 0x00007f9a2e72e647 uv_run (src/unix/core.c:439:5) #28 0x00007f9a2e64fdb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (uvloop/loop.c:18458:23) #29 0x00007f9a2e6b7e50 __pyx_f_6uvloop_4loop_4Loop__run (uvloop/loop.c:18876:18) #30 0x00007f9a2e6c8cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (uvloop/loop.c:31528:18) #31 0x00007f9a2e6c8cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (uvloop/loop.c:31331:13) #32 0x00007f9a7b065c25 PyObject_VectorcallMethod #33 0x00007f9a2e6ccd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (uvloop/loop.c:33768:23) #34 0x00007f9a2e6ce591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (uvloop/loop.c:33318:13) #35 0x00007f9a7b039358 PyObject_Vectorcall ``` ## Fix The `_after_fork` boolean field marks that this thread object is in a "post-fork zombie state." When the flag is set to true, Thread methods (e.g. `join`) become no-ops because the threads do not exist anymore so we should not try to do something with them. By checking that same flag, we can tell that we are trying to start a Thread that doesn't really exist and so we shouldn't try to do it. Co-authored-by: Thomas Kowalski <[email protected]>

## Description https://datadoghq.atlassian.net/browse/PROF-13112 This is an attempt to address the following crash. There seems to be a case (that I wasn't able to reproduce in a Docker image, but maybe my "code environment" didn't match the customer's exactly) where using `uvloop` results in a crash caused by `PeriodicThread_start` after `uvloop` tries to restart Threads after a fork. ``` #0 0x00007f9a7acdbefa cfree #1 0x00007f9a7accc6b5 pthread_create #2 0x00007f9a7a63aaa5 std::thread::_M_start_thread #3 0x00007f9a7a639d18 PeriodicThread_start #4 0x00007f9a2e71d565 __pyx_f_6uvloop_4loop_9UVProcess__after_fork (uvloop/loop.c:120214:3) #5 0x00007f9a2e6369a8 __pyx_f_6uvloop_4loop___get_fork_handler (uvloop/loop.c:163075:24) #6 0x00007f9a7ad17073 __fork #7 0x00007f9a2e732d62 uv__spawn_and_init_child_fork (src/unix/process.c:831:10) #8 0x00007f9a2e732d62 uv__spawn_and_init_child (src/unix/process.c:919:9) #9 0x00007f9a2e732d62 uv_spawn (src/unix/process.c:1013:18) #10 0x00007f9a2e71fb87 __pyx_f_6uvloop_4loop_9UVProcess__init (uvloop/loop.c:119056:19) #11 0x00007f9a2e711bf7 __pyx_f_6uvloop_4loop_18UVProcessTransport_new (uvloop/loop.c:126866:16) #12 0x00007f9a2e712aa7 __pyx_gb_6uvloop_4loop_4Loop_116generator16 (uvloop/loop.c:54030:28) #13 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #14 0x00007f9a2e699f8a __Pyx_Coroutine_AmSend (uvloop/loop.c:196492:18) #15 0x00007f9a2e69a052 __Pyx_Coroutine_Yield_From_Coroutine (uvloop/loop.c:197380:14) #16 0x00007f9a2e69b0e5 __Pyx_Coroutine_Yield_From (uvloop/loop.c:197408:16) #17 0x00007f9a2e69b0e5 __pyx_gb_6uvloop_4loop_4Loop_122generator18 (uvloop/loop.c:55002:15) #18 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #19 0x00007f9a2e69bb86 __Pyx_Generator_Next (uvloop/loop.c:196581:18) #20 0x00007f9a2e6398eb __Pyx_PyObject_Call (uvloop/loop.c:191431:15) #21 0x00007f9a2e6398eb __Pyx_PyObject_FastCallDict (uvloop/loop.c:191552:16) #22 0x00007f9a2e715a69 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66873:27) #23 0x00007f9a2e71996b __pyx_f_6uvloop_4loop_4Loop__on_idle (uvloop/loop.c:17975:25) #24 0x00007f9a2e713e52 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66927:24) #25 0x00007f9a2e715c88 __pyx_f_6uvloop_4loop_cb_idle_callback (uvloop/loop.c:87335:19) #26 0x00007f9a2e731311 uv__run_idle (unix/loop-watcher.c:68:1) #27 0x00007f9a2e72e647 uv_run (src/unix/core.c:439:5) #28 0x00007f9a2e64fdb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (uvloop/loop.c:18458:23) #29 0x00007f9a2e6b7e50 __pyx_f_6uvloop_4loop_4Loop__run (uvloop/loop.c:18876:18) #30 0x00007f9a2e6c8cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (uvloop/loop.c:31528:18) #31 0x00007f9a2e6c8cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (uvloop/loop.c:31331:13) #32 0x00007f9a7b065c25 PyObject_VectorcallMethod #33 0x00007f9a2e6ccd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (uvloop/loop.c:33768:23) #34 0x00007f9a2e6ce591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (uvloop/loop.c:33318:13) #35 0x00007f9a7b039358 PyObject_Vectorcall ``` ## Fix The `_after_fork` boolean field marks that this thread object is in a "post-fork zombie state." When the flag is set to true, Thread methods (e.g. `join`) become no-ops because the threads do not exist anymore so we should not try to do something with them. By checking that same flag, we can tell that we are trying to start a Thread that doesn't really exist and so we shouldn't try to do it. (cherry picked from commit 4c69fdd)

…15861) Backport 4c69fdd from #15798 to 4.0. ## Description https://datadoghq.atlassian.net/browse/PROF-13112 This is an attempt to address the following crash. There seems to be a case (that I wasn't able to reproduce in a Docker image, but maybe my "code environment" didn't match the customer's exactly) where using `uvloop` results in a crash caused by `PeriodicThread_start` after `uvloop` tries to restart Threads after a fork. ``` #0 0x00007f9a7acdbefa cfree #1 0x00007f9a7accc6b5 pthread_create #2 0x00007f9a7a63aaa5 std::thread::_M_start_thread #3 0x00007f9a7a639d18 PeriodicThread_start #4 0x00007f9a2e71d565 __pyx_f_6uvloop_4loop_9UVProcess__after_fork (uvloop/loop.c:120214:3) #5 0x00007f9a2e6369a8 __pyx_f_6uvloop_4loop___get_fork_handler (uvloop/loop.c:163075:24) #6 0x00007f9a7ad17073 __fork #7 0x00007f9a2e732d62 uv__spawn_and_init_child_fork (src/unix/process.c:831:10) #8 0x00007f9a2e732d62 uv__spawn_and_init_child (src/unix/process.c:919:9) #9 0x00007f9a2e732d62 uv_spawn (src/unix/process.c:1013:18) #10 0x00007f9a2e71fb87 __pyx_f_6uvloop_4loop_9UVProcess__init (uvloop/loop.c:119056:19) #11 0x00007f9a2e711bf7 __pyx_f_6uvloop_4loop_18UVProcessTransport_new (uvloop/loop.c:126866:16) #12 0x00007f9a2e712aa7 __pyx_gb_6uvloop_4loop_4Loop_116generator16 (uvloop/loop.c:54030:28) #13 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #14 0x00007f9a2e699f8a __Pyx_Coroutine_AmSend (uvloop/loop.c:196492:18) #15 0x00007f9a2e69a052 __Pyx_Coroutine_Yield_From_Coroutine (uvloop/loop.c:197380:14) #16 0x00007f9a2e69b0e5 __Pyx_Coroutine_Yield_From (uvloop/loop.c:197408:16) #17 0x00007f9a2e69b0e5 __pyx_gb_6uvloop_4loop_4Loop_122generator18 (uvloop/loop.c:55002:15) #18 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) #19 0x00007f9a2e69bb86 __Pyx_Generator_Next (uvloop/loop.c:196581:18) #20 0x00007f9a2e6398eb __Pyx_PyObject_Call (uvloop/loop.c:191431:15) #21 0x00007f9a2e6398eb __Pyx_PyObject_FastCallDict (uvloop/loop.c:191552:16) #22 0x00007f9a2e715a69 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66873:27) #23 0x00007f9a2e71996b __pyx_f_6uvloop_4loop_4Loop__on_idle (uvloop/loop.c:17975:25) #24 0x00007f9a2e713e52 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66927:24) #25 0x00007f9a2e715c88 __pyx_f_6uvloop_4loop_cb_idle_callback (uvloop/loop.c:87335:19) #26 0x00007f9a2e731311 uv__run_idle (unix/loop-watcher.c:68:1) #27 0x00007f9a2e72e647 uv_run (src/unix/core.c:439:5) #28 0x00007f9a2e64fdb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (uvloop/loop.c:18458:23) #29 0x00007f9a2e6b7e50 __pyx_f_6uvloop_4loop_4Loop__run (uvloop/loop.c:18876:18) #30 0x00007f9a2e6c8cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (uvloop/loop.c:31528:18) #31 0x00007f9a2e6c8cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (uvloop/loop.c:31331:13) #32 0x00007f9a7b065c25 PyObject_VectorcallMethod #33 0x00007f9a2e6ccd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (uvloop/loop.c:33768:23) #34 0x00007f9a2e6ce591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (uvloop/loop.c:33318:13) #35 0x00007f9a7b039358 PyObject_Vectorcall ``` ## Fix The `_after_fork` boolean field marks that this thread object is in a "post-fork zombie state." When the flag is set to true, Thread methods (e.g. `join`) become no-ops because the threads do not exist anymore so we should not try to do something with them. By checking that same flag, we can tell that we are trying to start a Thread that doesn't really exist and so we shouldn't try to do it. Co-authored-by: Thomas Kowalski <[email protected]>

## Description https://datadoghq.atlassian.net/browse/PROF-13112 This is an attempt to address the following crash. There seems to be a case (that I wasn't able to reproduce in a Docker image, but maybe my "code environment" didn't match the customer's exactly) where using `uvloop` results in a crash caused by `PeriodicThread_start` after `uvloop` tries to restart Threads after a fork. ``` #0 0x00007f9a7acdbefa cfree DataDog#1 0x00007f9a7accc6b5 pthread_create DataDog#2 0x00007f9a7a63aaa5 std::thread::_M_start_thread DataDog#3 0x00007f9a7a639d18 PeriodicThread_start DataDog#4 0x00007f9a2e71d565 __pyx_f_6uvloop_4loop_9UVProcess__after_fork (uvloop/loop.c:120214:3) DataDog#5 0x00007f9a2e6369a8 __pyx_f_6uvloop_4loop___get_fork_handler (uvloop/loop.c:163075:24) DataDog#6 0x00007f9a7ad17073 __fork DataDog#7 0x00007f9a2e732d62 uv__spawn_and_init_child_fork (src/unix/process.c:831:10) DataDog#8 0x00007f9a2e732d62 uv__spawn_and_init_child (src/unix/process.c:919:9) DataDog#9 0x00007f9a2e732d62 uv_spawn (src/unix/process.c:1013:18) DataDog#10 0x00007f9a2e71fb87 __pyx_f_6uvloop_4loop_9UVProcess__init (uvloop/loop.c:119056:19) DataDog#11 0x00007f9a2e711bf7 __pyx_f_6uvloop_4loop_18UVProcessTransport_new (uvloop/loop.c:126866:16) DataDog#12 0x00007f9a2e712aa7 __pyx_gb_6uvloop_4loop_4Loop_116generator16 (uvloop/loop.c:54030:28) DataDog#13 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) DataDog#14 0x00007f9a2e699f8a __Pyx_Coroutine_AmSend (uvloop/loop.c:196492:18) DataDog#15 0x00007f9a2e69a052 __Pyx_Coroutine_Yield_From_Coroutine (uvloop/loop.c:197380:14) DataDog#16 0x00007f9a2e69b0e5 __Pyx_Coroutine_Yield_From (uvloop/loop.c:197408:16) DataDog#17 0x00007f9a2e69b0e5 __pyx_gb_6uvloop_4loop_4Loop_122generator18 (uvloop/loop.c:55002:15) DataDog#18 0x00007f9a2e631419 __Pyx_Coroutine_SendEx (uvloop/loop.c:196315:14) DataDog#19 0x00007f9a2e69bb86 __Pyx_Generator_Next (uvloop/loop.c:196581:18) DataDog#20 0x00007f9a2e6398eb __Pyx_PyObject_Call (uvloop/loop.c:191431:15) DataDog#21 0x00007f9a2e6398eb __Pyx_PyObject_FastCallDict (uvloop/loop.c:191552:16) DataDog#22 0x00007f9a2e715a69 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66873:27) DataDog#23 0x00007f9a2e71996b __pyx_f_6uvloop_4loop_4Loop__on_idle (uvloop/loop.c:17975:25) DataDog#24 0x00007f9a2e713e52 __pyx_f_6uvloop_4loop_6Handle__run (uvloop/loop.c:66927:24) DataDog#25 0x00007f9a2e715c88 __pyx_f_6uvloop_4loop_cb_idle_callback (uvloop/loop.c:87335:19) DataDog#26 0x00007f9a2e731311 uv__run_idle (unix/loop-watcher.c:68:1) DataDog#27 0x00007f9a2e72e647 uv_run (src/unix/core.c:439:5) DataDog#28 0x00007f9a2e64fdb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (uvloop/loop.c:18458:23) DataDog#29 0x00007f9a2e6b7e50 __pyx_f_6uvloop_4loop_4Loop__run (uvloop/loop.c:18876:18) DataDog#30 0x00007f9a2e6c8cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (uvloop/loop.c:31528:18) DataDog#31 0x00007f9a2e6c8cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (uvloop/loop.c:31331:13) DataDog#32 0x00007f9a7b065c25 PyObject_VectorcallMethod DataDog#33 0x00007f9a2e6ccd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (uvloop/loop.c:33768:23) DataDog#34 0x00007f9a2e6ce591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (uvloop/loop.c:33318:13) DataDog#35 0x00007f9a7b039358 PyObject_Vectorcall ``` ## Fix The `_after_fork` boolean field marks that this thread object is in a "post-fork zombie state." When the flag is set to true, Thread methods (e.g. `join`) become no-ops because the threads do not exist anymore so we should not try to do something with them. By checking that same flag, we can tell that we are trying to start a Thread that doesn't really exist and so we shouldn't try to do it.

Configure CircleCI and Python packaging for auto-deploy

ef58626

Make CircleCI build the wheel (branch develop, master, tag vA.B.C) and publish it to our S3 repo

talwai reviewed Jul 5, 2016
View reviewed changes

ddtrace/__init__.py Outdated

Copy link

Contributor

talwai Jul 5, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo datadaog

Use mkwheelhouse to push built wheels to S3

4faf381

LotharSee force-pushed the benjamin/auto-deploy branch from 864a3ba to 4faf381 Compare July 5, 2016 12:36

LotharSee merged commit 62ccd2d into master Jul 5, 2016

LotharSee deleted the benjamin/auto-deploy branch July 5, 2016 14:33

labbati added a commit that referenced this pull request Sep 13, 2018

[tmp] #11

8f1b1cb

brettlangdon added a commit that referenced this pull request Feb 28, 2025

Revert "ci(serverless): force reimport of modules during serverless t…

d86174b

…est (#11…" This reverts commit 8c4d62d.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Automatic build and push to repo #11

Automatic build and push to repo #11

Uh oh!

LotharSee commented Jul 4, 2016 •

edited

Loading

Uh oh!

talwai Jul 5, 2016

Uh oh!

talwai commented Jul 5, 2016

Uh oh!

LeoCavaille commented Jul 5, 2016

Uh oh!

awang commented Jul 5, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Automatic build and push to repo #11

Automatic build and push to repo #11

Uh oh!

Conversation

LotharSee commented Jul 4, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

"Dev" versions

"Public" versions

Uh oh!

talwai Jul 5, 2016

Choose a reason for hiding this comment

Uh oh!

talwai commented Jul 5, 2016

Uh oh!

LeoCavaille commented Jul 5, 2016

Uh oh!

awang commented Jul 5, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

LotharSee commented Jul 4, 2016 •

edited

Loading