Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Worker should kill task handler process and its descendant processes cleanly when timeout #33

Merged
merged 3 commits into from
Sep 29, 2020

Conversation

everpeace
Copy link
Contributor

@everpeace everpeace commented Sep 29, 2020

$ cat test.sh
#! /bin/bash
sleep 1000

$ ./dist/pftaskqueue start-worker --default-command-timeout 10s --queue-name=test --name=test -- bash ./test.sh
9:18PM INF Start processing a task component=worker processUID=e1543f1f-4bed-463f-a4a7-23c24d2223f0 queueName=test queueUID=9d782669-acb2-4846-89ee-1ae859395c08 taskHandlerCommands=["bash","./test.sh"] taskSpec={"name":"task1","payload":"payload"} taskUID=90eb3dc8-9461-4194-ad9a-ad0f3970ff73 workerName=test workerUID=6e1dca1e-00bc-414f-90ee-71e4bcdf5fbd
9:18PM ERR Task handler process timeout error="context deadline exceeded" component=worker postHooks=[] processUID=e1543f1f-4bed-463f-a4a7-23c24d2223f0 queueName=test queueUID=9d782669-acb2-4846-89ee-1ae859395c08 taskHandlerCommands=["bash","./test.sh"] taskResult={"reason":"Timeout","type":"Failure"} taskSpec={"name":"task1","payload":"payload"} taskUID=90eb3dc8-9461-4194-ad9a-ad0f3970ff73 workerName=test workerUID=6e1dca1e-00bc-414f-90ee-71e4bcdf5fbd
9:18PM INF Completed As Failure because retry exhausted component=redis-backend failureCount=1 operation=RecordFailure queueName=test queueUID=9d782669-acb2-4846-89ee-1ae859395c08 retryLimit=0 workerName=test workerUID=6e1dca1e-00bc-414f-90ee-71e4bcdf5fbd
9:18PM INF Task marked Failure component=worker processUID=e1543f1f-4bed-463f-a4a7-23c24d2223f0 queueName=test queueUID=9d782669-acb2-4846-89ee-1ae859395c08 taskResult={"reason":"Timeout","type":"Failure"} taskSpec={"name":"task1","payload":"payload"} taskUID=90eb3dc8-9461-4194-ad9a-ad0f3970ff73 workerName=test workerUID=6e1dca1e-00bc-414f-90ee-71e4bcdf5fbd
9:18PM INF Queue is empty. retrying in 5 seconds. component=worker detectedQueueEmptyAt=2020-09-29T21:18:47+09:00 exitOnEmpty=false exitOnEmptyGracePeriod=10000 queueName=test queueUID=9d782669-acb2-4846-89ee-1ae859395c08 workerName=test workerUID=6e1dca1e-00bc-414f-90ee-71e4bcdf5fbd
...

# in another shell
# no subprocesses remains!!
$ ps -efj | egrep "[s]leep 1000"

@everpeace everpeace changed the title kill task handler process and its descendant processes cleanly Worker should kill task handler process and its descendant processes cleanly when timeout Sep 29, 2020
@everpeace everpeace added the release-note/bugfix Bugfix category in release note label Sep 29, 2020
@everpeace everpeace added release-note/breaking-change Braking Change category in release note and removed release-note/breaking-change Braking Change category in release note labels Sep 29, 2020
@everpeace everpeace merged commit d0ecdb4 into master Sep 29, 2020
@everpeace everpeace deleted the kill-task-subprocesses-cleanly branch September 29, 2020 13:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-note/breaking-change Braking Change category in release note release-note/bugfix Bugfix category in release note
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Worker doesn't kill taskHandler's subprocesses when timeout
1 participant