-
Notifications
You must be signed in to change notification settings - Fork 18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add timout to ssh #467
Add timout to ssh #467
Conversation
Codecov Report
@@ Coverage Diff @@
## master #467 +/- ##
=======================================
Coverage 93.46% 93.47%
=======================================
Files 99 99
Lines 8385 8393 +8
=======================================
+ Hits 7837 7845 +8
Misses 548 548
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any idea why the connection to AWS just hangs some times? If it hangs here, maybe it will for other subsequent ssh requests we send to?
ramp-engine/ramp_engine/aws/api.py
Outdated
logger.error('Unhandled exception happend when checking if there' | ||
f' is an instance action: {e}') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
logger.error('Unhandled exception happend when checking if there' | |
f' is an instance action: {e}') | |
logger.error('Unhandled exception occurred when checking for ' | |
f'instance action: {e}') |
thanks for your review @lucyleeow Yes. I spent quite some time investigating the whole hanging issue. It seems that curl was hanging for now, but there is always a possibility it will hang elsewhere as well. What other places would you suggest we need to take care of before making release? |
Thanks @lucyleeow |
Adds a timeout to curl and handles the exceptions in case there are any errors with subprocess output (before the connection just hanged sometimes stopping the work of the whole dispatcher).
The problem which remains and I don't know how to approach is that even if the connection is not established after (what is set now) 9 tries, with initial time 1sec doubled after each next try (ie max waiting time 8.51 min) the only thing we are able to return is either that the instance finished or not (ie True or False). If for any reason the instance is lost (shutting by hand, other scenarios (??) ) it will still try connecting to it the next time again.
This might be a problem for another PR though..