Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add timout to ssh #467

Merged
merged 6 commits into from
Nov 16, 2020
Merged

Add timout to ssh #467

merged 6 commits into from
Nov 16, 2020

Conversation

maikia
Copy link
Contributor

@maikia maikia commented Nov 16, 2020

Adds a timeout to curl and handles the exceptions in case there are any errors with subprocess output (before the connection just hanged sometimes stopping the work of the whole dispatcher).

The problem which remains and I don't know how to approach is that even if the connection is not established after (what is set now) 9 tries, with initial time 1sec doubled after each next try (ie max waiting time 8.51 min) the only thing we are able to return is either that the instance finished or not (ie True or False). If for any reason the instance is lost (shutting by hand, other scenarios (??) ) it will still try connecting to it the next time again.

This might be a problem for another PR though..

@codecov
Copy link

codecov bot commented Nov 16, 2020

Codecov Report

Merging #467 (ac76a37) into master (4ac7ce0) will increase coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #467   +/-   ##
=======================================
  Coverage   93.46%   93.47%           
=======================================
  Files          99       99           
  Lines        8385     8393    +8     
=======================================
+ Hits         7837     7845    +8     
  Misses        548      548           
Impacted Files Coverage Δ
ramp-engine/ramp_engine/tests/test_aws.py 63.33% <100.00%> (+3.57%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fe29f0c...075d281. Read the comment docs.

Copy link
Contributor

@lucyleeow lucyleeow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any idea why the connection to AWS just hangs some times? If it hangs here, maybe it will for other subsequent ssh requests we send to?

Comment on lines 953 to 954
logger.error('Unhandled exception happend when checking if there'
f' is an instance action: {e}')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
logger.error('Unhandled exception happend when checking if there'
f' is an instance action: {e}')
logger.error('Unhandled exception occurred when checking for '
f'instance action: {e}')

@maikia
Copy link
Contributor Author

maikia commented Nov 16, 2020

thanks for your review @lucyleeow

Yes. I spent quite some time investigating the whole hanging issue. It seems that curl was hanging for now, but there is always a possibility it will hang elsewhere as well.

What other places would you suggest we need to take care of before making release?

@maikia maikia merged commit 65adad5 into paris-saclay-cds:master Nov 16, 2020
@maikia
Copy link
Contributor Author

maikia commented Nov 16, 2020

Thanks @lucyleeow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants