Skip to content
This repository has been archived by the owner on Sep 18, 2024. It is now read-only.

Add timeout for web_channel in trial_runner #2710

Merged
merged 19 commits into from
Jul 24, 2020
Merged

Add timeout for web_channel in trial_runner #2710

Changes from 18 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
dcd2ffd
Merge pull request #251 from microsoft/master
SparkSnail May 29, 2020
3b8b6fb
Merge pull request #252 from microsoft/master
SparkSnail Jun 7, 2020
916e444
Merge pull request #253 from microsoft/master
SparkSnail Jun 15, 2020
caeffb8
Merge pull request #254 from microsoft/master
SparkSnail Jun 17, 2020
57c300e
Merge pull request #255 from microsoft/master
SparkSnail Jun 28, 2020
65660e6
Merge pull request #257 from microsoft/master
SparkSnail Jun 30, 2020
9376d6a
Merge pull request #258 from microsoft/master
SparkSnail Jul 1, 2020
5fef3cf
Merge pull request #259 from microsoft/master
SparkSnail Jul 3, 2020
5544ae8
Merge pull request #261 from microsoft/master
SparkSnail Jul 10, 2020
c5e26ef
add trial job detail link
SparkSnail Jul 19, 2020
785a324
Revert "add trial job detail link"
SparkSnail Jul 19, 2020
0f0567c
fix webchannel connection
SparkSnail Jul 19, 2020
bcab8a9
refactor logic
SparkSnail Jul 20, 2020
15ea95a
add timeout
SparkSnail Jul 21, 2020
a3b9d39
format
SparkSnail Jul 21, 2020
4a8dbcb
reset timeout value
SparkSnail Jul 21, 2020
6b9dd71
fix comments
SparkSnail Jul 23, 2020
ab8e130
unify code to use os._exit(1)
SparkSnail Jul 23, 2020
b8fbc9d
fix typo
SparkSnail Jul 24, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 11 additions & 7 deletions tools/nni_trial_tool/web_channel.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# Licensed under the MIT license.

import asyncio

import os
import websockets

from .base_channel import BaseChannel
Expand All @@ -16,19 +16,23 @@ def __init__(self, args):
self.args = args
self.client = None
self.in_cache = b""
self.timeout = 10

super(WebChannel, self).__init__(args)

self._event_loop = None

def _inner_open(self):
url = "ws://{}:{}".format(self.args.nnimanager_ip, self.args.nnimanager_port)
nni_log(LogType.Info, 'WebChannel: connected with info %s' % url)

connect = websockets.connect(url)
self._event_loop = asyncio.get_event_loop()
client = self._event_loop.run_until_complete(connect)
self.client = client
try:
connect = asyncio.wait_for(websockets.connect(url), self.timeout)
self._event_loop = asyncio.get_event_loop()
client = self._event_loop.run_until_complete(connect)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to hang at this line in some cases? Is a timeout needed for event loop?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This self._event_loop.run_until_complete(connect) is for connect method, and connect has already handled timeout.

self.client = client
nni_log(LogType.Info, 'WebChannel: connected with info %s' % url)
except asyncio.TimeoutError:
nni_log(LogType.Error, 'connect to %s timeout! Please make sure NNIManagerIP configured correclty, and accessable.' % url)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: correclty

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

os._exit(1)

def _inner_close(self):
if self.client is not None:
Expand Down