[Jobs] Allow logs for finished jobs and add sky jobs logs --refresh for restartin jobs controller
#4380
+151
−50
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Michaelvll
requested review from
cblmemo and
cg505
and removed request for
cblmemo
Michaelvll
changed the title
sky jobs logs --refresh for restartin jobs controller
cblmemo
reviewed
Nov 18, 2024
There was a problem hiding this comment.
Thanks for adding this @Michaelvll ! Left some discussions.
| handle: cloud_vm_ray_backend.CloudVmRayResourceHandle) -> None: | ||
| self, task_id: Optional[int], | ||
| handle: Optional[cloud_vm_ray_backend.CloudVmRayResourceHandle] | ||
| ) -> None: | ||
| """Downloads and streams the logs of the latest job. |
There was a problem hiding this comment.
Suggested change
| """Downloads and streams the logs of the latest job. | |
| """Downloads and streams the logs of the given task id. |
is this correct?
| refresh=False, | ||
| include_controller=False) | ||
| if clusters: | ||
| handle = clusters[0].get('handle') |
There was a problem hiding this comment.
Suggested change
| handle = clusters[0].get('handle') | |
| assert len(clusters) == 1 | |
| handle = clusters[0].get('handle') |
| @@ -382,15 +397,25 @@ def tail_logs(name: Optional[str], job_id: Optional[int], follow: bool, | |||
| sky.exceptions.ClusterNotUpError: the jobs controller is not up. | |||
| """ | |||
| # TODO(zhwu): Automatically restart the jobs controller | |||
| if name is not None and job_id is not None: | |||
| raise ValueError('Cannot specify both name and job_id.') | |||
| def set_local_log_file(job_id: int, task_id: Optional[int], | ||
| local_log_file: str): | ||
| """Set the local log file for a job.""" | ||
| task_str = '' if task_id is None else f' AND task_id={task_id}' |
There was a problem hiding this comment.
Not sure do we need to pass in task_id as ?.. Actually, is there any difference?
There was a problem hiding this comment.
IMO we should just use ? everywhere to avoid having to reason about where injection may be possible.
cg505
reviewed
Nov 22, 2024
| def set_local_log_file(job_id: int, task_id: Optional[int], | ||
| local_log_file: str): | ||
| """Set the local log file for a job.""" | ||
| task_str = '' if task_id is None else f' AND task_id={task_id}' |
There was a problem hiding this comment.
IMO we should just use ? everywhere to avoid having to reason about where injection may be possible.
|
|
||
| def get_local_log_file(job_id: int, task_id: Optional[int]) -> Optional[str]: | ||
| """Get the local log directory for a job.""" | ||
| task_str = '' if task_id is None else f' AND task_id={task_id}' |
There was a problem hiding this comment.
Comment on lines
+387
to
+394
| # Stream the logs to the console without reading the whole | ||
| # file into memory. | ||
| start_streaming = False | ||
| for line in f: | ||
| if log_lib.LOG_FILE_START_STREAMING_AT in line: | ||
| start_streaming = True | ||
| if start_streaming: | ||
| print(line, end='', flush=True) |
There was a problem hiding this comment.
Should we move this into some utils file?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently,
sky jobs logsdoes not work with finished jobs, although we do have the logs downloaded for failed jobs to the controller. We now enablesky jobs logsfor all finished jobs, including succeeded ones.Tested (run the relevant ones):
bash format.shpytest tests/test_smoke.pypytest tests/test_smoke.py::test_fill_in_the_nameconda deactivate; bash -i tests/backward_compatibility_tests.sh