Local output path does not work in script mode with s3 code location

[marcelgwerder]

Describe the bug
It seems like when setting a code location the output bucket is used to derive the kms key if the code and output buckets match.

sagemaker-python-sdk/src/sagemaker/estimator.py

Line 1732 in c58bd0f

output_bucket, _ = parse_s3_url(self.output_path)

The use of the parse_s3_url however makes it impossible to use a local output path in local mode because it expects a valid s3 url.

sagemaker-python-sdk/src/sagemaker/s3.py

Line 33 in c58bd0f

raise ValueError("Expecting 's3' scheme, got: {} in {}.".format(parsed_url.scheme, url))

To reproduce
When using the sagemaker.estimator.Framework class or any class extending it, use local mode with local output path and specify a code location on s3.

Expected behavior
Everything works just like when using the estimator class. I should be able to use a local output path even when setting an s3 code location.

Screenshots or logs

~/anaconda3/envs/sm/lib/python3.6/site-packages/sagemaker/estimator.py in fit(self, inputs, wait, logs, job_name, experiment_config)
    489 
    490         """
--> 491         self._prepare_for_training(job_name=job_name)
    492 
    493         self.latest_training_job = _TrainingJob.start_new(self, inputs, experiment_config)

~/anaconda3/envs/sm/lib/python3.6/site-packages/sagemaker/estimator.py in _prepare_for_training(self, job_name)
   1685             self.code_uri = self.uploaded_code.s3_prefix
   1686         else:
-> 1687             self.uploaded_code = self._stage_user_code_in_s3()
   1688             code_dir = self.uploaded_code.s3_prefix
   1689             script = self.uploaded_code.script_name

~/anaconda3/envs/sm/lib/python3.6/site-packages/sagemaker/estimator.py in _stage_user_code_in_s3(self)
   1730             code_s3_prefix = "/".join(filter(None, [key_prefix, self._current_job_name, "source"]))
   1731 
-> 1732             output_bucket, _ = parse_s3_url(self.output_path)
   1733             kms_key = self.output_kms_key if code_bucket == output_bucket else None
   1734 

~/anaconda3/envs/sm/lib/python3.6/site-packages/sagemaker/fw_utils.py in parse_s3_url(url)
    549         str: S3 bucket name str: S3 key
    550     """
--> 551     return s3.parse_s3_url(url)
    552 
    553 

~/anaconda3/envs/sm/lib/python3.6/site-packages/sagemaker/s3.py in parse_s3_url(url)
     31     parsed_url = urlparse(url)
     32     if parsed_url.scheme != "s3":
---> 33         raise ValueError("Expecting 's3' scheme, got: {} in {}.".format(parsed_url.scheme, url))
     34     return parsed_url.netloc, parsed_url.path.lstrip("/")
     35 

ValueError: Expecting 's3' scheme, got: file in file:///data/07_model_output/wnut17.

System information
A description of your system. Please provide:

• SageMaker Python SDK version: 1.57.0
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): Custom algorithm with Framework class
• Framework version:
• Python version: 3.6.10
• CPU or GPU: CPU
• Custom Docker image (Y/N): Y

[marcelgwerder]

So I found out how to use local code directly without the s3 upload by changing the session config

session = sagemaker.LocalSession()
session.config = {'local': {'local_code': True }}

and using an absolute path for the source_dir argument.

While this is a way to test locally it doesn't resolve the issue that code location and output path should not be hardwired like that and function independently.

[laurenyu]

sorry for the slow response here -

just to make sure I'm understanding how to reproduce this, the issue arises when using Local Mode with a local output_path and a code_location that requires a KMS key?

[marcelgwerder]

Yes, but the code_location doesn't necessarily require a KMS key. The problem is that the code used to derive the key (could end up being None) just assumes that the output path is an s3 location.

In general it would probably make the most sense to use the local code directly when using a local session. There is really not much of a point to upload code to s3 and download it again to a local container, especially if it doesn't seem to play nicely with a local output location. Also, I found this information about the config for local code rather hard to find.

[laurenyu]

Thanks for the clarification. I've opened #1516 to fix the bug and #1517 to document the config for local code

[laurenyu]

both PRs have been released in v1.59.0