TF 2.2.0 image is not available in eu-west-1

[yegortokmakov]

Describe the bug
TF 2.2.0 image is not available.
Region: eu-west-1

CalledProcessError: Command 'docker pull 763104351884.dkr.ecr.eu-west-1.amazonaws.com/tensorflow-training:2.2.0-cpu-py3' returned non-zero exit status 1.

at the same time docker pull 763104351884.dkr.ecr.eu-west-1.amazonaws.com/tensorflow-training:2.2.0-cpu-py37 image exists.

I haven't tested with other regions

To reproduce

estimator = TensorFlow(base_job_name='local',
                       entry_point='script.py',
                       source_dir=source_dir,
                       role=role,
                       framework_version='2.2.0',
                       py_version='py3',
                       hyperparameters=local_hyperparameters,
                       train_instance_count=1, train_instance_type=instance_type)

System information
A description of your system. Please provide:

• SageMaker Python SDK version: 1.60.0
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): tensorflow
• Framework version: 2.2.0
• Python version: 3.6.6
• CPU or GPU: both
• Custom Docker image (Y/N): N

[laurenyu]

apologies for the confusion - you need to specify "py37" for py_version

[yegortokmakov]

works with py37, thanks!
was trying py37 on the previous version of sdk, obviously it didn't work :)

[stoyan-dixa]

I have a similar problem when trying to use framework version 2.2.0 and I get an error when trying to specify py_version to 'py37':
ValueError: invalid py_version argument: py37

I created a new notebook instance few days ago and the sagemaker sdk version is 1.55.3
Would updating to a newer version solve this issue and what is the recommended way to update?

This is how my estimator definition looks like:

tf_estimator = TensorFlow(
    source_dir='src',
    entry_point='training.py',
    role=role,
    train_instance_count=1,
    train_instance_type='ml.p3.2xlarge',
    framework_version='2.2.0',
    py_version='py37',
    script_mode=True,
    hyperparameters=hyperparameters
)

[laurenyu]

@stoyan-dixa you can update your SDK version by running !pip install -U sagemaker in a cell and then restarting the notebook kernel

[stoyan-dixa]

Thank you for the response! I've updated sagemaker to version 1.67.1 and I can confirm that py_version now accepts py37 and works with framework_version=2.2.0
I was wondering if it is common that new notebook instances have outdated versions of the python packages and therefore it is a good practice to always update them when the notebook instance is created?
This article states that AWS is updating the notebook software once it is not in service. What do these software updates include?

[laurenyu]

@stoyan-dixa my understanding is that Notebook Instances are updated regularly to include updates from the SageMaker Python SDK (in addition to other libraries). Since we release the SDK every Monday-Thursday (if there are changes), this does mean that you may run into an outdated SDK even with a new notebook instance. You could also look into Notebook Instance Lifecycle Configs as a way to update libraries upon startup.

[kafka399]

In my case I had to restart my notebook, as kernel restart didn't fix the issue completely. Thanks!

[martinRenou]

Closing as answered, thanks!