docker worker does not have aws credentials

[xwjiang2010]

What is the problem?

I was just trying to see if sync to S3 works to bypass rsync issue for our customer.
The same set up as #17940, with the following change:

        sync_config=tune.SyncConfig(
            sync_to_driver=False,
            upload_dir="s3://xwjiang-test/keep_checkpoints_num/",
        ),

And wrap train in tune.durable.
In cluster.yaml's setup cmd, add
pip install awscli

Still the same workflow but ran into the following issue on worker:
Error message (1): fatal error: Unable to locate credentials

On Driver

(base) ray@ip-172-31-24-198:~$ aws configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key     ****************2RSP         iam-role
secret_key     ****************S2Jm         iam-role
    region                <not set>             None    None

On worker

(base) ray@ip-172-31-19-194:~$ aws configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key                <not set>             None    None
secret_key                <not set>             None    None
    region                <not set>             None    None

Is this expected? Am I missing something?

Reproduction (REQUIRED)

Please provide a short code snippet (less than 50 lines if possible) that can be copy-pasted to reproduce the issue. The snippet should have no external library dependencies (i.e., use fake or mock data / environments):

If the code snippet cannot be run by itself, the issue will be closed with "needs-repro-script".

• I have verified my script runs in a clean environment and reproduces the issue.
• I have verified the issue also occurs with the latest wheels.

[xwjiang2010]

@richardliaw

[xwjiang2010]

Per Richard's suggestion, added IamInstanceProfile to cluster.yaml.

worker_nodes:
    InstanceType: m5.xlarge
    ImageId: latest_dlami
    KeyName: xwjiang-test
    IamInstanceProfile:
        Arn: arn:aws:iam::YOUR_AWS_ACCOUNT_NUMBER:YOUR_INSTANCE_PROFILE

which grants S3 access to worker nodes.
More information here.

I will add some documentation to Ray.Tune's SyncConfig section. Keep the bug open to track that.