dask-cloudprovider doesnt pass first step of scheduler creation

[msalbego93]

Describe the issue:
When instantiating a new cluster using EC2Cluster the process don't pass the first step for creating the scheduler, the EC2 instance passes the checks correctly but no worker is ec2 is created after that. This behavior is checked even with no security group set (using dask-default).

The script is set forever to the following message:

Creating scheduler instance
Created instance i-0a1a624XXXXX01b76a as dask-21076af2-scheduler
Waiting for scheduler to run at XX.XXX.XX.XX:8786

Minimal Complete Verifiable Example:

    cluster = EC2Cluster(instance_type="m6i.large",
                            region='us-east-1',
                            security_groups = ['sg-092XXXX59b56f1b'],
                            n_workers=5 ,
                            env_vars=get_aws_credentials(), 
                            security = False
                         )

Anything else we need to know?:

Environment:

• Dask version:
• dask==2023.9.1
  dask-cloudprovider==2022.10.0
• Python version: 3.10
• Operating System: Ubuntu
• Install method (conda, pip, source): pipenv install

[jacobtomlinson]

Here are some steps to troubleshoot:

• SSH to the scheduler instance and checkout the /var/log/cloud-init-output.log file to ensure the scheduler has started correctly.
• Ensure you have network connectivity from your machine to the scheduler IP (theobfuscated XX.XXX.XX.XX:8786 in your example). The EC2Cluster class will need to connect to the scheduler.