Cloudstomp - Automated deployment of instances on EC2 via plugin for high CPU/GPU applications at the lowest price
Cloudstomp is a python application that is in active development and built to make deployment of high CPU/GPU applications in Amazon EC2 as spot instances for affordable, quick, and easy data analysis.
Plugins can be created in the json format that enable users to quickly deploy applications. Two plugins are included for demonstration, one for a hello world application, and one for hashcat.
Installation can be completed with Docker or natively. For a native local installation, start by downloading cloudstomp:
# cd /usr/bin
# wget https://raw.githubusercontent.com/Fmstrat/cloudstomp/master/cloudstomp.py
# chmod +x cloudstomp.py
or, if you would like to use Docker:
# alias cloudstomp.py='docker run -it --rm --name=cloudstomp -v ${HOME}/.cloudstomp:/data cloudstomp /cloudstomp.py'
The current documentation for plugins and avaialable plugins can be found at https://github.com/Fmstrat/cloudstomp-plugins and are:
- Hello World - The example shown below
- Hashcat - Password cracking in the cloud
To use Cloudstomp you must provide AWS credentials. To set up your credentials at the AWS Management Console:
- Sign in at https://console.aws.amazon.com/iam/
- Go to
Users - Then
Add User - Add the user
cloudstomp - Check the box for
Programmatic access - Click
Next: Permissions - If the
Administratorsgroup doesn't exist, clickAdd group - Enter
AdministratorsforGroup name - Check
Administrator Access - Click
Create groupwhich will return you to the add user screen - Back on the add user screen, choose
Add user to groupand checkAdministrators - Click
Next: Tags - Click
Next: Review - Click
Create user Access key IDandSecret access keyare displayed which are required for configuration ofCloudstomp
First, we need to download the latest plugins:
# cloudstomp.py --update
This will download all of the preconfigured plugins into the ~/cloudstomp folder (or the folder specified in the Docker command).
Then, we fire up cloudstomp:
# cloudstomp.py
The first time Cloudstomp runs, it requires the AWS credentials. This sets up a credentials file completely seperate from those used in your normal AWS administration to reduce conflicts.
AWS CLI is not configured, would you like to do this now? [y/N]: y Access key, Secret key, and default region must all be configured. AWS Access Key ID [None]: ********* AWS Secret Access Key [None]: ********* Default region name [None]: us-east-1 Default output format [None]:
Then, follow the prompts:
Sessions: ------------------------------------------- c) Create q) Quit ------------------------------------------- Select session: c Creating a new session. Plugins: ------------------------------------------- 1) Hashcat 2) Hello World ------------------------------------------- Please select a plugin: 2
The first time you choose a plugin, the AMI that instances will be created from needs to be built. This is a one-time item and the AMI is stored on S3 as they always are for faster future deployment.
The AMI for hello world needs to be built. Would you like to build it now? [Y/n]: y [2019-01-18 10:57:47] Build will take place on a t2.nano instance. [2019-01-18 10:57:47] Checking availability zones for t2.nano... [2019-01-18 10:57:48] Getting subnet... [2019-01-18 10:57:49] Using subnet subnet-0bea6a005f7f8dbed. [2019-01-18 10:57:49] Checking VPC... [2019-01-18 10:57:50] Using VPC vpc-02d8172ca01a496de. [2019-01-18 10:57:50] Checking security group... [2019-01-18 10:57:50] Creating new security group. [2019-01-18 10:57:51] Adding rules... [2019-01-18 10:57:52] Added port 22. [2019-01-18 10:57:52] Using security group sg-019fedf5084f3e068. [2019-01-18 10:57:54] Instance i-06f82a99547d8d66e started. [2019-01-18 10:57:54] Waiting for boot to complete... [2019-01-18 10:57:54] Instance in pending state, sleeping 15s... [2019-01-18 10:58:10] Instance has public IP of 18.212.124.241. [2019-01-18 10:58:10] Checking for SSH connectivity... [2019-01-18 10:58:13] Waiting for SSH to start, sleeping 15s... [2019-01-18 10:58:28] Waiting for SSH to start, sleeping 15s... [2019-01-18 10:58:46] Starting build scripts... [2019-01-18 10:58:46] Running build.sh... [2019-01-18 10:58:48] Uploaded. Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB] Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] ... [ Output of setting up the instance (apt-get/etc) ] ... Successfully built PyYAML Installing collected packages: docutils, jmespath, six, python-dateutil, urllib3, botocore, PyYAML, pyasn1, rsa, colorama, futures, s3transfer, awscli Successfully installed PyYAML-3.13 awscli-1.16.91 botocore-1.12.81 colorama-0.3.9 docutils-0.14 futures-3.2.0 jmespath-0.9.3 pyasn1-0.4.5 python-dateutil-2.7.5 rsa-3.4.2 s3transfer-0.1.13 six-1.12.0 urllib3-1.24.1 [2019-01-18 10:59:59] Stopping instance... [2019-01-18 11:00:00] Instance in stopping state, sleeping 15s... [2019-01-18 11:00:16] Instance in stopping state, sleeping 15s... [2019-01-18 11:00:37] Creating AMI image... [2019-01-18 11:00:38] Build is in pending state, sleeping 15s... [2019-01-18 11:00:54] Build is in pending state, sleeping 15s... [2019-01-18 11:01:09] Build is in pending state, sleeping 15s... [2019-01-18 11:01:25] Build is in pending state, sleeping 15s... [2019-01-18 11:01:41] Build is in pending state, sleeping 15s... [2019-01-18 11:01:57] Build is in pending state, sleeping 15s... [2019-01-18 11:02:13] Build is in pending state, sleeping 15s... [2019-01-18 11:02:33] Tagging with build version... [2019-01-18 11:02:34] Terminating source image... [2019-01-18 11:02:35] Build complete. Returning to session creation.
Once the build completes, we return to creating the new session. You will notice one of the questions below is about encryption. Output data is stored on S3 for later download when processes finish. This data can be encrypted with gocryptfs, that way if it is stored for a long period of time it is more secure. For the Hello World plugin, data is syncronized to S3 every 15 seconds. For other plugins, the reocmmendation is every 300 seconds.
Would you like to run the instance On-Demand or as a Spot Instance? [o/S]: s Instances: ------------------------------------------- 1) 1 vCPU, 1GB RAM [t2.micro] 2) 1 vCPU, 2GB RAM [t2.small] ------------------------------------------- Please select an instance: 1 Pricing: -------------- ---------- ---------- ---------- ------------ Zone Average Latest On Demand Reccomended -------------- ---------- ---------- ---------- ------------ 1) us-east-1c $0.003500 $0.003500 $0.011600 $0.004025 (best) 2) us-east-1f $0.003500 $0.003500 $0.011600 $0.004025 3) us-east-1b $0.003500 $0.003500 $0.011600 $0.004025 4) us-east-1a $0.003500 $0.003500 $0.011600 $0.004025 5) us-east-1e $0.003500 $0.003500 $0.011600 $0.004025 6) us-east-1d $0.003500 $0.003500 $0.011600 $0.004025 -------------- ---------- ---------- ---------- ------------ (The default is 15% over the average past two week spot price to keep your instance active.) Which zone would you like to run in? 1 What is the max price you would pay for a spot instance? [0.004025]: What would you like to call this instance? Testing How many seconds would you like to wait between printing 'hello world'? [10]: 10 Enter a password for gocryptfs encrypted output (leave blank for no encryption): Session Testing created. Sessions: ------------------------------------------- 1) Testing c) Create q) Quit ------------------------------------------- Select session:
Now you have the session Testing available. Next, we will start up an instance running the Hello World plugin.
Sessions: ------------------------------------------- 1) Testing c) Create q) Quit ------------------------------------------- Select session: 1 Testing: ------------------------------------------- 1) Start 2) Output s) Show variables d) Delete b) Back ------------------------------------------- Choose an option: 1 [2019-01-18 11:06:27] Getting subnet... [2019-01-18 11:06:28] Using subnet subnet-0bea6a005f7f8dbed. [2019-01-18 11:06:28] Checking VPC... [2019-01-18 11:06:29] Using VPC vpc-02d8172ca01a496de. [2019-01-18 11:06:29] Checking security group... [2019-01-18 11:06:30] Found existing security group. [2019-01-18 11:06:30] Using security group sg-019fedf5084f3e068. [2019-01-18 11:06:31] Waiting on spot id sir-9jcg74mg... [2019-01-18 11:06:31] Spot request in open state, sleeping 15s... [2019-01-18 11:07:03] Instance in running state. [2019-01-18 11:07:04] Instance has public IP of 54.88.86.34. [2019-01-18 11:07:04] Checking for SSH connectivity... [2019-01-18 11:07:07] Waiting for SSH to start, sleeping 15s... [2019-01-18 11:07:25] Uploading cloudstomp AWS credentials... [2019-01-18 11:07:26] Uploading any supporting files... [2019-01-18 11:07:26] Complete. [2019-01-18 11:07:26] Starting init script... [2019-01-18 11:07:26] Running remote.sh... [2019-01-18 11:07:27] Uploaded. [2019-01-18 11:07:33] Launch complete. Testing: ------------------------------------------- 1) Status 2) Connect 3) Stop 4) Output s) Show variables d) Delete b) Back ------------------------------------------- Choose an option:
Now we can check on the status of our instance, connect to it and attach to the screen session, stop the instance, or show the outputs from S3. Outputs are available even after the instance is stopped.
Testing: ------------------------------------------- 1) Status 2) Connect 3) Stop 4) Output s) Show variables d) Delete b) Back ------------------------------------------- Choose an option: 1 [2019-01-18 11:09:59] Instance has public IP of 54.88.86.34. [2019-01-18 11:09:59] Checking for SSH connectivity... Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Testing: ------------------------------------------- 1) Status 2) Connect 3) Stop 4) Output s) Show variables d) Delete b) Back ------------------------------------------- Choose an option: 3 [2019-01-18 11:10:28] Instance has public IP of 54.88.86.34. [2019-01-18 11:10:28] Checking for SSH connectivity... There is no screen to be resumed matching helloworld. There is no screen to be resumed matching sync. upload: cloudstomp/plain/output.txt to s3://cloudstomp-testing/output.txt upload: cloudstomp/plain/sync.txt to s3://cloudstomp-testing/sync.txt Connection to 54.88.86.34 closed. Sessions: ------------------------------------------- 1) Testing [shutting-down] c) Create q) Quit ------------------------------------------- Select session: 1 Testing: ------------------------------------------- 1) Output s) Show variables d) Delete b) Back ------------------------------------------- Choose an option: 1 download: s3://cloudstomp-testing/output.txt to ../../../tmp/cloudstomp-testing/plain/output.txt download: s3://cloudstomp-testing/sync.txt to ../../../tmp/cloudstomp-testing/plain/sync.txt Starting hello world... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... Hello world! Sleeping 10 seconds... ^C ------------------------------------------- Testing: ------------------------------------------- 1) Output s) Show variables d) Delete b) Back ------------------------------------------- Choose an option:
Many of these options can also be executed from the command line (check with cloudstomp.py --help) for automation.