This repository contains example code for getting started with EMR Serverless and using it with Apache Spark and Apache Hive.
In addition, it provides Container Images for both the Spark History Server and Tez UI in order to debug your jobs.
For full details about using EMR Serverless, please see the EMR Serverless documentation.
These demos assume you are using an Administrator-level role in your AWS account
-
Amazon EMR Serverless is now Generally Available! Check out the console to Get Started with EMR Serverless.
-
Create an Amazon S3 bucket in region where you want to use EMR Serverless (we'll assume
us-east-1).
aws s3 mb s3://BUCKET-NAME --region us-east-1- Create an EMR Serverless execution role (replacing
BUCKET-NAMEwith the one you created above)
This role provides both S3 access for specific buckets as well as read and write access to the Glue Data Catalog.
aws iam create-role --role-name emr-serverless-job-role --assume-role-policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "emr-serverless.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}'
aws iam put-role-policy --role-name emr-serverless-job-role --policy-name S3Access --policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ReadFromOutputAndInputBuckets",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::noaa-gsod-pds",
"arn:aws:s3:::noaa-gsod-pds/*",
"arn:aws:s3:::BUCKET-NAME",
"arn:aws:s3:::BUCKET-NAME/*"
]
},
{
"Sid": "WriteToOutputDataBucket",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::BUCKET-NAME/*"
]
}
]
}'
aws iam put-role-policy --role-name emr-serverless-job-role --policy-name GlueAccess --policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GlueCreateAndReadDataCatalog",
"Effect": "Allow",
"Action": [
"glue:GetDatabase",
"glue:GetDataBases",
"glue:CreateTable",
"glue:GetTable",
"glue:GetTables",
"glue:GetPartition",
"glue:GetPartitions",
"glue:CreatePartition",
"glue:BatchCreatePartition",
"glue:GetUserDefinedFunctions"
],
"Resource": ["*"]
}
]
}'Now you're ready to go! Check out the examples below.
-
Sample templates for creating an EMR Serverless application as well as various dependencies.
-
Template for creating a CloudWatch Dashboard for monitoring your EMR Serverless application.
-
Examples of building EMR Serverless environments with Amazon CDK.
-
Sample DAGs and preview version of the Airflow Operator. Check the releases page for updates.
-
This sample script shows how to use EMR Serverless to run a PySpark job that analyzes data from the open NOAA Global Surface Summary of Day dataset.
-
Shows how to package Python dependencies (Great Expectations) using a Virtualenv and
venv-pack. -
Shows how to use a different Python version than the default (3.7.10) provided by EMR Serverless.
-
This sample shows how to use EMR Serverless to combine both Python and Java dependencies in order to run genomic analysis using Glow and 1000 Genomes.
-
This sample script shows how to use Hive in EMR Serverless to query the same NOAA data.
You can call EMR Serverless APIs using standard AWS SDKs. The examples below show how to do this.
- EMR Serverless Estimator - Estimate the cost of running Spark jobs on EMR Serverless based on Spark event logs.
The following UIs are available in the EMR Serverless console, but you can still use them locally if you wish.
-
Spark UI- Use this Dockerfile to run Spark history server in a container.
-
Tez UI- Use this Dockerfile to run Tez UI and Application Timeline Server in a container.
See CONTRIBUTING for more information.
This library is licensed under the MIT-0 License. See the LICENSE file.