Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fixed broken link in spark observability module #214

Merged
merged 2 commits into from
May 30, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion analytics/terraform/emr-eks-karpenter/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ Checkout the [documentation website](https://awslabs.github.io/data-on-eks/docs/
| <a name="input_name"></a> [name](#input\_name) | Name of the VPC and EKS Cluster | `string` | `"emr-eks-karpenter"` | no |
| <a name="input_private_subnets"></a> [private\_subnets](#input\_private\_subnets) | Private Subnets CIDRs. 254 IPs per Subnet/AZ for Private NAT + NLB + Airflow + EC2 Jumphost etc. | `list(string)` | <pre>[<br> "10.1.1.0/24",<br> "10.1.2.0/24"<br>]</pre> | no |
| <a name="input_public_subnets"></a> [public\_subnets](#input\_public\_subnets) | Public Subnets CIDRs. 62 IPs per Subnet/AZ | `list(string)` | <pre>[<br> "10.1.0.0/26",<br> "10.1.0.64/26"<br>]</pre> | no |
| <a name="input_region"></a> [region](#input\_region) | region | `string` | n/a | yes |
| <a name="input_region"></a> [region](#input\_region) | Region | `string` | n/a | yes |
| <a name="input_secondary_cidr_blocks"></a> [secondary\_cidr\_blocks](#input\_secondary\_cidr\_blocks) | Secondary CIDR blocks to be attached to VPC | `list(string)` | <pre>[<br> "100.64.0.0/16"<br>]</pre> | no |
| <a name="input_tags"></a> [tags](#input\_tags) | Default tags | `map(string)` | `{}` | no |
| <a name="input_vpc_cidr"></a> [vpc\_cidr](#input\_vpc\_cidr) | VPC CIDR. This should be a valid private (RFC 1918) CIDR range | `string` | `"10.1.0.0/16"` | no |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,13 @@

# Script usage ./taxi-trip-execute my-s3-bucket us-west-2

if [ $# -ne 3 ]; then
# validating that use passes two arguments, if not return a message to pass the arguments
if [ $# -ne 2 ]; then
echo "Usage: $0 <S3_BUCKET> <REGION>"
exit 1
fi


S3_BUCKET="$1"
REGION="$2"

Expand Down
7 changes: 4 additions & 3 deletions website/docs/data-analytics/observability-spark-on-eks.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,13 @@ sidebar_label: Observability Spark on EKS
In this post, we will learn the Observability for Spark on EKS. We will use Spark History Server to watch Spark Applications logs and check the Spark job progress via the Spark Web UI. Amazon Managed Service for Prometheus is used to collect and store the metrics generated by Spark Applications and Grafana is used to build dashboards for monitoring use cases.

## Deploying the Solution
We will reuse the previous Spark on Operator example. Please follow [this link](https://awslabs.github.io/data-on-eks/docs/spark-on-eks/spark-operator-yunikorn#deploying-the-solution) to provision resources
We will reuse the previous Spark on Operator example. Please follow [this link](https://awslabs.github.io/data-on-eks/docs/data-analytics/spark-operator-yunikorn#deploying-the-solution) to provision resources


## Set up data and py script
let's navigate to one example folder under spark-k8s-operator and run the shell script to upload data and py script to the S3 bucket created by terraform above.
```bash
cd data-on-eks/analytics/terraform/spark-k8s-operator/examples/nvme-ephemeral-storage
cd data-on-eks/analytics/terraform/spark-k8s-operator/examples/cluster-autoscaler/nvme-ephemeral-storage

# replace <S3_BUCKET> with your S3 bucket and <REGION> with your region, then run
./taxi-trip-execute.sh
Expand All @@ -37,8 +37,9 @@ To try Spark web UI, let's update <S3_BUCKET> with your bucket name and <JOB_NAM
```

Then run port forward command to expose spark web service.

```bash
kubectl port-forward -n=spark &lt;SPARK_DRIVER_NAME&gt; 4040:4040
kubectl port-forward po/taxi-trip 4040:4040 -nspark-team-a
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

space is missing after -n

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It works anyways, but I'll fix it, thanks Vara

```

Then open browser and enter localhost:4040. You can view your spark application like below.
Expand Down