- The
main
branch of this repo is compatible with Rasa Open Source version 3.x - The bot for Rasa 1.x can be found in the rasa-1 branch.
This is an example chatbot demonstrating how to build AI assistants for financial services and banking. This starter pack can be used as a base for your own development or as a reference guide for implementing common banking-industry features with Rasa. It includes pre-built intents, actions, and stories for handling conversation flows like checking spending history and transferring money to another account.
Table of Contents
- Install dependencies
- Run the bot
- Overview of the files
- Things you can ask the bot
- Handoff
- Testing the bot
- Rasa X Deployment
- Action Server Image
- CI/CD
- Summary
- GitHub Secrets
- AWS Preparation
- EKS production cluster
- Appendix A: The AWS EKS cluster
- Appendix B: Manual Cleanup of AWS resources
- Appendix C: OCTANT
- Appendix D: AWS EKS references
Run:
pip install -r requirements.txt
To install development dependencies:
pip install -r requirements-dev.txt
pre-commit install
python -m spacy download en_core_web_md en
python -m spacy link en_core_web_md en
With pre-commit installed, the
black
anddoctoc
hooks will run on everygit commit
. If any changes are made by the hooks, you will need to re-add changed files and re-commit your changes.
Use rasa train
to train a model.
Then, to run, first set up your action server in one terminal window, listening on port 5056:
rasa run actions --port 5056
Note that port 5056 is used for the action server, to avoid a conflict when you also run the helpdesk bot as described below in the handoff
section.
In another window, run the duckling server (for entity extraction):
docker run -p 8000:8000 rasa/duckling
Then to talk to the bot, run:
rasa shell --debug
Note that --debug
mode will produce a lot of output meant to help you understand how the bot is working
under the hood. To simply talk to the bot, you can remove this flag.
You can also try out your bot locally using Rasa X by running
rasa x
Refer to our guided workflow in the Wiki page for how to get started with Rasa X in local mode.
data/nlu/nlu.yml
- contains NLU training data
data/nlu/rules.yml
- contains rules training data
data/stories/stories*.yml
- contains stories training data
actions.py
- contains custom action/api code
domain.yml
- the domain file, including bot response templates
config.yml
- training configurations for the NLU pipeline and policy ensemble
tests/
- end-to-end tests
The bot currently has five skills. You can ask it to:
- Transfer money to another person
- Check your earning or spending history (with a specific vendor or overall)
- Answer a question about transfer charges
- Pay a credit card bill
- Tell you your account balance
It also has a limited ability to switch skills mid-transaction and then return to the transaction at hand.
For the purposes of illustration, the bot recognises the following fictional credit card accounts:
emblem
justice bank
credit all
iron bank
It recognises the following payment amounts (besides actual currency amounts):
minimum balance
current balance
It recognises the following vendors (for spending history):
Starbucks
Amazon
Target
You can change any of these by modifying actions.py
and the corresponding NLU data.
If configured, the bot can also hand off to another bot in response to the user asking for handoff. More details on handoff below.
This bot includes a simple skill for handing off the conversation to another bot or a human. This demo relies on this fork of chatroom to work, however you could implement similar behaviour in another channel and then use that instead. See the chatroom README for more details on channel-side configuration.
Using the default set up, the handoff skill enables this kind of conversation with two bots:
The simplest way to use the handoff feature is to do the following:
- Clone chatroom and Helpdesk-Assistant alongside this repo
- In the chatroom repo, install the dependencies:
yarn install
- In the chatroom repo, build and serve chatroom:
yarn build
yarn serve
- In the Helpdesk-Assistant repo, install the dependencies and train a model (see the Helpdesk-Assistant README)
- In the Helpdesk-Assistant repo, run the rasa server and action server at the default ports (shown here for clarity)
In one terminal window:
In another terminal window:
rasa run --enable-api --cors "*" --port 5005 --debug
rasa run actions --port 5055 --debug
- In the Financial-Demo repo (i.e. this repo), run the rasa server and action server at the non-default ports shown below
In one terminal window:
In another terminal window:
rasa run --enable-api --cors "*" --port 5006 --debug
rasa run actions --port 5056 --debug
- Open
chatroom_handoff.html
in a browser to see handoff in action
Using chatroom, the general approach is as follows:
- User asks original bot for a handoff.
- The original bot handles the request and eventually
sends a message with the following custom json payload:
This message is not displayed in the Chatroom window.
{ "handoff_host": "<url of handoff host endpoint>", "title": "<title for bot/channel handed off to>" }
- Chatroom switches the host to the specified
handoff_host
- The original bot no longer receives any messages.
- The handoff host receives the message
/handoff{"from_host":"<original bot url">}
- The handoff host should be configured to respond to this message with something like, "Hi, I'm , how can I help you??"
- The handoff host can send a message in the same format as specified above to hand back to the original bot. In this case the same pattern repeats, but with the roles reversed. It could also hand off to yet another bot/human.
The "try it out" section doesn't require any further configuration; this section is for those who want to change or further understand the set up.
For this demo, the user can ask for a human, but they'll be offered a bot (or bots) instead, so that the conversation looks like this:
For handoff to work, you need at least one "handoff_host". You can specify any number of handoff hosts in the file actions/handoff_config.yml
.
handoff_hosts:
helpdesk_assistant:
title: "Helpdesk Assistant"
url: "http://localhost:5005"
## you can add more handoff hosts to this list e.g.
# moodbot:
# title: "MoodBot"
# url: "http://localhost:5007"
Handoff hosts can be other locally running rasa bots, or anything that serves responses in the format that chatroom accepts. If a handoff host is not a rasa bot, you will of course want to update the response text to tell the user who/what they are being handed off to.
The Helpdesk-Assistant bot has been set up to handle handoff in exactly the same way as Helpdesk-Assistant, so the simplest way to see handoff in action is to clone Financial-Demo alongside this repo.
If you list other locally running bots as handoff hosts, make sure the ports on which the various rasa servers & action servers are running do not conflict with each other.
You can test the bot on the test conversations by:
- start duckling
- running
rasa test
.
This will run end-to-end testing on the conversations in tests/test_stories.yml
.
All tests must pass.
To deploy financial-demo, it is highly recommended to make use of the one line deploy script for Rasa X.
As part of the deployment, you'll need to set up git integration to pull in your data and configurations, and build or pull an action server image.
You will need to have docker installed in order to build the action server image. If you haven't made any changes to the action code, you can also use the public image on Dockerhub instead of building it yourself.
To build & tag the image, run:
export ACTION_SERVER_DOCKERPATH=<dockerID>/<name-of-image>:<tag-of-image>
make docker-build
Run the action server container:
make docker-run
Perform a smoke test on the health endpoint:
make docker-test
Once you have confirmed that the container is working, push the container image to a registry:
# login to a container registry with your credentials
docker login
# check the registry logged into
docker system info | grep Registry
# push the action server image
make docker-push
As explained in the Setting up CI/CD section of the Rasa documentation, Continous Integration (CI) is the practice of merging in code changes frequently and automatically testing changes as they are committed. Continuous Deployment (CD) means automatically deploying integrated changes to a staging or production environment. Together, they allow you to make more frequent improvements to your assistant and efficiently test and deploy those changes.
A CI/CD pipeline is used to test, build and deploy the financial-demo bot to AWS EKS.
The pipeline uses GitHub Actions, defined in .github/workflows/cicd.yml
. It includes these jobs:
- Defines parameters for use by downstream jobs
- Prints the value of the parameters.
- Builds & Tests the docker image of the action server with tag:
<branch-name>
- Uploads the docker image to the AWS ECR repository:
financial-demo
- Trains & Tests the rasa model with name: `models/.tar.gz
- Uploads the trained model to the AWS S3 bucket:
rasa-financial-demo
- If not existing yet, creates an AWS EKS cluster with name:
financial-demo-<branch-name>
- Installs/Updates Rasa Enterprise, with the docker image created by the action_server job.
- Deploys the rasa model, trained by the rasa_model job.
- Performs smoke tests to ensure basic operations are all OK.
- Runs when pushing to the
main
branch, and all previous steps are successful. - Installs/Updates Rasa Enterprise, with the docker image created by the action_server job.
- Deploys the rasa model, trained by the rasa_model job.
- Performs smoke tests to ensure basic operations are all OK.
Secrets can be added to your GitHub repository by going to Settings > Secrets
and selecting New repository secret
.
When entering values, be sure to omit quotes.
To configure the aws cli in Github Actions, create IAM User API Keys as described below, and add them as GitHub secrets to the repository:
- AWS_ACCESS_KEY_ID =
Access key ID
- AWS_SECRET_ACCESS_KEY =
Secret access key
To define a pull secret in Github Actions for the private GCR repo, you'll need to retrieve the private values from your Rasa Enterprise license file (docs) and add them as GitHub secrets to the repository:
- GCR_AUTH_JSON_PRIVATE_KEY_ID =
private_key_id
- GCR_AUTH_JSON_PRIVATE_KEY =
private_key
- GCR_AUTH_JSON_CLIENT_EMAIL =
client_email
- GCR_AUTH_JSON_CLIENT_ID =
client_id
An alternative approach to GCR repo authentication would be with the gcloud credential helper.
To use safe_credentials in Github Actions through values.yml
(docs), add following GitHub Secrets to the repo, replacing each <safe credential>
with a different alphanumeric string and choosing a <username>
for the initial user.
(Please use safe credentials to avoid data breaches)
- GLOBAL_POSTGRESQL_POSTGRESQLPASSWORD =
<safe credential>
- GLOBAL_REDIS_PASSWORD =
<safe credential>
- RABBITMQ_RABBITMQ_PASSWORD =
<safe credential>
- RASAX_INITIALUSER_USERNAME =
<username>
- RASAX_INITIALUSER_PASSWORD =
<safe credential>
- RASAX_JWTSECRET =
<safe credential>
- RASAX_PASSWORDSALT =
<safe credential>
- GLOBAL_POSTGRESQL_POSTGRESQLPASSWORD =
<safe credential>
- GLOBAL_POSTGRESQL_POSTGRESQLPASSWORD =
<safe credential>
The CI/CD pipeline of financial-demo uses AWS for all the storage & compute resources.
After cloning or forking the financial-demo GitHub repository you must set up the following items before the pipeline can run.
The CI/CD pipeline uses the aws cli.
The aws cli needs a set of IAM User API keys for authentication & authorization:
In your AWS Console, go to the IAM dashboard to create a new set of API keys:
-
Click on Users
-
Click on Add user
-
User name = findemo (The actual name is not important, we will never use this name directly)
Choose "programmatic access." This allows you to use the aws cli to interact with AWS.
-
Click on Next: Permissions
-
Click on Attach existing policies directly
For IAM access, you can choose “AdministratorAccess”, or limit access to only what is needed by the CD pipeline.
-
-
Click on Next: Tags
-
Click on Next: Review
-
Click on Create user
-
Store in a safe location:
Access key ID
&Secret access key
-
To SSH into the EC2 worker nodes of the EKS cluster, you'll need an SSH Key Pair
- In your AWS Console, go to EC2 > Key Pairs, and create a Key Pair with the name
findemo
, and download the filefindemo.pem
which contains the private SSH key. Note that the namefindemo
is important, since it is used by the CI/CD pipeline when the cluster is created.
Before the CI/CD pipeline can run, you will use the AWS CLI locally to create some resources and need to install & configure the CLI locally.
See the installation instructions.
# AWS CLI version 2.1.26 or later
aws --version
# Configure AWS CLI
aws configure
AWS Access Key ID [None]: ----- # See above: IAM User API Keys
AWS Secret Access Key [None]: ------- # See above: IAM User API Keys
Default region name [None]: us-west-2 # The CI/CD pipeline uses us-west-2
Default output format [None]:
# Check your configuration
aws configure list [--profile profile-name]
# verify it works
aws s3 ls
The CI pipeline creates two artifacts:
- An action server docker image, which is pushed to an AWS ECR repository.
- A trained rasa model, which is copied to an AWS S3 bucket
Run these commands to create the storage for these artifacts. These commands run the AWS CLI and create an ECR repository and an S3 bucket:
# create ECR repository with name `financial-demo`
make aws-ecr-create-repository
# create S3 bucket with name `rasa-financial-demo`
make aws-s3-create-bucket
If the production cluster is already set up, you can skip this section.
This section describes the initial deployment of the financial-demo bot on an EKS production cluster.
This initial deployment is done manually. After that, the deployment is maintained & upgraded automatically by the CI/CD pipeline.
Before you can create or interact with an EKS cluster, you must install eksctl
, kubectl
, helm
, jp
, and define some environment variables.
See the installation instructions
If you use Ubuntu, you can issue the command:
make install-eksctl
See the installation instructions
If you use Ubuntu, you can issue the command:
make install-kubectl
See the installation instructions
If you use Ubuntu, you can issue the command:
make install-helm
See the installation instructions
If you use Ubuntu, you can issue the command:
make install-jp
There are many ways to set the required environment variables in your local environment.
One way is to create a file ./secret/envs_export
, with this content:
# source this file to set the environment variables the same as the GitHub secrets.
export GCR_AUTH_JSON_PRIVATE_KEY_ID=...
export GCR_AUTH_JSON_PRIVATE_KEY='-----BEGIN PRIVATE KEY-...-END PRIVATE KEY-----\n'
export GCR_AUTH_JSON_CLIENT_EMAIL='...'
export GCR_AUTH_JSON_CLIENT_ID=...
export GLOBAL_POSTGRESQL_POSTGRESQLPASSWORD=...
export GLOBAL_REDIS_PASSWORD=...
export RABBITMQ_RABBITMQ_PASSWORD=...
export RASAX_DEBUG_MODE=false
export RASAX_INITIALUSER_USERNAME=admin
export RASAX_INITIALUSER_PASSWORD=...
export RASAX_JWTSECRET=...
export RASAX_PASSWORDSALT=...
export RASAX_TOKEN=...
export RASA_TOKEN=...
Then, create the environment variables with the command:
source ./secret/envs_export
Create the EKS production cluster, with name financial-demo-production
:
make aws-eks-cluster-create AWS_EKS_CLUSTER_NAME=financial-demo-production
Some other useful 'Makefile functions' to interact with an EKS cluster:
make aws-eks-cluster-list-all
# in commands below, default name of cluster = financial-demo-<current branch name>
#
make aws-eks-cluster-describe AWS_EKS_CLUSTER_NAME=<name of cluster>
make aws-eks-cluster-describe-stacks AWS_EKS_CLUSTER_NAME=<name of cluster>
make aws-eks-cluster-delete AWS_EKS_CLUSTER_NAME=<name of cluster>
Add the EKS cluster information to ~/.kube/config
, and set the current-context
to that cluster:
make aws-eks-cluster-update-kubeconfig AWS_EKS_CLUSTER_NAME=financial-demo-production
# make sure kubectl is now looking at the correct EKS cluster
make kubectl-config-current-context
Get compatible versions for rasa-x, rasa and rasa-sdk from compatibility matrix
For the rasa-x-helm chart, the latest version should work. When installing a very old version, you might need to use an older helm chart version, selected with:
#.............NAME URL
helm repo add rasa-x https://rasahq.github.io/rasa-x-helm
helm repo update
# .................NAME /CHARTNAME
$ helm search repo rasa-x/rasa-x --versions
NAME CHART VERSION APP VERSION DESCRIPTION
rasa-x/rasa-x 2.0.0 0.41.1 Rasa X Helm chart for Kubernetes / Openshift
rasa-x/rasa-x 1.16.0 0.40.1 Rasa X Helm chart for Kubernetes / Openshift
..
rasa-x/rasa-x 1.7.0 0.33.0 Rasa X Helm chart for Kubernetes / Openshift
rasa-x/rasa-x 1.6.13 0.30.1 Rasa X Helm chart for Kubernetes / Openshift
# NOTE: APP VERSION = Rasa X version
# file: requirements.txt
rasa[spacy]==2.5.2
rasa-sdk==2.5.0
# file: Dockerfile
FROM rasa/rasa-sdk:2.5.0
# file: Makefile
RASAX_TAG := 0.39.3
RASAX_HELM_CHART_VERSION := 2.0.0
Build, run, test & push the action docker server image, with name:
<ECR URI>/financial-demo:<current branch name>
# check out the `main` branch.
# -> the branch name is used as the image tag
git checkout main
# build the image
make docker-build
# test it
make docker-run
make docker-test
make docker-stop
# Login & push the docker image to the AWS ECR repository
make aws-ecr-docker-login
make docker-push
Install/Upgrade Rasa Enterprise, using the action server docker image that was uploaded to the ECR repository:
# make sure kubectl is looking at the correct EKS cluster
make aws-eks-cluster-update-kubeconfig AWS_EKS_CLUSTER_NAME=financial-demo-production
# check it
make kubectl-config-current-context AWS_EKS_CLUSTER_NAME=financial-demo-production
# check out the `main` branch.
# -> The action server docker image with tag = current branch name will be installed
git checkout main
# if you want to re-install rasa enterprise completely from scratch:
make rasa-enterprise-uninstall
make rasa-enterprise-delete-pvc-all
make aws-eks-namespace-delete
# create namespace `my-namespace`
make aws-eks-namespace-create
# create/refresh gcr-pull-secret, for Rasa Enterprise image
make pull-secret-gcr-create
# create/refresh ecr-pull-secret, for action server image
make pull-secret-ecr-create
# Install/Upgrade Rasa Enterprise with the action server
make rasa-enterprise-install
# Check Rasa Enterprise Health
make rasa-enterprise-check-health
# To troubleshoot, highly recommended to use OCTANT, see Appendix C
# check out the `main` branch.
git checkout main
# Train the model: `models/<current branch>.tar.gz`
make rasa-train
# In another window, start duckling server
docker run -p 8000:8000 rasa/duckling
# Run the end-to-end tests
make rasa-test
# Upload `models/<current branch>.tar.gz` to S3
# Note: This does not mean the model is deployed to Rasa Enterprise,
# which is done in the next step.
make aws-s3-upload-rasa-model
# Configure ~/.kube/config and set current_context
make aws-eks-cluster-update-kubeconfig AWS_EKS_CLUSTER_NAME=financial-demo-production
# Deploy rasa model
make aws-s3-download-rasa-model
make rasa-enterprise-model-delete
make rasa-enterprise-model-upload
make rasa-enterprise-model-tag
# Wait about 1 minute, so rasa-production can download, upack & load the model
# Smoketest
make rasa-enterprise-smoketest
Optionally, if you want to access Rasa Enterprise at your own (sub)-domain name, define a DNS record of type CNAME with your domain service provider:
-
name of sub-domain:
aws-financial-demo
==> This example name will resolve to
aws-financial-demo.my-domain.com
-
Type:
CNAME
-
Content:
--------.us-west-2.elb.amazonaws.com
==> This is the hostname of the External Application Load Balancer that AWS EKS created during the deployment. You can get this hostname with the command:
make rasa-enterprise-get-loadbalancer-hostname
-
TTL (Time to Live): 1 Hour
-
Priority:
It might take some time for things to propagate, but you can verify it with commands like nslookup & dig:
########################################################################
# use `nslookup -type`
nslookup -type=CNAME aws-financial-demo.my-domain.com
aws-financial-demo.my-domain.com canonical name = ---.us-west-2.elb.amazonaws.com.
########################################################################
# use `dig`
dig CNAME aws-financial-demo.my-domain.com
...
;; ANSWER SECTION:
aws-financial-demo.my-domain.com. ---- IN CNAME ---.us-west-2.elb.amazonaws.com.
Once propagated, you can access Rasa Enterprise at http://aws-financial-demo.my-domain.com
If something goes wrong with the CI/CD pipeline, you should:
- Browse the workflow logs
- Peek in .github/workflows/cicd.yml to see what the pipeline is doing at the moment of trouble
- Trouble shoot it from your local computer
The following steps will most likely reveal what is going wrong:
-
Before you can use the Makefile commands locally, follow the Preparation steps:
- Install eksctl, kubectl, helm, jp
- Set the environment variables
-
Then, configure kubectl, so you are looking at the correct EKS cluster.
-
To troubleshoot the EKS test cluster for your current branch
make aws-eks-cluster-update-kubeconfig
-
To Troubleshoot the EKS production cluster
make aws-eks-cluster-update-kubeconfig AWS_EKS_CLUSTER_NAME=financial-demo-production
-
-
Make sure you are now looking at the correct cluster by checking the current context:
make kubectl-config-current-context
-
Then, issue these commands to check things out.
-
Trouble shoot the EKS cluster
# Check the available commands for the `aws-eks` cluster: $ make aws-eks # Do NOT press `Enter`, but press the `Tab` key twice aws-eks-cluster-create aws-eks-cluster-info aws-eks-cluster-delete aws-eks-cluster-list-all aws-eks-cluster-describe aws-eks-cluster-status aws-eks-cluster-describe-stacks aws-eks-cluster-update-kubeconfig aws-eks-cluster-exists aws-eks-namespace-create aws-eks-cluster-get-certificateAuthority aws-eks-namespace-delete aws-eks-cluster-get-endpoint # Trouble shoot the cluster make aws-eks-cluster-exists make aws-eks-cluster-status make aws-eks-cluster-info
-
Trouble shoot the S3 storage of the trained model
# Check the available commands for the `aws-s3` bucket: make aws-s3 # Do NOT press `Enter`, but press the `Tab` key twice aws-s3-create-bucket aws-s3-list-rasa-models aws-s3-delete-bucket aws-s3-rasa-model-exists aws-s3-download-rasa-model aws-s3-upload-rasa-model # Trouble shoot the S3 storage of the trained model make aws-s3-list-rasa-models make aws-s3-rasa-model-exists
-
Trouble shoot the ECR storage of the action server image
# Check the available commands for the `aws-ecr` repository: make aws-ecr # Do NOT press `Enter`, but press the `Tab` key twice aws-ecr-create-repository aws-ecr-get-repositoryUri aws-ecr-docker-login aws-ecr-image-exists aws-ecr-get-authorization-token aws-ecr-list-images # First, login to the ECR (This expires after a while, so redo it) make aws-ecr-docker-login # Then, Troubleshoot the ECR storage of the action server image make aws-ecr-list-images make aws-ecr-image-exists
-
Trouble shoot the Rasa Enterprise deployment
# Check the available commands for the `rasa-enterprise` deployment: $ make rasa-e # Do NOT press `Enter`, but press the `Tab` key twice rasa-enterprise-check-health rasa-enterprise-get-secrets-rabbit rasa-enterprise-get-access-token rasa-enterprise-get-secrets-redis rasa-enterprise-get-base-url rasa-enterprise-install rasa-enterprise-get-chat-token rasa-enterprise-model-delete rasa-enterprise-get-loadbalancer-hostname rasa-enterprise-model-tag rasa-enterprise-get-login rasa-enterprise-model-upload rasa-enterprise-get-pods rasa-enterprise-smoketest rasa-enterprise-get-secrets-postgresql rasa-enterprise-uninstall # Trouble shoot the rasa enterprise deployment make rasa-enterprise-get-pods make rasa-enterprise-check-health make rasa-enterprise-smoketest
-
The CI/CD pipeline creates unique artifacts and infrastructure for each branch. This is all cleaned up once a branch is deleted, by a special workflow .github/workflows/cleanup.yml
.
You can also do a manual cleanup, either from the AWS CLI, or from the AWS Console.
-
AWS ECR:
# list the action server images in the ECR make aws-ecr-list-images # delete the action server image of current branch make aws-ecr-delete-image # delete an action server image created by another branch make aws-ecr-delete-image GIT_BRANCH_NAME=my_other_branch
-
AWS S3:
# list the rasa models in the S3 bucket make aws-s3-list-rasa-models # delete the rasa model of current branch make aws-s3-delete-rasa-model # delete a rasa model created by another branch make aws-s3-delete-rasa-model GIT_BRANCH_NAME=my_other_branch
-
AWS EKS:
# list the EKS clusters make aws-eks-cluster-list-all # to delete ALL the test clusters make aws-eks-cluster-delete-all-test-clusters # delete the test cluster of the current branch make aws-eks-cluster-delete # delete a test cluster created by another branch make aws-eks-cluster-delete GIT_BRANCH_NAME=my_other_branch
-
AWS ECR:
In the AWS Console, go to the ECR repository
financial-demo
, and delete the action server images. -
AWS S3:
In the AWS Console, go to the S3 bucket
rasa-financial-demo
, and delete the rasa models. -
AWS EKS:
In the AWS Console, go to CloudFormation, and delete all the stacks in reverse order as created by the
eksctl
command.When a stack fails to delete due to dependencies, you have two options:
-
Option 1: Manually delete the resources that the stack is not able to delete. (RECOMMENDED)
You can do this by drilling down into the CloudFormation stack delete events messages and deleting items bottom-up the dependency tree.
One example of a bottom-up delete sequence is when deletion of the VPC fails:
- EC2 > Load Balancers: first, delete the ELB load balancer of the cluster. Look at the Tags to see what cluster a loadbalancer belongs to.
- VPC > Virtual Private Cloud: Subnets: then, delete the two subnets of the cluster. Look at the Name to see what cluster a subnet belongs to.
- This will also delete the EC2 > Network interfaces, named
eni-xxxx
- You cannot delete Subnets until the ELB load balancers are deleted
- This will also delete the EC2 > Network interfaces, named
- VPC > Virtual Private Cloud: Your VPCs: finally, delete the VPC of the cluster. Look at the Name to see what cluster a VPC belongs to.
- This will also delete all associated:
- security groups (
sg-xxx
) - internet gateways (
igw-xxx
) - subnets (
subnet-xxx
)
- security groups (
- This will also delete all associated:
After cleaning up, try again to delete the AWS CloudFormation stack.
If it still does not delete, iterate the manual cleanups until it does.
Again, this can be a painful process, but once the CloudFormation stacks delete properly, you are guaranteed that you have cleaned up all the EKS related resources created by the CI/CD pipeline.
-
Option 2: Select to retain the resources that have dependency errors. (NOT RECOMMENDED)
The stack delete operation will simply skip deleting them. This is NOT recommended, because you will clutter up your AWS account with many unused resources.
-
We use eksctl to create the clusters. It creates many AWS resources with CloudFormation templates.
Some of these AWS resources are:
- A VPC with
- Public & private subnets
- Internet & NAT Gateways
- Route Tables
- An IAM EKS service role
- An EKS Cluster Control Plane
- An EKS Managed nodegroup of EC2 instances
The cluster context is also added to the ~/.kube/config
file.
The VPC created looks something like this:
The EKS Control Plane interacts with the the EKS Data Plane (the nodes), like this:
Octant is a useful open sourced tool for visualizing workloads inside the cluster and troubleshooting issues when they arise.
cd ~
mkdir octant
cd octant
wget https://github.com/vmware-tanzu/octant/releases/download/v0.20.0/octant_0.20.0_Linux-64bit.deb
sudo dpkg -i octant_0.20.0_Linux-64bit.deb
# #################################################### #
# Configure kubectl to look at the correct EKS cluster #
# #################################################### #
# -> for a feature branch test cluster
git checkout my-feature-branch
# -> for the production cluster
git branch production # create temporary, local `production` branch
git checkout production
# Configure kubeconfig for the correct EKS cluster
make aws-eks-cluster-update-kubeconfig
# make sure kubectl is now looking at the correct EKS cluster
make kubectl-config-current-context
# #################################################### #
# Run Octant on default port 7777 and open the browser #
# #################################################### #
OCTANT_LISTENER_ADDR=0.0.0.0:7777 octant &
# Within octant, select the namespace `my-namespace`
There are two commands to create AWS EKS clusters; eksctl
& aws eks
.
-
The
eksctl
cli is the most complete, and it is what we ended up using. -
The
aws eks
cli does not support to launch worker nodes to the cluster control plane. This has to be done manually from the AWS Console, which makes it unsuited for a CI/CD pipeline where everything needs to be done via scripting (=> infrastructure as code).
The following references are useful for learning about AWS EKS, and it is highly recommended to manually build test-clusters with both the eksctl
& aws eks
commands to help demystify the AWS resources that are being generated:
- eksctl – the EKS CLI
- Getting started with Amazon EKS - eksctl
- Blog: EKS Cluster with One Command
- Blog: Demystifying cluster networking for EKS worker nodes
- https://docs.aws.amazon.com/eks/latest/userguide/create-public-private-vpc.html
- https://docs.aws.amazon.com/eks/latest/userguide/eks-compute.html
- https://logz.io/blog/amazon-eks-cluster/amp/
- https://www.eksworkshop.com/
- aws eks - Command Reference (Not used)