Skip to content

nnsutebu/ecommerce-app-deploy-eks

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Microservice Web Application Project Architecture

Online Shopping Application: This online shopping application was architected and built using cloud-first related principles and methodologies that promotes the adoption of application management strategies such as Microservices. The Online Shopping Application consists of about 11 microservices.

ProjectArch


Jenkins Microservices MultiBranch CI/CD Pipeline Automation Arch | One

PipelineArch1


Jenkins Microservices MultiBranch CI/CD Pipeline Automation Arch | Two

PipelineArch2

Continuous Integration

  1. Create a GitHub Repository with the name multi-microservices-application-project and push the code in this branch (main) to your remote repository (your newly created repository).
    • Go to GitHub: https://github.com
    • Login to Your GitHub Account
    • Create a Repository called multi-microservices-application-project
    • Clone the Repository in the Repository directory/folder on your local machine
    • Download the code in in this repository "multi-microservices-application-project main branch": https://github.com/awanmbandi/realworld-microservice-project.git
    • Unzip the code/zipped file
    • Copy and Paste everything from the zipped file into the repository you cloned in your local
    • Open your Terminal
      • Add the code to git, commit and push it to your upstream branch "main or master"
      • Add the changes: git add -A
      • Commit changes: git commit -m "adding project source code"
      • Push to GitHub: git push
    • Confirm that the code is now available on GitHub
  2. Download the code from ALL the other BRANCHES as well
  • Create the following branches in the Repository, you just created above
    • app-ad-serverice
    • app-cart-service
    • app-checkout-service
    • app-currency-service
    • app-database
    • app-email-service
    • app-frontend-service
    • app-loadgenerator-service
    • app-payment-service
    • app-product-catalog-service
    • app-recommendation-service
    • app-shipping-service
  • Download the Source Code of each microservice, from their respective branches from this Repository https://github.com/awanmbandi/realworld-microservice-project.git
  • And Push the Code based on the Microservice to the specific Branch you Created for that Service.
  1. Create An IAM Profile/Role For The Jenkins-CI Server
  • Create an EC2 Service Role in IAM with AdministratorAccess Privilege
  • Navigate to IAM IAM!
    • Click on Roles
    • Click on Create Role
    • Select Service Role
    • Use Case: Select EC2
    • Click on Next
    • Attach Policy: AdministratorAccess
    • Click Next
    • Role Name: AWS-EC2-Administrator-Role
    • Click Create
  1. Jenkins CI

⚠️ NOTE:ALERT ⚠️

  • ONLY VISIT THIS SECTION IF YOU STOPPED AND RESTARTED YOUR JENKINS SERVER
  • The above Jenkins Userdata includes a SonarQube container deployment task
    • As a result, we know containers are Ephemeral by natuure, so if you Stop your Jenkins CI Server at any point in time... You'll have to Deploy the Container again when you Start it back or bring the instance up again.
    • If you don't do this, you will not be able able to proceed with the project.
    • I have also Included a Docker Volume setup task as well for SonarQube, where the Container Data will be persisted to avoid Data lost.
# Volume inspection, confirm the docker volume exist
docker volume inspect volume sonarqube-volume

# Create a new conainter, provide your container name and deploy in the `Jenkins-CI` server
docker run -d --name PROVIDE_NAME_HERE -v sonarqube-volume:/opt/sonarqube/data -p 9000:9000 sonarqube:lts-community
  1. Slack
    • Go to the bellow Workspace and create a Private Slack Channel and name it "yourfirstname-jenkins-cicd-pipeline-alerts"
    • Link: https://join.slack.com/t/jjtechtowerba-zuj7343/shared_invite/zt-24mgawshy-EhixQsRyVuCo8UD~AbhQYQ
      • You can either join through the browser or your local Slack App
      • Create a Private Channel using the naming convention YOUR_INITIAL--multi-microservices-alerts
        • NOTE: (The Channel Name Must Be Unique, meaning it must be available for use)
      • Visibility: Select Private
      • Click on the Channel Drop Down and select Integrations and Click on Add an App
      • Search for Jenkins and Click on View
      • Click on Configuration/Install and Click Add to Slack
      • On Post to Channel: Click the Drop Down and select your channel above YOUR_INITIAL-multi-microservices-alerts
      • Click Add Jenkins CI Integration
      • Scrol Down and Click SAVE SETTINGS/CONFIGURATIONS
      • Leave this page open SlackConfig!

5A) Verify the Following Services are running in the Jenkins Instance

  • SSH into the Jenkins-CI server
    • Run the following commands and confirm that the services are all Running
# Confirm Java version
sudo java --version

# Confirm that Jenkins is running
sudo systemctl status jenkins

# Confirm that docker is running
sudo systemctl status docker

# Confirm that Terraform is running
terraform version

# Confirm that the Kubectl utility is running 
kubectl version --client

# Confirm that AWS CLI is running
aws --version

# Confirm that the SonarQube container is running
docker ps | grep sonarqube:lts-community

# Lastly confirm that the `sonarqube-volume docker volume` was created
docker volume inspect volume sonarqube-volume

5B) Deploy Your EKS Cluster Environment

  • UPDATE Your Terraform Provider Region to Your Choice REGION*
    • ⚠️NOTE:ALERT!⚠️: Do Not Use North Virginia, that's US-EAST-1
    • ⚠️NOTE:ALERT!⚠️: Also Confirm that The Selected Region Has A Default VPC You're Confident Has Internet Connection
    • ⚠️NOTE:ALERT!⚠️: The Default Terraform Provider Region Defined In The Config Is Ohio(US-EAST-2)
  • Confirm you're still logged into the Jenkins-CI Server via SSH
  • Run the following commands to deploy the EKS Cluster in the Jenkins-CI
  • NOTE: You Can As Well Deploy The Cluster Using Terraform From Your Local System
# Clone your project reporisoty
git clone https://github.com/awanmbandi/realworld-microservice-project.git

# cd and checkout into the DevSecOps project branch
cd realworld-microservice-project 
cd terraform/AWS/eks-cluster

# Deploy EKS Environment
terraform init
terraform plan
terraform apply --auto-approve
  • Navigate to EKS and confirm your Cluster was created successfully
  • Also confirmthere's no issue regarding your Terraform execution JenkinsSetup1! JenkinsSetup2!

5C) Once The Cluster Deployment Completes, Go Ahead and Enable The OIDC Connector/Provider

eksctl utils associate-iam-oidc-provider \
    --region us-east-2 \
    --cluster EKS_Cluster \
    --approve

Update/Get Cluster Credential:

aws eks update-kubeconfig --name <clustername> --region <region>

Update the EKS Cluster Security Group (Add A NodePort and Frontend Port)

  • Navigate to EC2
    • Select any of the Cluster Worker Nodes
    • Click on Security
    • Click on the EKS Cluster Security Group ID
    • Click on Edit Inbound Rules
    • Click on Add Rule
    • Port Number: 30000-32767, 80, 22 Source: 0.0.0.0/0
    • Click on SAVE

Jenkins setup

  1. Access Jenkins

    Copy your Jenkins Public IP Address and paste on the browser = ExternalIP:8080

    • Login to your Jenkins instance using your Shell (GitBash or your Mac Terminal)
    • Copy the Path from the Jenkins UI to get the Administrator Password
      • Run: sudo cat /var/lib/jenkins/secrets/initialAdminPassword
      • Copy the password and login to Jenkins JenkinsSetup1!
    • Plugins: Choose Install Suggested Plugings
    • Provide
      • Username: admin
      • Password: admin
      • Name and Email can also be admin. You can use admin all, as its a poc.
    • Click Continue
    • Click on Start using Jenkins JenkinsSetup2!
  2. Plugin installations:

    • Click on Manage Jenkins
    • Click on Plugins
    • Click Available
    • Search and Install the following Plugings and "Install"
      • SonarQube Scanner
      • Snyk
      • Multibranch Scan Webhook Trigger
      • Eclipse Temurin installer
      • Pipeline: Stage View
      • Docker
      • Docker Commons
      • Docker Pipeline
      • docker-build-step
      • Docker API
      • Kubernetes
      • Kubernetes CLI
      • Kubernetes Credentials
      • Kubernetes Client API
      • Kubernetes Credentials Provider
      • Kubernetes :: Pipeline :: DevOps Steps
      • Slack Notification
      • ssh-agent
      • BlueOcean
      • Build Timestamp
    • Click on Install
    • Once all plugins are installed
    • Select/Check the Box Restart Jenkins when installation is complete and no jobs are running PluginInstallation!
    • Refresh your Browser and Log back into Jenkins
    • Once you log back into Jenkins
  3. Global tools configuration:

    • Click on Manage Jenkins -->> Global Tool Configuration JDKSetup!

    • JDK

      • Click on Add JDK -->> Make sure Install automatically is enabled

      Note: By default the Install Oracle Java SE Development Kit from the website make sure to close that option by clicking on the image as shown below.

      • Name: JDK17
      • Click on Add installer
      • Select Install from adoptium.net
      • Version: jdk-17.0.8.1+1

      JDKSetup!

    • SonarQube Scanner

      • Click on Add SonarQube Scanner
      • Name: SonarScanner
      • Enable: Install automatically SonarQubeScanner!
    • Docker installations

      • Click on Add Docker
      • Name: Docker
      • Click on Add installer
        • Select Download from docker.com
        • Docker version: latest
      • Enable: Install automatically SonarQubeScanner!
    • Gradle Installation

      • Click on Add Gradle
      • Name: Gradle
      • Enable Install automatically
      • Version: Go with the latest GradleInstallation!
    • Snyk Installations

      • Click on ``Add Snyk`
      • Name: Snyk
      • Enable: Install automatically
        • Version: latest
        • Update policy interval (hours): 24
        • OS platform architecture: Auto-detection SnykInstallation!
  4. Credentials setup(SonarQube, Slack, DockerHub, Kubernetes and ZAP):

    • Click on Manage Jenkins

      • Click on Credentials
      • Click on Jenkins - System
      • Click on Global Credentials (Unrestricted)
      • Click on Add Credentials
      1. SonarQube secret token (SonarQube-Token)
        • Generating SonarQube secret token:
          • Login to your SonarQube Application (http://SonarServer-Sublic-IP:9000)

            • Default username: admin
            • Default password: admin
          • Click on Login

            • Old Password: admin

            • New Password: adminadmin

            • Confirm Password: adminadmin

            • Click on Manually (Create the app-shipping-service microservice test project)

              • Project display name: app-shipping-service
              • Display key: app-shipping-service
              • Main branch name: app-shipping-service
            • Click on Projects (Create the app-recommendation-service microservice test project)

              • Project display name: app-recommendation-service
              • Display key: app-recommendation-service
              • Main branch name: app-recommendation-service
            • Click on Projects (Create the app-product-catalog-service microservice test project)

              • Project display name: app-product-catalog-service
              • Display key: app-product-catalog-service
              • Main branch name: app-product-catalog-service
            • Click on Projects (Create the app-payment-service microservice test project)

              • Project display name: app-payment-service
              • Display key: app-payment-service
              • Main branch name: app-payment-service
            • Click on Projects (Create the app-loadgenerator-service microservice test project)

              • Project display name: app-loadgenerator-service
              • Display key: app-loadgenerator-service
              • Main branch name: app-loadgenerator-service
            • Click on Projects (Create the app-frontend-service microservice test project)

              • Project display name: app-frontend-service
              • Display key: app-frontend-service
              • Main branch name: app-frontend-service
            • Click on Projects (Create the app-email-service microservice test project)

              • Project display name: app-email-service
              • Display key: app-email-service
              • Main branch name: app-email-service
            • Click on Projects (Create the app-database microservice test project)

              • Project display name: app-database
              • Display key: app-database
              • Main branch name: app-database
            • Click on Projects (Create the app-currency-service microservice test project)

              • Project display name: app-currency-service
              • Display key: app-currency-service
              • Main branch name: app-currency-service
            • Click on Projects (Create the app-checkout-service microservice test project)

              • Project display name: app-checkout-service
              • Display key: app-checkout-service
              • Main branch name: app-checkout-service
            • Click on Projects (Create the app-cart-service microservice test project)

              • Project display name: app-cart-service
              • Display key: app-cart-service
              • Main branch name: app-cart-service
            • Click on Projects (Create the app-ad-serverice microservice test project)

              • Project display name: app-ad-serverice
              • Display key: app-ad-serverice
              • Main branch name: app-ad-serverice
            • Click on Set Up

          • Generate a Global Analysis Token This is the Token you need for Authorization

            • Click on the User Profile icon at top right of SonarQube
            • Click on My Account
            • Click Security
            • Generate Token: Generate this TOKEN and Use in the Next Step to Create The SonarQube Credential
              • Name: microservices-web-app-token
              • Type: Global Analysis Token
              • Expires in: 30 days Sonar!
            • Click on GENERATE
            • NOTE: Save The Token Somewhere...
        • Store SonarQube Secret Token in Jenkins:
          • Navigate back to Jenkins http://JENKINS_PUBLIC_IP:8080
          • Click on Manage Jenkins
            • Click on Jenkins System
            • Click Global credentials (unrestricted)
          • Click on Add Credentials
          • Kind: Secret text
          • Secret: Paste the SonarQube TOKEN value that we have created on the SonarQube server
          • ID: SonarQube-Credential
          • Description: SonarQube-Credential
          • Click on Create
      2. Slack secret token (slack-token)
        • Get The Slack Token:
        • Create The Slack Credential For Jenkins:
          • Click on Add Credentials
            • Click on Jenkins System
            • Click Global credentials (unrestricted)
          • Kind: Secret text
          • Secret: Place the Integration Token Credential ID (Note: Generate for slack setup)
          • ID: Slack-Credential
          • Description: Slack-Credential
          • Click on Create
      3. DockerHub Credential (Username and Password)
        • Login to Your DockerHub Account (You can CREATE one if you Don't have an Account)
          • Access DockerHub at: https://hub.docker.com/
          • Provide Username: YOUR USERNAME
          • Provide Username: YOUR PASSWORD
          • Click on Sign In or Sign Up
            • NOTE: If you have an account Sign in If not Sign up
        • DockerHub Credential (Username and Password)
          • Click on Add Credentials
            • Click on Jenkins System
            • Click Global credentials (unrestricted)
          • Kind: Username with password
          • Username: YOUR USERNAME
          • Password: YOUR PASSWORD
          • ID: DockerHub-Credential
          • Description: DockerHub-Credential
          • Click on Create
      4. Kubernetes Cluster Credential (kubeconfig)
      • Start By Increasing The EBS Volume Size of Your Kubernetes Cluster Worker Nodes
        • Navigate to EC2
        • Click on Volumes
        • Select and Modify Both Nodes Volumes
        • Size: 130 GB
        • Click Modify
      • Get Cluster Credential From Kube Config
        • SSH back into your Jenkins-CI server
        • RUN the command: aws eks update-kubeconfig --name <clustername> --region <region>
        • COPY the Cluster KubeConfig: cat ~/.kube/config
        • COPY the KubeConfig file content
          • Create a File Locally
          • RUN: touch ~/Downloads/kubeconfig-secret.txt
          • RUN: vi ~/Downloads/kubeconfig-secret.txt
          • PASTE and SAVE the KubeConfig content in the file
      • Create The Kubernetes Credential In Jenkins
        • Navigate back to Jenkins
        • Click on Add Credentials
          • Click on Jenkins System
          • Click Global credentials (unrestricted)
        • Kind: Secret File
        • File: Click Choose File
          • NOTE: Seletct the KubeConfig file you saved locally
        • ID: Kubernetes-Credential
        • Description: Kubernetes-Credential
        • Click on Create
      1. Create the ZAP Dynamic Application Security Testing Server Credential
        • Start by Copy the EC2 SSH Private Key File Content of your Jenkins-CI Server
          • Open your GitBash Terminal or MacOS Terminal
          • Navigate to the Location where your Jenkins-CI Server SSH Key is Stored (Usually in Downloads)
          • Run the Command cat YOUR_SSH_KEY_FILE_NAME.pem
          • COPY the KEY content and Navigate back to Jenkins to store it...
        • Create The ZAP Server SSH Key Credential in Jenkins
          • Navigate to the Jenkins Global Credential Dash
          • Click on Create Credentials
          • Scope: Select Global......
          • Type: Select SSH Username with Private Key
          • ID and Description: OWASP-Zap-Credential
          • Username: ubuntu
          • Private key: Select
            • Key: Click on Add
            • Key: Paste The Private Key Content You Copied
          • Click on Create
      2. Create Your Snyk Test (SCA) Credential
        • Navigate to: https://snyk.com/
          • Click on Sign Up
          • Select GitHub
            • Once you're login to your Snyk account
          • Click on Your Name below Help on the Botton left hand side of your Snyk Account
          • Click on Account Settings
          • Auth Token (KEY): Click on Click To Show
          • COPY the TOKEN and SAVE somewhere
      • Create SNYK Credential in Jenkins
        • Click on Add Credentials
        • Kind: Secret text
        • Secret: Paste the SNYK TOKEN
        • ID: Snyk-API-Token
        • Description: Snyk-API-Token
        • Click on Create KubeCredential!
  5. Configure system:

      • Click on Manage Jenkins
      • Click on System and navigate to the SonarQube Servers section
      • Click on Add SonarQube
      • Name: Sonar-Server
      • Server URL: http://YOUR_JENKINS_PRIVATE_IP:9000
      • Server authentication token: Select SonarQube-Credential SonarQubeServerSetup!
      • Still on Manage Jenkins and Configure System
      • Scroll down to the Slack Section (at the very bottom)
      • Go to section Slack
        • NOTE: Make sure you still have the Slack Page that has the team subdomain & integration token open
        • Workspace: Provide the Team Subdomain value (created above)
        • Credentials: select the Slack-Credential credentials (created above)
        • Default channel / member id: #PROVIDE_YOUR_CHANNEL_NAME_HERE
        • Click on Test Connection
        • Click on Apply and Save SlackSetup!

Update the EKS Cluster Security Group (Add A NodePort)

  • Navigate to EC2
    • Select any of the Cluster Worker Nodes
    • Click on Security
    • Click on the EKS Cluster Security Group ID
    • Click on Edit Inbound Rules
    • Click on Add Rule
    • Port Number: 30000, Source: 0.0.0.0/0
    • Click on SAVE

Pipeline creation (Make Sure To Make The Following Updates First)

  • UPDATE YOUR Jenkinsfiles...

  • Update your Frontend Service - OWASP Zap Server IP (Which is Jenkins IP) in the Jenkinsfile on Line 87

  • Update the EKS Worker Node IP with yours in the Jenkinsfile on Line 87

  • Update your Slack Channel Name in the Jenkinsfiles... - All Microservices

  • Update SonarQube projectName of you Microservices in the Jenkinsfiles... - All Microservices

  • Update the SonarQube projectKey of you Microservices in the Jenkinsfiles... - All Microservices

  • Update the DockerHub username of you Microservices in the Jenkinsfiles... - All Microservices, provide Yours

    • Log into Jenkins: http://Jenkins-Public-IP:8080/

    • Click on New Item

    • Enter an item name: Online-Shop-Microservices-CICD-Automation

    • Select the category as Multibranch Pipeline

    • Click OK

    • BRANCH SOURCES:

      • Git:
        • Project Repository
          • Repository URL: Provide Your Project Repo Git URL (the one you created at the beginning)
    • BEHAVIORS

      • Set it to: Discover Branches and
      • Click Add
        • Select: Filter by name (with wildcards)
        • Include: app-*
    • Property strategy: All branches get the same properties

    • BUILD CONFIGURATION

      • Mode: Select by Jenkinsfile
      • Script Path: Jenkinsfile
    • SCAN MULTIBRANCH PIPELINE TRIGGER

      • Select Scan by webhook
      • Trigger token: automation
    • Click on Apply and Save

    • CONFIGURE MULTIBRANCH PIPELINE WEBHOOK

      • Copy this URL and Update the Jenkins IP (to yours): http://PROVIDE_YOUR_JENKINS_IP:8080/multibranch-webhook-trigger/invoke?token=automation
      • Navigate to your Project Repository
        • Click on Settings in the Repository
        • Click on Webhooks
        • Click on Add Webhook
        • Payload URL: http://PROVIDE_YOUR_JENKINS_IP:8080/multibranch-webhook-trigger/invoke?token=automation
        • Content type: application/json
        • Which events would you like to trigger this webhook: Select Just the push event
        • Enable Active
        • Click ADD WEBHOOK

Navigate Back To Jenkins and Confirm That All 12 Pipeline Jobs Are Running (11 Microservices Jobs and 1 DB Job)

  • Click on the Jenkins Pipeline Job Name
  • Click on Scan Multibranch Pipeline Now MicroservicesPipelineJobs

Confirm That All Microservices Branch Pipelines Succeeded (If Not, Troubleshoot)

MicroservicesPipelineJobs MicroservicesPipelineJobs

SonarQube Code Inspection Result For All Microservices Source Code

SonarQubeResult!

Also Confirm You Have All Service Deployment/Docker Artifacts In DockerHub

DockerHubImages

PERFORM THE DEPLOYMENT IN THE STAGING ENVIRONMENT/NAMESPACE (EKS CLUSTER)

  • To perform the DEPLOYMENT in the staging Envrionment
  • You Just Have To UNCOMMENT the DEPLOY STAGE in the Jenkinsfiles..... and PUSH to GitHub
  • DEPLOY the Microservices in the STAGING Environment in the following ORDER (To Resolve DEPENDENCIES around the SERVICES)
  1. Redis DB
  2. Product Catalog Service
  3. Email Service
  4. Currency Service
  5. Payment Service
  6. Shipping Service
  7. Cart Service
  8. Ad Service
  9. Recommendation Service
  10. Checkout Service
  11. Frontend
  12. Load Generator

A. Test Application Access From the Test/Stagging-Environment Using NodePort of one of your Workers

  • SSH Back into your Jenkins-CI Server

    • RUN: kubectl get svc -n test-env
    • NOTE: COPY the Exposed NodePort Pod Number NodeportTestEnv
  • Access The Application Running in the Test Environment within the Cluster

  • Update the EKS Cluster Security Group (If you've not already)

    • To do this, navigate to EC2
    • Select one of the Worker Nodes --> Click on Security --> Click on The Security Group ID
    • Click on Edit Inbound Rules: Port = 30000 and Source 0.0.0.0/0
  • Open your Browser

  • Go to: http://YOUR_KUBERNETES_WORKER_NODE_IP:30000 TestEnv

  • Stage Deployment Succeeded TestEnv

PERFORM THE DEPLOYMENT NOW TO THE PRODUCTION ENVIRONMENT/NAMESPACE (EKS CLUSTER)

  • To perform the DEPLOYMENT to the Prod Envrionment
  • You Just Have To UNCOMMENT the DEPLOY STAGE in the Jenkinsfiles..... and PUSH to GitHub
  • DEPLOY the Microservices to the Prod Environment in the following ORDER (To Resolve DEPENDENCIES around the SERVICES)
  1. Redis DB
  2. Product Catalog Service
  3. Email Service
  4. Currency Service
  5. Payment Service
  6. Shipping Service
  7. Cart Service
  8. Ad Service
  9. Recommendation Service
  10. Checkout Service
  11. Frontend
  12. Load Generator
  • Confirm That Your Production Deployment Succeeded ProdEnv

    • To access the application running in the Prod-Env
    • Navigate back to the Jenkins-CI shell
    • RUN: kubectl get svc
    • Copy the LoadBalancer DNS and Open on a TAB on your choice Browser http://PROD_LOADBALANCER_DNS TestEnv
  • SonarQube Code Inspection Result For All Microservices Source Code SonarQubeResult!

  • Snyk SCA Test Result SnykResult!

  • Test/Scan Dockerfiles with Open Policy Agent (OPA) OPATest!

  • Slack Continuous Feedback Alert SlackResult!

Congratulations Your Deployment Was Successful

Home Page Checkout Page
Screenshot of store homepage Screenshot of checkout screen

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 23.4%
  • Go 20.9%
  • HCL 10.9%
  • C# 7.6%
  • HTML 6.9%
  • Shell 5.5%
  • Other 24.8%