Skip to content

AKS based jmeter testing framework with Grafana reporting

Notifications You must be signed in to change notification settings

petegrimsdale/aks_testing_fwk

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

53 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

AKS based scalable Jmeter Test Framework with Grafana

Latest Version: 1.5

Introduction

This Jmeter based testing framework provides a scalable test harness to support load testing of applications using Apache JmeterTM based test scripts. The framework excludes support for writing a Jmeter test plan but assumes a test plan in the form of a jmx files is available. The testing framework utilizes a master Jmeter node with one or more slave nodes used to run the tests. The deployment assumes a Jmeter backend listener is configured within the test plan to support writing metrics to the Influx database which can then be presented via a Grafana dashboard. The initial deployment only deploys a single jmeter-slave pod but can be scaled as needed to support the required number of client threads.

Architecture

Test Framework Architecure

The framework uses a Kubernetes based deployment of Apache Jmeter, InfluxDB and Grafana. The framework builds on Apache Jmeter's distributed load testing model whereby tests are initiated from a Jmeter master node(1) which then distributes the test script to the slaves (jmeter instances), the slaves are nodes/pods that carry out the load testing. Note: The test plan is replicated to all slaves so consideration of the overall client load is needed. For example, a test plan with 100 client threads distributed to 5 jmeter slaves will result in 500 active clients. You will use a distributed architecture like this if you want to carry out an intensive load test which can simulate hundreds and thousands of simultaneous users, this is the scenario we will be looking at in this blog post.The results of any load testing are sent to the InfluxDB using the built in BackEndLister available in Jmeter (see test plan construction below) and Grafana is used to render this data in an easily consumable dashboard.

Installation

Pre-requisities

  1. Clone the github repo

  2. Navigate to the installs directory

  3. Make the install.sh script executable

    • chmod +x install.sh
  4. Running install.sh

    • can we run with the following parameters

      validate

      install.sh validate -g {resource-group-name} -s {service-principal-name} -l {location}

      • This will validate the target environment, checking resource group, service principal name and AKS

      install

      Non Vnet based deployment install.sh install -g {resource-group-name} -s {service-principal-name} -l {location}

      Existing Vnet based deployment install.sh install -g {resource-group-name} -s {service-principal-name} -l {location} --vnetname {vnet} --subnetname {subnet}

      New Vnet based deployment install.sh install -g {resource-group-name} -s {service-principal-name} -l {location} --vnetname {vnet} --subnetname {subnet} -v {vnet cidr} -n {subnet cidr}

      Existing VNET with separate resource group for testing framework components install.sh install -g {resource-group-name} -s {service-principal-name} -l {location} --vnetname {vnet} --subnetname {subnet} --fwkrg {name of resource group}

      • This will deploy to the target environment using the resource group and location defined. It will create the resource group, service principal, Azure Container Registry, build and upload containers for Jmeter Master, Jmeter Slave and reporting, create an AKS cluster and deploy and configure all the elements required to run a test.

      delete

      install.sh delete -g {resource-group-name} -s {service-principal-name} --fwkrg {fwk-resource-group}

      This will remove all resources and the service principal. Note: If deploying to an alternative resource group with the --fmwkrg option then use this to remove that group

      kube_deploy

      install.sh kube_deploy -g {resource-group-name} -c {aks-cluster-name}

      • This will deploy and configure all the kubernetes elements required to deploy pods, configure Influx and Grafana on an existing Test Framework cluster.
  5. Once the deployment has completed successfully the Grafana dashboard can be accessed via the public IP of the provisioned loadbalancer. The easiest way to obtain this is to run

  • kubectl get svc

kubeclt result

The public IP address of the loadbalancer can then be used to access the Grafana login window ( access with default credentials admin/admin)

grafana login

Set up Jmeter Dashboard

Once the Grafana dashboad has been accessed an initial Grafana page should be loaded ( note: a default datasource has already been created )

An initial dashboard has been loaded as part of the installation as :

grafana dashboard

Select the Jmeter dashboard to open the default dashboard to support testing

If no dashboard is available follow the steps below to load a Jmeter dashboard to display testing data

grafana dashboard

Select Dashboards ....Manage as below:

grafana dashboard

This will open the Dashboard Management Window.....Select "Import"

grafana import

On the Import window enter 4026 as the Id for the dashboard as below

grafana uid

This will import the dashboard from Grafana labs and display the settings dialog as below - update the database name as shown and select Import to load the dashboard

grafana datasource

The Grafana Jmeter dashboard will be loaded with the default settings ready for your first Jmeter Test

grafana Jmeter dashboard

The key items to note are:

  1. The application Name - this will be linked to your Jmeter test plan configuration ( see below )
  2. Start/Stop marker - this will indicate when a jmeter test is started and stopped
  3. Time range - this will define the time period the dashboard displays data for.

Running your first Jmeter test

To validate the deployment a simple test script has been provided. To run this test navigate to the deploy directory. To run any tests the file starttest.sh is used in the form:

  • make the file starttest.sh executable
chmod +x starttest.sh
  • to execute a test run
./starttest.sh {testfilename}

for the example test script:

./starttest.sh simple.jmx

Example output will show the testplan being "distributed" to a single node ( as there is only 1 slave node with the default install )

simple test run

If you navigate to the Grafana dashboard you will see output data similar to: (note: change the timerange and application name)

grafana output

Writing your own Jmeter Test Plan

Jmeter Test Plan Setup

This page does not provide full details of how to create a Jmeter test plan - refer to the Apache Jmeter documentation for this purpose, however the key element to include into the test plan is a BackendListener configuration

BackendListener Configuration

The Jmeter BackEndListener needs to be configured to send data to InfluxDB to allow the real-time metrics

backendListener

Select the correct Backend Listener Implementation - this needs to be the Influx listener The influxDb URL must be set to: http://jmeter-influxdb:8086/write?db=jmeter Application - this is a name of your choice ( used in the Grafana dashboard to filter ) Measurement - should be set to jmeter

Running your own tests

Test Plan

After creating your testplan using a local Jmeter GUI instance save a copy of the testplan to the "deploy" directory Run the testplan with:

 ./starttest.sh {testplan.jmx}

Scaling the Test Framework

The Test Framework is deployed with a single Jmeter Slave node. To run a larger test ( more slaves ) scale out the jmeter-slaves deployment using:

kubectl scale deployments jmeter-slaves --replicas={number required}

This will scale the number of slaves ( note the configuration is set to autoscale the cluster and you will need to wait whilst the cluster scales out) Check the status with:

kubectl get deployments -w

Troubleshooting Notes

Remote Slave does not start

In some instances there may be a scenario where the remote slave cannot be contacted by the master node at test start. In this case the IP address of the pod will be the last one listed in the remote server list.

  • Using the Ip address of the remote slave find the pod using :
kubectl get pods -o wide |grep {ip address}

References

https://kubernauts.io

https://jmeter.apache.org/

About

AKS based jmeter testing framework with Grafana reporting

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published