Name	Name	Last commit message	Last commit date
parent directory ..
res	res
README.md	README.md
pythondoc.html	pythondoc.html

Python Code Coverage

The following document goes through running code coverage for python, automation with Jenkins and integration with SonarQube. The code coverage tool to be used for Python is coverage.py

This document is separated into 3 parts:

Manual Coverage - section will present you the supported method of running code coverage on python.
Jenkins Automation - section will introduce the basic steps in order to automate the coverage process using the web UI.
SonarQube Integration - section will teach you how to publish your results to SonarQube using the Jenkins web UI as well as manually.

⚔ Note: Additional advanced topics such as covering services for integration tests or JJB configuration are covered at the bottom of the document under the Advanced Topics section.

Python Code Coverage

Manual Coverage

Prerequisites

python 2.7+
coverage.py

⚔ Note: notice there are several methods and tools to run code coverage such as nose, pytest, pyunit as well as, testools. we will only introduce one way of going about it which we have concluded to be the most suitable and straightforward for this process.

Running code coverage manually

running code coverage coverage run <somefilename.py>

⚔ Note: you can focus your coverage to specific parts of the source by using the --include, --source and --omit flags interchangeably. Failing to add this parameter might yield running coverage against every used package including system packages. See Specifying source files for details.
outputting the coverage data to the screen after a successful code coverage run coverage report
exporting the report into an XML file after a successful code coverage run coverage xml

⚔ Note: The generated report will be in Cobertura format as required, in order to be processed by SonarQube.

Example

The following example includes encountering a failure and a successful run. Let's assume you have the following 2 python files:

a.py

def hello():
  return "Hello World"

def world():
  return hello() + ", I'ts me!"

main.py

import unittest
from a import world

class TestA(unittest.TestCase):

    def test_upper(self):
        self.assertEqual(world(), "Hello World, It's me!")

unittest.main()

⚔ Note: unittest or nose are required in order to run these tests.

run the command coverage run main.py

 F
 ======================================================================
 FAIL: test_upper (__main__.TestA)
 ----------------------------------------------------------------------
 Traceback (most recent call last):
  File "main.py", line 8, in test_upper
    self.assertEqual(world(), "Hello World, It's me!")
 AssertionError: "Hello World, I'ts me!" != "Hello World, It's me!"

 ----------------------------------------------------------------------
 Ran 1 test in 0.000s

 FAILED (failures=1)

as you can see, we've encountered an error due to an issue that arised in our tests!

⚔ Note: this issue is already resolved in the testing repository in order to simplify tthe implementation if this example.

we can see the following indicators:

F - for a failed test
E - for an error occurring during a test
dot(.) - for a successful test

Now that we've fixed the problem, you can see our tests running successfully!

.
--------------------------------------------------------------------------------

Ran 1 test in 0.000s

OK

you can now run the command coverage report which would output the coverage data collected like so:

 Name      Stmts   Miss  Cover
 -----------------------------
 a.py          4      0   100%
 main.py       6      0   100%
 -----------------------------
 TOTAL        10      0   100%

finally, you can export this report into an XML file by running the command coverage xml, creating the file coverage.xml in the current working directory.

Jenkins Automation

Prerequisites

Jenkins v1.653+
Cobertura plugin v1.9.8+
testing environment running:
- fedora v23
- Java v8+
- git
⚔ Note: you could either use a provisioning system such as docker daemon, OpenShift, OpenStack, Kubernetes, etc. or use a local environment.

⚔ Note: notice there are several methods and tools to manage CI and automation such as Gump, Hudson, Bamboo, Travis and more. We will only introduce and support Jenkins for this end as it is the standardized CI tool in RedHat.

Automating using the web UI

Continuing from the previous chapter, assuming our project files are held on a remote github repository https://github.com/RedHatQE/CodeQuality/tree/master/examples/python-test-repo.

Example

in the main Jenkins page, click to New Item button to create a new job
name your job, select the Freestyle Project radio button and save the new job
on the newly opened screen, scroll down and create a new bash script build step
paste the following deployment script onto the bash text editor
```
 # install deployment requirements
 dnf install -y python-devel.x86_64
 pip install coverage unittest2

 # fetch your codebase into your testing environment
 git clone https://github.com/RedHatQE/CodeQuality ${WORKSPACE}/some-project

 # run the coverage tests and export the results xml
 cd ${WORKSPACE}/some-project/examples/python-test-repo
 coverage run --source . main.py
 coverage xml
```
⚔ Note: the -y parameter in the dnf command approves installation prompts which is mandatory for automation purposes.

⚔ Note: the ${WORKSPACE} environment variable is used by Jenkins in order to point to the current build's working directory

let's have a look for a moment at our script, we can see it's divided into 3 main parts:
- installation of prerequisites
- fetching the code base
- running our tests with coverage to create a report (as seen on the previous chapter)
  
  ⚔ Note: in most cases, each of these parts will be more complicated and it's a good habit to break each part into it's own bash build step to ease troubleshooting
run a build of our newly created job

And we're done! on the next chapter you will learn how to publish your generated results into SonarQube to view them.

Uploading coverage results to Jenkins

Sometimes it's useful to have your coverage results uploaded to your Jenkins job which could ease troubleshooting procceses in case of large scale development efforts which might require several independant coverage jobs. For that purpose, we will use the Jenkins Cobertura plugin in order to preview this results in our job's web UI.

Example

Continuing from the previous section, assuming our newly created job has generated a coverage report onto ${WORKSPACE}/some-project/coverage.xml.

in the job's configuration screen, add a post-build action to publish to Cobertura
input a relative glob path to the generated report path and save the job
```
 **/some-project/coverage.xml
```
after rerunning our job you will be able to view the report's preview in your job's main screen

SonarQube Integration

Prerequisites

⚔ Note: in order to deploy a SonarQube instance, you can refer to the Installing Sonar Server v6.0.7 document

⚔ Note: for Jenkins Sonar plugin configuration see Analyzing with SonarQube Scanner for Jenkins for details.

Integrating SonarQube through the Jenkins web UI

As a direct continuation of the previous chapter, building on the same Jenkins job, we'll now add the SonarQube integration.

Example

in the job configuration, add a sonar runner builder

paste your sonar parameters onto the text editor and save the job

Now let's have a look at these parameters:

 # projectKey (string): SonarQube project identification key (unique)
 sonar.projectKey=some-project

 # projectName (string): SonarQube project name (NOT unique)
 sonar.projectName=Some Project

 # projectVersion (decimal): The analyzed project version (unique)
 sonar.projectVersion=1.0

 # sources (string): source code home directory
 sonar.sources=${WORKSPACE}/some-project

 # projectBaseDir (string): project home directory (same as sources)
 sonar.projectBaseDir=${WORKSPACE}/some-project

 # python.coverage.reportPath (string): relative coverage report file path
 sonar.python.coverage.reportPath=coverage.xml

 # language (string): project language(py)
 sonar.language=py

 # inclusions (string): file inclusion pattern
 sonar.inclusions=**/*.py

 # exclusions (string): file exclusion pattern
 sonar.exclusions=tests/**/*.py

 # ws.timeout (int): optional connection timeout parameter
 sonar.ws.timeout=180

⚔ Note: for further details on SonarQube analysis parameters, see Analysis Parameters.

run a build again to view the reported results

you'd now be able to see a link to the results on the job's page which will lead you to the SonarQube dashboard.

And we are done! you will now have a link to the published SonarQube report dashboard

Publishing to SonarQube manually

Sometimes it's useful to be able to publish our coverage report to SonarQube manually. Although it is not a recommended methodology, we will elaborate upon the needed steps for those ends.

⚔ Note: in this section we assume you are running an up-to-date RedHat distribution(Fedora, CentOS, RHEL)

Example

As a continuation of the previous examples and assuming our generated coverage report is located at /some-project/coverage.xml

⚔ Note: The generated report must be in Cobertura format in order to be processed by SonarQube.

install v2.6+ of SonarRunner, which is the client agent for the SonarQube server by running the following commands

 wget https://github.com/SonarSource/sonar-scanner-cli/releases/download/2.6-rc1/sonar-scanner-2.6-SNAPSHOT.zip
 unzip sonar-scanner-2.6-SNAPSHOT.zip

now, in addition to our previous scanning parameters while publishing to sonar through the Jenkins UI:

 # projectKey (string): SonarQube project identification key (unique)
 sonar.projectKey=some-project

 # projectName (string): SonarQube project name (NOT unique)
 sonar.projectName=Some Project

 # projectVersion (decimal): The analyzed project version (unique)
 sonar.projectVersion=1.0

 # sources (string): source code home directory
 sonar.sources=${WORKSPACE}/some-project

 # projectBaseDir (string): project home directory (same as sources)
 sonar.projectBaseDir=${WORKSPACE}/some-project

 # python.coverage.reportPath (string): relative coverage report file path
 sonar.python.coverage.reportPath=coverage.xml

 # language (string): project language(py)
 sonar.language=py

 # inclusions (string): file inclusion pattern
 sonar.inclusions=**/*.py

 # exclusions (string): file exclusion pattern
 sonar.exclusions=tests/**/*.py

 # ws.timeout (int): optional connection timeout parameter
 sonar.ws.timeout=180

we will now also include the SonarServer URL, in this example we are using the CentralCI test-lab instance:

 # host.url (string): the URL pointing to the SonarServer instance
 sonar.host.url=http://sonar_server_address:9000

all together, our final command should look as follows:

 sonar-scanner-2.6-SNAPSHOT/bin/sonar-scanner -X -e\
     -Dsonar.host.url=http://sonar_server_address\
     -Dsonar.projectKey=some-project\
     "-Dsonar.projectName=Some Project"\
     -Dsonar.projectVersion=1.0\
     -Dsonar.sources=${WORKSPACE}/some-project\
     -Dsonar.projectBaseDir=${WORKSPACE}/some-project\
     -Dsonar.python.coverage.reportPath=coverage.xml\
     -Dsonar.language=py\
     "-Dsonar.inclusions=**/*.py"\
     "-Dsonar.exclusions=tests/**/*.py"\
     -Dsonar.ws.timeout=180\

⚔ Note: we have parenthesized parameters which include white-space and special characters

⚔ Note: the -X -e flags are used to verbose and prompt runtime issues with the SonarScanner

finally, you should be able to see a success prompt with a link to your published coverage report dashboard such as this one:

 DEBUG: Upload report
 DEBUG: POST 200 http://sonar_server_address/api/ce/submit?projectKey=some-project&projectName=Some%20Project | time=34ms
 INFO: Analysis report uploaded in 41ms
 INFO: ANALYSIS SUCCESSFUL, you can browse http://sonar_server_address/dashboard/index/some-project
 INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
 INFO: More about the report processing at http://sonar_server_address/api/ce/task?id=AVpaB5_70YnVK7Pmb1mm
 DEBUG: Report metadata written to /some-project/.sonar/report-task.txt
 DEBUG: Post-jobs :
 INFO: ------------------------------------------------------------------------
 INFO: EXECUTION SUCCESS
 INFO: ------------------------------------------------------------------------
 INFO: Total time: 8.900s
 INFO: Final Memory: 51M/235M
 INFO: ------------------------------------------------------------------------

and your results have been published! (:

Advanced Topics

Covering services

In order to run coverage over a service or a process not launched manually from it's entry-point i.e multi-service products, API integration tests, multi-host integration tests, etc.
We are proposing the following solution, which inserts a pipeline hook to each python process.

⚔ Note: This process has been tested and is supporting bot multi-processing as well as multi-threading

Prerequisites

python 2.7+
coverage.py

Implementation steps

In order to run coverage over a service you must first configure your environment as follows.

create a file called sitecustomize.py at the site-packages directory, commonly located in /usr/lib/python2.7, containing:
```
 import coverage
 coverage.process_startup()
```
⚔ Note: This methodology uses the site module. In order to list it's paths, run python -m site. For more details see Site-wide Configuration.
create a .coveragerc configuration file wherever you'd like, containing:
```
 [run]
 source={source files directory path}
 data_file={results directory path}/.coverage
 parallel=true
 concurrency=multiprocessing

 [xml]
 output={results directory path}/coverage.xml
```
Notice this ini file is divided into coverage run parameters and report relevant parameters. other then setting the source and output directories, we've also set the parallel and concurrency parameters.
- parallel: whether to set a unique name for each generated report in cases of parallelism
- concurrency: which concurrency library is used
  
  ⚔ Note: We've used the multiprocessing module for concurrency, within our application and this parameter should be adapted for you scenario. For more information, see Configuration Files.
  
  ⚔ Warning: Not setting source might result in a system-wide python process coverage.

create a results folder

After this configuration, you should perform the following actions:

set the environment variable COVERAGE_PROCESS_START to the .coveragerc file we created
```
 export COVERAGE_PROCESS_START={path to .coveragerc file}
```
Before each process is being executed, the COVERAGE_PROCESS_START will be sampled. If the variable is set to a configuration file then these settings will be used for the coverage, otherwise, the default configuration will be used.

⚔ Note: You could also set COVERAGE_PROCESS_START in your .bashrc if you wish for coverage to run contentiously.
as mentioned above, this hook is ran before the process is launched, hence we need to restart the service
```
systemctl restart {service name}
```
after the service have shut down gracefully, you should be able to find your coverage results in the results folder. go to the results directory and combine the results generated into a single report
```
cd {results directory path}
coverage combine
```
⚔ Note: This step is only relevant for cases using parallelism, single-process cases will only have a single file within the results folder.

You should now have a .coverage report file in your results directory and we are done!

Jenkinsfile

Starting with Jenkins 2, automation configuration can mainitained using a Jenkinsfile which levrages the power of Grooveyscript to describe a jenkins job.

Prerequisites

⚔ Note: For more details on the Jenkinsfile format, see Using a Jenkinsfile

Example

The following file illustrates a possible Jenkinsfile configuration

pipeline {
    agent { node { label 'sonarqube-upshift' } }
    options {
      skipDefaultCheckout true
    }
    triggers {
      cron('0 8 * * *')
    }
    stages {
        stage('Deploy') {
            steps {
                // clone project
                git url: 'https://github.com/RedHatQE/CodeQuality.git'
            }
        }
        stage('Analyse') {
            steps {
                dir('examples/python-test-repo') {
                // run tests with coverage and export results to xml
                sh 'coverage run --source . main.py'
                sh 'coverage xml'
                }
            }
        }
        stage('Report') {
            /*
            sonar runner parameters, set sources and baseDir to project home
            ========================

            projectKey (string): SonarQube project identification key (unique)
            projectName (string): SonarQube project name (NOT unique)
            projectVersion (string): SonarQube project version (unique)
            sources (string): source code home directory
            projectBaseDir (string): project home directory (same as sources)
            python.coverage (string): relative xml coverage report path
            language (string): project language(py)
            inclusions (string): file inclusion pattern
            exclusions (string): file exclusion pattern
            login (string): SonarQube server user name
            password (string): SonarQube server user password
             */
            steps {
              writeFile file: "${pwd()}/sonar-project.properties", text: """
              sonar.projectKey=test-files_1_0_python_full-analysis
              sonar.projectName=Python Testfiles
              sonar.projectVersion=1.0
              sonar.sources=${pwd()}/examples/python-test-repo
              sonar.projectBaseDir=${pwd()}/examples/python-test-repo
              sonar.python.coverage.reportPath=${pwd()}/examples/python-test-repo/coverage.xml
              sonar.language=py
              sonar.inclusions=**/*.py
              sonar.exclusions=tests/**/*.py
              sonar.login=test
              sonar.password=test
              sonar.ws.timeout=180
              """

              // initite pre-configured sonar scanner tool on project
              // 'sonarqube_prod' is our cnfigured tool name, see yours
              // in the Jenkins tool configuration
              withSonarQubeEnv('sonarqube_prod') {
                sh "${tool 'sonar-scanner-2.8'}/bin/sonar-scanner"

              }
            }
        }
    }
}

Jenkins Job Builder

Jenkins automation configuration can also be done by using the Jenkins Job builder (JJB) which takes simple descriptions of Jenkins jobs in YAML or JSON format and uses them to configure Jenkins.

Prerequisites

⚔ Note: For more details on installing and configuring JJB, see the Quick Start Guide

Example

The following file illustrates a possible JJB configuration

- job:
    name: sonarqube_python_analysis


    #######################################################
    ############## SonarQube Parameters ###################
    #######################################################

    # sonarqube project parameters, set before build
    parameters:
      - string:
          name: SONAR_KEY
          default: sonarqube_python_analysis
          description: "SonarQube unique project key"
      - string:
          name: SONAR_NAME
          default: Python Analysis
          description: "SonarQube project name"
      - string:
          name: SONAR_PROJECT_VERSION
          default: "1.0"
          description: "SonarQube project version"

    #######################################################
    ############### Logging Aggregation ###################
    #######################################################

    # define how many days to kee build information
    properties:
      - build-discarder:
          days-to-keep: 60
          num-to-keep: 200
          artifact-days-to-keep: 60
          artifact-num-to-keep: 200

    #######################################################
    ################### Slave Image #######################
    #######################################################

    node: sonarqube-upshift

    #######################################################
    ################ Git Trigger Config ###################
    #######################################################

    # git repo to follow, skip-tag to not require auth
    scm:
      - git:
          url: https://github.com/RedHatQE/CodeQuality.git
          skip-tag: true

    # git polling trigger set to once an hour
    triggers:
      - pollscm:
          cron: "0 0 * * 0"
          ignore-post-commit-hooks: True

    #######################################################
    ################### Build Steps #######################
    #######################################################

    builders:

      # coverage tests initialization script
      - shell: |
          cd examples/python-test-repo
          coverage run --source . main.py
          coverage xml

      # sonar runner parameters, set sources and baseDir to project home
      # projectKey (string): SonarQube project identification key (unique)
      # projectName (string): SonarQube project name (NOT unique)
      # projectVersion (string): SonarQube project version (unique)
      # sources (string): source code home directory
      # projectBaseDir (string): project home directory (same as sources)
      # language (string): project language(ruby)
      # inclusions (string): file inclusion pattern
      # exclusions (string): file exclusion pattern
      # login (string): SonarQube server user name
      # password (string): SonarQube server user password
      - sonar:
          sonar-name: sonarqube_prod
          properties: |
            sonar.projectKey=$SONAR_KEY
            sonar.projectName=$SONAR_NAME
            sonar.projectVersion=$SONAR_PROJECT_VERSION
            sonar.sources=${WORKSPACE}/examples/python-test-repo
            sonar.projectBaseDir=${WORKSPACE}/examples/python-test-repo
            sonar.python.coverage.reportPath=coverage.xml
            sonar.language=py
            sonar.inclusions=**/*.py
            sonar.exclusions=tests/**/*.py
            sonar.login=test
            sonar.password=test
            sonar.ws.timeout=180

Job DSL

Example

def jobName = 'python-coverage-dsl-sample'
def giturl = 'https://github.com/RedHatQE/CodeQuality.git'
def sonarProperties = '''
    sonar.projectKey=sonarqube_python_analysis
    sonar.projectName=Python Analysis
    sonar.projectVersion=1.0
    sonar.sources=${WORKSPACE}/examples/python-test-repo
    sonar.projectBaseDir=${WORKSPACE}/examples/python-test-repo
    sonar.python.coverage.reportPath=coverage.xml
    sonar.language=py
    sonar.inclusions=**/*.py
    sonar.exclusions=tests/**/*.py
    sonar.login=test
    sonar.password=test
    sonar.ws.timeout=180
       '''.stripIndent()


job(jobName) {
    label('sonarqube-upshift')
    scm {
        git(giturl)
    }
    triggers {
        cron '0 8 * * *'
    }
    steps {
        shell '''
           cd examples/python-test-repo
           coverage run --source . main.py
           coverage xml
        '''
    }
    configure {
        it / 'builders' << 'hudson.plugins.sonar.SonarRunnerBuilder' {
            properties ("$sonarProperties")
    }
  }
}

Code Coverage With Integration Tests

Reference Docs:

How to generate coverage report for http based integration tests?

Requirements

We can implement the solution using the next steps:

Run the Server under Coverage mode.
Run the tests.
Ensure the Server coverage is written to file.
Read the coverage from this file and append it to the tests coverage report.

Http Server Example

HttpServer

The http_server.py file creates a simple http server that respond "Hello World" page on a GET requests.

# http_server.py
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer


class DummyHandler(BaseHTTPRequestHandler):

    def do_GET(self):
        self.send_response(200)
        self.send_header('Content-Type', 'text/html')
        self.end_headers()
        self.wfile.write('<html><body><h1>Hello World</h1></body></html>'.encode())


if __name__ == '__main__':
    HTTPServer(('127.0.0.1', 7000), DummyHandler).serve_forever()

test

A simple test that makes an HTTP request and verifies the response contains "Hello World":

# tests/test_httpserver.py
import requests


def test_get():
    respond = requests.get('http://127.0.0.1:7000')
    respond.raise_for_status()
    assert 'Hello World' in respond.text

Solution

Below it is a conftest.py file located under "tests" folder and from it we are running the server with a slightly modified environment using os.environ.copy() and Subprocess management. We are also reading the coverage data from a file using the python api coverage.data.CoverageData and appending it using a fixture provided by pytest-cov called cov. We marked the above four steps in the file below.

# tests/conftest.py
import os
import signal
import subprocess
import time
import coverage.data
import pytest



@pytest.fixture(autouse=True)
def run_server(cov):
    # 1.
    server_env = os.environ.copy()
    server_env['COVERAGE_FILE'] = '.coverage.http_server'
    serverprocess = subprocess.Popen(['coverage', 'run', 'http_server.py'],
                                     env=server_env,
                                     stdout=subprocess.PIPE,
                                     stderr=subprocess.PIPE,
                                     preexec_fn=os.setsid)
    time.sleep(3)
    yield  # 2.
    # 3.
    serverprocess.send_signal(signal.SIGINT)
    time.sleep(1)
    # 4.
    server_cov = coverage.data.CoverageData()
    with open('.coverage.http_server') as cov_f:
        server_cov.read_fileobj(cov_f)
    cov.data.update(server_cov)

Now we are Running the tests and adding the coverage of the httpServer.py to the overall coverage, although only tests selected. The --cov define which folder we will cover and "--cov-report term" tell pytest to output the coverage report to the terminal (we could choose xml or http format also):

$ pytest --cov=tests --cov-report term -vs
=============================== test session starts ===============================
platform linux2 -- Python 2.7.5, pytest-4.4.1, py-1.8.0, pluggy-0.9.0 -- /usr/bin/python2
cachedir: .pytest_cache
rootdir: /tmp/py_integ
plugins: cov-2.6.1
collected 1 item

tests/test_httpserver.py::test_get PASSED

---------- coverage: platform linux2, python 2.7.5-final-0 -----------
Name                       Stmts   Miss  Cover
----------------------------------------------
http_server.py                 9      0   100%
tests/conftest.py             18      0   100%
tests/test_httpserver.py       5      0   100%
----------------------------------------------
TOTAL                         32      0   100%


============================ 1 passed in 5.09 seconds =============================

Files

python

Directory actions

More options

Directory actions

More options

Latest commit

History

python

Folders and files

parent directory

README.md

Python Code Coverage

Table of Contents

Manual Coverage

Prerequisites

Running code coverage manually

Example

Jenkins Automation

Prerequisites

Automating using the web UI

Example

Uploading coverage results to Jenkins

Example

SonarQube Integration

Prerequisites

Integrating SonarQube through the Jenkins web UI

Example

Publishing to SonarQube manually

Example

Advanced Topics

Covering services

Prerequisites

Implementation steps

Jenkinsfile

Prerequisites

Example

Jenkins Job Builder

Prerequisites

Example

Job DSL

Example

Code Coverage With Integration Tests

Requirements

Http Server Example

HttpServer

test

Solution