The following document goes through running code coverage for c/c++, automation with Jenkins and integration with SonarQube. The code coverage tool to be used for c/c++ is gcov. Formatting and aggregation is managed with gcovr.
This document is separated into 3 parts:
- Manual Coverage - section will present you the supported method of running code coverage on python.
- Jenkins Automation - section will introduce the basic steps in order to automate the coverage process using the web UI.
- SonarQube Integration - section will teach you how to publish your results to SonarQube using the Jenkins web UI as well as manually.
⚔ Note: Additional advanced topics such as covering services for integration tests or JJB configuration are covered at the bottom of the document under the Advanced Topics section.
- C/C++ Code Coverage
- python 2.7+
- g++ (GCC) v6.3.1+
- gcov v6.3.1+ (supplied with GCC packages)
- gcovr v3.3+
⚔ Note: notice there are several methods and tools to run code coverage such as BullseyeCoverage, GCT and COVTOOL. we will only introduce one way of going about it which we have concluded to be the most suitable and straightforward for this process.
-
compile your project including the mapping and coverage flags
g++ -g --coverage <project files> -o <output file>⚔ Note: we included the -g mapping flag which adds line numbers to the output and the --coverage flag which instruments the output. If you are compiling C using gcc, you'd use -fprofile-arcs -ftest-coverage instead of --coverage.
-
run the compiled executable in order for the instrumented program to generate run-time logs
-
generate coverage reports from the run-time logs using gcov
gcov -o <reports target directory> <source files directory>⚔ Note: in most cases, both the reports and source file directory should be the same directory.
-
aggregate generated file coverage report into a single project coverage XML report using gcovr
gcovr -r <source files root directory> -x --object-directory=<gcov reports directory> > report.xml⚔ Note: we included the -r flag to signify the following path is the root path and the -x flag to indicate the expected output is XML.
⚔ Note: The generated report will be in Cobertura format as required, in order to be processed by SonarQube.
Let's assume you have the following C++ files:
-
main.cpp
#include "main.h" /** * our basic function * @param name the name of our friend * @return greeting for our friend */ string testMe(string name) { return "Hello " + name + "!"; }
-
main.h
#ifndef TESTCPP_MAIN_H #define TESTCPP_MAIN_H #include <iostream> using namespace std; // declaring our tested function string testMe(string name); #endif //TESTCPP_MAIN_H
-
tests.cpp
#include <cppunit/TestCase.h> #include <cppunit/ui/text/TextTestRunner.h> #include <cppunit/extensions/HelperMacros.h> #include <cppunit/TestResult.h> #include <cppunit/TestResultCollector.h> #include <cppunit/BriefTestProgressListener.h> #include <cppunit/CompilerOutputter.h> #include <cppunit/XmlOutputter.h> #include "main.h" using namespace CppUnit; class HelloWorldTest : public CppUnit::TestFixture { public: void test() { CPPUNIT_ASSERT( testMe("Bob").compare("Hello Bob!") == 0 ); } }; int main(int argc, char* argv[]) { CppUnit::TestCaller<HelloWorldTest> test( "test", &HelloWorldTest::test ); CppUnit::TestResult result; test.run( &result ); }
⚔ Note: CppUnit is required in order to run these tests.
-
run the command
g++ -g --coverage -lcppunit *.cpp -o testcppto compile the project with coverage and line mapping⚔ Note: we linked CppUnit statically using -lcppunit.
-
run our compiled executable
./testcppin order to generate our run-time .gcda/.gcno log files -
execute gcov by running
gcov -o $(pwd) $(pwd)in order to generate file coverage reports from our logging files -
finally, run gcovr with
gcovr -r $(pwd) -x --object-directory=$(pwd) > report.xmlin order to aggregate our generated file coverage reports into a singular XML report.
you can now find our generated report.xml report file in the working directory and we are done! (:
-
testing environment running:
- fedora v23
- Java v8+
- git
⚔ Note: you could either use a provisioning system such as docker daemon, OpenShift, OpenStack, Kubernetes, etc. or use a local environment.
⚔ Note: notice there are several methods and tools to manage CI and automation such as Gump, Hudson, Bamboo, Travis and more. We will only introduce and support Jenkins for this end as it is the standardized CI tool in RedHat.
Continuing from the previous chapter, assuming our project files are held on a remote github repository https://github.com/RedHatQE/CodeQuality.git.
-
in the main Jenkins page, click to
New Itembutton to create a new job
-
name your job, select the
Freestyle Projectradio button and save the new job
-
on the newly opened screen, set the SCM to git and paste in our repository
plain https://github.com/RedHatQE/CodeQuality.git
-
scroll down and create a new bash script build step
-
paste the following deployment script onto the bash text editor
# install deployment requirements dnf install -y gcc-c++ cppunit-devel.x86_64 cppunit.x86_64 unzip wget python-devel pip install gcovr # compile test files with coverage and mapping flags cd ${WORKSPACE}/examples/cpp-test-repo g++ -g --coverage -lcppunit *.cpp -o testcpp # generate runtime coverage metrics report ./testcpp gcov -o ${WORKSPACE} ${WORKSPACE} # aggregate generated reports to xml report gcovr -r ${WORKSPACE} -x --object-directory=${WORKSPACE} > report.xml
⚔ Note: the -y parameter in the dnf command approves installation prompts which is mandatory for automation purposes.
⚔ Note: the ${WORKSPACE} environment variable is used by Jenkins in order to point to the current build's working directory
let's have a look for a moment at our script, we can see it's divided into 3 main parts:
- installation of prerequisites
- running our tests with coverage to create file reports (as seen on the previous chapter)
- generating an aggregated report file
⚔ Note: in most cases, each of these parts will be more complicated and it's a good habit to break each part into it's own bash build step to ease troubleshooting
-
run a build of our newly created job
And we're done! on the next chapter you will learn how to publish your generated results into SonarQube to view them.
Sometimes it's useful to have your coverage results uploaded to your Jenkins job which could ease troubleshooting processes in case of large scale development efforts which might require several independent coverage jobs. For that purpose, we will use the Jenkins Cobertura plugin in order to preview this results in our job's web UI.
Continuing from the previous section, assuming our newly created job has generated a coverage report onto ${WORKSPACE}/report.xml.
-
in the job's configuration screen, add a post-build action to publish to Cobertura
-
input a relative glob path to the generated report path and save the job
**/report.xml
-
after rerunning our job you will be able to view the report's preview in your job's main screen
⚔ Note: for Jenkins Sonar plugin configuration see Analyzing with SonarQube Scanner for Jenkins for details.
As a direct continuation of the previous chapter, building on the same Jenkins job, we'll now add the SonarQube integration.
-
in the job configuration, add a sonar runner builder
-
paste your sonar parameters onto the text editor and save the job
Now let's have a look at these parameters:
# projectKey (string): SonarQube project identification key (unique) sonar.projectKey=some-project # projectName (string): SonarQube project name (NOT unique) sonar.projectName=Some Project # projectVersion (decimal): The analyzed project version (unique) sonar.projectVersion=1.0 # sources (string): source code home directory sonar.sources=${WORKSPACE} # projectBaseDir (string): project home directory (same as sources) sonar.projectBaseDir=${WORKSPACE} # cxx.coverage.reportPath (string): relative coverage report file path sonar.cxx.coverage.reportPath=${WORKSPACE}/report.xml # language (string): project language(c++) sonar.language=c++ # inclusions (string): file inclusion pattern sonar.inclusions=**/*.cpp # ws.timeout (int): optional connection timeout parameter sonar.ws.timeout=180
⚔ Note: for further details on SonarQube analysis parameters, see Analysis Parameters.
-
run a build again to view the reported results
you'd now be able to see a link to the results on the job's page which will lead you to the SonarQube dashboard.
And we are done! you will now have a link to the published SonarQube report dashboard
Sometimes it's useful to be able to publish our coverage report to SonarQube manually. Although it is not a recommended methodology, we will elaborate upon the needed steps for those ends.
⚔ Note: in this section we assume you are running an up-to-date RedHat distribution(Fedora, CentOS, RHEL)
As a continuation of the previous examples and assuming our generated coverage report report.xml is located at in out project's directory.
⚔ Note: The generated report must be in Cobertura format in order to be processed by SonarQube.
-
install v2.6+ of SonarRunner, which is the client agent for the SonarQube server by running the following commands
wget https://github.com/SonarSource/sonar-scanner-cli/releases/download/2.6-rc1/sonar-scanner-2.6-SNAPSHOT.zip unzip sonar-scanner-2.6-SNAPSHOT.zip
-
now, in addition to our previous scanning parameters while publishing to sonar through the Jenkins UI:
# projectKey (string): SonarQube project identification key (unique) sonar.projectKey=some-project # projectName (string): SonarQube project name (NOT unique) sonar.projectName=Some Project # projectVersion (decimal): The analyzed project version (unique) sonar.projectVersion=1.0 # sources (string): source code home directory sonar.sources=${WORKSPACE} # projectBaseDir (string): project home directory (same as sources) sonar.projectBaseDir=${WORKSPACE} # cxx.coverage.reportPath (string): relative coverage report file path sonar.cxx.coverage.reportPath=${WORKSPACE}/report.xml # language (string): project language(c++) sonar.language=c++ # inclusions (string): file inclusion pattern sonar.inclusions=**/*.cpp # ws.timeout (int): optional connection timeout parameter sonar.ws.timeout=180 ```
we will now also include the SonarServer URL, in this example we are using the CentralCI test-lab instance:
```shell
# host.url (string): the URL pointing to the SonarServer instance
sonar.host.url=<sonar-server-url>
```
all together, our final command should look as follows:
```shell
sonar-scanner-2.6-SNAPSHOT/bin/sonar-scanner -X -e\
-Dsonar.host.url=<sonar-server-url>\
-Dsonar.projectKey=some-project\
"-Dsonar.projectName=Some Project"\
-Dsonar.projectVersion=1.0\
-Dsonar.sources=${WORKSPACE}\
-Dsonar.projectBaseDir=${WORKSPACE}\
-Dsonar.cxx.coverage.reportPath=${WORKSPACE}/report.xml\
-Dsonar.language=c++\
"-Dsonar.inclusions=**/*.cpp"\
-Dsonar.ws.timeout=180\
```
> ⚔ Note: we have parenthesized parameters which include white-space and special characters
> ⚔ Note: the **-X -e** flags are used to verbose and prompt runtime issues with the SonarScanner
-
finally, you should be able to see a success prompt with a link to your published coverage report dashboard such as this one:
DEBUG: Upload report DEBUG: POST 200 INFO: Analysis report uploaded in 17ms INFO: ANALYSIS SUCCESSFUL, you can browse <sonar-server-url>/dashboard/index/sonarqube_testfiles_cpp_full_analysis INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report INFO: More about the report processing at <sonar-server-url>/api/ce/task?id=AVyIGhPDpIFAEL-ZWqjg DEBUG: Report metadata written to .sonar/report-task.txt DEBUG: Post-jobs : INFO: ------------------------------------------------------------------------ INFO: EXECUTION SUCCESS INFO: ------------------------------------------------------------------------ INFO: Total time: 6.189s INFO: Final Memory: 48M/184M INFO: ------------------------------------------------------------------------
and your results have been published! (:
Starting with Jenkins 2, automation configuration can mainitained using a Jenkinsfile which levrages the power of Grooveyscript to describe a jenkins job.
⚔ Note: For more details on the Jenkinsfile format, see Using a Jenkinsfile
The following file illustrates a possible Jenkinsfile configuration
pipeline {
agent { node { label 'sonarqube-upshift' } }
options {
skipDefaultCheckout true
}
triggers {
cron('0 8 * * *')
}
stages {
stage('Deploy') {
steps {
// clone project
git url: 'https://github.com/RedHatQE/CodeQuality.git'
}
}
stage('Analyse') {
steps {
dir('examples/cpp-test-repo') {
// compile test files with coverage and mapping flags
sh 'g++ -g --coverage -lcppunit *.cpp -o testcpp'
// generate runtime coverage metrics report
sh './testcpp'
sh 'gcov -o $(pwd) -f $(pwd)/main.cpp'
// aggregate generated reports to xml report
sh 'gcovr -r $(pwd) -x --object-directory=$(pwd) > report.xml'
}
}
}
stage('Report') {
steps {
/*
sonar runner parameters, set sources and baseDir to project home
=======================
projectKey (string): SonarQube project identification key (unique)
projectName (string): SonarQube project name (NOT unique)
projectVersion (string): SonarQube project version (unique)
sources (string): source code home directory
projectBaseDir (string): project home directory (same as sources)
language (string): project language(cpp)
inclusions (string): file inclusion pattern
cxx.coverage.reportPath (string): xml coverage report path
login (string): SonarQube server user name
password (string): SonarQube server user password
*/
writeFile file: "${pwd()}/sonar-project.properties", text: """
sonar.projectKey=test-files_1_0_cpp_full-analysis
sonar.projectName=CPP Testfiles
sonar.projectVersion=1.0
sonar.sources=${pwd()}/examples/cpp-test-repo
sonar.projectBaseDir=${pwd()}/examples/cpp-test-repo
sonar.language=c++
sonar.inclusions=**/*.cpp
sonar.cxx.coverage.reportPath=${pwd()}/examples/cpp-test-repo/report.xml
sonar.login=test
sonar.password=test
sonar.ws.timeout=180
"""
// initite sonar scanner tool on project
// 'sonarqube_prod' is our cnfigured tool name, see yours
// in the Jenkins tool configuration
withSonarQubeEnv('sonarqube_prod') {
sh "${tool 'sonar-scanner-2.8'}/bin/sonar-scanner"
}
}
}
}
}
Jenkins automation configuration can also be done by using the Jenkins Job builder (JJB) which takes simple descriptions of Jenkins jobs in YAML or JSON format and uses them to configure Jenkins.
⚔ Note: For more details on installing and configuring JJB, see the Quick Start Guide
The following file illustrates a possible JJB configuration
- job:
name: sonarqube_cpp_analysis
#######################################################
############## SonarQube Parameters ###################
#######################################################
# sonarqube project parameters, set before build
parameters:
- string:
name: SONAR_KEY
default: sonarqube_cpp_analysis
description: "SonarQube unique project key"
- string:
name: SONAR_NAME
default: Testfiles CPP Analysis
description: "SonarQube project name"
- string:
name: SONAR_PROJECT_VERSION
default: "1.0"
description: "SonarQube project version"
#######################################################
############### Logging Aggregation ###################
#######################################################
# define how many days to kee build information
properties:
- build-discarder:
days-to-keep: 60
num-to-keep: 200
artifact-days-to-keep: 60
artifact-num-to-keep: 200
#######################################################
################### Slave Image #######################
#######################################################
node: sonarqube-upshift
#######################################################
################ Git Trigger Config ###################
#######################################################
# git repo to follow, skip-tag to not require auth
scm:
- git:
url: https://github.com/RedHatQE/CodeQuality.git
skip-tag: true
# git polling trigger set to once an hour
triggers:
- pollscm:
cron: "0 0 * * 0"
ignore-post-commit-hooks: True
#######################################################
################### Build Steps #######################
#######################################################
builders:
# coverage tests initialization script
- shell: |
# compile test files with coverage and mapping flags
cd examples/cpp-test-repo/
g++ -g --coverage -lcppunit *.cpp -o testcpp
# generate runtime coverage metrics report
./testcpp
gcov -o $(pwd) -f $(pwd)/main.cpp
# aggregate generated reports to xml report
gcovr -r $(pwd) -x --object-directory=$(pwd) > report.xml
# sonar runner parameters, set sources and baseDir to project home
# projectKey (string): SonarQube project identification key (unique)
# projectName (string): SonarQube project name (NOT unique)
# projectVersion (string): SonarQube project version (unique)
# sources (string): source code home directory
# projectBaseDir (string): project home directory (same as sources)
# language (string): project language(ruby)
# inclusions (string): file inclusion pattern
# cxx.coverage.reportPath (string): xml coverage report path
# login (string): SonarQube server user name
# password (string): SonarQube server user password
- sonar:
sonar-name: sonarqube_prod
properties: |
sonar.projectKey=$SONAR_KEY
sonar.projectName=$SONAR_NAME
sonar.projectVersion=$SONAR_PROJECT_VERSION
sonar.sources=${WORKSPACE}/examples/cpp-test-repo
sonar.projectBaseDir=${WORKSPACE}/examples/cpp-test-repo
sonar.language=c++
sonar.inclusions=**/*.cpp
sonar.cxx.coverage.reportPath=${WORKSPACE}/examples/cpp-test-repo/report.xml
sonar.login=test
sonar.password=test
sonar.ws.timeout=180
########################################################
################## Report Publisher ####################
########################################################
# publishes aggregated results to Jenkins
publishers:
- cobertura:
report-file: "**/examples/cpp-test-repo/report.xml"
targets:
- line:
healthy: 0
unhealthy: 0
failing: 0def jobName = 'cpp-coverage-dsl-sample'
def giturl = 'https://github.com/RedHatQE/CodeQuality.git'
def sonarProperties = '''
sonar.projectKey=test-files_1_0_cpp_full-analysis
sonar.projectName=CPP Testfiles
sonar.projectVersion=1.0
sonar.sources=${WORKSPACE}/examples/cpp-test-repo
sonar.projectBaseDir=${WORKSPACE}/examples/cpp-test-repo
sonar.language=c++
sonar.inclusions=**/*.cpp
sonar.cxx.coverage.reportPath=${WORKSPACE}/examples/cpp-test-repo/report.xml
sonar.login=test
sonar.password=test
sonar.ws.timeout=180
'''.stripIndent()
job(jobName) {
label('sonarqube-upshift')
scm {
git(giturl)
}
triggers {
cron '0 8 * * *'
}
steps {
shell '''
# compile test files with coverage and mapping flags
cd examples/cpp-test-repo
g++ -g --coverage -lcppunit *.cpp -o testcpp
# generate runtime coverage metrics report
./testcpp
gcov -o $(pwd) -f $(pwd)/main.cpp
# aggregate generated reports to xml report
gcovr -r $(pwd) -x --object-directory=$(pwd) > report.xml
'''
}
configure {
it / 'builders' << 'hudson.plugins.sonar.SonarRunnerBuilder' {
properties ("$sonarProperties")
}
}
}
Reference links:
In order to get coverage report in cpp programs, the only thing you need is: 1. Programs must be compiled using gcc only to use gcov. 2. gcov must be invoked in the same directory as that where the source was compiled.
Below we will see a simple http server example in cpp, we will compile it with coverage ability and get the report after the server stops.
The foo.c file creates a simple http server that respond "Hello World" page on a GET requests.
foo.c file.
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h>
#include <arpa/inet.h>
#include <string.h>
#include <sys/stat.h>
#include <fcntl.h>
char webpage[] =
"HTTP/1.1 200 OK\r\n"
"Content-Type: text/html; charset=UTF-8\r\n\r\n"
"<!DOCTYPE html>\r\n"
"<html><head><title>Shay Test</title></head>\r\n"
"<body><center><h1>Hello World!</h1><br>\r\n"
"</center></body></html>\r\n";
int main(int argc, char *argv[])
{
struct sockaddr_in server_addr, client_addr;
socklen_t sin_len = sizeof(client_addr);
int http_server , http_client;
char buf[2048];
int on = 1;
http_server = socket(AF_INET, SOCK_STREAM, 0);
if(http_server < 0)
{
perror("socket");
exit(1);
}
setsockopt(http_server, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(int));
server_addr.sin_family = AF_INET;
server_addr.sin_addr.s_addr = INADDR_ANY;
server_addr.sin_port = htons(8080);
if(bind(http_server, (struct sockaddr *) &server_addr, sizeof(server_addr)) == -1)
{
perror("bind");
close(http_server);
exit(1);
}
if(listen(http_server, 10) == -1)
{
perror("listen");
close(http_server);
exit(1);
}
while(1)
{
http_client = accept(http_server, (struct sockaddr *) &client_addr, &sin_len);
if(http_client == -1)
{
perror("Connection faild....\n");
continue;
}
printf("Got client connection.....\n");
if (!fork())
{
/* child process */
close(http_server);
memset(buf, 0, 2048);
read(http_client, buf, 2047);
printf("%s\n", buf);
write(http_client, webpage, sizeof(webpage) - 1);
close (http_client);
printf("closing the server...\n");
exit(0);
}
/* parent process */
close(http_client);
}
return 0;
}Now we will complie it with coverage flags and run the executable file:
gcc -fprofile-arcs -ftest-coverage -o foo foo.c
./fooAfter browsing http://localhost:8080 URL and stoping this server, we can run the below command which creates the report gcov file:
gcov -b -c foo.cThe output need to be something like this:
File 'foo.c'
Lines executed:71.43% of 35
Branches executed:100.00% of 10
Taken at least once:60.00% of 10
Calls executed:64.00% of 25
Creating 'foo.c.gcov'If you need an XML report file, you can run this command:
gcovr -x > report.xmlThat's it!