Automated Testing

Automated Unit and Integration Testing

Description

This document describes how to run Eclipse JUnit plugin tests from the command line using Eclipse SDK JUnit test plug-ins and a testing framework. This method of running tests is intended for anyone who wishes to automate their testing process or reproduce the testing mechanism used in Eclipse builds by the Eclipse Platform Release Engineering team.

See the Eclipse Test Framework wiki page for a description of the testing framework in general, and how it can be used outside of the setup described here.

If you simply wish to write and run JUnit tests interactively from an Eclipse workbench, try one of the links below:

JUnit Support in Eclipse

Testing Eclipse Plug-ins with JUnit

Requirements

1. All test machines will require 1.5 and 1.6 java runtime environments installed to run all tests. See #testtargets for details on which tests require a 1.5 level jre. All other tests can be executed with either jre levels.
2. A properties file containing the following key-value pair: J2SE-5.0=<path to 1.5 java executable>. On Windows, the path must be enclosed with double quotes if it contains any space characters, e.g. J2SE-5.0="c:/program files/jdk1.5.0/jre/bin/java".
3. Info-ZIP UnZip version 5.41 or later (or equivalent zip/unzip programs) installed and added to the path.
4. The JUnit Plugin Tests eclipse-Automated-Tests-.zip from an Eclipse build. This ZIP contains the scripts to run the tests, the test framework plug-ins and the JUnit test plug-ins.
5. An Eclipse SDK distribution to test. Usually you will want to test a particular Eclipse build using a matching Automated Tests feature.
6. To run the pde build tests you will need the eclipse-delta-pack.zip from the Eclipse build.
7. To run the Team CVS tests you will need extssh access to a test CVS repository. The tests create a lot of junk modules and tags so you will want to keep the test repository separate from your development repository. While it is not necessary to clean up the repository between each Team test run you will probably want to clean it up periodically.
8. Some of the p2 tests require a copy of the Eclipse Platform Runtime Binary archive for the platform you are testing. More information is available from the Equinox Testing Wiki page.

Setup

1. Extract the eclipse-Automated-Tests-.zip or tar.gz. On Window, extract this file close to the root of the drive to prevent some path names from exceeding the 255 character limit. This will create a top-level directory eclipse-testing.
2. Place the Eclipse SDK distribution to test (and the Eclipse Platform Runtime Binary) in the eclipse-testing directory. Do not extract these files as the test scripts will take care of this.
3. To Run the PDE build scripts, save the eclipse-delta-pack.zip to the eclipse-testing directory.

Running Tests

From a command prompt or terminal:

1. cd to the eclipse-testing directory
2. [Until is fixed ...] Pick a platform specific testing.properties file from production/testScripts/configuration/sdk.tests/testConfigs and copy it here.
3. Execute the following command:

*runtests -properties [-os ] [-ws ] [-arch ] [-noclean] [-vm ][-Dtest.target=performance][*targetname1 targetname2 targetname3 etc...''] ''

All parameters and targets are optional on win32 systems with the exception of "-properties". On *nix systems, the os, ws and arch parameters must be specified in addition to the -properties parameter.

To remote-debug test runs, add this line to the properties files, run the tests, and then use Debug > Remote Java Application to attach the debugger to port 51234:

extraVMargs=-agentlib:jdwp=transport=dt_socket,suspend=y,server=y,address=localhost:51234

The JUnit test results are produced in individual xml files for each test plug-in with a test.xml. These results are copied to the eclipse-testing\results\xml directory. These results can be converted into html by running the "genHtml" target (if they are not automatically converted).

Please be aware that "Build Successful" printed to the console is NOT indicative of tests passing.

Parameters

	Parameter	Description
-os	The OS the tests are being run on. Such as, aix, hpux, linux, macosx, solaris, win32. Default win32
-ws	The windowing system the tests are being run on. Such as, cocoa, gtk, win32. Default win32
-arch	The architecture the tests are being run on. Such as, x86, x86_64, ppc, sparc. Default x86
-noclean	Run tests without installing a clean Eclipse (in "test-eclipse") between test targets. Default is to delete an existing test-eclipse directory and re-extract the Eclipse distribution between test targets.
-properties	As per requirements noted above, this is used to reference a properties file containing additional Ant properties used in running tests.Can be used to pass additional vm arguments to the Java virtual machine running the tests by adding the entry "extraVMargs=" to the specified properties file. At a minimum, the property file must include a "J2SE-5.0" property which specifies a path to a 1.5 java executable, for example: J2SE-5.0="c:/program files/jdk1.5.0_06/jre/bin/java" The path must be enclosed in double quotes if it contains spaces. This key-value pair is required to run JUnit plug-in tests that require a 1.5 level jvm.
-vm	When runtests is executed, two instances of eclipse are started: one to invoke the org.eclipse.ant.core.antRunner application to run a test.xml script in a test plug-in (outer instance) and another in which the JUnit tests are being executed (inner). By default, both will use the java executable on the system path. The -vm property is used to specify an alternate java executable. For example, "-vm c:\jdk1.7\jre\bin\java". However, the -vm value is ignored by the inner vm for JUnit plug-in tests requiring a 1.5 JRE to execute (see #testtargets). This property must be used in conjunction with the -properties parameter described above at all times.
-Dtest.target=performance	Runs the performance target only in each test plug-in's test.xml. When running tests on Windows, enclose this parameter in double quotes, such as, runtests "-Dtest.target=performance".
targetname1 targetname2 targetname3 etc..	Replace with a list of #testtargets to execute (space delineated). Each corresponds to an Ant target in the eclipse-testing/test.xml script. For example, runtests -os linux -ws gtk -arch x86 ant antui -properties vm.properties This will execute the "ant" and "antui" test targets only. If not specified, the default "all" target will be executed.

Appendix

Appendix A: Test Targets

Ant targets in the overall, main test.xml which execute specific test suites. Typically each corresponds to the execution of a test plug-in's test.xml. In addition to specific test suites, there are some targets that define a group of tests, for example quickTests is a set that runs relatively quickly whereas longRunningTests defines a set of tests that can take many hours to run.

All of these, groups and specific test suites, should be viewed as examples at a point in time, as the exact list changes over time, and the overall, main test.xml file has the definitive list.

	Target name (special jvm requirements)	Description / test plug-in
all (1.5 and 1.6 JREs required)	Runs all the targets below. Default.
ant	org.eclipse.ant.tests.core
antui	org.eclipse.ant.tests.ui
compare	org.eclipse.compare.tests
coreexpressions	org.eclipse.core.expressions.tests
coreresources	org.eclipse.core.tests.resources
coreruntime	org.eclipse.core.tests.runtime
coretestsnet	org.eclipse.core.tests.net
equinoxsecurity	org.eclipse.equinox.security.tests
filebuffers	org.eclipse.core.filebuffers.tests
jdtapt (JRE 1.5 required)	org.eclipse.jdt.apt.tests
jdtaptpluggable (JRE 1.6 required)	org.eclipse.jdt.apt.pluggable.tests
jdtcompilerapt (JRE 1.6 required)	org.eclipse.jdt.compiler.apt.tests
jdtcompilertool (JRE 1.6 required)	org.eclipse.jdt.compiler.tool.tests
jdtcorebuilder (JRE 1.5 required)	org.eclipse.jdt.core.tests.builder
jdtcorecompiler (JRE 1.5 required)	org.eclipse.jdt.core.tests.compiler
jdtcoremodel (JRE 1.5 required)	org.eclipse.jdt.core.tests.model
jdtcoreperf	org.eclipse.jdt.core.tests.performance
jdtdebug	org.eclipse.jdt.debug.tests
jdttext	org.eclipse.jdt.text.tests
jdtui	org.eclipse.jdt.ui.tests
jdtuirefactoring	org.eclipse.jdt.ui.tests.refactoring
jface	org.eclipse.jface.text.tests
jfacedatabinding	org.eclipse.jface.tests.databinding
ltkcorerefactoringtests	org.eclipse.ltk.core.refactoring.tests
ltkuirefactoringtests	org.eclipse.ltk.ui.refactoring.tests
osgi	org.eclipse.osgi.tests
pdeapitooling (JRE 1.5 required)	org.eclipse.pde.api.tools.tests
pdebuild	org.eclipse.pde.build.tests
pdeui	org.eclipse.pde.ui.tests
relEng	org.eclipse.releng.tests
swt	org.eclipse.swt.tests
teamcore	org.eclipse.team.tests.core
teamcvs	org.eclipse.team.tests.cvs.core
text	org.eclipse.text.tests
ua	org.eclipse.ua.tests
ui	org.eclipse.ui.tests
uieditors	org.eclipse.ui.editors.tests
uiforms	org.eclipse.ui.tests.forms
uinavigator	org.eclipse.ui.tests.navigator
uiperformance	org.eclipse.ui.tests.performance
uircp	org.eclipse.ui.tests.rcp
uiviews	org.eclipse.ui.tests.views.properties.tabbed
uiworkbenchtexteditor	org.eclipse.ui.workbench.texteditor.tests

Appendix B: Supported os, ws, and arch value combinations

Technically, any supported platform can be tested, but some are supported "out of the box" and others would require a change to the overall, main test.xml file. The following table contains the more common values that can be used with the -os, -ws, and -arch switches, but the overall, main test.xml file has the most comprehensive list for any point in time.

	Operating System (os)	Windowing System (ws)	Architecture (arch)
linux	gtk	x86
linux	gtk	x86_64
macosx	cocoa	x86_64
solaris	gtk	sparc
win32	win32	x86
win32	win32	x86_64